forked from kubernetes-sigs/kubespray
-
Notifications
You must be signed in to change notification settings - Fork 37
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[jjo] add kube-router support (kubernetes-sigs#3339)
* [jjo] add kube-router support Fixes cloudnativelabs/kube-router#147. * add kube-router as another network_plugin choice * support most used kube-router flags via `kube_router_foo` vars as other plugins * implement replacing kube-proxy (--run-service-proxy=true) via `kube_proxy_mode: none`, verified in a _non kubeadm_enabled_ install, should also work for recent kubeadm releases via `skipKubeProxyInstall: true` config * [jjo] address PR#3339 review from @woopstar * add busybox image used by kube-router to downloads * fix busybox download groups key * rework kubeadm_enabled + kube_router_run_service_proxy - verify it working ok w/the kubeadm_enabled and kube_router_run_service_proxy true or false - introduce `kube_proxy_remove` fact, to decouple logic from kube_proxy_mode (which affects kubeadm configmap settings, thus no-good to ab-use it to 'none') * improve kube-router.md re: kubeadm_enabled and kube_router_run_service_proxy * address @woopstar latest review * add inventory/sample/group_vars/k8s-cluster/k8s-net-kube-router.yml * fix kube_router_run_service_proxy conditional for kube-proxy removal * fix kube_proxy_remove fact (w/ |bool), add some needed kube-proxy tags on my and existing changes * update kube-router tolerations for 1.12 compatibility * add PriorityClass to kube-router DaemonSet
- Loading branch information
1 parent
c33e08c
commit a5edd0d
Showing
28 changed files
with
634 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| Kube-router | ||
| =========== | ||
|
|
||
| Kube-router is a L3 CNI provider, as such it will setup IPv4 routing between | ||
| nodes to provide Pods' networks reachability. | ||
|
|
||
| See [kube-router documentation](https://www.kube-router.io/). | ||
|
|
||
| ## Verifying kube-router install | ||
|
|
||
| Kube-router runs its pods as a `DaemonSet` in the `kube-system` namespace: | ||
|
|
||
| * Check the status of kube-router pods | ||
|
|
||
| ``` | ||
| # From the CLI | ||
| kubectl get pod --namespace=kube-system -l k8s-app=kube-router -owide | ||
| # output | ||
| NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE | ||
| kube-router-4f679 1/1 Running 0 2d 192.168.186.4 mykube-k8s-node-nf-2 <none> | ||
| kube-router-5slf8 1/1 Running 0 2d 192.168.186.11 mykube-k8s-node-nf-3 <none> | ||
| kube-router-lb6k2 1/1 Running 0 20h 192.168.186.14 mykube-k8s-node-nf-6 <none> | ||
| kube-router-rzvrb 1/1 Running 0 20h 192.168.186.17 mykube-k8s-node-nf-4 <none> | ||
| kube-router-v6n56 1/1 Running 0 2d 192.168.186.6 mykube-k8s-node-nf-1 <none> | ||
| kube-router-wwhg8 1/1 Running 0 20h 192.168.186.16 mykube-k8s-node-nf-5 <none> | ||
| kube-router-x2xs7 1/1 Running 0 2d 192.168.186.10 mykube-k8s-master-1 <none> | ||
| ``` | ||
|
|
||
| * Peek at kube-router container logs: | ||
|
|
||
| ``` | ||
| # From the CLI | ||
| kubectl logs --namespace=kube-system -l k8s-app=kube-router | grep Peer.Up | ||
| # output | ||
| time="2018-09-17T16:47:14Z" level=info msg="Peer Up" Key=192.168.186.6 State=BGP_FSM_OPENCONFIRM Topic=Peer | ||
| time="2018-09-17T16:47:16Z" level=info msg="Peer Up" Key=192.168.186.11 State=BGP_FSM_OPENCONFIRM Topic=Peer | ||
| time="2018-09-17T16:47:46Z" level=info msg="Peer Up" Key=192.168.186.10 State=BGP_FSM_OPENCONFIRM Topic=Peer | ||
| time="2018-09-18T19:12:24Z" level=info msg="Peer Up" Key=192.168.186.14 State=BGP_FSM_OPENCONFIRM Topic=Peer | ||
| time="2018-09-18T19:12:28Z" level=info msg="Peer Up" Key=192.168.186.17 State=BGP_FSM_OPENCONFIRM Topic=Peer | ||
| time="2018-09-18T19:12:38Z" level=info msg="Peer Up" Key=192.168.186.16 State=BGP_FSM_OPENCONFIRM Topic=Peer | ||
| [...] | ||
| ``` | ||
|
|
||
| ## Gathering kube-router state | ||
|
|
||
| Kube-router Pods come bundled with a "Pod Toolbox" which provides very | ||
| useful internal state views for: | ||
|
|
||
| * IPVS: via `ipvsadm` | ||
| * BGP peering and routing info: via `gobgp` | ||
|
|
||
| You need to `kubectl exec -it ...` into a kube-router container to use these, see | ||
| <https://www.kube-router.io/docs/pod-toolbox/> for details. | ||
|
|
||
| ## Kube-router configuration | ||
|
|
||
|
|
||
| You can change the default configuration by overriding `kube_router_...` variables | ||
| (as found at `roles/network_plugin/kube-router/defaults/main.yml`), | ||
| these are named to follow `kube-router` command-line options as per | ||
| <https://www.kube-router.io/docs/user-guide/#try-kube-router-with-cluster-installers>. | ||
|
|
||
| ## Caveats | ||
|
|
||
| ### kubeadm_enabled: true | ||
|
|
||
| If you want to set `kube-router` to replace `kube-proxy` | ||
| (`--run-service-proxy=true`) while using `kubeadm_enabled`, | ||
| then 'kube-proxy` DaemonSet will be removed *after* kubeadm finishes | ||
| running, as it's not possible to skip kube-proxy install in kubeadm flags | ||
| and/or config, see https://github.com/kubernetes/kubeadm/issues/776. | ||
|
|
||
| Given above, if `--run-service-proxy=true` is needed it would be | ||
| better to void `kubeadm_enabled` i.e. set: | ||
|
|
||
| ``` | ||
| kubeadm_enabled: false | ||
| kube_router_run_service_proxy: true | ||
| ``` | ||
|
|
||
| If for some reason you do want/need to set `kubeadm_enabled`, removing | ||
| it afterwards behave better if kube-proxy is set to ipvs mode, i.e. set: | ||
|
|
||
| ``` | ||
| kubeadm_enabled: true | ||
| kube_router_run_service_proxy: true | ||
| kube_proxy_mode: ipvs | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
37 changes: 37 additions & 0 deletions
37
inventory/sample/group_vars/k8s-cluster/k8s-net-kube-router.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # See roles/network_plugin/kube-router//defaults/main.yml | ||
|
|
||
| # Enables Pod Networking -- Advertises and learns the routes to Pods via iBGP | ||
| # kube_router_run_router: true | ||
|
|
||
| # Enables Network Policy -- sets up iptables to provide ingress firewall for pods | ||
| # kube_router_run_firewall: true | ||
|
|
||
| # Enables Service Proxy -- sets up IPVS for Kubernetes Services | ||
| # see docs/kube-router.md "Caveats" section | ||
| # kube_router_run_service_proxy: false | ||
|
|
||
| # Add Cluster IP of the service to the RIB so that it gets advertises to the BGP peers. | ||
| # kube_router_advertise_cluster_ip: false | ||
|
|
||
| # Add External IP of service to the RIB so that it gets advertised to the BGP peers. | ||
| # kube_router_advertise_external_ip: false | ||
|
|
||
| # Add LoadbBalancer IP of service status as set by the LB provider to the RIB so that it gets advertised to the BGP peers. | ||
| # kube_router_advertise_loadbalancer_ip: false | ||
|
|
||
| # Array of arbitrary extra arguments to kube-router, see | ||
| # https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md | ||
| # kube_router_extra_args: [] | ||
|
|
||
| # ASN numbers of the BGP peer to which cluster nodes will advertise cluster ip and node's pod cidr. | ||
| # kube_router_peer_router_asns: ~ | ||
|
|
||
| # The ip address of the external router to which all nodes will peer and advertise the cluster ip and pod cidr's. | ||
| # kube_router_peer_router_ips: ~ | ||
|
|
||
| # The remote port of the external BGP to which all nodes will peer. If not set, default BGP port (179) will be used. | ||
| # kube_router_peer_router_ports: ~ | ||
|
|
||
| # Setups node CNI to allow hairpin mode, requires node reboots, see | ||
| # https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode | ||
| # kube_router_support_hairpin_mode: false |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 22 additions & 0 deletions
22
roles/kubernetes-apps/network_plugin/kube-router/tasks/main.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| --- | ||
|
|
||
| - name: kube-router | Start Resources | ||
| kube: | ||
| name: "kube-router" | ||
| kubectl: "{{ bin_dir }}/kubectl" | ||
| filename: "{{ kube_config_dir }}/kube-router.yml" | ||
| resource: "ds" | ||
| namespace: "kube-system" | ||
| state: "latest" | ||
| when: | ||
| - inventory_hostname == groups['kube-master'][0] | ||
|
|
||
| - name: kube-router | Wait for kube-router pods to be ready | ||
| command: "{{bin_dir}}/kubectl -n kube-system get pods -l k8s-app=kube-router -o jsonpath='{.items[?(@.status.containerStatuses[0].ready==false)].metadata.name}'" | ||
| register: pods_not_ready | ||
| until: pods_not_ready.stdout.find("kube-router")==-1 | ||
| retries: 30 | ||
| delay: 10 | ||
| ignore_errors: yes | ||
| when: | ||
| - inventory_hostname == groups['kube-master'][0] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.