Add TCP/UDP load balancing example

nginxinc · Aug 17, 2018 · 94a669f · 94a669f
1 parent ef84a56
commit 94a669f
Show file tree

Hide file tree

Showing 6 changed files with 332 additions and 2 deletions.
diff --git a/docs/nginx-ingress-controllers.md b/docs/nginx-ingress-controllers.md
@@ -20,7 +20,7 @@ The table below summarizes the key difference between nginxinc/kubernetes-ingres
 | Merging Ingress rules with the same host | Supported | Supported | Supported |
 | HTTP load balancing extensions - Annotations | See the [supported annotations](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/annotations.md) | See the [supported annotations](https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/customization) | See the [supported annotations](https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/customization)|
 | HTTP load balancing extensions -- ConfigMap | See the [supported ConfigMap keys](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/configmap.md) | See the [supported ConfigMap keys](https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/customization) | See the [supported ConfigMap keys](https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/customization) |
-| TCP/UDP | Supported via a ConfigMap | Not supported | Not supported |
+| TCP/UDP | Supported via a ConfigMap | Supported via a ConfigMap with native NGINX configuration | Supported via a ConfigMap with native NGINX configuration |
 | Websocket | Supported | Supported via an [annotation](https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket) | Supported via an [annotation](https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket) |
 | TCP SSL Passthrough | Supported via a ConfigMap | Not supported | Not supported |
 | JWT validation | Not supported | Not supported | Supported |

diff --git a/examples/customization/README.md b/examples/customization/README.md
@@ -67,7 +67,7 @@ The table below summarizes all of the options. For some of them, there are examp
 | `nginx.com/health-checks-mandatory-queue` | N/A | When active health checks are mandatory, configures a queue for temporary storing incoming requests during the time when NGINX Plus is checking the health of the endpoints after a configuration reload. | `0` | [Support for Active Health Checks](../health-checks). |
 | `nginx.com/slow-start` | N/A | Sets the upstream server [slow-start period](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#server-slow-start). By default, slow-start is activated after a server becomes [available](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#passive-health-checks) or [healthy](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#active-health-checks). To enable slow-start for newly added servers, configure [mandatory active health checks](../health-checks). | `"0s"` | |
 | N/A | `external-status-address` | Sets the address to be reported in the status of Ingress resources. Requires the `-report-status` command-line argument. Overrides the `-external-service` argument. | N/A | [Report Ingress Status](../../docs/report-ingress-status.md). |
-| N/A | `stream-snippets` | Sets a custom snippet in stream context. | N/A | |
+| N/A | `stream-snippets` | Sets a custom snippet in stream context. | N/A | [Support for TCP/UDP Load Balancing](../tcp-udp). |
 | N/A | `stream-log-format` | Sets the custom [log format](http:https://nginx.org/en/docs/stream/ngx_stream_log_module.html#log_format) for TCP/UDP load balancing. | See the [template file](../../nginx-controller/nginx/nginx.conf.tmpl). | |
 
 ## Using ConfigMaps

diff --git a/examples/tcp-udp/README.md b/examples/tcp-udp/README.md
@@ -0,0 +1,188 @@
+# Support for TCP/UDP Load Balancing
+
+In this example we deploy the NGINX or NGINX Plus Ingress controller, a DNS server and then configure both TCP and UDP load balancing for the DNS server using the `stream-snippets` [ConfigMap key](../customization).
+
+The standard Kubernetes Ingress resources assume that all traffic is HTTP-based; they do not cater for the case of basic TCP or UDP load balancing. In this example, we use the `stream-snippets` ConfigMap key to embed the required TCP and UDP load-balancing configuration directly into the `stream{}` block of the NGINX configuration file. 
+
+With NGINX, we’ll use the DNS name or virtual IP address to identify the service, and rely on kube-proxy to perform the internal load-balancing across the pool of pods. With NGINX Plus, we can use a [headless](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) service and its DNS name to obtain the real IP addresses of the pods behind the service, and load-balance across these. NGINX Plus re-resolves the DNS name frequently, so will update automatically when new pods are deployed or removed.
+
+## Prerequisites
+
+* We use `dig` for testing. Make sure it is installed on your machine.
+* We use native NGINX configuration to configure TCP/UDP load balancing. If you'd like to better understand the example configuration, read about [TCP/UDP load balancing](https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/) and [DNS service discovery](https://www.nginx.com/blog/dns-service-discovery-nginx-plus/) in NGINX/NGINX Plus.
+
+## Running the Example
+
+### 1. Deploy the Ingress Controller
+
+1. Follow the installation [instructions](../../docs/installation.md) to deploy the Ingress controller. Make sure to expose port 5353 of the Ingress controller
+both for TCP and UDP traffic.
+
+2. Save the public IP address of the Ingress controller into a shell variable:
+ ```
+ $ IC_IP=XXX.YYY.ZZZ.III
+ ```
+ **Note**: If you'd like to expose the Ingress controller via a service with the type LoadBalancer, it is not allowed to create a type LoadBalancer service for both TCP and UDP protocols. To overcome this limitation, create two separate services, one for TCP and the other for UDP. In this case you will end up with two separate public IPs, one for TCP and the other for UDP. Use the former in Step 4.2 and the latter in Step 4.1.
+3. Save port 5353 of the Ingress controller into a shell variable:
+ ```
+ $ IC_5353_PORT=<port number>
+ ```
+
+### 2. Deploy the DNS Server
+
+We deploy two replicas of [CoreDNS](https://coredns.io/), configured to forward DNS queries to `8.8.8.8`. We also create two services for CoreDNS pods -- `coredns` and `coredns-headless`. The reason for that is explained in Steps 3.1 and 3.2.
+
+Deploy the DNS server:
+
+```
+$ kubectl apply -f dns.yaml
+```
+
+### 3. Configure Load Balancing
+
+We use `stream-snippets` ConfigMap key to configure TCP and UDP load balancing for our CoreDNS pods.
+
+1. Create load balancing configuration. In our example we create one server that listens for TCP traffic on port 5353 and one server that listens for UDP traffic on the same port. Both servers load balance the incoming traffic to our CoreDNS pods:
+
+ * For NGINX, we use the following configuration:
+ ```nginx
+ upstream coredns-udp {
+ server coredns.default.svc.cluster.local:53;
+ }
+
+ server {
+ listen 5353 udp;
+ proxy_pass coredns-udp;
+ proxy_responses 1;
+ }
+
+ upstream coredns-tcp {
+ server coredns.default.svc.cluster.local:53;
+ }
+
+ server {
+ listen 5353;
+ proxy_pass coredns-tcp;
+ }
+ ```
+
+ We define upstream servers using a DNS name. When NGINX is reloaded, the DNS name will be resolved into the virtual IP of the `coredns` service.
+
+ **Note**: NGINX will fail to reload if the DNS name `coredns.default.svc.cluster.local` cannot be resolved. To avoid that, you can define the upstream servers using the virtual IP of the `coredns` service instead of the DNS name.
+
+ * For NGINX Plus, we use a different configuration:
+ ```nginx
+ resolver kube-dns.kube-system.svc.cluster.local valid=5s;
+
+ upstream coredns-udp {
+ zone coredns-udp 64k;
+ server coredns-headless.default.svc.cluster.local service=_dns._udp resolve;
+ }
+
+ server {
+ listen 5353 udp;
+ proxy_pass coredns-udp;
+ proxy_responses 1;
+ status_zone coredns-udp;
+ }
+
+ upstream coredns-tcp {
+ zone coredns-tcp 64k;
+ server coredns-headless.default.svc.cluster.local service=_dns-tcp._tcp resolve;
+ }
+
+ server {
+ listen 5353;
+ proxy_pass coredns-tcp;
+ status_zone coredns-tcp;
+ }
+ ```
+ NGINX Plus supports re-resolving DNS names with the `resolve` parameter of the `upstream` directive, which we take an advantage of in our example. Additionally, when the `resolve` parameter is used, NGINX Plus will not fail to reload if the name of an upstream cannot be resolved, in contrast with NGINX. In addition to IP addresses, NGINX Plus will discover ports through DNS SRV records. 
+
+ To resolve IP addresses and ports, NGINX Plus uses the Kube-DNS, defined with the `resolver` directive. We also set the `valid` parameter to `5s` to make NGINX Plus re-resolve DNS names every 5s.
+
+ Instead of `coredns` service, we use `coredns-headless` service. This service is created as a [headless service](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services), meaning that no virtual IP is allocated for that service and NGINX Plus will be able to resolve the IP addresses of all the CoreDNS pods.
+
+ **Note**: NGINX Plus will fail to reload if the DNS name, specified in the `resolver` directive, cannot be resolved. To avoid that, you can define the resolver using the virtual IP of the `kube-dns` service instead of the DNS name.
+
+1. Update the ConfigMap with the `stream-snippets` containing the load balancing configuration:
+ * For NGINX, run:
+ ```
+ $ kubectl apply -f nginx-config.yaml
+ ```
+ * For NGINX Plus, run:
+ ```
+ $ kubectl apply -f nginx-plus-config.yaml
+ ```
+1. Make sure NGINX or NGINX Plus is successfully reloaded:
+ ```
+ $ kubectl describe configmap nginx-config -n nginx-ingress
+ . . .
+ Events:
+ Type Reason Age From Message
+ ---- ------ ---- ---- -------
+ Normal Updated 3s nginx-ingress-controller Configuration from nginx-ingress/nginx-config was updated
+ ```
+
+
+### 4. Test the DNS Server
+
+To test that the configured TCP/UDP load balancing works, we resolve the name `kubernetes.io` using our DNS server available through the Ingress Controller:
+
+1. Resolve `kubernetes.io` through UDP:
+ ```
+ $ dig @$IC_IP -p $IC_5353_PORT kubernetes.io
+
+ ; <<>> DiG 9.10.6 <<>> @<REDACTED>> -p 5353 kubernetes.io
+ ; (1 server found)
+ ;; global options: +cmd
+ ;; Got answer:
+ ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33368
+ ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
+
+ ;; OPT PSEUDOSECTION:
+ ; EDNS: version: 0, flags:; udp: 512
+ ;; QUESTION SECTION:
+ ;kubernetes.io. IN A
+
+ ;; ANSWER SECTION:
+ kubernetes.io. 299 IN A 45.54.44.100
+
+ ;; Query time: 111 msec
+ ;; SERVER:<REDACTED>#5353(<REDACTED>)
+ ;; WHEN: Fri Aug 17 12:49:54 BST 2018
+ ;; MSG SIZE rcvd: 71
+ ```
+
+1. Resolve `kubernetes.io` through TCP:
+ ```
+ $ dig @$IC_IP -p $IC_5353_PORT kubernetes.io +tcp
+
+ ; <<>> DiG 9.10.6 <<>> @<REDACTED> -p 5353 kubernetes.io +tcp
+ ; (1 server found)
+ ;; global options: +cmd
+ ;; Got answer:
+ ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49032
+ ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
+
+ ;; OPT PSEUDOSECTION:
+ ; EDNS: version: 0, flags:; udp: 512
+ ;; QUESTION SECTION:
+ ;kubernetes.io. IN A
+
+ ;; ANSWER SECTION:
+ kubernetes.io. 146 IN A 45.54.44.100
+
+ ;; Query time: 95 msec
+ ;; SERVER: <REDACTED>#5353(<REDACTED>)
+ ;; WHEN: Fri Aug 17 12:52:25 BST 2018
+ ;; MSG SIZE rcvd: 71
+ ```
+1. Look at the Ingress Controller logs:
+ ```
+ $ kubectl logs <nginx-ingress-pod> -n nginx-ingress
+ . . .
+ <REDACTED> [17/Aug/2018:11:49:54 +0000] UDP 200 71 42 0.016
+ <REDACTED> [17/Aug/2018:11:52:25 +0000] TCP 200 73 44 0.098
+ ```
+
diff --git a/examples/tcp-udp/dns.yaml b/examples/tcp-udp/dns.yaml
@@ -0,0 +1,86 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: coredns
+data:
+ Corefile: |
+ .:53 {
+ forward . 8.8.8.8:53
+ log
+ }
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+ name: coredns
+spec:
+ replicas: 2
+ selector:
+ matchLabels:
+ app: coredns
+ template:
+ metadata:
+ labels:
+ app: coredns
+ spec:
+ containers:
+ - name: coredns
+ image: coredns/coredns:1.2.0
+ args: [ "-conf", "/etc/coredns/Corefile" ]
+ volumeMounts:
+ - name: config-volume
+ mountPath: /etc/coredns
+ readOnly: true
+ ports:
+ - containerPort: 53
+ name: dns
+ protocol: UDP
+ - containerPort: 53
+ name: dns-tcp
+ protocol: TCP
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ add:
+ - NET_BIND_SERVICE
+ drop:
+ - all
+ readOnlyRootFilesystem: true
+ volumes:
+ - name: config-volume
+ configMap:
+ name: coredns
+ items:
+ - key: Corefile
+ path: Corefile
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: coredns 
+spec:
+ selector:
+ app: coredns 
+ ports:
+ - name: dns
+ port: 53
+ protocol: UDP
+ - name: dns-tcp
+ port: 53
+ protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: coredns-headless
+spec:
+ clusterIP: None
+ selector:
+ app: coredns 
+ ports:
+ - name: dns
+ port: 53
+ protocol: UDP
+ - name: dns-tcp
+ port: 53
+ protocol: TCP
diff --git a/examples/tcp-udp/nginx-config.yaml b/examples/tcp-udp/nginx-config.yaml
@@ -0,0 +1,25 @@
+kind: ConfigMap
+apiVersion: v1
+metadata:
+ name: nginx-config
+ namespace: nginx-ingress
+data:
+ stream-snippets: |
+ upstream coredns-udp {
+ server coredns.default.svc.cluster.local:53;
+ }
+
+ server {
+ listen 5353 udp;
+ proxy_pass coredns-udp;
+ proxy_responses 1;
+ }
+
+ upstream coredns-tcp {
+ server coredns.default.svc.cluster.local:53;
+ }
+
+ server {
+ listen 5353;
+ proxy_pass coredns-tcp;
+ } 
diff --git a/examples/tcp-udp/nginx-plus-config.yaml b/examples/tcp-udp/nginx-plus-config.yaml
@@ -0,0 +1,31 @@
+kind: ConfigMap
+apiVersion: v1
+metadata:
+ name: nginx-config
+ namespace: nginx-ingress
+data:
+ stream-snippets: |
+ resolver kube-dns.kube-system.svc.cluster.local valid=5s;
+
+ upstream coredns-udp {
+ zone coredns-udp 64k;
+ server coredns-headless.default.svc.cluster.local service=_dns._udp resolve;
+ }
+
+ server {
+ listen 5353 udp;
+ proxy_pass coredns-udp;
+ proxy_responses 1;
+ status_zone coredns-udp;
+ }
+
+ upstream coredns-tcp {
+ zone coredns-tcp 64k;
+ server coredns-headless.default.svc.cluster.local service=_dns-tcp._tcp resolve;
+ }
+
+ server {
+ listen 5353;
+ proxy_pass coredns-tcp;
+ status_zone coredns-tcp;
+ }