Chunked encoding with lots of small chunks appear to cause throughput issues on small core hosts

[stevej]

In trying to nail down why tests with echoheaders is problematic, @siggy and I decided to replicate the configuration of echoheaders: an nginx process with a built-in lua script. What we found was that lua scripting sends a chunked body for each line and that seems to cause latency issues and higher than normal cpu with linkerd.

For reference, here is the nginx config for echoheaders
https://github.com/kubernetes/contrib/blob/master/ingress/echoheaders/nginx.conf

First, here's a baseline of slow_cooker against the chunked backend directly:

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -qps 50 -concurrency 100 -host chunked http:https://perf-target-4:8080/chunked
# sending 5000 req/s with concurrency=100 to http:https://perf-target-4:8080/chunked ...
#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change
2017-02-08T23:36:50Z  49900/0/0 50000  99% 10s   0 [  2   5   7   18 ]   18 +
2017-02-08T23:37:00Z  50000/0/0 50000 100% 10s   0 [  2   4   5    9 ]    9
2017-02-08T23:37:10Z  50000/0/0 50000 100% 10s   0 [  3   6   7   24 ]   24
2017-02-08T23:37:20Z  50001/0/0 50000 100% 10s   0 [  3   6   8   11 ]   11
2017-02-08T23:37:30Z  50016/0/0 50000 100% 10s   0 [  2   5   7   10 ]   10
2017-02-08T23:37:40Z  50004/0/0 50000 100% 10s   0 [  2   4   4    7 ]    7

Now we try with nginx returning a chunk per line on in the backend, through linkerd (380% linkerd cpu, note there's only 4 cores on this host)

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -qps 50 -concurrency 100 -host chunked http:https://proxy-test-4b:4140/chunked
# sending 5000 req/s with concurrency=100 to http:https://proxy-test-4b:4140/chunked ...
#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change
2017-02-08T23:34:11Z  30439/0/0 50000  60% 10s   0 [ 25  68 130  353 ]  353 +
2017-02-08T23:34:21Z  36264/0/0 50000  72% 10s   0 [ 22  54  71  129 ]  129
2017-02-08T23:34:31Z  36561/0/0 50000  73% 10s   0 [ 22  52  68  122 ]  122
2017-02-08T23:34:41Z  36007/0/0 50000  72% 10s   0 [ 22  54  70  123 ]  123
2017-02-08T23:34:51Z  36390/0/0 50000  72% 10s   0 [ 22  54  69  117 ]  117

And then with chunked encoding off in the backend, through linkerd (280% linkerd cpu). I turned chunked encoding off via the chunked_transfer_encoding off; directive. The entire config file can be found at perf-target-4:/etc/nginx/nginx.conf

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -qps 50 -concurrency 100 -host chunked http:https://proxy-test-4b:4140/nochunked
# sending 5000 req/s with concurrency=100 to http:https://proxy-test-4b:4140/nochunked ...
#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change
2017-02-08T23:32:12Z  48803/0/0 50000  97% 10s   0 [ 14  30  41   91 ]   91 +
2017-02-08T23:32:23Z  49460/0/0 50000  98% 10s   0 [ 13  27  37   77 ]   77
2017-02-08T23:32:33Z  49369/0/0 50000  98% 10s   0 [ 13  26  38   98 ]   98
2017-02-08T23:32:43Z  49447/0/0 50000  98% 10s   0 [ 13  27  37   75 ]   75
2017-02-08T23:32:53Z  49526/0/0 50000  99% 10s   0 [ 13  24  36   74 ]   74
2017-02-08T23:33:03Z  49390/0/0 50000  98% 10s   0 [ 13  28  37   94 ]   94
2017-02-08T23:33:13Z  49300/0/0 50000  98% 10s   0 [ 13  29  38   92 ]   92
2017-02-08T23:33:23Z  49082/0/0 50000  98% 10s   0 [ 13  30  40   95 ]   95
2017-02-08T23:33:33Z  49060/0/0 50000  98% 10s   0 [ 14  29  40   95 ]   95
2017-02-08T23:33:43Z  48958/0/0 50000  97% 10s   0 [ 14  30  40   71 ]   71

Disabling chunked encoding allows us to get closer to our throughput targets. We should consider profiling linkerd with chunked encoding, especially if people are going to be testing us with echoheaders in the mix.

[wmorgan]

Nice work.

[adleong]

We should keep this in mind for our next round of performance regression testing/profiling.