Memory leak when client cache is evicting

[klingerf]

First reported by @ashald and @edio.

In a demo setup where the number of destinations exceeds the bindingCache, the linkerd process leaks memory as it adds/removes clients to/from the cache.

The easiest way to reproduce is with this docker-compose setup:

client-memory-issue.zip

Start the demo with:

docker-compose build && docker-compose up -d

Then load the linkerd admin dashboard on docker port 9990. The initial setup has 10 destinations and a client cache size of 10. You can see from the dashboard that the heap size for the process hovers at ~85mb.

Next stop the demo:

docker-compose stop && docker-compose rm -vf

And edit linkerd.yml to shrink the binding cache size from 10 to 9. Then restart the demo and reload the admin dashboard. You'll see the heap size start to creep up over time. Left running for about 30 minutes, heap will be ~400mb.

[chrisgoffinet]

I have confirmed this is a real leak. This exists in the latest build as well. I've taken a heap dump and working on a patch to fix this.

[dtacalau]

@adleong Please assign this issue to me.

[dtacalau]

I've tried to reproduce the memory leak with latest linkerd/namerd and had to modify the demo setup, here are the changes to initial demo setup:

1. I had to remove all code related to prometheus from the server.go because when trying to build the app using golang onbuild image I got a go module dependencies error:

Building app01
Step 1/2 : FROM golang:onbuild
# Executing 3 build triggers
 ---> Using cache
 ---> Running in df941fedd080
+ exec go get -v -d
github.com/prometheus/client_golang (download)
github.com/beorn7/perks (download)
github.com/cespare/xxhash (download)
package github.com/cespare/xxhash/v2: cannot find package "github.com/cespare/xxhash/v2" in any of:
    /usr/local/go/src/github.com/cespare/xxhash/v2 (from $GOROOT)
    /go/src/github.com/cespare/xxhash/v2 (from $GOPATH)
github.com/golang/protobuf (download)
github.com/prometheus/client_model (download)
github.com/prometheus/common (download)
github.com/matttproud/golang_protobuf_extensions (download)
github.com/prometheus/procfs (download)

But removing prometheus did not matter, the leak was still reproducing on linkerd 1.0.1.

2. Upped the versions, linkerd to 1.7.0, namerd to latest, slow_cooker to 1.2.0
3. Update linkerd.yaml to have linkerd admin bind on all interfaces, w/o this you'll get a connection refused when accessing admin from browser of docker host https://localhost:9990

I've attached the updated demo setup files.
client-memory-issue-mod.zip

[dtacalau]

Results of reproducing the leak with latest linkerd/namerd: screenshots with heap size after a 30 min run and cpu load.
pics.zip

After 30 minutes, the heap size increased a little nowhere near the ~400mb you get when reproducing with old linkerd 1.0.1. So, probably the leak is not a issue anymore, but I'd let the demo setup run for longer before reach a conclusion.

But I noticed another issue: the CPU load is very high after shrinking the binding cache size from 10 to 9. While before shrinking I noticed a 30% load max, after shrinking it averaged about 150%, so this would be another issue to investigate.

[dtacalau]

@ashald and @edio it seems the leak does not reproduce anymore with latest linkerd 1.7.0. Do you want to try and reproduce it on your end? According to @adleong the high cpu load is normal after shrinking the binding cache.

[ashald]

I don't think this issue was observed recently so most likely you're right.
I no longer work there so if anyone can confirm this - that would be @edio.

[koiuo]

Frankly speaking, I have hard time recalling any details, but as @ashald mentioned, we have not been observing any related symptoms for quite a while already.
I guess this issue can be closed.