Certain NameTree state transitions can cause the client to stop receiving name updates

[olix0r]

@obeattie reported an issue with the io.l5d.k8s namer where linkerd stops receiving updates.

Per @adleong:

Overview

In our QA veggies app, Lavender1 (the externally facing service) talks to Dill3. We will delete and then re-create the k8s service object for Dill3 which will cause Lavender1's linkerd to no longer be able to route to Dill3.

Reproduction Steps

Using our QA veggies topology, delete the dill3 service object:

kubectl --namespace=veggies delete svc/dill3

You will see this reflected in Lavender1's linkerd's logs:

D 0804 23:49:12.501 THREAD18: k8s deleted: dill3
I 0804 23:49:12.503 THREAD18: #/io.l5d.k8s/veggies/incoming/dill3: name resolution is negative (local dtab: Dtab())
D 0804 23:49:12.508 THREAD18: k8s ns veggies initial state: cayenne9, rutabaga6, rosemary5, potato7, pumpkin10, taro8, rutabaga4, lavender1, rutabaga2
D 0804 23:49:12.510 THREAD18: k8s ns veggies service dill3 missing

You will also see the corresponding loadbalancer removes in Lavender1's linkerd's metrics:

"rt/outgoing/dst/id/#/io.l5d.k8s/veggies/incoming/dill3/loadbalancer/removes": 3,

Then re-create the Dill3 service object:

kubectl --namespace=veggies create -f Dill3-svc.yml

You will see this reflected in Lavender1's linkerd's logs:

D 0804 23:49:34.296 THREAD18: k8s added: dill3
D 0804 23:49:34.296 THREAD18: k8s ns veggies initial state: dill3, cayenne9, rutabaga6, rosemary5, potato7, pumpkin10, taro8, rutabaga4, lavender1, rutabaga2
D 0804 23:49:34.297 THREAD18: k8s ns veggies service dill3 found
D 0804 23:49:34.297 THREAD18: k8s ns veggies service dill3 port incoming found + /

However, the new endpoints aren't added to loadbalancer, according to the metrics:

"rt/outgoing/dst/id/#/io.l5d.k8s/veggies/incoming/dill3/loadbalancer/size": 0,

And attempts to route requests to dill3 fail:

$ curl ....
No hosts are available for /http/dill3, Dtab.base=[/ns=>/#/io.l5d.k8s;/iface=>/ns/veggies;/srv=>/iface/incoming;/http=>/srv], Dtab.local=[]. Remote Info: Not Available

Restarting the Lavender1 linkerd fixes the issue and requests can be correctly routed to Dill3 again.

[olix0r]

Hypothesis: there are assumptions in the binding or load balancer factories about Activities not going Pending once they are OK.

[olix0r]

#567 provides a workaround--potentially a correct fix. We still need to get to the bottom of this behavior and document the contract between namers and linkerd.

[adleong]

Our current theory is that finagle expects that Activities never go pending once they have gone ok. As long as we obey that invariant, we should be ok (no pun intended). Closing this issue until something contradicts this theory.