Add docs for Calico, set the annotation by default.

[costinm]

Please provide a description of this PR:

Mostly readme additions, attempting to simplify the install for Calico.

For Ztunnel, the default is set to 0.0.0.0 - we have other docs suggesting a more complex install that
only allows impersonation for the pod range, but ztunnel may need to also set it for other clusters,
east-west and even ingress. So far not clear how IPv6 will be handled.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: costinm / name: Costin Manolache (15743c3, aefd154)

[howardjohn]

kubectl debug may be more appropriate here?

This is going to be a privilege escalation for users by default which doesn't seem great. debug is really just ubuntu, we may even only allow distroless as part of "ambient uses safe mode by default".

[costinm]

Would kubectl debug raise the capabilities of the pod ? I.e. if a cap is dropped, will it be added back ?

I tried and didn't seem to work - but if there is some option to add the caps I agree it would be far better.

[costinm]

Also "safe by default" is all good - but "debug" is neither the default nor supposed to be safe.

[howardjohn]

debug is the default. It shouldn't be in safe mode, of course.

I just tested and it worked. kubectl debug --image istio/base -it -n istio-system ztunnel-9xpvt then tcpdump in the shell had no issues. kubectl exec into ztunnel I got permission denied

[costinm]

Ok, will remove this.

Speaking of safe mode - for ztunnel it may be good to default to safe, and have debug optional. We should do
the same in the safe charts for istiod.

[costinm]

Also - if this works, maybe we should remove tcpdump, dig and other debug helpers from istio/base ?
I guess backward compat will prevent this - but we can create a second base for 'safe mode' at some point
and do a base cleanup too.

[AndreaM12345]

I think we may also need mention adding the IPs as mentioned in this doc?
https://github.com/istio/istio/tree/master/cni/pkg/ebpf

kubectl -n istio-system patch ds ztunnel --type='json' -p='[{"op": "add", "path": "/spec/template/metadata/annotations/cni.projectcalico.org~1allowedSourcePrefixes", "value": "[\"10.x.0.0/16\"]"}]'

See that you mention the IPs in the daemonset.yaml but thinking for clarity maybe mentioning it in the readme

[costinm]

I don't think patching ztunnel is right - user may reinstall it, upgrade, etc. For debugging or dev it may be ok, not prod.

I added the annotation to ztunnel - so it is added without extra patching ( non-calico will ignore it AFAIK ).

Also instead of 10.x... I set it to 0.0.0.0 - ztunnel may need to impersonate original dst from other clusters in MC or even from ingress.

Did you do any testing with IPv6 ? Is there an equivalent annotation ?

[AndreaM12345]

didn't test with IPv6. I think you are correct about not patching ztunnel. My comment was that the readme should point out that if a user wants to adjust the allowedSourcePrefix they should make the change to the daemonset.yaml Just for my own clarification are you suggesting that users should not set the allowedSourcePrefix?

[howardjohn]

ztunnel is only going to impersonate pods on its own node fwiw, so technically the setting on each ztunnel could be its node's CIDR.

Having a per-pod annotation based on the node is pretty hard to do though so not suggesting we actually do that, but I think it would be the minimal allowed from calico POV

[howardjohn]

Actually I am wrong :-) it will only impersonate to or from a node-local pod, but it may be random thing -> node local pod

[costinm]

Yes, that was my understanding as well - it should use 'x-forwarded-for' header, which may reflect a source IP from another cluster ( multi-network + E/W ), or the original pod sending request to the Waypoint or even the remote IP for ingress. Since we have no other way to convey this info ( and even when we have metadata endpoint, it's more efficient and compatible to have the real peer IP).

I am starting to believe we should implement HAProxy as well - it is VERY common in cloud infra. How to declare that a pod expects HAProxy remains TBD, maybe something GAMMA should define since others will have the same problem.