AWS EKS: Healthcheck probes don't work in ambient mode with enabled NetworkPolicy

[alex-k27]

Hi!
I'm working on migration from Istio sidecars to ambient mode on AWS EKS cluster (v1.30) and I faced an issue with failing Liveness and Readiness probes in ambient mode with enables NetworkPolicy. It worked fine with sidecars, but now I'm getting Client.Timeout for all the http and grpc health check probes for different apps in different namespaces, e.g.:
Warning Unhealthy 0s (x6 over 12s) kubelet Readiness probe failed: Get "https://10.X.X.X:8081/hub/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I use such or similar NetworkPolicies per namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-network-policy
  namespace: jupyterhub
spec:
  podSelector: {}
  policyTypes:
    - Egress
    - Ingress
  egress:
    - to: # allow traffic to kube-dns with specified protocol & port
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
    - to: # allow traffic to pods in following namespaces
        - namespaceSelector:
            matchExpressions:
              - key: kubernetes.io/metadata.name
                operator: In
                values: ["istio-system", "datadog", "jupyterhub"]
    - to:
      - ipBlock:
          cidr: 10.X.X.X/16
      - ipBlock:
          cidr: 127.0.0.1/32
  ingress:
    - from: # allow traffic from pods in following namespaces
        - namespaceSelector:
            matchExpressions:
              - key: kubernetes.io/metadata.name
                operator: In
                values: ["istio-ingress", "istio-system", "datadog", "jupyterhub"]

I also found that probes start to work if I update ingress rules to allow all the traffic in this way:

...
  ingress:
    - {}
...

But I would like to explicitly allow ingress traffic.
Could you advice on how to make health check probes work with strict network policies enabled.

My current istio components version is 1.22.1, but I faced the same issue with probes in ambient mode on version 1.21.2.

istioctl version
client version: 1.22.1
control plane version: 1.22.1
data plane version: 1.21.2 (8 proxies), 1.22.1 (14 proxies)

[howardjohn]

In ambient, traffic is tunneled over port 15008. This causes the netpol to see all traffic as coming in/out on port 15008.

However, this doesn't seem to be the case for what you are seeing.

I wouldn't expect health checks to be impacted by networkpolicies at all, regardless of Istio.

What network policy implementation are you using?

One other possibility is the health check is not denied by netpol, but the app is not being marked as healthy due to other reasons (like, perhaps, Istio blocking some traffic it needs to mark itself as ready)? inside the pod does curl https://127.0.0.1:8081/hub/health work?

[alex-k27]

Hello! Thanks for taking a look at this.

1. I use Amazon VPC CNI plugin for Kubernetes with { "enableNetworkPolicy": "true" } configuration (as it described here https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html ).
2. I tested healthcheck endpoint inside the pod in this way:

• I applied open network policy, restarted the pod and made sure that it's healthy
• run curl, it returns 200

curl https://127.0.0.1:8081/hub/health -i
HTTP/1.1 200 OK

• applied strict ingress net policy and continue testing with curl which continue to return 200 until the container crashes because of

Liveness probe failed: Get "https://10.x.x.x:8081/hub/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get "https://10.x.x.x:8081/hub/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I conclude that the health check endpoint works fine inside the pod even with strict net policy enabled, but the kubelet marks it as Unhealthy for some reason.

[howardjohn]

One other possibility is the health check is not denied by netpol, but the app is not being marked as healthy due to other reasons (like, perhaps, Istio blocking some traffic it needs to mark itself as ready)? inside the pod does curl https://127.0.0.1:8081/hub/health work?

Can you check this part if possible?

cc @bleggett

[bleggett]

@alex-k27 - similar to Cilium, if you create an ingress allowlist NetPol, your CNI (which enforces NetPol) will not know to ignore host-level healthchecks on the SNAT IP we use (169.254.7.127/32), and will block it.

The "allow" block you have for ingress doesn't consider host-level traffic and in general CNIs perform sleight-of-hand here usually, to silently exempt "some" host traffic from NetPol enforcement. But AWS EKS doesn't know to exempt host traffic with our SNAT IP, so it blocks it (I suspect, anyway).

You will need to allowFrom the ip 169.254.7.127/32 for every workload where you want host-node healthchecks to bypass your default-deny NetPol, and I suspect you should be good.

This should be better documented, and testing CNI and NetPol interactions directly in our integ suite is something we have long needed, but have not added.

[alex-k27]

I can confirm that the health check probes started to work after I applied this net policy ingress rule:

  ingress:
    - from:
      - ipBlock:
          cidr: 169.254.7.127/32

Thank you @bleggett !

[waynewong-ascenda]

Hi @bleggett @alex-k27, I had the same issue but I dont use network policy instead I use aws security group for in/outbound traffic. I tried to add 169.254.7.127/32 to security group rule but it's still network timeout. My istio version is 1.23.0. Thanks for your help.

[waynewong-ascenda]

Hi @howardjohn, do I need to add netpol for healthcheck? Thanks

[bleggett]

@waynewong-ascenda It might be good to open a separate issue, if you are not using netpol.

AFAIK AWS security groups do not apply to in-cluster traffic, and aren't relevant here.

[waynewong-ascenda]

Thanks @bleggett!