Running istio-proxy suddenly starts reporting authentication failure

[Purushotham233]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

Istio-proxy both as sidecar and ingress gateway previously running fine, suddenly started reporting authentication failure. And these are not recovering and needed to delete the pods.

Logs around the time once the error starts reporting. Need help in understanding why authentication is failing and hoping that could help find what is triggering this scenario.

istio-proxy 2024-06-20T22:37:07.895193Z info    xdsproxy        connected to upstream XDS server: istiod-istio-1-20-4.istio-system.svc.cluster.local:15012
istio-proxy 2024-06-20T23:05:44.762988Z info    xdsproxy        connected to upstream XDS server: istiod-istio-1-20-4.istio-system.svc.cluster.local:15012
istio-proxy 2024-06-20T23:33:19.854289Z info    xdsproxy        connected to upstream XDS server: istiod-istio-1-20-4.istio-system.svc.cluster.local:15012
istio-proxy 2024-06-20T23:39:49.165408Z info    ads     XDS: Incremental Pushing ConnectedEndpoints:2 Version:
istio-proxy 2024-06-20T23:39:49.336131Z info    cache   generated new workload certificate      latency=170.549629ms ttl=23h59m59.663872498s
istio-proxy 2024-06-21T00:02:05.302291Z info    xdsproxy        connected to upstream XDS server: istiod-istio-1-20-4.istio-system.svc.cluster.local:15012
istio-proxy 2024-06-21T00:02:05.308774Z warn    xdsproxy        upstream [26] terminated with unexpected error rpc error: code = Unauthenticated desc = authentication failure

Version

➜  ~ istioctl version
client version: 1.19.3
control plane version: 1.20.4
data plane version: 1.20.4 (* proxies)

➜  ~ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:14:10Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.12", GitCommit:"df63cd7cd818dd2262473d2170f4957c6735ba53", GitTreeState:"clean", BuildDate:"2023-12-19T13:32:17Z", GoVersion:"go1.20.12", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.24) and server (1.26) exceeds the supported minor version skew of +/-1

Additional Information

No response

[howardjohn]

istiod will show detailed error logs about why it was rejected

[Purushotham233]

Thanks @howardjohn . Somehow missed them earlier. Found the below logs

istiod-istio-1-20-4-5c4d56d97c-7drzb discovery 2024-06-22T17:48:58.137068Z      error   security        Failed to authenticate client from 10.52.19.152:34052: Authenticator ClientCertAuthenticator: no verified chain is found; Authenticator KubeJWTAuthenticator: failed to validate the JWT from cluster "*****": the service account authentication returns an error: [invalid bearer token, service account token has expired]

The bearer token here is same as istio-token mounted at kubernetes.io~projected/istio-token/istio-token ? These seem to have 12hours validity from the time of issuing.

As per documentation,

Istio agent monitors the expiration of the workload certificate. The above process repeats periodically for certificate and key rotation.

So istio agent failed to rotate the istio-token in time?

[howardjohn]

Kubernetes is responsible for mounting and rotating this. Istio just reads from the file and rotates it. If its stale, it is a k8s issue