istio_authn filter disappeared after upgrading from 1.19.7 to 1.20.6

[AleksanderBrzozowski]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

Currently, we use Istio 1.19.7, and we want to upgrade to 1.20.6.

We upgraded the version to 1.20.6, and we noticed that istio_authn filter is missing from a filter chain:

          - name: istio_authn # this filter is not added anymore
            typed_config:
              '@type': type.googleapis.com/udpa.type.v1.TypedStruct
              type_url: type.googleapis.com/io.istio.network.authn.Config
          - name: istio.metadata_exchange
            typed_config:
              '@type': type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange
              protocol: istio-peer-exchange
          - name: envoy.filters.network.http_connection_manager
            typed_config:
            ....

This causes issues when we want to downgrade to 1.19.7 - when control plane is running 1.19.7, but the data plane is running 1.20.6. We saw that the following log is found in the control plane:

Internal:Error adding/updating listener(s) virtualInbound: Didn't find a registered implementation for 'istio_authn' with type URL: 'io.istio.network.authn.Config'

As a result of this downgrade, when control plane is running 1.19.7, it adds istio_authn filter, and the data plane running version 1.20.6 complains that it cannot register it because there is no implementation.

Since Istio supports one version change between data plane and control plane, this use case should be supported.

At first, I thought that this is caused by this change - #47407, but it seems that this change was applied to Istio 1.21, and is not in Istio 1.20. Then, I thought that maybe this is caused by this - #46899. Any thoughts?

Version

istioctl version
client version: 1.20.0
control plane version: 1.20.6
data plane version: 1.19.7 (192 proxies), 1.20.6 (333 proxies)

kubectl version --short
Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.27.13-eks-3af4770

Additional Information

No response

[howardjohn]

This is not a supported version skew: https://istio.io/latest/docs/releases/supported-releases/#control-planedata-plane-skew.

If you want to rollback https://istio.io/latest/docs/setup/upgrade/canary/ is a much safer approach

[AleksanderBrzozowski]

@howardjohn Thanks for you response!