Envoy listener drain(NETWORK_FILTER change) should not cause data plane impact

[shonecyx]

We observed whenever apply below configure EnvoyFilter, it will cause listener drain in both ingressgateway and sidecar and will result to massive connection reset. Ideally xDS change should not cause data plane impact, is this expected behavior or an Envoy feature gap/bug?

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  creationTimestamp: "2023-08-10T21:09:34Z"
  generation: 2
  name: sample-200-response-code-accesslog-1.20
  namespace: istio-production
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
      proxy:
        proxyVersion: ^1\.20.*
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          access_log:
          - filter:
              and_filter:
                filters:
                - not_health_check_filter: {}
                - or_filter:
                    filters:
                    - and_filter:
                        filters:
                        - runtime_filter:
                            percent_sampled:
                              denominator: HUNDRED
                              numerator: 1
                            runtime_key: http_ok_response_sampling_fraction
                            use_independent_randomness: true
                        - status_code_filter:
                            comparison:
                              op: EQ
                              value:
                                default_value: 200
                                runtime_key: http_ok_response_sampling_status_eq
                    - and_filter:
                        filters:
                        - runtime_filter:
                            percent_sampled:
                              denominator: HUNDRED
                              numerator: 100
                            runtime_key: http_err_response_sampling_fraction
                            use_independent_randomness: true
                        - or_filter:
                            filters:
                            - status_code_filter:
                                comparison:
                                  op: LE
                                  value:
                                    default_value: 199
                                    runtime_key: http_err_response_sampling_status_le
                            - status_code_filter:
                                comparison:
                                  op: GE
                                  value:
                                    default_value: 201
                                    runtime_key: http_err_response_sampling_status_ge

Affected product area (please put an X in all that apply)

[ ] Ambient
[ ] Docs
[ ] Dual Stack
[ ] Installation
[ x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Additional context

[howardjohn]

This is something to discuss in Envoy repo, given:

• Draining behavior is driven by Envoy
• You are using raw EnvoyFilter
  Note they will want the issues described purely in terms of Envoy though without Istio abstraction typically.