crdclient's addCRD() is causing a panic when casting DeletedFinalStateUnknown to Object

[edieYoung]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

We met this panic error on the istiod pod(1.19.7)

{
  "level": "error",
  "time": "2024-06-12T20:04:38.398023Z",
  "scope": "klog",
  "msg": "Observed a panic: &runtime.TypeAssertionError{_interface:(*abi.Type)(nil), concrete:(*abi.Type)(0x2bf30c0), asserted:(*abi.Type)(0x2f25260), missingMethod:\"DeepCopyObject\"} (interface conversion: cache.DeletedFinalStateUnknown is not controllers.Object: missing method DeepCopyObject)\ngoroutine 467 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x2a95940?, 0xc156dc66c0})\n\tk8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc02244bba8?})\n\tk8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b\npanic({0x2a95940?, 0xc156dc66c0?})\n\truntime/panic.go:914 +0x21f\nistio.io/istio/pilot/pkg/config/kube/crdclient.(*Client).addCRD.func1({0x2bf30c0?, 0xc2b8ba17a0})\n\tistio.io/istio/pilot/pkg/config/kube/crdclient/client.go:356 +0x86\nistio.io/istio/pkg/kube/kclient.(*informerClient[...]).AddEventHandler.func1()\n\tistio.io/istio/pkg/kube/kclient/client.go:122 +0x2a\nk8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnDelete({0xc0028d9e90?, {0x3506e30?, 0xc000f01ef0?}}, {0x2bf30c0, 0xc2b8ba17a0})\n\tk8s.io/[email protected]/tools/cache/controller.go:323 +0x43\nk8s.io/client-go/tools/cache.(*processorListener).run.func1()\n\tk8s.io/[email protected]/tools/cache/shared_informer.go:979 +0x9f\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)\n\tk8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0027f4f38?, {0x34e3480, 0xc002923d70}, 0x1, 0xc002966900)\n\tk8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc0027f4f10?)\n\tk8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f\nk8s.io/apimachinery/pkg/util/wait.Until(...)\n\tk8s.io/[email protected]/pkg/util/wait/backoff.go:161\nk8s.io/client-go/tools/cache.(*processorListener).run(0xc002c47440)\n\tk8s.io/[email protected]/tools/cache/shared_informer.go:968 +0x69\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x4f\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 432\n\tk8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x73"
}

The culprit seems to be this line(https://github.com/airbnb/istio/blob/master/pilot/pkg/config/kube/crdclient/client.go#L356) where t.(controllers.Object) is casting a DeletedFinalStateUnknown and causing this panic error.

We need to properly handle this DeletedFinalStateUnknown case, similar issue happened in #36014

Version

$ istioctl version
client version: 1.19.7
istiod version: 1.19.7

Additional Information

No response

[keithmattix]

@MorrisLaw do you want to take a stab at this?

[hzxuzhonghu]

We should revisit all the case to xx.(controllers.Object)