You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a controller is holding various K8s leases it should clear the holderIdentity when shutting down.
Actual Behavior
The controller process doesn't wait for the leases to be released before exiting. The resulting behaviour is non-deterministic but the cascading of errors have been more pronounced with GKE 1.21 and the fact that we increased our default lease duration.
For example the API audit logs will show leases.update with a Gateway Timeout status. Serving has seen upgrade tests become flaky because when one controller shuts down there's a wait for the new controller to start reconciliation of resources.
Note: this isn't limited to controllers but any component that uses sharedmain
The text was updated successfully, but these errors were encountered:
Expected Behavior
When a controller is holding various K8s leases it should clear the
holderIdentity
when shutting down.Actual Behavior
The controller process doesn't wait for the leases to be released before exiting. The resulting behaviour is non-deterministic but the cascading of errors have been more pronounced with GKE 1.21 and the fact that we increased our default lease duration.
For example the API audit logs will show
leases.update
with aGateway Timeout
status. Serving has seen upgrade tests become flaky because when one controller shuts down there's a wait for the new controller to start reconciliation of resources.Note: this isn't limited to controllers but any component that uses
sharedmain
The text was updated successfully, but these errors were encountered: