Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Controllers (and related) should relinquish their leases when shutting down #2436

Closed
dprotaso opened this issue Feb 22, 2022 · 0 comments · Fixed by #2437
Closed

Controllers (and related) should relinquish their leases when shutting down #2436

dprotaso opened this issue Feb 22, 2022 · 0 comments · Fixed by #2437

Comments

@dprotaso
Copy link
Member

Expected Behavior

When a controller is holding various K8s leases it should clear the holderIdentity when shutting down.

Actual Behavior

The controller process doesn't wait for the leases to be released before exiting. The resulting behaviour is non-deterministic but the cascading of errors have been more pronounced with GKE 1.21 and the fact that we increased our default lease duration.

For example the API audit logs will show leases.update with a Gateway Timeout status. Serving has seen upgrade tests become flaky because when one controller shuts down there's a wait for the new controller to start reconciliation of resources.

Note: this isn't limited to controllers but any component that uses sharedmain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant