fix(controller): Handle quota issues during pod reconciliation #3175
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: terrytangyuan <[email protected]>
…iation Signed-off-by: terrytangyuan <[email protected]>
terrytangyuan
changed the title
Signed-off-by: terrytangyuan <[email protected]>
|
Kudos, SonarCloud Quality Gate passed!
|
simster7
reviewed
Jun 4, 2020
| @@ -819,8 +837,8 @@ func (woc *wfOperationCtx) podReconciliation() error { | |||
| // It is now impossible to infer pod status. The only thing we can do at this point is to mark | |||
| // the node with Error. | |||
| for nodeID, node := range woc.wf.Status.Nodes { | |||
| if node.Type != wfv1.NodeTypePod || node.Fulfilled() || node.StartedAt.IsZero() { | |||
| // node is not a pod, it is already complete, or it can be re-run. | |||
| if node.Type != wfv1.NodeTypePod || node.Fulfilled() || node.StartedAt.IsZero() || ExceededQuota(&node) || FailedQuota(&node) { | |||
There was a problem hiding this comment.
This would dead-lock the controller, making the solution in #2385 moot.
| @@ -111,6 +119,16 @@ type failedNodeStatus struct { | |||
| FinishedAt metav1.Time `json:"finishedAt"` | |||
| } | |||
|
|
|||
| // ExceededQuota checks if the error message indicates an exceeded quota in the namespace. | |||
| func ExceededQuota(n *wfv1.NodeStatus) bool { | |||
| return strings.Contains(n.Message, exceededQuotaString) | |||
There was a problem hiding this comment.
This is not a preferable way to diagnose this error as node strings could be arbitrary
|
Closing as this is already fixed |
|
Thanks for double checking. I was looking at an outdated version and porting some of the changes. Good to know that it’s fixed. |
|
No worries! Thanks for trying to fix |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related #721.
Checklist:
"fix(controller): Updates such and such. Fixes #1234".