-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
controller sending many pod delete requests that result in 404 response #12659
Comments
For reference, this was previously noted in Slack |
As far as I understand, the "agent" is a piece of the Executor that runs for certain built-in, non-
That sounds like it might be accidentally assuming that each Pod has an agent, when there are only certain types that do 🤔 |
argo-workflows/workflow/controller/operator.go Line 2369 in 66680f1
|
The agent pod will only be created if argo-workflows/workflow/controller/agent.go Lines 32 to 46 in 5c8062e
Only http and plugin template will be put into taskSet right now. |
@jswxstw do u want to create PR? |
I see you have created a PR and it looks good to me basically. |
Signed-off-by: oninowang <[email protected]>
Signed-off-by: oninowang <[email protected]>
Signed-off-by: oninowang <[email protected]>
Signed-off-by: oninowang <[email protected]>
- merge conflicts with tests removed in backport by agilgur5 Signed-off-by: oninowang <[email protected]> Signed-off-by: jswxstw <[email protected]> Co-authored-by: jswxstw <[email protected]> Co-authored-by: agilgur5 <[email protected]> (cherry picked from commit 825aacf)
Pre-requisites
:latest
What happened/what did you expect to happen?
although this seems to have no effect on the functioning of argoworkflows it could be potential stability/performance/k8s log cost issue at scale. it must be also filling up the cleanup queue so delaying cleanup of pods that really do exist
from checking the k8s api server the wfcontroller seems to be sending delete pod request for
<podname from a step>-agent
and getting not found response. i am using standard workflows like whalesay example. not sure what significance of agent suffix is (i did seeargo-workflows/workflow/controller/agent.go
Line 25 in 66680f1
seeing 1000s of these, seems for every pod run its sending this unrequired delete request for pod with -agent suffix?
Version
3.4.11
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
n/a
Logs from the workflow controller
Logs from in your workflow's wait container
The text was updated successfully, but these errors were encountered: