Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3.5.6 Logs flooded with Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io #13149

Open
3 of 4 tasks
vermaxik opened this issue Jun 6, 2024 · 6 comments
Open
3 of 4 tasks

3.5.6 Logs flooded with Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io #13149

vermaxik opened this issue Jun 6, 2024 · 6 comments
Labels
area/controller Controller issues, panics type/bug type/regression Regression from previous behavior (a specific type of bug)

Comments

@vermaxik
Copy link

vermaxik commented Jun 6, 2024

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

After upgrading to 3.5.6 we can see in logs a lot of warning of

{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-6gndn\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"customer-workflows","time":"2024-05-29T11:47:22.010Z","workflow":"workflow-6gndn"}

The issue was separated from #13140

Version

3.5.6

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

parallelism: 50
resourceRateLimit:
  limit: 100
  burst: 3
mainContainer:
  imagePullPolicy: IfNotPresent
  resources:
    limits:
      cpu: 0.3
      memory: 256M
    requests:
      cpu: 0.1
      memory: 128M
executor:
  imagePullPolicy: IfNotPresent
  resources:
    limits:
      cpu: 0.3
      memory: 256M
    requests:
      cpu: 0.1
      memory: 128M
artifactRepository:
  archiveLogs: true
  s3:
    bucket: argo-wf
    endpoint: s3.eu-west-1.amazonaws.com
    keyFormat: "{{workflow.labels.customer-id}}/{{workflow.name}}/{{pod.name}}"
    region: eu-west-1
persistence:
  archive: true
  nodeStatusOffLoad: true
  connectionPool:
    connMaxLifetime: 300s
  postgresql:
    host: argo-rds
    port: 5432
    database: argo_production
    tableName: argo_workflows
    userNameSecret:
      key: username
      name: db-secret
    passwordSecret:
      key: password
      name: db-secret

Logs from the workflow controller

kubectl logs -n argo deploy/argo-workflows-workflow-controller -c controller | grep workflow-c6d2s


{"key":"customer-workflows/workflow-c6d2s","level":"info","msg":"Workflow processing has been postponed due to max parallelism limit","time":"2024-06-05T14:04:46.775Z"}
{"level":"info","msg":"Updated phase  -\u003e Pending","namespace":"customer-workflows","time":"2024-06-05T14:04:46.775Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated message  -\u003e Workflow processing has been postponed because too many workflows are already running","namespace":"customer-workflows","time":"2024-06-05T14:04:46.775Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-c6d2s\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"customer-workflows","time":"2024-06-05T14:04:46.787Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Re-applying updates on latest version and retrying update","namespace":"customer-workflows","time":"2024-06-05T14:04:46.787Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Update retry attempt 1 successful","namespace":"customer-workflows","time":"2024-06-05T14:04:46.800Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Pending","resourceVersion":"245969123","time":"2024-06-05T14:04:46.800Z","workflow":"workflow-c6d2s"}
{"key":"customer-workflows/workflow-c6d2s","level":"info","msg":"Workflow processing has been postponed due to max parallelism limit","time":"2024-06-05T14:04:51.785Z"}
{"Phase":"Pending","ResourceVersion":"245969123","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:46.200Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":0,"time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated phase Pending -\u003e Running","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Updated message Workflow processing has been postponed because too many workflows are already running -\u003e ","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"DAG node workflow-c6d2s initialized Running","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-500067944, taskName task-02b979a3-e102-4e26-87ef-91ffe5c799a2","time":"2024-06-05T14:11:46.206Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-500067944, taskName task-02b979a3-e102-4e26-87ef-91ffe5c799a2","time":"2024-06-05T14:11:46.206Z"}
{"level":"info","msg":"All of node workflow-c6d2s.task-02b979a3-e102-4e26-87ef-91ffe5c799a2 dependencies [] completed","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Pod node workflow-c6d2s-500067944 initialized Pending","namespace":"customer-workflows","time":"2024-06-05T14:11:46.206Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Created pod: workflow-c6d2s.task-02b979a3-e102-4e26-87ef-91ffe5c799a2 (workflow-c6d2s-action-version-7142-task-02b979a3-e102-4e26-87ef-91ffe5c799a2-500067944)","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:46.237Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:46.237Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:46.237Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995160","time":"2024-06-05T14:11:46.277Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245995286","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:51.238Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":1,"time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-500067944","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:51.240Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:51.240Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:51.240Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995473","time":"2024-06-05T14:11:51.291Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245995601","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:11:56.286Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":1,"time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-c6d2s-500067944","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-05T14:11:56.287Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.287Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.288Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.290Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-1995060186, taskName task-15f3d4f5-0e50-4327-83cf-1923b124169a","time":"2024-06-05T14:11:56.290Z"}
{"level":"info","msg":"All of node workflow-c6d2s.task-15f3d4f5-0e50-4327-83cf-1923b124169a dependencies [task-02b979a3-e102-4e26-87ef-91ffe5c799a2] completed","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Pod node workflow-c6d2s-1995060186 initialized Pending","namespace":"customer-workflows","time":"2024-06-05T14:11:56.290Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Created pod: workflow-c6d2s.task-15f3d4f5-0e50-4327-83cf-1923b124169a (workflow-c6d2s-action-version-7917-task-15f3d4f5-0e50-4327-83cf-1923b124169a-1995060186)","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.318Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:11:56.318Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:11:56.318Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245995803","time":"2024-06-05T14:11:56.339Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7142-task-02b979a3-e102-4e26-87ef-91ffe5c799a2-500067944/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:11:58.341Z"}
{"Phase":"Running","ResourceVersion":"245995932","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:01.317Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":2,"time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-1995060186","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:01.319Z","workflow":"workflow-c6d2s"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.319Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.320Z"}
{"level":"warning","msg":"was unable to obtain the node for workflow-c6d2s-4280457137, taskName task-ec0efb20-7a0f-452f-996e-e8b86bd723ff","time":"2024-06-05T14:12:01.320Z"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:01.320Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:01.320Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996166","time":"2024-06-05T14:12:01.333Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245996323","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:06.337Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":2,"time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"PodInitializing","new.phase":"Pending","new.progress":"0/1","nodeID":"workflow-c6d2s-4280457137","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:06.338Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:06.339Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:06.339Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996490","time":"2024-06-05T14:12:06.370Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-15f3d4f5-0e50-4327-83cf-1923b124169a-1995060186/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:08.372Z"}
{"Phase":"Running","ResourceVersion":"245996679","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:11.369Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":3,"time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:11.371Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:11.372Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:11.372Z","workflow":"workflow-c6d2s"}
{"action":"terminateContainers","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:11.377Z"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245996843","time":"2024-06-05T14:12:11.384Z","workflow":"workflow-c6d2s"}
{"Phase":"Running","ResourceVersion":"245997016","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:16.386Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":3,"time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node unchanged","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"PodInitializing","new.phase":"Pending","new.progress":"0/1","nodeID":"workflow-c6d2s-2361177406","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-05T14:12:16.387Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2024-06-05T14:12:16.388Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245997196","time":"2024-06-05T14:12:16.405Z","workflow":"workflow-c6d2s"}
{"action":"deletePod","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:18.411Z"}
{"Phase":"Running","ResourceVersion":"245997390","level":"info","msg":"Processing workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:21.403Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Task-result reconciliation","namespace":"customer-workflows","numObjs":4,"time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-500067944","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-1995060186","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-4280457137","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"task-result changed","namespace":"customer-workflows","nodeID":"workflow-c6d2s-2361177406","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"node changed","namespace":"customer-workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-c6d2s-2361177406","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"reconcileAgentPod","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"customer-workflows","time":"2024-06-05T14:12:21.404Z","workflow":"workflow-c6d2s"}
{"action":"terminateContainers","key":"customer-workflows/workflow-c6d2s-exit-handler-2361177406/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:21.410Z"}
{"level":"info","msg":"https://172.20.0.1:443/api/v1/namespaces/customer-workflows/pods/workflow-c6d2s-exit-handler-2361177406/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec\u0026command=kill\u0026command=15\u0026command=1\u0026container=wait\u0026stderr=true\u0026stdout=true\u0026tty=false","time":"2024-06-05T14:12:21.410Z"}
{"level":"info","msg":"Workflow update successful","namespace":"customer-workflows","phase":"Running","resourceVersion":"245997561","time":"2024-06-05T14:12:21.423Z","workflow":"workflow-c6d2s"}
{"container":"wait","error":"unable to upgrade connection: container not found (\"wait\")","level":"info","msg":"signaled container","namespace":"customer-workflows","pod":"workflow-c6d2s-exit-handler-2361177406","stderr":null,"stdout":null,"time":"2024-06-05T14:12:21.438Z"}
{"level":"info","msg":"archiving workflow","namespace":"customer-workflows","time":"2024-06-05T14:12:25.441Z","uid":"59086eb4-e239-4e99-ac40-91a4a90f5632","workflow":"workflow-c6d2s"}
{"level":"info","msg":"Queueing Succeeded workflow customer-workflows/workflow-c6d2s for delete in 5m0s due to TTL","time":"2024-06-05T14:12:25.496Z"}
{"action":"killContainers","key":"customer-workflows/workflow-c6d2s-action-version-7917-task-ec0efb20-7a0f-452f-996e-e8b86bd723ff-4280457137/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:41.377Z"}
{"action":"killContainers","key":"customer-workflows/workflow-c6d2s-exit-handler-2361177406/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-05T14:12:51.439Z"}
{"level":"info","msg":"Queueing Succeeded workflow customer-workflows/workflow-c6d2s for delete in 2m29s due to TTL","time":"2024-06-05T14:14:56.080Z"}
{"level":"info","msg":"Deleting garbage collected workflow 'customer-workflows/workflow-c6d2s'","time":"2024-06-05T14:17:26.000Z"}
{"level":"info","msg":"Workflow already deleted 'customer-workflows/workflow-c6d2s'","time":"2024-06-05T14:17:26.014Z"}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
@vermaxik vermaxik added type/bug type/regression Regression from previous behavior (a specific type of bug) labels Jun 6, 2024
@Joibel Joibel added the area/controller Controller issues, panics label Jun 6, 2024
@Joibel Joibel mentioned this issue Jun 6, 2024
Closed
4 tasks
@agilgur5 agilgur5 changed the title 3.5.6 Logs flooded with "Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io" 3.5.6 Logs flooded with Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io Jun 6, 2024
@tooptoop4
Copy link
Contributor

@vermaxik have u tried on v3.5.8?

@vermaxik
Copy link
Author

@vermaxik have u tried on v3.5.8?

unfortunately, the issue is still persist in 3.5.8.

image

@tooptoop4
Copy link
Contributor

@vermaxik can you change the workflow controller command line to have --loglevel debug and share logs?

@Joibel
Copy link
Member

Joibel commented Jun 25, 2024

Seeing as no-one else is seeing this issue so far I suspect you have some local configuration causing a problem. Your cluster appears quite busy, does this also happen when you're running fewer workflows?

#12574 seems the most likely candidate for causing this, but I don't know why you're affected and others aren't, and I'm not sure it's the problem.

@reisei
Copy link

reisei commented Jun 25, 2024

Hello!

This is logs of the one workflow from ~100 with debug enabled.

{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:15.878Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:15.878Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:15.878Z"}
{"Phase":"Running","ResourceVersion":"257537302","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:18.525Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":5,"time":"2024-06-25T14:48:18.538Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.539Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:18.539Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:18.539Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:18.540Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:18.541Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:18.542Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1595781171","old.message":"","old.phase":"Pending","old.progress":"0/1","time":"2024-06-25T14:48:18.543Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:18.543Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.543Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.543Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:18.543Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42 of DAG is Running","namespace":"workflows","time":"2024-06-25T14:48:18.544Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.544Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.544Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67","time":"2024-06-25T14:48:18.544Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67)"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3: template: *v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3), boundaryID: workflow-21056-k6k42","namespace":"workflows","time":"2024-06-25T14:48:18.545Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.545Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.545Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:18.545Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3 of Pod is Running","namespace":"workflows","time":"2024-06-25T14:48:18.546Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.546Z","tmpl":"*v1alpha1.NodeStatus (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.547Z","tmpl":"*v1alpha1.NodeStatus (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:18.547Z","tmpl":"*v1alpha1.NodeStatus (root)"}
{"action":"terminateContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:18.582Z"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3 with container template: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3\n","namespace":"workflows","time":"2024-06-25T14:48:18.586Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipped pod workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3 (workflow-21056-k6k42-1595781171) creation: already exists","namespace":"workflows","podPhase":"Running","time":"2024-06-25T14:48:18.587Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.587Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.587Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0","time":"2024-06-25T14:48:18.587Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:18.589Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:18.590Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12","time":"2024-06-25T14:48:18.590Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:18.599Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:18.599Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"workflow-21056-k6k42-1595781171\":{\"hostNodeName\":\"ip-10-13-167-21.eu-west-1.compute.internal\",\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log\"}}],\"exitCode\":\"0\",\"parameters\":[{\"name\":\"podname\",\"value\":\"workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171\"}]},\"phase\":\"Running\"}},\"taskResultsCompletionStatus\":{\"workflow-21056-k6k42-1595781171\":true}}}","time":"2024-06-25T14:48:18.604Z"}
{"level":"warning","msg":"Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"workflow-21056-k6k42\": the object has been modified; please apply your changes to the latest version and try again Conflict","namespace":"workflows","time":"2024-06-25T14:48:18.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Re-applying updates on latest version and retrying update","namespace":"workflows","time":"2024-06-25T14:48:18.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Update retry attempt 1 successful","namespace":"workflows","time":"2024-06-25T14:48:18.676Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"257537489","time":"2024-06-25T14:48:18.676Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537489\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:18.676Z"}
{"Phase":"Running","ResourceVersion":"257537489","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:23.610Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":5,"time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:23.612Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1595781171","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:23.613Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.613Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.613Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:23.613Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42 of DAG is Running","namespace":"workflows","time":"2024-06-25T14:48:23.614Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.614Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.DAGTask (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12","time":"2024-06-25T14:48:23.615Z","tmpl":"*v1alpha1.NodeStatus (action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12)"}
{"level":"info","msg":"Outbound nodes of workflow-21056-k6k42 set to [workflow-21056-k6k42-1595781171 workflow-21056-k6k42-3182910293 workflow-21056-k6k42-4278140849]","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node workflow-21056-k6k42 phase Running -\u003e Succeeded","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node workflow-21056-k6k42 finished: 2024-06-25 14:48:23.615770727 +0000 UTC","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Checking daemoned children of workflow-21056-k6k42","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:23.615Z","workflow":"workflow-21056-k6k42"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:23.616Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:23.616Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: exit-handler","time":"2024-06-25T14:48:23.616Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"level":"debug","msg":"Initializing node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Pod node workflow-21056-k6k42-1349795579 initialized Pending","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.onExit with container template: exit-handler\n","namespace":"workflows","time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"","namespace":"workflows","needLocation":true,"time":"2024-06-25T14:48:23.616Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Creating Pod: workflow-21056-k6k42.onExit (workflow-21056-k6k42-exit-handler-1349795579)","namespace":"workflows","time":"2024-06-25T14:48:23.620Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Created pod: workflow-21056-k6k42.onExit (workflow-21056-k6k42-exit-handler-1349795579)","namespace":"workflows","time":"2024-06-25T14:48:23.654Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"}],\"nodes\":{\"workflow-21056-k6k42\":{\"finishedAt\":\"2024-06-25T14:48:23Z\",\"outboundNodes\":[\"workflow-21056-k6k42-1595781171\",\"workflow-21056-k6k42-3182910293\",\"workflow-21056-k6k42-4278140849\"],\"phase\":\"Succeeded\"},\"workflow-21056-k6k42-1349795579\":{\"displayName\":\"workflow-21056-k6k42.onExit\",\"finishedAt\":null,\"id\":\"workflow-21056-k6k42-1349795579\",\"name\":\"workflow-21056-k6k42.onExit\",\"nodeFlag\":{\"hooked\":true},\"phase\":\"Pending\",\"startedAt\":\"2024-06-25T14:48:23Z\",\"templateName\":\"exit-handler\",\"templateScope\":\"local/workflow-21056-k6k42\",\"type\":\"Pod\"},\"workflow-21056-k6k42-1595781171\":{\"finishedAt\":\"2024-06-25T14:48:17Z\",\"phase\":\"Succeeded\",\"resourcesDuration\":{\"cpu\":0,\"memory\":2}}}}}","time":"2024-06-25T14:48:23.655Z"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"257537587","time":"2024-06-25T14:48:23.696Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537587\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node workflow-21056-k6k42.task-8214a347-4a77-43a2-996e-57e4ab9e3da3","time":"2024-06-25T14:48:23.696Z"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537587\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node workflow-21056-k6k42","time":"2024-06-25T14:48:23.698Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:25.698Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:28.615Z"}
{"Phase":"Running","ResourceVersion":"257537600","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:28.663Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":6,"time":"2024-06-25T14:48:28.682Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.682Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:28.685Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:28.685Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:28.693Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.694Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1349795579  workflows  b6f80e49-1f52-43d4-89a8-b54467be6e29 257537676 2 2024-06-25 14:48:25 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:27 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-exit-handler-1349795579/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1349795579","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Running","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1349795579","old.message":"PodInitializing","old.phase":"Pending","old.progress":"0/1","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:28.695Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:28.695Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:28.695Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:28.695Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Node workflow-21056-k6k42 already completed","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:28.698Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:28.698Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:28.699Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: exit-handler","time":"2024-06-25T14:48:28.699Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.onExit of Pod is Running","namespace":"workflows","time":"2024-06-25T14:48:28.700Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Executing node workflow-21056-k6k42.onExit with container template: exit-handler\n","namespace":"workflows","time":"2024-06-25T14:48:28.700Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipped pod workflow-21056-k6k42.onExit (workflow-21056-k6k42-1349795579) creation: already exists","namespace":"workflows","podPhase":"Running","time":"2024-06-25T14:48:28.700Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"workflow-21056-k6k42-1349795579\":{\"message\":null,\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"default/workflow-21056-k6k42/workflow-21056-k6k42-exit-handler-1349795579/main.log\"}}],\"exitCode\":\"0\"},\"phase\":\"Running\"}},\"taskResultsCompletionStatus\":{\"workflow-21056-k6k42-1349795579\":true}}}","time":"2024-06-25T14:48:28.709Z"}
{"action":"terminateContainers","key":"workflows/workflow-21056-k6k42-exit-handler-1349795579/terminateContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:28.710Z"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Running","resourceVersion":"257537693","time":"2024-06-25T14:48:28.744Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537693\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node workflow-21056-k6k42.onExit","time":"2024-06-25T14:48:28.757Z"}
{"Phase":"Running","ResourceVersion":"257537709","level":"info","msg":"Processing workflow","namespace":"workflows","time":"2024-06-25T14:48:34.231Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Task-result reconciliation","namespace":"workflows","numObjs":6,"time":"2024-06-25T14:48:34.233Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1349795579  workflows  b6f80e49-1f52-43d4-89a8-b54467be6e29 257537676 2 2024-06-25 14:48:25 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:27 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-exit-handler-1349795579/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.233Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:34.233Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1349795579","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1349795579","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-2728974550  workflows  2b26e8f2-60ec-4a46-9d82-a8135fb1f074 257536296 2 2024-06-25 14:47:56 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:47:58 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-df57904b-a342-4892-bb45-5226e7eb3244-2728974550/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-2728974550","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-2728974550","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3451334822  workflows  17635c50-a7f1-45a6-b5f8-6b0be6b5e7cb 257537000 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3451334822","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3451334822","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-4278140849  workflows  aa6afaa7-8f90-456b-a5c7-b4430bda2258 257537003 2 2024-06-25 14:48:05 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-4278140849","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-4278140849","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-3182910293  workflows  2d8f409a-74e2-4ea4-a5b2-0039bb29783c 257536992 2 2024-06-25 14:48:06 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:07 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-3182910293","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-3182910293","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result:\n\u0026WorkflowTaskResult{ObjectMeta:{workflow-21056-k6k42-1595781171  workflows  7925fcd2-1155-4ab9-bcd2-30903ad93313 257537465 2 2024-06-25 14:48:16 +0000 UTC \u003cnil\u003e \u003cnil\u003e map[workflows.argoproj.io/report-outputs-completed:true workflows.argoproj.io/workflow:workflow-21056-k6k42] map[] [{argoproj.io/v1alpha1 Workflow workflow-21056-k6k42 30ca6b42-885b-407c-86c1-f89f91d2059c \u003cnil\u003e \u003cnil\u003e}] []  [{argoexec Update argoproj.io/v1alpha1 2024-06-25 14:48:17 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/report-outputs-completed\":{},\"f:workflows.argoproj.io/workflow\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"30ca6b42-885b-407c-86c1-f89f91d2059c\\\"}\":{}}},\"f:outputs\":{\".\":{},\"f:artifacts\":{},\"f:parameters\":{}}} }]},NodeResult:NodeResult{Phase:,Message:,Outputs:\u0026Outputs{Parameters:[]Parameter{Parameter{Name:podname,Default:nil,Value:*workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171,ValueFrom:nil,GlobalName:,Enum:[],Description:nil,},},Artifacts:[]Artifact{Artifact{Name:main-logs,Path:,Mode:nil,From:,ArtifactLocation:ArtifactLocation{ArchiveLogs:nil,S3:\u0026S3Artifact{S3Bucket:S3Bucket{Endpoint:,Bucket:,Region:,Insecure:nil,AccessKeySecret:nil,SecretKeySecret:nil,RoleARN:,UseSDKCreds:false,CreateBucketIfNotPresent:nil,EncryptionOptions:nil,CASecret:nil,},Key:default/workflow-21056-k6k42/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/main.log,},Git:nil,HTTP:nil,Artifactory:nil,HDFS:nil,Raw:nil,OSS:nil,GCS:nil,Azure:nil,},GlobalName:,Archive:nil,Optional:false,SubPath:,RecurseMode:false,FromExpression:,ArtifactGC:nil,Deleted:false,},},Result:nil,ExitCode:nil,},Progress:,},}","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"task result name:\nworkflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Marking task result complete workflow-21056-k6k42-1595781171","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"task-result changed","namespace":"workflows","nodeID":"workflow-21056-k6k42-1595781171","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"workflows","time":"2024-06-25T14:48:34.234Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"node changed","namespace":"workflows","new.message":"","new.phase":"Succeeded","new.progress":"0/1","nodeID":"workflow-21056-k6k42-1349795579","old.message":"","old.phase":"Running","old.progress":"0/1","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42: template: *v1alpha1.WorkflowStep (root), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: root","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (root)"}
{"level":"debug","msg":"Node workflow-21056-k6k42 already completed","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"TaskSet Reconciliation","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"reconcileAgentPod","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Running OnExit handler: exit-handler","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Evaluating node workflow-21056-k6k42.onExit: template: *v1alpha1.WorkflowStep (exit-handler), boundaryID: ","namespace":"workflows","time":"2024-06-25T14:48:34.235Z","workflow":"workflow-21056-k6k42"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Resolving the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"base":"*v1alpha1.Workflow (namespace=workflows,name=workflow-21056-k6k42)","level":"debug","msg":"Getting the template by name: exit-handler","time":"2024-06-25T14:48:34.235Z","tmpl":"*v1alpha1.WorkflowStep (exit-handler)"}
{"level":"debug","msg":"Node workflow-21056-k6k42.onExit already completed","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Task results completion status: map[workflow-21056-k6k42-1349795579:true workflow-21056-k6k42-1595781171:true workflow-21056-k6k42-2728974550:true workflow-21056-k6k42-3182910293:true workflow-21056-k6k42-3451334822:true workflow-21056-k6k42-4278140849:true]","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Updated phase Running -\u003e Succeeded","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Marking workflow completed","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"info","msg":"Marking workflow as pending archiving","namespace":"workflows","time":"2024-06-25T14:48:34.236Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537709\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowSucceeded' Workflow completed","time":"2024-06-25T14:48:34.236Z"}
{"level":"debug","msg":"Log changes patch: {\"metadata\":{\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Succeeded\",\"workflows.argoproj.io/workflow-archiving-status\":\"Pending\"}},\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"},{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2024-06-25T14:48:34Z\",\"nodes\":{\"workflow-21056-k6k42-1349795579\":{\"finishedAt\":\"2024-06-25T14:48:27Z\",\"phase\":\"Succeeded\",\"resourcesDuration\":{\"cpu\":0,\"memory\":3}}},\"phase\":\"Succeeded\"}}","time":"2024-06-25T14:48:34.237Z"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-1340600742-agent/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:34.242Z"}
{"level":"info","msg":"Workflow update successful","namespace":"workflows","phase":"Succeeded","resourceVersion":"257537770","time":"2024-06-25T14:48:34.256Z","workflow":"workflow-21056-k6k42"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"workflows\", Name:\"workflow-21056-k6k42\", UID:\"30ca6b42-885b-407c-86c1-f89f91d2059c\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"257537770\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node workflow-21056-k6k42.onExit","time":"2024-06-25T14:48:34.258Z"}
{"level":"info","msg":"Queueing Succeeded workflow workflows/workflow-21056-k6k42 for delete in 5m0s due to TTL","time":"2024-06-25T14:48:34.365Z"}
{"level":"info","msg":"archiving workflow","namespace":"workflows","time":"2024-06-25T14:48:34.462Z","uid":"30ca6b42-885b-407c-86c1-f89f91d2059c","workflow":"workflow-21056-k6k42"}
{"action":"deletePod","key":"workflows/workflow-21056-k6k42-exit-handler-1349795579/deletePod","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:36.462Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-51e19cea-5aa1-4088-8510-76395472ea67-3451334822/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:38.406Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8c2a4142-1f08-4997-a82e-5edfc2e8d6f0-3182910293/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:38.406Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-d6e520c4-8123-4169-b112-2167a2709b12-4278140849/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:38.474Z"}
{"key":"workflows/workflow-21056-k6k42","level":"debug","msg":"Won't process Workflow since it's completed","time":"2024-06-25T14:48:41.479Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-action-version-7917-task-8214a347-4a77-43a2-996e-57e4ab9e3da3-1595781171/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:48.587Z"}
{"action":"killContainers","key":"workflows/workflow-21056-k6k42-exit-handler-1349795579/killContainers","level":"info","msg":"cleaning up pod","time":"2024-06-25T14:48:58.712Z"}
{"level":"info","msg":"Deleting garbage collected workflow 'workflows/workflow-21056-k6k42'","time":"2024-06-25T14:53:35.000Z"}
{"level":"info","msg":"Workflow already deleted 'workflows/workflow-21056-k6k42'","time":"2024-06-25T14:53:35.017Z"}

@tooptoop4
Copy link
Contributor

@vermaxik do u have INFORMER_WRITE_BACK set to false by any chance?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/controller Controller issues, panics type/bug type/regression Regression from previous behavior (a specific type of bug)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Joibel @vermaxik @reisei @tooptoop4