Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v2.11: Steps (not DAG) always fails (with conflict error) and no helpful diagnostics (AWS only?) #4107

Closed
saranyaeu2987 opened this issue Sep 22, 2020 · 15 comments · Fixed by #4224
Closed

v2.11: Steps (not DAG) always fails (with conflict error) and no helpful diagnostics (AWS only?) #4107

saranyaeu2987 opened this issue Sep 22, 2020 · 15 comments · Fixed by #4224
Labels
type/bug
Milestone
v2.11

Comments

@saranyaeu2987
Copy link
Contributor

saranyaeu2987 commented Sep 22, 2020 •
edited
Loading

Summary

Trying Example https://argoproj.github.io/argo/examples/#output-parameters

Diagnostics

Argo version 2.11.0

2nd step(consume-parameter) failed because it was unable to get logs, while step 1(generate-parameter) was successful in pulling logs

Below are 2 API to made to get logs

http:https://internal-23e.us-east-1.elb.amazonaws.com:2746/artifacts/argo/output-parameters-wt294/output-parameters-wt294-3630452401/main-logs
(step : consume-parameter) Unsuccessful
Response: Artifact no found

http:https://internal-23e.us-east-1.elb.amazonaws.com:2746/artifacts/argo/output-parameters-wt294/output-parameters-wt294-2743065841/main-logs
(step : generate-parameter) Successful

Paste the workflow here, including status:
kubectl get wf -o yaml ${workflow} 

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2020-09-22T22:29:47Z"
  generateName: output-parameters-dummy-
  generation: 13
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Failed
  name: output-parameters-dummy-k28ls
  namespace: argo
  resourceVersion: "46048090"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/argo/workflows/output-parameters-dummy-k28ls
  uid: 590d6b5c-b75b-4d4f-a98d-8f88c7b1130f
spec:
  arguments: {}
  entrypoint: output-parameters
  templates:
  - arguments: {}
    inputs: {}
    metadata: {}
    name: output-parameters
    outputs: {}
    steps:
    - - arguments: {}
        name: generate-parameter
        template: whalesay
    - - arguments:
          parameters:
          - name: message
            value: '{{steps.generate-parameter.outputs.parameters.hello-param}}'
        name: consume-parameter
        template: print-message
    - - arguments: {}
        name: dummy
        template: dummy
  - arguments: {}
    container:
      args:
      - echo -n hello world > /tmp/hello_world.txt
      command:
      - sh
      - -c
      image: docker/whalesay:latest
      name: ""
      resources: {}
    inputs: {}
    metadata: {}
    name: whalesay
    outputs:
      parameters:
      - name: hello-param
        valueFrom:
          path: /tmp/hello_world.txt
  - arguments: {}
    container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - cowsay
      image: docker/whalesay:latest
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: message
    metadata: {}
    name: print-message
    outputs: {}
  - arguments: {}
    container:
      args:
      - test
      command:
      - echo
      image: selumalai/basicubuntu:latest
      name: ""
      resources: {}
    inputs: {}
    metadata: {}
    name: dummy
    outputs: {}
status:
  conditions:
  - status: "True"
    type: Completed
  finishedAt: "2020-09-22T22:31:00Z"
  message: child 'output-parameters-dummy-k28ls[1].consume-parameter' errored
  nodes:
    output-parameters-dummy-k28ls:
      children:
      - output-parameters-dummy-k28ls-898385737
      displayName: output-parameters-dummy-k28ls
      finishedAt: "2020-09-22T22:31:00Z"
      id: output-parameters-dummy-k28ls
      message: child 'output-parameters-dummy-k28ls[1].consume-parameter' errored
      name: output-parameters-dummy-k28ls
      phase: Failed
      startedAt: "2020-09-22T22:29:47Z"
      templateName: output-parameters
      templateScope: local/output-parameters-dummy-k28ls
      type: Steps
    output-parameters-dummy-k28ls-537766264:
      boundaryID: output-parameters-dummy-k28ls
      children:
      - output-parameters-dummy-k28ls-4052725204
      displayName: generate-parameter
      finishedAt: "2020-09-22T22:29:50Z"
      id: output-parameters-dummy-k28ls-537766264
      name: output-parameters-dummy-k28ls[0].generate-parameter
      outputs:
        artifacts:
        - archiveLogs: true
          name: main-logs
          s3:
            accessKeySecret:
              key: accesskey
              name: my-minio-cred
            bucket: my-bucket
            endpoint: minio:9000
            insecure: true
            key: output-parameters-dummy-k28ls/output-parameters-dummy-k28ls-537766264/main.log
            secretKeySecret:
              key: secretkey
              name: my-minio-cred
        parameters:
        - name: hello-param
          value: hello world
          valueFrom:
            path: /tmp/hello_world.txt
      phase: Succeeded
      startedAt: "2020-09-22T22:29:47Z"
      templateName: whalesay
      templateScope: local/output-parameters-dummy-k28ls
      type: Pod
    output-parameters-dummy-k28ls-898385737:
      boundaryID: output-parameters-dummy-k28ls
      children:
      - output-parameters-dummy-k28ls-537766264
      displayName: '[0]'
      finishedAt: "2020-09-22T22:29:51Z"
      id: output-parameters-dummy-k28ls-898385737
      name: output-parameters-dummy-k28ls[0]
      phase: Succeeded
      startedAt: "2020-09-22T22:29:47Z"
      templateName: output-parameters
      templateScope: local/output-parameters-dummy-k28ls
      type: StepGroup
    output-parameters-dummy-k28ls-4052725204:
      boundaryID: output-parameters-dummy-k28ls
      children:
      - output-parameters-dummy-k28ls-4217895726
      displayName: '[1]'
      finishedAt: "2020-09-22T22:31:00Z"
      id: output-parameters-dummy-k28ls-4052725204
      message: child 'output-parameters-dummy-k28ls[1].consume-parameter' errored
      name: output-parameters-dummy-k28ls[1]
      phase: Error
      startedAt: "2020-09-22T22:29:51Z"
      templateName: output-parameters
      type: StepGroup
    output-parameters-dummy-k28ls-4217895726:
      boundaryID: output-parameters-dummy-k28ls
      displayName: consume-parameter
      finishedAt: null
      id: output-parameters-dummy-k28ls-4217895726
      inputs:
        parameters:
        - name: message
          value: hello world
      name: output-parameters-dummy-k28ls[1].consume-parameter
      phase: Pending
      startedAt: "2020-09-22T22:31:00Z"
      templateName: print-message
      templateScope: local/output-parameters-dummy-k28ls
      type: Pod
  phase: Failed
  startedAt: "2020-09-22T22:29:47Z"


Paste the logs from the workflow controller:
kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name) | grep ${workflow}

 k kubectl logs -n argo $(k kubectl get pods -l app=workflow-controller -n argo -o name) | grep output-parameters-dummy-k28ls

 time="2020-09-22T22:29:47Z" level=info msg="Processing workflow" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:47Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:47Z" level=info msg="Steps node output-parameters-dummy-k28ls initialized Running" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:47Z" level=info msg="StepGroup node output-parameters-dummy-k28ls-898385737 initialized Running" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:47Z" level=info msg="Pod node output-parameters-dummy-k28ls-537766264 initialized Pending" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:47Z" level=info msg="Created pod: output-parameters-dummy-k28ls[0].generate-parameter (output-parameters-dummy-k28ls-537766264)" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:47Z" level=info msg="Workflow step group node output-parameters-dummy-k28ls-898385737 not yet completed" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:47Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46047464 workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:48Z" level=info msg="Processing workflow" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:48Z" level=info msg="Updating node output-parameters-dummy-k28ls-537766264 message: ContainerCreating"
time="2020-09-22T22:29:48Z" level=info msg="Skipped pod output-parameters-dummy-k28ls[0].generate-parameter (output-parameters-dummy-k28ls-537766264) creation: already exists" namespace=argo podPhase=Pending workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:48Z" level=info msg="Workflow step group node output-parameters-dummy-k28ls-898385737 not yet completed" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:48Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46047471 workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:49Z" level=info msg="Processing workflow" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:49Z" level=info msg="Skipped pod output-parameters-dummy-k28ls[0].generate-parameter (output-parameters-dummy-k28ls-537766264) creation: already exists" namespace=argo podPhase=Pending workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:49Z" level=info msg="Workflow step group node output-parameters-dummy-k28ls-898385737 not yet completed" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:49Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46047490 workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:50Z" level=info msg="Processing workflow" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:50Z" level=info msg="Workflow step group node output-parameters-dummy-k28ls-898385737 not yet completed" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:50Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46047498 workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:50Z" level=info msg="insignificant pod change" key=argo/output-parameters-dummy-k28ls-537766264
time="2020-09-22T22:29:51Z" level=info msg="Processing workflow" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:51Z" level=info msg="Workflow step group node output-parameters-dummy-k28ls-898385737 not yet completed" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:29:51Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46047512 workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Processing workflow" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="node output-parameters-dummy-k28ls-898385737 phase Succeeded -> Running" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Step group node &NodeStatus{ID:output-parameters-dummy-k28ls-898385737,Name:output-parameters-dummy-k28l[0],DisplayName:[0],Type:StepGroup,TemplateName:output-parameters,TemplateRef:nil,Phase:Running,BoundaryID:output-parameters-dummy-k28ls,Message:,StartedAt:2020-09-22 22:29:47 +0000 UTC,FinishedAt:2020-09-22 22:29:51 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[output-parameters-dummy-k28ls-537766264],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:local/output-parameters-dummy-k28ls,ResourcesDuration:ResourcesDuration{},HostNodeName:,MemoizationStatus:nil,} successful" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="node output-parameters-dummy-k28ls-898385737 phase Running -> Succeeded" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="SG Outbound nodes of output-parameters-dummy-k28ls-537766264 are [output-parameters-dummy-k28ls-537766264]" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Pod node output-parameters-dummy-k28ls-4217895726 initialized Pending" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Created pod: output-parameters-dummy-k28ls[1].consume-parameter (output-parameters-dummy-k28ls-4217895726)" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Workflow step group node output-parameters-dummy-k28ls-4052725204 not yet completed" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"output-parameters-dummy-k28ls\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Update retry attempt 1 successful" namespace=argo workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:00Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=46048090 workflow=output-parameters-dummy-k28ls
time="2020-09-22T22:31:03Z" level=info msg="insignificant pod change" key=argo/output-parameters-dummy-k28ls-4217895726

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@alexec
Copy link
Contributor

alexec commented Sep 23, 2020

Workflow does not have appeared to have finished. Pod node is still pending. Maybe attached logs from failed pod.

@saranyaeu2987
Copy link
Contributor Author

saranyaeu2987 commented Sep 23, 2020 •
edited
Loading

@alexec
Pod appears to be completed. But Argo says its still pending (but the step got failed due to failure to fetch logs)

Attached pod status from k8s dashboard

image

image

http:https://internal-23e.us-east-1.elb.amazonaws.com:2746/artifacts/argo/output-parameters-wt294/output-parameters-wt294-3630452401/main-logs
(step : consume-parameter) Unsuccessful
Response: Artifact no found

image

image

Logs from k8s pod
logs-from-main-in-output-parameters-dummy-k28ls-4217895726 - main.txt

logs-from-wait-in-output-parameters-dummy-k28ls-4217895726 - wait.txt

logs from argo workflow

argo -n argo logs output-parameters-dummy-k28ls                                                                         
output-parameters-dummy-k28ls-4217895726:  _____________ 
output-parameters-dummy-k28ls-4217895726: < hello world >
output-parameters-dummy-k28ls-4217895726:  ------------- 
output-parameters-dummy-k28ls-4217895726:     \
output-parameters-dummy-k28ls-4217895726:      \
output-parameters-dummy-k28ls-4217895726:       \     
output-parameters-dummy-k28ls-4217895726:                     ##        .            
output-parameters-dummy-k28ls-4217895726:               ## ## ##       ==            
output-parameters-dummy-k28ls-4217895726:            ## ## ## ##      ===            
output-parameters-dummy-k28ls-4217895726:        /""""""""""""""""___/ ===        
output-parameters-dummy-k28ls-4217895726:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
output-parameters-dummy-k28ls-4217895726:        \______ o          __/            
output-parameters-dummy-k28ls-4217895726:         \    \        __/             
output-parameters-dummy-k28ls-4217895726:           \____\______/

@saranyaeu2987
Copy link
Contributor Author

saranyaeu2987 commented Sep 23, 2020 •
edited
Loading

@alexec
It started happening for all of my workflows now where first step is successful but second step is suspended state as it was unable to pull logs. It happens only when workflow has steps.

@alexec alexec self-assigned this Sep 23, 2020
@alexec alexec added the P2 label Sep 23, 2020
@alexec alexec added this to the v2.11 milestone Sep 23, 2020
@alexec
Copy link
Contributor

alexec commented Sep 23, 2020

It is a bit unclear what happened here. Because of the conflict line in the logs, I think we had a race condition which resulted in invalid workflow state. You seem to say you can reproduce it? Can we jump on a Zoom?

@saranyaeu2987
Copy link
Contributor Author

saranyaeu2987 commented Sep 23, 2020 •
edited
Loading

@alexec Sure. Please let me know the zoom link

@alexec
Copy link
Contributor

alexec commented Sep 23, 2020

Note to self - we've had instances of errored pods going back into pending on certain providers.

@alexec
Copy link
Contributor

alexec commented Sep 23, 2020

@saranyaeu2987 ping me on Slack when you're ready.

@saranyaeu2987
Copy link
Contributor Author

@alexec
DMed on slack and in argo-workflows

@alexec alexec mentioned this issue Sep 23, 2020
Closed
@saranyaeu2987
Copy link
Contributor Author

saranyaeu2987 commented Sep 23, 2020 •
edited by alexec
Loading

`
❯ kubectl logs -n argo $( kubectl get pods -l app=workflow-controller -n argo -o name) | grep steps-nptfq

time="2020-09-23T23:31:47Z" level=info msg="Processing workflow" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Evaluating node steps-nptfq: template: *v1alpha1.WorkflowStep (hello-hello-hello), boundaryID: " namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Initializing node steps-nptfq: template: *v1alpha1.WorkflowStep (hello-hello-hello), boundaryID: " namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="Steps node steps-nptfq initialized Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Initializing node steps-nptfq[0]: template: *v1alpha1.Template (hello-hello-hello), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="StepGroup node steps-nptfq-3575724324 initialized Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Evaluating node steps-nptfq[0].hello1: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Initializing node steps-nptfq[0].hello1: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="Pod node steps-nptfq-838000797 initialized Pending" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:47Z" level=debug msg="Executing node steps-nptfq[0].hello1 with container template: &Template{Name:whalesay,Template:,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Inputs:Inputs{Parameters:[]Parameter{Parameter{Name:message,Default:nil,Value:*hello1,ValueFrom:nil,GlobalName:,},},Artifacts:[]Artifact{},},Outputs:Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{},Result:nil,ExitCode:nil,},NodeSelector:map[string]string{},Affinity:nil,Metadata:Metadata{Annotations:map[string]string{},Labels:map[string]string{},},Daemon:nil,Steps:[]ParallelSteps{},Container:&v1.Container{Name:,Image:docker/whalesay,Command:[cowsay],Args:[hello1],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:,ImagePullPolicy:,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},Script:nil,Resource:nil,DAG:nil,Suspend:nil,Volumes:[]Volume{},InitContainers:[]UserContainer{},Sidecars:[]UserContainer{},ArchiveLocation:nil,ActiveDeadlineSeconds:<nil>,RetryStrategy:nil,Parallelism:nil,Tolerations:[]Toleration{},SchedulerName:,PriorityClassName:,Priority:nil,ServiceAccountName:,HostAliases:[]HostAlias{},SecurityContext:nil,PodSpecPatch:,AutomountServiceAccountToken:nil,Executor:nil,ResubmitPendingPods:false,Metrics:nil,Synchronization:nil,Memoize:nil,Timeout:,}\n" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Creating Pod: steps-nptfq[0].hello1 (steps-nptfq-838000797)" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Setting s3 artifact repository information" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"argo\", Name:\"steps-nptfq\", UID:\"890ec74f-06fc-4b7d-b797-c62136fc82e7\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"46752633\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
time="2020-09-23T23:31:47Z" level=info msg="Failed pod steps-nptfq[0].hello1 (steps-nptfq-838000797) creation: already exists" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="Workflow step group node steps-nptfq-3575724324 not yet completed" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"steps-nptfq\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="Update retry attempt 1 successful" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:47Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46752642 workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=info msg="Processing workflow" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=debug msg="Evaluating node steps-nptfq: template: *v1alpha1.WorkflowStep (hello-hello-hello), boundaryID: " namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:48Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:48Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:48Z" level=debug msg="Executing node steps-nptfq of Steps is Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=debug msg="Evaluating node steps-nptfq[0].hello1: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:48Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:48Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:48Z" level=debug msg="Executing node steps-nptfq[0].hello1 of Pod is Pending" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:48Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:48Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:48Z" level=debug msg="Executing node steps-nptfq[0].hello1 with container template: &Template{Name:whalesay,Template:,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Inputs:Inputs{Parameters:[]Parameter{Parameter{Name:message,Default:nil,Value:*hello1,ValueFrom:nil,GlobalName:,},},Artifacts:[]Artifact{},},Outputs:Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{},Result:nil,ExitCode:nil,},NodeSelector:map[string]string{},Affinity:nil,Metadata:Metadata{Annotations:map[string]string{},Labels:map[string]string{},},Daemon:nil,Steps:[]ParallelSteps{},Container:&v1.Container{Name:,Image:docker/whalesay,Command:[cowsay],Args:[hello1],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:,ImagePullPolicy:,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},Script:nil,Resource:nil,DAG:nil,Suspend:nil,Volumes:[]Volume{},InitContainers:[]UserContainer{},Sidecars:[]UserContainer{},ArchiveLocation:nil,ActiveDeadlineSeconds:<nil>,RetryStrategy:nil,Parallelism:nil,Tolerations:[]Toleration{},SchedulerName:,PriorityClassName:,Priority:nil,ServiceAccountName:,HostAliases:[]HostAlias{},SecurityContext:nil,PodSpecPatch:,AutomountServiceAccountToken:nil,Executor:nil,ResubmitPendingPods:false,Metrics:nil,Synchronization:nil,Memoize:nil,Timeout:,}\n" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=debug msg="Creating Pod: steps-nptfq[0].hello1 (steps-nptfq-838000797)" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=debug msg="Setting s3 artifact repository information" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=info msg="Skipped pod steps-nptfq[0].hello1 (steps-nptfq-838000797) creation: already exists" namespace=argo podPhase=Pending workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=info msg="Workflow step group node steps-nptfq-3575724324 not yet completed" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:48Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46752657 workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=info msg="Processing workflow" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=debug msg="Evaluating node steps-nptfq: template: *v1alpha1.WorkflowStep (hello-hello-hello), boundaryID: " namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:49Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:49Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:49Z" level=debug msg="Executing node steps-nptfq of Steps is Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=debug msg="Evaluating node steps-nptfq[0].hello1: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:49Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:49Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:49Z" level=debug msg="Executing node steps-nptfq[0].hello1 of Pod is Pending" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:49Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:49Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:31:49Z" level=debug msg="Executing node steps-nptfq[0].hello1 with container template: &Template{Name:whalesay,Template:,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Inputs:Inputs{Parameters:[]Parameter{Parameter{Name:message,Default:nil,Value:*hello1,ValueFrom:nil,GlobalName:,},},Artifacts:[]Artifact{},},Outputs:Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{},Result:nil,ExitCode:nil,},NodeSelector:map[string]string{},Affinity:nil,Metadata:Metadata{Annotations:map[string]string{},Labels:map[string]string{},},Daemon:nil,Steps:[]ParallelSteps{},Container:&v1.Container{Name:,Image:docker/whalesay,Command:[cowsay],Args:[hello1],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:,ImagePullPolicy:,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},Script:nil,Resource:nil,DAG:nil,Suspend:nil,Volumes:[]Volume{},InitContainers:[]UserContainer{},Sidecars:[]UserContainer{},ArchiveLocation:nil,ActiveDeadlineSeconds:<nil>,RetryStrategy:nil,Parallelism:nil,Tolerations:[]Toleration{},SchedulerName:,PriorityClassName:,Priority:nil,ServiceAccountName:,HostAliases:[]HostAlias{},SecurityContext:nil,PodSpecPatch:,AutomountServiceAccountToken:nil,Executor:nil,ResubmitPendingPods:false,Metrics:nil,Synchronization:nil,Memoize:nil,Timeout:,}\n" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=debug msg="Creating Pod: steps-nptfq[0].hello1 (steps-nptfq-838000797)" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=debug msg="Setting s3 artifact repository information" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=info msg="Skipped pod steps-nptfq[0].hello1 (steps-nptfq-838000797) creation: already exists" namespace=argo podPhase=Pending workflow=steps-nptfq
time="2020-09-23T23:31:49Z" level=info msg="Workflow step group node steps-nptfq-3575724324 not yet completed" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:50Z" level=info msg="Processing workflow" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:50Z" level=debug msg="Evaluating node steps-nptfq: template: *v1alpha1.WorkflowStep (hello-hello-hello), boundaryID: " namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:50Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:50Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:50Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:50Z" level=debug msg="Executing node steps-nptfq of Steps is Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:50Z" level=debug msg="Evaluating node steps-nptfq[0].hello1: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:50Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:50Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:50Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:50Z" level=debug msg="Executing node steps-nptfq[0].hello1 of Pod is Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:50Z" level=info msg="Workflow step group node steps-nptfq-3575724324 not yet completed" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:50Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=46752681 workflow=steps-nptfq
time="2020-09-23T23:31:51Z" level=info msg="Processing workflow" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:51Z" level=debug msg="Evaluating node steps-nptfq: template: *v1alpha1.WorkflowStep (hello-hello-hello), boundaryID: " namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:51Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:51Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:51Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:31:51Z" level=debug msg="Executing node steps-nptfq of Steps is Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:51Z" level=debug msg="Evaluating node steps-nptfq[0].hello1: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:51Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:51Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:51Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:31:51Z" level=debug msg="Executing node steps-nptfq[0].hello1 of Pod is Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:31:51Z" level=info msg="Workflow step group node steps-nptfq-3575724324 not yet completed" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Processing workflow" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Evaluating node steps-nptfq: template: *v1alpha1.WorkflowStep (hello-hello-hello), boundaryID: " namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Executing node steps-nptfq of Steps is Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="node steps-nptfq-3575724324 phase Succeeded -> Running" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Evaluating node steps-nptfq[0].hello1: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Node steps-nptfq[0].hello1 already completed" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Step group node &NodeStatus{ID:steps-nptfq-3575724324,Name:steps-nptfq[0],DisplayName:[0],Type:StepGroup,TemplateName:hello-hello-hello,TemplateRef:nil,Phase:Running,BoundaryID:steps-nptfq,Message:,StartedAt:2020-09-23 23:31:47 +0000 UTC,FinishedAt:2020-09-23 23:31:53 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[steps-nptfq-838000797],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:local/steps-nptfq,ResourcesDuration:ResourcesDuration{},HostNodeName:,MemoizationStatus:nil,} successful" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="node steps-nptfq-3575724324 phase Running -> Succeeded" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="SG Outbound nodes of steps-nptfq-838000797 are [steps-nptfq-838000797]" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Evaluating node steps-nptfq[1].hello2a: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Initializing node steps-nptfq[1].hello2a: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Pod node steps-nptfq-1225588062 initialized Pending" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Executing node steps-nptfq[1].hello2a with container template: &Template{Name:whalesay,Template:,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Inputs:Inputs{Parameters:[]Parameter{Parameter{Name:message,Default:nil,Value:*hello2a,ValueFrom:nil,GlobalName:,},},Artifacts:[]Artifact{},},Outputs:Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{},Result:nil,ExitCode:nil,},NodeSelector:map[string]string{},Affinity:nil,Metadata:Metadata{Annotations:map[string]string{},Labels:map[string]string{},},Daemon:nil,Steps:[]ParallelSteps{},Container:&v1.Container{Name:,Image:docker/whalesay,Command:[cowsay],Args:[hello2a],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:,ImagePullPolicy:,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},Script:nil,Resource:nil,DAG:nil,Suspend:nil,Volumes:[]Volume{},InitContainers:[]UserContainer{},Sidecars:[]UserContainer{},ArchiveLocation:nil,ActiveDeadlineSeconds:<nil>,RetryStrategy:nil,Parallelism:nil,Tolerations:[]Toleration{},SchedulerName:,PriorityClassName:,Priority:nil,ServiceAccountName:,HostAliases:[]HostAlias{},SecurityContext:nil,PodSpecPatch:,AutomountServiceAccountToken:nil,Executor:nil,ResubmitPendingPods:false,Metrics:nil,Synchronization:nil,Memoize:nil,Timeout:,}\n" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Creating Pod: steps-nptfq[1].hello2a (steps-nptfq-1225588062)" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Setting s3 artifact repository information" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Created pod: steps-nptfq[1].hello2a (steps-nptfq-1225588062)" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Evaluating node steps-nptfq[1].hello2b: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.WorkflowStep (whalesay)"
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Initializing node steps-nptfq[1].hello2b: template: *v1alpha1.WorkflowStep (whalesay), boundaryID: steps-nptfq" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Pod node steps-nptfq-1208810443 initialized Pending" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=argo,name=steps-nptfq)" depth=0 tmpl="*v1alpha1.NodeStatus (hello-hello-hello)"
time="2020-09-23T23:32:03Z" level=debug msg="Executing node steps-nptfq[1].hello2b with container template: &Template{Name:whalesay,Template:,Arguments:Arguments{Parameters:[]Parameter{},Artifacts:[]Artifact{},},TemplateRef:nil,Inputs:Inputs{Parameters:[]Parameter{Parameter{Name:message,Default:nil,Value:*hello2b,ValueFrom:nil,GlobalName:,},},Artifacts:[]Artifact{},},Outputs:Outputs{Parameters:[]Parameter{},Artifacts:[]Artifact{},Result:nil,ExitCode:nil,},NodeSelector:map[string]string{},Affinity:nil,Metadata:Metadata{Annotations:map[string]string{},Labels:map[string]string{},},Daemon:nil,Steps:[]ParallelSteps{},Container:&v1.Container{Name:,Image:docker/whalesay,Command:[cowsay],Args:[hello2b],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},},VolumeMounts:[]VolumeMount{},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:,ImagePullPolicy:,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,},Script:nil,Resource:nil,DAG:nil,Suspend:nil,Volumes:[]Volume{},InitContainers:[]UserContainer{},Sidecars:[]UserContainer{},ArchiveLocation:nil,ActiveDeadlineSeconds:<nil>,RetryStrategy:nil,Parallelism:nil,Tolerations:[]Toleration{},SchedulerName:,PriorityClassName:,Priority:nil,ServiceAccountName:,HostAliases:[]HostAlias{},SecurityContext:nil,PodSpecPatch:,AutomountServiceAccountToken:nil,Executor:nil,ResubmitPendingPods:false,Metrics:nil,Synchronization:nil,Memoize:nil,Timeout:,}\n" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Creating Pod: steps-nptfq[1].hello2b (steps-nptfq-1208810443)" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=debug msg="Setting s3 artifact repository information" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Created pod: steps-nptfq[1].hello2b (steps-nptfq-1208810443)" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Workflow step group node steps-nptfq-421384857 not yet completed" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"steps-nptfq\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Update retry attempt 1 successful" namespace=argo workflow=steps-nptfq
time="2020-09-23T23:32:03Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=46752806 workflow=steps-nptfq

`

@alexec alexec changed the title Argo unable to pull log for a step v2.1: Steps (not DAG) always fails (with conflict error ) and no helpful diagnostics (AWS only?) Sep 23, 2020
@alexec alexec changed the title v2.1: Steps (not DAG) always fails (with conflict error ) and no helpful diagnostics (AWS only?) v2.1: Steps (not DAG) always fails (with conflict error) and no helpful diagnostics (AWS only?) Sep 23, 2020
@alexec
Copy link
Contributor

alexec commented Sep 23, 2020

Maybe related: #4058

@alexec
Copy link
Contributor

alexec commented Sep 23, 2020

Notes:

  • Occurs when running controllers: v2.8.2 and v2.10.2.
  • Cluster has mutatingwebhookconfigurations installed.

@alexec
Copy link
Contributor

alexec commented Sep 24, 2020

Created a dev image argoproj/workflow-controller:fix-4701-1 that added a 10s wait before processing workflow. This prevented this issue.

@alexec
Copy link
Contributor

alexec commented Sep 24, 2020

Note: while this dev image fixed this specific issue, I still don't know why this only affected this steps workflow. Why not others? Why at all?

@alexec alexec mentioned this issue Sep 24, 2020
Open
@alexec
Copy link
Contributor

alexec commented Sep 24, 2020 •
edited
Loading

image

Based on kubectl get wf--watch, I can see the following changes:

[
{"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"creationTimestamp":"2020-09-24T00:40:28Z","generateName":"steps-","generation":4,"labels":{"workflows.argoproj.io/phase":"Running"},"name":"steps-9khx2","namespace":"argo","resourceVersion":"46785889","selfLink":"/apis/argoproj.io/v1alpha1/namespaces/argo/workflows/steps-9khx2","uid":"b687e772-f844-4268-9fcf-04ae43c400cb"},"spec":{"arguments":{},"entrypoint":"hello-hello-hello","templates":[{"arguments":{},"inputs":{},"metadata":{},"name":"hello-hello-hello","outputs":{},"steps":[[{"arguments":{"parameters":[{"name":"message","value":"hello1"}]},"name":"hello1","template":"whalesay"}],[{"arguments":{"parameters":[{"name":"message","value":"hello2a"}]},"name":"hello2a","template":"whalesay"},{"arguments":{"parameters":[{"name":"message","value":"hello2b"}]},"name":"hello2b","template":"whalesay"}]]},{"arguments":{},"container":{"args":["{{inputs.parameters.message}}"],"command":["cowsay"],"image":"docker/whalesay","name":"","resources":{}},"inputs":{"parameters":[{"name":"message"}]},"metadata":{},"name":"whalesay","outputs":{}}]},"status":{"finishedAt":null,"nodes":{"steps-9khx2":{"children":["steps-9khx2-2475991523"],"displayName":"steps-9khx2","finishedAt":null,"id":"steps-9khx2","name":"steps-9khx2","phase":"Running","startedAt":"2020-09-24T00:40:28Z","templateName":"hello-hello-hello","type":"Steps"},"steps-9khx2-2475991523":{"boundaryID":"steps-9khx2","children":["steps-9khx2-3689613236"],"displayName":"[0]","finishedAt":null,"id":"steps-9khx2-2475991523","name":"steps-9khx2[0]","phase":"Running","startedAt":"2020-09-24T00:40:28Z","templateName":"hello-hello-hello","type":"StepGroup"},"steps-9khx2-3689613236":{"boundaryID":"steps-9khx2","displayName":"hello1","finishedAt":null,"hostNodeName":"ip-10-207-196-131.ec2.internal","id":"steps-9khx2-3689613236","inputs":{"parameters":[{"name":"message","value":"hello1"}]},"message":"ContainerCreating","name":"steps-9khx2[0].hello1","phase":"Pending","startedAt":"2020-09-24T00:40:28Z","templateName":"whalesay","type":"Pod"}},"phase":"Running","startedAt":"2020-09-24T00:40:28Z"}},
{"metadata":{"generation":5,"resourceVersion":"46785918"},"status":{"nodes":{"steps-9khx2-3689613236":{"hostNodeName":null,"message":null,"phase":"Running"}}}},
{"metadata":{"generation":6,"resourceVersion":"46785925"},"status":{"nodes":{"steps-9khx2-3689613236":{"hostNodeName":"ip-10-207-196-131.ec2.internal"}}}},
{"metadata":{"generation":7,"resourceVersion":"46785941"},"status":{"nodes":{"steps-9khx2-2475991523":{"finishedAt":"2020-09-24T00:40:34Z","phase":"Succeeded"},"steps-9khx2-2543249094":{"boundaryID":"steps-9khx2","children":["steps-9khx2-3046931061","steps-9khx2-2996598204"],"displayName":"[1]","finishedAt":null,"id":"steps-9khx2-2543249094","name":"steps-9khx2[1]","phase":"Running","startedAt":"2020-09-24T00:40:34Z","templateName":"hello-hello-hello","type":"StepGroup"},"steps-9khx2-2996598204":{"boundaryID":"steps-9khx2","displayName":"hello2b","finishedAt":null,"id":"steps-9khx2-2996598204","inputs":{"parameters":[{"name":"message","value":"hello2b"}]},"name":"steps-9khx2[1].hello2b","phase":"Pending","startedAt":"2020-09-24T00:40:34Z","templateName":"whalesay","type":"Pod"},"steps-9khx2-3046931061":{"boundaryID":"steps-9khx2","displayName":"hello2a","finishedAt":null,"id":"steps-9khx2-3046931061","inputs":{"parameters":[{"name":"message","value":"hello2a"}]},"name":"steps-9khx2[1].hello2a","phase":"Pending","startedAt":"2020-09-24T00:40:34Z","templateName":"whalesay","type":"Pod"},"steps-9khx2-3689613236":{"children":["steps-9khx2-2543249094"],"finishedAt":"2020-09-24T00:40:32Z","hostNodeName":null,"phase":"Succeeded"}}}},
{"metadata":{"generation":8,"resourceVersion":"46785950"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":"ip-10-207-196-131.ec2.internal"},"steps-9khx2-3046931061":{"hostNodeName":"ip-10-207-196-131.ec2.internal"}}}},
{"metadata":{"generation":9,"resourceVersion":"46785954"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":null,"message":"ContainerCreating"},"steps-9khx2-3046931061":{"hostNodeName":null,"message":"ContainerCreating"}}}},
{"metadata":{"generation":10,"resourceVersion":"46785964"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":"ip-10-207-196-131.ec2.internal"},"steps-9khx2-3046931061":{"hostNodeName":"ip-10-207-196-131.ec2.internal"}}}},
{"metadata":{"generation":11,"resourceVersion":"46785965"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":null},"steps-9khx2-3046931061":{"hostNodeName":null}}}},
{"metadata":{"generation":12,"resourceVersion":"46786007"},"status":{"nodes":{"steps-9khx2-3046931061":{"message":null,"phase":"Running"}}}},
{"metadata":{"generation":13,"resourceVersion":"46786013"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":"ip-10-207-196-131.ec2.internal"},"steps-9khx2-3046931061":{"hostNodeName":"ip-10-207-196-131.ec2.internal"}}}},
{"metadata":{"generation":14,"resourceVersion":"46786014"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":null},"steps-9khx2-3046931061":{"hostNodeName":null}}}},
{"metadata":{"generation":15,"resourceVersion":"46786026"},"status":{"nodes":{"steps-9khx2-2996598204":{"message":null,"phase":"Running"}}}},
{"metadata":{"generation":16,"resourceVersion":"46786029"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":"ip-10-207-196-131.ec2.internal"},"steps-9khx2-3046931061":{"hostNodeName":"ip-10-207-196-131.ec2.internal"}}}},
{"metadata":{"generation":17,"resourceVersion":"46786034"},"status":{"nodes":{"steps-9khx2-2996598204":{"hostNodeName":null},"steps-9khx2-3046931061":{"hostNodeName":null}}}},
{"metadata":{"generation":18,"labels":{"workflows.argoproj.io/completed":"true","workflows.argoproj.io/phase":"Succeeded"},"resourceVersion":"46786047"},"status":{"conditions":[{"status":"True","type":"Completed"}],"finishedAt":"2020-09-24T00:40:44Z","nodes":{"steps-9khx2":{"finishedAt":"2020-09-24T00:40:44Z","outboundNodes":["steps-9khx2-3046931061","steps-9khx2-2996598204"],"phase":"Succeeded"},"steps-9khx2-2543249094":{"finishedAt":"2020-09-24T00:40:44Z","phase":"Succeeded"},"steps-9khx2-2996598204":{"finishedAt":"2020-09-24T00:40:41Z","phase":"Succeeded"},"steps-9khx2-3046931061":{"finishedAt":"2020-09-24T00:40:40Z","phase":"Succeeded"}},"phase":"Succeeded"}},
{"metadata":{"generation":19,"resourceVersion":"46786052"},"status":{"nodes":{"steps-9khx2-2996598204":{"resourcesDuration":{"cpu":4,"memory":4}},"steps-9khx2-3046931061":{"resourcesDuration":{"cpu":4,"memory":4}}},"resourcesDuration":{"cpu":8,"memory":8}}},
]

@alexec
Copy link
Contributor

alexec commented Sep 24, 2020 •
edited
Loading

From busted version, why does generation 9 erase everything?

[
{"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"creationTimestamp":"2020-09-24T21:10:27Z","generateName":"steps-","generation":3,"labels":{"workflows.argoproj.io/phase":"Running"},"name":"steps-vqqhb","namespace":"argo","resourceVersion":"47362772","selfLink":"/apis/argoproj.io/v1alpha1/namespaces/argo/workflows/steps-vqqhb","uid":"686f3ff3-6374-4f2f-96d1-5aaabe86f4a1"},"spec":{"arguments":{},"entrypoint":"hello-hello-hello","templates":[{"arguments":{},"inputs":{},"metadata":{},"name":"hello-hello-hello","outputs":{},"steps":[[{"arguments":{"parameters":[{"name":"message","value":"hello1"}]},"name":"hello1","template":"whalesay"}],[{"arguments":{"parameters":[{"name":"message","value":"hello2a"}]},"name":"hello2a","template":"whalesay"},{"arguments":{"parameters":[{"name":"message","value":"hello2b"}]},"name":"hello2b","template":"whalesay"}]]},{"arguments":{},"container":{"args":["{{inputs.parameters.message}}"],"command":["cowsay"],"image":"docker/whalesay","name":"","resources":{}},"inputs":{"parameters":[{"name":"message"}]},"metadata":{},"name":"whalesay","outputs":{}}]},"status":{"finishedAt":null,"nodes":{"steps-vqqhb":{"children":["steps-vqqhb-1336869503"],"displayName":"steps-vqqhb","finishedAt":null,"id":"steps-vqqhb","name":"steps-vqqhb","phase":"Running","startedAt":"2020-09-24T21:10:27Z","templateName":"hello-hello-hello","templateScope":"local/steps-vqqhb","type":"Steps"},"steps-vqqhb-1278686064":{"boundaryID":"steps-vqqhb","displayName":"hello1","finishedAt":null,"hostNodeName":"ip-10-207-196-131.ec2.internal","id":"steps-vqqhb-1278686064","inputs":{"parameters":[{"name":"message","value":"hello1"}]},"message":"ContainerCreating","name":"steps-vqqhb[0].hello1","phase":"Pending","startedAt":"2020-09-24T21:10:27Z","templateName":"whalesay","templateScope":"local/steps-vqqhb","type":"Pod"},"steps-vqqhb-1336869503":{"boundaryID":"steps-vqqhb","children":["steps-vqqhb-1278686064"],"displayName":"[0]","finishedAt":null,"id":"steps-vqqhb-1336869503","name":"steps-vqqhb[0]","phase":"Running","startedAt":"2020-09-24T21:10:27Z","templateName":"hello-hello-hello","templateScope":"local/steps-vqqhb","type":"StepGroup"}},"phase":"Running","startedAt":"2020-09-24T21:10:27Z"}},
{"metadata":{"generation":4,"resourceVersion":"47362809"},"status":{"nodes":{"steps-vqqhb-1278686064":{"hostNodeName":null,"message":null,"phase":"Running"}}}},
{"metadata":{"generation":5,"resourceVersion":"47362810"},"status":{"nodes":{"steps-vqqhb-1278686064":{"hostNodeName":"ip-10-207-196-131.ec2.internal"}}}},
{"metadata":{"generation":6,"resourceVersion":"47362818"},"status":{"nodes":{"steps-vqqhb-1278686064":{"hostNodeName":null,"outputs":{"artifacts":[{"archiveLogs":true,"name":"main-logs","s3":{"accessKeySecret":{"key":"accesskey","name":"my-minio-cred"},"bucket":"my-bucket","endpoint":"minio:9000","insecure":true,"key":"steps-vqqhb/steps-vqqhb-1278686064/main.log","secretKeySecret":{"key":"secretkey","name":"my-minio-cred"}}}]}}}}},
{"metadata":{"generation":7,"resourceVersion":"47362819"},"status":{"nodes":{"steps-vqqhb-1278686064":{"hostNodeName":"ip-10-207-196-131.ec2.internal"}}}},
{"metadata":{"generation":8,"labels":{"workflows.argoproj.io/completed":"true","workflows.argoproj.io/phase":"Failed"},"resourceVersion":"47362836"},"status":{"conditions":[{"status":"True","type":"Completed"}],"finishedAt":"2020-09-24T21:10:34Z","message":"child 'steps-vqqhb[1].hello2a' errored","nodes":{"steps-vqqhb":{"finishedAt":"2020-09-24T21:10:34Z","message":"child 'steps-vqqhb[1].hello2a' errored","phase":"Failed"},"steps-vqqhb-1278686064":{"children":["steps-vqqhb-330359458"],"finishedAt":"2020-09-24T21:10:32Z","hostNodeName":null,"phase":"Succeeded"},"steps-vqqhb-1336869503":{"finishedAt":"2020-09-24T21:10:34Z","phase":"Succeeded"},"steps-vqqhb-330359458":{"boundaryID":"steps-vqqhb","children":["steps-vqqhb-3815304425"],"displayName":"[1]","finishedAt":"2020-09-24T21:10:34Z","id":"steps-vqqhb-330359458","message":"child 'steps-vqqhb[1].hello2a' errored","name":"steps-vqqhb[1]","phase":"Error","startedAt":"2020-09-24T21:10:34Z","templateName":"hello-hello-hello","type":"StepGroup"}},"phase":"Failed"}},
{"metadata":{"generation":9,"labels":{"workflows.argoproj.io/completed":null,"workflows.argoproj.io/phase":"Running"},"resourceVersion":"47363007"},"status":{"conditions":[{"status":"False","type":"Completed"}],"finishedAt":null,"message":null,"nodes":{"steps-vqqhb":{"finishedAt":null,"message":null,"phase":"Running"},"steps-vqqhb-330359458":{"finishedAt":null,"message":null,"phase":"Running"}},"phase":"Running"}},
{"metadata":{"generation":10,"labels":{"workflows.argoproj.io/completed":"true","workflows.argoproj.io/phase":"Failed"},"resourceVersion":"47363010"},"status":{"conditions":[{"status":"True","type":"Completed"}],"finishedAt":"2020-09-24T21:10:57Z","message":"child 'steps-vqqhb[1].hello2a' errored","nodes":{"steps-vqqhb":{"finishedAt":"2020-09-24T21:10:57Z","message":"child 'steps-vqqhb[1].hello2a' errored","phase":"Failed"},"steps-vqqhb-330359458":{"finishedAt":"2020-09-24T21:10:57Z","message":"child 'steps-vqqhb[1].hello2a' errored","phase":"Error"}},"phase":"Failed"}},
{"metadata":{"generation":11,"resourceVersion":"47363017"},"status":{"nodes":{"steps-vqqhb-330359458":{"children":["steps-vqqhb-3815304425","steps-vqqhb-3764971568"]},"steps-vqqhb-3764971568":{"boundaryID":"steps-vqqhb","displayName":"hello2b","finishedAt":null,"id":"steps-vqqhb-3764971568","inputs":{"parameters":[{"name":"message","value":"hello2b"}]},"name":"steps-vqqhb[1].hello2b","phase":"Pending","startedAt":"2020-09-24T21:10:57Z","templateName":"whalesay","templateScope":"local/steps-vqqhb","type":"Pod"},"steps-vqqhb-3815304425":{"boundaryID":"steps-vqqhb","displayName":"hello2a","finishedAt":null,"id":"steps-vqqhb-3815304425","inputs":{"parameters":[{"name":"message","value":"hello2a"}]},"name":"steps-vqqhb[1].hello2a","phase":"Pending","startedAt":"2020-09-24T21:10:57Z","templateName":"whalesay","templateScope":"local/steps-vqqhb","type":"Pod"}}}},
]

@alexec alexec removed their assignment Oct 2, 2020
@alexec alexec mentioned this issue Oct 6, 2020
Merged
6 tasks
@alexec alexec changed the title v2.1: Steps (not DAG) always fails (with conflict error) and no helpful diagnostics (AWS only?) v2.11: Steps (not DAG) always fails (with conflict error) and no helpful diagnostics (AWS only?) Oct 9, 2020
alexec pushed a commit that referenced this issue Oct 9, 2020
@saranyaeu2987
fix(controller): Make the delay before the first workflow reconciliat…
6ce6bf4 
…ion configurable. Fixes #4107 (#4224)
alexcapras pushed a commit to alexcapras/argo that referenced this issue Nov 12, 2020
@saranyaeu2987 @alexcapras
fix(controller): Make the delay before the first workflow reconciliat…
12b3446 
…ion configurable. Fixes argoproj#4107 (argoproj#4224)

Signed-off-by: Alex Capras <[email protected]>
@saranyaeu2987 saranyaeu2987 mentioned this issue Jan 3, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
v2.11
2 participants
@alexec @saranyaeu2987