persistentvolumeclaims already exists

[canadiannomad]

BUG REPORT

What happened:

$ argo submit --watch generic.yaml -n argo
Name:                generic-nf42h
Namespace:           argo
ServiceAccount:      default
Status:              Error
Message:             persistentvolumeclaims "generic-nf42h-workdir" already exists
Created:             Fri Dec 14 12:16:30 -0500 (1 second ago)
Started:             Fri Dec 14 12:16:30 -0500 (1 second ago)
Finished:            Fri Dec 14 12:16:30 -0500 (1 second ago)
Duration:            0 seconds
Parameters:
  revision:          master

STEP              PODNAME                  DURATION  MESSAGE
 ● generic-nf42h
 └---◷ provision  generic-nf42h-319909044  1s

What you expected to happen:
generic-nf42h-workdir didn't exist before running, and exists after running.. There shouldn't have been an error.

How to reproduce it (as minimally and precisely as possible):
I'm thinking this has to do with me running a bare metal kubernetes cluster with rook.. But I can't find any logs that clarify what is going wrong.

Anything else we need to know?:
The first step completes successfully but if I have a second step it never reaches it.

Environment:

• Argo version:

$ argo version
argo: v2.2.1
  BuildDate: 2018-10-11T16:25:59Z
  GitCommit: 3b52b26190163d1f72f3aef1a39f9f291378dafb
  GitTreeState: clean
  GitTag: v2.2.1
  GoVersion: go1.10.3
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: 2018-10-24T06:54:59Z
  compiler: gc
  gitCommit: 17c77c7898218073f14c8d573582e8d2313dc740
  gitTreeState: clean
  gitVersion: v1.12.2
  goVersion: go1.10.4
  major: "1"
  minor: "12"
  platform: darwin/amd64
serverVersion:
  buildDate: 2018-12-03T20:56:12Z
  compiler: gc
  gitCommit: ddf47ac13c1a9483ea035a79cd7c10005ff21a6d
  gitTreeState: clean
  gitVersion: v1.13.0
  goVersion: go1.11.2
  major: "1"
  minor: "13"
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

$ argo get <workflowname>
NAME            STATUS   AGE   DURATION
generic-9rq8z   Error    10s   0s

• executor logs:

$ kubectl logs <failedpodname> -c main
total 20
drwxr-xr-x    3 root     root          4096 Dec 14 17:33 .
drwxr-xr-x    1 root     root            28 Dec 14 17:33 ..
-rw-r--r--    1 root     root             0 Dec 14 17:33 file
drwx------    2 root     root         16384 Dec 14 17:33 lost+found
$ kubectl logs <failedpodname> -c wait
time="2018-12-14T17:33:22Z" level=info msg="Creating a docker executor"
time="2018-12-14T17:33:22Z" level=info msg="Executor (version: v2.2.0, build_date: 2018-08-30T08:52:54Z) initialized with template:\narchiveLocation: {}\ncontainer:\n  args:\n  - touch file; ls -al\n  command:\n  - sh\n  - -c\n  image: alpine/git\n  name: \"\"\n  resources: {}\n  volumeMounts:\n  - mountPath: /src\n    name: workdir\n  workingDir: /src\ninputs: {}\nmetadata: {}\nname: git-clone\noutputs: {}\n"
time="2018-12-14T17:33:22Z" level=info msg="Waiting on main container"
time="2018-12-14T17:33:23Z" level=info msg="main container started with container ID: 328ceb2261c0861eb60763c8531a9764a78898b1f715f71398586bd461606a43"
time="2018-12-14T17:33:23Z" level=info msg="Starting annotations monitor"
time="2018-12-14T17:33:23Z" level=info msg="docker wait 328ceb2261c0861eb60763c8531a9764a78898b1f715f71398586bd461606a43"
time="2018-12-14T17:33:23Z" level=info msg="Starting deadline monitor"
time="2018-12-14T17:33:23Z" level=info msg="Main container completed"
time="2018-12-14T17:33:23Z" level=info msg="No sidecars"
time="2018-12-14T17:33:23Z" level=info msg="No output artifacts"
time="2018-12-14T17:33:23Z" level=info msg="No output parameters"
time="2018-12-14T17:33:23Z" level=info msg="Annotations monitor stopped"
time="2018-12-14T17:33:23Z" level=info msg="Alloc=4412 TotalAlloc=11248 Sys=10086 NumGC=4 Goroutines=9"

• workflow-controller logs:

$ kubectl logs -n kube-system $(kubectl get pods -l app=argo-workflow-controller -n kube-system -o name)
time="2018-12-14T17:33:18Z" level=info msg="Processing workflow" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Creating pvc generic-9rq8z-workdir" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=error msg="generic-9rq8z pvc create error: persistentvolumeclaims \"generic-9rq8z-workdir\" already exists" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Updated message  -> persistentvolumeclaims \"generic-9rq8z-workdir\" already exists" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Marking workflow completed" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"generic-9rq8z\": the object has been modified; please apply your changes to the latest version and try again" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Re-appying updates on latest version and retrying update" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Update retry attempt 1 successful" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:18Z" level=info msg="Workflow update successful" namespace=argo workflow=generic-9rq8z
time="2018-12-14T17:33:39Z" level=info msg="Alloc=4414 TotalAlloc=35407 Sys=16650 NumGC=39 Goroutines=62"

Workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: generic-
spec:
  entrypoint: runtest
  arguments:
    parameters:
    - name: revision
      value: master
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
  templates:
  - name: runtest
    steps:
    - - name: provision
        template: git-clone

  - name: git-clone
    container:
      image: alpine/git
      workingDir: /src
      command: [sh, -c]
      args: [ "touch file; ls -al" ]
      volumeMounts:
      - name: workdir
        mountPath: /src

[canadiannomad]

Any ideas on where I can look to find out why it thinks the persistentvolumeclaim already exists despite it having created it? I'm still having this issue.

[alexmt]

Hello @canadiannomad . Thank you for providing sample workflow! I've tried to run it but could not reproduce issue.

I would suggest to watches for PVC using kubectl get pvc -w and start workflow in another terminal window. Before starting workflow there should be no PVCs with name generic- and after workflow started PVC should be created and deleted. Can you please attach kubectl get pvc -w output and controller logs?

[pplavetzki]

I am also receiving this error. Here is the output of the command kubectl get pvc -w just after running argo submit hello-world.yaml:

hello-world-bgmwd-service-data                                  Pending                                                                        standard       <invalid>
hello-world-bgmwd-service-data                                  Pending                                                                        standard       <invalid>
hello-world-bgmwd-service-data                                  Pending                                                                        standard       <invalid>
hello-world-bgmwd-service-data                                  Pending   pvc-c2f7d1e6-5cc4-11e9-abb7-0050569aba93   0                         standard       <invalid>
hello-world-bgmwd-service-data                                  Bound     pvc-c2f7d1e6-5cc4-11e9-abb7-0050569aba93   200Mi      RWO            standard       <invalid>

Controller Log:

time="2019-04-12T01:46:20Z" level=info msg="Processing workflow" namespace=jedi-operator workflow=hello-world-bgmwd
time="2019-04-12T01:46:20Z" level=info msg="Updated phase  -> Running" namespace=jedi-operator workflow=hello-world-bgmwd
time="2019-04-12T01:46:20Z" level=info msg="Creating pvc hello-world-bgmwd-service-data" namespace=jedi-operator workflow=hello-world-bgmwd
time="2019-04-12T01:46:21Z" level=info msg="Created pod: hello-world-bgmwd (hello-world-bgmwd)" namespace=jedi-operator workflow=hello-world-bgmwd
time="2019-04-12T01:46:21Z" level=info msg="Pod node hello-world-bgmwd (hello-world-bgmwd) initialized Pending" namespace=jedi-operator workflow=hello-world-bgmwd
time="2019-04-12T01:46:21Z" level=info msg="Workflow update successful" namespace=jedi-operator workflow=hello-world-bgmwd

What Happened:

Name:                hello-world-bgmwd
Namespace:           jedi-operator
ServiceAccount:      jedi-operator
Status:              Error
Message:             persistentvolumeclaims "hello-world-bgmwd-service-data" already exists
Created:             Thu Apr 11 18:46:16 -0700 (10 minutes from now)
Started:             Thu Apr 11 18:23:07 -0700 (12 minutes ago)
Finished:            Thu Apr 11 18:23:07 -0700 (12 minutes ago)
Duration:            0 seconds

STEP                  PODNAME            DURATION  MESSAGE
 ◷ hello-world-bgmwd  hello-world-bgmwd  10m 

Environment:

argo: v2.2.0
  BuildDate: 2018-08-30T08:51:40Z
  GitCommit: af636ddd8455660f307d835814d3112b90815dfd
  GitTreeState: clean
  GitTag: v2.2.0
  GoVersion: go1.10.3
  Compiler: gc
  Platform: darwin/amd64

Workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
spec:
  entrypoint: hello
  serviceAccountName: jedi-operator
  volumeClaimTemplates:
  - metadata:
      name: service-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: standard
      resources:
        requests:
          storage: 200Mi
  templates:
  - name: hello
    container:
      image: alpine
      command: ['/bin/sh', '-c']
      args: ['sleep 3;echo hello world']
      volumeMounts:
      - name: service-data
        mountPath: /var/data

K8S Version:

clientVersion:
  buildDate: "2019-03-25T15:53:57Z"
  compiler: gc
  gitCommit: 641856db18352033a0d96dbc99153fa3b27298e5
  gitTreeState: clean
  gitVersion: v1.14.0
  goVersion: go1.12.1
  major: "1"
  minor: "14"
  platform: darwin/amd64
serverVersion:
  buildDate: "2018-09-09T17:53:03Z"
  compiler: gc
  gitCommit: a4529464e4629c21224b3d52edfe0ea91b072862
  gitTreeState: clean
  gitVersion: v1.11.3
  goVersion: go1.10.3
  major: "1"
  minor: "11"
  platform: linux/amd64

Other Info:
This is using argo in a namespaced deployment without a cluster role:

Name:         argo-role
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","metadata":{"annotations":{},"name":"argo-role","namespace":"jedi-operator"},"r...
PolicyRule:
  Resources                         Non-Resource URLs  Resource Names  Verbs
  ---------                         -----------------  --------------  -----
  persistentvolumeclaims            []                 []              [create delete]
  pods/exec                         []                 []              [create get list watch update patch delete]
  pods                              []                 []              [create get list watch update patch delete]
  workflows.argoproj.io/finalizers  []                 []              [get list watch update patch delete]
  workflows.argoproj.io             []                 []              [get list watch update patch delete]
  configmaps                        []                 []              [get watch list]

Storage Class Info:

Name:            standard
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"standard"},"parameters":{"diskformat":"thin"},"provisioner":"kubernetes.io/vsphere-volume"}

Provisioner:           kubernetes.io/vsphere-volume
Parameters:            diskformat=thin
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
Events:                <none>

[jessesuen]

The fix is we need to be idempotent about pvc creations. The following logic does not tolerate when we have already created the PVC:

		pvc, err := pvcClient.Create(&pvcTmpl)
		if err != nil {
			return err
		}
		vol := apiv1.Volume{
			Name: refName,
			VolumeSource: apiv1.VolumeSource{
				PersistentVolumeClaim: &apiv1.PersistentVolumeClaimVolumeSource{
					ClaimName: pvc.ObjectMeta.Name,
				},
			},
		}
		woc.wf.Status.PersistentVolumeClaims[i] = vol

The fix should be if we attempt to create the PVC and it gets a AlreadyExists error, AND the existing PVC has our ownership reference, then we can ignore the error.

[audriusrudalevicius]

Im still getting persistentvolumeclaims "ml-work-rdbcq-workdir-1" already exists 1/10 time when i run my workflow.

What i want is to pre-download files before processing them, do it in parallel

argo get ml-work-rdbcq
Name:                ml-work-rdbcq
Namespace:           default
ServiceAccount:      default
Status:              Running
Message:             persistentvolumeclaims "ml-work-rdbcq-workdir-1" already exists
Created:             Thu Jan 23 14:52:13 +0200 (1 minute ago)
Started:             Thu Jan 23 14:52:13 +0200 (1 minute ago)
Finished:            Thu Jan 23 14:52:13 +0200 (1 minute ago)
Duration:            0 seconds
Parameters:
  parallel:          6

STEP                                          PODNAME                   DURATION  MESSAGE
 ● ml-work-rdbcq (download-images)
 └-·-◷ download-images-shard(0:1) (download)  ml-work-rdbcq-4015706135  1m
   ├-◷ download-images-shard(1:2) (download)  ml-work-rdbcq-2730545253  1m
   ├-◷ download-images-shard(2:3) (download)  ml-work-rdbcq-4012591687  1m
   ├-◷ download-images-shard(3:4) (download)  ml-work-rdbcq-232276801   1m
   ├-◷ download-images-shard(4:5) (download)  ml-work-rdbcq-1959765255  1m
   └-◷ download-images-shard(5:6) (download)  ml-work-rdbcq-1994131533  1m

Pvc:

 kubectl get pvc
NAME                              STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ml-work-rdbcq-workdir-1           Bound         pvc-b3dc6520-d332-4425-a7d0-b497cee0d8bb   1Gi        RWO            standard       2m44s
ml-work-rdbcq-workdir-2           Bound         pvc-3d07926b-c765-4627-98e8-ef9df4d173db   1Gi        RWO            standard       2m44s
ml-work-rdbcq-workdir-3           Bound         pvc-fc163a62-ea2c-473b-bc5f-0e4085ba7e3c   1Gi        RWO            standard       2m44s
ml-work-rdbcq-workdir-4           Bound         pvc-64a05440-195f-4748-97fe-a8b1effe4089   1Gi        RWO            standard       2m44s
ml-work-rdbcq-workdir-5           Bound         pvc-f6103cf7-60c8-4bd3-9880-49af98527755   1Gi        RWO            standard       2m44s
ml-work-rdbcq-workdir-6           Bound         pvc-e4b82cc8-f666-4f82-a08e-dd787b7accf2   1Gi        RWO            standard       2m44s

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T22:30:22Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.4-gke.22", GitCommit:"a6ba43f5a24ac29e631bb627c9b2a719c4e93638", GitTreeState:"clean", BuildDate:"2019-11-26T00:40:25Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

Argo version:
argoproj/workflow-controller:v2.4.3

Controller logs:

time="2020-01-23T12:52:13Z" level=info msg="Processing workflow" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Updated phase  -> Running" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Creating pvc ml-work-rdbcq-workdir-1" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Creating pvc ml-work-rdbcq-workdir-2" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Creating pvc ml-work-rdbcq-workdir-3" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Creating pvc ml-work-rdbcq-workdir-4" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Creating pvc ml-work-rdbcq-workdir-5" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Creating pvc ml-work-rdbcq-workdir-6" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Steps node ml-work-rdbcq (ml-work-rdbcq) initialized Pending" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="node ml-work-rdbcq (ml-work-rdbcq) phase Pending -> Running" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="StepGroup node ml-work-rdbcq[0] (ml-work-rdbcq-3903295395) initialized Running" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:13Z" level=info msg="Pod node ml-work-rdbcq[0].download-images-shard(0:1) (ml-work-rdbcq-4015706135) initialized Pending" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:15Z" level=info msg="Created pod: ml-work-rdbcq[0].download-images-shard(0:1) (ml-work-rdbcq-4015706135)" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:15Z" level=info msg="Pod node ml-work-rdbcq[0].download-images-shard(1:2) (ml-work-rdbcq-2730545253) initialized Pending" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:17Z" level=info msg="Created pod: ml-work-rdbcq[0].download-images-shard(1:2) (ml-work-rdbcq-2730545253)" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:17Z" level=info msg="Pod node ml-work-rdbcq[0].download-images-shard(2:3) (ml-work-rdbcq-4012591687) initialized Pending" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:18Z" level=info msg="Created pod: ml-work-rdbcq[0].download-images-shard(2:3) (ml-work-rdbcq-4012591687)" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:18Z" level=info msg="Pod node ml-work-rdbcq[0].download-images-shard(3:4) (ml-work-rdbcq-232276801) initialized Pending" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:20Z" level=info msg="Created pod: ml-work-rdbcq[0].download-images-shard(3:4) (ml-work-rdbcq-232276801)" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:20Z" level=info msg="Pod node ml-work-rdbcq[0].download-images-shard(4:5) (ml-work-rdbcq-1959765255) initialized Pending" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:22Z" level=info msg="Created pod: ml-work-rdbcq[0].download-images-shard(4:5) (ml-work-rdbcq-1959765255)" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:22Z" level=info msg="Pod node ml-work-rdbcq[0].download-images-shard(5:6) (ml-work-rdbcq-1994131533) initialized Pending" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:23Z" level=info msg="Created pod: ml-work-rdbcq[0].download-images-shard(5:6) (ml-work-rdbcq-1994131533)" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:23Z" level=info msg="Workflow step group node ml-work-rdbcq[0] (ml-work-rdbcq-3903295395) not yet completed" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:23Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"ml-work-rdbcq\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:23Z" level=info msg="Re-appying updates on latest version and retrying update" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:23Z" level=info msg="Update retry attempt 1 successful" namespace=default workflow=ml-work-rdbcq
time="2020-01-23T12:52:23Z" level=info msg="Workflow update successful" namespace=default workflow=ml-work-rdbcq

Pod description:

Name:               ml-work-rdbcq-4015706135
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-nnn-dask-pool-1-aadaae80-76z2/10.164.0.2
Start Time:         Thu, 23 Jan 2020 14:52:19 +0200
Labels:             workflows.argoproj.io/completed=false
                    workflows.argoproj.io/workflow=ml-work-rdbcq
Annotations:        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container wait
                    workflows.argoproj.io/node-name=ml-work-rdbcq[0].download-images-shard(0:1)
                    workflows.argoproj.io/template={"name":"download","arguments":{},"inputs":{"parameters":[{"name":"partition","value":"1"}]},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","c...
Status:             Succeeded
IP:                 10.40.126.49
Controlled By:      Workflow/ml-work-rdbcq
Containers:
  wait:
    Container ID:  docker:https://660bd400a5941c7d6b235d83f0b6b556be1aa7c8b48b67f2647cb75ef454ecfa
    Image:         argoproj/argoexec:v2.4.3
    Image ID:      docker-pullable:https://argoproj/argoexec@sha256:d7ab12ccc0c479cb856fa5aa6ab38c8368743f978bcbc4547bd8a67a83eb65f7
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 23 Jan 2020 14:52:53 +0200
      Finished:     Thu, 23 Jan 2020 14:52:55 +0200
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      ARGO_POD_NAME:  ml-work-rdbcq-4015706135 (v1:metadata.name)
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /mainctrfs/mnt/pv from workdir-1 (rw)
      /var/run/docker.sock from docker-sock (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jc2wb (ro)
  main:
    Container ID:  docker:https://d479bf67ffec8a58c7f67bda778db0166c1f4c037f39e81583bd9bde5c2e5005
    Image:         alpine:latest
    Image ID:      docker-pullable:https://alpine@sha256:ddba4d27a7ffc3f86dd6c2f92041af252a1f23a8e742c90e6e1297bfa1bc0c45
    Port:          <none>
    Host Port:     <none>
    Command:
      mkdir
      -p
      /mnt/pv/downloads_ml-work-rdbcq_1
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 23 Jan 2020 14:52:54 +0200
      Finished:     Thu, 23 Jan 2020 14:52:54 +0200
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /mnt/pv from workdir-1 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jc2wb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  Socket
  workdir-1:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ml-work-rdbcq-workdir-1
    ReadOnly:   false
  default-token-jc2wb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jc2wb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age              From                                                Message
  ----     ------                  ----             ----                                                -------
  Warning  FailedScheduling        7m (x3 over 7m)  default-scheduler                                   pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
  Normal   Scheduled               7m               default-scheduler                                   Successfully assigned default/ml-work-rdbcq-4015706135 to gke-nnn-dask-pool-1-aadaae80-76z2
  Normal   SuccessfulAttachVolume  7m               attachdetach-controller                             AttachVolume.Attach succeeded for volume "pvc-b3dc6520-d332-4425-a7d0-b497cee0d8bb"
  Normal   Pulled                  7m               kubelet, gke-nnn-dask-pool-1-aadaae80-76z2  Container image "argoproj/argoexec:v2.4.3" already present on machine
  Normal   Created                 7m               kubelet, gke-nnn-dask-pool-1-aadaae80-76z2  Created container wait
  Normal   Started                 7m               kubelet, gke-nnn-dask-pool-1-aadaae80-76z2  Started container wait
  Normal   Pulling                 7m               kubelet, gke-nnn-dask-pool-1-aadaae80-76z2  Pulling image "alpine:latest"
  Normal   Pulled                  7m               kubelet, gke-nnn-dask-pool-1-aadaae80-76z2  Successfully pulled image "alpine:latest"
  Normal   Created                 7m               kubelet, gke-nnn-dask-pool-1-aadaae80-76z2  Created container main
  Normal   Started                 7m               kubelet, gke-nnn-dask-pool-1-aadaae80-76z2  Started container main

Workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: ml-work-
  namespace: default
spec:
  arguments:
    parameters:
    - name: parallel
      value: '6'
  entrypoint: download-images
  templates:
  - name: download-images
    steps:
    - - arguments:
          parameters:
          - name: partition
            value: '{{item}}'
        name: download-images-shard
        template: download
        withSequence:
          end: '{{workflow.parameters.parallel}}'
          start: '1'
    - - arguments:
          parameters:
          - name: partition
            value: '{{item}}'
        name: tag-downloaded-images-shard
        template: tag-images
        withSequence:
          end: '{{workflow.parameters.parallel}}'
          start: '1'
    - - arguments:
          parameters:
          - name: partition
            value: '{{item}}'
        name: clean-shard
        template: cleanup
        withSequence:
          end: '{{workflow.parameters.parallel}}'
          start: '1'
  - container:
      command:
      - mkdir
      - -p
      - /mnt/pv/downloads_{{workflow.name}}_{{inputs.parameters.partition}}
      image: alpine:latest
      resources:
        limits:
          cpu: 0.1
          memory: 100Mi
        requests:
          cpu: 0.1
          memory: 100Mi
      volumeMounts:
      - mountPath: /mnt/pv
        name: workdir-{{inputs.parameters.partition}}
    inputs:
      parameters:
      - name: partition
    name: download
  - container:
      command:
      - ls
      - -lh
      - /mnt/pv/downloads_{{workflow.name}}_{{inputs.parameters.partition}}
      image: alpine:latest
      resources:
        limits:
          cpu: 0.1
          memory: 100Mi
        requests:
          cpu: 0.1
          memory: 100Mi
      volumeMounts:
      - mountPath: /mnt/pv
        name: workdir-{{inputs.parameters.partition}}
    inputs:
      parameters:
      - name: partition
    name: tag-images
  - container:
      command:
      - echo
      - Finished tagging, we can call webhook
      image: alpine:latest
      resources:
        limits:
          cpu: 0.1
          memory: 100Mi
        requests:
          cpu: 0.1
          memory: 100Mi
      volumeMounts:
      - mountPath: /mnt/pv
        name: workdir-{{inputs.parameters.partition}}
    inputs:
      parameters:
      - name: partition
    name: collect
  - container:
      command:
      - rm
      - -fr
      - /mnt/pv/downloads_{{workflow.name}}_{{inputs.parameters.partition}}
      image: alpine:latest
      resources:
        limits:
          cpu: 0.1
          memory: 100Mi
        requests:
          cpu: 0.1
          memory: 100Mi
      volumeMounts:
      - mountPath: /mnt/pv
        name: workdir-{{inputs.parameters.partition}}
    inputs:
      parameters:
      - name: partition
    name: cleanup
  volumeClaimTemplates:
  - metadata:
      name: workdir-1
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: workdir-2
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: workdir-3
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: workdir-4
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: workdir-5
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: workdir-6
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

Adding retry to download-images does not help. Feels like it doesn't wait for available resources before saying it failed. Also if it failed, status of workflow was still in running state even after >1h where pods had status "Completed". This resulting in stuck workflows.

NAME                       READY     STATUS      RESTARTS   AGE
ml-work-rdbcq-4015706135   0/2       Completed   0          86m