Argo v2.0 Demo doesn't work with Minikube

[cloudbring]

Following the Argo v2.0 Demo directions here (exact git hash link: https://github.com/argoproj/argo/blob/37f41eb7bf366cfe007d3ecce7b21f003d381e34/demo.md ) and using Minikube locally on a my MBP (OSX 10.12.6)

The tutorial as written isn't clear or doesn't work. I followed the instructions but, step 5 wasn't very clear.

When I ran kubectl edit configmap workflow-controller-configmap -n kube-system this is the complete config file saved:

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  config: |
    executorImage: argoproj/argoexec:v2.0.0-alpha2
    artifactRepository:
        s3:
            bucket: my-bucket
            endpoint: https://192.168.99.100:31443
            insecure: true
            accessKeySecret:
                name: argo-artifacts-minio-user
                key: "AKIAIOSFODNN7EXAMPLE"
            secretKeySecret:
                name: argo-artifacts-minio-user
                key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
kind: ConfigMap
metadata:
  creationTimestamp: 2017-12-09T04:40:09Z
  name: workflow-controller-configmap
  namespace: kube-system
  resourceVersion: "5975"
  selfLink: /api/v1/namespaces/kube-system/configmaps/workflow-controller-configmap
  uid: 0979807e-dc9b-11e7-a5f3-080027407c4d

I suspect that my challenges might come from using MiniKube. When I run kubectl get all -l release=argo-artifacts I get:

NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/argo-artifacts-minio   1         1         1            1           1h

NAME                                 DESIRED   CURRENT   READY     AGE
rs/argo-artifacts-minio-7c4fccc877   1         1         1         1h

NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/argo-artifacts-minio   1         1         1            1           1h

NAME                                 DESIRED   CURRENT   READY     AGE
rs/argo-artifacts-minio-7c4fccc877   1         1         1         1h

NAME                                       READY     STATUS    RESTARTS   AGE
po/argo-artifacts-minio-7c4fccc877-82dx7   1/1       Running   0          1h

NAME                           TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
svc/argo-artifacts-minio-svc   LoadBalancer   10.107.106.185   <pending>     9000:31443/TCP   1h

The external-ip was always pending and after some googling it seems that minikube doesn't do external ip's the same. Running minikube service list shows:

|-------------|--------------------------|-----------------------------|
|  NAMESPACE  |           NAME           |             URL             |
|-------------|--------------------------|-----------------------------|
| default     | argo-artifacts-minio-svc | https://192.168.99.100:31443 |
| default     | kubernetes               | No node port                |
| kube-system | argo-ui                  | https://192.168.99.100:32529 |
| kube-system | kube-dns                 | No node port                |
| kube-system | kubernetes-dashboard     | https://192.168.99.100:30000 |
| kube-system | tiller-deploy            | No node port                |
|-------------|--------------------------|-----------------------------|

The url and port for argo-artifacts-minio-svc https://192.168.99.100:31443 is what I used for the config under what was originally argo-artifacts-minio-svc.default:9000 to explain my changes in the config here:

    artifactRepository:
        s3:
            bucket: my-bucket
            endpoint: https://192.168.99.100:31443

The actual error happens when we run the last step (6):
argo submit https://raw.githubusercontent.com/argoproj/argo/master/examples/artifact-passing.yaml

Name:             artifact-passing-vb9l7
Namespace:        default
Status:           Pending
Created:          Fri Dec 08 21:58:15 -0800 (2 seconds from now)

Running argo get artifact-passing-clrtl

Name:             artifact-passing-vb9l7
Namespace:        default
Status:           Failed
Message:          step group artifact-passing-vb9l7[0] was unsuccessful
Created:          Fri Dec 08 21:58:15 -0800 (5 seconds ago)
Started:          Fri Dec 08 21:58:15 -0800 (5 seconds ago)
Finished:         Fri Dec 08 21:58:20 -0800 (now)
Duration:         5 seconds

STEP                       PODNAME                            MESSAGE
 ✖ artifact-passing-vb9l7
 └---⚠ generate-artifact   artifact-passing-vb9l7-2566534122  failed to save artifacts: Key [AKIAIOSFODNN7EXAMPLE argo-artifacts-minio-user] does not exists for secret %!s(MISSING)

Referencing the PODNAME artifact-passing-vb9l7-2566534122 and getting the logs from the wait container:
kubectl logs artifact-passing-sdrzz-2497688641 -c wait

time="2017-12-09T06:29:41Z" level=info msg="Executor (version: v2.0.0-alpha2) initialized with template:\narchiveLocation:\n  s3:\n    accessKeySecret:\n      key: AKIAIOSFODNN7EXAMPLE\n      name: argo-artifacts-minio-user\n    bucket: my-bucket\n    endpoint: https://192.168.99.100:31443\n    insecure: true\n    key: artifact-passing-sdrzz/artifact-passing-sdrzz-2497688641\n    secretKeySecret:\n      key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n      name: argo-artifacts-minio-user\ncontainer:\n  args:\n  - cowsay hello world | tee /tmp/hello_world.txt\n  command:\n  - sh\n  - -c\n  image: docker/whalesay:latest\n  name: \"\"\n  resources: {}\ninputs: {}\nname: whalesay\noutputs:\n  artifacts:\n  - name: hello-art\n    path: /tmp/hello_world.txt\n"
time="2017-12-09T06:29:41Z" level=info msg="Waiting on main container"
time="2017-12-09T06:29:43Z" level=info msg="[docker wait 10e0350ee40e2f1251df6014e04a2c5cff7969b9986714592f8e65bb5366113c]"
time="2017-12-09T06:29:43Z" level=info msg="Waiting completed"
time="2017-12-09T06:29:43Z" level=info msg="Killing sidecars"
time="2017-12-09T06:29:43Z" level=info msg="Saving output artifacts"
time="2017-12-09T06:29:43Z" level=info msg="Main container identified as: 10e0350ee40e2f1251df6014e04a2c5cff7969b9986714592f8e65bb5366113c"
time="2017-12-09T06:29:43Z" level=info msg="Saving artifact: hello-art"
time="2017-12-09T06:29:43Z" level=info msg="Archiving 10e0350ee40e2f1251df6014e04a2c5cff7969b9986714592f8e65bb5366113c:/tmp/hello_world.txt to /argo/outputs/artifacts/hello-art.tgz"
time="2017-12-09T06:29:43Z" level=info msg="[sh -c docker cp -a 10e0350ee40e2f1251df6014e04a2c5cff7969b9986714592f8e65bb5366113c:/tmp/hello_world.txt - | gzip > /argo/outputs/artifacts/hello-art.tgz]"
time="2017-12-09T06:29:44Z" level=info msg="Archiving completed"
time="2017-12-09T06:29:44Z" level=fatal msg="Error saving output artifacts, Key [AKIAIOSFODNN7EXAMPLE argo-artifacts-minio-user] does not exists for secret %!s(MISSING)\ngithub.com/argoproj/argo/errors.New\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:47\ngithub.com/argoproj/argo/errors.Errorf\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:53\ngithub.com/argoproj/argo/errors.InternalErrorf\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:63\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).getSecrets\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:47\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).InitDriver\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:235\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:170\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:29\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:702\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:783\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:736\nmain.main\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:11\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:185\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2337"

I was trying out Argo for the first time and maybe I'm doing something wrong but, I followed the directions carefully. If there's something I'm missing, I'd be glad to hear it. Otherwise, please test and fix the directions so that they work locally with Minikube.

[jessesuen]

Hi @cloudbring, the issue is that you are storing the plain-text secret value in the configmap, instead of a kubernetes secret key reference. This is incorrect:

            accessKeySecret:
                name: argo-artifacts-minio-user
                key: "AKIAIOSFODNN7EXAMPLE"
            secretKeySecret:
                name: argo-artifacts-minio-user
                key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

This is what should be stored in the config map literally:

        accessKeySecret:
          name: argo-artifacts-minio-user
          key: accesskey
        secretKeySecret:
          name: argo-artifacts-minio-user
          key: secretkey

The two fields, accessKeySecret and secretKeySecret are k8s secret selectors. argo-artifacts-minio-user refers to the secret name (which helm install stable/minio created for you). accesskey and secretKey refer to to the key in the secret.

Please try changing the configmap to the literal values in the demo.md and let us know if you have any further issues.

[jessesuen]

That said, this error message is confusing and should be fixed.

failed to save artifacts: Key [AKIAIOSFODNN7EXAMPLE argo-artifacts-minio-user] does not exists for secret %!s(MISSING)

It should probably read something like:

secret 'argo-artifacts-minio-user' does not have the key 'AKIAIOSFODNN7EXAMPLE'