bug: cannot get artifact log with state "pod termination" (502 Bad Gateway)

[clementlecorre]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:

In argo UI I want to see an artifact (from the log download link) but I have a 502 (502 Bad Gateway) error.

workflow > logs > Still waiting for data or an error? Try getting logs from the artifacts.

What you expected to happen:

I'm expecting to get my workflow logs.

How to reproduce it (as minimally and precisely as possible):

I don't know

Anything else we need to know?:

After the bug, I looked at the server logs.

2020/04/22 15:38:26 http: panic serving 10.0.187.94:52380: runtime error: invalid memory address or nil pointer dereference
goroutine 1807 [running]:
net/http.(*conn).serve.func1(0xc008643e00)
	/usr/local/go/src/net/http/server.go:1767 +0x139
panic(0x1b9bbe0, 0x3537a20)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/argoproj/argo/pkg/apis/workflow/v1alpha1.(*Outputs).GetArtifactByName(...)
	/go/src/github.com/argoproj/argo/pkg/apis/workflow/v1alpha1/workflow_types.go:1659
github.com/argoproj/argo/server/artifacts.(*ArtifactServer).getArtifact(0xc00004dec0, 0x263d020, 0xc00090f920, 0xc00017d000, 0xc000eb1a74, 0x44, 0xc000eb1ab9, 0x9, 0x0, 0x0, ...)
	/go/src/github.com/argoproj/argo/server/artifacts/artifact_server.go:129 +0x11c
github.com/argoproj/argo/server/artifacts.(*ArtifactServer).GetArtifact(0xc00004dec0, 0x2636160, 0xc0014961c0, 0xc0006ddf00)
	/go/src/github.com/argoproj/argo/server/artifacts/artifact_server.go:57 +0x509
net/http.HandlerFunc.ServeHTTP(0xc0001f38b0, 0x2636160, 0xc0014961c0, 0xc0006ddf00)
	/usr/local/go/src/net/http/server.go:2007 +0x44
net/http.(*ServeMux).ServeHTTP(0xc00047b340, 0x2636160, 0xc0014961c0, 0xc0006ddf00)
	/usr/local/go/src/net/http/server.go:2387 +0x1bd
net/http.serverHandler.ServeHTTP(0xc0005e0000, 0x2636160, 0xc0014961c0, 0xc0006ddf00)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc008643e00, 0x263cf60, 0xc002c4f4c0)
	/usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

I wanted to see my pod logs with the status "pod termination".

Environment:

• Argo version:

$ argo version
argo: v2.7.5+ede163e.dirty
  BuildDate: 2020-04-21T01:01:37Z
  GitCommit: ede163e1af83cfce29b519038be8127664421329
  GitTreeState: dirty
  GitTag: v2.7.5
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-11-14T04:24:29Z"
  compiler: gc
  gitCommit: b3cbbae08ec52a7fc73d334838e18d17e8512749
  gitTreeState: clean
  gitVersion: v1.16.3
  goVersion: go1.12.13
  major: "1"
  minor: "16"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-02-26T01:12:54Z"
  compiler: gc
  gitCommit: bac3690554985327ae4d13e42169e8b1c2f37226
  gitTreeState: clean
  gitVersion: v1.15.10-eks-bac369
  goVersion: go1.12.12
  major: "1"
  minor: 15+
  platform: linux/amd64

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

We should never see panics. This will cause the container to restart. I’ll investigate.

[alexec]

Can you please share your artifact configuration?

[clementlecorre]

I don't have a problem with other workflows. I have more than 1000 finished workflows or I had no problem.

kubectl get configmap argo-workflow-controller-configmap -o yaml

apiVersion: v1
data:
  config: |
    instanceID: argo-ID
    containerRuntimeExecutor: docker
    artifactRepository:
      archiveLogs: true
      s3:
        accessKeySecret:
          key: accesskey
          name: argo-s3
        secretKeySecret:
          key: secretkey
          name: argo-s3
        bucket: argo-s3-example
        endpoint: s3.amazonaws.com
        insecure: false
...

[alexec]

The reason you get a 502 is the controller panics due to a nil deref. This crashes it and is a bug. But it is coincident - it is not why you do not have any logs - but it will hide the reason. So we need to fix it.

Are you able to use kubectl logs to get the logs of the wait and main containers and paste them here please?

[clementlecorre]

Unfortunately I don't have the crash pod logs.... I feel like it's been deleted.