argoproj/argoexec:latest sometimes only contains linux/amd64

[omus]

Summary

Submitting any Argo workflow fails when running on an M1 Mac due to "argoproj/argoexec:latest" not including a linux/arm64 image.

The failures message from kubectl describe:

Failed to pull image "argoproj/argoexec:latest": rpc error: code = Unknown desc = no matching manifest for linux/arm64/v8 in the manifest list entries

Diagnostics

What Kubernetes provider are you using?

❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T20:58:09Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm64"}

What version of Argo Workflows are you running?

❯ argo version
argo: v3.1.3+9337abb.dirty
  BuildDate: 2021-07-28T08:20:43Z
  GitCommit: 9337abb002d3c505ca45c5fd2e25447acd80a108
  GitTreeState: dirty
  GitTag: v3.1.3
  GoVersion: go1.16.6
  Compiler: gc
  Platform: darwin/arm64

What executor are you running? Docker

Did this work in a previous version? I.e. is it a regression? Yes, worked yesterday on the same versions

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[omus]

I could work around this by specifying an alternate image such as "argoproj/argoexec:latest-linux-arm64" but I don't believe there is a way to change via argo submit.

[omus]

As of this moment the "latest" tag includes all of the expected OS/ARCH combos:

To me this points to there being a Docker build issue where "latest" is being updated too early resulting in runtime failures for those using linux/arm64 or windows/amd64 for a small period of time.

[wanghong230]

Do you see the same problem with the released images? not latest, eg: v3.1.x image/

[omus]

I haven't noticed it myself personally. Based upon how latest behaved there's probably only a ~9 minute window where someone would notice this issue

[sarabala1979]

something in your node or pod configuration which doesn't match with the default image Linux/arm64/v8 I am not sure about v8.

[omus]

Just happened again:

Error message seen from Argo submit:

ErrImagePull: rpc error: code = Unknown desc = no matching manifest for linux/arm64/v8 in the manifest list entries

[omus]

Looks like the "latest-windows" and "latest-linux-arm64" images have been updated "latest-linux-amd64" has not yet been updated:

And the "latest" manifest has been updated to include all architectures again:

[sarabala1979]

@omus Is this issue resolved?

[omus]

@omus Is this issue resolved?

No, the issue is not resolved. The problem is that Argo workflows break on non-AMD64 platforms every time a new latest build is initially released.

Until the build system is updated to only push a argoexec:latest manifest once (which includes all supported platforms) the issue will keep occurring.

[alexec]

@omus this would need fixing here:

https://github.com/argoproj/argo-workflows/blob/master/.github/workflows/release.yaml

Do you want to try and submit a PR?