Building Wasm image with specific tags makes podman looks for (potentially non-existent) registry instead of picking the locally available image

[mh4ck-Thales]

Issue Description

When trying to run a wasm container with podman, a specific combination of flags for the build and the run command makes podman looks for the non-existent localhost registry instead of picking the image available locally. This bug seems close of #7699 or #12509 in the behavior. This does not only impact simple tags like -t mycontainer but also remote tags like -t myregistry.corp.net/mycontainer, which means you either are not able to run images (because the localhost registry does not exist) or you need to be logged into the specified registry and the image must be present (instead of picking the locally available image which have the same name).

Steps to reproduce the issue

0. (OPTIONAL) generate and build a demo Wasm app to put inside the container. With Rust, you can simply do :

cargo new app && cd app
rustup target add wasm32-wasi
cargo build --target wasm32-wasi

App will then be in target/wasm32-wasi/debug/app.wasm and you can check it works with running it with a Wasm runtime, e.g :

wasmedge target/wasm32-wasi/debug/app.wasm

Build the Wasm container image using this Containerfile (adapt if you're not using the Rust demo app) :

FROM scratch

COPY target/wasm32-wasi/debug/app.wasm /app.wasm
CMD ["/app.wasm"]

1. WARNING This will delete all of your podman images, containers, and configuration. Do this only in an environment where you do not care about deleting everything. Run :

podman system reset

to clean your podman environment. See why this step is needed in the "Additional information" section.

2. Build with :

podman build -t demo-wasm --annotation "module.wasm.image/variant=compat" .

3. Try to run this newly built image. For this, you'll need crun with wasm support. You can check with crun --version if you have the +WASM:wasmedge (or other wasm runtime) flag, otherwise you need to get a version of crun supporting it, see How to build crun for wasm? crun#1108 (comment) or use Fedora which ships in its repos a podman / crun version already built with wasm. You can then run :

podman run --platform wasi/wasm localhost/demo-wasm

4. See that podman fails to find the image tagged with localhost, searching for a registry in localhost instead. You can check the image is indeed present and using the good tag with podman image ls

Describe the results you received

podman tries to contact a non-existent localhost registry instead of using the existing image stored locally :

Trying to pull localhost/demo-wasm:test...
WARN[0000] Failed, retrying in 1s ... (1/3). Error: initializing source docker:https://localhost/demo-wasm:test: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
WARN[0001] Failed, retrying in 1s ... (2/3). Error: initializing source docker:https://localhost/demo-wasm:test: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
WARN[0002] Failed, retrying in 1s ... (3/3). Error: initializing source docker:https://localhost/demo-wasm:test: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
Error: initializing source docker:https://localhost/demo-wasm:test: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

Describe the results you expected

Podman succeeds in starting the wasm container.

podman info output

I confirmed this bug on multiple setups and OSes, including Debian 11 with old (3.4.2) podman version, and Fedora 38 / Archlinux (rootless) running both the latest (4.5.0) version of podman. Below is the output of podman info for my Fedora setup.

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 77.53
    systemPercent: 5.08
    userPercent: 17.39
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: cloud
    version: "38"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.2.9-300.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 3843981312
  memTotal: 8312483840
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8312057856
  swapTotal: 8312057856
  uptime: 23h 1m 43.00s (Approximately 0.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  mycustomreg.5gcn.kingsecret.net:5000:
    Blocked: false
    Insecure: true
    Location: mycustomreg.5gcn.kingsecret.net:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: mycustomreg.5gcn.kingsecret.net:5000
    PullFromMirror: ""
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 36421218304
  graphRootUsed: 2929225728
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486942
  BuiltTime: Fri Apr 14 15:42:22 2023
  GitCommit: ""
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Bare metal on Archlinux, qemu/KVM VMs for Debian 11 and Fedora 38

Additional information

I had a lot of trouble reproducing this bug because the cache can make the bug reproduction fails. For example building the same image with flags that works and run it, and then do the bug reproduction makes the bug disappear and the container work correctly. But if you clear completely the cache and try to do only the bug reproduction, the bug will appear.

1. Reproduce the bug reproduction steps 0 and 1 (don't forget podman system reset).

2. Try to build with :

podman build --annotation "module.wasm.image/variant=compat" --platform wasi/wasm -t localhost/demo-wasm:annotation-and-platform .

(notice the added --platform wasi/wasm flag)

3. Run this image with :

podman run --platform wasi/wasm localhost/demo-wasm:annotation-and-platform

Notice that it works.

4. Now rebuild the image according to the bug reproduction instructions :

podman build --annotation "module.wasm.image/variant=compat" -t localhost/demo-wasm:annotation-only .

5. Try to run this image with :

podman run --platform wasi/wasm localhost/demo-wasm:annotation-only

Notice that it works.

6. Now clean again podman with :

podman system reset

7. Try to build only with the bug repoduction :

podman build --annotation "module.wasm.image/variant=compat" -t localhost/demo-wasm:annotation-only .

8. Run this image :

podman run --platform wasi/wasm localhost/demo-wasm:annotation-only

Notice that it is now failing as described above.

I don't know if this problem is related or another unrelated bug. I noticed that when you build the 2 images (the good one and the bad one) one after the other, the same container image is tagged with both test test and test2 as they are indeed the same image with the exact same content. However, they do not have the same configuration (one is using the wasi/wasm platform, and the other the default one). The mix-up between the two is maybe the root cause for this problem of reproducibility, but I don't know if it's a real issue or if I did something incorrectly.

[Luap99]

@flouthoc PTAL

[flouthoc]

This looks like an issue in libimage since it tried to re-pull images with the format wasi/wasm , I'll check this thanks fo reporting.

[mh4ck-Thales]

Any news on this @flouthoc ? Did you find the origin of the bug ?

[flouthoc]

@mh4ck-Thales I know where it is but i was not able to work on it last week. I'll try checking it this week 😄

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@flouthoc What is going on with this one?

[flouthoc]

I'll check this missed this one last week.

[flouthoc]

Hi @mh4ck-Thales , I checked after #18542 we don't support annotations based builds anymore, I suggest building image with <podman/buildah> build --platform=wasi/wasm ....., see updated docs here: https://github.com/containers/crun/blob/main/docs/wasm-wasi-example.md#running-oci-wasm-compat-images-with-buildah-and-podman

I have verified and it will work just fine with new --platform=wasi/wasmflag for building image, therefore I am closing this issue. Please feel free to re-open if you see any more such issues :)

[mh4ck-Thales]

Hi @flouthoc , thanks for your answer. I have some questions to better understand the decisions and the situation if you may.

I stumbled on this issue when working on listing, comparing and mixing the different ways I found to build and run a Wasm image. I used the doc you mentioned and updated to remove the legacy Wasm annotations. Is deprecating the module.wasm.image and other Wasm-related annotations a global Wasm decision or just a podman one ?

Also, #18542 mentions that "propagating annotations may have security consequences". I've never heard of that and I'd like to know more. I'd be glad if you could point me on some content on the topic that prompted podman to not include image annotations at all.

[flouthoc]

Hi @mh4ck-Thales , Sure I'll try to answer your concerns

I stumbled on this issue when working on listing, comparing and mixing the different ways I found to build and run a Wasm image. I used the doc you mentioned and updated to remove the legacy Wasm annotations. Is deprecating the module.wasm.image and other Wasm-related annotations a global Wasm decision or just a podman one ?

I don't think this is any global decision but as per the discussion happened in past a more standard way to classify an OCI image as WASM image exists now i.e platform string is a better option as compared to some white-listed annotation. Maybe see discussion here: opencontainers/image-spec#964 and threads listed in same PR. The annotation was a way to do it when nothing was standardized.

Also, #18542 mentions that "propagating annotations may have security consequences". I've never heard of that and I'd like to know more. I'd be glad if you could point me on some content on the topic that prompted podman to not include image annotations at all.

There is no security consequences for wasm annotation specifically it was in general where image is changing behavior of runtime via propagating annotations ( runtime-spec does not documents this) , in PR itself it is written that we might do exception for wasm annotations but I am just not sure if there are enough old users who are using module.wasm.image annotations, idea is to promote new way i.e platform string which is more standard.

But of-course if there are more reports of breaking users using module.wasm.image then I think slow deprecation can be done, TBH i am just not sure if anybody is stuck there and if they are maybe its a good time to switch to platform string.