Can't use attached --device in rootless container "Permission denied"

[frroool]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Can not use an "attached" USB device in rootless container via --device because of "Permission denied" on the USB device. I have tried using --annotation run.oci.keep_original_groups=1 but the original groups does not follow along to the container - or - I am misunderstanding and doing something wrong. Would be really glad for some help.

Sidenote:
--annotation run.oci.keep_original_groups=1 does not appear to make any difference in root container either - but I have not investigated that further. Maybe it is intended?

Steps to reproduce the issue:

On host:

[podman@localhost ~]$ ls -al /dev/ttyACM0
crw-rw----. 1 root dialout 166, 0 Mar 13 14:45 /dev/ttyACM0

[podman@localhost ~]$ cat /dev/ttyACM0
^C

Rootless container without --annotation run.oci.keep_original_groups=1:

[podman@localhost ~]$ podman run --rm -it --device=/dev/ttyACM0 ubuntu:latest /bin/bash 

root@67ad11ddc944:/# ls -al /dev/ttyACM0
crw-rw----. 1 nobody nogroup 166, 0 Mar 13 13:45 /dev/ttyACM0

root@67ad11ddc944:/# cat /dev/ttyACM0
cat: /dev/ttyACM0: Permission denied

Rootless container with --annotation run.oci.keep_original_groups=1:

[podman@localhost ~]$ podman run --rm -it --annotation run.oci.keep_original_groups=1 --device=/dev/ttyACM0 ubuntu:latest /bin/bash

root@0861ba6c9e41:/# ls -al /dev/ttyACM0
crw-rw----. 1 nobody nogroup 166, 0 Mar 13 13:45 /dev/ttyACM0

root@0861ba6c9e41:/# cat /dev/ttyACM0
cat: /dev/ttyACM0: Permission denied

Describe the results you received:
I can not use the attached --device in the container because of the "Permission denied".

Describe the results you expected:
I was hoping to be able to use the --device in a rootless container.

Additional information you deem important (e.g. issue happens only occasionally):
Output of podman container inspect goofy_stonebraker | grep Annotations -A 9 on rootless container started with --annotation run.oci.keep_original_groups=1:

            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2021-03-13T15:11:46.597121188+01:00",
                "io.kubernetes.cri-o.TTY": "true",
                "io.podman.annotations.autoremove": "TRUE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15",
                "run.oci.keep_original_groups": "1"

I have checked issue #4477 which I thought could help me, but it does not help.

Output of podman version:

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.8
Built:        Fri Feb 19 17:56:17 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.26-1.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 777074ecdb5e883b9bec233f3630c5e7fa37d521'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.8.15-301.fc33.x86_64
  linkmode: dynamic
  memFree: 230502400
  memTotal: 2074468352
  ociRuntime:
    name: crun
    package: crun-0.18-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.18
      commit: 808420efe3dc2b44d6db9f1a3fac8361dde42a95
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 1027862528
  swapTotal: 1037037568
  uptime: 4h 23m 27.13s (Approximately 0.17 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/podman/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1001/containers
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1613753777
  BuiltTime: Fri Feb 19 17:56:17 2021
  GitCommit: ""
  GoVersion: go1.15.8
  OsArch: linux/amd64
  Version: 3.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.0.1-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes, 20) in https://github.com/containers/podman/blob/master/troubleshooting.md seems to be my problem.

Additional environment details (AWS, VirtualBox, physical, etc.):
Tested on "bare metal" on RPI 3B+ and in QEMU/KVM with the same result.

[rhatdan]

Did you check SELinux?

podman run --rm -it --security-opt label=disable --device=/dev/ttyACM0 ubuntu:latest /bin/bash

[frroool]

[podman@localhost ~]$ ls -alh /dev/ttyACM0
crw-rw----. 1 root dialout 166, 0 Mar 16 19:54 /dev/ttyACM0
[podman@localhost ~]$ podman run --rm -it --security-opt label=disable --device=/dev/ttyACM0 ubuntu:latest /bin/bash

root@6b7fc6397848:/# ls -alh /dev/ttyACM0
crw-rw----. 1 nobody nogroup 166, 0 Mar 16 18:54 /dev/ttyACM0
root@6b7fc6397848:/# cat /dev/ttyACM0
cat: /dev/ttyACM0: Permission denied

[podman@localhost ~]$ podman run --rm -it --security-opt label=disable --device=/dev/ttyACM0 --annotation run.oci.keep_original_groups=1 ubuntu:latest /bin/bash
root@fc2f74b02b78:/# ls -alh /dev/ttyACM0
crw-rw----. 1 nobody nogroup 166, 0 Mar 16 18:54 /dev/ttyACM0
root@fc2f74b02b78:/# cat /dev/ttyACM0
^C

--security-opt label=disable together with --annotation run.oci.keep_original_groups=1 seems to do the trick! I really thought that I tried that before, but maybe that was before I found out about the run.oci.keep_original_groups=1. Thanks!

I will check in the future if I can get it to work with SELinux too.

[rhatdan]

Could you attach the AVCs?

[frroool]

Could you attach the AVCs?

Haven’t had time to check SELinux yet. Will try to do it this weekend. When I do I can attach the AVCs.

[faern]

I have this issue now in Fedora 36 with podman 4.2.0

On host (I have created udev rules to give me the ownership of this device. But I got the exact same results when the device was owned by root:dialout):

$ ls -l /dev/hidraw6
crw-rw-rw-+ 1 faern dialout 239, 6 Sep 22 23:43 /dev/hidraw6

In container (--group-add keep-groups makes absolutely no difference):

$ podman run --rm -it --device /dev/hidraw6 ubuntu:latest /bin/bash

root@4ba2beb8c274:/# ls -l /dev/
ls: cannot access '/dev/hidraw6': Permission denied
...
-?????????? ? ?      ?            ?            ? hidraw6
...

root@db8e623387da:/# ls -l /dev/hidraw6
ls: cannot access '/dev/hidraw6': Permission denied

Running podman with --security-opt label=disable as pointed out above works. So seems to be SELinux related..

In SELinux logs on host:

$ sudo ausearch -m AVC,USER_AVC,SELINUX_ERR -ts recent
...
time->Thu Sep 22 23:56:22 2022
type=AVC msg=audit(1663883782.134:4550): avc:  denied  { getattr } for  pid=1270019 comm="ls" path="/dev/hidraw6" dev="devtmpfs" ino=10793 scontext=system_u:system_r:container_t:s0:c17,c528 tcontext=system_u:object_r:usb_device_t:s0 tclass=chr_file permissive=0