[garden/seccomp] Unable to run 32-bit binaries in concourse containers

[kallisti5]

Summary

After upgrading to concourse 7.4.0 to from 7.2.0, and the container host to Fedora 34 from 33, unable to run 32-bit binaries in concourse

Originally reported here:
https://bugzilla.redhat.com/show_bug.cgi?id=1998392

They mentioned an issue with seccomp.

Maybe something is missing here?
https://github.com/concourse/concourse/blob/master/worker/runtime/spec/seccomp.go

Steps to reproduce

Container host: Fedora 34, x86_64

Image: docker.io/fedora:34

yum install git nasm autoconf automake texinfo flex bison gcc gcc-c++ make glibc-devel zlib-devel xorriso curl-devel byacc libstdc++-static glibc-devel.i686 libstdc++-devel.i686 libstdc++-devel python36

Steps to Reproduce:

1. echo "void main(void) {}" > test.c
2. gcc -m32 test.c -o test-32bit
3. ./test-32bit

Expected results

It to work

Actual results

./test-32bit: error while loading shared libraries: libc.so.6: cannot stat shared object: Operation not permitted

Triaging info

• Concourse version: 7.4.0

[kallisti5]

strace -vvFf ./test-32bit
strace: deprecated option -F ignored
execve("./test-32bit", ["./test-32bit"], ["DISTTAG=f34container", "PWD=/tmp/build/7e93e057", "FBR=f34", "HOME=/root", "LANG=C.UTF-8", "LS_COLORS=rs=0:di=01;34:ln=01;36"..., "FGC=f34", "TOOLARCH=x86_gcc2", "TERM=xterm-256color", "LESSOPEN=||/usr/bin/lesspipe.sh "..., "USER=root", "SHLVL=1", "which_declare=declare -f", "PATH=/root/.local/bin:/root/bin:"..., "BASH_FUNC_which%%=() {  ( alias;"..., "_=/usr/bin/strace"]) = 0
[ Process PID=40050 runs in 32 bit mode. ]
brk(NULL)                               = 0x9e89000
arch_prctl(0x3001 /* ARCH_??? */, 0xff9ae608) = -1 EINVAL (Invalid argument)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf7f26000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, 0xff9ad920) = -1 EPERM (Operation not permitted)
close(3)                                = 0
openat(AT_FDCWD, "/lib/tls/i686/sse2/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/lib/tls/i686/sse2", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/lib/tls/i686/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/lib/tls/i686", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/lib/tls/sse2/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/lib/tls/sse2", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/lib/tls/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/lib/tls", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/lib/i686/sse2/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/lib/i686/sse2", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/lib/i686/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/lib/i686", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/lib/sse2/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
statx(AT_FDCWD, "/lib/sse2", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
openat(AT_FDCWD, "/lib/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\353\1\0004\0\0\0"..., 512) = 512
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0T\244\243\"\215\254\372k\247\24\v'\210\321\365\217"..., 108, 468) = 108
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, 0xff9ad8d0) = -1 EPERM (Operation not permitted)
close(3)                                = 0
writev(2, [{iov_base="./test-32bit", iov_len=12}, {iov_base=": ", iov_len=2}, {iov_base="error while loading shared libra"..., iov_len=36}, {iov_base=": ", iov_len=2}, {iov_base="libc.so.6", iov_len=9}, {iov_base=": ", iov_len=2}, {iov_base="cannot stat shared object", iov_len=25}, {iov_base=": ", iov_len=2}, {iov_base="Operation not permitted", iov_len=23}, {iov_base="\n", iov_len=1}], 10./test-32bit: error while loading shared libraries: libc.so.6: cannot stat shared object: Operation not permitted
) = 114
exit_group(127)                         = ?
+++ exited with 127 +++

Looks like statx isn't allowed?

[kallisti5]

wait.. statx is allowed (and was added in 2020) @muntac

However, this looks sus:

arch_prctl(0x3001 /* ARCH_??? */, 0xff9ae608) = -1 EINVAL (Invalid argument)

Is concourse failing to detect 32-bit?

[kallisti5]

I see arch_prctl mentioned in #6044 @muntac Maybe related?

[taylorsilva]

Muntac is off right now so don't expect a reply from him any time soon :)

Could this be a bug with Fedora 34? Have you tested running 7.4.0 on Fedora 33? Would be interesting to see if the same error occurs there. From what I can tell, briefly looking at the seccomp.go and your comments, it seems like Concourse's seccomp profile should allow the statx and arch_prctl calls. I'm not immediately sure what we would need to change to make this work for you on Fedor 34.

[fweimer-rh]

As I said on the Fedora bug, it's a bug in the container environment. Fedora 34 simply uses slightly different parts of the Linux system call interface than previous versions. This comment above hints to the reason why:

statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, 0xff9ad920) = -1 EPERM (Operation not permitted)

Container environments need to return ENOSYS for system calls that they do not implement or are not prepared to handle, like regular Linux does. There have been extensive discussions about this in various places:

• Change seccomp default error from EPERM to ENOSYS
• seccomp filter should return ENOSYS for unknown syscalls

[kallisti5]

@fweimer-rh I'm a bit confused here. Isn't the EPERM error coming from the kernel ioctl and not anything concourse is doing?

Unless i'm missing something, it seems that the github.com/opencontainers/runtime-spec/specs-go go module or the kernel is the cause? I'm not seeing any ENOSYS returns anywhere in the concourse code.

These are all basic runc containers

[kallisti5]

@fweimer-rh @taylorsilva

opencontainers/runtime-spec#960 this seems related. I guess the "default" in the runtime-spec go module is EPERM, and that needs adjusted to ENOSYS on "all platforms"? (or just Fedora platforms?)

[kallisti5]

I opened opencontainers/runtime-spec#1122 about potentially changing the default in runtime-spec since it causes breakages. The alternative is I just stop using Fedora 😆

[rcsalome]

Thanks for making this issue @kallisti5 , it came at the perfect time! We have noticed that the latest Debian Bullseye release has also had issues due to the statx system call.

After some testing, we have been able to get around this by changing our container runtime to containerd.

CONCOURSE_WORKER_RUNTIME=containerd

https://concourse-ci.org/concourse-worker.html#containerd-runtime

Not sure if it will also help with Fedora, but it might be worth a try.

[kallisti5]

I love workarounds. Trying this out now. :-)

[kallisti5]

confirming that switching from the default garden engine to containerd solves this issue! 🎆 Thanks @rcsalome !
I'll leave it open for potential investigation of the garden code.

[taylorsilva]

Stuff like this is actually why we worked on the containerd runtime. I don't see the garden team keeping up with the recent OCI changes. They already don't support cgroupsV2 and that's been the default on Fedora for a while now and starting with Debian Bullseye as well.