[Bug]: Timeout running testlibrary in a podman container

[smcv]

Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for a bug that matches the one I want to file, without success.
• If this is an issue with a particular app, I have tried filing it in the appropriate issue tracker for the app (e.g. under https://github.com/flathub/) and determined that it is an issue with Flatpak itself.
• This issue is not a report of a security vulnerability (see here if you need to report a security issue).

Flatpak version

1.15.4-92-g9d63abef0 plus #5591 and #5590

What Linux distribution are you using?

Debian

Linux distribution version

unstable

What architecture are you using?

x86_64

How to reproduce

Originally reproduced in Debian sbuild (which uses schroot) while trying to release 1.15.6, but it can be reproduced in a more debuggable way with a container:

$ mkdir _build
$ podman run --rm -it -w $(pwd) -v $(pwd):$(pwd):ro -v $(pwd)/_build:$(pwd)/_build:rw debian:sid-slim
# apt update
# apt full-upgrade
# (edit /etc/apt/sources.list.d/ to enable deb-src)
# apt install build-essential meson
# apt build-dep flatpak
# meson setup -C _build
# meson compile -C _build
# meson test -C _build

Expected Behavior

Tests run successfully (most are skipped because bubblewrap doesn't work, but that's OK)

Actual Behavior

testlibrary takes a long time. Even when killed while in progress with kill -SEGV, Meson (version 1.2.3) does not acknowledge that it has finished until I also kill web-server.py.

Additional Information

Release blocker: I'm not going to do a release that I can't package.

[smcv]

I think perhaps it's genuinely just being too slow in my container, maybe because it takes time to run each sandboxed trigger (which fails, because bwrap doesn't work in this environment).

[smcv]

When using schroot with #5591 applied, the build succeeds, so this isn't a release blocker and I was able to release 1.15.6. It still seems bad that this fails in podman, though. Pull requests would be appreciated!

One possible improvement in this direction would be to start breaking up testlibrary, which has become a bit of a monster, into more/smaller test-cases that can be run separately, or even in parallel.

[dbnicholson]

I recently revamped our interal CI builder at Endless to use podman containers. Flatpak tests were definitely difficult. I can't remember everything, but I do recall hanging tests. Have you tried bind mounting a real directory to /var/tmp? If you're still on a kernel without unprivileged overlayfs, fuse-overlayfs is incredibly slow and not totally compliant to real filesystem semantics. I also added --device /dev/fuse specifically for flatpak since it causes issues for tests if the device isn't available (even if it isn't usable).