[Bug]: race condition in test-summaries: File 'httpd-log' doesn't match regexp

[smcv]

Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for a bug that matches the one I want to file, without success.
• If this is an issue with a particular app, I have tried filing it in the appropriate issue tracker for the app (e.g. under https://github.com/flathub/) and determined that it is an issue with Flatpak itself.
• This issue is not a report of a security vulnerability (see here if you need to report a security issue).

Flatpak version

1.14.8

What Linux distribution are you using?

Debian

Linux distribution version

testing/unstable rolling release

What architecture are you using?

x86_64

How to reproduce

Run the Flatpak test suite repeatedly, either directly on the host system or in a virtual machine, and either as "as-installed" tests or at build-time.

Running the test suite in a chroot or container will not necessarily reproduce this failure mode, because many tests cannot work and are skipped in that situation.

Expected Behavior

All tests pass

Actual Behavior

Rarely, one of the instantiations of test-summaries.sh fails like this:

378s ++ assert_not_file_has_content httpd-log
summaries/cb09c0ee49abd2a14a6a556463d4cf696e5f58a2341112c612ea533ac18531f9.gz
378s ++ assert_file_has_content httpd-log
summaries/7d82d86befacc5608749cb851a2deaa04156345146dbfd8970b6ff97a321db23-cb09c0ee49abd2a14a6a556463d4cf696e5f58a2341112c612ea533ac18531f9.delta
378s File 'httpd-log' doesn't match regexp
'summaries/7d82d86befacc5608749cb851a2deaa04156345146dbfd8970b6ff97a321db23-cb09c0ee49abd2a14a6a556463d4cf696e5f58a234
1112c612ea533ac18531f9.delta' at test-summaries.sh:258
378s OK closing connection
378s fusermount: entry for /tmp/test-flatpak-0Wl0GX/runtime/doc not found in /etc/mtab
378s 127.0.0.1 - - [09/May/2024 00:24:18] "GET /test/summaries/7d82d86befacc5608749cb851a2deaa04156345146dbfd8970b6ff97a321db23-cb09c0ee49abd2a14a6a556463d4cf696e5f58a2341112c612ea533ac18531f9.delta HTTP/1.1" 200 -
378s FAIL: Flatpak/[email protected] (Child process exited with code 1)

In practice I have never seen this during build-time testing, only when Flatpak is configured with --enable-installed-tests (equivalent to -Dinstalled_tests in 1.15.x) and installed, and the "as-installed" tests are subsequently run using gnome-desktop-testing. As I said, this is a rare failure, but flaky tests are troublesome for CI frameworks like Debian's autopkgtest, because they can falsely detect updated packages like GLib as being broken when, in fact, they are not.

This is fairly obviously a race condition: the desired request does get written to the web server log, but not until after the test script has checked for it, not found it, and made the test fail. The test probably needs to loop until the desired output is seen in the log, or make some other distinctive request as a marker and loop until that request is seen in the log.

Additional Information

No response