[Bug]: Flatpaks don't run while a kbfs filesystem is mounted on "/keybase"

[besendorf]

Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for a bug that matches the one I want to file, without success.
• If this is an issue with a particular app, I have tried filing it in the appropriate issue tracker for the app (e.g. under https://github.com/flathub/) and determined that it is an issue with Flatpak itself.
• This issue is not a report of a security vulnerability (see here if you need to report a security issue).

Flatpak version

1.15.4

What Linux distribution are you using?

Arch Linux

Linux distribution version

rolling

What architecture are you using?

x86_64

How to reproduce

No response

Expected Behavior

flatpaks start the application

Actual Behavior

flatpaks run and stop after allowing pulseaudio access

$ flatpak -vv run im.riot.Riot 
F: No installations directory in /etc/flatpak/installations.d. Skipping
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/22.08/5c8d805b97240ba857b1a63e64fb27fdcf23bb3c2926a5cf6c0d25de23c277fb/files/lib32 does not exist
F: Cleaning up unused container id 3282511126
F: Cleaning up per-app-ID state for im.fluffychat.Fluffychat
F: Allocated instance id 3191513932
F: Add defaults in dir /im/riot/Riot/
F: Add locks in dir /im/riot/Riot/
F: Trying to export read/write: /run/user/1000/pipewire-0
F: /run is not a symlink
F: /run/user is not a symlink
F: /run/user/1000 is not a symlink
F: /run/user/1000/pipewire-0 is not a symlink
F: Will export read/write: /run/user/1000/pipewire-0
F: Trying to export read/write: /run/user/1000/keyring
F: /run is not a symlink
F: /run/user is not a symlink
F: /run/user/1000 is not a symlink
F: /run/user/1000/keyring is not a symlink
F: Will export read/write: /run/user/1000/keyring
F: Trying to replace with tmpfs: /home/user/.var/app
F: /home is not a symlink
F: /home/user is not a symlink
F: /home/user/.var is not a symlink
F: /home/user/.var/app is not a symlink
F: Will replace with tmpfs: /home/user/.var/app
F: Trying to export read/write: /home/user/.var/app/im.riot.Riot
F: /home is not a symlink
F: /home/user is not a symlink
F: /home/user/.var is not a symlink
F: /home/user/.var/app is not a symlink
F: /home/user/.var/app/im.riot.Riot is not a symlink
F: Will export read/write: /home/user/.var/app/im.riot.Riot
F: Trying to replace with tmpfs: /home/user/.local/share/flatpak
F: /home is not a symlink
F: /home/user is not a symlink
F: /home/user/.local is not a symlink
F: /home/user/.local/share is not a symlink
F: /home/user/.local/share/flatpak is not a symlink
F: Will replace with tmpfs: /home/user/.local/share/flatpak
F: Trying to ensure existence of directory: /home/user
F: /home is not a symlink
F: /home/user is not a symlink
F: Will ensure existence of directory: /home/user
F: Converting FlatpakExports to bwrap arguments...
F: "/home/user" is meant to be a directory
F: Ensuring "/home/user" is created as a directory
F: "/home/user/.local/share/flatpak" is meant to be a tmpfs or empty directory
F: Parent of "/home/user/.local/share/flatpak" is not mapped, creating empty directory
F: "/home/user/.var/app" is meant to be a tmpfs or empty directory
F: Parent of "/home/user/.var/app" is not mapped, creating empty directory
F: "/home/user/.var/app/im.riot.Riot" is meant to be shared (ro or rw) with the container
F: "/run/user/1000/keyring" is meant to be shared (ro or rw) with the container
F: "/run/user/1000/pipewire-0" is meant to be shared (ro or rw) with the container
F: Allowing wayland access
F: Allowing x11 access
F: Allowing pulseaudio access

Additional Information

No response

[misbahally]

Hello,

I have the same issue, but on Fedora 38, flatpak run hangs, I cannot do CTRL+C, I tried running strace on the process it also hangs.

This behaviour happens with all application I have installed.

[besendorf]

Today it worked again. I am really not sure what the issue is. I can provide any logs that would be helpful

[smcv]

flatpak -vv run is as much information as we can easily get, so unless someone can reproduce this issue and dig into it themselves, I suspect this will forever remain a mystery.

[besendorf]

Right now I have the problem again. Could you give me any hints what I can do to provide more information @smcv ?

$ flatpak -vv run im.riot.Riot 
F: No installations directory in /etc/flatpak/installations.d. Skipping
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/22.08/5c8d805b97240ba857b1a63e64fb27fdcf23bb3c2926a5cf6c0d25de23c277fb/files/lib32 does not exist
F: Allocated instance id 514792909
F: Add defaults in dir /im/riot/Riot/
F: Add locks in dir /im/riot/Riot/
F: Trying to export read/write: /run/user/1000/pipewire-0
F: /run is not a symlink
F: /run/user is not a symlink
F: /run/user/1000 is not a symlink
F: /run/user/1000/pipewire-0 is not a symlink
F: Will export read/write: /run/user/1000/pipewire-0
F: Trying to export read/write: /run/user/1000/keyring
F: /run is not a symlink
F: /run/user is not a symlink
F: /run/user/1000 is not a symlink
F: /run/user/1000/keyring is not a symlink
F: Will export read/write: /run/user/1000/keyring
F: Trying to replace with tmpfs: /home/user/.var/app
F: /home is not a symlink
F: /home/user is not a symlink
F: /home/user/.var is not a symlink
F: /home/user/.var/app is not a symlink
F: Will replace with tmpfs: /home/user/.var/app
F: Trying to export read/write: /home/user/.var/app/im.riot.Riot
F: /home is not a symlink
F: /home/user is not a symlink
F: /home/user/.var is not a symlink
F: /home/user/.var/app is not a symlink
F: /home/user/.var/app/im.riot.Riot is not a symlink
F: Will export read/write: /home/user/.var/app/im.riot.Riot
F: Trying to replace with tmpfs: /home/user/.local/share/flatpak
F: /home is not a symlink
F: /home/user is not a symlink
F: /home/user/.local is not a symlink
F: /home/user/.local/share is not a symlink
F: /home/user/.local/share/flatpak is not a symlink
F: Will replace with tmpfs: /home/user/.local/share/flatpak
F: Trying to ensure existence of directory: /home/user
F: /home is not a symlink
F: /home/user is not a symlink
F: Will ensure existence of directory: /home/user
F: Converting FlatpakExports to bwrap arguments...
F: "/home/user" is meant to be a directory
F: Ensuring "/home/user" is created as a directory
F: "/home/user/.local/share/flatpak" is meant to be a tmpfs or empty directory
F: Parent of "/home/user/.local/share/flatpak" is not mapped, creating empty directory
F: "/home/user/.var/app" is meant to be a tmpfs or empty directory
F: Parent of "/home/user/.var/app" is not mapped, creating empty directory
F: "/home/user/.var/app/im.riot.Riot" is meant to be shared (ro or rw) with the container
F: "/run/user/1000/keyring" is meant to be shared (ro or rw) with the container
F: "/run/user/1000/pipewire-0" is meant to be shared (ro or rw) with the container
F: Allowing wayland access
F: Allowing x11 access
F: Allowing pulseaudio access

[smcv]

What exactly do you mean by "stop after allowing pulseaudio access"? There are two different symptoms that "stop" could refer to: do you mean the flatpak process exits and returns control to your shell (a crash, or some other unexpected exit), or do you mean it continues to run but does not make any apparent progress towards running the app (a hang)?

Running flatpak under strace and capturing the last part of the output (everything from Allowing x11 access onwards) would probably be helpful: that would give us more of an idea of what it's doing.

[besendorf]

What exactly do you mean by "stop after allowing pulseaudio access"? There are two different symptoms that "stop" could refer to: do you mean the flatpak process exits and returns control to your shell (a crash, or some other unexpected exit), or do you mean it continues to run but does not make any apparent progress towards running the app (a hang)?

Running flatpak under strace and capturing the last part of the output (everything from Allowing x11 access onwards) would probably be helpful: that would give us more of an idea of what it's doing.

Thanks for reopening the issue.
The process continues to run and does not make any progress. Also Ctrl+C does not kill the process.

Here's the strace output:
strace_flatpak.txt

the last parenthesis is not closed I did not leave anything out

[misbahally]

@besendorf I had the same issue, I found that keybase is causing the problem, I don't know why.
In strace output you provided the last line is openat(40, "keybase", O_RDONLY....

Try closing keybase, make sure its not running in system tray. Then run a flatpak application.

[smcv]

the last parenthesis is not closed I did not leave anything out

This is useful and meaningful: it means that the fstatfs() call made by Flatpak started, but never finished.

What is "keybase"? Is it an application? Does it provide a FUSE filesystem?

My best guess from the strace log provided is that there is some FUSE filesystem mounted on /keybase which hangs forever when we try to query its filesystem type and properties. df would probably have a similar problem.

At first glance this seems like a wrong FUSE filesystem implementation: it should either return results quickly, or return an error quickly.

[smcv]

If my guess is correct, you might think that you would only see this problem for apps that have access to --filesystem=host. However, Flatpak also needs to use the equivalent of something very similar to --filesystem=host when it runs the trusted xdg-dbus-proxy subprocess, so in fact this would affect all apps equally, even if they do not have --filesystem=host themselves.

[smcv]

What is "keybase"? Is it an application? Does it provide a FUSE filesystem?

The answer seems to be: https://github.com/keybase/client, yes, and yes.

My best guess from the strace log provided is that there is some FUSE filesystem mounted on /keybase which hangs forever when we try to query its filesystem type and properties.

This seems to be keybase/client#12458.

[smcv]

This seems unlikely to be fixable from the Flatpak side, so I've labelled this as wontfix - but if someone can provide a merge request that somehow solves this, I'd review it.

[ashleysommer]

I've been seeing this same bug, and I have found an odd workaround:

Weirdly, simply restarting the Keybase Filesystem (KBFS) task fixes the issue:

systemctl --user restart keybase-redirector.service kbfs.service

Flatpak suddenly works again. So something about how keybase-redirector initially creates that directory is what triggers the Flatpak freeze. Restarting kbfs and keybase-redirector re-creates the mountpoint, then Flatpak runs fine after that.

Could it have something to do with the ordering of startup tasks in SystemD?

[leleobhz]

I've been seeing this same bug, and I have found an odd workaround:

Weirdly, simply restarting the Keybase Filesystem (KBFS) task fixes the issue:

systemctl --user restart keybase-redirector.service kbfs.service

Flatpak suddenly works again. So something about how keybase-redirector initially creates that directory is what triggers the Flatpak freeze. Restarting kbfs and keybase-redirector re-creates the mountpoint, then Flatpak runs fine after that.

Could it have something to do with the ordering of startup tasks in SystemD?

Also affected by this issue with the same workarround working well!

[oryband]

Can confirm issue reproduces on Ubuntu 22.04 LTS with Keybase and flatpak installed, and restart keybase-redirector and kbfs service workarounds the issue temporarily as mentioned in #5496 (comment)

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

# uname -a
Linux roach 6.1.0-1027-oem #27-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 15:08:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux