Flatpak cannot handle empty /var/lib/flatpak

[AdrianVovk]

Linux distribution and version

carbonOS 2021.1 (my own in-dev distro)

Flatpak version

1.9.2

Description of the problem

I'm trying to have systemd-tmpfiles do some fs-related setup on /var/lib/flatpak. Basically, I have tmpfiles A) create a btrfs subvolume at /var/lib/flatpak and then B) enable compression on /var/lib/flatpak. Doing it this way ensures that whatever flatpak ends up writing to there gets compressed. However, flatpak breaks in various ways whenever /var/lib/flatpak exists and is empty. It seems like something goes wrong with the system helper (see example below).

My best guess that some parts of flatpak just check for the existence of /var/lib/flatpak and then assume that flatpak is set-up, while other parts actually go through and try to initialize the contents of /var/lib/flatpak and fail due to lack of privileges.

Steps to reproduce

$ sudo rm -rf /var/lib/flatpak
$ sudo mkdir /var/lib/flatpak        OR       $ sudo btrfs subvolume create /var/lib/flatpak
$ sudo chmod 0755 /var/lib/flatpak
$ sudo chown root:root /var/lib/flatpak
$ flatpak repair
error: Creating repo: mkdirat: Permission denied

Similar problems also occur with flatpak install, with errors like "Ref not found" or other permission errors. Running flatpak repair with root privilege does, indeed, work, and also fixes the rest of Flatpak. However, when /var/lib/flatpak does not exist at all, no such issues occur: flatpak repair does not need root access and flatpak install just works without repairing.

[tinywrkb]

I'm also hitting this issue.
Can we have a flatpak init command that initializes the user/system Flatpak installation?
This way we can have a dedicated polkit action-id so we would able to allow it for the wheel group and it doesn't have to be enabled by default on vanilla distros.
Also, it should (optionally) add remotes from /etc/flatpak/remotes.d.

My setup is a little different. My Flatpak system installation lives in /apps as I have a (semi-)stateless system where the system state is only saved in /var (/etc is a symlink to /usr/etc) as I like to be able to reset the system but keep installed apps.
The mounted rootfs is read-only so /apps must exist, even if empty.

My current workaround is to package an empty flatpak installation, created with flatpak remote-add --system --if-not-exists ... and then use tmpfiles.d to copy the repo folder to /apps/repo.

[AdrianVovk]

@tinywrkb that's what flatpak repair does when the system installation dir is not present, at least on my system. The ostree repo is init'd, the remotes are loaded properly, and it seems to use the system helper because, as a member of wheel, I don't even get a polkit prompt.

However, when /var/lib/flatpak exists but is empty, all that breaks

[mkrajnak]

I am hitting this on a distro that comes without a default system remote, thus the /var/lib/flatpak is empty.

At first, I through it's some gnome-software issue, because it was where I first saw the error message. Then I found this issue and I moved on Fedora. Based on the reproducer in the first comment I tried to replicate the "no default system remote" situation (I want to get rid of system remotes so I can have all flatpaks in my home partition).

$ rpm -q flatpak 
flatpak-1.11.1-1.fc35.x86_64
$ sudo rm -rf /var/lib/flatpak
$ sudo mkdir /var/lib/flatpak  
$ sudo chmod 0755 /var/lib/flatpak
$ sudo chown root:root /var/lib/flatpak
$ pkill gnome-software

Gnome-software starts and shows error: Creating repo: mkdirat: Permission denied

I wondered where the error is comming from, so I added flathub and that is where I've seen the message coming from flatpak:

$ flatpak remote-add --user --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
$ flatpak remote-info flathub 
error: Creating repo: mkdirat: Permission denied
$ flatpak remote-ls flathub 
error: Creating repo: mkdirat: Permission denied
$ flatpak remote-delete flathub
error: Creating repo: mkdirat: Permission denied

Of course, it went away after I executed # flatpak repair. Is there a mechanism that can be implemented so flatpak won't fail to execute those actions on --user installations without existence of /var/lib/flatpak ? Or some kind of post install hook that will populate it ?

[smcv]

#4732 should resolve at least the parts of this that interact with the flatpak(1) CLI.

$ sudo chown root:root /var/lib/flatpak
$ flatpak repair

I don't think this is necessarily doing what you think it's doing. flatpak repair is equivalent to flatpak repair --system, which won't work unless you have root privileges. #4732 swaps the order of operations around so that you get the message Privileges are required to make changes before it attempts to do privileged things, so that you have a hint that you would need sudo flatpak repair for this to actually do anything useful.

Can we have a flatpak init command that initializes the user/system Flatpak installation?

There are already various harmless commands that have this as a side-effect.

Most commands will already initialize the user installation in ~/.local/share/flatpak as a side-effect.

If you are root, most commands will already initialize the system installation. For example, the Debian packaging uses flatpak remote-list --system > /dev/null || : to achieve this.

If you are not root, the first operation that writes to the system installation, such as flatpak remote-add [--system] or flatpak install [--system], will initialize the system repo as a side-effect, making use of the same privileged helper (flatpak-system-helper) that does the actual remote-add or install operation. Until then, I think the ideal situation is that everything will treat the system installation as though it had a libostree repository but was empty (no remotes, no apps, no runtimes). #4732 implements that (or at least, implements most of that - there might be a few that I missed).

One thing that doesn't currently exist (I don't think) is a way for non-root users to trigger inclusion of /etc/flatpak/remotes.d/ into a system installation that is an empty directory, using the system helper if necessary, so that they can subsequently use it for read-only commands like flatpak search. It might make sense for that to be a side-effect of flatpak update as an unprivileged user - at the moment, flatpak update will install /etc/flatpak/remotes.d/ into a system installation that already has a libostree repo, but will not install them into a system installation that is completely empty.

[initializing the repo is] what flatpak repair does when the system installation dir is not present

I think flatpak repair is a big hammer that people should ideally never need to use. It lets you recover from bugs, but it would be best if it was never necessary to run it.

Is there a mechanism that can be implemented so flatpak won't fail to execute those actions on --user installations without existence of /var/lib/flatpak ?

Yes, see #4732.

Gnome-software starts and shows error: Creating repo: mkdirat: Permission denied

It's not clear to me whether this has already been resolved, or whether it will be resolved by #4732, or whether it's a separate libflatpak issue not resolved by #4732, or whether this is something that will have to be looked at by gnome-software developers.

[smcv]

One thing that doesn't currently exist (I don't think) is a way for non-root users to trigger inclusion of /etc/flatpak/remotes.d/ into a system installation that is an empty directory, using the system helper if necessary, so that they can subsequently use it for read-only commands like flatpak search

Updated version of #4732 resolves that, as well:

$ cat /etc/flatpak/remotes.d/flathub.flatpakrepo
... same contents as https://dl.flathub.org/repo/flathub.flatpakrepo ...
$ sudo rm -fr /var/lib/flatpak ~/.local/share/flatpak && sudo mkdir /var/lib/flatpak
$ ./flatpak search Steam
... metadata is downloaded, the list is non-empty ...