[Feature request]: System extensions from Flatpak

[chergert]

Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Suggestion

This is meant to be a tracking issue for implementing system extensions from Flatpak.

In particular, we can, without much changes to Flatpak (potentially just triggers) extend the host system from a --system installation using modern systemd-sysext features.

I've laid out the gist of this at https://blogs.gnome.org/chergert/2024/05/07/system-extensions-from-flatpak/ which shows a prototype of things working for Sysprof. It allows a /usr/libexec/sysprofd and associated infrastructure to be provided by a Sysprof Flatpak and integrated into the host using systemd-sysext.

Some notable things to work out:

• How to handle SELinux enabled systems without selinux=0. Presumably this will result in improving selinux policies.
• How to handle writability of /usr after a systemd-sysext merge. This may be handled by systemd-sysext: Implement optional mutability for extensions systemd/systemd#31000
• Managing symlinks from flatpak trigger to remove/update symlinks from /var/lib/extensions for apps (honestly runtimes could too) containing extensions.

Though if the trigger bits were implemented it would basically work on GNOME OS out of the box.

[pothos]

• How to handle writability of /usr after a systemd-sysext merge

I think the distro should set the symlinks up like this

$ ls -l /var/lib/extensions.mutable/
total 8
lrwxrwxrwx. 1 root root 4 28. Feb 22:59 etc -> /etc
lrwxrwxrwx. 1 root root 4 28. Feb 23:00 usr -> /usr

and also set the SYSTEMD_SYSEXT_MUTABLE_MODE=auto env var for systemd-sysext.service¹.
Flatpak should only set up the /run/extensions/ symlink and trigger systemctl restart systemd-sysext.service when /var/lib/extensions.mutable/usr exists to prevent causing troubles (read-only /usr or worse, e.g., split state when creating it as regular folder with …MUTABLE_MODE=yes if the user had set this up but stopped using sysext again).
Doing the /var/lib/extensions.mutable/ symlink setup and changing systemd-sysext.service to use …=auto when flatpak --system runs might also be ok but I wonder if it should be opt-in or opt-out because this is surprising behavior.

¹ We don't have a config file yet but it could look like this, e.g., in /etc/systemd/extensions.conf (@krnowak had a patch for that but dropped it from the PR until the need arises which I think is the case now, also because the overlay would ideally be set up in the initrd for the final system to make udev rules, kernel module params, etc. take effect and to run systemd-sysext from there with the --root=/sysroot/ flag we should use the same mutable setting, i.e., we need the config file because it's not possible to load the env var from a unit drop-in under /sysroot/whereever):

[Confext]
Mutable=auto
[Sysext]
Mutable=auto

[pothos]

About building: If static binaries or custom library paths are troublesome or would bloat the size, one can also ship a small bwrap wrapper script in the sysext image (/usr/bin/BIN) as entry point into the Flatpak chroot (I think plain bwrap is better for root than flatpak run --command=/usr/bin/BIN). However, that would mean mounting everything again and maybe we can rather reuse the existing chroot mounts. I think nsenter -m -t PID could work if the original /proc was somewhere available for bind mounting (nsenter --pid results in a valid /proc but that doesn't have the host PIDs of course). Maybe bwrap could be changed to first create a mount namespace with the original /proc and start a "target" process for nsenter there, and then create a second mount namespace after unsharing the PID namespace where a new /proc gets mounted for the app. This way the system service wrappers could use nsenter to land in the mount namespace with the host /proc.
Edit: Scratch that, nsenter would limit this to root and we want regular users to execute this, too. A new bwrap call is better.

FYI: I've explored rebundling non-static binaries into sysext images with two approaches, one is ELF patching, one is using a full chroot with bwrap: flatcar/sysext-bakery#74 For an MVP the ELF patching approach could be helpful but in the end it would be nice to avoid the duplication in Flatpak.

Edit: If the Flatpak chroot is available via OSTree one could also point bwrap there? Or even do FLATPAK/lib/ld-linux-x86-64.so.2 --library-path FLATPAK/lib:FLATPAK/usr/lib….