"bwrap: can't make symlink at /var/run: File exists" when /media is a symlink to /var/run/media

[tallero]

Linux distribution and version

Archlinux, Builder v3.36.0, 3.34, 3.32

Flatpak version

1.6.2

Description of the problem

When I try to run to run any project in (distro's) Builder I do get after correctly building the app

Application started at 04:59:29 AM
bwrap: Can't make symlink at /var/run: File exists
Application exited

It happened through 3.34 branch and even on 3.32 but at the time it went away with

systemctl start --user xdg-desktop-portal

If I try to run flatpak'd Builder instead, I do get

➜  Projects flatpak --verbose run org.gnome.Builder
F: No installations directory in /etc/flatpak/installations.d. Skipping
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/user/.local/share/flatpak
F: Opening system flatpak installation at path /var/lib/flatpak
F: Cleaning up unused container id 2123384349
F: Allocated instance id 2731448443
F: Add values in dir '/org/gnome/builder/', prefix is '/org/gnome/Builder/'
F: Add defaults in dir /org/gnome/Builder/
F: Add locks in dir /org/gnome/Builder/
F: writing D-Conf values to /home/user/.var/app/org.gnome.Builder/config/glib-2.0/settings/keyfile
F: Allowing dri access
F: Allowing host-fs access
F: Allowing homedir access
F: Allowing wayland access
F: Allowing x11 access
F: Allowing session-dbus access
F: Allowing system-dbus access
F: Running 'bwrap --args 27 xdg-dbus-proxy --args=33'
F: Running 'bwrap --args 30 gnome-builder'
bwrap: Can't make symlink at /var/run: File exists

Steps to reproduce

1. Create a new GNOME application in GNOME Builder (any language)
2. Click on run

Maybe something is wrong in my /var. Do you know where should I look?

References

• Fix issues with XDG_RUNTIME_DIR being /var/run which is a symlink by alexlarsson · Pull Request #2710 · flatpak/flatpak
• FS#65876 : Can't run applications with GNOME Builder
• Can't run any project: "bwrap: can't make symlink at /var/run: File exists" (#1164) · Issues · GNOME / gnome-builder · GitLab

[probonopd]

Even without GNOME Buidler, I've been running into the same issue since basically forever (preventing me from testing Flatpak on Ubuntu Live ISOs). E.g., on the Xubuntu 18.04.2 LTS Live ISO:

me@host:~$ flatpak --version
Flatpak 1.6.2 # from PPA

me@host:~$ flatpak run --user --command=bash org.gnome.gedit

Note that the directories 

'/var/lib/flatpak/exports/share'
'/home/me/.local/share/flatpak/exports/share'

are not in the search path set by the XDG_DATA_DIRS environment variable, so
applications installed by Flatpak may not appear on your desktop until the
session is restarted.

bwrap: Can't make symlink at /var/run: File exists

[tallero]

@probonopd Did you try with systemctl --user start xdg-desktop-portal?

[probonopd]

Originally I hadn't, but even if I do, the result is the same. Thanks @tallero

[tallero]

I am sorry. On 18.04's GNOME release that solved the bug for me at the time.

[tallero]

Temporarly moving /var/run solves the problem, but isn't it created at boot?

[probonopd]

E.g., on Xubuntu Bionic Live ISO:

me@host:~$ ls -lh  /var/run
lrwxrwxrwx 1 root root 4 Feb  3 19:22 /var/run -> /run

me@host:~$ mount | grep "on /run"
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/user/999 type tmpfs (rw,nosuid,nodev,relatime,size=1007308k,mode=700,uid=999,gid=999)
gvfsd-fuse on /run/user/999/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=999,group_id=999)

[Yiannis128]

Running into this issue as well...

Also FYI for anyone following advice about messing with /var/run, don't it will cause your system to not boot because GDM needs NetworkManager and NetworkManager relies on that symlink.

[tallero]

@Yiannis128 wait I said temporarily

[Yiannis128]

@tallero Yeah, in hindsight I can see that what you said is correct, the issue is that I didn't know that it was "temporarily as in move it back or you soft brick your OS", I decided to let other people know by making it extra clear :) Also, while it does fix some apps like Gnome Builder not launching, it breaks NetworkManager, so working on apps like Gnome Web is impossible without directly solving this issue.

[tallero]

Yes, I specified it because an year ago I've temporarily broken my system, too.

[smcv]

Running the org.gnome.Builder Flatpak app works fine for me (with Flatpak 1.10.2 on Debian 11).

Do you have any Flatpak permissions overrides, either for GNOME Builder or globally? I suspect this can be triggered by sharing more filesystems with the app. Try:

$ flatpak override --show
$ flatpak override --show org.gnome.Builder
$ flatpak override --user --show
$ flatpak override --user --show org.gnome.Builder

Also try running flatpak run -vv (-vv instead of --verbose) which would show the full bubblewrap command-line.

Also FYI for anyone following advice about messing with /var/run, don't

Certainly don't remove /var/run, it's a necessary part of the OS. On most Linux distributions the expected situation is that you have a symlink /var/run -> /run and a tmpfs on /run.

[smcv]

@Yiannis128, you haven't said what version of Flatpak you are using, or which operating system you are using?

[Yiannis128]

@smcv I'm using Manjaro running kernel 5.11.2-1-MANJARO. For Flatpak I have version 1.10.1 installed.

[Yiannis128]

Running the org.gnome.Builder Flatpak app works fine for me (with Flatpak 1.10.2 on Debian 11).

Do you have any Flatpak permissions overrides, either for GNOME Builder or globally? I suspect this can be triggered by sharing more filesystems with the app. Try:

$ flatpak override --show
$ flatpak override --show org.gnome.Builder
$ flatpak override --user --show
$ flatpak override --user --show org.gnome.Builder

Also try running flatpak run -vv (-vv instead of --verbose) which would show the full bubblewrap command-line.

Also FYI for anyone following advice about messing with /var/run, don't

Certainly don't remove /var/run, it's a necessary part of the OS. On most Linux distributions the expected situation is that you have a symlink /var/run -> /run and a tmpfs on /run.

I don't know what Flatpak permission overrides are so I assume I don't have any...

Here is the output of each of the requested commands

Here is the output of the -vv run command, it's big so I dumped it at TextBin:
https://textbin.net/rn8aclzcvb

[smcv]

This is correct:

--symlink ../run /var/run

but then, later, this also appears in the command-line:

--symlink ../var/../run /var/run

and I think that might be what is breaking it for you.

It looks as though your /media is a symbolic link to /var/run/media.

If you change that to be a symbolic link to /run/media, does that help?

[smcv]

OK, I can reproduce this by changing my /media (which is normally a real directory on Debian) to be a symlink to /var/run/media, and creating that directory.

Replacing /media with a symlink to /run/media seems to be successful as a workaround for this.

[Yiannis128]

OK, I can reproduce this by changing my /media (which is normally a real directory on Debian) to be a symlink to /var/run/media, and creating that directory.

Replacing /media with a symlink to /run/media seems to be successful as a workaround for this.

The thing is that wasn't a directory in my original Manjaro install, I created it because I wanted an easy way to access /run/media since I was used to /media before using Systemd. I'll remap /media to /run/media....

[Yiannis128]

OK, I can reproduce this by changing my /media (which is normally a real directory on Debian) to be a symlink to /var/run/media, and creating that directory.

Replacing /media with a symlink to /run/media seems to be successful as a workaround for this.

Thanks this fixed it. But can you explain why this was an issue in the first place? I want to understand the reason this happened.

[smcv]

But can you explain why this was an issue in the first place?

Not easily, and I'd prefer to spend that time writing a merge request to fix this for everyone permanently.

If you want to, you have access to the complete source code and can trace through exactly what Flatpak does, though :-)

[Yiannis128]

But can you explain why this was an issue in the first place?

Not easily, and I'd prefer to spend that time writing a merge request to fix this for everyone permanently.

If you want to, you have access to the complete source code and can trace through exactly what Flatpak does, though :-)

Understandable, thanks for the help regardless.

[mirkobrombin]

I'm facing the same issue in a setup where /var is a symlink of /.system/var. What exactly is going on under the hood that makes Flatpak/bwrap so sensitive to symlinks?

[smcv]

I'm facing the same issue in a setup where /var is a symlink of /.system/var.

Please report that separately, with flatpak -vv output, and mention that /var is a symlink of /.system/var in the issue title. Without seeing a log, I can't say whether a change intended to fix the /var/run/media issue would work for you or not.

The symptom is the same, but the root cause is different (similar to the relationship between #2200 and this issue).

[smcv]

What exactly is going on under the hood that makes Flatpak/bwrap so sensitive to symlinks?

The short version? "Containers".

Flatpak uses Linux bind-mounts to set up its sandbox environment, but there are several factors that make bind-mounts and symlinks conflict.

The most obvious one is that the sandbox environment is a different mount namespace, so if we set up a symlink inside the sandbox environment pointing to something that isn't available in the sandbox, it won't work. We have to figure out what the symlink points to on the host, bind-mount that into the sandbox instead, and then replicate the symlink structure that points to it.

This is made harder by the fact that the Linux kernel doesn't let us bind-mount a symlink into the sandbox, or mount on top of a symlink: it will always follow the symlink and act on its target instead (it's like stat() where what we want is more like lstat()).

It's also made harder by the fact that some parts of the filesystem are special to the Flatpak runtime, notably /usr, /etc, /run and /var/run, all of which we need to set up in specific ways otherwise the app won't work. It's easy for something we do for --filesystem to conflict with something we do for the Flatpak runtime, and in particular that's what is happening when /media is a symlink to /var/run/media.

[mirkobrombin]

Thanks for the detailed answer. I fixed it by binding the path /var instead of using a symlink.

[smcv]

I'm facing the same issue in a setup where /var is a symlink of /.system/var.

#5213 might help with that, actually.

I fixed it by binding the path /var instead of using a symlink

I would always recommend using mount points rather than symlinks for this sort of OS-level filesystem aliasing, where the existence of /.system is (presumably) an implementation detail and you still want /var to be treated as the canonical path.

[smcv]

@tallero: Was this symptom caused for you by the same setup that @Yiannis128 reported (/media a symlink to /var/run/media)?

If not, please provide the information I asked for in #3477 (comment), similar to #3477 (comment).

[mirkobrombin]

I would always recommend using mount points rather than symlinks for this sort of OS-level filesystem aliasing, where the existence of /.system is (presumably) an implementation detail and you still want /var to be treated as the canonical path.

I figured it out the hard way.

[deftdawg]

I overrode /var/run/docker.sock with a link to the host and now my flatpak won't start because it already exists and I can find no way of removing the override...

How in do I nuke the override? Flatseal couldn't reset, even uninstalling the flatpak doesn't reset it...

flatpak override --show com.visualstudio.code

[Context]
filesystems=/var/run/docker.sock;/run/docker.sock;

(this is with the flatpak no longer installed)

[smcv]

I overrode /var/run/docker.sock

This is off-topic in this issue report, despite the similar error message, because this issue report is specifically about /media being a symlink to /var/run/media. I'm trying to keep to 1 issue = 1 report, to reduce confusion and avoid having issue reports that can never be closed because nobody knows whether all their root causes have been dealt with.

To remove filesystems=/var/run/docker.sock;/run/docker.sock; overrides, use:

flatpak override --nofilesystem=/run/docker.sock --nofilesystem=/var/run/docker.sock com.visualstudio.code

The four possible sources of overrides are:

1. flatpak override ... com.example.app will add or remove overrides in /var/lib/flatpak/overrides/com.example.app
2. flatpak override --user ... com.example.app will add or remove overrides in ~/.local/share/flatpak/overrides/com.example.app
3. flatpak override ... without an app ID will add or remove overrides in /var/lib/flatpak/overrides/global
4. flatpak override --user ... without an app ID will add or remove overrides in ~/.local/share/flatpak/overrides/global

For each of those four types of override, you can use flatpak override --show (with or without an app ID, and with or without --user) to show the overrides that exist.

[deftdawg]

Thank you, this helped me find the flatpak overrides were in /var/lib/flatpak/overrides, I ended up doing sudo rm /var/lib/flatpak/overrides/com.visualstudio.code to reset the package overrides (the --nofilesystem override explicitly excluded the files with ! prefix which inverts the override but doesn't remove it)