Goofys Experiences SystemD Ordering Cycle issues during dismounting at OS Poweroff

[oven8Mitts]

Hello,

Correct me if I am posting this in the wrong place, as I'm unsure exactly which piece of software could be causing this. I have an EC2 instance with EFS/NFS mounts, and S3 mounts using Goofys.

During shutdowns of the OS (Debian 10, in AWS), we experience issues with the server gracefully shutting down. Visible in the syslog, these are the first problem I see
(All .mount files here that are referenced are Goofys mounts)

Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found ordering cycle on network-online.target/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found dependency on networking.service/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found dependency on local-fs.target/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found dependency on mnt-www\x2dtrades\x2dgate-TradeInFiles.mount/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Job network-online.target/stop deleted to break ordering cycle starting with mnt-www\x2dtrades\x2dgate-TradeInFiles.mount/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found ordering cycle on mnt-www\x2ddata\x2dgate-computescripts.mount/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found dependency on network.target/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found dependency on networking.service/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found dependency on local-fs.target/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Job mnt-www\x2ddata\x2dgate-computescripts.mount/stop deleted to break ordering cycle starting with local-fs.target/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found ordering cycle on mnt-www\x2ddata\x2dgate-ib_meetings.mount/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found dependency on network.target/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found dependency on networking.service/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Found dependency on local-fs.target/stop
Aug  5 20:21:17 localhost systemd[1]: local-fs.target: Job mnt-www\x2ddata\x2dgate-ib_meetings.mount/stop deleted to break ordering cycle starting with local-fs.target/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found ordering cycle on network.target/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found dependency on networking.service/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found dependency on local-fs.target/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Found dependency on mnt-www\x2dtrades\x2dgate-TradeInFiles.mount/stop
Aug  5 20:21:17 localhost systemd[1]: mnt-www\x2dtrades\x2dgate-TradeInFiles.mount: Job network.target/stop deleted to break ordering cycle starting with mnt-www\x2dtrades\x2dgate-TradeInFiles.mount/stop

Logs stop getting recorded shortly hereafter. A poweroff/reboot event is not recorded.

After research and investigation, I found that Goofys mounts appear to order strangely compared to other network mounts(AWS EFS).

A Goofys requests to be attached prior to both "local-fs.target" as well as "remote-fs.target" is reached. There is also an "after" requirement for "local-fs-pre.target" and "remote-fs-pre.target". If I understand correctly, these two occur at different parts of the boot cycle and might conflict. (Correct me if I am wrong).

myUserName@redactedHostname:~$ systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After mnt-myS3BucketName.mount
Requires=system.slice -.mount
Requisite=
Wants=network-online.target
BindsTo=
PartOf=
Before=remote-fs.target umount.target local-fs.target
After=system.slice remote-fs-pre.target -.mount network.target local-fs-pre.target network-online.target systemd-journald.socket

From this stackoverflow page, it appears that removing the local-fs related entries should fix it, although I have not figured out how to do that through fstab:
https://unix.stackexchange.com/questions/519221/how-can-i-solve-this-ordering-cycle-in-a-mount-unit

Here is my mount statement:
goofys#redactedBucket /mnt/BucketName fuse allow_other,--uid=0,--gid=1017,--file-mode=0660,--dir-mode=0770,_netdev 0 0

I've removed the _netdev statement and observed that the local-fs related targets remain, while remote targets are removed. It appears that the presence of _netdev does not remove local-fs, but it only adds remote-fs targets.

Here is what an EFS mount looks like:

myUserName@redactedHostname:~$ systemctl show -p Requires,Wants,Requisite,BindsTo,PartOf,Before,After var-redacted-mount-name.mount
Requires=system.slice -.mount
Requisite=
Wants=network-online.target
BindsTo=
PartOf=
Before=remote-fs.target umount.target
After=system.slice systemd-journald.socket network-online.target remote-fs-pre.target -.mount network.target

I've attempted to add "x-systemd.*" flags as listed here, but setting x-systemd.defaultdependencies=no had no affect, and setting dependencies such as x-systemd.after=remote-fs.target did not have an effect on the entry.

Setting x-systemd.after=basic.target removed all local-fs targets and retained remote-fs targets, although I don't know what will break if that is set. Removing the basic.target statement also did not re-add local-fs targets until the next reboot.

All assistance or workarounds are appreciated. My next workaround is creating .mount files rather than using fstab, although it would be nice to avoid that.

[oven8Mitts]

I've done more testing.

There appears to be something happening when Systemd attaches the mount. The mount options are correctly parsed by systemd before a mount is mounted, and after it is dismounted. The mount options appear to change while the mount is attached though. Disabling default dependencies in systemd fixed this issue (see below).

Steps to reproduce:

1. Create an fstab entry or a mount unit file for an S3 bucket using goofys. I will use a mount unit file since I can add the DefaultDependencies flag in this file. (In this example, goofys#yourBucketName /mnt/yourBucketName fuse _netdev,allow_other 0 0 or a unit file equivalent is used to generate the settings)
2. Once the entry is created, run systemctl daemon-reload to have systemd update the mount's settings.
3. Run systemctl show -p Options,Requires,Wants,Before,After mnt-yourMountName.mount
4. Observe the correct settings being listed:

Options=_netdev,allow_other
Requires=-.mount system.slice
Wants=network-online.target
Before=remote-fs.target umount.target
After=network.target systemd-journald.socket network-online.target remote-fs-pre.target -.mount system.slice

5. Actually mount the share by running systemctl start mnt-yourMountName.mnt
6. Rerun the show command systemctl show -p Options,Requires,Wants,Before,After mnt-yourMountName.mount to observe the options changing:

Options=rw,relatime,user_id=0,group_id=0,default_permissions,allow_other
Requires=-.mount system.slice
Wants=network-online.target
Before=umount.target local-fs.target remote-fs.target
After=-.mount network-online.target remote-fs-pre.target system.slice systemd-journald.socket local-fs-pre.target network.target

7. Dismount the directory by running systemctl stop mnt-yourMountName.mnt
8. Run systemctl daemon-reload again (despite not making any changes)
9. Observe the options change back..:

Options=_netdev,allow_other
Requires=-.mount system.slice
Wants=network-online.target
Before=remote-fs.target umount.target
After=-.mount network-online.target network.target remote-fs-pre.target system.slice systemd-journald.socket

The options change can be prevented by manually creating a systemd unit mount file that includes the DefaultDependencies=no flag. Please note then you must add in your own dependencies such as network-online.target.

1. Copy the generated unit file from the location listed from this command systemctl show -p FragmentPath "mnt-yourMountName.mount" to a working directory.
2. Modify the unit file to comment out the SourcePath and Documentation statements.
3. Add in these statements in the unit:

[Install]
RequiredBy=remote-fs.target
[Unit]
DefaultDependencies=no
Requires=system.slice -.mount
Wants=network-online.target
Conflicts=umount.target
Before=remote-fs.target umount.target
After=remote-fs-pre.target system.slice network-online.target -.mount network.target systemd-journald.socket

4. Copy the file into /etc/systemd/system/mnt-yourMountName.mount and comment out the fstab entry.
5. Run systemctl daemon-reload
6. Enable the mount for startup using systemctl enable mnt-yourMountName.mount
7. Reboot or start the mount using systemctl start mnt-yourMountName.mount

This otherwise fixed my ordering issue.
I have no idea why systemd overrides _netdev in mount options and changes variables that are already defined, while not doing this to other mounts. Not sure if it's a goofys or systemd or fuse bug, but I've only experienced it with Goofys. Disclaimer:I take no responsibility for anything my workaround might break.

Edit: I just realized I forgot to actually add "DefaultDependencies=no" into the unit file example. Fixed