Add workaround for SFTP servers that don't support O_APPEND

[adam-nielsen]

I recently upgraded my system and in the process went from sshfs 3.0.0 to 3.3.1 and I can no longer use git over an sshfs mount.

$ git pull
error: cannot open .git/FETCH_HEAD: Permission denied

The file itself has no permission issues:

$ rm .git/FETCH_HEAD
$ echo create > .git/FETCH_HEAD
$ cat .git/FETCH_HEAD
create
$ git pull
error: cannot open .git/FETCH_HEAD: Permission denied
$ cat .git/FETCH_HEAD
[now a zero-length file]

Running git through strace says:

openat(AT_FDCWD, ".git/FETCH_HEAD", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 EACCES (Permission denied)

Why would openat() be returning EACCES when whatever functions the shell is using work just fine on the same file path?

Is anyone else able to run git fetch on a git repository accessed over sshfs?

[Nikratio]

Thanks for the report! Most likely the problem is with O_APPEND. There's been some code churn related to that.

[Nikratio]

Specifically, only since version 3.2.0 SSHFS actually supports O_APPEND (by forwarding it to the server). Before that, this flag was a no-op. I suspect your SFTP server does not support it. Could you run with --debug and post the output? And maybe comment out https://github.com/libfuse/sshfs/blob/master/sshfs.c#L2526 and see if that works around the problem?

[adam-nielsen]

Many thanks for the quick response! You are correct - commenting out the O_APPEND line does indeed fix the problem.

The debug log I am getting doesn't look terribly useful, but here are what appears to be the relevant bits, let me know if you would prefer the full thing:

FUSE library version: 3.2.1
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.26
flags=0x001ffffb
max_readahead=0x00020000
   INIT: 7.26
   flags=0x0000f039
   max_readahead=0x00020000
   max_write=0x00020000
   max_background=0
   congestion_threshold=0
   time_gran=1000000000
   unique: 1, success, outsize: 80
...
unique: 624, opcode: OPENDIR (27), nodeid: 32, insize: 48, pid: 25392
   unique: 623, success, outsize: 16
   unique: 624, success, outsize: 32
unique: 625, opcode: READDIRPLUS (44), nodeid: 32, insize: 80, pid: 25392
   unique: 625, success, outsize: 912
unique: 626, opcode: READDIR (28), nodeid: 32, insize: 80, pid: 25392
   unique: 626, success, outsize: 16
unique: 627, opcode: RELEASEDIR (29), nodeid: 32, insize: 64, pid: 0
   unique: 627, success, outsize: 16
unique: 628, opcode: LOOKUP (1), nodeid: 4, insize: 51, pid: 25392
   unique: 628, success, outsize: 144
unique: 629, opcode: OPEN (14), nodeid: 30, insize: 48, pid: 25392
   unique: 629, error: -13 (Permission denied), outsize: 16
unique: 630, opcode: FLUSH (25), nodeid: 21, insize: 64, pid: 25392
   unique: 630, success, outsize: 16
unique: 631, opcode: RELEASE (18), nodeid: 21, insize: 64, pid: 0
   unique: 631, success, outsize: 16

If I telnet to the server on port 22, I am greeted with: SSH-2.0-mod_sftp/0.9.7 if that is useful for any sort of feature blacklist.

[Nikratio]

Arg, sorry. The correct option is -o sshfs_debug. That should be more helpful.

That said, there isn't much we can do at the SSHFS side. The server is refusing to open files with O_APPEND, sshfs is just forwarding that error. Just stripping away that option doesn't seem like a good idea - the application is probably using it for a reason.

[Nikratio]

Best solution is probably to add another workaround that emulates O_APPEND by always seeking to EOF before writing (similar to what NFS does).

[mschilli87]

Are there still plans to implement this?

It appears I ran into this trying to commit to a Git repository that is mounted over SSH.

It boils down to this:

sshfs -V

SSHFS version 3.7.0
FUSE library version 3.9.1
using FUSE kernel interface version 7.31
fusermount3 version: 3.9.1

grep /etc/fstab <remote-machine>

<remote-user>@<remote-machine>: /mnt/ssh/<remote-machine>  sshfs  _netdev,user,idmap=user,allow_other  0 0

cat /etc/fuse.confs

user_allow_other

1. Mount remote with debug output:

 mount -o sshfs_debug MOUNTPOINT

SSHFS version 3.7.0
executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-2> <USER@SERVER> <-s> <sftp>
USER@SERVER's password:
Server version: 3
Extension: versions <2,3,4,5,6>
Extension: [email protected] <1>
Extension: [email protected] <1>
Extension: [email protected] <2>
Extension: [email protected] <2>
Extension: [email protected] <1>
remote_uid = 0

2. In a different terminal, access the mounted share:

cd MOUNTPOINT/DIR_WITH_WRITE_PERMISSIONS

[00002] LSTAT
  [00002]          ATTRS       45bytes (188ms)

3. Verify regular writing works:

echo foo > foobar

[00003] LSTAT
  [00003]         STATUS       38bytes (46ms)
[00004] LSTAT
  [00004]         STATUS       38bytes (32ms)
[00005] LSTAT
  [00005]          ATTRS       45bytes (242ms)
[00006] OPENDIR
  [00006]         HANDLE       29bytes (31ms)
[00007] READDIR
[00008] READDIR
  [00007]           NAME      668bytes (58ms)
[00009] READDIR
[00010] READDIR
  [00008]           NAME      483bytes (65ms)
[00011] READDIR
[00012] READDIR
  [00009]         STATUS       37bytes (27ms)
  [00010]         STATUS       37bytes (27ms)
[00013] CLOSE
[00014] LSTAT
  [00011]         STATUS       37bytes (27ms)
  [00012]         STATUS       37bytes (27ms)
  [00013]         STATUS       28bytes (26ms)
  [00014]         STATUS       38bytes (31ms)
[00015] OPEN
[00016] LSTAT
  [00015]         HANDLE       29bytes (153ms)
  [00016]          ATTRS       45bytes (158ms)
[00017] FSTAT
  [00017]          ATTRS       45bytes (29ms)
[00018] WRITE
  [00018]         STATUS       28bytes (28ms)
[00019] CLOSE
  [00019]         STATUS       28bytes (28ms)

4. Trigger error by attempting to append:

echo bar >> foobar

[00020] LSTAT
  [00020]         STATUS       38bytes (74ms)
[00021] LSTAT
  [00021]         STATUS       38bytes (57ms)
[00022] LSTAT
  [00022]          ATTRS       45bytes (52ms)
[00023] OPENDIR
  [00023]         HANDLE       29bytes (53ms)
[00024] READDIR
[00025] READDIR
  [00024]           NAME      668bytes (68ms)
[00026] READDIR
[00027] READDIR
  [00025]           NAME      597bytes (77ms)
[00028] READDIR
[00029] READDIR
  [00026]         STATUS       37bytes (47ms)
[00030] CLOSE
  [00027]         STATUS       37bytes (47ms)
[00031] OPEN
[00032] LSTAT
  [00028]         STATUS       37bytes (47ms)
  [00029]         STATUS       37bytes (47ms)
  [00030]         STATUS       28bytes (26ms)
  [00031]         STATUS       43bytes (28ms)
  [00032]          ATTRS       45bytes (29ms)

zsh: permission denied: foobar

---

Note: I asked for help over at StackExchange first in the hope of a work-around but all I got was a pointer to this issue.
Please have a look there if you think I am stupid for trying to use Git over SSHFS and feel free to suggest a work-around for my particular use case. 😉

[Nikratio]

Thanks for the detailed report!

I am afraid as far as I'm aware there have never been concrete plans to implement this. However, if someone wanted to work on this, then I'd be happy to review patches and (hopefully) merge them.

[mschilli87]

@Nikratio: Thank you for the prompt reply and the heads up.

I have a feeling this is probably not the kind of issue for a first contribution. If you think it is doable by somebody unfamiliar with the codebase who used C only for small hobbyist-type one-man projects (couple of hundred lines a year at best), I could give it a shot though. Some pointers of where to start would be great in that case.

@adam-nielsen: Did you ever find a viable work-around (other than patching Git)?

[devidasjadhav]

@mschilli87 I can help you debug this issue.
I need your help in two ways.

1. provide server which I can clone and reproduce issue.
2. try adding logs to find root cause. i.e.

error is EACCES.
there are two instances of this.
both in sshfc.c possible candidate is
function sftp_error_to_errno.
which intern used in 3 places.

1. sftp_request_wait
2. sshfs_read_end
3. sftp_check_root

I think sshfs_read_end related to O_APPEND.
which is part of sshfs_send_read

I need help adding logs to verify this hypothesis.

[devidasjadhav]

I tried using sshfs to clone github repo. but no success.

dev@debian:~/a/sshfs-3.7.0/builddir$ ./sshfs [email protected]:devidasjadhav/blog.git ./mnt -d -o no_readahead
SSHFS version 3.7.0
executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-2> <[email protected]> <-s> <sftp>
subsystem request failed on channel 0
remote host has disconnected

maybe I am not using it right way. I will attempt a patch. if I am successful.
an interesting issue.

[adam-nielsen]

@mschilli87 When I was testing I disabled O_APPEND and have been using that ever since to work with git:

diff --git a/sshfs.c b/sshfs.c
index 7fe9c6e..ddabe5e 100644
--- a/sshfs.c
+++ b/sshfs.c
@@ -2523,8 +2523,8 @@ static int sshfs_open_common(const char *path, mode_t mode,
        if (fi->flags & O_TRUNC)
                pflags |= SSH_FXF_TRUNC;
 
-       if (fi->flags & O_APPEND)
-               pflags |= SSH_FXF_APPEND;
+//     if (fi->flags & O_APPEND)
+//             pflags |= SSH_FXF_APPEND;
        
        sf = g_new0(struct sshfs_file, 1);
        list_init(&sf->write_reqs);

Definitely a dodgy fix but it has been working for a while now.

[mschilli87]

@devidasjadhav:

There is no need to get git involved to see this error. See my previous mesage how to reproduce the issue with >> only.

I need your help in two ways.

1. provide server which I can clone and reproduce issue.

If that would be my server, I could probably active support for O_APPEND on the server side and be happy.
As it is not, I can't give anyone else access.
Icould look into trying to reproduce this locally though. I should be able to provide some code to run an SFTO server and mount via localhost. Let me know if this is be something that would help you help me.

2. try adding logs to find root cause

All the logs I can think of are already included in my previous mesage.

---

@adam-nielsen:

Thank you for getting back at me.
This is a good start.

---

@Nikratio:

Would @adam-nielsen's hack be an acceptable solution via an 'offical' -o workaround (similar to -o unreliable_append that got removed in d193b19 - by you actually as it turns out 😉)?
Is there an easy way to test on the client side if a server supports O_APPEND so the if statement could be expanded to test for that (and maybe issue a warning when dropping the flag)?
If not, we could achieve the same behaviour using try/catch (if O_APPEND results in EACCESS, issue a warning and try again without O_APPEND).