mv fails with EPERM when moving a file between different remote FS's

[g-raud]

Moving files on a same sshfs mount but between different FS's on the
remote host fails. The server is OpenSSH 7.4p1-10+deb9u1 from Debian
with default config (sftp-server enabled). The sshfs command is also
used with the default options; note that the option workaround=rename
does not change this error case:

$ sshfs -V
SSHFS version 2.8
FUSE library version: 2.9.7
fusermount version: 2.9.7
using FUSE kernel interface version 7.19
$ sshfs -o debug,sshfs_debug,loglevel=debug -f host:/ /home/user/sshfs/ | vipe
unique: 518, opcode: RENAME (12), nodeid: 27, insize: 114, pid: 20235
rename /mnt/fs1/dir1/file /mnt/fs2/dir2/file
[00074] EXTENDED
  [00074]         STATUS       28bytes (0ms)
   unique: 518, error: -1 (Operation not permitted), outsize: 16
$ strace mv file ../../fs2/dir2/ 2>&1 | egrep '^(rename|open|ioctl|read|write)\([^0-2]' | vipe
rename("file", "../../fs2/dir2/file") = -1 EPERM (Operation not permitted)

Relevant output of the sftp-server DEBUG3 log in /var/log/auth.log:

Oct  4 11:47:03 localhost sftp-server[27991]: debug3: request 74: posix-rename
Oct  4 11:47:03 localhost sftp-server[27991]: posix-rename old "/mnt/fs1/dir1/file" new "/mnt/fs2/dir2/file"
Oct  4 11:47:03 localhost sftp-server[27991]: debug3: request 74: sent status 4
Oct  4 11:47:03 localhost sftp-server[27991]: sent status Failure

Moving on the remote works as expected:

$ strace mv file ../../fs2/dir2/ 2>&1 | egrep '^(rename|open|ioctl|read|write)\([^0-2]' | vipe
rename("file", "../../fs2/dir2/file") = -1 EXDEV (Invalid cross-device link)
open("file", O_RDONLY|O_LARGEFILE|O_) = 3
open("../../fs2/dir2/file", O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 4
ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = -1 EXDEV (Invalid cross-device link)
read(3, "\nThe programs included with the "..., 131072) = 286
write(4, "\nThe programs included with the "..., 286) = 286
read(3, "", 131072)                     = 0

Here the error code returned by rename(3) is EXDEV which makes mv
attempt to actually move the data. I wonder why POSIX does not allow
mv(1posix) to attempt a move after other kinds of failure. What other
similar command is able to force a move of data in this case (while
preserving times and attributes) that is simpler to type than:
bash -c 'cp -a "$@" && rm "${@:1:$#-1}"' arg0

I suppose that sshfs returns EPERM instead of EXDEV because sftp does
not return a significant error code. Would this be by design of sftp or
a shortcoming of OpenSSH's sftp-server?

As a fix maybe sshfs could always return EXDEV when rename fails and it
does not know precisely which error was triggered. Or does sftp by
chance make it possible to detect mount points?

Regards

[g-raud]

Here's a workaround implemented as an LD_PRELOAD library.

/* preload_rename.c: LD_PRELOAD library to make rename() fail only with EXDEV
 * make: gcc -fPIC -shared -opreload_rename.so preload_rename.c -ldl
 * run: LD_PRELOAD=$(pwd)/preload_rename.so command
 */

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <errno.h>

/* Function pointer to the value of the glibc function */
int (*real_rename)(const char *oldpath, const char *newpath) = NULL;

/* Wrapping function */
int rename(const char *oldpath, const char *newpath)
{
	int rc;
	/* resolve and call the real function from glibc */
	real_rename = dlsym(RTLD_NEXT, "rename");
	rc = real_rename(oldpath, newpath);
	/* set errno to EXDEV to incite file managers to try to actually move the
	 * file */
	if (rc < 0) errno = EXDEV;
	return(rc);
}

[Nikratio]

Thanks for the report! You are right, the EXDEV error should come from the OpenSSH server and then be forwarded all the way to the mv command. Not sure why it isn't working. First steps to debug this would be to figure out what OpenSSH actually returns.

[g-raud]

Isn't the return status from OpenSSH given in its log: "status 4" "sent status Failure"? Or do you think that another layer mingles the data?

[Nikratio]

I don't know. You could start from the assumption that OpenSSH does return the right value, and then try to determine why SSHFS doesn't forward that to the kernel.