No way to access snapshots within the .zfs/snapshot virtual folder from Samba share mounted via MacOS 14.6.1

[kimono-koans]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	24.04
Kernel Version	6.8.0-39-generic
Architecture	amd64
OpenZFS Version	zfs-2.2.2-0ubuntu9

Describe the problem you're observing

No way to access snapshots within the .zfs/snapshot virtual folder from Samba share mounted via MacOS 14.6.1

Describe how to reproduce the problem

Mount Samba share and try to browse mount at .zfs/snapshot. This used to work without setting snapdir=visible before an upgrade to MacOS 14.6 and Ubuntu 24.04.

Include any warning/errors/backtraces from the system logs

Attempts to access from within the .zfs/snaphot directory , even with sudo, gives errors like:

% ls -al
ls: snap_2024-07-24-16÷02÷10_HomeSync: No such file or directory
ls: snap_2024-07-25-03÷05÷53_HomeSync: No such file or directory
ls: snap_2024-07-25-16÷00÷55_HomeSync: No such file or directory
ls: snap_2024-07-25-16÷57÷58_HomeSync: No such file or directory
ls: snap_2024-07-26-16÷06÷29_HomeSync: No such file or directory
ls: snap_2024-07-27-16÷49÷33_HomeSync: No such file or directory
ls: snap_2024-07-28-15÷53÷12_HomeSync: No such file or directory
ls: snap_2024-07-28-16÷00÷15_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷05÷12_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷05÷52_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷12÷29_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷14÷47_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷16÷31_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷19÷44_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷19÷55_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷20÷42_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷20÷57_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷27÷19_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷35÷18_HomeSync: No such file or directory
ls: snap_2024-07-29-16÷00÷07_HomeSync: No such file or directory
ls: snap_2024-07-30-13÷19÷01_HomeSync: No such file or directory
ls: snap_2024-07-30-17÷15÷54_HomeSync: No such file or directory
ls: snap_2024-07-30-22÷43÷18_HomeSync: No such file or directory
ls: snap_2024-07-31-16÷01÷02_HomeSync: No such file or directory
ls: snap_2024-08-01-16÷11÷34_HomeSync: No such file or directory
total 64
drwxrwxrwx  1 kimono  staff  16384 Aug  6 23:45 .
drwxrwxrwx  1 kimono  staff  16384 Aug  3 00:44 ..
ls: fts_read: No such file or directory

smb.conf contains:

zfsacl:expose_snapdir = True
veto files = /.windows/.mac/

When dataset is zfs set snapdir=visible ..., then most directories are visible but contents are not:

ls: snap_2024-07-29-14÷19÷55_HomeSync: No such file or directory
ls: snap_2024-07-30-17÷15÷54_HomeSync: No such file or directory
total 800
drwxrwxrwx  1 kimono  staff  16384 Aug  6 23:45 .
drwxrwxrwx  1 kimono  staff  16384 Aug  3 00:44 ..
drwxrwxrwx  1 kimono  staff  16384 Jul 24 16:02 snap_2024-07-24-16÷02÷10_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 25 03:05 snap_2024-07-25-03÷05÷53_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 25 16:00 snap_2024-07-25-16÷00÷55_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 25 16:57 snap_2024-07-25-16÷57÷58_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 26 16:06 snap_2024-07-26-16÷06÷29_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 27 16:49 snap_2024-07-27-16÷49÷33_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 28 15:53 snap_2024-07-28-15÷53÷12_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 28 16:00 snap_2024-07-28-16÷00÷15_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:05 snap_2024-07-29-14÷05÷12_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:05 snap_2024-07-29-14÷05÷52_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:12 snap_2024-07-29-14÷12÷29_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:14 snap_2024-07-29-14÷14÷47_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:16 snap_2024-07-29-14÷16÷31_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:19 snap_2024-07-29-14÷19÷44_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:20 snap_2024-07-29-14÷20÷42_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:20 snap_2024-07-29-14÷20÷57_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:27 snap_2024-07-29-14÷27÷19_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 14:35 snap_2024-07-29-14÷35÷18_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 29 16:00 snap_2024-07-29-16÷00÷07_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 30 13:19 snap_2024-07-30-13÷19÷01_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 30 22:43 snap_2024-07-30-22÷43÷18_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Jul 31 16:01 snap_2024-07-31-16÷01÷02_HomeSync
drwxrwxrwx  1 kimono  staff  16384 Aug  1 16:11 snap_2024-08-01-16÷11÷34_HomeSync

It seems as though ZFS no longer auto-mounts snapshots when a request is made from a SMB client.

However, if I stir the snapshots on the host, and let the auto-mounter work, the guest system now shows the contents of re: ls -alR but not re: a normal statx!

[robn]

I'll start by saying that OpenZFS 2.2.2 never had support for kernel 6.8; Ubuntu backported experimental patches that we hadn't completed, and I don't know what they've done since. This is not to blame them, but rather to say, this isn't a combination we technically support, and it might just be busted. But let's assume the best for the moment.

You say it worked before, so what actually changed? You mention client and OS; I suppose the latter means you got new kernel, OpenZFS and Samba? What versions of each did you have before?

It's not clear to me if you're saying that it's working if you do it directly on the host? I don't know what you're doing in the last paragraph ("stir the snapshots").

If Samba is doing something different than it used to, then that might be it. If it's doing the same thing then yes, perhaps a response has changed. You mention statx, how exactly has its response changed?

It might be a real bug since fixed, but I didn't quickly find a PR that sounded like it. Could easily mean I missed something.

Basically, divide and conquer. See if you can narrow it down!

[kimono-koans]

I'll start by saying that OpenZFS 2.2.2 never had support for kernel 6.8; Ubuntu backported experimental patches that we hadn't completed, and I don't know what they've done since.

Understood. Hope no offense is taken. I see a thumbs down emoji! It seemed like a possible bug in the auto-mounter, like perhaps it should mount on a larger variety of system calls.

You say it worked before, so what actually changed? You mention client and OS; I suppose the latter means you got new kernel, OpenZFS and Samba? What versions of each did you have before?

I upgraded from Ubuntu 22.04 to 24.04 while using MacOS 13, except it required the use of the zfsacl:expose_snapdir = True setting in the smb.conf which it didn't previously.

This config seemed to work fine until an update to MacOS 14+.

It's not clear to me if you're saying that it's working if you do it directly on the host? I don't know what you're doing in the last paragraph ("stir the snapshots").

On the host, I get the auto-mouter to mount the snapshots. Like -- if I do a read_dir on the .zfs/snapshot folder to give me all the snapshot directories, and statx on a file (in this case .zshrc) in each snapshot directory, by joining the snapshot directories with the relative path, via the httm command.

This causes the auto-mounter to mount all snapshot directories.

If Samba is doing something different than it used to, then that might be it. If it's doing the same thing then yes, perhaps a response has changed. You mention statx, how exactly has its response changed?

Yes, it could be. Samba version is: Version 4.19.5-Ubuntu.

Basically I'd like to be able to do what I did previously: On the client, I'd like to do a read_dir on the .zfs/snapshot folder on the client, and statx on a file (in this case .zshrc) in each snapshot directory, via httm, and have these requests auto-mount the snapshot directories.

It might be a real bug since fixed, but I didn't quickly find a PR that sounded like it. Could easily mean I missed something.

I searched the issues for something similar but didn't discover anything.

[kimono-koans]

Okay I have a workaround: If I do an opendir and read_dir on the snapshot directory of the file to be statx-ed and iterate over at least one entry, before I fstat that file, I can now fstat each file. Of course that's kinda bizarre, and very slow, but it works.

Also -- I think smbd is using fstat instead of statx. And I think that may be the difference? Could it be the auto-mounter is not mounting on fstat calls? That kind of makes sense if perhaps the auto-mounter is looking for paths not fds.

In any event, this perfectly describes my problem and the ask. I'd like snapshots to auto-mount when any stat type call is made instead of having to wait to complete the opendir into read_dir iterator. This was once the behavior of Samba + ZFS but it is apparently not any longer. It makes sense to do this because it is much faster than waiting 10-100x the time for the other system calls to complete. It's also more correct. When I stat a file in a snapshot directory, it should show up. We shouldn't have to enter that directory first.

[kimono-koans]

FYI to confirm the source of this behavior seems to be Samba on the Linux server. See:

Starting with version 4.14 Samba provides core infrastructure code that allows basing all access to the server's filesystem on file handles and not on paths. An example of this is using fstat() instead of stat(), or SMB_VFS_FSTAT() instead of SMB_VFS_STAT() in Samba parlance.

From: https://wiki.samba.org/index.php/The_New_VFS

@robn would you advise?

[robn]

This is good info, thank you. Unfortunately I don't know anything much about Samba; my questions were mostly for triage.

The next thing I suppose is to see if you can reproduce it on a stock OpenZFS., to rule in/out Ubuntu's version. You could possibly even just ask Ubuntu, since they should be supporting their whole stack.

Alternately, a simple reproduction script or program that doesn't require installing Samba or Ubuntu. We'll need to be able to reproduce it to fix it if there is a bug anyway, and if it doesn't reproduce it on stock OpenZFS, then you'll have narrowed it down.

[kimono-koans]

Minimal reproducer shows my hypothesis might not be the cause:

use std::fs::OpenOptions;
use std::os::fd::AsRawFd;
use std::path::PathBuf;
use std::thread::sleep;
use std::time::Duration;

pub type GenericResult<T> = Result<T, Box<dyn std::error::Error + Send + Sync>>;

fn main() -> GenericResult<()> {
    let path = PathBuf::from("/.zfs/snapshot/autosnap_2024-06-01_00:00:29_monthly/etc/passwd");

    let path_md = nix::sys::stat::stat(&path).ok();

    eprintln!("{:?}", path_md);

    let file = OpenOptions::new()
        .read(true)
        .open("/home/kimono/.zfs/snapshot/autosnap_2024-08-11_10:05:28_hourly/.zshrc")?;

    sleep(Duration::from_secs(30));

    let fd = file.as_raw_fd();

    let file_md = nix::sys::stat::fstat(fd).ok();

    eprintln!("{:?}", file_md);

    Ok(())
}