Flake paths should use virtual store directories

[PigeonF]

When using a flake with --store, builtins.storeDir contains the wrong path. I ran into this while trying to use lib.fileset.gitTracked in combination with --store. See NixOS/nixpkgs#283843 for the original issue I filed.

@infinisil suggests this should be fixed on the flake side, rather than by adjusting lib.fileset.gitTracked. I don't know how this would play with the other types of stores, since I have only tried the chroot one.

Priorities

Add 👍 to issues you find important.

[Ericson2314]

 $ nix --extra-experimental-features nix-command eval --store 'dummy:https://?store=/asdf' --expr
 'builtins.storeDir'
"/asdf"

seems to work for me.

builtins.storeDir isn't supposed to respond to changes in the physical store dir, only the virtual store dir --- i.e. chroot stores which mount the store as /nix/store will make builtins.storeDir still return /nix/store.

 $ nix --extra-experimental-features nix-command eval --store '~/foo' --expr 'builtins.storeDir' --read-only
"/nix/store"

[Ericson2314]

I can reproduce what @infinisil said, but the problem is not with builtins.storeDir.

[infinisil]

Yeah it should be fixed the other way around, paths in Flakes should use the virtual store dir:

{
  outputs = { ... }: {
    path = ./.;
    storeDir = builtins.storeDir;
  };
}

$ nix eval .#storeDir
"/nix/store"
$ nix eval .#path
/nix/store/f5dc7kgs97658c024mlnx0wn9j95ns91-source
$ nix eval .#storeDir --store /tmp/nix-store/
"/nix/store"
$ nix eval .#path --store /tmp/nix-store/
/tmp/nix-store/nix/store/f5dc7kgs97658c024mlnx0wn9j95ns91-source

I expect the last value to also be /nix/store/f5dc7kgs97658c024mlnx0wn9j95ns91-source

[Ericson2314]

{
  outputs = { ... }: {
    path = builtins.trace ./. "";
    storeDir = builtins.storeDir;
  };
}

 $ nix --extra-experimental-features 'nix-command flakes' --store "$PWD/store" eval ./foo#path --show-trace
error:
       … while calling the 'throw' builtin

         at /tmp/tmp.fAS3q6Eefn/store/nix/store/223nxwxxvicnbgpm24a0xp5wgl9bd9is-source/flake.nix:3:12:

            2|   outputs = _: {
            3|     path = builtins.throw ./foo.;
             |            ^
            4|     storeDir = builtins.storeDir;

       error: getting status of '/tmp/tmp.fAS3q6Eefn/store/nix/store/223nxwxxvicnbgpm24a0xp5wgl9bd9is-source/foo.': No such file or directory

it's not specific to outputs.

[Ericson2314]

this is the one Store::toRealPath call I got with

gdb --args nix --extra-experimental-features 'nix-command flakes' --store "$PWD/store" e
val ./foo#path --show-trace

(gdb) break Store::toRealPath
Breakpoint 1 at 0x7ffff79416d0 (4 locations)
(gdb) r
Starting program: /nix/store/yv1r6brw5mczs0ab1fxvx4b0i31w8hy1-system-path/bin/nix --extra-experimental-features nix-command\ flakes --store /tmp/tmp.fAS3q6Eefn/store eval ./foo\#path --show-trace
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libthread_db.so.1".
[New Thread 0x7ffff5ae46c0 (LWP 2905559)]
[New Thread 0x7fffdd1636c0 (LWP 2905560)]

Thread 1 "nix" hit Breakpoint 1.3, 0x00007ffff7b66110 in nix::Store::toRealPath[abi:cxx11](nix::StorePath const&) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixfetchers.so
(gdb) bt
#0  0x00007ffff7b66110 in nix::Store::toRealPath[abi:cxx11](nix::StorePath const&) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixfetchers.so
#1  0x00007ffff7b6b660 in nix::fetchers::Input::fetch(nix::ref<nix::Store>) const ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixfetchers.so
#2  0x00007ffff7dff1f2 in nix::FlakeRef::fetchTree(nix::ref<nix::Store>) const ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixexpr.so
#3  0x00007ffff7dee736 in nix::flake::fetchOrSubstituteTree(nix::EvalState&, nix::FlakeRef const&, bool, std::vector<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> >, std::allocator<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> > > >&) [clone .lto_priv.0] ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixexpr.so
#4  0x00007ffff7df4590 in nix::flake::getFlake(nix::EvalState&, nix::FlakeRef const&, bool, std::vector<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> >, std::allocator<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) () from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixexpr.so
#5  0x00007ffff7df567a in nix::flake::getFlake(nix::EvalState&, nix::FlakeRef const&, bool, std::vector<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> >, std::allocator<std::pair<nix::FlakeRef, std::pair<nix::fetchers::Tree, nix::FlakeRef> > > >&)
    () from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixexpr.so
#6  0x00007ffff7df5ab5 in nix::flake::lockFlake(nix::EvalState&, nix::FlakeRef const&, nix::flake::LockFlags const&) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixexpr.so
#7  0x00007ffff775f5aa in nix::InstallableFlake::getLockedFlake() const ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixcmd.so
#8  0x00007ffff77609a4 in nix::InstallableFlake::getCursors(nix::EvalState&) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixcmd.so
#9  0x00007ffff775cefb in nix::InstallableValue::getCursor(nix::EvalState&) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixcmd.so
#10 0x00007ffff775c7d8 in nix::InstallableFlake::toValue(nix::EvalState&) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixcmd.so
#11 0x0000555555664bd6 in CmdEval::run(nix::ref<nix::Store>, nix::ref<nix::InstallableValue>) ()
#12 0x00007ffff774fb34 in nix::InstallableValueCommand::run(nix::ref<nix::Store>, nix::ref<nix::Installable>) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixcmd.so
#13 0x00007ffff776e28a in nix::InstallableCommand::run(nix::ref<nix::Store>) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixcmd.so
#14 0x00007ffff774e3b7 in nix::StoreCommand::run() ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixcmd.so
#15 0x000055555568633a in nix::mainWrapped(int, char**) ()
#16 0x00007ffff7bf071e in nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) ()
   from /nix/store/xwshrzv0dccs2sq5wan2q8zq78jbfix1-nix-2.18.1/lib/libnixmain.so
#17 0x00005555555e19b7 in main ()

[Ericson2314]

That function call is gone on master, but the bug still persists :(

 $ outputs/out/bin/nix --store ./store --extra-experimental-features 'nix-command fl
akes' eval "path:$PWD/foo#path"
/home/jcericson/src/nix-master/store/nix/store/4afiiial3gnghb3bgz03zmm1i62vzp1f-source

[Ericson2314]

Oh nevermind, it's all the toRealPath in getFlake (which also seems to be struggling to prevent symlink issues).

The solution is probably to avoid calling state.rootPath, and instead call state.store->getFSAccessor and build a SourcePath from that.

[Ericson2314]

Now that #10088 is merged, the fix is slightly different but morally still the same.

[fricklerhandwerk]

Potential fix: #10345

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-options-of-type-pathinstore-break-when-running-nixos-install/47429/2

[WizardUli]

Potential fix: #10345

I'm not sure whether it has really been fixed of whether the problem is with actually with https://github.com/NixOS/nixpkgs/blob/85a13515a5fd98ca1a4aeefcaedcb6417f841e13/lib/types.nix#L538 in my case:

https://discourse.nixos.org/t/nixos-options-of-type-pathinstore-break-when-running-nixos-install/47429/4