Cache the result of getFlake

[thufschmitt]

Add an evaluation caching mechanism

Add the caching mechanism already present for cli arguments to the evaluation loop.

This means that an expression of the form (builtins.getFlake foo).bar.baz will be cached if it can be (and also slightly more complex stuff like let pkgs = (builtins.getFlake foo).packages.x86_64; in pkgs.bar.

This isn't as useful as it could be because flake inputs aren't cached (yet). But it opens the way for it.

[thufschmitt]

I've update this to cache a few more things (in particular the stringification of derivations, which is probably the most important thing to cache).

I've also run some quick benchmarks, and the results are not really nice as the caching adds a noticeable overhead to the evaluation (but it's very likely that it can be optimized. I'll look into that in the coming days). The results are below.

The command I ran for the benchmarks is

nix build --experimental-features "nix-command flakes" --expr "$EXPR"

where $EXPR is

let
  pkgs = (builtins.getFlake "github:NixOS/nixpkgs?rev=ad0d20345219790533ebe06571f82ed6b034db31").legacyPackages.x86_64-linux;
in
  pkgs.runCommandNoCC "foo" {
    buildInputs = [
      pkgs.firefox
      pkgs.pandoc
    ];
  }
  "echo bar > $out"

Command	Mean [s]	Min [s]	Max [s]	Relative
this branch, cache disabled	2.982 ± 0.012	2.962	3.002	2.20 ± 0.04
master	2.152 ± 0.025	2.129	2.211	1.59 ± 0.03
this branch, cache hot	1.353 ± 0.022	1.339	1.414	1.00

[thufschmitt]

After a lot of struggle and rewriting the cache partially from scratch, I eventually managed to get some much nicer benchmark results: On the evaluation part the overhead is down to between 0 and 10% (generally around 5%) when the cache is disabled or cold. On the exact same benchmark, I get:

Command	Mean [s]	Min [s]	Max [s]	Relative
cached	1.380 ± 0.350	1.257	2.377	0.63
coldCache	2.366 ± 0.012	2.351	2.397	1.08
noCache	2.279 ± 0.006	2.269	2.289	1.04
systemNix	2.183 ± 0.014	2.163	2.210	1

(with the caveat that the relative speed of the cached run is a bit arbitrary as it depends more on how much of the input expression can be cached than on its actual complexity. So it's possible to build benchmarks where it will be arbitrarily faster than an uncached run, or conversely where they will both perform as fast. I chose this one as a reasonable scenario where some expensive stuff is cached (firefox and pandoc), while some other isn't (runCommandNoCC).

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-7/11552/1

[Mic92]

Do you see a way how we could partition nixpkgs evaluation in nixpkgs-review using the cache?
Currently doing something like nix-env --meta --json -f . -qa '*' will consume something like 5GB of RAM. It is possible to evaluate smaller sets but than the whole execution time takes longer before depending derivation need to be re-evaluated.

[thufschmitt]

Do you see a way how we could partition nixpkgs evaluation in nixpkgs-review using the cache?
Currently doing something like nix-env --meta --json -f . -qa '*' will consume something like 5GB of RAM. It is possible to evaluate smaller sets but than the whole execution time takes longer before depending derivation need to be re-evaluated.

Good question… This PR alone wouldn't do it, but I guess this + a --json flag to nix flake show to make it display more information (and in an easier to parse format) could do the trick.

[Mic92]

Do you see a way how we could partition nixpkgs evaluation in nixpkgs-review using the cache?
Currently doing something like nix-env --meta --json -f . -qa '*' will consume something like 5GB of RAM. It is possible to evaluate smaller sets but than the whole execution time takes longer before depending derivation need to be re-evaluated.

Good question… This PR alone wouldn't do it, but I guess this + a --json flag to nix flake show to make it display more information (and in an easier to parse format) could do the trick.

Ok. All we would need for nixpkgs-review is the store path and if possible filter output to only include architecture. However from what I currently see nix flake show --legacy also consumes tons of memory - so I don't see how the evaluation cache can help me here.

[stale]

I marked this as stale due to inactivity. → More info