Allow specifying alternative paths for reading/writing flake locks #8042

lheckemann · 2023-03-13T20:13:32Z

Motivation

This allows having multiple separate lockfiles for a single project, which can be useful for testing against different versions of nixpkgs; it also allows tracking custom input overrides for remote flakes without requiring local clones of these flakes.

For example, if I want to build Nix against my locally pinned nixpkgs, and have a lock file tracking this override independently of future updates to said nixpkgs:

nix flake lock --output-lock-file /tmp/nix-flake.lock --override-input nixpkgs flake:nixpkgs
nix build --reference-lock-file /tmp/nix-flake.lock

Context

Hydra's current flake support is mediocre at best, and we think supporting alternative lock files would be extremely helpful to allow Hydra (and other CI systems!) to build jobsets against alternative input sets, update inputs (individually or all at once), etc.

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests - tests/nixos/*
documentation in the manual
code and comments are self-explanatory
commit message explains why the change was made
new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

This allows having multiple separate lockfiles for a single project, which can be useful for testing against different versions of nixpkgs; it also allows tracking custom input overrides for remote flakes without requiring local clones of these flakes. For example, if I want to build Nix against my locally pinned nixpkgs, and have a lock file tracking this override independently of future updates to said nixpkgs: nix flake lock --output-lock-file /tmp/nix-flake.lock --override-input nixpkgs flake:nixpkgs nix build --reference-lock-file /tmp/nix-flake.lock Co-Authored-By: Will Fancher <[email protected]>

knedlsepp · 2023-03-14T06:38:53Z

Awesome! Also fixes: #5673

src/libexpr/flake/flake.hh

src/libcmd/installables.cc

edolstra · 2023-03-14T10:49:09Z

src/libcmd/installables.cc

@@ -102,6 +102,28 @@ MixFlakeOptions::MixFlakeOptions()
 }}
 });

+ addFlag({
+ .longName = "reference-lock-file",


I'm not sure if there is a real use case for a separate --reference-lock-file and --output-lock-file. With a unified --lock-file, the caller can always write the input lock file to that location to be overwritten.

I like to keep the lock files of my projects in sync with that of my main deployment. With the separated options, I can use nix flake lock --reference-lock-file ~/deploy/flake.lock to keep them in sync (provided my deployment has a superset of the inputs of the project I'm updating).

It's also useful for updating individual inputs (but leaving everything else unmodified) without having a checkout of the whole repository, e.g. nix flake lock github:NixOS/patchelf --output-lock-file flake.lock --update-input nixpkgs or nix flake lock github:NixOS/patchelf --output-lock-file flake.lock --override-input nixpkgs github:nixos/nixpkgs/nixos-22.11. This is what I had in mind for Hydra, since it invokes multiple commands on the flake for a single evaluation and should use the same set of inputs each time.

without having a checkout of the whole repository

Just for clarity: you mean checkout out the repository into a temporary directory because it will still land "checked out" in the nix store.

Not necessarily in the future (#6530).

Co-authored-by: Eelco Dolstra <[email protected]>

knedlsepp · 2023-03-14T13:52:18Z

src/libcmd/installables.cc

@@ -102,6 +102,28 @@ MixFlakeOptions::MixFlakeOptions()
 }}
 });

+ addFlag({
+ .longName = "reference-lock-file",


If we go for --output-lock-file, why not call it --input-lock-file?

"reference" seems more appropriate here, since the lock file may be completely foreign (e.g. I'm working on a project that uses nixos-22.11 and want it to use my deployment's lock file, so I run nix flake lock --reference-lock-file ~/deploy/flake.lock). This is very much my personal opinion though and if there's broad agreement that it should be called "input" I'd be willing to change it.

edolstra · 2023-03-15T09:31:53Z

Can you add some tests? In particular, we need to test that --reference-lock-file requires --impure (since specifying a different lock file essentially modifies the flake).

ElvishJerricco · 2023-03-15T14:21:14Z

@edolstra I wouldn't want --impure to be required. That enables a lot more impurity than I would be looking for if I just want to provide a different lock file. e.g. I still don't want builtins.currentTime to exist just because I want to change the lock file.

lheckemann · 2023-03-15T14:23:34Z

Yes, I'll have a look at adding some tests.

I don't think --reference-lock-file should require --impure though. It's not an ambient impurity (unlike the Flake registry for example, which doesn't even require --impure!), but one explicitly specified at invocation time and which can't be set globally (which is why I prefer this command-line arg approach over an environment variable as previously implemented by @colemickens).

Requiring disabling all purity would also be terrible for e.g. the Hydra use case, allowing environment variables or nixpkgs config to leak into eval (because restrict-eval has to be disabled for any non-github flake inputs to be supported, see #7098).

ElvishJerricco · 2023-03-15T14:28:47Z

I think command line flags in general should be treated more like function arguments rather than impurities. They're explicit and require you to deliberately include them, as opposed to the implicit impurities like environment variables or out-of-tree files. So specifying a different lock file seems a lot more like a function argument than an impurity to me. It's effectively equivalent to making an ad-hoc parent flake and using the follows mechanism, which isn't considered impure.

edolstra · 2023-03-15T14:47:33Z

@lheckemann Yeah I said that wrong. I didn't mean impure but that allow-dirty should be enabled. I.e. the top-level input must be considered dirty, in the same way as when you do nix build and the lock file gets updated without using --commit-lock-file.

SuperSandro2000 · 2023-03-19T15:39:19Z

src/libexpr/flake/flake.cc

+ if (lockFlags.outputLockFilePath) {
+ throw Error("--commit-lock-file and --output-lock-file are currently incompatible");
+ }


If output-lock-file ends up under the git repository it could be commited like flake.lock. Any none absolute path that flattened does not start with .. should be commitable.

This does not apply if symlinks are in use, and generating accurate commit messages would require reading the original lock file (if one already exists at the given path) as well as the reference lock file in order to generate an accurate commit message. Most of the value from this PR comes with the basic functionality and I'd welcome if someone else were to implement committing support in the future, but do not want to implement it myself at this time.

lheckemann requested a review from edolstra as a code owner March 13, 2023 20:13

lheckemann mentioned this pull request Mar 14, 2023

Allow overriding flake.lock file location via command-line #5673

Closed

edolstra reviewed Mar 14, 2023

View reviewed changes

Apply review suggestions

3a1de4c

Co-authored-by: Eelco Dolstra <[email protected]>

knedlsepp reviewed Mar 14, 2023

View reviewed changes

lheckemann added 2 commits March 19, 2023 14:11

Add tests for alternate lockfile path functionality

ea207a2

Only allow reference lock files when allow-dirty is set

f1c9d83

github-actions bot added the with-tests Issues related to testing. PRs with tests have some priority label Mar 19, 2023

Create test lockfiles in TEST_ROOT

3c3bd07

SuperSandro2000 reviewed Mar 19, 2023

View reviewed changes

edolstra merged commit 81491e1 into NixOS:master Apr 3, 2023

thufschmitt mentioned this pull request Apr 4, 2023

Build failing on arrch64-linux #8159

Closed

lheckemann deleted the alt-lockfiles branch April 4, 2023 11:25

colemickens mentioned this pull request May 11, 2023

flakes: support the new nix feature --reference-lock-file nix-community/nix-eval-jobs#209

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow specifying alternative paths for reading/writing flake locks #8042

Allow specifying alternative paths for reading/writing flake locks #8042

lheckemann commented Mar 13, 2023 •

edited

Loading

knedlsepp commented Mar 14, 2023

edolstra Mar 14, 2023

lheckemann Mar 14, 2023 •

edited

Loading

SuperSandro2000 Mar 19, 2023

lheckemann Mar 25, 2023

knedlsepp Mar 14, 2023

lheckemann Mar 25, 2023

edolstra commented Mar 15, 2023

ElvishJerricco commented Mar 15, 2023

lheckemann commented Mar 15, 2023 •

edited

Loading

ElvishJerricco commented Mar 15, 2023

edolstra commented Mar 15, 2023

SuperSandro2000 Mar 19, 2023

lheckemann Mar 19, 2023

Allow specifying alternative paths for reading/writing flake locks #8042

Allow specifying alternative paths for reading/writing flake locks #8042

Conversation

lheckemann commented Mar 13, 2023 • edited Loading

Motivation

Context

Checklist for maintainers

Priorities

knedlsepp commented Mar 14, 2023

edolstra Mar 14, 2023

Choose a reason for hiding this comment

lheckemann Mar 14, 2023 • edited Loading

Choose a reason for hiding this comment

SuperSandro2000 Mar 19, 2023

Choose a reason for hiding this comment

lheckemann Mar 25, 2023

Choose a reason for hiding this comment

knedlsepp Mar 14, 2023

Choose a reason for hiding this comment

lheckemann Mar 25, 2023

Choose a reason for hiding this comment

edolstra commented Mar 15, 2023

ElvishJerricco commented Mar 15, 2023

lheckemann commented Mar 15, 2023 • edited Loading

ElvishJerricco commented Mar 15, 2023

edolstra commented Mar 15, 2023

SuperSandro2000 Mar 19, 2023

Choose a reason for hiding this comment

lheckemann Mar 19, 2023

Choose a reason for hiding this comment

lheckemann commented Mar 13, 2023 •

edited

Loading

lheckemann Mar 14, 2023 •

edited

Loading

lheckemann commented Mar 15, 2023 •

edited

Loading