libstore: generalize signature scheme and introduce remote signing #9076

RaitoBezarius · 2023-09-30T21:25:45Z

Motivation

We would like to hide the signing key on another machine to onboard more folks to help on infrastructure of NixOS.

Context

As moving to HSM/PKCS#11 is out of scope in the current timeline we are operating in, we are proposing a simpler solution, albeit very flexible one, which is remote signing technology for Nix.

For this, we need to generalize the signature scheme that assumes we always possess the private key material.

Now, signature schemes use an abstract Signer object that can be instantiated anywhere
and passed around in the hierarchy for signing relevant objects.

The old local signing scheme is now implemented as the LocalSigner.

The abstract Signer always require a local PublicKey to make verification cheap.

In addition, we introduce another signer: the RemoteSigner, adhering to a "Nix remote signing API",
which is a need for the operations of the NixOS.org infrastructure community, to move the signing key location
in a separate piece of infrastructure with increased scrutiny and security.

The Nix remote signing API has 2 endpoints:

/sign : takes as input a string and reply with its ED25519-SHA512 signature
/publickey: reply the public key material

There is no authentication neither authorization, except for maybe a special string (not yet implemented) that enables to ensure we do not let, by mistake, an open signing server which could sign any path and create a bigger disaster.
Such things are left to firewalling and reverse proxies measures.

Impact evaluation on signature speed, management of timeouts (exponential backoff), error handling are all out of scope for now, as we are still working on this with @cole-h and potentially more folks who may join us.

Perl bindings will not contain a remote signing version as it's hard to abstract properly the store()->signDetached(...), for whatever is worth, a store could not have a signer, and we could not sign things with it.
The only way is to peek and downcast to a store known to have signers. Such a change would be invasive.

TODO list

Rework getPublicKey idea for remote signer to avoid paying a penalty all the time for nothing
Validate the remote-signing URL format for easier buildup of API endpoints
(Integration?) tests
Binary cache store implementation?
Communication key (the "secret special string", which is by no means a serious protection, but defense in depth!)
Fix Perl bindings (for Hydra)

Priorities

Add 👍 to pull requests you find important.

src/libstore/local-store.cc

src/libstore/signature/local-keys.cc

src/libstore/signature/local-keys.hh

src/libstore/globals.hh

src/libstore/signature/signer.cc

lheckemann

Love to see things happening here!

I wonder if this is fundamentally the right approach, as opposed to sticking to local signatures (with a per-machine key, and the signing done by another machine based on its trust of those per-machine keys). Such an approach would provide somewhat better traceability.

RaitoBezarius · 2023-10-02T11:11:53Z

I wonder if this is fundamentally the right approach, as opposed to sticking to local signatures (with a per-machine key, and the signing done by another machine based on its trust of those per-machine keys). Such an approach would provide somewhat better traceability.

Wouldn't this require copying the paths to the signing box, then getting those paths back, incurring a potential high network traffic?

lheckemann · 2023-10-02T14:02:50Z

No, it could take the fingerprint as signed by the builder and re-sign that.

RaitoBezarius · 2023-10-02T14:20:15Z

No, it could take the fingerprint as signed by the builder and re-sign that.

Wouldn't this imply making Nix a signing server itself?

lheckemann · 2023-10-03T07:53:08Z

This functionality wouldn't have to be part of Nix. It could just as well be implemented in a cache server like attic, or as a service that uses Nix to copy and re-sign narinfos from an intermediate flat-file binary cache to the "final" one.

EDIT: and Attic is already 90% of the way there: it stores signatures from the upload, but doesn't currently validate them or deliver them to clients, instead generating a signature using its own per-cache signing key when the narinfo is requested.

RaitoBezarius · 2023-10-03T11:44:13Z

This functionality wouldn't have to be part of Nix. It could just as well be implemented in a cache server like attic, or as a service that uses Nix to copy and re-sign narinfos from an intermediate flat-file binary cache to the "final" one.

EDIT: and Attic is already 90% of the way there: it stores signatures from the upload, but doesn't currently validate them or deliver them to clients, instead generating a signature using its own per-cache signing key when the narinfo is requested.

Gotcha, but, in practice, for our needs right now, what is the plan you propose with this idea?

I guess this is pushing the problem to an extra piece of software we have to run on the NixOS org machines, right?

Ericson2314 · 2023-10-03T14:19:03Z

I think this machinery should go in libutil because it is just about signing arbitrary data, and not anything store-specific?

RaitoBezarius · 2023-10-03T16:08:30Z

I think this machinery should go in libutil because it is just about signing arbitrary data, and not anything store-specific?

I agree with this.

zimbatm · 2023-10-21T06:02:49Z

So move this in libutil and then this can be merged?

Ericson2314 · 2023-10-21T14:12:48Z

Yeah if it is moved I support this. Maybe some more rounds of little review after but only little things.

nixos-discourse · 2023-11-20T14:47:36Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-11-17-nix-team-meeting-minutes-104/35753/1

Ericson2314 · 2023-11-20T23:07:09Z

Idea sounds good to people in the nix team meeting. Just need to fix the more minor things addressed in the thread.

doc/manual/src/protocols/signature.md

src/libutil/hash.hh

src/libutil/signature/local-keys.cc

src/libstore/keys.hh

src/libutil/signature/signer.cc

Ericson2314 · 2024-01-03T15:41:05Z

src/libutil/util.cc

+const std::string nixVersion = PACKAGE_VERSION;
+


Is this used?

Well, yes, the symbol needs to be defined in that library? It's used for set_user_agent.

src/libutil/util.cc

Ericson2314 · 2024-01-03T15:41:53Z

src/libutil/util.hh

+struct Hash;
+
 void initLibUtil();

+/**
+ * This Nix version.
+ */
+extern const std::string nixVersion;
+
+


no changes to this file are needed

Why? How do I access nixVersion ?

As it makes more sense to make it available to more consumers. In this commit, we create a new util: `randomSha1` because `libstore` depended on sodium to do that in the past. We moved the sodium initialization and dependency to `libutil`. Moreover, we keep `getDefaultPublicKeys` inside `libstore` as it relies on the global variable `settings` to derive this information and is really only but a consumer of the `libutil`'s signature code.

As we are only reading the string data, this is more idiomatic.

We document the format and the remote signature protocol.

This allows someone oeprating a binary cache store to setup a remote signing path for remote signing operations.

The rest should work out of the box.

This avoids useless roundtrips every time we want to inquire about the public key.

Ericson2314 · 2024-01-03T17:31:56Z

doc/manual/src/protocols/signature.md

+# Signature
+
+In Nix, signatures are used to trust store paths using asymmetric cryptography,
+in this instance: Curve25519-based cryptography.
+
+## Store path signature format
+
+Store path signatures are ED25519 signatures, there are two types of
+signatures:
+
+- `ValidPathInfo` will assemble a fingerprint in the form of `1;<store
+ path>;<NAR hash in base32 including its own type>;<NAR size in bytes>;<NAR
+ references separated by ,>` and sign this information with the ED25519
+ private key.
+- `Realisation` will assemble a fingerprint in the form of a JSON string: `{
+ id, outPath, dependentRealisations }` and sign this information with the
+ ED25519 private key.
+
+# Remote signature protocol
+
+The remote signature protocol is a mechanism to offload the signature of your
+store paths to another machine that can possess the secret key material in a
+secure way.
+
+In this setup, Nix will contact a remote signing URL that you specified and ask
+to sign fingerprints over the wire.
+
+The protocol expects a UNIX domain socket to force you to handle proper
+authentication and authorization. `socat` is a great tool to manipulate all
+sorts of sockets.
+
+## Semantics of the APIs
+
+- `POST /sign`: expects a fingerprint as input and will return the signature
+ associated to that fingerprint.
+- `POST /sign-store-path`: expects a store path as a parameter and will attempt
+ to sign that specific store path which is expected to be present on the
+ signer's machine and return the signature in the response.
+- `GET /publickey`: receives the public key as a string in the response.
+
+## A note on the security of that scheme
+
+You are responsible to ensure that `/sign` cannot be abused to sign anything
+and everything, for this, a simple setup could involve setting up a TCP service
+that requires authentication, e.g. SSH or something on the HTTP level and you
+can run a highly privileged daemon on the machine that wants to benefit from
+signatures presenting a UNIX domain socket to Nix.


This file should be one sentence per line (or other punctuation / semantic boundary), not fixed line length wrapped.

Does that mean that you want to remove the fixed line length wrapping and leave it the existing punctuation or ?

I'll do it. @fricklerhandwerk wrote the non-fixed length lines somewhere in the contributions docs but I forget where. You can find it if you are curious.

…ization

This sets up infrastructure in libutil to allow for signing other than by a secret key in memory. NixOS#9076 uses this to implement remote signing. (Split from that PR to allow reviewing in smaller chunks.) Co-Authored-By: Raito Bezarius <[email protected]>

…n-prep Signer infrastructure: Prep for #9076

@Ericson2314

…ization Master included NixOS#9688 prep PR, shrinking the diff. Also the merge includes a number of slight adjustments from @Ericson2314.

Ericson2314 · 2024-01-03T22:35:35Z

doc/manual/src/protocols/signature.md

+- `POST /sign-store-path`:
+
+ Expects a store path as a parameter and will attempt to sign that specific store path which is expected to be present on the signer's machine and return the signature in the response.


This doesn't exist at the moment, but I wonder whether it should. Yes, I talked about moving things into libutil before, but my experience with hardware parse-display-and-sign devices make me wonder if we should always send over a "nice, JSON" form of what is to be signed, and the other end is responsible for concatenating the actual signature data.

If nothing else, forcing the remote side to munge the bytes is a good defense against lazy blind-singing implementations.

Well, it "exists" in that I implemented it in my service-side of this PR (https://github.com/cole-h/nixos-cache-signing-server), heh.

I do think sending the bits necessary for signing as a well-formed JSON document is a good idea, and will make it harder to misuse.

(The only reason /sign-store-path exists is because I initially implemented it in terms of shelling out to nix, because at the time I couldn't figure out how to do the signing as Nix does it. This has been since rectified, so honestly I wouldn't be opposed to removing it altogether, unless there's a reason to keep it around?)

As said privately to @Ericson2314, I don't believe this sign store path API is good for us yet because we are looking at a high frequency signature system, e.g. hydra.nixos.org and there's no distributed filesystems that will make those stores paths available to the signer in time so they can sign them.

For technical reasons, we need a blind signer that is sufficiently resistant to misuse to avoid disasters for random people and ourselves.

Sending a well-formed JSON document looks good to me, but how is it harder to misuse? The only way to make it better is to introduce authnz or auth inside the server and the protocol, this was obviously out of scope because we cannot represent all the usecases in such a simple PR.

Once this lands, my opinion is that we should focus our efforts on understanding how to move the signing to the remote server using PKCS#11 API (which is almost ready with cryptoki and rust-cryptoki thanks to @baloo) and then think from the HSM perspective. There are many missing pieces and knobs around this system of signature: no revocation, no "timestamping", no certification paths, etc. This is understandable why but all those years have passed and those missing features are limiting our operational range in regards to the cache security.

fricklerhandwerk · 2024-01-03T23:35:52Z

doc/manual/src/protocols/signature.md

+The protocol expects a UNIX domain socket to force you to handle proper authentication and authorization.
+`socat` is a great tool to manipulate all sorts of sockets.


This is missing an example of how any of this looks in action. Right now I wouldn't even know where to start in order to use it.

I wonder if I should really give an example because I don't want to give the trivial example that will make people run it insecurely, so I can give non-trivial examples to run it securely but with imaginary TCP services.

Yeah I plan on revising the docs a good bit

It would be enough to show how to invoke Nix to listen on a socket. I don't see from the docs which interfaces this change even exposes.

mschwaig · 2024-01-04T23:07:59Z

I wonder if this is fundamentally the right approach, as opposed to sticking to local signatures (with a per-machine key, and the signing done by another machine based on its trust of those per-machine keys). Such an approach would provide somewhat better traceability.

I agree and would like to add to this point. I see it as a big design issue in the existing signing scheme that the signatures do not make any statements about the origin of a path.
An updated signature format could include a cryptographically verifiable statement that the signing host (claims to have) built a given path itself. I think such a signature would serve as a good basis for another host (remote signer) to also produce a signature for the given path.

See #9644 for a more detailed description of that idea.

baloo · 2024-01-05T18:47:39Z

doc/manual/src/protocols/signature.md

+ 1;<store path>;<NAR hash in base32 including its own type>;<NAR size in bytes>;<NAR references separated by ,>
+ ```
+ and sign this information with the ED25519 private key.


I'm not sure what the exact signature scheme is, but if we intend to sign prehashed payload (and use the NAR hash in base32 specified here), ed25519 requires it to be sha512.
(https://docs.rs/ed25519-dalek/latest/ed25519_dalek/trait.DigestSigner.html#impl-DigestSigner%3CD,+Signature%3E-for-Context%3C'_,+'_,+SigningKey%3E)

github-actions bot added new-cli Relating to the "nix" command store Issues and pull requests concerning the Nix store labels Sep 30, 2023

RaitoBezarius commented Sep 30, 2023

View reviewed changes

src/libstore/local-store.cc Outdated Show resolved Hide resolved

RaitoBezarius commented Sep 30, 2023

View reviewed changes

src/libstore/signature/local-keys.cc Outdated Show resolved Hide resolved

RaitoBezarius commented Sep 30, 2023

View reviewed changes

src/libstore/signature/local-keys.hh Outdated Show resolved Hide resolved

RaitoBezarius changed the title ~~libstore: generalize signature scheme~~ libstore: generalize signature scheme and introduce remote signing Sep 30, 2023

RaitoBezarius force-pushed the signature-generalization branch from a71db16 to 78e091b Compare September 30, 2023 21:41

cole-h reviewed Sep 30, 2023

View reviewed changes

src/libstore/globals.hh Outdated Show resolved Hide resolved

cole-h reviewed Sep 30, 2023

View reviewed changes

src/libstore/signature/signer.cc Outdated Show resolved Hide resolved

lheckemann reviewed Oct 2, 2023

View reviewed changes

src/libstore/signature/signer.cc Outdated Show resolved Hide resolved

lheckemann reviewed Oct 2, 2023

View reviewed changes

src/libstore/signature/signer.cc Outdated Show resolved Hide resolved

lheckemann reviewed Oct 2, 2023

View reviewed changes

src/libstore/signature/signer.cc Outdated Show resolved Hide resolved

lheckemann reviewed Oct 2, 2023

View reviewed changes

src/libstore/signature/signer.cc Outdated Show resolved Hide resolved

lheckemann reviewed Oct 2, 2023

View reviewed changes

thufschmitt assigned Ericson2314 Nov 17, 2023

Ericson2314 added the idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. label Nov 20, 2023

thufschmitt removed the new-cli Relating to the "nix" command label Dec 20, 2023

RaitoBezarius force-pushed the signature-generalization branch from 78e091b to c5e06a4 Compare January 2, 2024 01:23

github-actions bot added the new-cli Relating to the "nix" command label Jan 2, 2024

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

doc/manual/src/protocols/signature.md Show resolved Hide resolved

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

src/libutil/hash.hh Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

src/libutil/signature/local-keys.cc Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

src/libstore/keys.hh Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

src/libstore/keys.hh Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

src/libutil/signature/signer.cc Show resolved Hide resolved

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

src/libutil/util.cc Outdated Show resolved Hide resolved

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

RaitoBezarius added 7 commits January 3, 2024 17:37

fixup! chore: move signature code from libstore to libutil

bf42bf4

libutil/signature: use string_view instead of references to string

d924a13

As we are only reading the string data, this is more idiomatic.

doc: add some words about signatures

51b96f2

We document the format and the remote signature protocol.

libstore/binary-cache-store: add remote-signing-path setting

dce47c0

This allows someone oeprating a binary cache store to setup a remote signing path for remote signing operations.

libstore/binary-cache-store: perform remote signing if available

90a5783

The rest should work out of the box.

libstore/signature: memoize the public key in the remote signer

b912333

This avoids useless roundtrips every time we want to inquire about the public key.

RaitoBezarius force-pushed the signature-generalization branch from 474f63c to b912333 Compare January 3, 2024 16:43

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/master' into signature-general…

28af2b8

…ization

Ericson2314 mentioned this pull request Jan 3, 2024

Signer infrastructure: Prep for #9076 #9688

Merged

Ericson2314 added a commit that referenced this pull request Jan 3, 2024

Merge pull request #9688 from obsidiansystems/signature-generalizatio…

65fad9b

…n-prep Signer infrastructure: Prep for #9076

Merge remote-tracking branch 'upstream/master' into signature-general…

a58585c

…ization Master included NixOS#9688 prep PR, shrinking the diff. Also the merge includes a number of slight adjustments from @Ericson2314.

Ericson2314 force-pushed the signature-generalization branch from 286ea7d to a58585c Compare January 3, 2024 22:29

Ericson2314 reviewed Jan 3, 2024

View reviewed changes

fricklerhandwerk requested changes Jan 3, 2024

View reviewed changes

baloo reviewed Jan 5, 2024

View reviewed changes

RaitoBezarius mentioned this pull request Mar 3, 2024

Hydra signs with >= 2 sets of keys NixOS/hydra#1366

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libstore: generalize signature scheme and introduce remote signing #9076

libstore: generalize signature scheme and introduce remote signing #9076

RaitoBezarius commented Sep 30, 2023 •

edited

Loading

lheckemann left a comment

RaitoBezarius commented Oct 2, 2023

lheckemann commented Oct 2, 2023

RaitoBezarius commented Oct 2, 2023

lheckemann commented Oct 3, 2023 •

edited

Loading

RaitoBezarius commented Oct 3, 2023

Ericson2314 commented Oct 3, 2023

RaitoBezarius commented Oct 3, 2023

zimbatm commented Oct 21, 2023

Ericson2314 commented Oct 21, 2023

nixos-discourse commented Nov 20, 2023

Ericson2314 commented Nov 20, 2023

Ericson2314 Jan 3, 2024

RaitoBezarius Jan 3, 2024

Ericson2314 Jan 3, 2024

RaitoBezarius Jan 3, 2024

Ericson2314 Jan 3, 2024

RaitoBezarius Jan 3, 2024

Ericson2314 Jan 3, 2024

Ericson2314 Jan 3, 2024

cole-h Jan 3, 2024

RaitoBezarius Jan 3, 2024

fricklerhandwerk Jan 3, 2024

RaitoBezarius Jan 3, 2024

Ericson2314 Jan 4, 2024

fricklerhandwerk Jan 5, 2024

mschwaig commented Jan 4, 2024

baloo Jan 5, 2024

		- `POST /sign-store-path`:

		Expects a store path as a parameter and will attempt to sign that specific store path which is expected to be present on the signer's machine and return the signature in the response.

		The protocol expects a UNIX domain socket to force you to handle proper authentication and authorization.
		`socat` is a great tool to manipulate all sorts of sockets.

libstore: generalize signature scheme and introduce remote signing #9076

Are you sure you want to change the base?

libstore: generalize signature scheme and introduce remote signing #9076

Conversation

RaitoBezarius commented Sep 30, 2023 • edited Loading

Motivation

Context

TODO list

Priorities

lheckemann left a comment

Choose a reason for hiding this comment

RaitoBezarius commented Oct 2, 2023

lheckemann commented Oct 2, 2023

RaitoBezarius commented Oct 2, 2023

lheckemann commented Oct 3, 2023 • edited Loading

RaitoBezarius commented Oct 3, 2023

Ericson2314 commented Oct 3, 2023

RaitoBezarius commented Oct 3, 2023

zimbatm commented Oct 21, 2023

Ericson2314 commented Oct 21, 2023

nixos-discourse commented Nov 20, 2023

Ericson2314 commented Nov 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mschwaig commented Jan 4, 2024

Choose a reason for hiding this comment

RaitoBezarius commented Sep 30, 2023 •

edited

Loading

lheckemann commented Oct 3, 2023 •

edited

Loading