Add artifact sign/verify via libsodium. #357

bookshelfdave · 2016-04-07T20:18:34Z

This PR adds artifact sign/verify via libsodium.

The following subcommands have been implemented:

hab artifact sign
hab artifact verify
hab origin key generate

hab archive has been renamed to hab artifact.

Note, this does not implement user or service keys, and also excludes encryption/decryption. I'm going to leave some commented out code in (any code that includes "box"), please ignore for now and I'll update it when this story is completed next week.

1 minute tutorial

Generate an origin key

./target/debug/hab origin key generate habitat

Make a tarfile that you want to sign:

tar cvJf chef-zlib.-1.2.8-20160310193228.xz /opt/bldr/pkgs/chef/zlib/1.2.8/20160310193228/

Create a signed artifact:

./target/debug/hab artifact sign chef-zlib.-1.2.8-20160310193228.xz chef-zlib.-1.2.8-20160310193228.hab --origin habitat

OR

export HABITAT_ORIGIN=habitat
./target/debug/hab artifact sign chef-zlib.-1.2.8-20160310193228.xz chef-zlib.-1.2.8-20160310193228.hab

Look at the header:

head -3 chef-zlib.-1.2.8-20160310193228.hab

Skip verification and just access the tarball:

tail -n +4 chef-zlib.-1.2.8-20160310193228.hab | tar tvJ

Verify the artifact and extract it's contents

./target/debug/hab artifact verify ./chef-zlib.-1.2.8-20160310193228.hab ./out.xz

Make sure it worked:

tar tvJf out.xz

TODO:

Development notes

GPG still exists in the repo

I'll remove references to GPG in a later PR

sodiumoxide + libsodium-sys

While we are waiting for the generichash functions from this PR to be merged, we're pointing to a fork. The annoying thing about this is that we have to include the entire repo as a cargo override via components/.cargo/config and Cargo.toml . Once the PR is merged to the official repo, we can remove these files and continue in peace.

Compiling

In order to compile this today, you'll need the libsodium package installed and Rust needs to know how to link to libsodium via SODIUM_LIB_DIR.

export SODIUM_LIB_DIR=/opt/studios/src/opt/bldr/pkgs/chef/libsodium/1.0.8/20160406134120/lib
export LD_LIBRARY_PATH=${SODIUM_LIB_DIR}

I updated the depot and sup plans to include libsodium, but we'll need to address the plan that builds hab asap.

These are a prettier version of the docs I've embedded inside of hab_crypto.rs:

Habitat uses libsodium and it's Rust counterpart sodiumoxide for cryptographic operations.

Concepts and terminology:

All public keys/certificates/signatures will be referred to as public.
All secret or private keys will be referred to as secret.
The word key by itself does not indicate public or secret. The only exception is if the word key appears as part of a file suffix, where it is then considered the secret key file.
Origin - refers to build-time operations, including signing and verifification of an artifact.
Organization - refers to run-time operations that can happen in Habitat, such as deploying a package signed in a different origin into your own organization.
Signing keys - aka sig keys. These are used to sign and verify packages. Contains a sig.key file suffix. Sig keys are NOT compatible with box keys.
Box keys - used for encryption/decryption of arbitrary data. Contains a .box.key file suffix. Box keys are NOT compatible with sig keys.
Key revisions - Habitat can use several keys for any given user, service, or origin via different revision numbers. Revision numbers appear following the key name and are in the format
{year}{month}{day}{hour24}{minute}{second}. For all user-facing cryptographic operations (sign/verify/encrypt/decrypt), the latest key is tried first, and upon failure, Habitat will try keys in reverse chronological order until success or there are no more keys. _TODO: key revisions are generated as part of a filename, but only the most recent key is used during crypto operations._

Example origin key file names ("sig" keys):

 habitat-201603312016.pub
 habitat-201603312016.sig.key
 your_company-201604021516.pub
 your_company-201604021516.sig.key

Example user keys ("box" keys)

 [email protected]
 [email protected]

Example Service keys:

 [email protected]

Habitat signed artifact format

A signed .hab artifact has 3 plaintext lines followed by a binary blob of data, which is the unsigned tarfile.

The first plaintext line is the name of the origin signing key that was used to sign this artifact.
The second plaintext line is the hashing algorithm used, which will be BLAKE2b unless our use of crypto is expanded some time in the future.
The third plaintext line is a base64 signed value of the binary blob's base64 file hash. Signing uses a secret origin key, while verifying uses the public origin key. Thus, it it safe to distribute public origin keys.

Example header:

habitat-20160405144945
BLAKE2b
signed BLAKE2b signature
<binary-blob>

https://download.libsodium.org/doc/hashing/generic_hashing.html

It's possible to examine the contents of a .hab file from a Linux shell:

$ head -3 /opt/bldr/cache/pkgs/chef-glibc-2.22-20160310192356.bldr
habitat-20160405144945
BLAKE2b
w4yC7/QADdC+NfH/wgN5u4K94nMieb1TxTVzbSfpMwRQ4k+YwhLs1nDXSIbSC8jHdF/7/LqLWtgPvGDmoKIvBDI0aGpIcGdlNDJhMDBnQ3lsMVVFM0JvRlZGSHhXcnBuWWF0SllXTXo1ZDg9
# Note that this is an example signature only

It is also possible to extract a plain tarball from a signed .hab artifact using the following command:

tail -n +4 /tmp/somefile.hab > somefile.tar
# start at line 4, skipping the first 3 plaintext lines.

bookshelfdave · 2016-04-07T21:11:26Z

I'll make the verify subcommand return an appropriate exit code instead of spitting out the tar file.

fnichol · 2016-04-08T14:23:04Z

Dockerfile

+ iproute2 \
+ libsodium-dev \
+ sudo \
+ vim \


Could you alphabetize into the list? It should help with future merging

fnichol · 2016-04-08T14:58:56Z

@metadave I realize this might still be finishing up, but great work! You must have had fun with Rust/IO pushing all those files around.

Because we're using components/.cargo/config to override sodiumoxide to point at a custom fork, we need to CD into each directory instead of specifying the manifest location to allow the build to complete. - add iproute2 dep for `ip` command

reset · 2016-04-08T20:12:06Z

Makefile

- $(run) cargo build --manifest-path components/hab/Cargo.toml
- $(run) cargo build --manifest-path components/sup/Cargo.toml
- $(run) cargo build --manifest-path components/depot/Cargo.toml
+ $(run) sh -c 'cd components/hab && cargo build'


Did this become a problem? Why change directory instead of just pass the manifest-path?

yes, because of the components/.cargo/config file

Thanks Cargo

chef-delivery · 2016-04-08T20:13:59Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

reset · 2016-04-08T20:14:05Z

components/core/src/crypto.rs

+
+/// If an env var is set, then return it's value.
+/// If it's not, return the default
+fn env_var_or_default(env_var: &str, default: &str) -> String {


This might be better served in the util module instead of crypto

agreed, I'll pull it out in my next few crypto PR's, as I have some more work do to w/ env vars etc.

reset · 2016-04-08T20:30:25Z

@metadave great work man, I'm looking forward to when they merge your changes ;). My comments are mostly just tips on how to be more fluent in Rust

reset · 2016-04-08T20:33:41Z

blast-off when ready

bookshelfdave · 2016-04-08T21:16:37Z

thanks so much for the review @reset @fnichol
review fixes pushed, getting ready to merge soon

bookshelfdave · 2016-04-08T21:17:10Z

Note, this needs to be merged after: #361

chef-delivery · 2016-04-08T21:28:58Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

bookshelfdave · 2016-04-08T21:30:43Z

@delivery approve

chef-delivery · 2016-04-08T21:30:47Z

Change: b7477faa-3da9-4c5d-b8a4-0697e52ae948 approved by: @metadave

chef-delivery added the Delivery Change Created label Apr 7, 2016

bookshelfdave force-pushed the dp_hab_nacl branch from 97e26f3 to 22596a5 Compare April 7, 2016 20:27

bookshelfdave force-pushed the dp_hab_nacl branch 2 times, most recently from 80a527a to df8f586 Compare April 8, 2016 13:51

fnichol reviewed Apr 8, 2016
View reviewed changes

Dockerfile

iproute2 \

libsodium-dev \

sudo \

vim \

Copy link

Collaborator

fnichol Apr 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you alphabetize into the list? It should help with future merging

Dave Parfitt added 4 commits April 8, 2016 13:15

artifact sign/verify

68cfff5

add libsodium dep to other hab components

106e051

update embedded docs

5c9c68a

bookshelfdave force-pushed the dp_hab_nacl branch from e9cc7ab to f58804e Compare April 8, 2016 17:16

crypto review cleanup

45a6776

bookshelfdave force-pushed the dp_hab_nacl branch from 6c58788 to 45a6776 Compare April 8, 2016 19:36

update hab plan to build w/ libsodium

5ffaa28

reset reviewed Apr 8, 2016
View reviewed changes

review cleanup, add libsodium to LD_LIBRARY_PATHs

d5c9839

chef-delivery merged commit d4eccfb into master Apr 8, 2016

chef-delivery deleted the dp_hab_nacl branch April 8, 2016 21:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add artifact sign/verify via libsodium. #357

Add artifact sign/verify via libsodium. #357

bookshelfdave commented Apr 7, 2016

bookshelfdave commented Apr 7, 2016

fnichol Apr 8, 2016

fnichol commented Apr 8, 2016

reset Apr 8, 2016

bookshelfdave Apr 8, 2016

reset Apr 8, 2016

chef-delivery commented Apr 8, 2016

reset Apr 8, 2016

bookshelfdave Apr 8, 2016

reset commented Apr 8, 2016

reset commented Apr 8, 2016

bookshelfdave commented Apr 8, 2016

bookshelfdave commented Apr 8, 2016

chef-delivery commented Apr 8, 2016

bookshelfdave commented Apr 8, 2016

chef-delivery commented Apr 8, 2016

Add artifact sign/verify via libsodium. #357

Add artifact sign/verify via libsodium. #357

Conversation

bookshelfdave commented Apr 7, 2016

1 minute tutorial

TODO:

Development notes

GPG still exists in the repo

sodiumoxide + libsodium-sys

Compiling

Concepts and terminology:

Habitat signed artifact format

bookshelfdave commented Apr 7, 2016

fnichol Apr 8, 2016

Choose a reason for hiding this comment

fnichol commented Apr 8, 2016

reset Apr 8, 2016

Choose a reason for hiding this comment

bookshelfdave Apr 8, 2016

Choose a reason for hiding this comment

reset Apr 8, 2016

Choose a reason for hiding this comment

chef-delivery commented Apr 8, 2016

reset Apr 8, 2016

Choose a reason for hiding this comment

bookshelfdave Apr 8, 2016

Choose a reason for hiding this comment

reset commented Apr 8, 2016

reset commented Apr 8, 2016

bookshelfdave commented Apr 8, 2016

bookshelfdave commented Apr 8, 2016

chef-delivery commented Apr 8, 2016

bookshelfdave commented Apr 8, 2016

chef-delivery commented Apr 8, 2016