Switch hashing from openssl to libsodium, drop openssl dependency

[t184256]

If anybody's interested in dropping the openssl dependency in favor of libsodium, here's my attempt at it.

Marking as a draft because libsodium doesn't have MD5 and SHA-1, so we'd have to deprecate them first.

[edolstra]

Dropping MD5 or SHA1 is not an option because it would make it impossible to build old Nixpkgs versions.

[edolstra]

Note that we used to have our own copy of MD5 and SHA1 in-tree (removed in a6ca68a), so we could restore those.

[t184256]

Thanks, I'll look into it.

[t184256]

OK, taking the suggested mix-and-match way of restoring bundled SHA-1/MD5, but using SHA-2 from libsodium. Removing Draft because in this form it doesn't sound too scary to land.

Do we have at least some plans to retire MD5/SHA-1? Gate them behind a config option and flip it to 0 in Nix 3.0, maybe?

[t184256]

re CI failure: tests/gc-auto passes on my machine and doesn't look related to me. flaky test?

[edolstra]

@t184256 Yes, you can ignore that.

[edolstra]

Unfortunately this slows down hashing a lot (except for md5). Tested on a 2.9 GiB file, before:

md5: 3.80user 0.66system 0:04.51elapsed 98%CPU (0avgtext+0avgdata 20080maxresident)k
sha1: 1.46user 0.61system 0:02.13elapsed 97%CPU (0avgtext+0avgdata 20224maxresident)k
sha256: 1.55user 0.61system 0:02.22elapsed 97%CPU (0avgtext+0avgdata 20072maxresident)k
sha512: 3.63user 0.64system 0:04.33elapsed 98%CPU (0avgtext+0avgdata 20272maxresident)k

After:

md5: 4.05user 0.60system 0:04.70elapsed 98%CPU (0avgtext+0avgdata 20312maxresident)k
sha1: 4.44user 0.60system 0:05.10elapsed 99%CPU (0avgtext+0avgdata 20056maxresident)k
sha256: 9.92user 0.58system 0:10.56elapsed 99%CPU (0avgtext+0avgdata 20048maxresident)k
sha512: 6.21user 0.58system 0:06.84elapsed 99%CPU (0avgtext+0avgdata 20076maxresident)k

Especially sha256 is critical, we can't slow it down.

Edit: it appears that libsodium doesn't have SSE/AVX-optimized versions of sha256/sha512.

[edolstra]

It's also worth noting that dropping the openssl dependency in itself does not reduce Nix's closure size, since libcurl still depends on it.

[t184256]

Sigh. OK, I tried =) Feel free to close it for the better times.

[t184256]

Yeah, I've noticed that, but that depends on what libcurl's compiled with. For my purposes, I was going to switch libcurl to something else as well.

[stale]

I marked this as stale due to inactivity. → More info