stdenv/freebsd: revive #311795

rhelmot · 2024-05-14T23:57:34Z

Description of changes

part of #296581

Make it work. Depends on all the other PRs I've submitted in the past few hours.

Things done

Add a 👍 reaction to pull requests you find important.

alyssais · 2024-05-15T08:59:03Z

Can you ping me when all the dependency PRs are merged and this is ready for review?

rhelmot · 2024-06-17T05:33:46Z

@Ericson2314 @alyssais all dependencies have been merged, so this is now ready for review!

Some thoughts:

Since writing this I have learned that we can maybe get around the part where I have to statically compile a bunch of separate utilities in order to unpack the bootstrap tarball, since, well, the nar format can just represent a directory directly. This means we can probably drop mkdir, unzx, tar, and chmod, leaving bash and patchelf. However, I don't know how to do this! Help!
Is the goal to merge the bootstrap files code first, then build them and have them in the cache, and then merge stdenv? I'm really not sure how this works.

paparodeo · 2024-06-17T20:38:21Z

there is some documentation about the bootstrap-tools process. https://github.com/NixOS/nixpkgs/tree/master/maintainers/scripts/bootstrap-files

if you put everything in a nar file you don't need static builds. darwin does this with unpack.nar.gz

nixpkgs/pkgs/stdenv/darwin/make-bootstrap-tools.nix

Line 269 in 96b134d

dumpnar $out/unpack | xz -9 -T $NIX_BUILD_CORES > $out/on-server/unpack.nar.xz

tho not sure why they use dumpnar rather than nix-store --dump. the files in unpack.nar.xz contain rpaths relative to an $ORIGIN like anchor.

rhelmot · 2024-06-20T06:18:56Z

I pushed an update which switches to a single NAR of dynamic executables, and also uses recursive nix in order to drastically simplify the process of determining the runtime closure of the bootstrap files while also just using nix-store to dump the NAR as previously mentioned. It needs to merge with staging before it can actually build the native stdenv on the FreeBSD side, but it does still work!

rhelmot · 2024-06-20T06:25:45Z

notable: my freebsd nix doesn't seem to have xz or even gz support so I left it uncompressed. it's about 1.2GiB rn, but with #321150 it'll be probably around 1.0GiB

paparodeo · 2024-06-20T07:38:39Z

notable: my freebsd nix doesn't seem to have xz or even gz support so I left it uncompressed. it's about 1.2GiB rn, but with #321150 it'll be probably around 1.0GiB

nix/fetchurl.nix should handle nar.xz. can try the command bellow. the url can be a file:https:// on darwin but not linux.

nix-build -E - <<NIX
import <nix/fetchurl.nix> {
    url = "http:https://tarballs.nixos.org/stdenv/x86_64-apple-darwin/d03a4482228d4d6dbd2d4b425b6dfcd49ebe765f/unpack.nar.xz";
    hash = "sha256-93GK8LjjgUBknxsylfGVr0DG4AbWVIQEIWrwxhDW07k=";
    name = "unpack";
    unpack = true;
}
NIX

the darwin bootstrap tools are 46M compressed, 257 uncompressed. llvm takes up a ton of space. you can probably remove all of the .a files from llvm except for some in compiler-rt to trim things down. having atime enabled on your filesystem and then using find /nix/store/expanded/nar -anewer reference and running a bootstrap can show what needs to be trimmed.

can you use $ORIGIN in RPATH or some permutation $ORIGIN/../lib or $ORIGIN/./ and update the shared objects before dumping them into a nar? it would prevent having 2 copies of the expanded nar -- 1 with bad RPATHS and the other with fixed RPATHS.

rhelmot · 2024-06-20T07:45:10Z

I can do that (referring to the latter request) but I don't think it's possible or even worth it to not have a patching phase for the bootstrap tools.

Possible: I'll still have to patch shebangs and other misc references to /nix/store (there are MANY) and I don't think mixing symlinks and copies in the patches derivation is a good idea

Worth it: doesn't the size of the bootstrap files literally only matter to hydra, since they're not included in the build closure for anything using the final stdenv?

paparodeo · 2024-06-20T08:33:53Z

I can do that (referring to the latter request) but I don't think it's possible or even worth it to not have a patching phase for the bootstrap tools.

yeah -- that is an issue on darwin too. the libs need to be signed with the correct path and the full path written into the libs as well as fixing up she shebangs. this is why only bash, tar, mkdir, xz + needed libs are in the nar file and everything else is just packed into a tarball. using the tarball allows for the modifications to happen in-place at extraction time.

Worth it: doesn't the size of the bootstrap files literally only matter to hydra, since they're not included in the build closure for anything using the final stdenv?

this sounds correct unless building the stdenv. given darwin and freebsd use llvm i'll guess 225M is around the lower bound for the size of uncompressed tools. i think darwin tools were 500M before the static llvm libs were removed. keeping storage down seems worthwhile for any package, 1G just seems like a lot for the bootstrap tools compared to other systems and given the darwin comparison, perhaps wrongly, it seems like 3/4 of the space is unnecessary.

rhelmot · 2024-06-20T23:01:13Z

Good news everyone! All of @paparodeo's concerns have been addressed. There are now stage0.nar.xz (1.8MB compressed, 5MB decompressed) and stage1.tar.xz (50MB compressed, 292MB decompressed). I was able to accomplish this by generating and committing a large list of unused files. I wasn't able to come up with a good method for doing it more generically, but this has the best savings anyway. I did verify that stdenv builds fine with the shrunken files.

paparodeo

not an exhaustive list but just a few things i noticed.

paparodeo · 2024-06-21T06:58:44Z

pkgs/stdenv/freebsd/bootstrap-files.nix

@@ -0,0 +1,105 @@
+{ system }:


Q: what is the command use to build the bootstrap files?

for linux can do nix-build -A freshbootstrapTools.build or nix-build pkgs/top-level/release.nix -A stdenvBootstrapTools.x86_64-unknown-linux-gnu.build

the args probably need to be adjusted when comparing this with the darwin/make-bootstrap-tools.nix but i don't know the correct approach off hand.

It's supposed to be nix-build with the full path to that exact nix file. I didn't realize the other bootstrap tools were accessible from the nixpkgs default entry point. I'll set up an entry for them soon.

paparodeo · 2024-06-21T07:00:32Z

pkgs/stdenv/freebsd/bootstrap-files.nix

+ pack-all =
+ packCmd: name: pkgs: fixups:
+ (runCommand name {
+ requiredSystemFeatures = [ "recursive-nix" ];


this needs to be enabled in the experimental-features list of nix.conf, which seems like it could be problematic, tho i do like this approach.

paparodeo · 2024-06-21T07:01:20Z

pkgs/stdenv/freebsd/bootstrap-files.nix

+ nix_store=${lib.getBin buildPackages.nix}/bin/nix-store
+ rsync=${lib.getExe buildPackages.rsync}


these can be added to nativeBuildInputs and then just can use the command name in the script rather than an env var.

paparodeo · 2024-06-21T07:05:50Z

pkgs/stdenv/freebsd/bootstrap-files.nix

+ for dir in $requisites; do
+ cd "$dir/nix-support" 2>/dev/null || continue
+ for f in $(find . -type f); do
+ mkdir -p "$base/nix-support/$(dirname $f)"
+ cat $f >>"$base/nix-support/$f"
+ done
+ done


is re-creating nix-support required? it looks like it adds propogated-build-inputs and setup.

Yes, cacert needs the setup hook.

paparodeo · 2024-06-21T07:12:21Z

pkgs/stdenv/freebsd/unpack-source.sh

+ *.tar.xz | *.tar.lzma | *.txz)
+ # Don't rely on tar knowing about .xz.
+ xz -d < "$fn" | tar xf - --warning=no-timestamp
+ ;;


I don't think this is the case anymore. tar works (for me) fine on .xz

This function is adapted from stdenv/generic/setup.sh _defaultUnpack but made to work with BSD cp, which can handle the nuances of BSD filesystem attributes a lot better. iirc some packages simply won't unpack without overriding the basic unpacker. The verbiage on the comment from the origin seems to have changed since I copied it but it says the same thing.

paparodeo · 2024-06-21T07:17:34Z

pkgs/stdenv/freebsd/bootstrap-files.nix

+ ${packCmd}
+ '');
+ nar-all = pack-all "$nix_store --dump . | xz -9 -T $NIX_BUILD_CORES >$out";
+ tar-all = pack-all "XZ_OPT=\"-9 -T $NIX_BUILD_CORES\" tar cJf $out .";


i think there are some more flags needed so the tarball is reproducible

nixpkgs/pkgs/stdenv/linux/make-bootstrap-tools.nix

Line 203 in ed0af8c

XZ_OPT="-9 -e" tar cvJf $out/on-server/bootstrap-tools.tar.xz --hard-dereference --sort=name --numeric-owner --owner=0 --group=0 --mtime=@1 -C $out/pack .

paparodeo · 2024-06-21T07:24:14Z

pkgs/stdenv/freebsd/bootstrap-files.nix

+ # - manually remove from the list the following; they are not marked as atime'd even though they are used
+ # - bin/strings # used only during bootstrap
+ # - plop it here
+ ) "xargs rm -f <${./bootstrap-files-spurious.txt}";


Suggested change

) "xargs rm -f <${./bootstrap-files-spurious.txt}";

) "xargs rm -f < <(grep -v '\<include/' ${./bootstrap-files-spurious.txt}");

maybe leaving the header files alone makes this slightly more robust? adds 19M/1.9M uncompressed / compressed. idk. mostly saying this because i see that include/{stddef,stdbool}.h are in the spurious list.

rhelmot · 2024-06-21T07:46:06Z

Everything I haven't commented on so far I'll fix in a day or two. Feel free to keep reviewing in the meantime!

graylog-5_2: 5.2.7 -> 5.2.9

Fixes: Traceback (most recent call last): File "/nix/store/km8nzjccd4r0g704is31q18qzl101g89-safeeyes-2.1.9/bin/.safeeyes-wrapped", line 6, in <module> from safeeyes.__main__ import main File "/nix/store/km8nzjccd4r0g704is31q18qzl101g89-safeeyes-2.1.9/lib/python3.12/site-packages/safeeyes/__main__.py", line 32, in <module> from safeeyes import utility File "/nix/store/km8nzjccd4r0g704is31q18qzl101g89-safeeyes-2.1.9/lib/python3.12/site-packages/safeeyes/utility.py", line 35, in <module> from distutils.version import LooseVersion ModuleNotFoundError: No module named 'distutils'

cargo-make: 0.37.13 -> 0.37.14

Mic92 · 2024-07-20T04:24:00Z

The other depending pr has been merged.

The old stdenv didn't work, and was also impure. The new one works, and is pure. Presently, the bootstrap tools are cross compiled into one small nar and one large tar, which is then unpacked, patched, and split into smaller derivations. Efforts were made to make the boot process as short as possible - there are only two clangs built, and as many packages are propagated between stages as possible while leaving the bootstrap tools out of the final stdenv's closure.

rhelmot requested review from alyssais and Ericson2314 as code owners May 14, 2024 23:57

github-actions bot added 6.topic: stdenv Standard environment 6.topic: flakes The experimental Nix feature 6.topic: lib The Nixpkgs function library labels May 14, 2024

rhelmot added the 6.topic: bsd Running or building packages on BSD label May 15, 2024

rhelmot force-pushed the freebsd-minimal3/stdenv/freebsd branch 2 times, most recently from 50cc9aa to 59e1636 Compare May 27, 2024 19:54

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels May 27, 2024

rhelmot force-pushed the freebsd-minimal3/stdenv/freebsd branch from 59e1636 to b51eda5 Compare May 31, 2024 17:29

rhelmot mentioned this pull request May 31, 2024

rustc: Add support for FreeBSD #316257

Merged

14 tasks

rhelmot force-pushed the freebsd-minimal3/stdenv/freebsd branch from b51eda5 to 71171d7 Compare June 4, 2024 15:57

AndersonTorres added the 5. scope: tracking Long-lived issue tracking long-term fixes or multiple sub-problems label Jun 17, 2024

rhelmot force-pushed the freebsd-minimal3/stdenv/freebsd branch from 71171d7 to 909b5ea Compare June 20, 2024 06:15

rhelmot force-pushed the freebsd-minimal3/stdenv/freebsd branch from 909b5ea to cd24f3d Compare June 20, 2024 22:55

paparodeo reviewed Jun 21, 2024

View reviewed changes

wegank added the 2.status: merge conflict label Jul 4, 2024

emilytrau and others added 2 commits July 11, 2024 10:37

cewler: init at 1.2.0

335505c

jmusicbot: 0.4.1 -> 0.4.2

42e552d

pbsds and others added 4 commits July 20, 2024 03:05

Merge pull request NixOS#327730 from bbenno/graylog-5_2/update

cee1917

graylog-5_2: 5.2.7 -> 5.2.9

go: support FreeBSD

8a6b616

Merge pull request NixOS#328510 from r-ryantm/auto-update/cargo-make

82cbb28

cargo-make: 0.37.13 -> 0.37.14

rhelmot added 3 commits July 19, 2024 21:27

[squashme] do the requested changes

d586b61

[squashme] Remove some hacks

765ebb0

rhelmot force-pushed the freebsd-minimal3/stdenv/freebsd branch from 369f402 to 765ebb0 Compare July 20, 2024 04:28

rhelmot requested review from RaitoBezarius, kalbasit, katexochen, Mic92, zowoq, adisbladis, mweinelt, K900, NickCao, SuperSandro2000, ttuegel and emilylange as code owners July 20, 2024 04:28

github-actions bot added 6.topic: python 6.topic: nixos 8.has: module (update) 6.topic: emacs 6.topic: policy discussion 6.topic: golang 8.has: maintainer-list (update) labels Jul 20, 2024

rhelmot closed this Jul 20, 2024

rhelmot mentioned this pull request Jul 20, 2024

FreeBSD stdenv: re-init #328582

Open

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stdenv/freebsd: revive #311795

stdenv/freebsd: revive #311795

rhelmot commented May 14, 2024 •

edited

Loading

alyssais commented May 15, 2024

rhelmot commented Jun 17, 2024

paparodeo commented Jun 17, 2024

rhelmot commented Jun 20, 2024

rhelmot commented Jun 20, 2024 •

edited

Loading

paparodeo commented Jun 20, 2024

rhelmot commented Jun 20, 2024

paparodeo commented Jun 20, 2024 •

edited

Loading

rhelmot commented Jun 20, 2024

paparodeo left a comment

paparodeo Jun 21, 2024

rhelmot Jun 21, 2024

paparodeo Jun 21, 2024

paparodeo Jun 21, 2024

paparodeo Jun 21, 2024

rhelmot Jun 21, 2024

paparodeo Jun 21, 2024

rhelmot Jun 21, 2024

paparodeo Jun 21, 2024

paparodeo Jun 21, 2024

rhelmot commented Jun 21, 2024

Mic92 commented Jul 20, 2024

		nix_store=${lib.getBin buildPackages.nix}/bin/nix-store
		rsync=${lib.getExe buildPackages.rsync}

	) "xargs rm -f <${./bootstrap-files-spurious.txt}";
	) "xargs rm -f < <(grep -v '\<include/' ${./bootstrap-files-spurious.txt}");

stdenv/freebsd: revive #311795

stdenv/freebsd: revive #311795

Conversation

rhelmot commented May 14, 2024 • edited Loading

Description of changes

Things done

alyssais commented May 15, 2024

rhelmot commented Jun 17, 2024

paparodeo commented Jun 17, 2024

rhelmot commented Jun 20, 2024

rhelmot commented Jun 20, 2024 • edited Loading

paparodeo commented Jun 20, 2024

rhelmot commented Jun 20, 2024

paparodeo commented Jun 20, 2024 • edited Loading

rhelmot commented Jun 20, 2024

paparodeo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhelmot commented Jun 21, 2024

Mic92 commented Jul 20, 2024

rhelmot commented May 14, 2024 •

edited

Loading

rhelmot commented Jun 20, 2024 •

edited

Loading

paparodeo commented Jun 20, 2024 •

edited

Loading