Hostname being set too late during bootup causes mismatching log metadata.

[eduarrrd]

Describe the bug

Hostname being unset (resulting in localhost) causes issues for logging. Specifically, with journald, the same boot (identified by _BOOT_ID (see systemd.journal-fields(7)) has both _HOSTNAME=localhost and _HOSTNAME=<intended hostname> with the switch happening at the rootfs pivot of the boot.

Why does this matter?

This makes filtering by hostname difficult. Even more so, when doing log aggregation from multiple machines, filtering by hostname will yield incomplete results. Instead one must build a pipeline that makes looks up all boot_ids of a hostname and querying all those instead.

Steps To Reproduce

Steps to reproduce the behavior:

1. boot
2. inspect journalctl -o cat --output-fields=_BOOT_ID,_HOSTNAME -b
3. see behavior described above

Expected behavior

journal contains the the same _HOSTNAME as long as it not changed by the user.

Hostname should be generally available, even during early boot(?).

Screenshots

N/A

Additional context

I'm filing this as a bug instead of a pull request because I'm not sure if there are any unintended consequences of doing this. Aside from the usual https://xkcd.com/1172/. Hoping for your input here.

There's a reference in the docs giving an example of setting the hostname by hand:

nixpkgs/nixos/modules/system/boot/systemd/initrd.nix

Line 163 in e2dd4e1

"/etc/hostname".text = "mymachine";

(from 5653209). Why this was chosen as an example and not just standard behavior, I don't know.

Notify maintainers

CC/FYI @dasJ , @roberth who added the example/was tagged in the commit.

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

╰$ nix-shell -p nix-info --run "nix-info -m"
- system: `"x86_64-linux"`
- host os: `Linux 6.9.3, NixOS, 24.11 (Vicuña), 24.11.20240607.051f920`
- multi-user?: `yes`
- sandbox: `yes`
- version: `nix-env (Nix) 2.18.2`
- channels(root): `""`
- channels(...): `""`
- nixpkgs: `/nix/store/3dr5pyja36lvvrszhzffww1jwyrx6i09-source`

---

Add a 👍 reaction to issues you find important.

[roberth]

Why this was chosen as an example and not just standard behavior, I don't know.

Me neither. I don't see a reason not to have that file. @dasJ may be more knowledgeable, but I'd say go for it.
Also systemd-based initrd is somewhat young, so I feel like the probability for spacebar heating is somewhat low.

[dasJ]

I think the main reason was to have the initrds be more substitutable between different hosts as this would be a difference between all of them (when having different systems with only different hostnames).

I'm not sure how well adding the hostname file would work. My memory is a bit fuzzy on the details (I did a lot of other projects at that time) but I think an issue was that when journald starts up, it imports the kernel logs BEFORE the hostname was set (causing the pre-systemd lines to be mapped to localhost). An alternative implementation that would probably solve that as well (if it's still/even an issue) would be adding the hostname= kernel parameter rather than adding the file to the initrd.

[eduarrrd]

If I read the code correctly, journald reads the hostname from /proc: server_init()->server_open_hostname():

https://github.com/systemd/systemd/blob/9afc6ac87c01c70917e996e7d3e2d381bf2b4f07/src/journal/journald-server.c#L2030-L2033

So the hostname needs to be set system-wide before journald starts up. Systemd does this in main()->initialize_runtime()->hostname_setup():

https://github.com/systemd/systemd/blob/9afc6ac87c01c70917e996e7d3e2d381bf2b4f07/src/shared/hostname-setup.c#L158-L211

So on bare metal we can either set systemd.hostname on the kernel cmdline or /etc/hostname it seems.

[roberth]

I think the main reason was to have the initrds be more substitutable between different hosts as this would be a difference between all of them (when having different systems with only different hostnames).

set systemd.hostname on the kernel cmdline

It seems this is the best choice then, but maybe also

adding the hostname= kernel parameter

as that's more standard and picked up even earlier in the boot process. Unless you're running an old kernel though. (<5.19?)
We could just set both.

[eduarrrd]

I'll run an experiment to see if /etc/hostname even solves my problem or one must use hostname=.

On that note, I also realized there's an equivalent (or slightly harder?) problem with machine_id as well :-.

[eduarrrd]

(So, setting /etc/hostname in the initrd works. Every message from that boot has the correct hostname.)

I was also thinking about your folks' usecase some more, and was wondering about one thing ... how are you approaching the UKI/secureboot? None of the mentioned approaches work for that, right?

[dasJ]

Why not? Most people use https://github.com/nix-community/lanzaboote for that which signs whatever UKI you have locally with a local key

[eduarrrd]

Because changing files or the kernel cmdline will cause different images to be created? Your design shares a single one among multiple hosts, no?

[dasJ]

Yes, that's the downside. Although, the cmdline is only baked into the UKI which is not in the nix store anyway. This way we could substitute the initrd and have differing hostnames.

I must mention, however, that I personally don't care about the substitutability of the initrd. My main focus was to keep the state as it was before, but if people decide it's time to change that, I'm not opposed to that ;)