Incorrect linker flags passed to Rust riscv32imc-unknown-none-elf cross-compiler

[SFrijters]

Describe the bug

I've been trying to expand on @n8henrie blog post "Compiling Rust for the ESP32 with Nix" and update nixpkgs to a more recent version. Changing the nixpkgs input from release-23.05 to release-23.11 and updating the lock file broke the linker.

After some bisecting it looks like 6980e6b is the last working commit, after which the cross compiler infrastructure experiences a regression and the gcc compiler doesn't build and I don't even get to the Rust part. This is fixed in 36f5b2e but then I get the linker error on my final executable that I still see on master:

       >   = note: x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-flavor'
       >           x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '--as-needed'; did you mean '-mno-needed'?
       >           x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '--gc-sections'; did you mean '--data-sections'?
       >           
       >
       > error: could not compile `http-client` (bin "http-client") due to 1 previous error

Steps To Reproduce

The simplified flake I use is available at https://github.com/SFrijters/esp32-c3-nix-rust-test and nix build github:SFrijters/esp32-c3-nix-rust-test reproduces the error - at least from my x86_64-linux base system.

Using the dev shell and cargo build works correctly.

Expected behavior

The program links correctly.

Additional context

Add any other context about the problem here.

Notify maintainers

@amjoseph-nixpkgs

Maybe @yu-re-ka or @alyssais know anything about this, recently having worked on the rust build-support files?

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.7.0, NixOS, 24.05 (Uakari), 24.05.20240115.56f85cb`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(frijters): `""`
 - channels(root): `""`
 - nixpkgs: `/etc/nixpkgs`

---

Add a 👍 reaction to issues you find important.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/packaging-qemu-with-support-for-esp32c3/38285/6

[SFrijters]

I looked a bit more into this: the problem is fixed when "CARGO_TARGET_${stdenv.hostPlatform.rust.cargoEnvVarTarget}_LINKER=${linkerForHost}" is not set explicitly. But I don't know what default decision cargo build makes when this variable is not present, or why it doesn't figure out the correct flags when it is. Any help is still appreciated.

[SFrijters]

Alternatively, forcing the use of LLD also fixes the issue. However, platform.isRiscV is false for stdenv.targetPlatform even though rustcTarget = "riscv32imc-unknown-none-elf" so I had to use shouldUseLLD = platform: true; to test. This also seems weird to me?

EDIT: I understand this a bit better now: I didn't set the platform value of crossSystem, so it was the same as my build machine. But if I set it to something more sensible, like

            crossSystem = lib.systems.examples.riscv32-embedded // {
              rustc.config = "riscv32imc-unknown-none-elf";
            };

the flags in my .cargo/config.toml file are no longer respected, but the values in a .config/config file that nixpkgs generates takes precedence and that causes compilation of my Rust dependencies to fail.

[SFrijters]

Workaround that works for me:

  cargoBuildFlags = [
    "--config target.riscv32imc-unknown-none-elf.linker='${pkgsCross.pkgsBuildHost.llvmPackages.bintools}/bin/${pkgsCross.stdenv.cc.targetPrefix}ld.lld'"
  ];

as used in https://github.com/SFrijters/nix-qemu-esp32c3-rust-example .

[liarokapisv]

Bump. I also had this issue when cross compiling to arm-none-eabihf. I had to basically go through the whole debugging process that @SFrijters did.

Some thoughts:

• Is it a good idea to default linkerFor* to ccFor* here ?
• Is it a good idea to ignore the project's .cargo/config.toml file ?

Another workaround is to patch the cargo-build-hook directly:

cargoBuildHook = (rustPlatform.cargoBuildHook).overrideAttrs
  (
    let envVars = pkgsCross.rust.envVars; in {
      setEnv = ''
        ${envVars.setEnv} \
        CARGO_TARGET_THUMBV7EM_NONE_EABIHF_LINKER=arm-none-eabi-ld \
      '';
    }
  );
buildRustPackage = rustPlatform.buildRustPackage.override {
  inherit cargoBuildHook;
};

[n8henrie]

Huh, I have no idea how I missed the notification on this post.

I've recently tried to get a new project for the esp32c3 compiling, and I've had to review my own blog post multiple times. I'm still having to use the oxalica overlay (in order to get a rust compiler with the appropriate toolchain), but I've been able to get it to compile with the new (crates.io-available) esp-wifi 0.6.0.

I would be interested in dropping the oxalica dependency and building with pure nixpkgs / rust if possible. However, it looks like https://github.com/SFrijters/esp32-c3-nix-rust-test is using oxalica as well, I'm not sure if there is currently a process to get an aarch64-darwin rustc with the riscv32imc-unknown-none-elf toolchain with just nixpkgs.

Changing the nixpkgs input from release-23.05 to release-23.11 and updating the lock file broke the linker.

I'm primarily on aarch64-darwin (though I also use aarch64-linux and x86_64-linux), and I've had to pin nixpkgs to 23.05 due to this issue. Interesting that a separate issue is blocking gcc-based toolchains from updating as well.

[SFrijters]

@n8henrie The https://github.com/SFrijters/esp32-c3-nix-rust-test repo contains the information about my debugging process, but a much cleaner example currently lives in https://github.com/SFrijters/nix-qemu-esp32c3-rust-example/ . I recently bumped the toolchain to 1.78.0, esp-hal to 0.18.0, uses a pretty recent nixpkgs, and I cleaned up the Cargo.toml file a bit (and I have another example project - also still based on your blog post - in a private repo that also managed the 0.6.0 version bump for esp-wifi in addition to these changes).

But, it also still uses the oxalica overlay. I still don't know enough to want to try and get rid of that as well, I'm pretty happy with the current situation, apart from the linker flag hack.

@liarokapisv It feels like there must be some smallish tweak to the code you linked and/or adjacent code that can be made to fix the issue, but I don't know enough about the impacts on other platforms that I don't use myself.

[n8henrie]

Looking at https://github.com/SFrijters/nix-qemu-esp32c3-rust-example/blob/master/flake.nix

We do not set system to something riscv related, which is kinda weird

Because we're using rust to cross-compile, not nix. If you were just using a standard rustup / rust install, you'd have an x86_64-linux rust binary, rustup target add riscv32imc-unknown-none-elf (and probably something about nightly and rust-src), and then compile using your x86_64-linux rustc. We're basically doing the same thing -- the oxalica stuff is taking care of getting the hostSystem-targeted binary with the toolchain, we're just using crossSystem to tell nix to build with --target=my-esp32c3-target.

Setting system to something espressif in crossSystem would be telling it to use a rustc compiled natively for the esp32c3 architecture. At least I think that's what's going on.

Speaking of which, I had used rustc.config = "riscv32imc-unknown-none-elf"; but I notice you changed that for rust.rustcTarget = "riscv32imc-unknown-none-elf";, which is admittedly more readable / intuitive. Any other benefits to changing this?

[SFrijters]

Re: weirdness - Ah, I see, I think :)

Regarding the rustc.config change: I had just stumbled upon #319624 and supposedly "This is the new way (tm)", following #271707 .

[liarokapisv]

@n8henrie The https://github.com/SFrijters/esp32-c3-nix-rust-test repo contains the information about my debugging process, but a much cleaner example currently lives in https://github.com/SFrijters/nix-qemu-esp32c3-rust-example/ . I recently bumped the toolchain to 1.78.0, esp-hal to 0.18.0, uses a pretty recent nixpkgs, and I cleaned up the Cargo.toml file a bit (and I have another example project - also still based on your blog post - in a private repo that also managed the 0.6.0 version bump for esp-wifi in addition to these changes).

But, it also still uses the oxalica overlay. I still don't know enough to want to try and get rid of that as well, I'm pretty happy with the current situation, apart from the linker flag hack.

@liarokapisv It feels like there must be some smallish tweak to the code you linked and/or adjacent code that can be made to fix the issue, but I don't know enough about the impacts on other platforms that I don't use myself.

The main issue is that linkerFor* variables are later used for CARGO_TARGET_*_LINKER. This is then directly set when calling cargo build in cargo's build hook through the setEnv string. The linker is set to ccForTarget by default. This means that the build will fail if ever passed linker-only flags that are not known by the active compiler. For example lto ends up appending --gc-sections to the linker which is not usable by gcc durectly, it needs to be passed to ld instead. Using cc as the linker is most likely wrong for most configurations and only works as long as no linker-specific flags are passed by the build system.

[n8henrie]

Regarding the rustc.config change

I see, thank you!