Discussion: The future of SDKs on Darwin

[reckenrode]

With #234710 getting close to done and the LLVM bump imminent, #229210 should be able to move forward adding new SDKs. We’ve had several conversations on Matrix regarding how we’d like to handle multiple SDKs. The purpose of this issue is to capture those conversations and work towards a consensus.

I’m pinging @toonn and @emilazy because we have discussed this on Matrix before. I am also pinging @jtojnar because @drupol mentioned having talked with him about the SDK issue. If there is anyone I forget, feel free to ping them.

Assumptions

• There need to be multiple SDKs. Unfortunately, if we want to build software that uses features on new platforms, we can’t just use a newer SDK by default because static feature detection may result in binaries that don’t run on older platforms.
• x86_64-darwin will continue to maintain an older default than aarch64-darwin. That is currently 10.12 but will need to increase to 10.13 because libc++ 17 has dropped support for 10.12.
• It should be easy to use in the simple case while also composing well with language and other complex frameworks.
• It should be possible to mix dependencies using different SDKs, and switching an SDK should not result in a complete rebuild of the stdenv just to use it. All stdenvs should share a common libc++.

Current Situation

There are two SDKs currently in nixpkgs: the 10.12 SDK and the 11.0 SDK. 10.12 is the default on x86_64-darwin, and 11.0 is the default on aarch64-darwin. The 10.12 SDK is built from Apple’s source releases where possible while the 11.0 SDK is derived from the upstream SDK. There is some shared implementation but also a lot of custom implementation between the two.

The default SDK is darwin.apple_sdk. Packages that need to access SDK packages typically use the package and the inherit (darwin) <package> pattern. To use a different SDK, the common pattern is to use apple_sdk_<version>.callPackage, which overrides some parts of Darwin to provide versions from the new SDK. However, explicitly inheriting packages also requires updating those to reference the new SDK.

Friction Points

The SDK typically ends up providing SDK-specific stdenvs and language frameworks (like rustPlatform). The SDK should not have to concern itself with how it is being used downstream, and those frameworks should just worked based on the chosen SDK (picking it up as well as the updated compilers).

xcbuild hardcodes the SDK it uses. It should be possible to override this in a way that does not require rebuilding xcbuild every time a new SDK is required.

propagatedBuildInputs do not work well with SDK overrides. For example, it is not possible to mix libGLU, which propagates a couple of SDK frameworks, in a package that needs a different SDK because it results in headers being used from the wrong SDK.

The top level of darwin attribute set contains some SDK-specific items (like Libsystem). These need to be moved to their appropriate SDKs. The top level should contain only packages that are Darwin-specific and part of or associated with an SDK.

CF is a massive pain in the ass. It should either be dropped or made to build on aarch64-darwin. Having two different build paths through the stdenv bootstrap complicates things. The way CF propagates also makes the stdenv bootstrap very sensitive to the order things are built on x86_64-darwin. There are also risks for cyclical dependencies¹.

Updating SDKs and making them source-based is difficult. This is going to be a problem for x86_64-darwin when it comes time to bump to clang 18 next year because libc++ will have dropped support for the platform, and additional post-10.12 APIs may be used (meaning it may not be feasible just to revert the commit that removed to 10.12 support).

---

¹: There are two examples I have encountered while working on the stdenv rework.

• CF depended on curl, which depends on CoreFoundation. The old stdenv avoided this issue by not including the rpath, causing nix not to find the dependency on the linked CF from a prior bootstrap stage. This was fixed in the new stdenv by removing the curl usage from CF. Fortunately, it is not needed in our usage because we do not build swift-corelibs Foundation.
• Bash depends on CF, but CF depends on libxml2 and ICU, which provide scripts that include shebang lines that need patched to use bash from nixpkgs. This worked in the old stdenv because only it used darwin.ICU, and libxml2 just happened to work with the system bash on Darwin. The reworked stdenv uses the nixpkgs icu, which caused problems with packages that used the script to get icu’s build flags. This was fixed by making bash depend on CoreFoundation instead of CF.

[reckenrode]

I’m replying with my perspective separately to keep it from biasing the OP. I would like to see a singular approach to overriding the SDK and a pattern that plays nicely with it. It seems to me that the SDK is analogous to the libc, and so should live as an attribute on the stdenv. When you need an SDK package, you should reference stdenv.apple_sdk in your derivation. This has two benefits:

• It makes overriding the SDK be a stdenv adapter. This should allow the SDK to be overriden for any stdenv without requiring it to provide a special, vendored stdenv as it does today.
• It plays nicely with callPackage and the simple package paths proposal because references to specific frameworks would be handled in the derivation.

For example, given an adapter might like:

overrideSDK = stdenv: new_sdk:
  let mkCC = cc:
    cc.override {
      bintools =s tdenv.cc.bintools.override { libc = new_sdk.Libsystem; };
      libc = new_sdk.Libsystem;
    };
  in
  (overrideCC stdenv (mkCC stdenv.cc)).override {
    extraBuildInputs = [ new_sdk.CF ];
    hostPlatform = stdenv.hostPlatform // {
      darwinSdkVersion = new_sdk.version;
    };
  } // {
    apple_sdk = new_sdk;
  };

Given the following derivation:

{ stdenv, zlib }:

stdenv.mkDerivation {
  name = "my-package";
  version = "1.0.0";
  buildInputs = [ zlib ]
    ++ lib.optionals stdenv.isDarwin (with stdenv.apple_sdk.frameworks; [ CoreFoundation ]);
}

There are two ways to use this. With an explicit callPackage:

my-package = callPackage ./path/to/my-package {
  stdenv = if stdenv.isDarwin then overrideSDK stdenv darwin.apple_sdk_13 else stdenv;
};

Auto-called inside the derivation:

{ stdenv, darwin, overrideSDK, zlib }:

let
  stdenv' = if stdenv.isDarwin then overrideSDK stdenv darwin.apple_sdk_latest else stdenv;
in
stdenv'.mkDerivation {
  pname = "my-package";
  buildInputs = [ zlib ]
    ++ lib.optionals stdenv'.isDarwin (with stdenv'.apple_sdk.frameworks; [ CoreFoundation ]);
  # etc
}

Which pattern is preferred should follow how different stdenv requirements are handled after the simple package paths RFC is implemented.

[reckenrode]

Regarding SDK evolution, I would like to see the 10.12 SDK retrofitted into the structure being implemented #229210. The source SDK components should be implemented as overlays. I think being able to incrementally convert an SDK will make it easier to move forward on that front. We should also be aiming for an ideal of “source-based headers plus tbd files” for purity. Trying to build Apple’s OSS stuff can be pretty painful, especially newer releases.

For non-SDK applications (e.g., adv_cmds, system_cmds, etc), though should be treated as normal packages and updated to be as new as will build on the default SDK for a platform. When they are common packages that are already maintained in nixpkgs, they should be dropped in favor of the nixpkgs versions (e.g., ICU and libiconv).

[toonn]

Since I agree with most of this, I don't really have much to add. I don't think we need to worry about 10.13, that work is progressing, I will be opening PRs soon.

This might be out of scope but I want to bring it up anyway.

To me the ideal scenario would be packages simply specify what minimum macOS version they support; potentially a maximum too but that's pretty much a non-issue at this point. The Nixpkgs infrastructure would then provide whatever this minimum version is, probably preferring source releases over the SDK distributed by Apple.

Now, while this minimum version information seems appropriate for a package's meta attribute, that would be quite challenging to get at, since it's a derivation attribute and the SDK inputs are a prerequisite to evaluating the derivation.

Maybe darwin should be a function, taking the minimum version as an argument? Inputs would then be specified as something along the lines of inherit (darwin "13.2") Libsystem. Overriding a package's minimum version would be done by essentially passing in _: darwin "14.0".
I wouldn't be surprised if there were better ways of doing this though.

Is it feasible for us to take something like this into account? In the simple case it'd mean maintainers just look up what the minimum supported macOS version is upstream, pass it in however we decide on doing it and be done with it. In the probably more common case it'd mean looking up what version of macOS symbols were introduced in. This could be scripted to reduce the pain though.

> macOSMinimum.sh symbol1 symbol2 ... symbolN
Oldest version containing all symbols: 11.0.1

[reckenrode]

Since I agree with most of this, I don't really have much to add. I don't think we need to worry about 10.13, that work is progressing, I will be opening PRs soon.

Will that introduce a new apple_sdk_10_13 or modify the existing apple_sdk? I ask to illustrate the direction I’d like to see the SDKs go, which is a separate release for each version. apple_sdk is then set to the default SDK for the architecture — similar to LLVM where there are multiple llvmPackages_X but only one llvmPackages that is the default for the platform.

[reckenrode]

This might be out of scope but I want to bring it up anyway.

It seems in scope since it pertains to how the SDK interacts with nixpkgs.

To me the ideal scenario would be packages simply specify what minimum macOS version they support; potentially a maximum too but that's pretty much a non-issue at this point. The Nixpkgs infrastructure would then provide whatever this minimum version is, probably preferring source releases over the SDK distributed by Apple.

Is that the minimum runtime or build requirement?

MoltenVK supports back to 10.11, but it really should be built with the latest SDK. It can’t even be built with the 10.11 (or 10.12) SDKs, and building with pre-latest SDKs negatively impacts feature availability when run on newer macOS releases.

Now, while this minimum version information seems appropriate for a package's meta attribute, that would be quite challenging to get at, since it's a derivation attribute and the SDK inputs are a prerequisite to evaluating the derivation.

Maybe darwin should be a function, taking the minimum version as an argument? Inputs would then be specified as something along the lines of inherit (darwin "13.2") Libsystem. Overriding a package's minimum version would be done by essentially passing in _: darwin "14.0". I wouldn't be surprised if there were better ways of doing this though.

How would it interact with language frameworks and alternate stdenvs? Would the SDK still have to vend e.g., gcc12Stdenv, if you need a non-standard stdenv?

Is it feasible for us to take something like this into account? In the simple case it'd mean maintainers just look up what the minimum supported macOS version is upstream, pass it in however we decide on doing it and be done with it. In the probably more common case it'd mean looking up what version of macOS symbols were introduced in. This could be scripted to reduce the pain though.

> macOSMinimum.sh symbol1 symbol2 ... symbolN
Oldest version containing all symbols: 11.0.1

The script sounds handy regardless. It would also be nice to know if some of those symbols are weakly linked (because it might allow well-behaved applications to be built with a newer SDK with an older deployment target).

[abathur]

x86_64-darwin will continue to maintain an older default than aarch64-darwin. That is currently 10.12 but will need to increase to 10.13 because libc++ 17 has dropped support for 10.12.

Tangent to the main point here (happy to rm if it's a distraction): I guess this implies some thinking/coordination with installers and about what should happen on any existing installs under this floor? (I have no clue if there actually are any left in the wild...)

• The Nix installer uses 10.12.6 as a minimum version. (See installer: update macOS version check to 10.12.2 nix#2594 for the last bump, though I'm not sure why there's a discrepancy between the 10.12.2 in the title and 10.12.6 as implemented.)

• I don't see the same test in the detsys installer. I do see a version test in https://github.com/DeterminateSystems/nix-installer/blob/d076888f8822d30ffe17f51ba6e81714d5c92796/nix-installer.sh#L331-L382, but it looks like it's just related to the help command. They might technically be installing on ~unsupported versions?

  (If so, anyone with access to a macOS this old might be able to use it to test how clear Nix/nixpkgs failure mode(s) are here? I'll ask on Matrix if the detsys folks can shed any light on macOS versions used, though TBH I'd be surprised if pre 10.13 devices are still getting fresh Nix installs even if they're still in service...)

• Depending on the failure modes, maybe something to ensure it's clear or telegraph deprecation? (IDK. Feels silly to waste much time on this if no one's still using it, but a sudden ~EOL on a channel update would suck if the devices are still load-bearing somewhere.)

[reckenrode]

x86_64-darwin will continue to maintain an older default than aarch64-darwin. That is currently 10.12 but will need to increase to 10.13 because libc++ 17 has dropped support for 10.12.

Tangent to the main point here (happy to rm if it's a distraction): I guess this implies some thinking/coordination with installers and about what should happen on any existing installs under this floor? (I have no clue if there actually are any left in the wild...)

No, I think it’s something that needs consideration too. If Darwin is going to regularly add SDKs and update LLVM, we need to determine what the deprecation process looks like for packages and platforms and document it.

• The Nix installer uses 10.12.6 as a minimum version. (See installer: update macOS version check to 10.12.2 nix#2594 for the last bump, though I'm not sure why there's a discrepancy between the 10.12.2 in the title and 10.12.6 as implemented.)
• I don't see the same test in the detsys installer. I do see a version test in https://github.com/DeterminateSystems/nix-installer/blob/d076888f8822d30ffe17f51ba6e81714d5c92796/nix-installer.sh#L331-L382, but it looks like it's just related to the help command. They might technically be installing on ~unsupported versions?

I assume coordinating with the installers would be part of the deprecation process. What channel do they use by default? That would also affect things since the unstable channel would change over before the stable one.

(If so, anyone with access to a macOS this old might be able to use it to test how clear Nix/nixpkgs failure mode(s) are here? I'll ask on Matrix if the detsys folks can shed any light on macOS versions used, though TBH I'd be surprised if pre 10.13 devices are still getting fresh Nix installs even if they're still in service...)

• Depending on the failure modes, maybe something to ensure it's clear or telegraph deprecation? (IDK. Feels silly to waste much time on this if no one's still using it, but a sudden ~EOL on a channel update would suck if the devices are still load-bearing somewhere.)

It would be nice if there were a place in the release notes for Darwin-related changes, especially deprecations and incompatibilities. Given an LLVM 18 bump won’t happen until this time next year for the 24.11 release, an announcement in 23.11 would provide a year’s notice of the upcoming change.

[abathur]

What channel do they use by default?

The official installer uses unstable. I don't understand what exactly the detsys installer does here.

They do cite what they don't do:

• nix-channel --update is not run, ~/.nix-channels is not provisioned

But I haven't gone through it to understand the full implications.

[reckenrode]

What channel do they use by default?

The official installer uses unstable.

That’s what I thought, so we have less time than until the 24.11 release. Can we get a deprecation notice or some changes in the installer to let users know that 10.12 support will be dropping next year?

If necessary we might be able to push it out to 2025 by reverting the commit from libc++ that drops 10.12 support, but we’d also need to make sure it’s not using 10.13 APIs anywhere else (and the LLVM maintainers are willing to carry the patch).

I don't understand what exactly the detsys installer does here.
They do cite what they don't do:

• nix-channel --update is not run, ~/.nix-channels is not provisioned

But I haven't gone through it to understand the full implications.

I assume the lack of channels is because they assume people will be using flakes, but I reached out on Matrix to see if anyone had any experience with it to confirm.

[reckenrode]

Another topic worth discussing is how to handle open source releases that are out of date or no longer available.

I ran into this while testing the clang 16 update. Ruby fails to build with clang 16 on x86_64-darwin because it needs __cospi and __sinpi. These were added in 10.9, so they should be available in the 10.12 SDK, but the math.h header in the 10.12 source-based SDK is dated from 2002 and does not have those definitions. The reason why Ruby detects and tries to use them is because autoconf only checks that functions with those names can be linked.

I’m currently planning to use the 11.0 SDK to build Ruby, but it would be better if there were a way to provide the up-to-date headers for the 10.12 SDK.

[toonn]

I think the plan to overlay source releases on the SDKs kinda solves the outdated/unavailable source problem. If there's no source for something, we get it from the SDK, like XPC for example. Unless an open source reimplementation comes along.

[uri-canva]

I think SDKs and platforms are strongly related, is there some way how we can align the darwin sdks with the concept of nix platforms? For example the platforms in lib/systems/examples.nix have attributes such as sdkVer = "14.3"; xcodeVer = "12.3"; xcodePlatform = "iPhoneSimulator";, but I think they're only used when cross compiling. Can we use the same pattern to configure the regular non-cross compile flows as well?

We might need to represent the distinction between the host and target platform: if I understand the MoltenVK case correctly for example, the host platform needs the latest SDK, but the target platform can be as low as 10.11.

[uri-canva]

Because we don't use source SDKs in all cases in practice we can target macOS without nix from nixpkgs easily at the moment, by using pkgsStatic: #214611.

Is this a use case we want to support in nixpkgs, or something we should defer to external solutions such as https://github.com/DavidEGrayson/nixcrpkgs/?

[reckenrode]

I think SDKs and platforms are strongly related, is there some way how we can align the darwin sdks with the concept of nix platforms? For example the platforms in lib/systems/examples.nix have attributes such as sdkVer = "14.3"; xcodeVer = "12.3"; xcodePlatform = "iPhoneSimulator";, but I think they're only used when cross compiling. Can we use the same pattern to configure the regular non-cross compile flows as well?

Would it be possible to mix packages from different platforms under that scheme?

We might need to represent the distinction between the host and target platform: if I understand the MoltenVK case correctly for example, the host platform needs the latest SDK, but the target platform can be as low as 10.11.

What would that look like in a derivation? Something like this?

let
  stdenv' = stdenv.override {
    targetPlatform = stdenv.targetPlatform // { darwinMinVersion = "13.3"; };
    hostPlatform = stdenv.hostPlatform // { darwinMinVersion = "10.11"; };
  };
in
stdenv.mkDerivation {
 depsBuildTarget = [ /* ??? */ ];
  /* MoltenVK stuff */
}

[reckenrode]

Because we don't use source SDKs in all cases in practice we can target macOS without nix from nixpkgs easily at the moment, by using pkgsStatic: #214611.

The source SDK ought to be able to do that too, but work needs to be done to make it possible.

Is this a use case we want to support in nixpkgs, or something we should defer to external solutions such as https://github.com/DavidEGrayson/nixcrpkgs/?

Ideally, we should not need to defer to external projects. It seems like something we can support and should.

[reckenrode]

#244471 is an example of where slight differences in the source SDK caused an issue. Foundation should reexport libobjc, but it did not. I opened NixOS/darwin-stubs#10 regarding the issue.

If tapi alone isn’t giving us what we need, we may have to supplement it with other tools (e.g., scraping otool -L output for reexports).

[uri-canva]

@reckenrode

Would it be possible to mix packages from different platforms under that scheme?

It should be, the same way we can currently build container images on darwin by using a linux package set. Not sure if that pattern fits the use cases you have in mind but at least it's possible, and we can add helper functions around it to make it more ergonomic.

What would that look like in a derivation? Something like this?

Yeah, something like that. I'm not sure about the specifics, it looks like they're the wrong way around, but maybe if we add darwinVersion in addition to darwinMinVersion it would be clearer. This brings up an interesting question, what version of the sdk should be provided at runtime if the two don't match? The min version / version distinction only really makes sense for macOS without nix, since the package and the runtime sdk are decoupled.

[reckenrode]

It should be, the same way we can currently build container images on darwin by using a linux package set. Not sure if that pattern fits the use cases you have in mind but at least it's possible, and we can add helper functions around it to make it more ergonomic.

Here is an example of what I have in mind: Wine is built with the 10.12 SDK. It depends on MoltenVK, which is built with the 11.0 SDK. MoltenVK is a builtInput. Would that change under this scheme?

Yeah, something like that. I'm not sure about the specifics, it looks like they're the wrong way around, but maybe if we add darwinVersion in addition to darwinMinVersion it would be clearer. This brings up an interesting question, what version of the sdk should be provided at runtime if the two don't match? The min version / version distinction only really makes sense for macOS without nix, since the package and the runtime sdk are decoupled.

As I understand it, the host is what uses the dylibs, which for MoltenVK can be as old as 10.11. The build environment would need to be 13.3. I went with targetPlatform because I wasn’t sure how using buildPlatform would interact with cross-compilation from non-Darwin systems, which is something we’d like to support eventually.

Regarding darwinVersion, that already exists. darwinSdkVersion specifies the SDK version. darwinMinVersion is the deployment target. If MoltenVK didn’t already hardcode the deployment target in its Xcode project, it should be set darwinMinVersion in the stdenv to 10.11 to reflect that it builds dylibs that should run on 10.11 systems.

Currently, not many packages bother to change the minimum version. mongodb-6_0 does (setting it to 10.14), and I have a patch in my queue to have pybind11 set it to 10.13 to silence (arguably bogus) checks clang 16 does when using aligned allocations. It’s currently set on x86_64-darwin to 10.12 to reflect the fact that’s the default SDK and target version.

[uri-canva]

Here is an example of what I have in mind: Wine is built with the 10.12 SDK. It depends on MoltenVK, which is built with the 11.0 SDK. MoltenVK is a builtInput. Would that change under this scheme?

That should be ok if whichever package uses the non-default SDK does so explicitly in the package derivation, like the stdenv' example you posted above.

However what does that mean for a consumer of those packages? Does that mean that changing the default SDK at the package set level would only apply to certain packages? Hopefully packages using a specific non-default SDK would be the exception, and once we have many SDKs, we can have package specify the minimum SDK version, and then they can use either that, or the default SDK version, if the default SDK version is new enough. That would let consumers override both the host and target platforms to be newer than the current nixpkgs default, and have the packages pick it up.

Example of which SDK would be used to build packages:

Package	apple_sdk_10_12	apple_sdk_11_0	apple_sdk_12_0
moltenvk	11.0	11.0	12.0
wine	10.12	11.0	12.0

[reckenrode]

However what does that mean for a consumer of those packages? Does that mean that changing the default SDK at the package set level would only apply to certain packages? Hopefully packages using a specific non-default SDK would be the exception, and once we have many SDKs, we can have package specify the minimum SDK version, and then they can use either that, or the default SDK version, if the default SDK version is new enough. That would let consumers override both the host and target platforms to be newer than the current nixpkgs default, and have the packages pick it up.

I’d like to work through the MoltenVK to make sure I understand how this would work (especially since I do now realize I had it backwards regarding host and target). Assume that requireSdk sets the darwinSdkVersion to at least the requested SDK version but allows higher versions if that is the default.

{ ... }: args elided

let
  stdenv' = stdenv.override (old: {
    hostPlatform = requireSdk "13.3" old.hostPlatform;
  });
  # or even the following, and it takes care of overriding `hostPlatform`
  # stdenv' = requireSdk "13.3" stdenv;
in
stdenv'.mkDerivation (finalAttrs: {
  pname = "MoltenVK";
  version = "1.2.4";
  
  src = fetchFromGitHub { /* -- 8< -- snip  -- 8< -- */ };
  patches = [ /* -- 8< -- snip  -- 8< -- */ ];

  # These are for the target platform (10.11+).
  depsTargetTarget = [ cereal glslang simd spirv-tools vulkan-headers ];
  depsBuildTarget = [ cctools sigtool ];

  # These are for the host platform (latest SDK).
  buildInputs = [ AppKit, Foundation, MacOSX-SDK, Metal, QuartzCore ];
  nativeBuildInputs = [ xcbuild ];

  env.NIX_CFLAGS_COMPILE = toString [
    "-isystem ${lib.getDev pkgsTargetTarget.libcxx}/include/c++/v1"
    "-I${lib.getDev pkgsTargetTarget.spirv-cross}/include/spirv_cross"
    "-I${lib.getDev pkgsTargetTarget.spirv-headers}/include/spirv/unified1/"
  ];

  outputs = [ "out" "bin" "dev" ];

  dontConfigure = true;

  postPatch = '' # -- 8< -- snip  -- 8< --'';

  buildPhase = '' # -- 8< -- snip  -- 8< --'';

  installPhase = '' # -- 8< -- snip  -- 8< --'';

  # This would execute on the build host against the target target.
  postFixup = ''
    install_name_tool -id "$out/lib/libMoltenVK.dylib" "$out/lib/libMoltenVK.dylib"
    codesign -s - -f "$out/lib/libMoltenVK.dylib"
    codesign -s - -f "$bin/bin/MoltenVKShaderConverter"
  '';

  passthru = { /* -- 8< -- snip  -- 8< -- */ };

  meta = { /* -- 8< -- snip  -- 8< -- */ };
}

That’s very similar to the status quo today. The 11.0 SDK stdenv sets targetPlatform to the selected SDK, but that’s not right. It should be hostPlatform. Would the targetPlatform then be responsible for setting the deployment target? The generic stdenv currently uses darwinMinVersion for the hostPlatform.

The other question I have is how those dependencies are determined. Would they be spliced into the derivation? You just use darwin.apple_sdk, and depending on where you use it, you get the required versions?

[bestlem]

I am used to MacPorts I am not certain that the last assumption in your original post can be done.

Macports works and can provide current builds going back to Tiger (10.4) in some cases. The limit tends to be what upstream does.

What they do is for each different version of OSX/macOS they use a different version of Xcode and Apple libraries including clang.
The downside is that you need separate builds for each OS version and probaly separate builder machines for each version ie you build for version X on a version X machine - that is probaly doable with virtual machines now, just requires a lot more compute and storage.

For most software the build instructions are the same independent of the macOS version but they can be changed so that new Apple APIs can be used on newer macOS versions.

In general this makes sense to me as each version of macOS has a different stdenv.

[reckenrode]

I am used to MacPorts I am not certain that the last assumption in your original post can be done.

Why not? That’s how things work on Darwin normally outside of nixpkgs. An application can link a framework regardless of which SDK was used to build it as long as any common dependencies are shared (e.g., they link the same zlib, libxml2, etc).

MoltenVK is my go-to example because it supports 10.11 at runtime even though it cannot be built with older SDKs. It currently builds with the 11.0 SDK, but it really should be built with the 13.3 one. Xcode 11 support was recently dropped from MoltenVK. It’s only a matter of time before it no longer builds with any SDK in nixpkgs (without #229210).

Macports works and can provide current builds going back to Tiger (10.4) in some cases. The limit tends to be what upstream does.

What they do is for each different version of OSX/macOS they use a different version of Xcode and Apple libraries including clang. The downside is that you need separate builds for each OS version and probaly separate builder machines for each version ie you build for version X on a version X machine - that is probaly doable with virtual machines now, just requires a lot more compute and storage.

I like MacPorts, but that approach isn’t feasible for nixpkgs. There already are not enough Darwin builders, and many Linux maintainers have no access to Darwin hardware. If they had to make sure packages built across a matrix of platforms, I fear even fewer would bother. The pool of active Darwin maintainers is also pretty small. I expect it’d be a recipe for bitrot. 😕

In general this makes sense to me as each version of macOS has a different stdenv.

The only things used from the system are the libc and system frameworks (because there is no other way). The system libc++ is not used nor is Apple’s toolchain (except¹ for cctools and ld64, which are built from source). nixpkgs builds LLVM from upstream (not Apple’s fork but upstream LLVM) and uses that with the SDK headers and stubs to build packages.

Theoretically, this setup should allow cross-compilation to Darwin. It’s not there yet. Currently, only x86_64-darwin to aarch64-darwin works (and static compilation on aarch64-darwin).

---

¹ I started replacing cctools with equivalents from LLVM as part of the Darwin stdenv rework. Eventually, it might be possible to drop Apple’s tools and make Darwin a useLLVM platform. We’re not there yet. I’d like to see if it lld 18 can be used to link the stdenv when it’s bumped against next year. I tried it with lld 16, but it fails linking the x86_64-darwin stdenv.

[Atemu]

The 11.0 SDK stdenv sets targetPlatform to the selected SDK, but that’s not right. It should be hostPlatform. Would the targetPlatform then be responsible for setting the deployment target?

I don't understand why target and host would ever be different here. That should only be relevant for things like building the cross-compiler itself from Darwin to some other platform and that's not a thing yet. Am I missing something?

---

I really like the idea of making the SDK part of the platform; it feels like the obvious place to put it. Overriding it should then only be a matter of overriding stdenv.hostPlatform.

As for dependencies which propagate a specific SDK version (i.e. libGLU), perhaps stdenv.mkDerivation could automatically set the hostPlatform's SDK to that propagated SDK version (throwing an error if there are multiple different required SDK versions).

[reckenrode]

I don't understand why target and host would ever be different here. That should only be relevant for things like building the cross-compiler itself from Darwin to some other platform and that's not a thing yet. Am I missing something?

The SDK used to build MoltenVK is not necessarily the same as the one used at runtime. It can be newer than the host system. That’s why I assume targetPlatform would be different because buildPlatform could be something like Linux¹. This is assuming the cross-based approach that was proposed. I’m not sure that’s a good idea because that’s not how deployment target in build systems normally. It’s just a variable you set. I’m wary of the extra complexity.

I really like the idea of making the SDK part of the platform; it feels like the obvious place to put it. Overriding it should then only be a matter of overriding stdenv.hostPlatform.

The way it works currently is the SDK version is specified by stdenv.hostPlatform.darwinSdkVersion. The problem is this has absolutely nothing to do with what you get from darwin.apple_sdk. Is it possible to splice the required SDK version when the derivation is evaluated (and is that a good idea), or should the SDK be accessed through an associated attribute like stdenv.hostPlatform.apple_sdk?

---

¹: Given that pkgsStatic and x86_64-darwin to aarch64-darwin cross both work now, it seems like only a matter of time before Linux to Darwin cross is also possible. I don’t even want to look at that before the stdenv is updated nor before some other housekeeping is done on the Darwin side (improving the SDK situation generally, etc).

[Atemu]

The SDK used to build MoltenVK is not necessarily the same as the one used at runtime. It can be newer than the host system.

I had assumed we explicitly link it against our frameworks packaged in Nix but, looking at the outputs of otool -L, apparently the frameworks are impure; linked against /System/Library/Frameworks/*.framework/?! Is it supposed to be like that?

Host platform wouldn't mean much for frameworks then; it'd be more of an intermediate between build and host platform.

That’s why I assume targetPlatform would be different because buildPlatform could be something like Linux1.

Build and target/host platform might be different at some point, yes but I don't see why host and target platform would be different except for special cases like building a x86_64-darwin to aarch64-darwin compiler and the likes.

The problem is this has absolutely nothing to do with what you get from darwin.apple_sdk. Is it possible to splice the required SDK version when the derivation is evaluated (and is that a good idea), or should the SDK be accessed through an associated attribute like stdenv.hostPlatform.apple_sdk?

Doing that somehow doesn't feel right to me.

Perhaps the stdenv could do some magic here; taking the correct SDK out of darwin based on hostPlatform.darwinSdkVersion.

[reckenrode]

I had assumed we explicitly link it against our frameworks packaged in Nix but, looking at the outputs of otool -L, apparently the frameworks are impure; linked against /System/Library/Frameworks/*.framework/?! Is it supposed to be like that?

Yes. We link against stub frameworks, which point to the system frameworks but restrict the symbols to those available in that particular SDK version. Even the source-based SDKs still link against stubs for system frameworks other than for a handful that can be built from source.

Host platform wouldn't mean much for frameworks then; it'd be more of an intermediate between build and host platform.

Build and target/host platform might be different at some point, yes but I don't see why host and target platform would be different except for special cases like building a x86_64-darwin to aarch64-darwin compiler and the likes.

That sounds more or less like the current situation with darwinSdkVersion, which I think is fine. I’ve been trying to steelman the cross-based approach suggested to suss out the details though I’m skeptical of it.

Doing that somehow doesn't feel right to me.

The splicing or the attribute?

Perhaps the stdenv could do some magic here; taking the correct SDK out of darwin based on hostPlatform.darwinSdkVersion.

How would that look in a derivation? Would they continue to use darwin.apple_sdk or some other mechanism to access the required frameworks?

[reckenrode]

To provide an example of what I mean, ideally the callPackage for MoltenVK would change from:

  moltenvk = pkgs.darwin.apple_sdk_11_0.callPackage ../os-specific/darwin/moltenvk {
    inherit (apple_sdk_11_0.frameworks) AppKit Foundation Metal QuartzCore;
    inherit (apple_sdk_11_0) MacOSX-SDK Libsystem;
    inherit (pkgs.darwin) cctools sigtool;
  };

To something like this:

  moltenvk = callPackage ../os-specific/darwin/moltenvk {
    stdenv = overrideAppleSDK stdenv apple_sdk_latest;
  };

Inside the derivation, it would access the frameworks in some way. That pattern is what needs to be established. Is that darwin.apple_sdk which maps to the latest one via some magic? Is that stdenv.apple_sdk? Some other approach?

[reckenrode]

I’m adding overrideSDK as a stdenv adapter as another way of changing the SDK version used to build a package.

stdenvAdapters: add overrideSDK

I ended up having to do something because curl needs to export the frameworks it links, and too many things (like Nix) link against libcurl.4.dylib to force them to manually propagate the required frameworks, and ignoring the breakage was unlikely to be okay due to the visibility of curl and its history of Darwin difficulties (compared to libGLU).

The overrideSDK adapter is based on the things discussed here. It uses the same approach as the SDK-specific stdenv to override the stdenv, but it also modifies the derivation’s build inputs to use the frameworks from the requested SDK. This modification also extends to any propagated build inputs (allowing packages to use a different SDK and curl).

Packages that want to use the adapter should just use the unversioned SDK frameworks. Once there are more SDKs with newer frameworks not in the base SDK, we can start to figure out just how to handle that situation. In theory, if you use a newer SDK’s frameworks, it should also replace those with the requested one, but it might be confusing to use both the adapter and an SDK-specific framework attribute.

I also wanted to thank everyone who participated in this discussion since it helped inform the approach taken in #263598.

[reckenrode]

I mentioned this on Matrix, but I wanted to document it here. These are the known limitations of overrideSDK.

• It doesn’t support all dependency attributes (only common ones like buildInputs, nativeBuildInputs),
• It doesn’t check other files for propagated inputs (e.g., the CMake file in abseill-cpp propagating CF);
• It needs more special casing added source-based components that have different names;
• It doesn’t support private frameworks (which have a different name and fail the heuristic); and
• It doesn’t support non-framework SDK components like libs (and it needs to update apple_sdk.version).

[uri-canva]

It doesn’t support non-framework SDK components like libs.

This sounds like something we could fix, since the dylibs in the SDK are known? Or am I misunderstanding?

[reckenrode]

It doesn’t support non-framework SDK components like libs.

This sounds like something we could fix, since the dylibs in the SDK are known? Or am I misunderstanding?

It can be fixed. The issue is the SDKs expose different attributes under libs. Those need cleaned up and harmonized, but that’s not needed for staging-next, so it’s deferred until after the release and #229210.