Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

swift: 5.8 -> 5.9.2 #256956

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

swift: 5.8 -> 5.9.2 #256956

wants to merge 4 commits into from

Conversation

stephank
Copy link
Contributor

@stephank stephank commented Sep 23, 2023

Description of changes

https://www.swift.org/blog/swift-5.9-released/

Incomplete. Opening a draft PR to indicate I'm working on it, and share progress.

Current state:

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@mattpolzin
Copy link
Contributor

I have not had time to look at this at all yet, but figured I would jot down a note that I hit the following error building on x86_64-darwin:

error: builder for '/nix/store/jd7bhp58khf973rj2xagvny6zwv434hn-swift-5.9.drv' failed with exit code 1;
       last 10 log lines:
       > In file included from /private/tmp/nix-build-swift-5.9.drv-0/src/swift/include/swift/Frontend/Frontend.h:22:
       > In file included from /private/tmp/nix-build-swift-5.9.drv-0/src/swift/include/swift/AST/DiagnosticEngine.h:21:
       > In file included from /private/tmp/nix-build-swift-5.9.drv-0/src/swift/include/swift/AST/ActorIsolation.h:19:
       > In file included from /private/tmp/nix-build-swift-5.9.drv-0/src/swift/include/swift/AST/Type.h:23:
       > In file included from /private/tmp/nix-build-swift-5.9.drv-0/src/swift/include/swift/AST/LayoutConstraint.h:22:
       > /private/tmp/nix-build-swift-5.9.drv-0/src/swift/include/swift/AST/PrintOptions.h:585:33: warning: '\c' command has no word arguments, expected 1 [-Wdocumentation]
       >   /// Whether or not to print \c @attached(extension) attributes on
       >                               ~~^
       > 1 warning generated.
       > ninja: build stopped: subcommand failed.
       For full logs, run 'nix-store -l /nix/store/jd7bhp58khf973rj2xagvny6zwv434hn-swift-5.9.drv'.
error: 1 dependencies of derivation '/nix/store/jrzlc42q94ckg1bf02j5zqppyddziqg8-swift-wrapper-5.9.drv' failed to build

@stephank
Copy link
Contributor Author

Updated this to Swift 5.9.2.

Whatever issue I was having that was causing the compiler to segfault seems to have been fixed between 3.9 and 3.9.2.

I have not had time to look at this at all yet, but figured I would jot down a note that I hit the following error building on x86_64-darwin:

This cuts off the actual error, but I got a similar end-of-log on Linux which was caused by the wrapper changes in this PR. That should be fixed now.

Haven't yet tested Darwin again, though.

Now fighting a weird issue in swift-format:

error: 'swift-argument-parser': Invalid manifest (compiled with: ["/nix/store/w9rwhbbrckjv1xhha5f278b782py2cj6-swift-wrapper-5.9.2/bin/swiftc", "-vfsoverlay", "/build/TemporaryDirectory.HCMalP/vfs.yaml", "-L", "/nix/store/az7zdncpcpj1hzfrgv4yzhyfsgd479z9-swiftpm-5.9.2/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/nix/store/az7zdncpcpj1hzfrgv4yzhyfsgd479z9-swiftpm-5.9.2/lib/swift/pm/ManifestAPI", "-swift-version", "5", "-I", "/nix/store/az7zdncpcpj1hzfrgv4yzhyfsgd479z9-swiftpm-5.9.2/lib/swift/pm/ManifestAPI", "-package-description-version", "5.8.0", "/build/source/.build/checkouts/swift-argument-parser/[email protected]", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/build/TemporaryDirectory.GSzyO9/swift-argument-parser-manifest"])
/build/TemporaryDirectory.GSzyO9/swift-argument-parser-manifest: error while loading shared libraries: libdispatch.so: cannot open shared object file: No such file or directory

I wonder if this is specific to swift-argument-parser 1.3.0. It's used by other components in the toolchain, but they're still on older versions.

@stephank stephank changed the title swift: 5.8 -> 5.9 swift: 5.8 -> 5.9.2 Jan 14, 2024
@mattpolzin
Copy link
Contributor

mattpolzin commented Jan 14, 2024

The only error I hit now when building the Swift 5.9.2 derivation on x86_64-darwin is

[1275/1387] Compiling /tmp/nix-build-swift-5.9.2.drv-0/build/swift/stdlib/public/libexec/swift-backtrace//OSX/x86_64/swift_backtrace_macosx_x86_64.o
FAILED: stdlib/public/libexec/swift-backtrace/OSX/x86_64/swift_backtrace_macosx_x86_64.o /tmp/nix-build-swift-5.9.2.drv-0/build/swift/stdlib/public/libexec/swift-backtrace/OSX/x86_64/swift_>
cd /tmp/nix-build-swift-5.9.2.drv-0/build/swift/stdlib/public/libexec/swift-backtrace && /nix/store/101zq7jbr3w3q0v2f1xyz9y6d488f4lv-cmake-3.27.8/bin/cmake -E make_directory /tmp/nix-build->
<unknown>:0: warning: -link-objc-runtime is no longer supported on Apple platforms
<unknown>:0: warning: overriding '-mmacos-version-min=10.15' option with '-target x86_64-apple-macosx10.13'
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/libexec/swift-backtrace/main.swift:141:19: error: cannot find 'CLOCK_MONOTONIC' in scope
    clock_gettime(CLOCK_MONOTONIC, &startTime)
                  ^~~~~~~~~~~~~~~
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/libexec/swift-backtrace/main.swift:143:19: error: cannot find 'CLOCK_MONOTONIC' in scope
    clock_gettime(CLOCK_MONOTONIC, &endTime)
                  ^~~~~~~~~~~~~~~

@stephank
Copy link
Contributor Author

Fixed swift-format by downgrading the swift-argument-parser dependency to 1.2.3 for now.

I'm not sure why Darwin fails like that. CLOCK_MONOTONIC should be available in >=10.12. Hoping I'll be able to debug sometime soon.

@stephank
Copy link
Contributor Author

#282851 should fix the CLOCK_MONOTONIC issue, but has to go through staging.

Next error: (aarch64-darwin)

FAILED: stdlib/public/BackDeployConcurrency/CMakeFiles/swift_Concurrency-macosx-arm64.dir/TaskStatus.cpp.o
/private/tmp/nix-build-swift-5.9.2.drv-0/build/llvm/bin/clang++ -DGTEST_HAS_RTTI=0 -DSWIFT_CONCURRENCY_BACK_DEPLOYMENT -DSWIFT_INLINE_NAMESPACE=__runtime -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Dswift_Concurre
ncy_macosx_arm64_EXPORTS -I/tmp/nix-build-swift-5.9.2.drv-0/build/swift-concurrency-backdeploy/stdlib/public/BackDeployConcurrency -I/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency -I/private/tmp/nix-build-swi
ft-5.9.2.drv-0/src/swift/stdlib/include -I/private/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/SwiftShims -I/tmp/nix-build-swift-5.9.2.drv-0/build/swift-concurrency-backdeploy/include -I/private/tmp/nix-build-swift-5.9.2.drv-0/s
rc/swift/include -I/tmp/nix-build-swift-5.9.2.drv-0/src/llvm-project/llvm/include -I/tmp/nix-build-swift-5.9.2.drv-0/build/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wn
o-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-
indentation -Wctad-maybe-unsupported -fdiagnostics-color -Werror=switch -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -O3 -DNDEBUG -std=c++17 -arch arm64 -isysroot /nix/store
/azbzc5yrfnsf40jf5rx95xnhw27vzl96-SDKs/MacOSX10.15.sdk -fPIC -Werror=gnu -Werror=c++98-compat-extra-semi -Dswift_Concurrency_EXPORTS -fswift-async-fp=never -D__STDC_WANT_LIB_EXT1__=1 -I/private/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdl
ib/include -DSWIFT_TARGET_LIBRARY_NAME=swift_Concurrency -DSWIFT_RUNTIME -DSWIFT_LIB_SUBDIR=\"macosx\" -DSWIFT_ARCH=\"arm64\" -target arm64-apple-macosx10.15 -isysroot /nix/store/azbzc5yrfnsf40jf5rx95xnhw27vzl96-SDKs/MacOSX10.15.sdk -F/nix/
store/azbzc5yrfnsf40jf5rx95xnhw27vzl96-SDKs/MacOSX10.15.sdk/../../../Developer/Library/Frameworks -O2 -g0 -DNDEBUG -DSWIFT_LIBRARY_EVOLUTION=1 -DSWIFT_STDLIB_SUPPORT_BACK_DEPLOYMENT -DSWIFT_ENABLE_REFLECTION -DSWIFT_STDLIB_HAS_DLADDR -DSWIF
T_STDLIB_HAS_DLSYM=1 -DSWIFT_STDLIB_HAS_FILESYSTEM -DSWIFT_STDLIB_HAS_DARWIN_LIBMALLOC=1 -DSWIFT_STDLIB_HAS_ASL -DSWIFT_STDLIB_HAS_STDIN -DSWIFT_STDLIB_HAS_ENVIRON -DSWIFT_STDLIB_HAS_LOCALE -DSWIFT_THREADING_DARWIN -DSWIFT_RUNTIME_OS_VERSIO
NING -DSWIFT_STDLIB_SHORT_MANGLING_LOOKUPS -DSWIFT_STDLIB_ENABLE_VECTOR_TYPES -DSWIFT_STDLIB_HAS_TYPE_PRINTING -DSWIFT_STDLIB_SUPPORTS_BACKTRACE_REPORTING -DSWIFT_STDLIB_ENABLE_UNICODE_DATA -DSWIFT_STDLIB_CONCURRENCY_TRACING  -fno-exception
s -fno-rtti -MD -MT stdlib/public/BackDeployConcurrency/CMakeFiles/swift_Concurrency-macosx-arm64.dir/TaskStatus.cpp.o -MF stdlib/public/BackDeployConcurrency/CMakeFiles/swift_Concurrency-macosx-arm64.dir/TaskStatus.cpp.o.d -o stdlib/public
/BackDeployConcurrency/CMakeFiles/swift_Concurrency-macosx-arm64.dir/TaskStatus.cpp.o -c /tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.cpp
In file included from /tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.cpp:18:
In file included from /tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/CompatibilityOverride.h:21:
In file included from /tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/ConcurrencyRuntime.h:21:
In file included from /tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskGroup.h:26:
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.h:102:26: error: no member named 'Deadline' in 'swift::TaskStatusRecordKind'; did you mean simply 'Deadline'?
      : TaskStatusRecord(TaskStatusRecordKind::Deadline), Deadline(deadline) {}
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                         Deadline
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.h:98:16: note: 'Deadline' declared here
  TaskDeadline Deadline;
               ^
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.h:102:9: error: no matching constructor for initialization of 'TaskStatusRecord'
      : TaskStatusRecord(TaskStatusRecordKind::Deadline), Deadline(deadline) {}
        ^                                      ~~~~~~~~
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.h:45:3: note: candidate constructor not viable: no known conversion from 'TaskDeadline' to 'TaskStatusRecordKind' for 1st argument
  TaskStatusRecord(TaskStatusRecordKind kind,
  ^
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.h:51:3: note: candidate constructor not viable: no known conversion from 'TaskDeadline' to 'const TaskStatusRecord' for 1st argument
  TaskStatusRecord(const TaskStatusRecord &) = delete;
  ^
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.h:107:55: error: no member named 'Deadline' in 'swift::TaskStatusRecordKind'
    return record->getKind() == TaskStatusRecordKind::Deadline;
                                ~~~~~~~~~~~~~~~~~~~~~~^
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.cpp:448:30: error: no member named 'Deadline' in 'swift::TaskStatusRecordKind'
  case TaskStatusRecordKind::Deadline:
       ~~~~~~~~~~~~~~~~~~~~~~^
/tmp/nix-build-swift-5.9.2.drv-0/src/swift/stdlib/public/BackDeployConcurrency/TaskStatus.cpp:532:30: error: no member named 'Deadline' in 'swift::TaskStatusRecordKind'
  case TaskStatusRecordKind::Deadline:
       ~~~~~~~~~~~~~~~~~~~~~~^
5 errors generated.

@stephank
Copy link
Contributor Author

stephank commented Feb 24, 2024

I think we have to remove Concurrency backdeploy, but that's unfortunate. It doesn't build with Swift 5.9 anymore, and sources were removed in the Swift 5.10 branch: swiftlang/swift@e54b183

Because of this, I'm not sure if Swift code that uses Concurrency can be built to target macOS <12 using our toolchain. I don't see a way to remedy this. If I understand correctly, Xcode relies on old binaries to accomplish this.


Without Concurrency backdeploy, the build now completes (aarch64-darwin), but installation fails:

CMake Error at stdlib/public/libexec/swift-backtrace/cmake_install.cmake:41 (file):
  file INSTALL cannot find
  "/tmp/nix-build-swift-5.9.2.drv-0/build/swift/./libexec/swift/macosx/swift-backtrace":
  No such file or directory.
Call Stack (most recent call first):
  stdlib/public/libexec/cmake_install.cmake:42 (include)
  stdlib/public/cmake_install.cmake:142 (include)
  stdlib/cmake_install.cmake:47 (include)
  cmake_install.cmake:42 (include)


FAILED: CMakeFiles/install.util 
cd /tmp/nix-build-swift-5.9.2.drv-0/build/swift && /nix/store/0i350l1k6a0nh6a46kasawki08qsz081-cmake-3.27.9/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

I've been trying to debug this a number of times already, but the CMake is dense here. I don't understand why this fails.

We no longer need the swift-driver patch to handle bash fds.
This doesn't build with Swift 5.9 anymore, and sources were removed in
the Swift 5.10 branch:
swiftlang/swift@e54b183

Because of this, I'm not sure if Swift code that uses Concurrency can be
built to target macOS <12 using our toolchain. That's very unfortunate,
but I don't see a way to remedy this. If I understand correctly, Xcode
relies on old binaries to accomplish this.
@mattpolzin
Copy link
Contributor

mattpolzin commented Mar 30, 2024

I was not able to figure out why the build path for swift-backtrace in the cmake_install.cmake file was off in the first place, but I do seem to have gotten past the problem (though my confidence in this being a solution rather than a smoke screen is not high) with the following patch:

diff --git a/pkgs/development/compilers/swift/compiler/default.nix b/pkgs/development/compilers/swift/compiler/default.nix
index 38c40106388b..c56edcbbeaef 100644
--- a/pkgs/development/compilers/swift/compiler/default.nix
+++ b/pkgs/development/compilers/swift/compiler/default.nix
@@ -519,6 +519,9 @@ in stdenv.mkDerivation {
     mv llvm/bin/clang-16{-unwrapped,}
     mv swift/bin/swift-frontend{-unwrapped,}
 
+    substituteInPlace swift/stdlib/public/libexec/swift-backtrace/cmake_install.cmake \
+      --replace "./libexec/swift/macosx/swift-backtrace" "stdlib/public/libexec/swift-backtrace/OSX/x86_64"
+
     mkdir $out $lib
 
     # Install clang binaries only. We hide these with the wrapper, so they are

This patch is clearly specific to my macos x86_64 system, but it could easily be generalized if it was even the right path to take. I more just wanted to post this finding in case it helped unblock others. I am going to keep trying to get farther in the meantime (I did hit another failure later on in the process, but nothing that looks too mysterious yet).

[EDIT] the next issue was missing the CryptoKit library. I tried simply adding it to build inputs which did not fix the error and that's when I ran out of time for now.

@dvtkrlbs
Copy link

dvtkrlbs commented May 26, 2024

@mattpolzin you fix seems to work on arm64 as well. I am now trying to tackle the missing cryptokit issue. Also is there way to reuse the cache when changing the nix file. Waiting for rebuilds everytime on my m1 is annoying

[EDIT] looking further it feels like we also need to package CryptoKit. just to be sure i will try a linux build as well. if this issue doesnt happen on linux it might be about the apple sdk

@matteo-pacini matteo-pacini mentioned this pull request Jun 10, 2024
13 tasks
@reckenrode
Copy link
Contributor

If it makes things easier re: CryptoKit, the 12.3 SDK was added in #326406.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants