Rewrite the build system with CMake

This commit is an attempt to provide a concrete path forward on
WebAssembly#425. I personally think it's pretty important to
get the ability to have more architectures here but at the same time I
also think it's important to to take this as an opportunity to refactor
and improve the build system of this repository. To that end this
represents my attempt to improve the status quo.

This removes the old `Makefile` and replaces it with a CMake-based
system to build all these projects. Overall this is intended to be a "no
functional change" intended sort of refactoring. Changing build systems
inevitably causes issues, however, so this change additionally has a
very high likelihood of needing follow-up fixes. At a high enough level
this commit introduces two major changes to how this repository is
built:

1. The `make`-based system (the root `Makefile`) is replaced with CMake.
   This additionally updates tests to use CMake.
2. A single "build" is split into either building a toolchain or
   building a sysroot. This enables builds to only build one or the
   other as necessary.

The first change, using CMake, is due to the fact that using `make` on
Windows basically is not pleasant coupled with the fact that more
advanced logic, such as changing flags, compilers, etc, is much easier
with a CMake-based system. The second change is intended to cover the
use case of WebAssembly#425 in addition to refactoring the current build.

Throughout this change I have intentionally not tried to keep a 1:1
correspondance with behaviors in the old `Makefile` because much of this
PR is intended to address shortcomings in the old build system. A list
of changes, improvements, etc, made here are:

* CMake provides a much nicer portability story to Windows than `make`.
  This is moving towards the direction of not needing `bash`, for
  example, to build an SDK. Currently `wasi-libc` still requires this,
  but that's now the only "hard" dependency.

* The set of targets built can now be configured for smaller builds
  and/or debugging just a single target. All WASI targets are still
  built by default but it's much easier to add/remove them.

* Different targets are now able to be built in parallel as opposed to
  the unconditional serial-nature of the `Makefile`.

* Use of `ninja` is no longer required and separate build systems can be
  used if desired.

* The sysroot and the toolchain can now be built with different CMake
  build profiles. For example the `Makefile` hardcoded `MinSizeRel` and
  `RelWithDebInfo` and this can now be much more easily customized by
  the SDK builder.

* Tarballs are now more consistently produced and named. For a tarball
  of the name `foo.tar.gz` it's guaranteed that there's a single folder
  `foo` created when unpacking the tarball.

* The macOS binaries are no longer hybrid x64/arm64 binaries which
  greatly inflates the size of the SDK. There's now a separate build for
  each architecture.

* CI now produces arm64-linux binaries. The sysroot is not built on the
  arm64-linux builder and the sysroot from the x86_64-linux builder is
  used instead.

* Windows now executes tests in CI.

* Tests are now integrated into CMake. This means that the wasm binaries
  are able to be built in parallel and the tests are additionally
  executed in parallel with `ctest`. It is possible to build/run a
  single test. Tests no longer place all of their output in the source
  tree.

* Out-of-tree builds are now possible and the build/installation
  directories can both be customized.

* CI configuration of Windows/macOS/Linux is much more uniform by having
  everything in one build matrix instead of separate matrices.

* Linux builds are exclusively done in docker containers in CI now. CI
  no longer produces two Linux builds only for one to be discarded when
  artifacts are published.

* Windows 32-bit builds are no longer produced in CI since it's expected
  that everyone actually wants the 64-bit ones instead.

* Use of `ccache` is now automatically enabled if it's detected on the
  system.

* Many preexisting shell scripts are now translated to CMake one way or
  another.

* There's no longer a separate build script for how to build wasi-sdk in
  docker and outside of docker which needs to be kept in sync,
  everything funnels through the same script.

.github/workflows/main.yml

@@ -6,42 +6,39 @@ on:
  push:
  branches:
  - main
+ - cmake
  pull_request:
  workflow_dispatch:
 
 jobs:
  build:
- name: Native Build
+ name: Build ${{ matrix.artifact }}
  runs-on: ${{ matrix.os }}
  strategy:
  fail-fast: false
  matrix:
- os:
- - ubuntu-latest
- - macos-latest
+ include:
+ - artifact: x86_64-linux
+ os: ubuntu-latest
+
+ - artifact: arm64-linux
+ os: ubuntu-latest
+ rust_target: aarch64-unknown-linux-gnu
+
+ - artifact: arm64-macos
+ os: macos-latest
+ llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=arm64
+ rust_target: aarch64-apple-darwin
+
+ - artifact: x86_64-macos
+ os: macos-latest
+ llvm_cmake_flags: -DCMAKE_OSX_DEPLOYMENT_TARGET=10.12 -DCMAKE_OSX_ARCHITECTURES=x86_64
+ rust_target: x86_64-apple-darwin
+ skip_sysroot: 1
+
+ - artifact: x86_64-windows
+ os: windows-latest
  steps:
- - uses: actions/cache@v4
- with:
- path: ~/.cache/ccache
- # Bump the prefix number to evict all previous caches and
- # enforce a clean build, in the unlikely case that some
- # weird build error occur and ccache becomes a potential
- # suspect.
- key: 0-cache-ubuntu-latest-${{ github.run_id }}
- restore-keys: |
- 0-cache-ubuntu-latest
- if: matrix.os == 'ubuntu-latest'
- - uses: actions/cache@v4
- with:
- path: ~/Library/Caches/ccache
- key: 0-cache-macos-latest-${{ github.run_id }}
- restore-keys: |
- 0-cache-macos-latest
- if: matrix.os == 'macos-latest'
- - name: Setup `wasmtime` for tests
- uses: bytecodealliance/actions/wasmtime/setup@v1
- with:
- version: "18.0.2"
  - uses: actions/checkout@v4
  with:
  fetch-depth: 0
@@ -53,136 +50,165 @@ jobs:
  # Server does not allow request for unadvertised object" in the
  # future.
  - run: git submodule update --init --depth 32 --jobs 3
- - name: Install ccache, ninja (macOS)
- run: brew install ccache ninja
- if: matrix.os == 'macos-latest'
- - name: Install ccache, ninja (Linux)
- run: sudo apt install ccache ninja-build
- if: matrix.os == 'ubuntu-latest'
- - name: Build
- run: NINJA_FLAGS=-v make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON
- shell: bash
- - name: Run the testsuite
- run: NINJA_FLAGS=-v make check RUNTIME=wasmtime
- - name: Upload artifacts
- uses: actions/upload-artifact@v4
- with:
- # Upload the dist folder. Give it a name according to the OS it was built for.
- name: ${{ format( 'dist-{0}', matrix.os) }}
- path: dist
 
- winbuild:
- name: Windows Build
- runs-on: windows-latest
- strategy:
- fail-fast: false
- matrix:
- include:
- - arch: x64
- sys: clang64
- env: clang-x86_64
- - arch: x86
- sys: clang32
- env: clang-i686
- steps:
+ # Persist ccache-based caches across builds. This directory is configured
+ # via the CCACHE_DIR env var below for ccache to use.
+ #
+ # Bump the prefix number to evict all previous caches and enforce a clean
+ # build, in the unlikely case that some weird build error occur and ccache
+ # becomes a potential suspect.
  - uses: actions/cache@v4
+ id: cache-restore
  with:
- path: ~/AppData/Local/ccache
- key: 0-${{ format( 'cache-windows-latest-{0}', matrix.arch) }}-${{ github.run_id }}
+ path: ${{ runner.tool_cache }}/ccache
+ key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }}
  restore-keys: |
- 0-${{ format( 'cache-windows-latest-{0}', matrix.arch) }}
- - uses: msys2/setup-msys2@v2
- with:
- install: >-
- base-devel
- git
- mingw-w64-${{ matrix.env }}-ccache
- mingw-w64-${{ matrix.env }}-cmake
- mingw-w64-${{ matrix.env }}-ninja
- mingw-w64-${{ matrix.env }}-toolchain
- msystem: ${{ matrix.sys }}
- update: true
- release: false
- path-type: inherit
- - uses: actions/checkout@v4
+ 0-cache-${{ matrix.artifact }}-
+ - run: |
+ mkdir -p '${{ runner.tool_cache }}/ccache'
+ echo 'CCACHE_DIR=${{ runner.tool_cache }}/ccache' >> $GITHUB_ENV
+ shell: bash
+
+ # Configure CMake flags for `ci/build.sh` as necessary for each
+ # matrix entry.
+ - run: echo WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS=${{ matrix.llvm_cmake_flags }} >> $GITHUB_ENV
+ if: matrix.llvm_cmake_flags != ''
+ shell: bash
+ - run: |
+ cmake_args=-DWASI_SDK_ARTIFACT=${{ matrix.artifact }}
+ if [ "${{ matrix.rust_target }}" != "" ]; then
+ rustup target add ${{ matrix.rust_target }}
+ cmake_args="$cmake_args -DRUST_TARGET=${{ matrix.rust_target }}"
+ fi
+ echo WASI_SDK_CI_TOOLCHAIN_CMAKE_ARGS="$cmake_args" >> $GITHUB_ENV
+ shell: bash
+ - run: echo WASI_SDK_CI_SKIP_SYSROOT=1 >> $GITHUB_ENV
+ if: matrix.skip_sysroot != ''
+
+ # Add some extra installed software on each runner as necessary.
+ - name: Setup `wasmtime` for tests
+ uses: bytecodealliance/actions/wasmtime/setup@v1
  with:
- fetch-depth: 0
- - run: git fetch --tags --force
- name: Force-fetch tags to work around actions/checkout#290
- - run: git submodule update --init --depth 32 --jobs 3
- - name: Build
- shell: msys2 {0}
- run: |
- make package LLVM_CMAKE_FLAGS=-DLLVM_CCACHE_BUILD=ON
- make check
- - name: Does it work sans msys2?
- run: |
- C:\wasi-sdk\bin\clang.exe --version
- C:\wasi-sdk\bin\llvm-ar.exe --version
- C:\wasi-sdk\bin\wasm-ld.exe --version
+ version: "18.0.2"
+ - name: Install ccache, ninja (macOS)
+ run: brew install ccache ninja
+ if: runner.os == 'macOS'
+ - name: Install ccache, ninja (Windows)
+ run: choco install ccache ninja
+ if: runner.os == 'Windows'
+ - name: Install ccache, ninja (Linux)
+ run: sudo apt install ccache
+ if: runner.os == 'Linux'
+
+ - name: Build and test (macOS)
+ run: ./ci/build.sh
+ if: runner.os == 'macOS'
+
+ - name: Build and test (Linux)
+ run: ./ci/docker-build.sh ${{ matrix.artifact }}
+ if: runner.os == 'Linux'
+
+ # Use a shorter build directory than the default on Windows to avoid
+ # hitting path length and command line length limits.
+ - name: Build and test (Windows)
+ run: ./ci/build.sh C:/wasi-sdk
+ if: runner.os == 'Windows'
+
+ # Upload the `dist` folder from the build as the artifacts for this
+ # runner.
  - name: Upload artifacts
  uses: actions/upload-artifact@v4
  with:
- # Upload the dist folder. Give it a name according to the OS it was built for.
- name: ${{ format( 'dist-windows-latest-{0}', matrix.arch) }}
- path: dist
+ name: ${{ format( 'dist-{0}', matrix.artifact) }}
+ path: build/dist
 
- dockerbuild:
- name: Docker Build
- runs-on: ubuntu-latest
- steps:
- - uses: actions/cache@v4
+ # Help debug ccache issues by showing what happened.
+ - if: always()
+ name: Show ccache statistics
+ run: ccache --show-stats
+
+ # Always save a cache, even if the build failed. This ensures that if
+ # live-debugging via CI the build gets to pick up where it left off last
+ # time instead of having to recreate everything each time a failure
+ # happens.
+ - if: always() && steps.cache-restore.outputs.cache-hit != 'true'
+ uses: actions/cache/save@v4
  with:
- path: ~/.ccache
- key: 0-cache-ubuntu-bionic-${{ github.run_id }}
- restore-keys: |
- 0-cache-ubuntu-bionic
+ path: ${{ runner.tool_cache }}/ccache
+ key: 0-cache-${{ matrix.artifact }}-${{ github.run_id }}
 
+ # Once all of the above matrix entries have completed this job will run and
+ # assemble the final `wasi-sdk-*` artifacts by fusing the toolchain/sysroot
+ # artifacts.
+ finalize:
+ name: Finalize wasi-sdk artifacts
+ needs: build
+ runs-on: ubuntu-latest
+ steps:
  - uses: actions/checkout@v4
+ - uses: actions/download-artifact@v4
+ - run: ./ci/merge-artifacts.sh
+ - uses: actions/upload-artifact@v4
  with:
- fetch-depth: 0
- - run: git fetch --tags --force
- name: Force-fetch tags to work around actions/checkout#290
+ name: release-artifacts
+ path: dist
 
- - run: git submodule update --init --depth 32 --jobs 3
+ # dockerbuild:
+ # name: Docker Build
+ # runs-on: ubuntu-latest
+ # steps:
+ # - uses: actions/cache@v4
+ # with:
+ # path: ~/.ccache
+ # key: 0-cache-ubuntu-bionic-${{ github.run_id }}
+ # restore-keys: |
+ # 0-cache-ubuntu-bionic
 
- - uses: docker/login-action@v2
- with:
- registry: ghcr.io
-  username: ${{ github.actor }}
-  password: ${{ secrets.GITHUB_TOKEN }}
+ #  - uses: actions/checkout@v4
+ #  with:
+ #  fetch-depth: 0
+ # - run: git fetch --tags --force
+ # name: Force-fetch tags to work around actions/checkout#290
 
- - uses: docker/setup-qemu-action@v2
- - uses: docker/setup-buildx-action@v2
+ # - run: git submodule update --init --depth 32 --jobs 3
 
- - uses: docker/metadata-action@v4
- id: meta
- with:
- images: ghcr.io/${{ github.repository }}
- tags: |
- type=schedule
- type=ref,event=branch
- type=ref,event=tag
- type=ref,event=pr
- type=sha
-
- - name: Run docker_build script
- run: ./docker_build.sh
- - name: Upload artifacts
- uses: actions/upload-artifact@v4
- with:
- # Upload the dist folder. Give it a name according to the OS it was built for.
- name: dist-ubuntu-bionic
- path: dist
+ # - uses: docker/login-action@v2
+ # with:
+ # registry: ghcr.io
+ # username: ${{ github.actor }}
+ # password: ${{ secrets.GITHUB_TOKEN }}
 
- - name: Build and push wasi-sdk docker image
- uses: docker/build-push-action@v3
- with:
- context: .
- file: docker/Dockerfile
- push: ${{ github.event_name != 'pull_request' && github.event_name != 'workflow_dispatch' }}
- platforms: linux/amd64,linux/arm64
- tags: ${{ steps.meta.outputs.tags }}
- labels: ${{ steps.meta.outputs.labels }}
- cache-from: type=gha
- cache-to: type=gha,mode=max
+ # - uses: docker/setup-qemu-action@v2
+ # - uses: docker/setup-buildx-action@v2
+
+ # - uses: docker/metadata-action@v4
+ # id: meta
+ # with:
+ # images: ghcr.io/${{ github.repository }}
+ # tags: |
+ # type=schedule
+ # type=ref,event=branch
+ # type=ref,event=tag
+ # type=ref,event=pr
+ # type=sha
+
+ # - name: Run docker_build script
+ # run: ./docker_build.sh
+ # - name: Upload artifacts
+ # uses: actions/upload-artifact@v4
+ # with:
+ # # Upload the dist folder. Give it a name according to the OS it was built for.
+ # name: dist-ubuntu-bionic
+ # path: dist
+
+ # - name: Build and push wasi-sdk docker image
+ # uses: docker/build-push-action@v3
+ # with:
+ # context: .
+ # file: docker/Dockerfile
+ # push: ${{ github.event_name != 'pull_request' && github.event_name != 'workflow_dispatch' }}
+ # platforms: linux/amd64,linux/arm64
+ # tags: ${{ steps.meta.outputs.tags }}
+ # labels: ${{ steps.meta.outputs.labels }}
+ # cache-from: type=gha
+ # cache-to: type=gha,mode=max

CMakeLists.txt

@@ -0,0 +1,45 @@
+# Build logic for building both a toolchain and a sysroot for WASI.
+#
+# This top level `CMakeLists.txt` file can be used either to build a clang
+# toolchain or a WASI sysroot. Note that this can't be done at the same time.
+# A toolchain build requires a compiler for the target architecture. A
+# WASI sysroot build requires this previous compiler and must be runnable on
+# the host.
+
+cmake_minimum_required(VERSION 3.26)
+project(wasi-sdk)
+include(ExternalProject)
+
+set(WASI_SDK_TARGETS "wasm32-wasi;wasm32-wasip1;wasm32-wasip2;wasm32-wasip1-threads;wasm32-wasi-threads"
+ CACHE STRING "List of WASI targets to build")
+option(WASI_SDK_BUILD_TOOLCHAIN "Build a toolchain instead of the sysroot" OFF)
+
+set(llvm_proj_dir ${CMAKE_CURRENT_SOURCE_DIR}/src/llvm-project)
+set(wasi_libc ${CMAKE_CURRENT_SOURCE_DIR}/src/wasi-libc)
+
+list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")
+include(wasi-sdk-enable-ccache)
+
+find_program(PYTHON python3 python REQUIRED)
+
+# Set some variables based on the `version.py` script
+set(version_script ${CMAKE_CURRENT_SOURCE_DIR}/version.py)
+execute_process(
+ COMMAND ${PYTHON} ${version_script} llvm-major --llvm-dir=${llvm_proj_dir}
+ OUTPUT_VARIABLE clang_version
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
+execute_process(
+ COMMAND ${PYTHON} ${version_script}
+ OUTPUT_VARIABLE wasi_sdk_version
+ OUTPUT_STRIP_TRAILING_WHITESPACE)
+
+message(STATUS "wasi-sdk toolchain LLVM version is ${clang_version}")
+message(STATUS "wasi-sdk version is ${wasi_sdk_version}")
+
+# Only include one version of the build logic as pulling in both isn't
+# supported at this time.
+if(WASI_SDK_BUILD_TOOLCHAIN)
+include(wasi-sdk-toolchain)
+else()
+include(wasi-sdk-sysroot)
+endif()