feat: build relases on gh-actions & support linux arm64 #2629

hegerdes · 2024-09-03T21:20:31Z

IMPORTANT: Please do not create a Pull Request adding a new feature without discussing it first.

The place to discuss new features is the forum: https://zola.discourse.group/
If you want to add a new feature, please open a thread there first in the feature requests section.

Sanity check:

Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Code changes

(Delete or ignore this section for documentation changes)

Are you doing the PR on the next branch?

What changed

Fixes #2620

This ports the azure-devops pipeline to GH-Actions to allow for easy docker arm build all in one platform.
This adds a build & test job that that runs cargo test --all, cargo fmt --check and cargo build --all. All transferred from Azure Pipelines. This is run for the following targets:

x86_64-unknown-linux-gnu
aarch64-unknown-linux-gnu
x86_64-pc-windows-msvc
x86_64-apple-darwin
aarch64-apple-darwin

All target are naively compiled (also macos arm since GH has M1 runners) except for linux arm64. Due to GitHub limiting hosted arm runner to paid plans we need to cross-compile.

A test job can be found here

The same setup is implemented for release which runs only on tags matching v*. All binaries are build uploaded as an artifact, singed via GitHub attestations and finally one release is created. See demo workflow run

The Docker images are build after that. There is an optional build arg USE_GH_RELEASE which defaults to false. If set to true it downloads the latest zola version (in the right ARCH) in the dockerfile instead of building it itself.
This allows the (re)usage of already signed artifacts and avoids Docker QEMU emulation which would result in 1h+ build times. Users should be fine building it locally with the current source by running docker build --tag zola .

To discuss:

In order to cross-compile I needed to create a file .cargo/config.toml and setting the liker to aarch64-linux-gnu-gcc for that arch. I have no real knowledge in rust and rust cross-compiling. Is there a better way, can we avoid that file? -> FIXED by using CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER in GH Action only - this does not affect anyone build
Check the file names in the releases. I created a demo release on my fork (will delete later). Please double check: https://github.com/hegerdes/zola/releases
Is the approach with the optional arg in docker fine to download existing releases?

hegerdes · 2024-09-03T21:21:37Z

Do you have runtime stats for azure-devops?

Just to compare:

Note: It does not take 45min. All jobs except for the last two run in parallel.

Signed-off-by: Henrik Gerdes <[email protected]>

Keats · 2024-09-05T08:29:50Z

The last release:

Keats · 2024-09-05T08:30:40Z

.github/workflows/build-and-test.yml

+          - aarch64-unknown-linux-gnu
+          - x86_64-pc-windows-msvc
+          - x86_64-apple-darwin
+          - aarch64-apple-darwin


Do we need to run tests on different archs for the same OS?

GitHubs Action matrix is a little counter-intuitive. A job for each target (currently 5) and each rustup_toolchain (currently 1) is created. The include part sets the os. We use the native os for each target, except linux arm. I thought testing on the native arch is best when ever possible - but I know very litte about the rust build standards.

If you think testing with cross compiling is better or also needed required let me know the exact test matrix.

.github/workflows/build-and-test.yml

.github/workflows/release.yml

Keats · 2024-09-05T08:36:37Z

Dockerfile

-
+RUN if [ "${USE_GH_RELEASE}" = "true" ]; then \
+    mkdir -p target/$(uname -m)-unknown-linux-gnu/release && \
+    export ZOLA_VERSION=$(curl -sL https://api.github.com/repos/getzola/zola/releases/latest | jq -r .name) && \


This is not reproducible, can you pass the version to the Dockerfile instead?

Totally agree. Just wanted to check if this approach is fine before going all the way.
I would pass an extra arg like ZOLA_VERSION with. I would set the default to latest and override in CI to the current tag. To enforce reproducibility we could also just fail if USE_GH_RELEASE is set but ZOLA_VERSION is not. Any takes on that?

Sounds good

Keats · 2024-09-05T08:37:54Z

.github/workflows/release.yml

+            ```
+          token: ${{ secrets.GITHUB_TOKEN }}
+          prerelease: ${{ contains(github.ref, '-pre') }}
+          files: artifacts/*


What's the reason behind splitting the Release-Build and Release jobs? Why not do it in 1 job?

The build for each target run all on their own isolated runner. So the targets can be build native and in parallel. The runners don't share any storage so wee need to get all build assets on one runner. This is done via upload/download artifacts. The artifacts assets are temporary and will auto delete in one week.

When each runner creates a release we can have race conditions or partly failed release jobs. This ensures the release build for every target worked and only then one release is created. If one target fails -> no release.

The argument prerelease would allow you to create pre-releases when the tag looks like v0.1.0-pre.0 It will not be marked as latest.

Keats · 2024-09-05T08:38:36Z

.github/workflows/release.yml

+            You can verify the signatures using the `GitHub` CLI.
+
+            ```shell
+            gh attestation verify --owner ${{ github.repository_owner }} <my-artifact>


Right now Azure generates a list of commits since the last release (https://github.com/getzola/zola/releases/tag/v0.19.2), is it possible to keep it?

Yeah it should generate release notes with generate_release_notes: true. My demo release didn't have any real changes since it was the first release and I'm not working with PRs for active development and debugging.

I'm also using this action here: https://github.com/hegerdes/joplin-plugin-remote-note-pull/releases/tag/v1.3.0 and Pulumi also uses uses with some more changes than my small project, see https://github.com/pulumi/pulumi-awsx/releases

I think they might look a litte different to azures notes though.

Feel fee to add whatever you want to that release body you are the owner, it was just an example for now.

Signed-off-by: Henrik Gerdes <[email protected]>

Keats · 2024-09-13T07:29:59Z

Nice, thanks!

hegerdes · 2024-09-13T08:18:23Z

Thanks for merging this @Keats :) I really checked for errors but I would still recommend doing a pre-release once before switching completely over to the new release setup. A tag containing -pre should do the trick.

If anything does not look right feel free to tag me anytime

feat: port zola release build to gh actions & support linux arm64

6422779

Signed-off-by: Henrik Gerdes <[email protected]>

hegerdes force-pushed the feat/gh-action-and-arm-build branch from eb63ddc to 6422779 Compare September 3, 2024 21:27

hegerdes mentioned this pull request Sep 3, 2024

Docker image for arm64 #2620

Open

hegerdes added 3 commits September 3, 2024 23:36

fix: add buildx support for multi-arch docker images

2904f63

Signed-off-by: Henrik Gerdes <[email protected]>

fix: pin gh release action to sha & allow pre-releases

5bcf7c6

Signed-off-by: Henrik Gerdes <[email protected]>

fix: use env in gh action for linux arm build

0ec2840

Signed-off-by: Henrik Gerdes <[email protected]>

Keats reviewed Sep 5, 2024

View reviewed changes

chore: switch to dtolnay/rust-toolchain action for rust setup

ea4b901

Signed-off-by: Henrik Gerdes <[email protected]>

hegerdes marked this pull request as ready for review September 5, 2024 15:37

fix: windows archive step

d411bd9

Signed-off-by: Henrik Gerdes <[email protected]>

Keats merged commit 481135f into getzola:next Sep 13, 2024
5 checks passed

filmor mentioned this pull request Oct 21, 2024

Zola not working on aarch64 linux #2332

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: build relases on gh-actions & support linux arm64 #2629

feat: build relases on gh-actions & support linux arm64 #2629

hegerdes commented Sep 3, 2024 •

edited

Loading

hegerdes commented Sep 3, 2024 •

edited

Loading

Keats commented Sep 5, 2024

Keats Sep 5, 2024

hegerdes Sep 5, 2024

Keats Sep 5, 2024

hegerdes Sep 5, 2024

Keats Sep 8, 2024

Keats Sep 5, 2024

hegerdes Sep 5, 2024

Keats Sep 5, 2024

hegerdes Sep 5, 2024

Keats commented Sep 13, 2024

hegerdes commented Sep 13, 2024

feat: build relases on gh-actions & support linux arm64 #2629

feat: build relases on gh-actions & support linux arm64 #2629

Conversation

hegerdes commented Sep 3, 2024 • edited Loading

Code changes

What changed

To discuss:

hegerdes commented Sep 3, 2024 • edited Loading

Keats commented Sep 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Keats commented Sep 13, 2024

hegerdes commented Sep 13, 2024

hegerdes commented Sep 3, 2024 •

edited

Loading

hegerdes commented Sep 3, 2024 •

edited

Loading