Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container building changes #3958

Merged
merged 4 commits into from
Oct 22, 2023

Conversation

BlackDex
Copy link
Collaborator

@BlackDex BlackDex commented Oct 5, 2023

Rework the container building.

  • Use docker buildx bake instead of custom build/push scripts
  • Just two Dockerfile's one for Debian and one for Alpine
  • Pinned openssl-sys version to v0.9.92, higher versions break Alpine builds ([REGRESSION] lib atomic is loaded dynamic again on armv6 with openssl-sys v0.9.93 sfackler/rust-openssl#2043)
  • Updated release workflow
  • Created documentation on how to use bake
  • Created a helper script bake.sh to easily bake/build the images (Read the documentation)
  • Created a podman helper script podman-bake.sh which uses podman instead of docker.
  • Updated the web-vault to v2023.9.1

I needed to change some way's to install the packages for MariaDB/MySQL, since on armv6/armel Debian it causes issues.
By downloading them and force installing via dpkg the build process still works on all platforms.

All images are tested using QEMU and also run the Favicon DDoS on all these images to verify it doesn't segfault.
The Alpine build images are now downloaded from ghcr.io instead of docker.io, maybe it helps in speed, but probably doesn't matter.

The runtime images are no longer a version maintained by Balena, they were sometimes outdated like there Alpine version was for a long time. And it also had a static qemu binary in there which only makes the image larger. Now we use the official images from both Debian and Alpine.

Also switched to the -slim rust building container, this has all the needed packages pre-installed (except for pkg-config).

Also:

  • Updated Rust to v1.73.0 (and MSRV to v1.71.1)
  • Updated all the crates possible

Fixes #3839
Fixes #3912
Fixes #3957

@BlackDex BlackDex force-pushed the release-build-revision branch 9 times, most recently from 649d6d4 to d59fb1f Compare October 7, 2023 11:48
@BlackDex BlackDex marked this pull request as ready for review October 7, 2023 11:59
@BlackDex BlackDex changed the title WIP: Container building changes Container building changes Oct 7, 2023
@BlackDex BlackDex force-pushed the release-build-revision branch 3 times, most recently from 06ed516 to 5c47965 Compare October 11, 2023 13:52
Copy link
Contributor

@stefan0xC stefan0xC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not yet really looked into docker buildx bake (except that you have to install the plugin for your docker installation, i.e. with sudo apt install docker-buildx-plugin which might be missing from this documentation).

Since I am using podman locally I wonder if we could add a script which builds vaultwarden locally with podman build, i.e. (based on the docker/bake.sh script) running

podman build --format=docker -f "${BASEDIR}/Dockerfile.debian" "${BASEDIR}/.."

will build an image (but it's missing the labels that are set by docker/docker-bake.hcl and maybe some other stuff).

Also the built vaultwarden image is missing the git version info. I'm not sure if this is a further limitation of my feeble attempt to getting it to work with podman or if it's a general issue. I'll have to setup a virtual machine to further test the docker buildx bake method (my cloud instance that is running docker was not powerful enough to compile Vaultwarden).

docker/amd64/Dockerfile Outdated Show resolved Hide resolved
docker/README.md Outdated Show resolved Hide resolved
docker/README.md Outdated Show resolved Hide resolved
.github/workflows/build.yml Show resolved Hide resolved
docker/Dockerfile.j2 Outdated Show resolved Hide resolved
docker/Dockerfile.j2 Outdated Show resolved Hide resolved
docker/Dockerfile.j2 Outdated Show resolved Hide resolved
docker/Dockerfile.debian Show resolved Hide resolved
docker/bake.sh Outdated Show resolved Hide resolved
@stefan0xC
Copy link
Contributor

Okay, I've compiled it in a virtual machine using docker/bake.sh and the version info is missing too:

$ docker run --rm -ti --name vaultwarden docker.io/vaultwarden/server:testing-amd64 /vaultwarden --version
vaultwarden (Version info from Git not present)

@BlackDex
Copy link
Collaborator Author

Hmm, good find, seems git is missing from the build image of rust.
An other strange item is that the alpine build does, but, it looks like it's using a wrong version for some reason, the branch name is correct, but not the version.

Ill have to look into this strange item.
With podman, i think installing moby buildkit should be enough with some other commands. Ill see if i can test that also, but my prio was, to have the CI build nicely with the docker tools first.

@stefan0xC
Copy link
Contributor

stefan0xC commented Oct 16, 2023

Installing git in the build image solves the issue with the version number (the tag is from your repository since I did a fresh clone of your repository):

docker run --rm -ti --name vaultwarden docker.io/vaultwarden/server:testing-amd64 /vaultwarden --version
vaultwarden 1.22.2-5c479659 (release-build-revision)

@BlackDex
Copy link
Collaborator Author

Installing git in the build image solves the issue with the version number (the tag is from your repository since I did a fresh clone of your repository):

docker run --rm -ti --name vaultwarden docker.io/vaultwarden/server:testing-amd64 /vaultwarden --version
vaultwarden 1.22.2-5c479659 (release-build-revision)

Ah! That explains it, so, that is ok then.

@BlackDex
Copy link
Collaborator Author

@stefan0xC please check again, i updated everything and as i mentioned on Matrix, there now is a podman-bake.sh script which you can use.

Copy link
Contributor

@stefan0xC stefan0xC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding a script for podman which seems to work just fine.

Wouldn't it be more consistent to call SOURCE_VERSION VW_VERSION and using the same value as label and inside vaultwarden?

docker/Dockerfile.j2 Show resolved Hide resolved
docker/README.md Outdated Show resolved Hide resolved
@BlackDex BlackDex force-pushed the release-build-revision branch 4 times, most recently from beda546 to b217af0 Compare October 20, 2023 17:54
@BlackDex
Copy link
Collaborator Author

Thanks again @stefan0xC, i think i have addressed all your comments and more.
It should be ready to go now :)

@BlackDex BlackDex force-pushed the release-build-revision branch 3 times, most recently from efaca6a to e81496e Compare October 21, 2023 11:54
- Updated to rust 1.73.0
- Updated crates
- Updated documentation
- Added a bake.sh script to make baking easier
- Updated workflow to use qemu and buildx bake

In the future i would like to extract the alpine based binaries and add
them as artifacts to the release.
- Addressed review remarks
- Added `podman-bake.sh` script to build Vaultwarden with podman
- Updated README
- Updated crates
- Added `VW_VERSION` support
- Added annotations
- Updated web-vault to v2023.9.1
@dani-garcia dani-garcia merged commit d722328 into dani-garcia:main Oct 22, 2023
3 checks passed
@BlackDex BlackDex deleted the release-build-revision branch October 23, 2023 07:56
arthurgeek pushed a commit to arthurgeek/vaultwarden-fly-template that referenced this pull request Nov 12, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [vaultwarden/server](https://togithub.com/dani-garcia/vaultwarden) |
stage | minor | `1.29.2-alpine` -> `1.30.0-alpine` |

---

### Release Notes

<details>
<summary>dani-garcia/vaultwarden (vaultwarden/server)</summary>

###
[`v1.30.0`](https://togithub.com/dani-garcia/vaultwarden/releases/tag/1.30.0)

[Compare
Source](https://togithub.com/dani-garcia/vaultwarden/compare/1.29.2...1.30.0)

⚠️ **Note:** The WebSockets service for live sync has been integrated in
the main HTTP server, which means simpler proxy setups that don't
require a separate rule to redirect WS traffic to port 3012. Please
check the updated examples in the
[wiki](https://togithub.com/dani-garcia/vaultwarden/wiki/Proxy-examples).
It's recommended to migrate to this new setup as using the old server on
port 3012 is deprecated, won't receive new features and will be removed
in a future release.

#### Major changes and New Features

- Added `passkey` support, allowing the browser extensions to store and
use your `passkeys`, make sure the extension is updated to version
`2023.10.0` or newer for passkey support.
-   Updated web vault to 2023.10.0.
-   Fixed crashes in ARMv6 devices
- Fixed crashes when trying to create/edit a cipher in the mobile
applications.

#### What's Changed

- Update Rust and Crates by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3808
- update web-vault to v2023.8.2 by
[@&#8203;stefan0xC](https://togithub.com/stefan0xC) in
[dani-garcia/vaultwarden#3821
- Fix Login With Device without MasterPassword by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3831
- Update GitHub Workflow by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3910
- Fix arm builds by [@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3911
- Fix typos by [@&#8203;tuhanayim](https://togithub.com/tuhanayim) in
[dani-garcia/vaultwarden#3959
- csp: rename anonaddy.com to addy.io by
[@&#8203;stefan0xC](https://togithub.com/stefan0xC) in
[dani-garcia/vaultwarden#3950
- filter handlebars logs by
[@&#8203;stefan0xC](https://togithub.com/stefan0xC) in
[dani-garcia/vaultwarden#3859
- Remove unnecessary variable clone by
[@&#8203;mvalois](https://togithub.com/mvalois) in
[dani-garcia/vaultwarden#3981
- README.md: Fix grammar nit by
[@&#8203;AndreasHGK](https://togithub.com/AndreasHGK) in
[dani-garcia/vaultwarden#3965
- Fix small issues by [@&#8203;BlackDex](https://togithub.com/BlackDex)
in
[dani-garcia/vaultwarden#3964
- Adds LastActive on /admin/users API route by
[@&#8203;mvalois](https://togithub.com/mvalois) in
[dani-garcia/vaultwarden#3951
- Reopen log file on SIGHUP by
[@&#8203;tobiasmboelz](https://togithub.com/tobiasmboelz) in
[dani-garcia/vaultwarden#3909
- Fix External ID not set during DC Sync by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3804
- New config option disable email change by
[@&#8203;admav](https://togithub.com/admav) in
[dani-garcia/vaultwarden#3986
- 2FA Confirmation Code Email subject line change to fix triggering
Google spam blocker by
[@&#8203;aureateflux](https://togithub.com/aureateflux) in
[dani-garcia/vaultwarden#3572
- Implement cipher key encryption by
[@&#8203;dani-garcia](https://togithub.com/dani-garcia) in
[dani-garcia/vaultwarden#3990
- Container building changes by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3958
- Fix issue with MariaDB/MySQL migrations by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3994
- feat: Working passkeys storage by
[@&#8203;GeekCornerGH](https://togithub.com/GeekCornerGH) in
[dani-garcia/vaultwarden#4025
- ci: add trivy workflow by
[@&#8203;mightyBroccoli](https://togithub.com/mightyBroccoli) in
[dani-garcia/vaultwarden#3997
- Fix importing Bitwarden exports by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#4030

#### New Contributors

- [@&#8203;tuhanayim](https://togithub.com/tuhanayim) made their first
contribution in
[dani-garcia/vaultwarden#3959
- [@&#8203;mvalois](https://togithub.com/mvalois) made their first
contribution in
[dani-garcia/vaultwarden#3981
- [@&#8203;AndreasHGK](https://togithub.com/AndreasHGK) made their first
contribution in
[dani-garcia/vaultwarden#3965
- [@&#8203;tobiasmboelz](https://togithub.com/tobiasmboelz) made their
first contribution in
[dani-garcia/vaultwarden#3909
- [@&#8203;admav](https://togithub.com/admav) made their first
contribution in
[dani-garcia/vaultwarden#3986
- [@&#8203;aureateflux](https://togithub.com/aureateflux) made their
first contribution in
[dani-garcia/vaultwarden#3572
- [@&#8203;mightyBroccoli](https://togithub.com/mightyBroccoli) made
their first contribution in
[dani-garcia/vaultwarden#3997

**Full Changelog**:
dani-garcia/vaultwarden@1.29.2...1.30.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "on saturday" (UTC), Automerge - At
any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/arthurgeek/vaultwarden-fly-template).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40Ni4wIiwidXBkYXRlZEluVmVyIjoiMzcuNDYuMCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
arthurgeek pushed a commit to arthurgeek/vaultwarden-fly that referenced this pull request Nov 12, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [vaultwarden/server](https://togithub.com/dani-garcia/vaultwarden) |
stage | minor | `1.29.2-alpine` -> `1.30.0-alpine` |

---

### Release Notes

<details>
<summary>dani-garcia/vaultwarden (vaultwarden/server)</summary>

###
[`v1.30.0`](https://togithub.com/dani-garcia/vaultwarden/releases/tag/1.30.0)

[Compare
Source](https://togithub.com/dani-garcia/vaultwarden/compare/1.29.2...1.30.0)

⚠️ **Note:** The WebSockets service for live sync has been integrated in
the main HTTP server, which means simpler proxy setups that don't
require a separate rule to redirect WS traffic to port 3012. Please
check the updated examples in the
[wiki](https://togithub.com/dani-garcia/vaultwarden/wiki/Proxy-examples).
It's recommended to migrate to this new setup as using the old server on
port 3012 is deprecated, won't receive new features and will be removed
in a future release.

#### Major changes and New Features

- Added `passkey` support, allowing the browser extensions to store and
use your `passkeys`, make sure the extension is updated to version
`2023.10.0` or newer for passkey support.
-   Updated web vault to 2023.10.0.
-   Fixed crashes in ARMv6 devices
- Fixed crashes when trying to create/edit a cipher in the mobile
applications.

#### What's Changed

- Update Rust and Crates by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3808
- update web-vault to v2023.8.2 by
[@&#8203;stefan0xC](https://togithub.com/stefan0xC) in
[dani-garcia/vaultwarden#3821
- Fix Login With Device without MasterPassword by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3831
- Update GitHub Workflow by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3910
- Fix arm builds by [@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3911
- Fix typos by [@&#8203;tuhanayim](https://togithub.com/tuhanayim) in
[dani-garcia/vaultwarden#3959
- csp: rename anonaddy.com to addy.io by
[@&#8203;stefan0xC](https://togithub.com/stefan0xC) in
[dani-garcia/vaultwarden#3950
- filter handlebars logs by
[@&#8203;stefan0xC](https://togithub.com/stefan0xC) in
[dani-garcia/vaultwarden#3859
- Remove unnecessary variable clone by
[@&#8203;mvalois](https://togithub.com/mvalois) in
[dani-garcia/vaultwarden#3981
- README.md: Fix grammar nit by
[@&#8203;AndreasHGK](https://togithub.com/AndreasHGK) in
[dani-garcia/vaultwarden#3965
- Fix small issues by [@&#8203;BlackDex](https://togithub.com/BlackDex)
in
[dani-garcia/vaultwarden#3964
- Adds LastActive on /admin/users API route by
[@&#8203;mvalois](https://togithub.com/mvalois) in
[dani-garcia/vaultwarden#3951
- Reopen log file on SIGHUP by
[@&#8203;tobiasmboelz](https://togithub.com/tobiasmboelz) in
[dani-garcia/vaultwarden#3909
- Fix External ID not set during DC Sync by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3804
- New config option disable email change by
[@&#8203;admav](https://togithub.com/admav) in
[dani-garcia/vaultwarden#3986
- 2FA Confirmation Code Email subject line change to fix triggering
Google spam blocker by
[@&#8203;aureateflux](https://togithub.com/aureateflux) in
[dani-garcia/vaultwarden#3572
- Implement cipher key encryption by
[@&#8203;dani-garcia](https://togithub.com/dani-garcia) in
[dani-garcia/vaultwarden#3990
- Container building changes by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3958
- Fix issue with MariaDB/MySQL migrations by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#3994
- feat: Working passkeys storage by
[@&#8203;GeekCornerGH](https://togithub.com/GeekCornerGH) in
[dani-garcia/vaultwarden#4025
- ci: add trivy workflow by
[@&#8203;mightyBroccoli](https://togithub.com/mightyBroccoli) in
[dani-garcia/vaultwarden#3997
- Fix importing Bitwarden exports by
[@&#8203;BlackDex](https://togithub.com/BlackDex) in
[dani-garcia/vaultwarden#4030

#### New Contributors

- [@&#8203;tuhanayim](https://togithub.com/tuhanayim) made their first
contribution in
[dani-garcia/vaultwarden#3959
- [@&#8203;mvalois](https://togithub.com/mvalois) made their first
contribution in
[dani-garcia/vaultwarden#3981
- [@&#8203;AndreasHGK](https://togithub.com/AndreasHGK) made their first
contribution in
[dani-garcia/vaultwarden#3965
- [@&#8203;tobiasmboelz](https://togithub.com/tobiasmboelz) made their
first contribution in
[dani-garcia/vaultwarden#3909
- [@&#8203;admav](https://togithub.com/admav) made their first
contribution in
[dani-garcia/vaultwarden#3986
- [@&#8203;aureateflux](https://togithub.com/aureateflux) made their
first contribution in
[dani-garcia/vaultwarden#3572
- [@&#8203;mightyBroccoli](https://togithub.com/mightyBroccoli) made
their first contribution in
[dani-garcia/vaultwarden#3997

**Full Changelog**:
dani-garcia/vaultwarden@1.29.2...1.30.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "on saturday" (UTC), Automerge - At
any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/arthurgeek/vaultwarden-fly).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40Ni4wIiwidXBkYXRlZEluVmVyIjoiMzcuNDYuMCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants