Meta issue about building packages for experimental platforms (aarch64-apple-darwin and armv6l-linux-*)

[giordano]

There is a large demand to build packages for the "experimental" platforms aarch64-apple-darwin and armv6l-linux-*, but there are two NON-NEGOTIABLE constraints

1. the JLL package must require at least Julia v1.6, because only Julia v1.6+ is able to deal with the presence of these platforms in the Artifacts.toml file
2. most existing packages are marked as compatible with Julia v1.0, but as a consequence of the point 1 above if we want to rebuild an existing package for the new platform the generated JLL package must have a new version number. No exceptions whatsoever to this rule, otherwise we'll break the ecosystem for Julia v1.5-

Also, if a package has binary dependencies, we need to build all of them before being able to build it. The fact that we need to require Julia v1.6 means that we need to cut out of support old versions of Julia, and not all package maintainers will be happy about that decision. The rebuild of the packages of a consistent part of the ecosystem will reasonably take a few months, certainly not before a new long-term support (LTS) version of Julia is officially announced.

Examples of commonly requested packages:

• FFTW: it doesn't have dependencies, but (a) we need to build a new version (b) the maintainer needs to agree to require Julia v1.6 (ARM Issue (Raspberry Pi Zero) JuliaMath/FFTW.jl#168, Darwin ARM: libfftw3 not defined JuliaMath/FFTW.jl#187, FFTW_jll for Mac M1 (ARM) #2738; related to the issue of requiring Julia v1.6: Eliminate build step, use Preferences JuliaMath/FFTW.jl#188)
• zeromq (ZMQ jll for Apple Silicon platform needed JuliaInterop/ZMQ.jl#210, Please build macos :aarch64 for ZeroMQ to support Apple Silicon BinaryBuilder.jl#1072; built in [ZeroMQ] Update to v4.3.4 and build for experimental platforms #3194)
• openspecfun: no dependencies ([openspecfun] Update to 0.5.4 #2892)
• Hwloc: no dependencies, but needs to require Julia v1.6 for a new version (Darwin ARM: libhwloc not defined JuliaParallel/Hwloc.jl#44, Hwloc_jll for Mac M1 (ARM) #2733, done in [Hwloc] Build for experimental platforms #3190)
• Rmath (asked a few times in Darwin/ARM64 tracking issue JuliaLang/julia#36617 because it's an indirect dependency of Distributions.jl): built with [Rmath] Build for experimental platforms #2732
• GR: lots of dependencies ([GR] Build on Apple M1 #2760)
• libsodium: done in [libsodium] Build for experimental platforms #2768
• bzip2: [Bzip2] Upgrade to v1.0.8 and build for experimental platforms #2957
• freetyp2: [FreeType2] Upgrade to v2.10.4 and build for experimental platforms #2994
• expat: [Expat] Upgrade to v2.2.10 and build for experimental platforms #2995
• libuuid: [Libuuid] Upgrade to v2.36 and build for experimental platforms #2997
• fontconfig: [Fontconfig] Upgrade to v2.13.93 and build for experimental platforms #2998
• libiconv: [Libiconv] Build for experimental platforms and add pkg-config file #3004
• libmount: [Libmount] Upgrade to v2.35 and build for experimental platforms #3006
• libffi: [Libffi] Build for experimental platforms #3007
• xml2: [XML2] Upgrade to v2.9.12 #3009
• pcre: [PCRE] Upgrade to v8.44 and build for experimental platforms #3010
• lz4: [Lz4] Upgrade to v1.9.3 and build for experimental platforms #3035
• gettext: [Gettext] Upgrade to v0.21 and build for experimental platforms #3037
• glib: [Glib] Upgrade builder to v2.68.1 #91
• lzo: [LZO] Build for experimental platforms #3047
• pixman: [Pixman] Build for experimental platforms #3048
• libpng: [libpng] Build for experimental platforms #3049
• libgpg_error: [Libgpg_error] Upgrade to v1.42 and build for experimental platforms #3050
• libgcrypt: [Libgcrypt] Upgrade to v1.8.7 and build for experimental platforms #3056
• xslt: [XSLT] Upgrade to v1.1.34 and build for experimental platforms #3055
• ICU: [ICU] Upgrade to v69.1 and build for experimental platforms #3068
• cairo: [Cairo] Build for aarch64-apple-darwin #3062
• graphite2: [Graphite2] Upgrade to v1.3.14 and build for experimental platforms #3070
• fribidi: [FriBidi] Upgrade to v1.0.10 and build for experimental platforms #3071
• ncurses: [Ncurses] Upgrade to v6.2 and build for experimental platforms #3075
• harfbuzz: [HarfBuzz] Upgrade to v2.8.1, split ICU, build for experimental platforms #3069
• pango: [Pango] Upgrade to v1.47.0 and build for experimental platforms #3079
• lame: [LAME] Build for experimental platforms #3093
• libvpx: [LibVPX] Upgrade to v1.10.0 and build for experimental platforms #3094
• openssl: [OpenSSL] Upgrade to v1.1.1k and build for experimental platforms #3096
• opus: [Opus] Build for experimental platforms #3097
• ogg: [Ogg] Build for experimental platforms #3098
• libass: [libass] Upgrade to v0.15.1 and build for experimental platforms #3100
• libfdk_aac: [libfdk_aac] Upgrade to v2.0.2 and build for experimental platforms #3101
• libvorbis: [libvorbis] Upgrade to v1.3.7 and build for experimental platforms #3102
• x264: [x264] Upgrade to latest revision and build for experimental platforms #3103
• x265: [x265] Upgrade to v3.5 and build for experimental platforms #3104
• ffmpeg: [FFMPEG] Upgrade to v4.4 and build for experimental platforms #3105
• jpegturbo: [JpegTurbo] Upgrade to v2.1.0 and build for experimental platforms #3107
• libtiff: [Libtiff] Upgrade to v4.3.0 and build for experimental platforms #3110
• cuba: [Cuba] Build for experimental platforms #3114
• glfw: [GLFW] Build for aarch64-apple-darwin #3118
• qt5base: [Qt5Base] Build for aarch64-apple-darwin #3119
• libwebp: [libwebp] Upgrade to v1.2.0 and build for experimental platforms #3306
• leptonica: 3308
• littlecms: [LittleCMS] Upgrade to v2.12.0 and build for experimental platforms #3309
• openjpeg: 3310
• graphene: 3393
• libepoxy: [Libepoxy] Update to v1.5.8 and build for experimental platforms #3392
• atk: [ATK] Build for experimental platforms #3397
• gdk-pixbuf 3398
• gtk3: [GTK3] Update to v3.24.30 and build for experimental platforms #3401
• SymEngine

[giordano]

Note: I've built bzip2 for the new architectures with #2957, but that changed the soname of the library, which means we can't have downstream libraries depending on both v1.0.6 and v1.0.8 (registration has been put on hold: JuliaRegistries/General#36391), they must be rebuilt against v1.0.8. It should be all fixed now

[mkborregaard]

This seems to happen for ImageMagick as well

julia> using ImageMagick
ERROR: InitError: UndefVarError: libwand not defined
Stacktrace:
 [1] magickgenesis
   @ ~/.julia/packages/ImageMagick/b8swT/src/libmagickwand.jl:21 [inlined]
 [2] __init__()
   @ ImageMagick ~/.julia/packages/ImageMagick/b8swT/src/libmagickwand.jl:27
 [3] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base ./loading.jl:768
 [4] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base ./loading.jl:854
 [5] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1097
 [6] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1013
 [7] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:997
during initialization of module ImageMagick

Is this thread the right place to report?

[giordano]

Is this thread the right place to report?

Yes, but there was already a PR open (and already merged a couple of weeks ago) for this: #3424

[jeremiahpslewis]

Sorry for creating a duplicate issue, reposting the content:

There are currently 214 instances of platforms = supported_platforms() in the Yggdrasil code base, i.e. cases where (mainly because the script has yet to be updated), experimental platforms like Apple Silicon are not supported. A rough grep of build_tarballs(ARGS suggests this is ~25% (214 / 926) of binary files.

This migration is a very specific change which is repetitive and creates (what I assume to be) an added burden on maintainers.

Would it make sense to get a few people together on a couple of afternoons and do an 'experimental platforms hackathon' to move this number from 25% towards ~0%? This could be planned so that it is maximally convenient for maintainers and so that open PRs don't pollute the repo.

Thoughts? @giordano

[giordano]

Answered in #3805 (comment)

[mkborregaard]

Thanks @giordano , I didn't see that issue above. Great!

[gbaraldi]

libhdf5 isn't supported on either the M1 and also lots of other platforms :s . But it looks to be really awful to build.

[giordano]

We can't cross-compile HDF5, we repackage native builds from the Python ecosystem

[imciner2]

@giordano is there a reason to keep this open now that the list of experimental platforms is actually empty?

[giordano]

Uh, sorry, I missed the message. No, probably this can be closed now.