RFC: break up monolithic sysimg build work #38119
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This alters the way that we build the system image, in preparation for future work in this direction. We've been talking about having better stacked and incremental system images for some time now. While this doesn't yet aim to fully accomplish that, I believe this does functionally simulate it quite well. Along the way, this lets us start identifying and fixing mistaken code where some piece accidentally depended on another piece, without declaring that dependency. It may also be seen as a template for PackageCompiler developers, as they should now better recognize some of the code involved here (particularly stdlib/generate_stdlib.jl) as being already in use. The general compile policies can now be seen to be split here between compile processes (such as Base.jl), linker processes (generate_minimal.jl and generate_full.jl), and their driver scripts (generate_sysimg.jl and generate_precompile.jl).
Implementation wise, this currently works a bit of slight-of-hand and knows that the default image "usr/lib/sys.so" has a better copy "usr/lib/fullsys.so" and will secretly open that one instead when you're using Julia interactively (in the future, we want this to be a bit smoother by delaying the decision until later). But otherwise (i.e. when you're using it non-interactively), it will default to opening the more minimal version, and thus be able to develop and replace the extra stdlibs. The results should be cross-compatible however, so the precompile caches for julia-{minimal|full}-{debug|release} can all be shared.
Opening as draft, since I'm running into this Pkg bug when attempting doctests: JuliaLang/Pkg.jl#2146 and this one too #38116
This also includes #37844, because those sort of introspection-based hacks to bypass correct Project.toml dependency usage are now intentionally not possible.