Make precompile files relocatable

[fatteneder]

Fixes #45215
Fixes #47943
Fixes #45541
Fixes #50918

---

Updated description

Hi.

I tried my luck with #47943 and I implemented what was suggested by OP.

String replacement with @depot when serializing out happens with any paths that are located inside a DEPOT_PATH (first match wins). If no match, then we emit the absolute file path as before. Right now we only emit one token @depot.

String replacement of @depot when loading happens now on a .ji file basis and only if all the listed include dependencies can be resolved to files located in one and the same depot on DEPOT_PATH (again, first match wins). If we can't resolve, then the cache is invalided with stale_cachefile.

This PR does not explicitly address any of #4630 or #33065 some suggested would be nice to fix in one swipe with #47943. It remains to check if this was accidentally solved.

I also did not understand what was meant in #47943 (comment) with the deps/build.jl and artifacts issue and how that should be addressed here.

Status of this PR

The following works for me locally,

• build julia, run it and compile a few packages

$ cd <git-root-julia>
$ export JULIA_DEPOT_PATH="$(pwd)/usr/share/julia"
$ ./julia
julia> using Pkg; Pkg.add("UnPack")
julia> using UnPack

• exit julia, copy the depot (<git-root>/usr/share/julia/) somewhere else, set JULIA_DEPOT_PATH accordingly, e.g.

$ cp usr/share/julia /tmp
$ export JULIA_DEPOT_PATH="/tmp/julia"

• restart julia and verify the package can be loaded without compilation by running

$ ./julia
pkg = Base.identify_package("UnPack")
path = Base.locate_package(pkg)
iscached = @lock Base.require_lock begin
    m = Base._require_search_from_serialized(pkg, path, UInt128(0))
    m isa Module
end
iscached == true # can load from cache

Open questions

From docs:

The first entry is the "user depot" and should be writable by and owned by the
current user. The user depot is where: registries are cloned, new package versions
are installed, named environments are created and updated, package repos are cloned,
newly compiled package image files are saved, log files are written, development
packages are checked out by default, and global configuration data is saved. Later
entries in the depot path are treated as read-only and are appropriate for
registries, packages, etc. installed and managed by system administrators.

• Does ... are treated as read-only ... guarantee that Julia will never write anything to a path other than DEPOT_PATH[1]?
• [Need to test this myself] If we load packages from any of the read-only paths, their precompiles also end up in DEPOT_PATH[1]/compiled?

If both questions can be answered with yes, then I guess its save to replace @depot always with DEPOT_PATH[1] when loading. The path for @stdlib is replaced with Sys.STDLIB, because that seems to be tied to where the julia binary sits.

I think I could work around all of that by following more closely what was proposed in #47943 (comment):

If we have a package Foo.jl which has been precompiled, that cache should reference the source files as something like @depot/packages/Foo_jll/a2b2c3/src/Foo.jl. During cache header verification, we string-replace ^@depot to the same depot that the cache file is currently being loaded from. I do not think it's worthwhile to allow this path to resolve to other depots on the depot path; it's too easy to find the "wrong" files, and I think it's reasonable to expect an atomic move of all associated resources for a package image.

See top of this post on how it is implemented now.

TODO

• Get basic functionality working.
• Write basic test.
• PkgEval -- first run looked good.
• Fix depot relocation logic.
• Write more tests.

I would appreciate any feedback!

[fatteneder]

I guess I did miss something, because the first test is already failing with
LoadError("sysimg.jl", 20, ErrorException("failed to identify a depot path: '/cache/build/default-armageddon-0/julialang/julia-master/usr/share/julia/stdlib/v1.10/ArgTools/src/ArgTools.jl'"))

[fatteneder]

This needs more work because it breaks this MWE

# from test/compiler/contextual.jl

load_path = mktempdir()
depot_path = mktempdir()

prev_DEPOT_PATH = deepcopy(DEPOT_PATH)
prev_LOAD_PATH = deepcopy(LOAD_PATH)

pushfirst!(LOAD_PATH, load_path)
pushfirst!(DEPOT_PATH, depot_path)

write(joinpath(load_path, "Foo.jl"),
    """
    module Foo
    end
    """)

try
    Foo = Base.require(Main, :Foo)
finally
    copy!(DEPOT_PATH, prev_DEPOT_PATH)
    copy!(LOAD_PATH, prev_LOAD_PATH)
end

[DilumAluthge]

This is very exciting. Thanks for working on this!

[fatteneder]

Regarding builds & tests:

I see random (?) build failures (e.g. curl: Operation too slow) I think are unrelated to this PR, since builds passed on all platforms at least yesterday.

There are some tests failing too where I don't know what is causing that.

• On Linux there are (random ?) LinearAlgebra tests failing, although they pass when run locally.
• Mac and Windows tests fail with ArgumentError: Invalid checksum in cache file (on Mac it happens in test/threads.jl). I don't have a Mac to debug but can setup a VM to test on Windows.

[vtjnash]

Some CI tests / machines are flaky, and those seem like they may be some of them. I think this is simple enough that I say we just go for it? We might have other issues still (like stacktraces and Strings in the source code that accidentally capture @__DIR__) but we can address those later.

Maybe @KristofferC can help make sure this gets merged?

[fatteneder]

I hacked together a MWE for a unit test.
IIUC its not possible to unload modules, so I thought I just call a test twice with some env vars set:
Once to precompile something and verify that its there. Then copy the whole depot into the test folder. And run a second time with the JULIA_DEPOT_PATH adjusted to verify that we can load again and that no recompilation was needed.

But I think it is not yet doing the right thing ...

[fatteneder]

Regarding test failures:

• The Windows and Mac ones are not random, so there is still something wrong in the PR.
• I tried to debug Windows with a mingw build, but there I could run selected tests like test/compiler and they passed. The CI stack traces did not give me a clue what to look for ...
• The test I added fails on the local mingw because of problems with cp.

[fatteneder]

I could identify and fix one more problem related to stale_cachefile:
Moving a depot causes the mtimes of package code to change and become out of sync with the times written in the .ji files.
To work around this I extended the .ji files to also contain the file size and hash next to the mtime of the sources they are pointing to as suggested here: #45541. The file size is added to have a fail-fast branch.

[IanButterworth]

I’ve not reviewed this yet but I just wanted to say thanks. It would be great to have this.

[IanButterworth]

Looks like this was hitting repeated precompilation on windows. This Pkg test was failing

https://github.com/JuliaLang/Pkg.jl/blob/master/test/api.jl#L339

If you want to investigate why the caches are being rejected you can set JULIA_DEBUG=loading

[vchuravy]

@nanosoldier runtests()

[vchuravy]

Let's do one more PkgEval and then merge this!

[nanosoldier]

The package evaluation job you requested has completed - possible new issues were detected.
The full report is available.

[IanButterworth]

🎉🎉🎉

[DilumAluthge]

This is really exciting. Thank you so much @fatteneder!

And thanks to @vchuravy, @staticfloat, @IanButterworth, @KristofferC, and anyone else that I missed that helped with reviewing this PR.

This PR crosses off two things (#45215 and #45541) off of my Julia wishlist.

[DilumAluthge]

@fatteneder @vchuravy Could we add this to NEWS?

[fatteneder]

Thanks everyone for the assistance!

[Keno]

This broke Revise, which is unfortunate. There's a PR open to fix it, but it'd have been nice to merge that first, since Revise is pretty essential for debugging for those of us that live on master.