Add preference for version named manifest files

[IanButterworth]

(Updated to latest)

This makes julia prefer manifest files in the format (Julia)Manifest-v{major}.{minor}.toml over a simple (Julia)Manifest.toml, to allow maintaining multiple manifests for different julia versions concurrently.

A new manifest will still save as (Julia)Manifest.toml (the Julia prefix is used if you have a JuliaProject.toml), so it's up to the user to manually rename the manifest to enable this. So Manifest-v1.11.toml would work for v1.11.

A downside of this approach is adding two file stat operations on-top of the most common scenarios of just having a Manifest.toml or no manifest saved at all, which might be non-negligible on some filesystems. There are design optimizations possible there though (see #43845 (comment))

An alternative is to store multiple julia-versioned manifests within a single Manifest.toml

[jpsamaroo]

An alternative is to store multiple julia-versioned manifests within a single Manifest.toml

If we need to do a breaking change to the Manifest spec in the future (surely we will), then this would be problematic. It's also often easiest to just manage versions as separate files on the filesystem. For example, if I'm developing my package (which supports Julia 1.7) with 1.8 because I like it more, it doesn't mean I want that 1.8-specific bit of Manifest committed with the rest of the versions in the Manifest when I go to push a change.

[IanButterworth]

Good point.

Also, I don't think this would need backporting to 1.6, you'd just include the 1.6 one as Manifest.toml

[pfitzseb]

Manifest.1.8.toml/Manifest-1-8.toml look better imho.

[IanButterworth]

This order probably makes more sense

Suggested change

	const manifest_names = ("JuliaManifest$(VERSION.major).$(VERSION.minor).toml", "JuliaManifest.toml",
	"Manifest$(VERSION.major).$(VERSION.minor).toml", "Manifest.toml")
	const manifest_names = ("JuliaManifest$(VERSION.major).$(VERSION.minor).toml",
	"Manifest$(VERSION.major).$(VERSION.minor).toml",
	"JuliaManifest.toml",
	"Manifest.toml")

[vtjnash]

This looks great to me. I hope we are not wasting time now with a stat call (aka isfile), since it is substantially faster (and more accurate) to just try to open the file name.

[DilumAluthge]

@IanButterworth I think this is resolved by 6c36a8d, right?

[IanButterworth]

The first issue is resolved, but Jameson's point is worth keeping visible. Currently loading does do the bad thing of isfile testing rather than just opening because it happens in different places.

[DilumAluthge]

I prefer the format Manifest.1.8.toml, as I find it to be much easier to read.

I'd also be okay with Manifest-1.8.toml, but I don't love it. Manifest.1.8.toml is my first choice.

I really dislike Manifest1.8.toml or Manifest-1-8.toml.

[DilumAluthge]

In the General registry¹, we've already implemented the pattern of Manifest.1.6.toml², Manifest.1.7.toml³, Manifest.1.8.toml⁴, etc. And I would really like to keep that same naming convention.

Footnotes

1. https://github.com/JuliaRegistries/General/tree/master/.ci ↩

2. https://github.com/JuliaRegistries/General/blob/master/.ci/Manifest.1.6.toml ↩

3. https://github.com/JuliaRegistries/General/blob/master/.ci/Manifest.1.7.toml ↩

4. https://github.com/JuliaRegistries/General/blob/master/.ci/Manifest.1.8.toml ↩

[DilumAluthge]

If I recall correctly (from the last time I looked at this code), you'll need to make a few small changes to Pkg, but nothing terribly difficult.

[vtjnash]

bump?

[IanButterworth]

It might be good to get some more Pkg views

@KristofferC @StefanKarpinski @fredrikekre

Perhaps run by triage too?

[DilumAluthge]

It could also be useful to have a Pkg call.

[IanButterworth]

bumping for Pkg team opinions.

Given there's a little work to do in Pkg if this merges, it would be good to get it decided before the 1.8 freeze

[IanButterworth]

I think it's fair to say from a slack discussion in #pkg-dev that there's some dislike of this approach, and a suggestion by @GunnarFarneback to add a --manifest option to julia to point to a custom manifest file/name convention was well received.

Potential use models are:

1. --manifest=Manifest.1.8.toml: a strict name
2. --manifest=path/to/Manifest.1.8.toml: a strict relative path
3. --manifest=Manifest.#.#.toml: a magic name where the version is filled into the #'s
4. --manifest=path/to/Manifest.#.#.toml: same magic but a relative path

There was also the idea of doing the same via a JULIA_MANIFEST_NAME env var, which might sidestep potential issues with #'s messing up the arg handling on some platforms?

cc. @GunnarFarneback @DilumAluthge @fredrikekre @StefanKarpinski

[DilumAluthge]

I think we only need to worry about use cases 1 and 3, which is exactly what your PR #44286 does.

[IanButterworth]

This has come up a few times recently, and seems to be the preferred approach over #44286

Stale

[DilumAluthge]

I would prefer Stefan's original suggestion over the hyphenated variant.

[StefanKarpinski]

How about instead of top-level files (which feels like it's going to get crowded), use a Manifest directory like this: Manifest/v1.9.toml?

[LilithHafner]

Questions for traige

1. Do we want to support specifying different manifests for different versions of Julia
2. What approach do we want to take: Canonical file paths / environment variables / startup args / Manifest.toml contains multiple versions / other
3. If we go with canonical file paths, what names should they have?

[IanButterworth]

Triage generally agrees that the best worst is Manifest-v1.10.toml and has given the ok to go ahead with that

[IanButterworth]

This is RTM from my perspective. I will merge this and bump Pkg immediately after merge.

[StefanKarpinski]

LGTM

[ufechner7]

Shall this be back-ported? If yes, to which Julia versions?

[LilithHafner]

Features are not usually backported.

If we don't backport this, and you are using both 1.9 and 1.10, then you'll have to keep dealing with the problems for now.

If you are using 1.10 and 1.11, though, then you can have a Manifest.toml file that is for 1.10 and a Manifest-v1.11.toml file that is for 1.11, and each Julia version will know to use the correct manifest file.

[vchuravy]

Is there a way to cause Julia to automatically create versioned Manifests? Would make switching versions a lot nicer.

Also an empty Manifest file is treated as v1

touch Manifest-v1.11.toml

┌ Warning: The active manifest file at `/home/vchuravy/src/GPUCompiler/Manifest-v1.11.toml` has an old format that is being maintained.
│ To update to the new format, which is supported by Julia versions ≥ 1.6.2, run `import Pkg; Pkg.upgrade_manifest()` which will upgrade the format without re-resolving.
│ To then record the julia version re-resolve with `Pkg.resolve()` and if there are resolve conflicts consider `Pkg.update()`.
└ @ Pkg.Types ~/.julia/juliaup/julia-nightly/share/julia/stdlib/v1.11/Pkg/src/manifest.jl:318