Add vtjnash/Pidfile.jl as a new external stdlib

[IanButterworth]

Pidfile.jl will be helpful for improving multiprocess file IO concurrency across code loading precompilation, Distributed and Pkg operations to name a few
This replaces #42471

This PR started as the output of the contrib/new-stdlib.sh wizard.

I tried to add the checksums but hit:

julia % make -f contrib/refresh_checksums.mk Pidfile
make: *** No rule to make target `Pidfile'.  Stop.

So I just added the MacOS ones for now.

Before merging this I guess it would be appropriate to move Pidfile to the JuliaLang org?

[DilumAluthge]

We should get the contrib/refresh_checksums.mk stuff working before we merge this.

Did you make this PR manually? I think there's a script (contrib/new-stdlib.sh) that should automate this process.

[IanButterworth]

This PR started as the output of the contrib/new-stdlib.sh wizard.

All it did was create the Pidfile.version file, and add Pidfile to to STDLIBS_EXT

[KristofferC]

I still don't think this should be an external stdlib since we want to use it in Base.

Regarding #42471 (comment)

TOML is not an external stdlib. It has a separate repo so people can use it before it was an stdlib but it is completely in repo. Also, only the extra parts of logging (the ones not used in Base) are in the stdlib. This looks like it should just go into file system, and if filsystem gets moved out, this can go with it.

[KristofferC]

Triage for what the best way is to incorporate this into julia.

[JeffBezanson]

From triage:

• First we'll wait until we have a use case ready to go, so we're sure we will actually use this in Base.
• Move to JuliaLang org, add it as an external stdlib so we can keep separate issues/testing.

[IanButterworth]

First we'll wait until we have a use case ready to go, so we're sure we will actually use this in Base.

JuliaLang/Pkg.jl#2793 appears successful, bar some unrelated windows io issues that the intensive testing uncovered

Move to JuliaLang org, add it as an external stdlib so we can keep separate issues/testing.

It was also suggested in the above PR to rename it to PidFiles.jl? @vtjnash what do you think? Once that's done I can update this PR

[IanButterworth]

Bump @vtjnash

[StefanKarpinski]

Also: can we call it PidFiles instead of Pidfile?

[vtjnash]

Renaming packages sounds quite hard

[IanButterworth]

juliahub stats for reference https://juliahub.com/ui/Packages/Pidfile/wlmRx/1.2.0?page=2

Perhaps instead of moving the repo to JuliaLang, a new repo is created there called PidFiles and Pidfile could be deprecated?

[IanButterworth]

bump. (I thought there was support for this idea, but it feels a bit like I'm dragging it along..)

[Sacha0]

Echoing Ian, friendly bump :). IIUC JuliaLang/Pkg.jl#2793 depends upon this work, and #42405 depends upon that, which is biting builds on master.

[StefanKarpinski]

Agree that making a new PidFiles stdlib and deprecating the Pidfile package seems good. Let's just do that. Mixing stdlib and non-stdlib versions is ok but probably better to avoid since we'd like a name change anyway.

[fredrikekre]

Maybe I missed something, but why not just include the functions in e.g. Base.Filesystem? A whole new stdlib seems pretty "heavy" for this when it is just a couple of functions. In addition, I thought there were plans to use this for code loading and then it can't be in a stdlib anyway(?). (Edit: Also avoids the need of a name change and the potential confusion with both Pidfile and Pidfiles existing.)

[StefanKarpinski]

Yeah, that's actually also a good approach.

[IanButterworth]

Just for the record, that was the original proposal, albeit FileWatching rather than Filesystem #42471

The point was raised in triage about the julia repo issue/PR overload, so adding it as an external stdlib would be more aligned with helping that.

See #42571 (comment)

[fredrikekre]

Okay, but how many issues/PRs can an extra ~150 LOC generate?

[IanButterworth]

I guess this needs another discussion in triage?

[StefanKarpinski]

Does it depend on stuff in FileWatching?

[IanButterworth]

Indeed https://github.com/vtjnash/Pidfile.jl/blob/fa5b193df95c67cdb8cf148c52513ecb25783565/src/Pidfile.jl#L210

[IanButterworth]

Summary for triage:

The plan from the previous triage call was #42571 (comment)

• First we'll wait until we have a use case ready to go, so we're sure we will actually use this in Base.
• Move to JuliaLang org, add it as an external stdlib so we can keep separate issues/testing.

But that plan stalled at moving the repo, then the idea to rename Pidfile to PidFiles was blocked because of existing dependents.

The suggestion to deprecate vtjnash/Pidfile.jl and start a new repo JuliaLang/PidFiles.jl seemed reasonable to me.

Then objection was made to adding a new stdlib for ~150 LOC, and the original PR was brought up #42471

It seems to me the options boil down to:

1. Add a new stdlib, JuliaLang/PidFiles.jl which is a clean new repo (deprecate Pidfile.jl). Keeps any issues separate
2. Absorb the code into an existing stdlib e.g. #42471. Issues are tracked in the main julia repo
   • Might be backportable to 1.6?
   • The stdlib it's absorbed into would likely eventually move to an external repo, as I believe that's the general plan?

[vtjnash]

triage says PidFile makes sense to rename it to, though I don't know how to do that, I have opened an issue here vtjnash/Pidfile.jl#10

[fredrikekre]

Why is the name so important? I think renaming will just be confusing.

In any case, if this is added as a stdlib it can't be used for e.g. code loading anyway(?), so then it would only be Pkg that will make use of it? In that case, why not just vendor the 150 LOC into an internal Pkg module and let other users be unaffected. Stdlibs are the worst, IMO we should make everything we can to avoid making more packages into stdlibs.

[IanButterworth]

if this is added as a stdlib it can't be used for e.g. code loading anyway(?)

That's not true apparently. It was pointed out in triage that things like this are already done.

Why is the name so important? I think renaming will just be confusing.

Triage concluded that adding a new stdlib that breaks the PackageNamingConvention would be odd.

There was a lot of discussion in both triage calls. I can summarize further if needed, but I think the triage process is generally to move forward once triage has concluded?
Otherwise we'd be breaking process twice here.

Sorry if that suggestion is frustrating @fredrikekre, I suggested re-triage and wrote the summary up hoping that interested parties would be able to join the second triage call.
I just want to get this moved forward.

The course of action I see is:

• Make a new repo at JuliaLang/PidFile.jl with the contents of vtjnash/Pidfile.jl. New UUID, so different package. (@vtjnash I can do that if you give this a thumbs up)
• Deprecate vtjnash/Pidfile.jl (there's only two direct dependents registered, as far as I can tell from juliahub https://juliahub.com/ui/Packages/Pidfile/wlmRx/1.2.0?page=2)
• Update this PR
• Get the benefits into 1.8 so we can fix issues like https://github.com/JuliaLang/julia/issues/43286 and potentially the ideas around making parallel precompilation a background process??

[fredrikekre]

Triage concluded that adding a new stdlib that breaks the PackageNamingConvention would be odd.

REPL could be REPLs, SHA could be SHAs etc.

Let me ask like this then: What will become better with this as a stdlib compared to vendored in Pkg? (I just really dislike stdlibs, thats where packages go to die.)

[IanButterworth]

Things that have been discussed

• fix Pkg usage file handling
• make precompilation background & async with code loading
• avoid unnecessary re-precompilation when un precompiled packages are loaded on distributed workers that share the same depot

There may be more?

[fredrikekre]

Right, there are a number of cases in Pkg were this would be useful, but that would have been fixed by vendoring it too.

That's not true apparently. It was pointed out in triage that things like this are already done.

How? For e.g. Distributed when using --procs I guess? But with pidfiles it would be different, IMO, because Julia would be broken without this stdlib (if you can't load packages), and that seems to suggest that it should be part of base and not pretend to be an external thing?

If this is added as a stdlib people that wan't to support LTS can't use it unless it is also released as a new package in General. Then we have three versions of the package: the existing Pidfile, the stdlib PidFiles, the regular package PidFiles. This also means that bugfixes can only be fixed every ~3 months and new features every ~6 months or something. If this was vendored in Pkg 2 months ago all the Pkg usecases could already have been fixed, and users of the package Pidfile would never be affected by it.

[IanButterworth]

Putting it in Pkg doesn't seem right to me given at least @vtjnash has expressed interest in using it elsewhere in Base/other stdlibs. Perhaps more that I don't understand.

Into FileWatching makes more sense I think? It's a smaller stdlib that makes sense to be used in other stdlibs & Base?

The point about adding more PRs/Issues to the base repo could be abated if in the same move FileWatching is converted to an external stdlib?

Although, I think there was objection to growing existing stdlibs?

[fredrikekre]

Putting it in Pkg doesn't seem right to me given at least @vtjnash has expressed interest in using it elsewhere in Base/other stdlibs. Perhaps more that I don't understand.

Sure, but if the plan is to use this for this to be used for such important things as precompilation and code loading it seems like it should be properly integrated into Base instead of using some hack to load it from an external stdlib? An additional benefit of that is that the API doesn't have to be stable. Putting this in a stdlib will essentially cement the package until Julia 2.0. This could be very annoying if there are necessary changes needed for Pidfile in order to make it work with all the usecases envisioned above?

[IanButterworth]

The package is dependent on FileWatching, so putting it in Base would still introduce using a stdlib in Base

[IanButterworth]

I think someone needs to make a decision here. I don't feel qualified to do so and I'm hesitant to suggest putting it to triage again.

[IanButterworth]

A last hope bump before the 1.8 cutoff. Does anyone see a clear strategy here that everyone can live with?

[fredrikekre]

Since the only current usecase is Pkg I think it should be vendored there for now and this can be revisited once there are other usecases.

[IanButterworth]

Ok. That sounds reasonable and is prepared here JuliaLang/Pkg.jl#2793

[vtjnash]

We also need this for loading.jl, and I would not be surprised if there are other users already.

[IanButterworth]

Options:

• Add as a Base submodule
• Add to FileWatching stdlib (and at some point make it an external stdlib)
• Add as new ext stdlib
• Add as new ext stdlib with new name PidFile and new UUID
• Vendor in Pkg

[vtjnash]

After discussion in triage, we concluded that it fits best into FileWatching, being small and heavily dependent on it already anyways

[IanButterworth]

Replaced by #44367