RFC: Official deprecation policy

[IainNZ]

Inspired by discussion here and elsewhere. First stab at writing said policy.

I think this "policy" is simple and unlikely to cause issues, and will provide a stable mechanism to trim dead packages from METADATA over time. If packages are truly abandoned (i.e. the maintainer isn't willing/able to do this themselves) we can always tag a new release ourselves (and file an issue with the maintainer so they will be aware of the issue).

The main thing that needs to be clarified are when exactly things get done. The options are

• release/one Tag max version as current Julia minor release, remove packages when next minor release comes out (aggressive) (e.g. tag v0.2 as max, remove when v0.3 released)
• release/two Tag max version as current Julia minor release, remove packages when next-next minor release comes out (less agressive, my favourite) (e.g. tag v0.2 as max, remove when v0.4 released)
• unstable/one Tag max version as current Julia minor unstable release, remove packages when next minor release comes out (less aggressive) (e.g. tag v0.3 as max, remove when v0.4 released)
• unstable/two Tag max version as current Julia minor unstable release, remove packages when next-next minor release comes out (slow) (e.g. tag v0.3 as max, remove when v0.5 released)

I think release/two is best because if it really is deprecated then presumably no one is using it right now, so no need for it to work on the unstable version - tagging max version as release won't cause any grief. Anyone still using it when two more versions come around shouldn't be surprised when a package is removed.

There is a question of whether things ever need to actually be removed from METADATA - if the max version is set, you can't add it anyway, so its effectively dead. I'm in favor of removing them to avoid ambiguity to the passing observer but up for discussion.

[ivarne]

There should probably be some standard way of marking a package as deprecated in METADATA as well. The package list would benefit and users could get a warning when they do Pkg.status() and Pkg.update().

[IainNZ]

@kmsquire would be good to hear your thoughts

[kmsquire]

@IainNZ, thanks for doing this.

Ideally, I'd prefer not to have packages removed from METADATA at all. If an unmaintained package still works with, e.g., Julia v0.2, I'd like to be able to download that version of Julia and install the package easily.

This is important when trying to replicate results from a slightly dated scientific paper, for example.

The downside is that METADATA.jl would get crufty, and I think the only way to solve this is to bump the version of METADATA.jl itself, where earlier versions of Julia would point to the older version.

The BioConductor R repository is a good example of how this could work.

Thoughts?

[johnmyleswhite]

I think Kevin's right to argue against removing packages from METADATA. I think our desire to have some kind of quality control is best handled by a secondary layer on top of METADATA, which recommends the best package for any given purpose.

[IainNZ]

@kmsquire @johnmyleswhite agreed, removing things from METADATA is probably unnecessary, and the versioning-for-replication is indeed a topic I care about.

I guess the part about "removing from METADATA" in the text above could be - we bump METADATA to a new branch for the new release and only bring non-deprecated packages with us. But then I worry we will go back to that awkward phase of having to run a script to back-populate METADATA.

[kmsquire]

I think that, for now, we can forgo the back-populate script. There are at least two alternatives I can think of:

1. Only bump older METADATA versions explicitly, upon request. Package maintainers can submit pull requests for this if they want to support a previous METADATA.jl version
2. Allow the Pkg.publish() framework to create pull requests for older metadata repos

These are not necessarily mutually exclusive, and either way, users can always check out the latest master by hand (although that's not guaranteed to work with older versions of Julia).

[StefanKarpinski]

Quoth myself:

If you remove a package, you have to remove all versions of other packages that depend on it – otherwise those versions will obviously be broken and uninstallable. I'm not sure how else things could possibly work. What you can do if you want to remove a package entirely is simultaneously remove all the versions of other packages that depend on it, or if they didn't really depend on it, remove the dependency from their requirements. Then everything should work.

[kmsquire]

What you can do if you want to remove a package entirely is simultaneously remove all the versions of other packages that depend on it, or if they didn't really depend on it, remove the dependency from their requirements. Then everything should work.

Say a user uses a package to implement a solution to a problem, and simply wants to continue using that version of Julia with that package. She probably wouldn't be too happy to find the package removed from METADATA, when she gets a new computer or her colleague attempts to install and use the same solution.

This is not just a hypothetical scenario--I've see it happen quite frequently in academic bio labs where users come up with solutions to problems and continue to use those same solutions for years, using the same old versions of code (until it breaks for good and they have to fix it).

[StefanKarpinski]

So my question would be if a version of some package is perfectly functional, why are we removing it?

[StefanKarpinski]

I think the issue of reproducing old setups is orthogonal to the METADATA stuff. You basically want to be able to snapshot a set of packages and maybe julia itself and get back to that exact setup easily. I was aiming for that as part of the original Pkg design, but it ended up getting in the way and getting discarded. But I do still think we need some mechanism for that.

[kmsquire]

So my question would be if a version of some package is perfectly functional, why are we removing it?

General reorganization. The current maintainer of a package doesn't want to maintain it anymore. It only works on an old version of Julia. The package functionality is being merged with other packages. There's any number of valid reasons why a function could still be perfectly functional (for some configuration/version of Julia), and someone could want to remove it.

Anyway, I'm just suggesting that whenever a package is removed from METADATA, someone's setup will break, and that I think there are ways to avoid this.

[ntessore]

Just dreaming here, but imagine how great it would be to share code if you could fingerprint your current configuration (Julia, packages, the whole deal) and then store that configuration in a central repository. Ideally, if everything is still available from some git repositories, this would allow you to publish the fingerprint and someone else could replicate your results in a matter of minutes.

[ivarne]

@ntessore You share that dream with most of the Julia developers. To get "the whole deal" to work, you'd need a virtual machine environment, and you need to transfer gigabytes rather than a fingerprint. There isn't really any way around that, because we rely on system software. The good news is that virtualization has always worked and will always work.

We currently have a package manager that is awfully slow. We think this is because we fork out to command line git, and yesterday a PR was merged that starts the process of getting libgit2 included in Julia, so that we can rewrite the package manager and it will hopefully be faster. When this is finished there will likely be more interest in adding features like "per project dependencies" and other virtualenv like features.

[timholy]

I'm not sure it's as big as all that: wouldn't versioninfo() and Pkg.status() basically give you the information you're asking for? It would be just a question of writing a couple of tools to go back to that state. (@ntessore, I encourage you to dive into base/pkg and see if you can implement them.)

[ntessore]

@timholy This is what I (maybe naively) thought as well: Go through the installed packages, get the current state of each repository, write it to a dictionary (basically Pkg.status() with commits instead of versions). Once those are gathered, create a gist (?) with this dictionary. For going back, maybe create a clean directory under ~/.julia, switch to it somehow (?) and Pkg.add() each item in the dictionary at the given commit. I think I will try that, if only for education.

EDIT: Of course, this ignores how to manage the base julia version. No ideas how this could be done.

[timholy]

As long as the julia commit is recorded (use @which versioninfo() to see how to get at that), then anyone who is using a source build you could just check out the relevant commit and build it.

In bisecting #8631, I did notice that julia 0.1 does not seem to be buildable anymore, due to what appears to be changes in zlib download URLs. So it may take some effort to resurrect old versions.

[ntessore]

I thought more about this. I don't suppose there is any way to separate the Julia runtime from the Julia standard library? In that case, going back to any Julia version would simply mean checking out the corresponding version of the standard library (if the runtime is kept backward compatible, which seems easier to achieve).

[quinnj]

I wonder if this is relevant any more? I'm not sure if we have anything documented (in which case this would be nice), but I believe our current practice is to try and transfer ownership of abandoned packages, otherwise move them to the JuliaArchive organization, along with tagging final releases and adding final deprecation warnings. Anything else I'm missing?

[tkelman]

What's missing here is applying a Julia upper version bound to a single new tag doesn't actually accomplish the desired result of deprecating the package. To mark a package as deprecated, all past tags of it in METADATA need to have Julia upper version bounds added to them, otherwise the Pkg resolver will just say "oh the latest tag doesn't work on my current Julia version, I'll use the second-to-last tag of the package instead."

As we saw with Grid.jl, when deprecating a package all of its dependencies dependents effectively get deprecated by proxy, so it's best to give all your dependent package authors fair warning and guidance on a transition plan to use something else before doing this.

[ViralBShah]

Maybe close this?