[BUG] IJuliaExt causes problems with IJulia in default environment

[oschulz]

Details

IJulia is a bit peculiar in that you can really only have one version of it at the same time, otherwise one get's these "IJulia version doesn't match ..." warnings. This is because the (default) Jupyter kernelspec is bound to a specific IJulia version (since "kernel.json" points to ".julia/packages/IJulia/SOME_VERSION/src/kernel.jl"). So one (at least me) typically keeps IJulia in the default (e.g. "v1.9") environment and out of specific project environment like the the "Project/Manifest.toml" kept next to a notebook.

This leads to the following problem with the current Plots version on Julia v1.9, though:

using Plots

[ Info: Precompiling IJuliaExt [2f4121a4-3b3a-5ce6-9c5e-1f2673ce168a]
ERROR: LoadError: ArgumentError: Package IJulia [7073ff75-c697-5162-941a-fcdaad2a7d2a] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

This is very inconvenient, the only solution seems to be having IJulia in the notebook environment, but this would require having a separate Jupyter kernel for each notebook environment as well (or at lease having quite a few kernels). It's also really confusion to (esp. newer) users, because they can run using IJulia in the same notebook before using Plots cell without error, so IJulia is "there" (just not in an extension-compatible way).

IJulia is special here because it's automatically loaded without being part of the notebook, and because the "kernel.json" issue, Plot's other Pkg-extensions don't have to be this problem.

I would suggest the we handle IJulia via Requires again. Most (all) ..._ijulia_... function in "IJuliaExt.jl" could be defined in Plots itself, since they don't depend on IJulia, so Requires would only need to load a few lines IJulia-specific code.

Backends

Not backend-specific.

Versions

Plots.jl version: v1.38.9
Backend version (]st -m <backend(s)>): Not backend-specific
Output of versioninfo():

Julia Version 1.9.0-rc2
Commit 72aec423c2a (2023-04-01 10:41 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)

[BeastyBlacksmith]

I tend to agree. @t-bltg what is your opinion?

[t-bltg]

Thanks for the detailed explanation.
I do not use IJulia myself, so my port of the IJulia code to weak deps was probably naive even if I remembered having checked this, probably too lightly. We should then revert to Requires if there is no other choice, with added comments in the code to notify developers why we can't do otherwise.

[t-bltg]

So AFAIK you rely on environment stacks in order for the kernel to find IJulia, is it something common ? As you say, normally, if your project uses IJulia, it should IMO be put in the project Project.toml.

[oschulz]

So AFAIK you rely on environment stacks in order for the kernel to find IJulia, is it something common ? As you say, normally, if your project uses IJulia, it should IMO be put in the project Project.toml.

Indeed it should. The problem is, one often has quite a few Project.toml files (different deps in different projects/notebooks), and so they will require different IJulia versions (esp. with tutorials, that have longer-lived Project.toml/Manifest.toml files, of with output "archived" for data preservations resp. scientific reproducibility). Unfortunately. only one of these IJulia versions can be referred to by the Jupyter IJulia "kernel.json" file (few people use lots of separate custom kernel files). This results in warnings if the IJulia version used by a given project differs from the one in the Jupyter "kernel.json" - and can result in trouble if the versions are too different.

Keeping IJulia in the default environment and out of project-specific environments has worked well for me so far as a way of avoiding this. It's also natural approach, I would say, because the notebook code rarely used IJulia, so it's not really a dependency of the project. IJulia is unique here, I would argue, since it get's injected as an additional dependency in notebooks (without any import IJulia in the code), novice users might not even be aware of it happening.

[BeastyBlacksmith]

Should be fixed on julia 1.9 with JuliaLang/julia#48352 I think. We probably should update compat once that is released.

[oschulz]

Should be fixed on julia 1.9 with JuliaLang/julia#48352 I think.

Oh, that is very nice!

[oschulz]

Seems fixed now with v1.9-rc3.

[oschulz]

I was mistaken, the problem persists in Julia v1.9-rc3. :-(

[AndrewSwann]

Also a problem in Julia v1.9.0

[svilupp]

Same error as described above on 1.9.0.

We see it everywhere, because we use Quarto to produce our stakeholder reports (which uses IJulia under the hood).

What are some temporary solutions?

• Downgrade to some version before the EXT were introduced?
• Is there a way to block the loading of the extension? Preferences? (probably negatively affecting functionality)
• Or is it simply migrating to Plotly/Makie world?

[oschulz]

Can we just use Requires for now until this is sorted out on the Julia side? This issue is really annoying and hits new users especially hard.

#4749

[t-bltg]

Where is the julia issue tracking this ?

[oschulz]

I guess it was supposed to be JuliaLang/julia#48352, only for some reason that doesn't seem to do the trick ...

[t-bltg]

Someone should open a dedicated issue in julialang/julia in order to make upstream Julia devs aware of this problem which might not have been taken into account while designing extensions.
Else, saying "until this problem is fixed on the Julia side" sounds like never to me.

[oschulz]

I'll do it.

[t-bltg]

@KristofferC , is there a workaround for this problem in the extensions world it should we fall back to using Requires again on 1.9 ?

[oschulz]

If there's no other workaround I think #4749 should do the job for now. I play a bit with it, TTFP doesn't noticeably increase (subjectively).

[KristofferC]

I'll have to dig into what is going on here before being able to provide a fix/workaround. For now, going back to Requires if that works seems like the best solution indeed.

[KristofferC]

If you want to keep the extension, I think a workaround is to replace using IJulia with

const IJulia = Base.require(Base.PkgId(Base.UUID("7073ff75-c697-5162-941a-fcdaad2a7d2a"), "IJulia"))

[t-bltg]

Thanks.
@oschulz , can you confirm that this fixes your issue and update #4749 accordingly? I would be much more inclined to keeping the extension with a workaround than going back to Requires.

[oschulz]

Sorry for the delay - @KristofferC , your workaround seem to do it's job perfectly! :-)

@t-bltg I've updated #4749 accordingly.

[oschulz]

I think we can close this now. :-)

[oschulz]

@BeastyBlacksmith could we release a Plot v1.38.14 to get this fix out? There is e4f1978 but Registrator doesn't seem to have been run on it.

[t-bltg]

JuliaRegistries/General#84406

[oschulz]

Thanks @t-bltg !

[nilshg]

I'm seeing this again in 1.10-beta3, IJulia 1.24.2 (in default env) and Plots 1.39.0:

[svilupp]

This sounds more like JuliaLang/julia#51280 (comment)

[nilshg]

Ah yes it might well be, I saw the pipe_writer world age error for precompilation of other packages (I think it has snuck back into 1.10-beta3?), didn't clock that IJulia extension might trigger it (presumably as that doesn't get precompiled when the environment is precompiled in the REPL which I do to work around the IJulia bug).

[svilupp]

Yes, it's any precompilation. Btw. it has been fixed but the fix hasn't made it into beta3 (see the issue). So, hopefully, it will make it to beta4!

A simple fix that works most of the time is to call Pkg.precompile() proactively.