julep: conditional modules

[vtjnash]

currently, some modules (incorrectly and unfortunately) use Pkg.installed to decide whether it should include some additional functionality. I would like to propose the following extension to module initialization that could help simplify this situation of optional dependencies:

---

Rules:

1. During module parse time, a module can declare a dependency on another module, given by name
2. Following the closing of a module (without error), and after all __init__ functions have run, any modules that declared module B require Main.C will be (re-)loaded.

Example:

module A
  module B requires Main.C
    # code in here is part of A iff Main.C is defined as a module
    # it's execution time is defined as after the call to A.__init__ and C.__init__
    using Main.C
    import ..A.something
    something(::C.CType) = 2
  end
  # code in here is always part of A
  something(::Any) = 1
end
# Accessing A.B here would throw an UndefRef-like error, with a message that B requires C
module C
  type CType end
end
# Now A.B exists, and A.something() has extra functionality

[JeffBezanson]

Using modules to delimit optional code seems like a potentially very good idea.
It's not good for it to inherently require reloading code though.

[timholy]

I was mentally going over similar ground yesterday. Would it be reasonable to declare that interacting code can't be loaded until after the types are defined?

[ivarne]

Very similar to #2025

I do not get the crucial point here. Why does anything need to be reloaded? I would assume that most of this could be solved with a defer_require(mod::String, file::String) method that defers loading of the file until a module with name mod is loaded into Main.

[vtjnash]

The comment about reloading would only be triggered in the case where the user replaced the module. However, perhaps that condition does not need to be defined.

Using modules to define the separation allows Julia to pre-compile and cache the results (in the future). Thus, it is almost just as flexible, but it can be optimized to be fast.

[StefanKarpinski]

I wonder if parametric modules are relevant here. Consider that Main.C could be viewed as a parameter to A.B – you're declaring, in effect, a module template A.B{C} where the actual C to specialize that template on is to be provided later by some other code.

[vtjnash]

@StefanKarpinski that sounds rather clever. however, who would be responsible for instantiating the necessarily templates?

[StefanKarpinski]

Julia?

[vtjnash]

I mean, how would Julia know which templates to instantiate if the user wrote:

module X{A}
end

what is A allowed to be?

[StefanKarpinski]

That would be up to the user. If they wrote using X{Foo} then A would be Foo. This would inherently allow multiple instances of modules.

[quinnj]

What's the reasoning of using modules for optional code vs. arbitrary blocks of code? Something like:

when A
    using A
    # do stuff with module A
    # define functions, call methods, optionally wrap them in a module, etc.
elsewhen B
    using B
    # if A isn't loaded, but B is; do stuff with B
else
    # fallback if A or B can't be found; could just throw a "Need A or B error"
end

There's probably some good implementation detail to require modules, but I can imagine there are a lot of cases where you really only need a few lines, so not needing the submodule would be convenient.

[vtjnash]

1. modules are the only interface we have for reloading code (relatively) cleanly
2. this functionality has nothing to do with what modules can be found (state), and operates instead against the event of loading a module. an else clause is meaningless in such context.
3. creating a new module is the only interface we have for adding entirely new exports & functions after 'closing' a module

this is all covered in the discussion on the pull-request thread

[jiahao]

I've run into a use case which could really benefit from conditional package loading in JuliaLinearAlgebra/IterativeSolvers.jl#35

[rleegates]

Any progress on this topic? I really feel the need for conditional imports :)

[StefanKarpinski]

I've been wanting this to be merged for about six months now.

[mbauman]

Requires.jl is a very nice stop-gap that has worked well for my purposes.

[rleegates]

What's standing in its way? I'm having trouble seeing how your pull request helps my issue. Basically I have a module Big that does a bunch of stuff even when a dependency Dep is missing, however if it's installed, additional functionality is provided. As far as I understood your pull request, the whole module Big requires the dependency, if it's missing, the module Big isn't loaded. Is this interpretation correct?

[StefanKarpinski]

@vtjnash, have you abandoned this issue? You haven't commented on it since almost a year ago.

[vtjnash]

not entirely, i was just finally getting back to even older issues (#2818) and have no shortage of other unresolved proposals (https://github.com/JuliaLang/julia/pulls/vtjnash)

[sbromberger]

Yeah. We had to break out a bunch of things into a separate module in order to keep LightGraphs from imploding due to fragility issues with all the dependencies. We also now have a policy in place with respect to PRs that severely restricts what we allow in LightGraphs now (not a good thing):

• PRs introducing dependencies on core Julia packages are ok.
• PRs introducing dependencies on non-core "leaf" packages (no subdependencies except for core Julia packages) are less ok.
• PRs introducing dependencies on non-core non-leaf packages require strict scrutiny and will likely not be accepted without some compelling reason (urgent bugfix or much-needed functionality).

Conditional dependencies would go a long way to reducing the problems of "dependence fragility" within Julia packages. (We don't want to get to a point where Node found itself a few months back.)

[tkelman]

#15705 is the latest thinking. needs an implementation that ties into precompilation correctly. in the meantime, when it comes to conditional dependencies, either disable precompilation or don't do things conditionally. Make separate packages for features that have additional dependencies that won't always be needed by a "main" package.

[StefanKarpinski]

The latest latest thinking (based on a discussion @JeffBezanson, @vtjnash and I had last week), is that what we want is very similar to what I originally proposed in #2025 – i.e. have a mechanism whereby packages, when they're loaded, can register a module for later loading once some set of modules have all been loaded. Note that if you want code for A that depends on both B and C then you'll want to wait until all of A, B and C have been loaded, so this isn't just a binary relation.

[tbreloff]

This is the "Requires" model, and if that was built-in and played well with
precompilation it would be a big win!

The other use case is when you want to do something only when a package is
installed (not just when it's explicitly loaded).

I think both uses would share the tricky parts of the implementation,
namely how to defer code execution until some event, and how to make it
play well with precompilation. I just hope the "on install" event is also
able to be supported, in addition to the "on import" event.

On Tuesday, October 11, 2016, Stefan Karpinski [email protected]
wrote:

The latest latest thinking (based on a discussion @JeffBezanson
https://github.com/JeffBezanson, @vtjnash https://github.com/vtjnash
and I had last week), is that what we want is very similar to what I
originally proposed in #2025
#2025 – i.e. have a mechanism
whereby packages, when they're loaded, can register a module for later
loading once some set of modules have all been loaded. Note that if you
want code for A that depends on both B and C then you'll want to wait until
all of A, B and C have been loaded, so this isn't just a binary relation.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#6195 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA492sUprUHIe7TxkDpchkQBPG_zmm6xks5qzB0SgaJpZM4Bqetc
.

[sbromberger]

The problem we had with Requires.jl (and the if defined :packagename workaround) was that any use of macros from an external package meant that it couldn't be conditionally loaded. This affected us primarily because of JuMP.

[tbreloff]

any use of macros from an external package meant that it couldn't be conditionally loaded

But wasn't that because Requires was an external hack to try to get the functionality we wanted? If this was built into the core of the language, presumably macros could be deferred until they were ready to be run.

[JeffBezanson]

A bit more on @StefanKarpinski 's description above: #15705 (comment)

[richardreeve]

I see this has been moved to a 1.0 milestone, so I presume this is a way off now, but is it still in someone's plans, because it would be incredibly useful!

[StefanKarpinski]

Yes, we really need this in 1.0 – it's a big problem for the package ecosystem currently.

[richardreeve]

Good to know, thanks.

[richardreeve]

@StefanKarpinski So does this mean conditional modules are not going into 1.0 any more? Or are they already implemented somewhere?

[hayd]

@richardreeve see #2025 (comment)