cli: Add infrastructure for new CLI drivers juliax and juliac

[Keno]

Following the discussion in #50974, it became clear that there is significant objection to changes to the behavior of the julia CLI driver. Some commenters found changes to the behavior acceptable if they were unlikely to impact existing code (e.g. in the case of #50974 using __main__ instead of main), while other were worried about the reputational risks of even changing behavior in the corner case. In subsequent discussion it became relatively clear that the only way forward that did not raise significant objection was to introduce a new CLI driver for the new behavior. This may seem a bit drastic just for the change in #50974, but I actually think there's a number of other quality-of-life improvements that this would enable us to do over time, for example:

• Autoloading/caching of all packages in the manifest
• auto-selection of julia versions (integrate juliaup?)

In addition, it doesn't seem so bad to have some CLI design flexibility to make sure that the juliac driver is aligned with what we need.

This PR is the minimal infrastructure to add the new drivers. In particular, it does the following:

1. Adds two new cli drivers, juliax and juliac. At the moment, juliac is a placeholder and just errors, while juliax behaves the same as julia (except to error on the deprecated --math-mode=fast, just as an example of a behavior difference).

2. Documents that the behavior of julia (but not juliax or juliac) is pat of the julia public API.

3. Switches the cli mode based on the argv[0] of the binary. I.e. all three binaries are identical, except for their name, the same way that, e.g. clang and clang++ are the same binary just with different names. On Unix systems, these are symlinks. On windows, separate copies of the same (small) binary. There is a fallback cli option --cli-mode that can be used in case the argv[0] detection is not available (e.g. for some fringe embedded use cases).

4. There is currently no separate -debug version of the new drivres. My intention is to make this dependent on the ordinary debug flags, rather than having a separate driver.

Once this is merged, I intend to resubmit #50974 (chaning juliax only), and then finish and hook up juliac shortly thereafter.

[PallHaraldsson]

juliax behaves the same as julia (except to error on the deprecated --math-mode=fast, just as an example of a behavior difference).

What are some other examples, and more importantly, can I propose some more, since "experimental" anyway?

It seems juliax is meant for scripting and/or is experimental (for now), so I suggest --startup-file=no as its default, at a minimum. I was shot down previously on that new default (even for non-interactive).

Since it's experimental, not committing to an API(?), does that imply 2.0, or allowing some such? I then suggest a radically smaller sysimage for it, e.g. Pkg out right away (I've done it, it's not difficult if you just want to disable it), and then julia implies juliax plus doing:

using Pkg
and/or
using Julia1 # stdlib, which can imply Pkg.

If this is ok, then I would suggest excising LinearAlgebra too (and OpenBLAS, neither very useful for scripting, both recovered with using LinearAlgebra and/or using Julia1). BLIS.jl (and/or MKL.jl) is actually better than OpenBLAS, so another reason to get rid of it, also for Juliac it seems... I would even like to excise Array (Julia1 and/or LinearAlgebra would add it), just keep the new Memory, except to construct 1D Vector, possibly leave out push!, at least for nD Array. Would excising Array make the sysimage smaller radically? I'm not sure, I suspect Julia itself doesn't need Array, as opposed to Vector and Dict, that better uses the new Memory type.

On windows, separate copies of the same (small) binary.

If this is meant for scripting, then the separate console and non-console apps are encoded into the .exe, so they can't be the same, thus there is at least that need for two for Julia source code, as opposed to compiled.

[Python has a separate .pyw ending available for "Windows console" apps. I.e. Windows needs two types of apps, and thus two drivers. It's unclear to me if Julia needs a new ending or just such a driver, or maybe if PackageCompiler.jl covers this. And now juliac, it that its future replacement?]

Autoloading/caching of all packages in the manifest

Great future plan, does that mean you would potentially download compiled pkgimages for packages from the Julia registry, not just source code not precompiled?

Typos "pat of the julia public API", "drivres", "chaning".

juliac - Compiler driver for the julia programming language

juliax - Experimental driver for the julia programming language
--cli-mode
In juliax, this option is an error

I'm confused, is there any need to error? Rather just silently ignore? Or both could allow on and off, just with different defaults?

[MasonProtter]

I'm a bit sad to see this being bundled into a new CLI driver instead of an entrypoint marker or something else where the meaning is clear in the code itself.

[Seelengrab]

In addition to the entrypoint point (which I agree with), I have two additional things:

Does this prevent a REPL-then-deploy style workflow? I think I mentioned it elsewhere, but I'm currently able to compile a binary for AVR/microcontrollers from the REPL, change a thing in my source code, reload through Revise, inspect existing IR, and only have to (pre)compile the new addition, keeping compilation extremely snappy. That's something traditional AoT compiled languages had to fight hard for to get from their toolchains, and is a workflow I'd very much like to keep. So even with a seperate driver, I doubt I'd use it (except for CI deployment, I guess?).

Documents that the behavior of julia (but not juliax or juliac) is pat of the julia public API.

What does this mean? Which "julia" is this referring to, if that is supposed to "go away" in the future (be replaced by juliax)?

[Keno]

What are some other examples, and more importantly, can I propose some more, since "experimental" anyway?

At present there are none. Yes, you may suggest additional behavior, but please not in this issue. The CLI is expected to be unstable for multiple releases.

[Keno]

I'm a bit sad to see this being bundled into a new CLI driver instead of an entrypoint marker or something else where the meaning is clear in the code itself.

There's no instead here. Entrypoints in projects, multiple entrypoints, complex deployment setups are all build system concerns (which for us is Pkg probably) and should be addressed there.

[Keno]

What does this mean? Which "julia" is this referring to, if that is supposed to "go away" in the future (be replaced by juliax)?

The current julia driver. The point is to document that it won't go away or change behavior in 1.x since that was everyone's complaint.

[DilumAluthge]

I think it's okay to say that the documented behaviors of the julia driver are public API, since that's consistent with the rest of this FAQ entry (which says that documented behaviors of public symbols are public API). But "semantically observable behaviors" seems too vague and hard to define.

Suggested change

	In addition, the documented and semantically observable behaviors of the `julia` CLI driver
	In addition, the documented behaviors of the `julia` CLI driver

[Keno]

That would have prevented the --math-mode=fast removal.

[DilumAluthge]

I think I'm misunderstanding what you've written here. Which of the following interpretations is correct?

Interpretation 1:

• The documented behaviors of the julia driver are public API.
• The semantically observable behaviors of the julia driver are public API.

Interpretation 2:

• A behavior of the julia driver is public API iff the behavior is documented and the behavior is semantically observable.

When I read the above text, I interpreted it as interpretation 1.

[Keno]

Interpretation 2 is what I intended. In particular options that are non-semantic (optimization levels, fastmath, etc.) are not guaranteed to work the same across versions, even if the documentaiton tells you what they do.

[DilumAluthge]

Ah, I see. Interpretation 2 sounds good to me.

Let's reword the sentence to make it more clear?

[DilumAluthge]

How about something along the lines of:

A behavior of the julia CLI driver is only part of the public API if the behavior is documented and the behavior is semantically observable.

[DilumAluthge]

And it might also be helpful to add some form of the sentence that you wrote above, since I think it gives some good illustrative examples:

For example, options that are non-semantic (optimization levels, fastmath, etc.) are not guaranteed to work the same across versions, even if the documentation tells you what they do.

[mbauman]

It feels very strange to me to have two subtly different executables that mostly do the same thing. juliac I get: it'd be a hugely different workflow. Maybe it's hard for me to see past the color of a bikeshed to the bigger things, but this feels like blame-shifting for breakage: the only reason I'd not alias julia=juliax is if it breaks my code.

A key point in #50974 was that the name of the entry point should be easy-to-use and obvious. I don't want to be telling people that the feature they want is tucked away in an "experimental" executable that doesn't have stability guarantees. This seems to be throwing out the baby with the bathwater.

[Keno]

It feels very strange to me to have two subtly different executables that mostly do the same thing.

Well, this gives is freedom to diverge, but yeah, obviously this it not perfect, but there were enough people angry enough about the prospect of changing the behavior of julia, that I don't think it's tenable at this point. I think this is the best we're gonna get at this point. We really need to be moving past this particular change, so I can move on to the rest of the static compiler changes.

[MasonProtter]

I guess this just seems like a rather extreme approach, and as far as I can tell from reading the public conversations, it seems to be motivated by a purely aesthetic desire to write

function main(ARGS)
    ...
end

without any other signals that main is special.

[adienes]

is having a kinda-new-kinda-fragile-slightly-different-but-mostly-the-same-but-cant-remember-how juliax really more aesthetic than __main__?

now I must have two sets of code because I must be afraid that users won't know whether the code I delivered should be run with julia or juliax ?

[quinnj]

Sorry if this is a dumb or already-answered question, but what's the medium/long-term plan for juliax? Is it just to provide alternative option defaults?

I think I agree w/ @mbauman and @MasonProtter here that it seems a bit excessive. I get juliac for an entirely different workflow. I still like the __main__ idea though.

Just as an FYI, I'm probably the only person out there that juliax would break, but I keep my own alias of juliax pointing to a latest-master build of julia (normal julia points to current release; I have a few other aliases for various versions too). Not sure that really matters, but if we're brining up rare corner cases that affect users....?

[mbauman]

The uproar was nearly entirely over breakage with Main.main; there exist plenty of other colors to paint the bikeshed as it currently exists in julia without introducing a whole new safe confinement called juliax around the thing.

I know you didn't like __main__, but pretty much every argument against it in #50974 (comment) also applies to juliax. And there's a much bigger problem with juliax: you need to be able to convince someone that they're going to run their code with an "experimental" executable. But __main__ isn't the only option.

• A macro @main function foo(ARGS) could assert the arg list looks ok and assign const __main__ = foo in addition to defining foo itself as usual. The macro name could be painted different colors, as could the name that the function is assigned to.
• It could even be anonymous: @main function(ARGS) works just as well.
• The syntax function @main(ARGS) is available; making this also work as @main(ARGS) = println("hello world") would be more tenuous but is also doable with some magic.

[Keno]

motivated by a purely aesthetic desire to write

That sounds a bit reductive to me, but it's not wrong exactly. I played with a bunch of options, and that's the one that I thought worked best. Of course it's possible that I'm wrong and the design is dumb, but hey with this at least we can delete the cli driver again in that case.

is having a kinda-new-kinda-fragile-slightly-different-but-mostly-the-same-but-cant-remember-how juliax really more aesthetic than __main__?

I don't know about aesthetic, but several people objected to changing the behavior at all, even if the name is __main__, so it's not really an option.

Sorry if this is a dumb or already-answered question, but what's the medium/long-term plan for juliax? Is it just to provide alternative option defaults?

Well kind of, but some of the options would be considered to breaking to change as defaults in the julia driver, so the medium term plan is that juliax can evolve to meet current demands, while keeping julia the same for people who need stability. Down the road we might decide to add a cli versioning mechanism to juliax, or decide in 2.0 to replace julia by juliax.

[adienes]

I worry this will bifurcate / fragment the ecosystem yet more if you have some code that only works with julia and other code that only works with juliax

And there must be such code, because otherwise if everything behaves the same with each driver, no point in separating!

[Keno]

But __main__ isn't the only option.

Yes, the macro options were discussed, but people didn't like them, because it wasn't clear how the scoping and export of them would work. Also, there's scenarios where the main binding is not a generic function, which was unclear how it would be supported in that use case.

However, I want to re-emphasize again that the main thing isn't the only thing that the new cli driver is for. There's other changes I want to make to the behavior of the cli driver - the main discussion just made it clear that each one of them would be a huge fight. What juliax gives us is a few releases of reprieve to work out exactly how the cli driver should work. It's no different from Base.Experimental in that regard.

[Keno]

I worry this will bifurcate / fragment the ecosystem yet more if you have some code that only works with julia and other code that only works with juliax

I think this is a valid concern. I would be fine with disallowing juliax script.jl for the time being, unless script.jl has a juliax shebang line. @vtjnash had previously proposed autoswitching the CLI based on the shebang line, which I didn't like, but I think requiring it as an explicit opt-in is fine. That way any interactive juliax features can be explores as usual, but there wouldn't be a risk of not knowing which driver a script was for.

[mbauman]

Ok, I could be convinced. I may be missing the forest for a single __tree__ here. I see that this unblocks your static compilation work — which is great — but I also want to get that into julia proper and could see this as a slippery slope away from that.

[Seelengrab]

Yes, the macro options were discussed, but people didn't like them, because it wasn't clear how the scoping and export of them would work. Also, there's scenarios where the main binding is not a generic function, which was unclear how it would be supported in that use case.

Lots of people in this and previous PR seemed very amicable to the entrypoint idea, which would solve that problem handily without requiring bifurcation of the CLI. Maybe the discussions you're referencing happened elsewhere, in which case it would be great if you could share them in more detail.

There's other changes I want to make to the behavior of the cli driver - the main discussion just made it clear that each one of them would be a huge fight. What juliax gives us is a few releases of reprieve to work out exactly how the cli driver should work. It's no different from Base.Experimental in that regard.

This sounds like there are a lot of changes coming that rely on your exact version/implementation? It would be great if you could share those plans with the community before dismissing the (imo valid) concerns people have about the current approach. Otherwise it feels like you're trying to force us to accept your vision, which admittedly doesn't feel nice.

but I think requiring it as an explicit opt-in is fine. That way any interactive juliax features can be explores as usual, but there wouldn't be a risk of not knowing which driver a script was for.

That also sounds like something an explicit entrypoint declaration would handle amicably - I don't think there were big arguments against it in the previous PR, and lots of people 👍 comments mentioning it.

[Keno]

Otherwise it feels like you're trying to force us to accept your vision, which admittedly doesn't feel nice.

I'm gonna ignore this, but as I've said before, I really don't like this kind of comment, and I would again ask you to please refrain from it. It's just so unnecessarily confrontational. There's no sides here. We're all trying to work together to find a good design here.

This sounds like there are a lot of changes coming that rely on your exact version/implementation?

I don't have any concrete plans for additional changes, but there have been various such changes discussed over the years that we might want to do. I mentioned two of them in the original post, but there may be others.

Maybe the discussions you're referencing happened elsewhere, in which case it would be great if you could share them in more detail.

There was some discussion about it on slack, some discussion on the triage call, and I also talked to @vtjnash about it in person after he first proposed it, but the summary is basically the following. There's two ways that this could work

1. There is a special internal symbol that the macro just expands (the @main) idea.
2. The macro directly declares the entrypoint using some sort of internal mechanism.

The first is undesirable, because it's unclear how it should work with entrypoints that are not generic functions, and the import/export of the special internal symbol is weird. Obviously there can be syntax for it, but that's like 3 or 4 different special cases that need to be learned just to avoid using the name that everybody actually wants to call it.

The second option isn't great, because it's not clear what to do with conflicting entrypoint definitions and it again needs special syntax to support entrypoints that are not generic functions. It also doesn't support any form of entrypoint import/export or the interactive version, particularly in a world where the manifest is autoloaded.

[DilumAluthge]

If we change our mind in the future, we can just straight up remove juliax and/or juliac in future releases, right? Since we documented that they aren't public API?

So it seems that this PR doesn't lock us into anything that we can't change or get rid of in the future. And if it's going to be helpful to Keno and others as far as unblocking other work they want to do, I think it makes sense to move forward with this PR.

[DilumAluthge]

And the design discussions around the entrypoint macro (@main function f(args) etc) can still continue while separate experimental work is being done on juliax, right? One doesn't need to preclude the other.

[adienes]

a clarifying question for my own benefit:

is this intended to be

• latest and greatest, and those who are typically early adopters should expect to start using juliax pattern for most things, and only the users who need guaranteed stability should remain on julia

or

• parallel, experimental work necessary for some rather broad features which may or may not get folded back into the stable API in the future

I suspect this is one of those situations where the tone/intentions of a proposal can have a big impact on its reception, as it appears to me that much of the concern is uncertainty about the medium-long term vision

[tecosaur]

Two cents from me:

1. From the discussion so far, I can't see a clear reason to motivate juliax over julia -x say (and I would think the latter preferable).
2. Something I've admired for some time is how with Ion you can embed the dependencies of a script within the script itself (resulting in a single, self-contained file). Would this be something a "julia script mode" could include?

[Seelengrab]

I'm gonna ignore this, but as I've said before, I really don't like this kind of comment, and I would again ask you to please refrain from it. It's just so unnecessarily confrontational. There's no sides here. We're all trying to work together to find a good design here.

I'm not trying to be confrontational here (though I'm happy to chat on slack/VC if there's a particular issue with me expressing my concerns here); I'm trying to figure out why a split of juliac/juliax in particular is good for continued development, and hence I'm trying to understand what the plan is to do with this. If there is no plan - great, that's a message too! Not one I'd hope for, but at least something 🤷

The reason I personally dislike the split is because I worry that this causes static compilation (and hence cross compilation) to be segregated away from julia and purely into juliax, which would seemingly preclude the workflow I outlined above. I'm trying to figure out if your plans for static compilation prevent such a workflow, or make it substantially harder to keep around.

I don't have any concrete plans for additional changes, but there have been various such changes discussed over the years that we might want to do. I mentioned two of them in the original post, but there may be others.

Ok, so (please correct me if I'm wrong) I'm interpreting this as "it just seems like something neat, but there are no concrete reasons this split needs to be done in order to achieve the changes/things we wished for in the past". So I guess that leaves the big question - what does this split, on its own merits, allow us to do, that couldn't be done with a regular API hook or commandline switch?

The first is undesirable, because it's unclear how it should work with entrypoints that are not generic functions, and the import/export of the special internal symbol is weird. Obviously there can be syntax for it, but that's like 3 or 4 different special cases that need to be learned just to avoid using the name that everybody actually wants to call it.

I don't follow. What is an entry point that is not a generic function? The entry point is special, and hence should be a special thing that ought to be learned. Julia currently doesn't have entry points as a concept, and so we're free to define what it means to be an entry point - there's no requirement I'm aware of that means only generic functions can be entry points.

Making entry points special, e.g. via @main or an entry point declaration in Project.toml was suggested (and 👍ed) multiple times. The first case of mentioning changing the driver executable was by you, and that didn't get as many 👍, so my question is - why not directly go for that? Your argument was that this can be built on top of whatever solution is merged, but I disagree - if it's not the standard/default way to do so, it's not going to catch on and whatever Base exposes will be the "standard" (or not used at all, through obscurity).

The second option isn't great, because it's not clear what to do with conflicting entrypoint definitions and it again needs special syntax to support entrypoints that are not generic functions. It also doesn't support any form of entrypoint import/export or the interactive version, particularly in a world where the manifest is autoloaded.

I think this is conflating multiple different modes of execution unnecessarily. I don't think there's an expectation of compiling standalone scripts into a statically compilable & runnable executable. Conflicting entrypoint definitions should error, at least in a first iteration. Multi-entrypoint support can be easily built on top of it, by providing a singular multi-entrypoint entrypoint.

Not sure what you mean with entrypoint import/export, can you clarify what that is? What is an interactive version of an entrypoint?

Again, I'm just trying to understand what the goal here is. There seem to be a lot of distinct topics that get mixed up and end up in a very confusing ball of yarn.

---

So it seems that this PR doesn't lock us into anything that we can't change or get rid of in the future. And if it's going to be helpful to Keno and others as far as unblocking other work they want to do, I think it makes sense to move forward with this PR.

I mean, as with lots of things in Base.Experimental, people will start to rely on it being there, locking us in. I'd love to know what the plan for this future work is though, that requires this kind of seperation 🤷

[jariji]

Since this issue seems to be reviving the main conversation I'll just mention - There is another option for defining entry points, which is to specify them in a toml, enabling multiple entry points with different names. A number of languages do it that way nowadays. Takafumi in JuliaLang/Pkg.jl#1962 (comment) explained how it could look in Julia.

[Keno]

If we change our mind in the future, we can just straight up remove juliax and/or juliac in future releases, right? Since we documented that they aren't public API?

Yes, that's the point of doing it this way.

And the design discussions around the entrypoint macro (@main function f(args) etc) can still continue while separate experimental work is being done on juliax, right? One doesn't need to preclude the other.

Yes, if somebody wants to do the work, and show that it's better for some reason that wasn't considered and put up a subsequent PR that's always an option of course.

• latest and greatest, and those who are typically early adopters should expect to start using juliax pattern for most things, and only the users who need guaranteed stability should remain on julia

Probably this one, but it's uncertain and depends on how the static compilation work goes, whether it's good enough that most scripts will default to juliax and what other changes end up being made for juliax.

From the discussion so far, I can't see a clear reason to motivate juliax over julia -x say (and I would think the latter preferable).

This is basically that julia --cli-mode=juliax, except that there's symlink shortcut for situations where passing extra command line arguments is cumbersome (like how clang++ is just a shortcut for clang -x c++).

Something I've admired for some time is how with Ion you can embed the dependencies of a script within the script itself (resulting in a single, self-contained file). Would this be something a "julia script mode" could include?

Yes, this is part of the design objectives for juliac. But that's a few months down the road.

The reason I personally dislike the split is because I worry that this causes static compilation (and hence cross compilation) to be segregated away from julia and purely into juliax,

The compilation part is in juliac anyway. This is mostly about the alignment between the compiled workflow and the interpreted workflow. Having some flexibility in the CLI semantics is good for that as it allows us to more closely align the CLI options between the two modes.

Ok, so (please correct me if I'm wrong) I'm interpreting this as "it just seems like something neat, but there are no concrete reasons this split needs to be done in order to achieve the changes/things we wished for in the past". So I guess that leaves the big question - what does this split, on its own merits, allow us to do, that couldn't be done with a regular API hook or commandline switch?

The concrete reason is there is significant uncertainty about the correct design direction and significant opposition to doing such experiments with the main julia driver, so this is where we're at. As for a command line switch, that's basically what this is, as I discussed above, just with the extra convenience of being able to alias it with a symlink, which is a common thing for compiler drivers (clang, gcc, etc).

As for the other changes that might be made, there are no concrete designs, but I will defer again to the two examples in the original proposal that we wanted to do, but couldn't because of concerns about CLI compatibility

• Auto-loading of the whole Project (Faster incremental sysimg rebuilds #40414 and related PRs)
• Auto version selection (Start specific Julia version if it is specified in Project.toml/Manifest.toml juliaup#10). More generally the need for the existence of juliaup as a separate is a failure of Base to address user needs.

None of these have concrete designs, but that's precisely why having a driver that's experimental and unstable for a few releases is useful to prototype these things. If doing language design for the past decade has taught me anything is that the way you get to a good solution is that you start with an ok one, implement it, see how it feels and then iterate from there. The only issue here is that reliance issues now restrict the way in which we can do these experiments, so we need to come up with some creative solutions to enable this experimentation.

I will also point out that none of these discussions are new. We've been having them for the better part of the past decade in various fora (e.g. see #15864). I've been doing my best to summarize, but there's just a lot of context and previous discussion.

don't follow. What is an entry point that is not a generic function?

Generic functions in julia are not special. The current design for juliac for example has

const main = enforce_static(:runtime => false) do ARGS
    println("Hello World")
end

as an API. Whether or not that's the right API is TBD, but it should at least be possible.

Making entry points special, e.g. via @main or an entry point declaration in Project.toml

Requiring a Project.toml just to build a binary is a non-starter, as was mentioned multiple times. The issues with the macro I discussed above.

Your argument was that this can be built on top of whatever solution is merged, but I disagree - if it's not the standard/default way to do so, it's not going to catch on and whatever Base exposes will be the "standard" (or not used at all, through obscurity).

That's like saying that nobody will use make because gcc exists. There can be multiple options. An entrypoint in Project.toml can override the Main.main default for the entry binding (and wouldn't it be nice to have some semantic flexibility to make this change down the line ;) ). Things that are even more complicated that that (complex web deployment, multi-entrypoint apps, split-assurance apps, etc.), will need more sophisticated build system support anyway. Obviously it'll be nice to align things as much as possible, but let's not barricade ourselves from the simple solutions because it doesn't cover all use cases.

Not sure what you mean with entrypoint import/export, can you clarify what that is? What is an interactive version of an entrypoint?

The module MyApp; export main; end, and juliax -e using MyApp thing that people wanted.

Again, I'm just trying to understand what the goal here is. There seem to be a lot of distinct topics that get mixed up and end up in a very confusing ball of yarn.

Well, I think it'd be a lot clearer if people would just let me finish implementing it, so that they can actually try it out but here we are ;).

was suggested (and 👍ed) multiple times.

We don't make design decisions by majority vote. That's how you end up with UTF16. The way this works is that everybody contributes ideas, we think about the technical pro/cons, collect technical objections, solutions, etc. And, yes, to some extent it's up to the technical and aesthetic discretion of the person putting in the work. Even triage is just a fast-pathed version of this. If something is really controversial with many committers disagreeing, it sometimes goes into small discussion, but I don't really think we're there yet.

[PallHaraldsson]

what does this split, on its own merits, allow us to do, that couldn't be done with a regular API hook or commandline switch?

I don't know (all of Keno's) goals. But only one julia executable can't work, if you want to support the two types of Windows apps. So I at least welcome this new infrastructure.

[MasonProtter]

Apologies because I don't want to drag this out and cause more strife, but I'm still having trouble understanding the reasoning against an entrypoint/main macro.

Generic functions in julia are not special. The current design for juliac for example has

const main = enforce_static(:runtime => false) do ARGS
    println("Hello World")
end

as an API. Whether or not that's the right API is TBD, but it should at least be possible.

Couldn't this just be written as something like

const main = @entrypoint enforce_static(:runtime => false) do ARGS
     println("Hello World")
end

or

const main = enforce_static(:runtime => false) do ARGS
     println("Hello World")
end
@entrypoint main

The module MyApp; export main; end, and juliax -e using MyApp thing that people wanted.

and similarly couldn't this be

module MyApp
@entrypoint function main(ARGS)
    ...
end
export main
end

?

I promise I'll drop it after this as I can tell I'm probably not helping, and likely just bogging things down, but I just want to understand if there's a technical barrier here I'm not understanding.

[Keno]

const main = @entrypoint enforce_static(:runtime => false) do ARGS
     println("Hello World")
end

Do you have concrete semantics in mind? Obviously that could be written, but what does it mean ;)? I think as a new rule anybody that proposes a new macro needs to tell me what the macro expands to ;). This is not meant as a dismissal, it's just really hard to be precise in my thoughts if I don't know exactly what the proposal is. In the main proposal, the semantics are simple: The entrypoint is whatever is bound to Main.main. What are the semantics here that you're envisioning?

[MasonProtter]

Sure, that's a good rule. What I'm picturing here is that in _start() (

julia/base/client.jl

Line 536 in cc4108b

function _start()

), instead of

        if isdefined(Main, :main) && !is_interactive
            if Core.Compiler.generating_sysimg()
                precompile(Main.main, (typeof(ARGS),))
            else
                ret = invokelatest(Main.main, ARGS)
            end

we'd write something more like

        the_entrypoint = Core.maybe_get_main_entrypoint()
        if !isnothing(the_entrypoint) && !is_interactive
            if Core.Compiler.generating_sysimg()
                precompile(the_entrypoint, (typeof(ARGS),))
            else
                ret = invokelatest(the_entrypoint, ARGS)
            end

and then would have two functions

Core.register_entrypoint!
Core.maybe_get_entrypoint

which would (in a hopefully type stable way(?)) be able to store and retrieve an object to be called as an entrypoint. Then the macro would expand from

@entrypoint enforce_static(:runtime => false) do ARGS
     println("Hello World")
end

to something like

begin
    local _entrypoint = enforce_static(:runtime => false) do ARGS
         println("Hello World")
    end
    Core.register_entrypoint!(_entrypoint)
    _entrypoint
end

I know that's not the most complete possible specification, but hopefully it at least shows what I was picturing.

[Keno]

Right, that's the "magic side effect" option that was proposed in various forms. The general objection to that was that we don't necessarily want merely loading a package that defines an entrypoint to automatically cause its execution.

There's a couple of different options for semantics that I think work better, but I'm still skeptical of.

1. The macro wraps the rhs in a magic object and the startup code checks all bindings in Main for the special object and uses that as the entry point.

I think this would work for the const main = case, but it's a bit awkward for the @entrypoint main(ARGS) = ... case, because it's not clear what the binding refers to. Does it refer to the special magic object or does it refer to the original function. If the former, it doesn't work for import/export. If the latter, you can't extend it using regular syntax, which is a bit awkward.

2. Bindings gain a special entrypoint flag that the macro can set. On startup, we can scan all bindings in Main and execute the whichever binding has the entrypoint flag set, assuming it is unique (error if not).

I don't think this has been proposed so far in this form, but I think of all the possible options, that's the probably the best alternative so far. The primary issues I see with this are:

• It's a little hard to explain how it work - relying on a lot of internal details, binding resolution, etc.
• It steals a bit from the bindings for something that we expect to have exactly one of in the whole system (though I suppose we could always treat it as a flag, but store it elsewhere).

Still I think that variant might work out ok. I'll mock it up and see if it works.

[MasonProtter]

The above is probably somewhat more convoluted than it needs to be with the function stubs and whatnot. Even simpler would just be something like

        if isdefined(Base, :main_entrypoint) && !is_interactive
            if Core.Compiler.generating_sysimg()
                precompile(Base.main_entrypoint, (typeof(ARGS),))
            else
                ret = invokelatest(Base.main_entrypoint, ARGS)
            end

for Base.start and then have

@entrypoint ex

macroexpand to

begin
    const Base.main_entrypoint = ex
end

[MasonProtter]

I see, thank you for laying out those thoughts.

The general objection to that was that we don't necessarily want merely loading a package that defines an entrypoint to automatically cause its execution.

Yeah that's definitely a real concern. Though I think for somewhat similar (but maybe more restricted reasons), we might also not want a package exporting a name to automatically cause its execution too. Maybe it's fine, but I could certainly imagine a world where it's decided that it's bad style for a package to ever export main for this reason.

That is, it's probably best for the marking to happen in the Main module even if we don't have magic side effects.

Bindings gain a special entrypoint flag that the macro can set. On startup, we can scan all bindings in Main and execute the whichever binding has the entrypoint flag set, assuming it is unique (error if not).

that's definitely an interesting idea. I think maybe to some intermediate users it might be a bit confusing because it's exotic, but I think for the majority of users they shouldn't have too much trouble understanding that writing @entrypoint foo makes the binding foo special in some way, and a lot of people wouldn't be too bothered if it's hard to understand exactly how.

E.g. analogously someone might ask "okay, but how exactly does @inline work, and the answer is basically just "it's complicated and you can't do it with a regular macro, but we can sorta explain what it does", which seems to satisfy most people.

[Keno]

#51435 for the potluck variant.