Document reservation of double underscore names (dunders)

[mkitti]

In the recent discussion about main, __main__ was proposed. One reason in favor of this is that double underscore, or dunder, names are reserved names in Julia. This however is not documented.

This pull request adds a paragraph reserving double underscore names for Julia.

They are added to Base -> Essentials -> Keywords where other reserved names are documented.

https://docs.julialang.org/en/v1.11-dev/base/base/#Keywords

cc: @StefanKarpinski

[3f6a]

In the recent discussion about main, __main__ was proposed.

Just for future reference, this is talking about this proposal:

• https://discourse.julialang.org/t/on-hold-if-you-have-a-function-called-main-you-may-need-to-tweak-it/103519/39.
• Standardize the entry-point for Julia execution #50974

[3f6a]

These reservations are not currently enforced by the parser.

Should they be?

[mkitti]

@c42f would there be a way to enforce this via JuliaSyntax.jl?

[ericphanson]

fwiw I don't think lack of enforcement should hold up this PR, since documenting existing expectations is a definite improvement over not documenting them, and the text could easily be updated in a future PR if some enforcement is added

[rfourquet]

Enforcing this would be somewhat breaking, no? e.g. in Revise you can use __revise_mode__ , in ReTest you can use __retest_defaults__, and there are probably other instances.

[3f6a]

One could forbid new names with __, with some exceptions that are already in use. So there could be a list of allowed dunders names.

[mkitti]

One could forbid new names with __, with some exceptions that are already in use. So there could be a list of allowed dunders names.

I think the enforcement matter should be opened as a new issue once this is merged.

[Tokazama]

Maybe I'm missing something, but this syntax still doesn't have a consistent functionality or semantic meaning. The most consistent thing I've seen is that it is expands to a new value informed by the context of a macro call. But Base.__init__ and __precompile__ are a generic functions, Base.__toplevel__ is a module, and macros like @__MODULE__ just return their __lowercase__ counterparts.

Is the intent here just to guarantee that Base can always use this syntax without potentially breaking package code?

[3f6a]

Is the intent here just to guarantee that Base can always use this syntax without potentially breaking package code?

Yes. See for instance discussion at #50974.

[Tokazama]

Is the intent here just to guarantee that Base can always use this syntax without potentially breaking package code?

Yes. See for instance discussion at #50974.

I've read through that. Keno pointed out that it doesn't have any unique meaning in Julia. If there's a plan to use this for something that makes sense. However, there currently isn't any technical reason users shouldn't use this syntax and there doesn't appear to be any consensus how this should be uniformly unique in the future. If we aren't going to change how this syntax is parsed this is escalating stylistic convention to formal API.

[mkitti]

@StefanKarpinski asserted that "__ names are reserved" here:
https://discourse.julialang.org/t/on-hold-if-you-have-a-function-called-main-you-may-need-to-tweak-it/103519/23?u=mkitti

However, we noticed that this is not documented anywhere. Independent of if we use __main__, I think it's still useful to make this reservation for future use if needed. If anything is not clear, please suggest a change.

[Tokazama]

But why are they reserved?

[mkitti]

They are reserved to provide a namespace for the Julia Language in case we need it to expand for matters that we cannot currently anticipate.

Python has a fair number of these at the moment, many of which Julia does not need.

https://docs.python.org/3/reference/lexical_analysis.html?highlight=dunder

There, they issue this warning:

Any use of * names, in any context, that does not follow explicitly documented use is subject to breakage without warning.

It seems we have also started to follow this convention without clearly stating it.

[mkitti]

Apparently ISO C and C++ also make some reservations for identifiers with double underscores.

Names containing double underscore or beginning with an underscore and a capital letter are reserved for implementation (compiler, standard library) and should not be used (e.g. __reserved or _Reserved)

https://en.wikipedia.org/wiki/Naming_convention_%28programming%29?wprov=sfla1

https://en.cppreference.com/w/c/language/identifier#:~:text=Note%3A%20in%20C%2B%2B%2C%20identifiers,a%20double%20underscore%20are%20reserved.

[Tokazama]

I'm not arguing that we need to have everything figured out right now.
But before we start changing public API let's figure out a couple things:

1. Is this pattern a good candidate for special parsing. I'm not sure that the current special parsing in macros for a few explicit symbols is proof that this works well as a generic pattern. I'm definitely not the one to make the call on this but this isn't like bike-shedding a single name based on how people use it in other languages. This is a very large sweeping statement that assumes this syntax could eventually be evaluated completely different than it currently is.
2. Implementation of special parsing cant't be breaking. There are definitely packages that use the __variable pattern. I'm not sure if there many that use the __variable__ pattern, but making previous permissible symbols illegal sounds breaking. This would also be breaking if it changed evaluating of __precompile__ and __init__.

I actually think it would be really cool to have something that provides better contextual parsing outside of macros. But we already have an issue with properly documenting and enforcing public/private APIs. I think it would make more sense to first ensure this could even be implemented prior to Julia 2.0

[mkitti]

First, let me emphasize a very important aspect of this. I'm not changing public API. I'm merely documenting part of the API that was not previously stated clearly. While it might not be stated explicitly, it could be inferred from currently documented identifiers and the tradition of languages proceeding Julia that double underscore identifiers were reserved. Languages do not come from a vacuum but rather draw prior art.

I can understand the perspective that this can not be public API since it was not documented. It is because of this perspective that I propose this pull request with some urgency.

1. On special parsing, I'm having a difficult time figuring out what you are discussing. I infer you mean the use of __module__ and __source__ within macros and if these should be available outside of macros. To answer a your question, my answer is that I do not know. If we introduced more special parsing, then the macro case seems to provide some precedence. As per the preamble, we are not changing anything. We are documenting an existing concept
2. The __variable pattern is not being documented here as reserved for the language. Only identifiers that start and end with double underscores are being documented as reserved here. I can see you how you got that that from my PR introduction, so I edited it. Please refer to the text of the diff. Again, my position here is that using identifiers preceding and ending with double underscores were not actually permissible previously.

As for existing identifiers using that use the reserved pattern, there is a fair case that these should be grandfathered in. Additionally, I note that Revise.jl and ReTest.jl may have special license to act on behalf of the language.

[3f6a]

Friendly bump. Not sure if a conclusion was reached here?

[c42f]

Late to the discussion here sorry... but FWIW I don't think we should involve the parser here - the Juila parser allows many constructs which aren't valid Julia code, but may be consumed by macros and turned into something valid.

I think the "correct" place to emit a warning (or error, or whatever is decided) would be in lowering.

[kimikage]

Just to be sure, is this just a double, or at least a double?
E.g., ___is_a_triple_reserved___?

This may be a silly question, but given that several packages use their dunders, I don't think it's a bad idea to warrant their substitutions.

[mkitti]

I would go with the following operational definition.

julia> function isreserved(name)
           name = string(name)
           return startswith(name, "__") && endswith(name, "__")
       end
isreserved (generic function with 1 method)

julia> isreserved(:___is_a_triple_reserved___)
true

Triple underscores could be delegated later after establishing this convention.

[mkitti]

However, there is an ambiguity because of the idea that the reserved words should be minimalized and the fact that we already distinguish between _single_ and __double__.

Do we mention anything about _single_ anywhere?

[mkitti]

I am unclear what the ambiguity is then.

• _single_ does not start or end with __ and is thus not reserved.
• __double__ does start and end with __ and is thus reserved.
• ___triple___ does start and end with __ and is thus reserved.

Please make a specific suggestion to improve the pull request.

[mkitti]

I simply do not understand the issue you are raising and thus cannot fix it. I'm not sure if we are still talking about N underscores for N > 2 or if there is a overriding issue regarding "the idea that the reserved words should be minimalized". If you think there is specific text that can be added here, please clearly state what text that might be.