Tracking issue for procedural macros and "hygiene 2.0"

[alexcrichton]

This is intended to be a tracking issue for enabling procedural macros to have "more hygiene" than the copy/paste hygiene they have today. This is a very vague tracking issue and there's a very long and storied history to this. The intention here though is to create a fresh tracking issue to collect the current state of play, in hopes that we can close some of the much older and very long issues which contain lots of outdated information.

[petrochenkov]

To give some background, feature(proc_macro_hygiene) tracked by this issue was merged from multiple features (proc_macro_mod, proc_macro_expr, proc_macro_non_items, proc_macro_gen) in #52121.

So, it's pretty heterogeneous and is not even entirely about hygiene.

[petrochenkov]

Copypasting what I wrote in #52121:

None of these features is blocked on hygiene, except for proc_macro_gen.

• proc_macro_mod is, IIRC, about expanding inner attributes and expanding non-inline modules (mod foo;). I'm not sure why #[my_macro] mod m { ... } with outer attribute and inline module is still gated, I'm not aware of any reasons to not stabilize it.
• proc_macro_expr/proc_macro_non_items have no reasons to be gated as well and can be stabilized after some implementation audit. Proc macros have well-defined "copy-paste" Span::call_site() hygiene, this includes "unhygienic" use of local variables.
  IIRC, @nrc assumed that additional feature gates would prevent interactions of that copy-paste hygiene with local variables, but that's not true - you can easily expand an item and inject tokens from the outside into its body (e.g. TokenStream::parse does not resolve with Span::call_site() #50050), causing exactly the same interactions.

What is really blocked on hygiene is Span::def_site() and decl_macro (which is also blocked on a ton of other stuff).

[petrochenkov]

proc_macro_gen is more about interactions between Span::call_site hygiene (entirely unhygienic) and macro_rules hygiene (unhygienic except for local variables and labels).

[CodeSandwich]

This looks like the right branch for this question. Sorry about copy-paste, but I really need to know.

It seems that since rustc 1.30.0-nightly (63d51e89a 2018-09-28) it is no longer possible to traverse code inside module in separate file. If you process mod module;, you get TokenStream containing mod module;, WYSIWYG.

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code. Is there or will be any way to work with content of modules in separate files? Its lack may be confusing for end users when trivial refactoring of module organization causes change in macro behavior.

[petrochenkov]

@CodeSandwich
This may be a regression from #52319 perhaps?

cc @tinco

[petrochenkov]

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code.

From what I remember, this is an open question.
The transformation mod m; -> mod m { ... } is an expansion of sorts, and we may treat it both as an early expansion (like e.g. #[cfg], which is also expanded before passing the item to attribute-like macros) and as a late expansion (the item is not expanded before being passed to macros).

The "early expansion behavior" looks preferable to me, and this change in behavior looks like a bug rather than a fix.

[tinco]

I lack some context, so I can't be certain, what exactly is "process"? If it relies on rustc's pretty printing, then yes this would be a consequence of #52319, and it should be simple to fix by changing the invocation I think. #52319 does two things:

1. Add information to the AST (a single bool, tracking if a mod is inline)
2. Make pretty print no longer print the external file if that bool is false

If you need the expanded file, then that should be made explicit in the invocation of the pretty printer.

[petrochenkov]

Yes, the change is likely due to the pretty-printing change.
So it can be fixed either by reverting the change, or making it more fine-grained so it prints the contents regardless of the inline flag if pretty-printing is done "for proc macros" (whatever that means).

[tinco]

If you can point me at the line of code that invokes the pretty printer, I can suggest a fix.

[CodeSandwich]

By "processing" I mean parsing TokenStream with Syn to an Item, modifying its content and converting it back to TokenStream. The precise spot of parsing is here and content traversal is there. I haven't dug into implementation of Syn yet, so I can't tell for sure if it uses pretty printing. I've set nightly feature in Syn, which as far as I know skips step of printing and parsing the string again.

[alexcrichton]

I personally feel that what a proc macro receives as input tokens when attached to a module is an open questions, as @petrochenkov said. I do not think that one answer is clearly correct over another (mod foo; vs mod foo { ... }) at this time. That's why it's unstable!

[eddyb]

I'm confused why proc macros expanding to macro_rules! definitions is gated.
(especially when the new macros are only invoked by other expanded items)

[petrochenkov]

@eddyb
Well, the answer is "not enough confidence in correctness of hygiene interactions between transparent (procedural on stable) and semi-transparent (macro_rules) macros".

I hoped to at least get rid of the "context transplantation trick" before stabilizing it, but got distracted by the edition stuff.

It generally works though, I'm not aware of any specific bugs manifesting in normal cases.

[petrochenkov]

$crate in generated macros concerns me in particular.
(I don't remember whether it works correctly now, need to test.)

[corbinu]

@SergioBenitez Just wanted to check is #68717 the last needed piece for stabilizing Rocket or is something more needed?

[joshtriplett]

There are a number of features tracked in this issue. It sounds like some have been stabilized, while others (like def_site) haven't. Would it be possible to get a summary of the remaining work needed for proc macro hygiene? (It may also make sense to break this issue up into multiple tracking issues, or alternatively, get an issue with a checklist of other issues.)

[TheButlah]

I came here from: danakj/dynpath#1

But reading this issue, I have very little clue how "non-inline modules in proc macro input" relates to this issue

[antimora]

I'm looking forward to this feature. Do you have any updates on the progress or timeline for this feature to become stable?

Specifically I would like this subfeature to be released:

error[E0658]: custom attributes cannot be applied to statements

[JohnScience]

The users of my qualifier_attr library would benefit from this because then library authors can write

use qualifier_attr::qualifiers;

#[cfg_attr(semver_exempt, qualifiers(pub))]
mod private_mod1;
#[cfg_attr(semver_exempt, qualifiers(pub))]
mod private_mod2;
// ...

instead of

#[cfg(semver_exempt)]
pub mod private_mod1;
#[cfg(not(semver_exempt))]
mod private_mod1;

#[cfg(semver_exempt)]
pub mod private_mod2;
#[cfg(not(semver_exempt))]
mod private_mod2;
// ...

where private_mod1 and private_mod2 are non-inline modules.