Tracking issue for RFC 2582, &raw [mut | const] $place (raw_ref_op)

[Centril]

This is a tracking issue for the RFC "an operator to take a raw reference" (rust-lang/rfcs#2582), feature(raw_ref_op).

Steps:

• Implement the RFC (see this comment for a detailed checklist)
• Adjust documentation (see instructions on rustc-guide)
• Stabilization PR (see instructions on rustc-guide)

Unresolved questions:

• Maybe the lint should also cover cases that look like &[mut] <place> as *[mut|const] ?T in the surface syntax but had a method call inserted, thus manifesting a reference (with the associated guarantees). The lint as described would not fire because the reference actually gets used as such (being passed to deref). However, what would the lint suggest to do instead? There just is no way to write this code without creating a reference.

Implementation history:

• Initial implementation: Add a raw "address of" operator #64588

[matthewjasper]

I'll try to get a PR open for this soon.

[Centril]

I would suggest splitting the implementation work up into phases to make each part thoroughly reviewed and tested. However, think of this list as a bunch of tasks that need to be done at some point.

1. Implement &raw [mut | const] $expr in the parser and AST.
   • Ensure the unused_parens lint works alright.
   • Handle interactions in librustc_resolve.
2. Push things down from AST => HIR
   • Handle lowering
   • Handle type checking
3. Push things down from HIR => HAIR
4. Push things down from HAIR => MIR -- here, the actual MIR operation is introduced.
5. Introduce lints for statically detectable UB as per the RFC.
6. Make safe_packed_borrows a hard error when the feature gate is active, and make this the case as well for unsafe equivalents.

---

...but it seems @matthewjasper already has a PR heh.

[Aaron1011]

Make safe_packed_borrows a hard error when the feature gate is active, and make this the case as well for unsafe equivalents.

I'm not sure that this is a good idea. This would make it impossible to gradually migrate a codebase over to using raw references - once the feature gate is enabled, all references to packed structs must be fixed at once in order to get the crate building again.

[moshg]

I wonder why not &raw $expr but &raw const $expr. Is there a discussion about this?

[mjbshaw]

@moshg Yes, this was discussed. See here (and the following comments) for an initial discussion. And here (and the following comments) for a signoff from grammar-wg. And here for an example that &raw (expr) can be ambiguous.

I think it might be worth amending the RFC to state why &raw const $expr was chosen over &raw $expr.

[Ixrec]

The RFC thread noted a couple times (including in the FCP comment) that &raw expr without the const conflicts with existing syntax and would break real code. rust-lang/rfcs#2582 (comment) goes into some detail. My understanding is that it is still possible to migrate to &raw expr with breaking changes in an edition, and I didn't see much explicit discussion of that, but I think everyone agrees that there's enough urgency here that we definitely prefer making &raw const expr a thing over blocking any solution on waiting for another edition.

[moshg]

@mjbshaw @lxrec
Thank you for your explanations! I've understood why &raw $expr is breaking change.

[RalfJung]

I think it might be worth amending the RFC to state why &raw const $expr was chosen over &raw $expr.

rust-lang/rfcs#2764

[red75prime]

@RalfJung I'm addressing the points you've made in rust-lang/rfcs#2582 (comment)

promotion being confusing here seems mostly orthogonal to raw-vs-ref, doesn't it?

While it would be helpful to have at least a warning for a promotion in &mut <rvalue> case, disabling it is a backward incompatible change to the language. In &raw mut <rvalue> case it can be done from the beginning.

If no promotion happens with raw ptrs, what would you expect to happen instead?

I would expect &raw mut <rvalue> to result in compilation error, as &<rvalue> does in C.

I would expect &raw mut (1+2) to be promoted the same way &mut (1+2) is; not doing that seems even more confusing and very error-prone

I have an impression that silently allowing such promotions is more error prone, as it violates the principle of least surprise.

trait AddTwo {
    fn add_two(&mut self);
}

impl AddTwo for u32 {
    fn add_two(&mut self) { *self += 2; }
}

const N: u32 = 0;
// somewhere else
assert_eq!(N, 0);
// N += 2; // doesn't compile
N.add_two(); // compiles with no warnings
assert_eq!(N, 2); // fails

[oli-obk]

promotion being confusing here seems mostly orthogonal to raw-vs-ref, doesn't it?

yes, promotion has nothing to do with it as far as I can tell.

I would expect &raw mut to result in compilation error, as & does in C.

I agree. We should error on any &raw place where place

1. has no Deref projections (*) or Index ([i]) projections
2. and has a temporary local as base

since we still want to permit &raw *foo() I would assume. Or more concretely &raw foo().bar when foo() returns a reference.

If "erroring if not" has too many negations (i confused myself in the process of writing this), here's a version of what I think we should permit (and only that):

We should permit &raw place only if place

1. has Deref projections or Index projections
2. or has a static or local variable as the base

[RalfJung]

While it would be helpful to have at least a warning for a promotion in &mut case, disabling it is a backward incompatible change to the language. In &raw mut case it can be done from the beginning.

Note that not promoting here means this becomes a very short-lived temporary and the code has UB (or fails to compile, when only safe references are used)! Is that really what you want? ptr as T is not a place like ptr is and cannot be used as such; in particular it cannot be mutated.

I have an impression that silently allowing such promotions is more error prone, as it violates the principle of least surprise.

I have a hard time imagining that silently causing UB is less of a surprise than promoting...

N.add_two(); // compiles with no warnings

You are conflating some things here; what you are seeing here are implicit deref coercions. Promotion is not involved in this example.

We should permit &raw place only if place

Oh I see, you want to cause some static errors. Yeah I can imagine that there are reasonable things we could do here.

[red75prime]

You are conflating some things here; what you are seeing here are implicit deref coercions. Promotion is not involved in this example.

There's a bit of misunderstanding. I used a wrong term. When I said "rvalue promotion", I had "creation of a temporary memory location for an rvalue" in mind, not "promotion of a temporary to 'static".

[RalfJung]

@red75prime oh I see. Well currently, with promotion, we do not create a temporary memory location, but just a global one. But I suppose your desired behavior is more like what @oli-obk described, where we have static compiler errors?

I am not sure to what extend we can reliably produce those though. But MIR construction should know when it builds temporaries, so couldn't it refuse to do that when &raw is involved? Cc @matthewjasper

[HadrienG2]

Is there any plan to eventually add &raw pointer support to pattern matching? It could enable getting rid of various forms of deliberate UB in abomonation.

[RalfJung]

@HeroicKatora actually wrote an RFC for that, though it was originally meant as an alternative to &raw: rust-lang/rfcs#2666

[HeroicKatora]

I'd be fine with it being interpreted as complementary, and can rewrite portions of it as necessary :)

[mjbshaw]

I can't recall if this was discussed in the RFC (and failed to find any comments on it), but can <place> be a pointer (to allow pointer-to-field computations)? Example:

struct Struct {
    field: f32,
}

let base_ptr = core::mem::MaybeUninit::<Struct>::uninit().as_ptr();
let field_ptr = &raw const base_ptr.field; // Is this allowed by this RFC?

Currently field_ptr has to be initialized with unsafe { &(*base_ptr).field as *const _ }. Using unsafe { &raw const(*base_ptr).field } is a bit better, but avoiding the *base_ptr dereference entirely would be ideal.

I assume the RFC doesn't permit this (since base_ptr.field isn't a place), but I just wanted to confirm (to be clear: I'm not trying to advocate for this RFC to do that if it doesn't already; I would like to draft a separate RFC exploring ways to avoid the *base_ptr dereference).

[RalfJung]

I can't recall if this was discussed in the RFC (and failed to find any comments on it), but can be a pointer (to allow pointer-to-field computations)?

That question is ill-typed. A pointer is a kind of value, which is a class of things separate from places.

let field_ptr = &raw const base_ptr.field; // Is this allowed by this RFC?

No, because there is no deref coercion for raw ptrs. But &raw const (*base_ptr).field is allowed, because (*base_ptr).field is a place.

avoiding the *base_ptr dereference entirely would be ideal.

That has nothing to do with places though, that's just a deref coercion. When you write similar things with references, they work because the compiler adds the * for you -- and not because of anything having to do with places.

since base_ptr.field isn't a place

It's not even well-typed.

I would like to draft a separate RFC exploring ways to avoid the *base_ptr dereference).

Agreed, the current syntax is bad -- but mostly because of the parenthesis and the use of a prefix operator. I think we should have posfix-deref, e.g. base_ptr.*.field. That has been discussed before... somewhere...

[mjbshaw]

That question is ill-typed. A pointer is a kind of value, which is a class of things separate from places.

You're right. Sorry for my poor choice of words.

let field_ptr = &raw const base_ptr.field; // Is this allowed by this RFC?

No

Thanks for answering my question. I'll follow up to your other points in a new thread on internals.rust-lang.org.

[eddyb]

Small nit: a deref coercion is &T to &U via deref(s). Dot access has its own implicit deref.

[A1-Triard]

If it is UB even with &raw, then why this RFC and &raw construction exist? It does not have any sense.

[RalfJung]

This is described in the RFC: rust-lang/rfcs#2582.

[joshtriplett]

@RalfJung Re-reading the RFC, I do think it makes sense to switch to just plain getelementptr when raw refs are involved, and we can always switch to a new nowrap variant if LLVM adds one.

[RalfJung]

IIRC this came up as a future possibility during the RFC. The problem is that this means the semantics of * differ depending on the syntactic context, which seems rather tricky and could be surprising to programmers -- in particular when that context is introduced implicitly by the compiler (through auto-ref).

[A1-Triard]

I've read the RFC. There are many interesting in there, especially about getelementptr inbounds.

So, even with this new feature there is no any possibility to calculate field offset, right?

[RalfJung]

So, even with this new feature there is no any possibility to calculate field offset, right?

No, this feature is enough to implement offset_of!. In fact, https://github.com/Gilnaa/memoffset/ already does so if you set the unstable_raw feature. (Note that this has been discussed here in this very thread before. The discussion here is not very long yet, please read it before posting. :)

[A1-Triard]

No, this feature is enough to implement offset_of!.

Through MaybeUninit? I would not call it "enough to implement".

[SimonSapin]

The problem is that this means the semantics of * differ depending on the syntactic context, which seems rather tricky and could be surprising to programmers -- in particular when that context is introduced implicitly by the compiler (through auto-ref).

I’m not sure if by "this" you mean a future possibility, but I think the RFC as-is introduces exactly that problem.

Consider this code with UB:

struct A(u8, i32);

let dangling: *mut A = std::ptr::NonNull::dangling().as_ptr();
let field_ref: &i32 = unsafe { &(*dangling).1 };

The RFC seems to be written on the premise that problematic part of this code is the reference-creating operator &, and proposes a new operator that can be used instead. However that does not match my mental model at all. My understanding of raw pointers in current stable Rust is that the dereference operator * is the only source of unsafety / potential UB. Using it requires the raw pointer to point to valid memory. Indeed, copying a field without using the reference-creating operator causes just as much UB:

struct A(u8, i32);

let dangling: *mut A = std::ptr::NonNull::dangling().as_ptr();
let copied_field: i32 = unsafe { (*dangling).1 };

With this RFC’s addition to the language, dereferencing with * a raw pointer to invalid memory may or may not be UB, depending on whether the expression eventually also uses &raw. This makes the unsafety of raw pointers harder to reason about. I’m sorry to only say it this late (the RFC is already accepted), but I feel strongly that this is an important enough flaw that we should consider alternative designs that focus on avoiding the * operator instead of avoiding the & or &mut operator.

If we need to do projection of a raw pointer to a struct to a raw pointer to a field of that struct, could we have an operator doing exactly that? Strawman not fully thought through:

struct A(u8, i32);

let dangling: *mut A = std::ptr::NonNull::dangling().as_ptr();
let field_ptr: *mut i32 = dangling->1;

---

Aside, from the RFC:

If one wants to avoid creating a reference to uninitialized data (which might or might not become part of the invariant that must be always upheld), it is also currently not possible to create a raw pointer to a field of an uninitialized struct: again, &mut uninit.field as *mut _ would create an intermediate reference to uninitialized data.

But what is the type of uninit in that expression? Presumably it is not SomeStruct but something like *mut SomeStruct from MaybeUninit::as_mut_ptr, otherwise the example would have UB before we even get to the given code. So that example might be missing an operator for dereferencing a raw pointer: &mut (*uninit).field as *mut _.

[SimonSapin]

With this RFC’s addition to the language, dereferencing with * a raw pointer to invalid memory may or may not be UB, depending on whether the expression eventually also uses &raw.

By the way, shouldn’t raw pointer field projection always be safe? &raw mut (*dangling).1 requires an unsafe block in current Nightly.

[RalfJung]

@A1-Triard

Through MaybeUninit? I would not call it "enough to implement".

I mean, it literally is enough to implement it. We have constructive proof of this fact. I am not sure what else you are asking for. Maybe you consider the implementation too complicated, but that is a separate problem -- and a much less pressing one.

---

@SimonSapin

The RFC seems to be written on the premise that problematic part of this code is the reference-creating operator &, and proposes a new operator that can be used instead. However that does not match my mental model at all. My understanding of raw pointers in current stable Rust is that the dereference operator * is the only source of unsafety / potential UB.

What you say is orthogonal to the RFC. It is true that for raw pointers, * is the only source of UB. The issue is that, without this RFC, it is sometimes impossible to even use raw pointers, so one has to use references, and for those there is much more UB than for raw pointers.

With this RFC’s addition to the language, dereferencing with * a raw pointer to invalid memory may or may not be UB, depending on whether the expression eventually also uses &raw.

That is not correct, the RFC says nothing like this. @joshtriplett proposed to make further changes that would have the effect. I am confused as to why you think the RFC does this, when it does not.

This code is UB with and without this RFC:

struct A(u8, i32);

let dangling: *mut A = std::ptr::NonNull::dangling().as_ptr();
let field_ref: &i32 = unsafe { &(*dangling).1 };

And this code (which can only be written after the RFC) is UB:

struct A(u8, i32);

let dangling: *mut A = std::ptr::NonNull::dangling().as_ptr();
let field_ref: &i32 = unsafe { &raw const (*dangling).1 };

The reason is the same in both cases -- * was used on a dangling raw pointer. The UB in the first example has nothing to do with the &, and thus this entire issue has nothing to do with the RFC.

By the way, shouldn’t raw pointer field projection always be safe? &raw mut (*dangling).1 requires an unsafe block in current Nightly.

Now I am really confused, because this is the exact opposite of what you just asked for! If we want to make this safe, we have to make * special when used below &raw. And you just said you were rather opposed to that.

[SimonSapin]

Now I am really confused, because this is the exact opposite of what you just asked for! If we want to make this safe, we have to make * special when used below &raw. And you just said you were rather opposed to that.

I thought this was what the RFC proposes, so it looks like I deeply misunderstand it :/

[RalfJung]

I thought this was what the RFC proposes, so it looks like I deeply misunderstand it :/

There is, as far as I can see, not a single instance of the deref operator * in the RFC (except for "future possibilities", where they are implicit though through auto-deref). So no, this is not what the RFC proposes. The "offsetof woes" paragraph in "Future possibilities" mentions such a change could be desirable in the future, but the rest of the RFC is specifically about creating pointers, not about dereferencing them.

[A1-Triard]

@RalfJung , OK, You have convinced me. I will use memoffset crate.

[withoutboats]

The RFC seems to be written on the premise that problematic part of this code is the reference-creating operator &, and proposes a new operator that can be used instead. However that does not match my mental model at all. My understanding of raw pointers in current stable Rust is that the dereference operator * is the only source of unsafety / potential UB.

That's not correct; the use case that the RFC immediately addresses has to do with packed structs and not dereferencing raw pointers.

#[repr(packed)]
struct A(u8, i32);

let x: A = A(0, 0);
let r: &A = &x;

// IMMEDIATE UB
let ub: &i32 = &x.1;

This code contains UB, as I understand it, because it creates an unaligned reference to an i32. This despite the fact that it contains no raw pointers at all. Eventually we'd like to make this a compiler error.

The raw ref macros like you get an *mut i32 without constructing an &i32; that avoids the UB, because you never create an unaligned reference. The problem today is that there's no reasonable to create a raw pointer without first creating a reference, which must be aligned.

[QuineDot]

For what it's worth, I too got the wrong impression from this paragraph and the two that follow.

If one wants to avoid creating a reference to uninitialized data (which might or might not become part of the invariant that must be always upheld), it is also currently not possible to create a raw pointer to a field of an uninitialized struct: again, &mut uninit.field as *mut _ would create an intermediate reference to uninitialized data.

[RalfJung]

What was that wrong impression that you got?

[joshtriplett]

It looks like the new raw ref syntax has been implemented. Are there any blocking concerns that would prevent us from stabilizing this?

[RalfJung]

AFAIK the lint specified in the RFC has never been implemented. But I guess that is not a blocker for the syntax.

We introduced the addr_of! macro instead of stabilizing the &raw [mut|const] syntax immediately due to concerns around extending the surface syntax and avoiding syntax bikeshed and grammar concerns. No idea if anything changed on that front -- Cc @rust-lang/wg-grammar

[CAD97]

It's also my recollection that there were no blocking concerns on the grammar or parsing side.

There were three questions I recall without a definitive answer:

• &raw $place versus &raw const $place, and whether the syntax should be evoking *const T or &T.
  • From a grammar side, &raw (expr) still requires making raw a contextual keyword here. But the question is whether const is just a concession to parsing that we'd prefer to remove, or if it's desirable as evocative of *const T.
• &raw const $place giving a read-only pointer even when used on a writable place is contentious.
• It's unclear what &raw const (*ptr).field requires from ptr — is it gepi or gep, and the related generality/optimization toggles.
  • The problem here is that dereferencing into "place mode" is potentially the wrong abstraction.

Additionally, there is one big "shiny future" possibility: &raw const T (or just &raw T, or &'unsafe T, or any other number of possibilities) for a "better raw pointer" type. The problems and footguns with working with the existing *[const|mut] T and ptr::NonNull<T> are well documented and I won't rehash them here. If in a shiny future we get a built-in nicer raw pointer type, &raw [const|mut] T is currently the most promising syntax for creating a raw pointer (and that's why it's used here), and it'd be a shame to have worse syntax for the better thing, and better syntax for the worse thing.

(But seriously, go read every single thing @Gankra has written about pointers.)

To be clear: I'm not arguing we shouldn't improve what we have because we might be able to do it better in the future. Rather, there are existing proposals that show we really should consider

If we had a time machine, what would we want Rust pointers to look like?

and how close can we realistically get to that ideal?

I think we can get close enough that Rust is a contender for the best language to write C in. This isn't the case currently because of how easy it is to incidentally introduce reference semantics into pointer code (along with other papercuts), but I have faith that we can make it there.

So I guess the conclusion of this is: &raw hasn't stabilized yet because while it is agreeably the best implementation of this solution to this problem, we're still not necessarily in agreement that this is the correct problem to be solving.

Given a magical grant of bandwidth, this is what I personally think should be the path forward:

• Collect a T-lang design notes document for "pointer pain points"
• From the pain points, T-lang decides on which path we want to pursue: new pointer type(s) or old pointer types
• If old pointer types:
  • Stabilize &raw $place syntax 🎉
• If new pointer type(s)
  • If &raw T:
    • &raw $place makes them and not * [const|mut] T
  • If a new keyword/sigil/etc:
    • Too much of an unknown to predict
  • If a named type (e.g. ptr::NonNull<T>)
    • Determine if place-to-new-pointer is an important operation that wants the & {?} $place syntax space
    • If not: stabilize &raw $place syntax to make today's Rust significantly better 🎉

(And to try to predict the future: I predict that a far-future pointer-perfect Rust has refinement typing for pointers and that ptr::NonNull and ptr::WellFormed are aliases for the namable refined pointer types that can cross function boundaries. Prefix deref is deprecated in favor of .read()/.write()/.as_ref()/.as_mut() and . for safe field offsetting, ~ for unsafe GEPi field offsetting. container_of! is still, as usual, a controversial nightmare.)

[RalfJung]

• Collect a T-lang design notes document for "pointer pain points"
• From the pain points, T-lang decides on which path we want to pursue: new pointer type(s) or old pointer types

Seems like we are still at this stage. So if someone wants to push this forward, working on such a document would be the best way to help. :)

[RalfJung]

On the other hand, if we are getting a better raw pointer type in the future, we always have the option of using an edition transition to change the meaning of &raw. So maybe we should just stabilize what we got so that we can finally stop using a macro for this?