fix LinkedList invalidating mutable references #60072

RalfJung · 2019-04-18T09:50:34Z

The test test_insert_prev failed in Miri due to what I consider a bug in LinkedList: in various places, NonNull::as_mut got called to modify the prev/next pointers of existing nodes. In particular, the unstable insert_next has to modify the next pointer of the node that was last handed out by the iterator; to this end it creates a mutable reference to the entire node that overlaps with the mutable reference to the node's content that was handed out by the iterator! Thus, the next use if said mutable reference is UB.

In code:

            loop {
                match it.next() { // mutable reference handed to us
                    None => break,
                    Some(elt) => {
                        it.insert_next(*elt + 1); // this invalidates `elt` because it creates an overlapping mutable reference
                        match it.peek_next() {
                            Some(x) => assert_eq!(*x, *elt + 2), // this use of `elt` now is a use of an invalid pointer
                            None => assert_eq!(8, *elt),
                        }
                    }
                }
            }

This PR fixes that by using as_ptr instead of as_mut. This avoids invalidating the mutable reference that was handed to the user. I did this in all methods called by iterators, just to be sure.

Cc @gankro

rust-highfive · 2019-04-18T09:50:38Z

r? @shepmaster

(rust_highfive has picked a reviewer for you, use r? to override)

shepmaster · 2019-04-18T23:11:10Z

src/liballoc/collections/linked_list.rs

- Some(mut head) => head.as_mut().prev = node,
+ // Not creating new mutable (unique!) references to not invalidate
+ // references we handed out.
+ Some(head) => (*head.as_ptr()).prev = node,


I'm not sure I even understand what is going on here...

Previously, as_mut returned a &mut Node<T>, which we then modified the prev field of. That much makes sense.

Now, we get a *mut Node<T>, dereference it to get a Node<T> then set the prev field of.

I've been under the impression that setting the field of a struct implicitly created a mutable reference — where am I wrong? If I'm not wrong, why aren't we creating a second mutable reference here?

I believe a mutable borrow is established only if you create an &mut that isn't immediately explicitly cast to a *mut. It's very janky immediate-action rules, with similar logic to why you can take a reference to the output of arr[x] and it's not a reference to a temporary.

Regardless for a lot of these the issue isn't a mutable borrow, but rather a mutable borrow that overlaps with node.elem. By writing the code this way we never claim unique ownership of the whole type, just that we can mutate the next/prev links.

I believe a mutable borrow is established only if you create an &mut that isn't immediately explicitly cast to a *mut.

No, that would be rust-lang/rfcs#2582 which is not yet in.

I've been under the impression that setting the field of a struct implicitly created a mutable reference

Writing to a field has all the same effects (in terms of invalidating other references) as creating a mutable reference to it, yes.

The key point is that we only modify the prev field, and the aliasing reference we are worried about points to the data field (element or whatever it is called). I will expand the comments to clarify this.

By writing the code this way we never claim unique ownership of the whole type, just that we can mutate the next/prev links.

s/type/node/. But yes.

@shepmaster I tried to improve the comments explaining why these changes are needed; does this make more sense now?

Writing to a field has all the same effects (in terms of invalidating other references) as creating a mutable reference to it.

Can you remove the pronoun here? Do you mean:

Writing to a struct's field has the same effects as creating a mutable reference to

the struct

the field

I mean "to the field".

It's really the other way around -- creating a mutable reference has the effects of a write to the memory that is "covered" by the reference (size_of::<T> bytes starting at where the pointer points).

Gankra

seems fine, r=me, but I'm not actually a reviewer, so

Gankra · 2019-04-18T16:50:41Z

src/liballoc/collections/linked_list.rs

@@ -143,14 +146,18 @@ impl<T> LinkedList<T> {
 /// Adds the given node to the front of the list.
 #[inline]
 fn push_front_node(&mut self, mut node: Box<Node<T>>) {
+ // This method takes care not to create mutable references, to maintain
+ // validity of aliasing pointers into existing nodes.


could you clarify that the aliasing pointers are to the elements so it is specifically fine for us to prod at all the next/prev links with wild abandon?

Good point, done.

lqd · 2019-04-19T09:49:19Z

src/liballoc/collections/linked_list.rs

+ None => self.list.push_back(element), // push_back is okay with aliasing nodes
+ Some(head) => unsafe {
+ let prev = match head.as_ref().prev {
+ // push_back is okay with aliasing nodes


should that be "push_front is okay with aliasing nodes" ?

Indeed, thanks! Fixed.

shepmaster · 2019-04-19T16:32:25Z

Thank you for the clarifications! r=me after squashing ❤️

shepmaster · 2019-04-19T16:35:24Z

@bors r+ rollup

bors · 2019-04-19T16:35:25Z

📌 Commit 8b09d04 has been approved by shepmaster

bors · 2019-04-19T17:12:24Z

⌛ Testing commit 8b09d04 with merge 130dc3e...

fix LinkedList invalidating mutable references The test `test_insert_prev` failed in Miri due to what I consider a bug in `LinkedList`: in various places, `NonNull::as_mut` got called to modify the `prev`/`next` pointers of existing nodes. In particular, the unstable `insert_next` has to modify the `next` pointer of the node that was last handed out by the iterator; to this end it creates a mutable reference to the *entire node* that overlaps with the mutable reference to the node's content that was handed out by the iterator! Thus, the next use if said mutable reference is UB. In code: ```rust loop { match it.next() { // mutable reference handed to us None => break, Some(elt) => { it.insert_next(*elt + 1); // this invalidates `elt` because it creates an overlapping mutable reference match it.peek_next() { Some(x) => assert_eq!(*x, *elt + 2), // this use of `elt` now is a use of an invalid pointer None => assert_eq!(8, *elt), } } } } ``` This PR fixes that by using `as_ptr` instead of `as_mut`. This avoids invalidating the mutable reference that was handed to the user. I did this in all methods called by iterators, just to be sure. Cc @gankro

bors · 2019-04-19T20:15:04Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: shepmaster
Pushing 130dc3e to master...

rust-highfive assigned shepmaster Apr 18, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 18, 2019

RalfJung mentioned this pull request Apr 18, 2019

add LinkedList test and mention the bug Miri found there rust-lang/miri#697

Merged

shepmaster reviewed Apr 18, 2019

View reviewed changes

Gankra approved these changes Apr 19, 2019

View reviewed changes

lqd reviewed Apr 19, 2019

View reviewed changes

fix LinkedList invalidating mutable references

8b09d04

RalfJung force-pushed the linked-list branch from 416fe81 to 8b09d04 Compare April 19, 2019 16:33

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 19, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 19, 2019

bors merged commit 8b09d04 into rust-lang:master Apr 19, 2019

kpp mentioned this pull request Apr 26, 2019

DoubleLinkedList implementation violates aliasing rules rust-rosetta/rust-rosetta#587

Open

RalfJung mentioned this pull request May 28, 2019

Stacked Borrows: asserting uniqueness too early? Should we allow the optimizer to add spurious stores? rust-lang/unsafe-code-guidelines#133

Open

RalfJung deleted the linked-list branch June 10, 2019 11:04

This was referenced Aug 16, 2019

Fix validity problem contain-rs/linked-hash-map#96

Merged

Audit the sentinel node for Unsafe Code Guidelines Violations contain-rs/linked-hash-map#97

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix LinkedList invalidating mutable references #60072

fix LinkedList invalidating mutable references #60072

RalfJung commented Apr 18, 2019

rust-highfive commented Apr 18, 2019

shepmaster Apr 18, 2019 •

edited

Gankra Apr 19, 2019 •

edited

RalfJung Apr 19, 2019

RalfJung Apr 19, 2019

shepmaster Apr 19, 2019

RalfJung Apr 19, 2019

Gankra left a comment

Gankra Apr 18, 2019

RalfJung Apr 19, 2019

lqd Apr 19, 2019

RalfJung Apr 19, 2019

shepmaster commented Apr 19, 2019

shepmaster commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

fix LinkedList invalidating mutable references #60072

fix LinkedList invalidating mutable references #60072

Conversation

RalfJung commented Apr 18, 2019

rust-highfive commented Apr 18, 2019

shepmaster Apr 18, 2019 • edited

Choose a reason for hiding this comment

Gankra Apr 19, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gankra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shepmaster commented Apr 19, 2019

shepmaster commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

bors commented Apr 19, 2019

shepmaster Apr 18, 2019 •

edited

Gankra Apr 19, 2019 •

edited