own_ref RFC initial commit

[CheaterCodes]

View all comments

Introduce owning references &own into the language, which allows passing ownership of the pointee without moving its value.
Owning references can be moved out of (including partial moves), and drop their pointee when dropped.

fn print_all(producers: &own [&own dyn FnOnce() -> String]) {
    for producer in producers {
        println!("Got value: {}", producer());
    }
}

Important

Since RFCs involve many conversations at once that can be difficult to follow, please use review comment threads on the text changes instead of direct comments on the RFC.

If you don't have a particular section of the RFC to comment on, you can click on the "Comment on this file" button on the top-right corner of the diff, to the right of the "Viewed" checkbox. This will create a separate thread even if others have commented on the file too.

Rendered

[CheaterCodes]

To create the appropriate cross-links to relevant issues:

• Issue for owned references: Owned references to contents in an earlier stack frame (&own, &move, etc) #998
• Issue for moving/owned deref: Extending deref/index with ownership transfer: DerefMove, IndexMove, IndexSet #997

[clarfonthey]

Here's my thought on this: I like the concept, but feel like requiring a third reference type is probably not what we want for API explosion issues.

Here's another thought: what if this were a property of lifetimes instead? Really, what you want is to be able say is that a lifetime is maximal for a given type. This would mean that generally, as long as the lifetime is kept the same through methods, you don't have to worry about "ownership" being tracked through all involved methods, only ones that would explicitly need to know about the drop behaviour.

We can bikeshed the syntax however we want, but how I see it, the main distinction is that whenever a lifetime is explicitly tagged as "maximal", it has ownership mechanics. This also allows an interesting case not covered by this RFC: mutability of a reference simply controls whether the data is allowed to be modified before it's moved, rather than after. While I'm struggling to imagine API scenarios where this is useful, it is technically an interesting option.

Also commenting a bit on the syntax bikeshed, maybe something like:

fn box_move<final 'a>(data: &'a T) -> Box<T> {
    // ...
}

Which could maybe have some shorthand syntax like:

fn box_move(data: &final T) -> Box<T> {
    // ...
}

Obviously requires a lot of extra detail/bikeshedding, but, I feel like focusing on the lifetime having properties rather than the reference itself makes a lot more sense. We could also maybe use it to extend some existing logic to allow "move and put back" type actions:

let value = *x;
*x = value + 1;

Where the difference on a maximal lifetime is that you are not allowed to "put back" the value; it has to be moved out and dropped.

View changes since the review

[ChayimFriedman2]

This isn't really a property of lifetimes, which only considers for how long a value lives. This is a property of ownership, similar to & vs. &mut. As an example, what is the meaning of Type: final 'a?

[ais523]

I don't think this has any more to do with lifetimes than the difference between & and &mut has to do with lifetimes.

In the past, I have considered the question of writing methods that are generic over reference type, and one possibility I considered is to express the difference between & and &mut as a lifetime (i.e. you can consider a mutable reference to be one that has a mutable lifetime). It kind-of works, so I'd expect expressing the difference between & and &own as a lifetime, or between &mut and &own as a lifetime, to also kind-of work. But it doesn't really capture the fundamental differences between the types of references.

In any case, most methods that could reasonably be & or &mut are not meaningful with &own. For example, you can borrow an &Vec<T> as an &[T], or an &mut Vec<T> as an &mut [T]. It does not make sense to borrow a &own Vec<T> as a &own [T] (because you cannot drop the Vec while keeping the memory it owns alive). Most other methods with an & and an &mut version would at minimum create a memory leak if written as an &own version. (The "put the value back" idea in the original post is actually an exception to this, but I think it corresponds to take_mut rather than to &own and the two are significantly different features which should be handled by different RFCs.)

A good way to think about it is that any method that takes an &'a own (to some type) should probably also return an &'a own (to some, possibly different, type) – if it does not, the memory that contained it will be unusable for the rest of the lifetime 'a. This means that the APIs that make sense for owning references usually look significantly different from those that make sense for mutable references. (The iterator example in the RFC is a good one: a mutable-borrow iterator looks like fn next(&mut self) -> Option<T>, whereas an owning-reference iterator looks like fn next(&own self) -> Option<(T, &own Self)> or perhaps even fn next(&own self) -> Result<(T, &own Self), &mut MaybeUninit<Self>> so that the memory could be reused after the iterator finishes.)

[CheaterCodes]

I agree that lifetimes are mostly unrelated to this. Especially since here, it would mean that lifetimes affect post-mono behavior (whether to drop), which just doesn't work in Rust.

Regarding API explosion: This is of course a valid concern, like it is with most pointer types. However, I also believe that it is nowhere near as bad as with & and &mut: After all, e.g., writing an accessor does not really make sense with &own, at least not in the current formulation. I imagine that &own will see significantly less API compared with other references.

I am happy to mention API explosion in drawbacks though, if there are others with similar concern?

[ChayimFriedman2]

This interacts heavily with the Beyond the & project goals family. See also the #t-lang/custom-refs Zulip channel. People have been discussing this for a long time now, I don't think we should ignore all that discussion (while the RFC has a section for the history of &own/&move, which is good, it does not include those).

View changes since the review

[CheaterCodes]

I have been somewhat involved in those discussions as well, so I believe I have a mostly complete picture of said proposals. My argument here (which I thought I put into the RFC, but maybe forgot) is that we would likely want an owning reference with the proposed semantics in any case, even if we support custom reference types. This just means that this proposal would change from a lang feature to a libs API, with the same contents (except syntax).

[ais523]

This RFC is missing what is to me the most important use for &own: creating references that can change the type of their target. For me this is an extremely strong motivation for &own, and so I'd like to see it mentioned.

A toy example of this sort of API would be fn try_as_nonzero(n: &own usize) -> Result<&own NonZeroUsize, &own usize>: you give it an owning reference to a usize and get back an owning reference to it as a NonZeroUsize, if possible.

More useful examples include things like fn drop_in_place(t: &own T) -> &own MaybeUninit<T> (i.e. "drop a value we own, while keeping the memory containing it"). This makes it possible for safe code to use the same memory to store multiple different types of objects; currently, doing that in safe code without a memory leak requires storing the object via MaybeUninit::<Option<T>>::write (allowing Option::take to drop it), but this has both the memory overhead of Option and the "it's hard to prove this program doesn't panic" overhead of Option).

View changes since the review

[programmerjake]

fn drop_in_place(t: &own T) -> &own MaybeUninit<T> seems unsound, the &mut T equivalent is unsound because you can use it to overwrite an enum tag that was niche-optimized so is in the same memory as T, which you can then overwrite by writing MaybeUninit::uninit()

[ais523]

@programmerjake This was discussed recently at rust-lang/unsafe-code-guidelines#618. It has not yet been decided for &mut whether or not it is legal to overwrite an enum discriminant from within one of its own variants, if the original reference is never used again (but I was hoping for a decision to be made that it would be).

For &own I think there is less downside than with &mut in allowing it (creating an &own reference to an enum's fields conceptually requires destroying the enum and thus it no longer has a discriminant to overwrite), and more upside than with &mut in allowing it (because writing this sort of code would otherwise require unsafe Rust).

[CheaterCodes]

One thing we did talk about was having some type, let me call it AlreadyDropped<T>, which behaves like ManuallyDrop but is known to be dropped. With such a type, I could see APIs like drop_in_place<T>(&own T) -> &own AlreadyDropped<T> and AlreadyDropped<T>::write(&own self, val: T) -> &own T.
Here, AlreadyDropped would guarantee a valid bit pattern for T, so that these operations can be safe and sound.

I'm definitely happy to put fn try_as_nonzero(n: &own usize) -> Result<&own NonZeroUsize, &own usize> as an interesting API in the appendix, but if you think this should receive a bigger spotlight in the Motivation section, I'm sure we can come up with something compelling there as well.

[ais523]

I believe that, unlike &'a mut T, &'a own T should not have the well-formedness condition that T: 'a (in other words, it should be legal to have an &'a own T for which the T is not valid for entire lifetime 'a. With &'a mut T, at least in safe code the reference has to store a T for the entire lifetime 'a, so it makes sense to require T to live that long. With &'a own T, it is possible to drop the T before the lifetime 'a ends, and so it is both reasonable and useful to store a short-lived value of type T into long-lived memory that lasts for 'a (many of my programs that want to do this are temporarily storing short-lived values into long-lived memory, but want to remember that the memory is long-lived).

I think, but am not totally sure, that this is compatible with the requirement to drop the T when an &'a own T is dropped (because you can't drop a type outside the lifetime of any of its generic parameters, so even though an &'a own T could be alive during parts of the lifetime 'a where T is dead, the reference couldn't be dropped at such times, only leaked or forgotten).

If adopted, the "there is no requirement that T: 'a" condition should be added to this list, as it is a difference from mutable references that is not currently listed.

View changes since the review

[programmerjake]

you can drop things where their internal lifetimes have expired, that's what #[may_dangle] does (used in container types in std): https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=42e06a037f1711bed836023f2b6fb163

[ais523]

I forgot about #[may_dangle], but I don't think it matters here: if a wrapper Wrapper does not use #[may_dangle] (regardless of what else might be using it), you can't drop a Wrapper<T> after T's lifetime has expired except in cases where it's safe to drop a T after T's lifetime has expired. Thus, in any situation where you drop an &'a own T after T's lifetime has expired, it should be safe to drop the T.

[CheaterCodes]

This briefly came up in discussions prior to this RFC, though I don't think we elaborated much on it. One thing is sure though: Even if we drop the implicit bound &'a own T implies T: 'a, in many cases this will be required by dropcheck. In particular, any function that takes a generic &'a own T and drops it inside its scope must require that T: 'a, since it otherwise calling drop(T) would be called on dangling lifetimes.

This can be relaxed only if we know we do not need a valid T for dropping, e.g. if it is copy or marked with may_dangle in the appropriate drop implementation. (Or we never drop it in the generic function.)

[ais523]

I don't agree that this is required by dropcheck. If you take a generic &'a own T, then even without the well-formedness requirement, you must be both inside the lifetime 'a and inside the lifetime of T (because a function/method can only be called if all its generic parameters are alive, both types and lifetimes). Thus, it should be OK to drop it.

This is making me think that Own<'a, T> may be the right name for this sort of reference, because it makes it more obvious that the lifetime of the reference is tied to both 'a and T rather than just 'a.

[ais523]

Another drawback to implementing this with a library implementation (based on experience with trying to implement it): there is no way to convey the fact that an &own is owned to the compiler, so it will generate code assuming that it might have aliases. This leads to worse-quality code generation.

View changes since the review

[CheaterCodes]

Good point, though this could be fixed by using Box<T, Noop<'a>> as the pointer type, which currently provides noalias.

[ais523]

Strangely enough, it doesn't: putting noalias on a Box<T, A> is actually unsound in the current Rust operational semantics unless A allocates only fresh memory from entirely outside the Rust abstract machine, and is also unsound in the current LLVM operational semantics unless A is entirely opaque to LLVM. (The current resolution to these problems are that LLVM does not inline or otherwise optimize around calls to global memory-allocation functions, and Rust does not put noalias on Box unless it is using the global allocator). There's at least one RFC on LLVM's side to try to address the situation (and I've been working on a not-yet-posted proposal to do so from the Rust side, too).

[ais523]

It doesn't: such a reference would be able to implement DerefMut, and it would almost certainly be a good idea to. With that trait implementation, Own<'_, T> would reborrow automatically as a receiver, and would allow manual reborrowing (e.g. in argument position) using the syntax &mut *o (where o is the Own<'_, T>). I am inclined to think that automatically reborrowing in argument position is a bad idea regardless of the syntax used for the reference (basically because reborrowing an owning reference does something significantly different from moving it, and both are plausible uses for an owning reference in argument position), so this syntax actually doesn't have any syntax overhead for reborrowing.

View changes since the review

[CheaterCodes]

Maybe this wasn't well formulated here, (re-)borrowing as &mut is certainly possible.
(Re-)borrowing of expressions as &own is what I'm concerned about, i.e., how to replace &own expression.

(I should remove the "re-" part here anyway, since that likely doesn't make much sense for owning references)

[ais523]

I see the &own operator as more like a move than a borrow: I guess you can think of it as a delayed move (because you are creating an object with the ability to move the thing it points to; from a borrow-checking point of view, &own as an operator can be thought of as moving a value and borrowing the memory that previously contained it). We probably need a good name for the operator that makes it clear what it does, because Rust's current terminology doesn't seem to be up to the task of naming it unambiguously.

Having a good name for the operator would both be important for teaching people how this sort of reference works, and help to avoid misunderstandings like this.

[ais523]

There has been discussion about splitting the concept of an allocator from the concept of a deallocator (and allowing a Box to contain just a deallocator). Although this isn't currently implemented in allocator_api, it is as I understand it a possibility that is intentionally being left open.

I believe that &own is the equivalent of a Box that has a no-op deallocator that is not usable as an allocator. In the world where deallocators and allocators are separate, some of Box's methods/traits would require Box's second type parameter to be both an allocator and deallocator, whereas others would be usable with just a deallocator; and in that world, Box::clone would require a full allocator, so it would not be implemented on Own<'a, T> regardless of whether T were Clone or not.

That said, I think the "implement &own in terms of Box" technique is backwards; Box probably can, and probably should, be implemented in terms of &own instead (although there are likely to be some opsem issues with this, I am hopeful that they can be resolved).

View changes since the review

[CheaterCodes]

Interesting, I have not seen such proposals. Certainly, if "Box that doesn't allocate and does nothing when deallocating" fits into the design of Box, then we could do that, although it still feels a bit hacky.

I agree that we could probably implement Box as a struct Box<T, A>(unsafe<'a> &'a own T, A) (or similar), since &own mostly represents what we want Unique to be.

[ais523]

I think Pin<&'static own T> is sound with regards to the Pin drop guarantee (basically because if the &'static own gets forgotten, there is no way to ever access the memory again).

Pin<&'a own T> with non-static 'a is unsound, but this is in the same way (and for the same reason) that Pin<Box<T, A>> is unsound unless A: StaticAllocator. As such, I think it conceptually has the same solution, "it's unsound to pin through something that owns memory if the memory could be repurposed outside its lifetime, so doing so requires unsafe unless the memory can be proven to live for 'static". To me this is a strong argument against remote drop flags: there isn't really a reason to handle this any differently from how Box handles it.

View changes since the review

[maxdexh]

Yes, &'static own is safe to pin. I think this was meant in comparison to what the moveit crate does for its owning refs, which support pinning via remote drop

[CheaterCodes]

Yes, a Pin<&'static own T> is sound, which i briefly mentioned in the drawbacks section. And by the Box<T, Noop<'a>> analogy, we can see how that relates to StaticAllocator. And while I agree that it has the same solution, with Box you can make a Box<T, A> sound if you're careful around creating it (by using a proper allocator), whereas a Pin<&'a own T> is always unsound (unless 'a: 'static). This to me is a meaningful difference.

While I'm not a fan of adding drop flags, I don't understand how this is a strong argument against them though: Sure, we can just say "its unsound, don't do it", but with drop flags it can be sound, so surely that's an argument for it? Can you elaborate on this maybe?

[ais523]

The argument is basically that I wouldn't expect pinning an &own to have more functionality on owned references than Box::pin/Box::into_pin do on boxes; it would be much easier to fit a drop flag into a Box than it would be to fit it into a reference, so if such an API were provided, I would expect to see it on Box first.

[kennytm]

🤔 So raw can be contextual in &raw const because of the full keywords const and mut which prevents existing code from using the same sequence. But &own standing on its own would really need own to be a full keyword and otherwise we'll get very weird syntax or type errors if people aren't migrating via cargo fix.

Example:

• &own[1..] previously means &(own[1..]) of type &[T], after this RFC it means &own([1..]) of type &own [RangeFrom<i32>].
• &own() previously means &(own()) calling the own function and gets a temporary reference, after this RFC it is an owned reference of the unit type
• if x != &own { stmt; } previously is a normal conditional statement, after this RFC it compares x with the owned reference of the result of { stmt; } (and ends up with syntax error because the if expression is missing a block)
• &own!(expr) previously is a reference of the result of the own! macro, after this RFC this becomes &own (std::ops::Not::not(expr)).

View changes since the review

[CheaterCodes]

Oh hm, I thought they were contextual because they appeared after the &... So this still works, but migration would be a bit of a pain if not done automatically...
I guess in some weird edge cases this change could even go unnoticed and open up soundness holes and such.

[CheaterCodes]

Would be interesting to see how often this appears in real code, but probably occasionally, especially with the macro and function, which to comply with naming guidelines

[CheaterCodes]

Actually, making own a full keyword doesn't help with this at all, does it? You would still get all the problems you mentioned, just that we'd do it over an edition, presumably while linting in older editions.

In that sense, a contextual keyword would still significantly reduce the breakage.

[ais523]

&move I think is likely to produce less breakage; are there any ambiguities other than with &move || {…}? You could resolve that ambiguity using precedence rules (i.e. interpreting the ambiguous case as &(move || {…}) for backwards compatibility, and writing the other possibility as &move (|| {…})).

&own with a full keyword would work better in a new edition, but would frequently have to be written as &k#own in older editions, which is a bit of a mess (and possibly also not 100% backwards compatible, although that particular sequence of tokens is unlikely to appear in a macro argument).

There's also the possibility of decoupling the syntax for the operator and for the type, e.g. Own<'a, T> for the type and .move for the operator.

[ais523]

This section is missing prior art from other languages.

Most notably, C++ has the concept of an "xvalue", which is a value that has identity and can be moved out of; an "xvalue reference" would be analogous to the concept discussed in this RFC. (It isn't exactly the same, because moves in C++ work differently to how they work in Rust; in Rust, a moved-from object is unsound to use from safe code, whereas in C++, a moved-from object is re-initialized to a valid but unspecified value. But it seems to be the same if you allow for the difference in how moves work.)

C++ doesn't have xvalue references, but it does have rvalue references, which are like xvalue references with the exception that you cannot observe their address ("rvalues" include xvalues and temporaries, and the "cannot observe their address" restriction appears to be intended to prevent you reusing memory that was used to store a temporary). These use the syntax T&& for the type (C++'s normal reference type is written T& and is the equivalent of Rust's &UnsafeCell<T>), and do not have an explicit operator to create them (rather, they are created automatically when you attempt to move a T into a variable or function argument that expects a T&&). The usual way to intentionally create an rvalue reference is to add a call to std::move, which is the identity function on type T&& (so the argument gets moved into the function as an rvalue reference, and then the rvalue reference is returned).

Rust already provides a way to see the address of a temporary (you can apply &mut to a temporary in Rust), and thus it doesn't need a distinction between xvalues and other types of rvalues (nor a way to prevent observing the address of an rvalue that is not an xvalue). As such, translating the C++ design to Rust would look something like this: an value of type T can be coerced into a value of type &own T, which has a borrow-checker effect similar to moving the value and borrowing the memory containing it (and the resulting &'a own T represents both ownership of the value, and a borrow of the memory for lifetime 'a).

It's notable that C++ went from having one reference type to two once rvalue references were added, which seems like even more of an extreme change than adding a new reference type to Rust. (Rust has two "main" reference types, &T and &mut T, but also a number of more minor reference types like Pin<&mut T> and &Cell<T>, and also Box which is not technically a reference type but acts a lot like one in practice.)

View changes since the review