This design is prototyped in #1924. An older, more limited design is prototyped in transmute-slice-experiment.

Currently, transmute_ref! and transmute_mut! only support sized referents, and ensure that the source and destination types are the same size. This enforcement relies on core::mem::transmute, which is special-cased in the compiler (no other language or standard library API provides a compile-time size equality check, except via post-monomorphization error (PME)).

Users often request the ability to transmute dynamically-sized referents – specifically slices and slice DSTs. "Size equality" for slices and slice DSTs is a far more nuanced concept than for sized types. In particular, there are four things we might mean when we say that two slice DSTs "have the same size":

• When performing a fat pointer cast, the destination encodes the same referent size as the source
• When performing a fat pointer cast, the destination encodes a referent size no larger than that of the source
• When performing a fat pointer cast, it is possible to do a "metadata fix-up" operation at runtime such that the destination encodes the same referent size as the source
• When performing a fat pointer cast, it is possible to do a "metadata fix-up" operation at runtime such that the destination encodes a referent size no larger than that of the source

The "no larger than" cases are referred to as "shrinking" casts, and they are tracked separately in #2701. The designed outlined in this issue only supports size-preserving casts, and is the default behavior. Shrinking casts will require a user opt-in to avoid foot guns.

We can of course view casts which do not require a metadata fix-up as a special cast of casts which do require a metadata fix-up, with the caveat that the former will be marginally cheaper at runtime compared to the latter. That said, as we explore below, the fix-up operation itself is very cheap, and so we do not believe that the cognitive complexity of requiring our users to reason about the distinction between these casts is justified by the marginal performance gain. Users who care enough about performance to think about it are of course free to take other measures to ensure that they don't accidentally perform casts which require metadata fix-up.

Progress

Upstream Rust language semantics changes

• Guarantee that raw pointer conversions preserve slice element count rust-lang/reference#1417
• [ptr] Document maximum allocation size rust-lang/rust#116675
  • Required in order to guarantee the soundness post-condition on pointer metadata fix-up operations
• References refer to allocated objects rust-lang/rust#116677
  • Required in order to guarantee the soundness post-condition on pointer metadata fix-up operations
• Clarify ManuallyDrop bit validity rust-lang/rust#115522
  • Required in order to support ManuallyDrop in generic transmutability
• Define raw pointer transmute behavior rust-lang/reference#1661
  • An older version of this design would have supported non-metadata-fix-up casts in a const context using mem::transmute. The current version of this design no longer relies on this language semantics change.
• Clarify atomic bit validity rust-lang/rust#121943
  • Required in order to support atomics in generic transmutability
• Expand mem::offset_of! docs rust-lang/rust#117512
  • KnownLayout's original design involved a de-risking strategy of "racing" multiple design approaches. One of those approaches was predicated on stabilizing offset_of!. That approach didn't end up winning the race, so this design no longer relies on offset_of! being stable.

Externally-tracked designs

• Support KnownLayout trait and custom DSTs #29
• Generic transmutability #1359

PRs

• SizeEq supports casts via method
  • [pointer] SizeEq supports raw pointer transmutes #2427
• Support slice DST transmutations that preserve referent size exactly
  • Teach transmute_{ref,mut}! to handle slice DSTs #2428
• Support sized-to-slice DST transmutations that preserve referent size exactly
  • Can build on @jswrenn's size specialization autoref prototype

High-level design

In the rest of this issue, we will refer to transmute_ref! for brevity. Everything we say of transmute_ref! applies to transmute_mut! as well.

Currently, transmute_ref! validates:

• Src: Sized and Dst: Sized have the same size
• Src has alignment at least as large as Dst

We can view this condition as a special case of a more general condition, which is that Src and Dst have the same size as defined by KnownLayout::LAYOUT.size_info. That condition means, precisely, that either of the following hold:

• Src: Sized, Dst: Sized, and size_of::<Src>() == size_of::<Dst>() (what we currently support)
• Src and Dst are slice DSTs with the same TrailingSliceLayout – the same trailing slice offset and the same trailing slice element size

(Note a subtlety here: It would be tempting to say that, in order to avoid a difference in trailing padding, alignment must be equal. However, this is not the case as described below.)

So long as either of these conditions hold, then a &Src to &Dst conversion will preserve referent size (with the exception of trailing padding in the slice DST case – again, see below).

Metadata fix-up

We can, in turn, view this slice DST case as a special case of a more general notion of size equality. In particular, it is sometimes possible to modify the metadata of a fat reference such that the size is preserved even if KnownLayout::LAYOUT.size_info is not the same. Consider, for example, transmuting &[u16] -> &[u8]: if we double the slice length, then the referent size is preserved exactly.

We can even support cases in which the trailing slice offset differs between types. Consider, for example:

#[repr(C)]
struct Src(u32, [u32]);

#[repr(C)]
struct Dst(u16, [u16]);

A &Src with 2 trailing slice elements has size 4 + 2 * 4 = 12. Thus, we can convert a &Src with 2 trailing slice elements to a &Dst with 5 trailing slice elements.

In the general case, the &Src to &Dst transformation is an affine function. The difference between the slice offsets (in this case, the difference between the 4-byte offset of the trailing [u32] in Src and the 2-byte offset of the trailing [u16] in Dst) must be a multiple of the element size of Dst's trailing slice. Regardless of the metadata of Src, there will always be this fixed component (in this case, the 0th element of the trailing [u16] corresponds to bytes 2 and 3 in Src, and this is true even when Src has a trailing slice of length 0). The remainder is a linear transformation from the trailing slice in Src to the trailing slice in Dst. In this particular example, dst_meta = 1 + (src_meta * 2) – the fixed part is 1, and the linear part is * 2.

Detailed design

Bit validity and transmutation

The most straightforward approach to implementing this feature would be to require the appropriate trait bounds on the Src and Dst types and then to write the relevant unsafe code by hand internally. However, as we've seen in e.g. #2226, that's risky and error-prone. It also requires duplicating the same safety reasoning in multiple places (e.g. in transmute_mut! and in various TryFromBytes and FromBytes methods), which increases long-term maintenance burden and increases the likelihood of soundness holes like #2226 slipping through.

Instead, we're going to use this as an opportunity to unify our transmutability analysis around a generic transmutability framework on Ptr (#1359). This issue is blocked on that unification work, and thus concerns itself only with the pointer metadata aspects of slice DST support, but not with the bit validity aspects.

SizeEq trait

Building on generic transmutability means that the pointer metadata fix-up operation has to live in a location which is reachable from the Ptr transmute machinery. Currently, Dst: SizeEq<Src> is a marker trait that denotes that a *Src to *Dst pointer cast will preserve referent size. In order to support metadata fix-up, SizeEq needs to gain the ability to perform that metadata fix-up. Dst: SizeEq<Src> will no longer denote that some other operation has a particular behavior – instead, it will provide a method that provides the metadata fix-up functionality.

PME bound on trait implementation

Here we run into a seeming contradiction between three unavoidable aspects of this design:

• The code which executes at compile time to compute the size equality predicate (ie, to determine whether a particular &Src to &Dst transmutation is valid from a size perspective) must be written as const code, and failure must be expressed as a const panic, otherwise known as a post-monormphization error (PME).*
• This code must operate on generic types. The alternative would be for all of the code to be generated by macro_rules! macro, and thus to operate on concrete types, but since the code in question is quite complex, that would make for extremely difficult debugging and horrendous error messages for users.
• Building atop the generic transmutability machinery requires that SizeEq be implemented for the Src and Dst types.

* Technically, since Rust's type system is turing complete, it would be possible to encode this predicate in the trait system itself. For myriad, hopefully self-evident reasons, we're not taking that approach.

Thus, we need to support code that looks something like:

fn transmute_ref<Src, Dst>(src: &Src) -> &Dst
where
    Src: IntoBytes + Immutable,
    Dst: FromBytes + Immutable,
{ ... }

Note that transmute_ref appears infallible, but it is not: errors are surfaced at compile time via PME. In order to support transmutability, we need Dst: SizeEq<Src>, but we can't actually implement that! We don't actually know that Dst: SizeEq<Src> until we have computed the relevant predicate. But there's no way in Rust – in a generic context – to say "emit this trait impl only if this const code executes without panicking."

In order to work around this limitation, we introduce the following macro:

/// Invokes `$blk` in a context in which `$src<$t>` and `$dst<$u>` implement
/// `SizeEq`.
///
/// This macro emits code which implements `SizeEq`, and ensures that the impl
/// is sound via PME.
///
/// # Safety
///
/// Inside of `$blk`, the caller must only use `$src` and `$dst` as `$src<$t>`
/// and `$dst<$u>`. The caller must not use `$src` or `$dst` to wrap any other
/// types.
macro_rules! unsafe_with_size_eq {
    (<$src:ident<$t:ident>, $dst:ident<$u:ident>> $blk:expr) => { ... };
}

The idea is that $t and $u are the generic Src and Dst types which are parameters on transmute_ref. The PME logic is called inside the macro body, which ensures that it is sound to emit $dst<$u>: SizeEq<$src<$t>>. However, this is done by emitting a generic impl: impl<T, U> SizeEq<$src<T>> for $dst<U>>. Note that this impl is unsound!! It's only guaranteed to be sound for the particular T and U types $t and $u. That's why unsafe_with_size_eq! has a safety precondition requiring the caller to promise to only use $src and $dst with $t and $u.

Backwards-compatibility with const execution

Today, transmute_ref! works in a const context. However, the machinery we're proposing here relies on trait methods, and those can't be called in a const context.

In order to maintain backwards-compatibility, we will use autoref specialization to delegate to one of two implementations. When Src: Sized and Dst: Sized, we will delegate to a const-friendly implementation. When Src: ?Sized or Dst: ?Sized, we will delegate to the new implementation, which does not work in a const context. Since transmute_ref! only currently supports Sized types, this ensures backwards-compatibility.

The PME delayed compilation footgun

TODO: Explain the footgun and explain why we should require concrete types even though we don't need to

Core algorithm: Metadata fix-up

TODO

Alignment mismatch

TODO: Explain why it's acceptable for align_of::<Src>() >= align_of::<Dst>() for slice DSTs - ie, why it's acceptable for trailing padding to be larger in Src than in Dst

Old design

trait TransmuteRefHelper {
    type Elem;
}

impl<T: Sized> TransmuteRefHelper for T {
    type Elem = T;
}

impl<T: Sized> TransmuteRefHelper for [T] {
    type Elem = T;
}

fn from_elem<T: TransmuteRefHelper>(e: T::Elem) -> &'static T

// Infers `t` as either equal to the sized source value or
// the element type of the source slice.
let mut t = loop {}; 
e = from_elem(t);