Skip to content

self-referential data support (v2) - #181

Merged
nbdd0121 merged 20 commits into
mainfrom
dev/selfref
Oct 11, 2026
Merged

self-referential data support (v2)#181
nbdd0121 merged 20 commits into
mainfrom
dev/selfref

Conversation

@nbdd0121

@nbdd0121 nbdd0121 commented Sep 29, 2026

Copy link
Copy Markdown
Member

This adds self-referential data support, making this possible with safe code:

use pin_init::*;

#[pin_data]
struct SelfRef {
    part: &'str str,
    str: String,
}

fn use_self_ref() {
    stack_pin_init!(let foo = pin_init!(SelfRef {
        str: "hello world".to_owned(),
        part: &str[..5],
    }));
}

@nbdd0121
nbdd0121 marked this pull request as draft September 29, 2026 17:04
@nbdd0121
nbdd0121 force-pushed the dev/selfref branch 4 times, most recently from 3c50015 to 9768388 Compare October 7, 2026 11:10
@nbdd0121
internal: pin_data: infer self-referential struct
44e77f1 
As a first step towards adding self-referential data structures in
pin-init, add parsing support.

Scan all field types for unbounded lifetimes, and if the names that of
fields, it is inferred as a self-referential field lifetime. No explicit
annotations are supported yet.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
nbdd0121 marked this pull request as ready for review October 7, 2026 14:19
@nbdd0121
nbdd0121 force-pushed the dev/selfref branch 2 times, most recently from ec2771f to 31ae04d Compare October 8, 2026 11:53
@nbdd0121
internal: pin_data: rewrite fields that borrow others
d3e89e0 
Fields that borrow other fields have lifetimes that are within the struct
and these are not part of the struct generics. Therefore, these fields need
to have their lifetime erased.

A naive implementation would be to replace their lifetimes with `'static`.
However, doing so is unsound for multiple reasons:
* Users may directly access such field with field access syntax, and get
  exposed with wrong lifetime;
* Auto trait implementations will cause the struct to be implementing auto
  traits when the type only implements the auto trait for specific
  lifetime. This is similar to how specialization can be unsound if
  specialized on lifetime.

Create a `Erase` type, which has `for<'a> fn(&'a ()) -> Foo<'a>` as generic
parameter. Internally, it uses a helper trait to resolve that to
`Foo<'static>`. The first issue is solved by not exposing any public
accessor on that type. The second issue is solved by add custom `Send` and
`Sync` implementations that requires `Send` to be implemented for all
lifetimes, thus closing the lifetime specialization hole. The actual
implementation is a bit more convoluted because it supports erasing
multiple lifetimes.

This is more or less a stable polyfill of the unstable `unsafe_binder`
feature, without `unsafe_binders`'s no drop glue requirement.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: pin borrowed fields with wrapper
ab3dce4 
For fields that are borrowed, a mutable reference to the struct no longer
mean that it has the permission to access these fields. Therefore, the
memory that they refer to must be pinned.

Wrap these fields inside a `Borrowed` struct which pins it. They may be
accessed directly (if they're not themselves referencing other struct
fields), so implement a `Deref`.

As such fields are always pinned, there is no need to generate a
conditional `Unpin` implementations that implements `Unpin` when all fields
are. Simply generate a never satisfiable `Unpin` implementation to prevent
user from adding their own.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: teach drop check about generics that cannot dangle
897485f 
Lifetimes not needed by drop glue are considered by Rust's drop check to be
considered `#[may_dangle]`. In case for a self-referential struct, we may
have fields which need lifetime of borrowed fields in their drop glue, so
compiler's automatic check is insufficient.

Code like this:

    #[pin_data]
    struct SelfRef<'a> {
        borrow: PrintOnDrop<&'owner str>,
        owner: &'a str,
    }

may access `owner` during the drop, however Rust will determine that since
`'a` only is used in `owner`, the `'a` lifetime may dangle during drop.

This is undesirable for pin-init self references, because `&'a str` could
be coerced to `&'owner str` and this could further coerce if there're
implied outlives, e.g. `&'earlier_field &'owner ()` would allow `&'owner
str` to further coerce to `&'earlier_field`.

Thus, if any self-referential field require field lifetime access in `Drop`
impl, we would need to ensure that the all generic parameters visible by
self-referential fields would strictly outlive the struct. And this can be
done by a simple `Drop` impl that does nothing. Without a dropck eye patch,
presence of `Drop` impl, albeit empty, tells the drop check that the strict
outlive relation is needed.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
nbdd0121 force-pushed the dev/selfref branch 2 times, most recently from ff8c573 to f9fbad8 Compare October 9, 2026 10:11
@nbdd0121
internal: pin_data: self-referential drop order checks
feae46e 
Check drop order to ensure that usage of lifetime inside self-referential
struct is consistent with the order that the fields will dropped in drop
glue.

First, fields are checked according to their index to ensure that if `a`
borrows from `b`, `b` must outlive `a`. This is simple and produces a very
good diagnostic when misused.

Lifetime bounds can also be indirectly crafted with implied bounds that
make fields well-formed. For example, in this struct

    struct Foo {
        x: &'b &'a (),
        a: String,
        y: PrintOnDrop<&'b str>,
        b: String,
    }

`&'b &'a ()` will imply that `a` outlive `b`, which is inconsistent with
the actual drop order. For this case, create a `__drop_order_check`
function with field lifetimes and outlive relationship of them as generic
parameter, and ask Rust to prove that the types are well-formed inside the
generated function, to ensure that the bad implied bounds cannot happen.

The `__drop_order_check` also need to correlate lifetimes or types captured
by generics and the field lifetimes. Do this by inserting outlive bounds
when a field mentions a specific type or lifetime parameter.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: check covariance of self-referential fields
17e5fc8 
We implicitly infer covariance for fields that self-references. This needs
to be checked to ensure that the fields are really covariant, so the rest
of expansion code can rely on this fact.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: implement initialization of borrowed structs
237b17a 
We now have the checks to ensure that lifetime relations are what is
expected, we can generate the slot projections in `generate_pin_data` so
self-referential struct can be implemented.

New slot and guard types are defined (`SelfRefSlot` and `SelfRefDropGuard`)
which gives the generated let bindings longer lifetime than the guard
themselves.

Have `__make_closure` take `data` back as an argument. This gives
`#[pin_data]` an opportunity to change the type to add lifetimes.
Higher-ranked trait bound on `__make_closure` is used to ensure that the
initialization closure cannot make arbitrary assumptions of those
lifetimes.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: project self-referential fields
9ee8803 
This adds the projection for fields that are shared borrowed or that
borrows other fields but is covariant. Both cases allow a shared reference
to be accessed.

No mutable references can be created for these cases for different reasons:
* For fields that are shared borrowed, aliasing restriction prevents
  creation of mutable reference
* For fields that borrow other fields, their proper type contains field
  lifetimes. These lifetimes cannot be made available in the returned
  `project` struct (because there is no way to represent existential
  lifetime in return position). For covariant types, it is possible to
  shorten these lifetimes to that of `&self`; but doing so requires the
  reference to also be covariant over the pointee type, so we cannot give
  out `&mut` as it is invariant over the pointee.

Due to field-referencing fields being wrapped inside `Erase`, the normal
accessor syntax stop working; create accessor methods for these fields
instead.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: add with_project method
ca0985e 
The `project` method needs to perform covariant coercion on covariant
fields, causing them to no longer being mutable. Implement a `with_project`
that does not require covariant coercion by using higher-ranked trait
bounds, thus allow the fields to be assignable inside the callback.

This mechanism can also be used to access non-covariant fields.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: enable self-referential support
b66eb3a 
Enable self-referential support, and add example and test cases.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: allow lifetime to be shortened per field drop order
2238c9d 
Add the outlive relations per field drop order. This allows a single
lifetime to be used when a field potentially borrow from two different
fields, by allowing the longer-living field lifetime to be shortened to a
shorter-living field lifetime.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: parse explicit #[borrowed] annotation
7b76ee2 
`#[borrowed]` attribute explicitly marks a field as potentially being
borrowed by other fields.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: support mutable borrows
ebf2b0a 
Allow fields to be mutably referenced by other fields in addition to shared
references. In order for this to be sound, the fields that can be mutably
borrowed are blocked from being accessed via field access syntax or
projection to maintain the aliasing requirements.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: parse explicit #[uses] annotation
d23c5bc 
Add support for explicit self-referential annotations.

`#[uses]` attribute is used to mark what other field lifetimes are
captured by this field, and also the variance of the type in respect to the
field lifetimes. For example,

    #[uses('a: covariant, 'b: invariant)]

indicates that the field captures the field lifetime `'a` covariantly, and
field lifetime `'b` invariantly. Many types are covariant, so this is the
default variance if the variance is omitted (e.g. `#[uses('a)]`),
consistent with the automatically inferred borrow.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: make field lifetime invariance imply type invariance
582ac96 
If invariant field captures a field lifetime, then we also need to make
sure that field is also invariant. Imagine this struct:

    #[pin_data]
    struct SelfRef<'a> {
        #[uses('outer: invariant)]
        part: Mutex<&'outer str>,
        outer: &'a String,
    }

    fn new<'a>(str: &'a String) -> impl PinInit<SelfRef<'a>, Infallible> {
        pin_init!(SelfRef {
            outer: str,
            part: Mutex::new(*outer),
        })
    }

If we make this struct covariant over `'a`, then we can have the following
case:

    let mut long = "hello world".to_owned();
    let s = Box::pin_init(new(&long)).unwrap();
    {
        let mut short = "hello world".to_owned();
        // If `s` is covariant, this would be okay, because we shorten from
        // `SelfRef<'long>` to `SelfRef<'short>`.
        s.with_project_ref(|p| {
            *p.part.lock().unwrap() = &short;
        });
    }

Conceptually, a field's type must outlive the field's lifetime, so if we
have a covariant `'a`, we can have a shortened `SelfRef<'short_a>` where
`'outer` outlives `'short_a`. But the wellformedness requirement of the
field will imply `'short_a: 'outer`, which enables the `&'short_a` to
`'outer` coercion, effectively making the field lifetime `'f` behave
covariantly, too, breaking the requirement that it is invariant.

Therefore, compute an invariant closure and use an additional generics on
`Borrowed` to allow capturing things invariantly. Note that we do capture
all parameters explicitly rather than capture the field type invariantly,
because if type aliases are involved, we might be syntactically determining
that the field uses a type parameter but actually not, causing the
invariance enforcement to be missed.

Outlive bounds between field lifetimes can also cause the same issue (an
invariant field lifetime cannot be a lower bound of a covariant field
lifetime). Since we cannot deduce whether any implied bounds from type
wellformness would create such outlive relationships, prevent such bounds
from happening by using a split outlive chain.

Note that this is not a soundness hole in itself in absence of
`with_project_ref`, because with single field accessors only, the field
lifetimes of the fields are not connected; in methods like `with_project`
lifetimes are invariant so shortening cannot happen. However, such method
is likely desirable, so include the variance rule before it has been
heavily relied upon.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: complete invariant borrow support
cc92c50 
Non-covariant types can already be accessed inside projections. As
projections are only generated for `Pin<&mut T>`, they're not accessible
otherwise. Add `with_{field_name}` methods so fields can be accessed using
closures with just `&T`.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: perform AST lifetime replacement if possible
63cd2f7 
Currently lifetimes are replaced via function type and `FnOutput` trait.
This is very general approach as it uses generic associated type to replace
lifetime, so it can even work when macros are involved. This does cause
more generated code, and does not render in documentation nicely.

Thus, just replace the lifetime in the AST if no macros are involved.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: support shared projection
3a36d0e 
For fields that self-references, it is desirable that projection can
happen on shared references too, so lifetime between different fields can
be correlated. Add support for that with `with_project_ref`.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
internal: pin_data: support existential lifetimes
315a807 
There are many cases where structs that have interior mutability, but they
do not need the invariance over captured lifetimes as the lifetime is
captured upon construction, and new data of that particular lifetime does
not flow back into the struct.

For these use cases, the same mechanism as pin-init self-reference may be
used. Add support for existential lifetimes, introduced by having where
clauses such as

    exists<'a>: 'b

The lifetime `'a` above is minted similar to field lifetimes. Because `'a`
is an erased lifetime living longer than `'b`, the only variance
requirement that we have is that `'b` cannot be contravariant; that defense
is fulfilled by adding `PhantomData<&'b ()>` so the struct is either
covariant or invariant over `'b`.

Signed-off-by: Gary Guo <gary@garyguo.net>
@nbdd0121
tests: add self reference test suites
bb0c889 
These are test suites that are gathered when developing pin-init
self-reference and is a collection of unsoundness issues discovered along
the process.

Signed-off-by: Gary Guo <gary@garyguo.net>
nbdd0121 added a commit to nbdd0121/linux that referenced this pull request Oct 10, 2026
@nbdd0121
rust: pin-init: create self references safely
e611ffc 
This is a big series that add support for one field to reference a sibling
field in a safe and ergnonomic way. Unlike many other crates in the
userspace Rust ecosystem, no additional allocation is required, thus it
requires the struct to be pinned, which is exactly what pin-init provides.

This is a very powerful feature, and thus can raise concerns about whether
it is sound. I spent a lot of time studying the rules to make this sound,
and presented durirng Kangrejos [1].

The simple use case looks like this:

    #[pin_data]
    struct MyDriver<'bound> {
        dev: &'bound Device,
        #[pin]
        irq: irq::Registration<'bound, MyIrqHandler<'bound, 'bar>>,
        bar: Bar<'bound, BAR_SIZE>,
    }

You simply need to mention a lifetime that shares the name with the field.
There are more advanced use cases which require annotations like

    #[uses('bar: invariant)]

to explicitly declare that the lifetime is used in invariant manner, and
also

    where
        exists<'dev>: 'a

to create new existential lifetimes as a way to erase invariant lifetimes
that would otherwise bubble up to users. More info can be seen in my
Plumbers slide [2]. Note that the syntax for explicit variance annotation
has changed following discussions during Plumbers.

The initial plan was to upstream the simple use case only and iron out the
advanced features subsequently; however it turns out that DRM jobqueue
would need the variance annotation feature, and Nova `Cmdq` would need to
use the existential lifetime feature.

During Plumbers the upstream schedule is discussed, and instead it was
agreed that all the features should be upstreamed at once, but the usage of
the advanced feature usage limited to the pre-agreed users only to limit
the blast radius in case the feature needs to be reworked.

Detailed documentation about the pin-init self-reference feature would be
added the following cycle, when the usage of them become more clear.

The pull request on GitHub [3] has a bunch of test suites, which are not
synchronized to kernel tree.

To: Benno Lossin <lossin@kernel.org>
To: Miguel Ojeda <ojeda@kernel.org>
To: Boqun Feng <boqun@kernel.org>
To: Björn Roy Baron <bjorn3_gh@protonmail.com>
To: Andreas Hindborg <a.hindborg@kernel.org>
To: Alice Ryhl <aliceryhl@google.com>
To: Trevor Gross <tmgross@umich.edu>
To: Danilo Krummrich <dakr@kernel.org>
To: Daniel Almeida <daniel.almeida@collabora.com>
To: Tamir Duberstein <tamird@kernel.org>
To: Alexandre Courbot <acourbot@nvidia.com>
To: Onur Özkan <work@onurozkan.dev>
Cc: linux-kernel@vger.kernel.org
Cc: rust-for-linux@vger.kernel.org
Link: https://kangrejos.com/2026/Self%20referential%20pin-init.pdf [1]
Link: https://lpc.events/event/20/contributions/2498/attachments/2200/4855/presentation.pdf [2]
Link: Rust-for-Linux/pin-init#181 [3]
Signed-off-by: Gary Guo <gary@garyguo.net>

---
Changes in v3:
- EDITME: describe what is new in this series revision.
- EDITME: use bulletpoints and terse descriptions.
- Link to v2: https://patch.msgid.link/20261008-dev-selfref-v2-0-e280b3c8fba5@garyguo.net

Changes in v2:
- Add where bounds to covariance check (Sashiko)
- Use `NotVisible` mechanism for `with_project_ref` instead of filtering (Sashiko)
- Drop `pub` from `SelfRefSlot` (Sashiko)
- Fixed some stale comments (Sashiko)
- Picked up Benno's Ack.
- Link to v1: https://patch.msgid.link/20261008-dev-selfref-v1-0-6c1eb269fe57@garyguo.net

--- b4-submit-tracking ---
# This section is used internally by b4 prep for tracking purposes.
{
  "series": {
    "revision": 3,
    "change-id": "20261006-dev-selfref-27c37abfa849",
    "prefixes": [],
    "presubject": "",
    "history": {
      "v1": [
        "20261008-dev-selfref-v1-0-6c1eb269fe57@garyguo.net"
      ],
      "v2": [
        "20261008-dev-selfref-v2-0-e280b3c8fba5@garyguo.net"
      ]
    }
  }
}
@nbdd0121

nbdd0121 commented Oct 11, 2026

Copy link
Copy Markdown
Member Author

Merging with Benno's Ack on mailing list. There are some further cleanups possible and this should probably be extensively documented, but these could be done later, and would benefit from testing from real user testing from early adopters.

@nbdd0121
nbdd0121 merged commit de29203 into main Oct 11, 2026
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nbdd0121