Skip to content

Conversation

lrh2000
Copy link
Contributor

@lrh2000 lrh2000 commented Oct 18, 2024

Emiting vptrs for empty/auto traits is unnecessary (#114942) and causes unsoundness in trait_upcasting (#131813). This PR should ensure that we never emit vtables for such traits. See the linked issues for more details.

I'm not sure if I can add tests for the vtable layout. So this PR only adds tests for the soundness hole (i.e., the segmentation fault will disappear after this PR).

Fixes #114942
Fixes #131813

Cc #65991 (tracking issue for trait_upcasting)

r? @WaffleLapkin (per #131813 (comment))

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Oct 18, 2024
@rust-log-analyzer

This comment has been minimized.

@WaffleLapkin
Copy link
Member

You can write tests for the vtable layout (see the failing tests that already do that, not sure if there is a need to add more after blessing them)

@lrh2000
Copy link
Contributor Author

lrh2000 commented Oct 18, 2024

You can write tests for the vtable layout (see the failing tests that already do that, not sure if there is a need to add more after blessing them)

Yeah, the failed CI already told me the answer to that question. Will take a look at it.

@rust-log-analyzer

This comment has been minimized.

@WaffleLapkin
Copy link
Member

@bors r+

@bors
Copy link
Collaborator

bors commented Oct 18, 2024

📌 Commit 781bff0 has been approved by WaffleLapkin

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 18, 2024
bors added a commit to rust-lang-ci/rust that referenced this pull request Oct 18, 2024
Rollup of 3 pull requests

Successful merges:

 - rust-lang#126207 (std::unix::stack_overflow::drop_handler addressing todo through libc …)
 - rust-lang#131864 (Never emit `vptr` for empty/auto traits)
 - rust-lang#131870 (compiletest: Store test collection context/state in two structs)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 765e8c7 into rust-lang:master Oct 18, 2024
6 checks passed
@rustbot rustbot added this to the 1.84.0 milestone Oct 18, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Oct 18, 2024
Rollup merge of rust-lang#131864 - lrh2000:upcast_reorder, r=WaffleLapkin

Never emit `vptr` for empty/auto traits

Emiting `vptr`s for empty/auto traits is unnecessary (rust-lang#114942) and causes unsoundness in `trait_upcasting` (rust-lang#131813). This PR should ensure that we never emit vtables for such traits. See the linked issues for more details.

I'm not sure if I can add tests for the vtable layout. So this PR only adds tests for the soundness hole (i.e., the segmentation fault will disappear after this PR).

Fixes rust-lang#114942
Fixes rust-lang#131813

Cc rust-lang#65991 (tracking issue for `trait_upcasting`)

r? `@WaffleLapkin`  (per rust-lang#131813 (comment))
bors added a commit that referenced this pull request Aug 28, 2025
…mpiler-errors

When determining if a trait has no entries for the purposes of omitting vptrs from subtrait vtables, consider its transitive supertraits' entries, instead of just its own entries.

When determining if a non-first supertrait vptr can be omitted from a subtrait vtable, check if the supertrait or any of its (transitive) supertraits have methods, instead of only checking if the supertrait itself has methods.

This fixes the soundness issue where a vptr would be omitted for a supertrait with no methods but that itself had a supertrait with methods, while still optimizing the case where the supertrait is "truly" empty (it has no own vtable entries, and none of its (transitive) supertraits have any own vtable entries).

Fixes <#145752>

-----

Old description:

~~Treat all non-auto traits as non-empty (possibly having methods) for purposes of determining if we need to emit a vptr for a non-direct supertrait (and for new "sibling" entries after a direct or non-direct supertrait).~~

This fixes (I believe) the soundness issue, ~~but regresses vtable sizes and possibly upcasting perf in some cases when using trait hierarchies with empty non-auto traits (see `tests/ui/traits/vtable/multiple-markers.stderr`) since we use vptrs in some cases where we could re-use the vtable.~~

Fixes <#145752>

Re-opens (not anymore) <#114942>

Should not affect <#131813> (i.e. the soundness issue is still fixed, ~~though the relevant vtables in the `trait Evil` example will be larger now~~)

cc implementation history <#131864> <#113856>

-----

~~It should be possible to check if a trait has any methods from itself *or* supertraits (instead of just from itself), but to fix the immediate soundness issue, just assume any non-auto trait could have methods. A more optimistic check can be implemented later (or if someone does it soon it could just supercede this PR 😄).~~ Done in latest push

`@rustbot` label A-dyn-trait F-trait_upcasting
github-actions bot pushed a commit to rust-lang/miri that referenced this pull request Aug 30, 2025
…mpiler-errors

When determining if a trait has no entries for the purposes of omitting vptrs from subtrait vtables, consider its transitive supertraits' entries, instead of just its own entries.

When determining if a non-first supertrait vptr can be omitted from a subtrait vtable, check if the supertrait or any of its (transitive) supertraits have methods, instead of only checking if the supertrait itself has methods.

This fixes the soundness issue where a vptr would be omitted for a supertrait with no methods but that itself had a supertrait with methods, while still optimizing the case where the supertrait is "truly" empty (it has no own vtable entries, and none of its (transitive) supertraits have any own vtable entries).

Fixes <rust-lang/rust#145752>

-----

Old description:

~~Treat all non-auto traits as non-empty (possibly having methods) for purposes of determining if we need to emit a vptr for a non-direct supertrait (and for new "sibling" entries after a direct or non-direct supertrait).~~

This fixes (I believe) the soundness issue, ~~but regresses vtable sizes and possibly upcasting perf in some cases when using trait hierarchies with empty non-auto traits (see `tests/ui/traits/vtable/multiple-markers.stderr`) since we use vptrs in some cases where we could re-use the vtable.~~

Fixes <rust-lang/rust#145752>

Re-opens (not anymore) <rust-lang/rust#114942>

Should not affect <rust-lang/rust#131813> (i.e. the soundness issue is still fixed, ~~though the relevant vtables in the `trait Evil` example will be larger now~~)

cc implementation history <rust-lang/rust#131864> <rust-lang/rust#113856>

-----

~~It should be possible to check if a trait has any methods from itself *or* supertraits (instead of just from itself), but to fix the immediate soundness issue, just assume any non-auto trait could have methods. A more optimistic check can be implemented later (or if someone does it soon it could just supercede this PR 😄).~~ Done in latest push

`@rustbot` label A-dyn-trait F-trait_upcasting
github-actions bot pushed a commit to rust-lang/compiler-builtins that referenced this pull request Sep 1, 2025
…mpiler-errors

When determining if a trait has no entries for the purposes of omitting vptrs from subtrait vtables, consider its transitive supertraits' entries, instead of just its own entries.

When determining if a non-first supertrait vptr can be omitted from a subtrait vtable, check if the supertrait or any of its (transitive) supertraits have methods, instead of only checking if the supertrait itself has methods.

This fixes the soundness issue where a vptr would be omitted for a supertrait with no methods but that itself had a supertrait with methods, while still optimizing the case where the supertrait is "truly" empty (it has no own vtable entries, and none of its (transitive) supertraits have any own vtable entries).

Fixes <rust-lang/rust#145752>

-----

Old description:

~~Treat all non-auto traits as non-empty (possibly having methods) for purposes of determining if we need to emit a vptr for a non-direct supertrait (and for new "sibling" entries after a direct or non-direct supertrait).~~

This fixes (I believe) the soundness issue, ~~but regresses vtable sizes and possibly upcasting perf in some cases when using trait hierarchies with empty non-auto traits (see `tests/ui/traits/vtable/multiple-markers.stderr`) since we use vptrs in some cases where we could re-use the vtable.~~

Fixes <rust-lang/rust#145752>

Re-opens (not anymore) <rust-lang/rust#114942>

Should not affect <rust-lang/rust#131813> (i.e. the soundness issue is still fixed, ~~though the relevant vtables in the `trait Evil` example will be larger now~~)

cc implementation history <rust-lang/rust#131864> <rust-lang/rust#113856>

-----

~~It should be possible to check if a trait has any methods from itself *or* supertraits (instead of just from itself), but to fix the immediate soundness issue, just assume any non-auto trait could have methods. A more optimistic check can be implemented later (or if someone does it soon it could just supercede this PR 😄).~~ Done in latest push

`@rustbot` label A-dyn-trait F-trait_upcasting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

trait_upcasting unsoundness due to reordered super traits Imperfect vtable layout with an empty super trait comming after non-empty one
5 participants