Repository navigation
Conversation
|
@bors try @rust-timer queue |
|
@joshlf: 🔑 Insufficient privileges: not in try users |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@bors try @rust-timer queue |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Optimize DST `size_of_val`, `align_of_val`, and field projection codegen
|
The use of |
Thanks @saethlin! |
This comment has been minimized.
This comment has been minimized.
Ah okay cool thanks for the heads up. Any suggestions for how to do it better? |
Previously, `rustc_codegen_ssa::size_of_val::size_and_align_of_dst` (and its counterpart in `rustc_codegen_cranelift`) used a naive recursive algorithm that simultaneously computed both size and alignment at every level of struct/tuple nesting. This caused several codegen inefficiencies: 1. Calling `size_of_val` or `align_of_val` computed both values and discarded half the work. In `Layout::for_value_raw` (used when dropping `Box<T>`, `Rc<T>`, and `Arc<T>`), calling `size_of_val_raw` followed by `align_of_val_raw` emitted the entire vtable load and alignment-rounding sequence twice (rust-lang#152773). 2. Nested slice-tailed DSTs emitted `O(n)` sequential alignment-rounding steps (`add`, `sub`, `neg`, `and`) instead of folding the layout hierarchy at compile time. 3. Zero-offset `align(1)` wrappers around `dyn Trait` (such as `ManuallyDrop<dyn Trait>`, `UnsafeCell<dyn Trait>`, and `Cell<dyn Trait>`) loaded `vtable.align` and rounded `vtable.size` up to `vtable.align` even though `vtable.size` is already guaranteed to be a multiple of `vtable.align`. 4. `PlaceRef::project_field` called `size_and_align_of_dst` and emitted dynamic alignment-rounding IR whenever projecting to any unsized field at a non-zero offset, even for slice-tailed DST fields whose alignment (and therefore field offset) is a compile-time constant. Replace `size_and_align_of_dst` with decoupled `size_of_dst`, `align_of_dst`, and `dst_field_offset` routines in `rustc_codegen_ssa` and `rustc_codegen_cranelift`: - Decouple `align_of_dst` from `size_of_dst`: - For slice-tailed DSTs (`[T]`, `str`), `layout.align.abi` already incorporates the element alignment alongside all outer fields, `#[repr(align)]`, and `#[repr(packed)]`, so `align_of_dst` returns it as a compile-time constant. - For `dyn Trait`-tailed DSTs, by distributivity of `max` and `min` on power-of-two alignments, the alignment of an arbitrarily nested wrapper chain collapses to `max(layout.align.abi, min(vtable.align, min_pack))`. When `min_pack <= layout.align.abi`, this is statically `layout.align.abi` with no vtable load. - Distill slice-tailed DSTs into `O(1)` parameters in `size_of_dst`: - Walk the wrapper chain from the `[T]`/`str` tail outward at compile time, folding steps into `outer_offset + align_to(base + offset, align)` via the identities `align_to(c + align_to(x, a1), a2) == align_to(align_to(c, a1) + x, a2)` (when `a1 <= a2`), `align_to(c + align_to(x, a1), a2) == align_to(c, a2) + align_to(x, a1)` (when `a2 < a1`), and divisibility by `elem_size`. - Emit at most one `mul nuw nsw` (omitted when `elem_size == 1`), one pre-mask `add nuw nsw` (omitted when the addend is 0), one bitwise `and` (omitted when `elem_size % align == 0`), and (only when an outer `#[repr(packed)]` struct places a dynamically-padded inner DST at an unaligned offset) one post-mask `add nuw nsw`. When `elem_size == 0`, fold the entire DST size to a compile-time constant. - Because DST sizes fit in `isize` and `isize::MAX + 1` is a multiple of every valid alignment, `base + offset + (align - 1) <= isize::MAX`, allowing the use of `unchecked_suadd` (`add nuw nsw`). For DSTs with a non-zero header, this lets LLVM prove `size > 0` and eliminate the dead `size == 0` branch before `__rust_dealloc` in `Box::drop`, `Rc::drop`, and `Arc::drop`. - Collapse `dyn Trait` wrapper chains in `size_of_dst`: - Stream steps from innermost to outermost, skipping inner `offset == 0 && align == 1` wrappers (`ManuallyDrop<dyn Trait>`, `UnsafeCell<dyn Trait>`, etc.) so `size_of_val` emits a single `vtable.size` load (a 2.18x speedup at `-O3`). - Merge adjacent wrappers when `prev.align <= cur.align` and either `cur.offset == 0 && cur.pack == prev.pack` or both steps have statically capped alignment (`pack <= align`). - When the innermost step has `offset == 1 && align == 1` (e.g. `struct S(u8, dyn Trait)`), `full_align` divides `vtable.align` and thus divides `vtable.size`, so `align_to(1 + vtable.size, full_align)` simplifies to `vtable.size + full_align` without a bitmask. - Unify unsized field offset calculation into `dst_field_offset`: - Return `None` (using the static `layout.fields.offset(i)`) for all slice-tailed DST fields and for `dyn Trait` fields whose static offset is already a multiple of `min_pack`. - When dynamic rounding is needed for a `dyn Trait` field, exploit the fact that `offset` is already aligned to `field.align.abi`: when `offset == field.align.abi`, `align_to(offset, effective_align)` is `effective_align` itself; when `offset > field.align.abi`, all bits below `field.align.abi` in `-offset` are already 0, so clamping `vtable.align` up to `field.align.abi` before computing `offset + (-offset & (align - 1))` is unnecessary. - Extend MIR `InstSimplify` (`simplify_size_or_align_of_val`): - Fold `align_of_val` on `str` and single-field unaligned/unpacked wrapper structs and 1-tuples around `[T]` or `str` (such as `Path`, `OsStr`, and `CStr`) to `align_of::<Elem>()` in MIR (a 1.50x speedup on `Layout::for_value::<Path>` at `-O3`). Fixes rust-lang#152773 Validation: ./x test --stage 1 tests/codegen-llvm tests/mir-opt tests/ui/packed tests/ui/extern tests/ui/dst tests/ui/layout Agent-Authored-By: AI agent acting on Josh Liebow-Feeser's behalf
4aa98df to
839bc23
Compare
|
The job Click to see the possible cause of the failure (guessed by this bot) |
This comment has been minimized.
This comment has been minimized.
|
Finished benchmarking commit (dfc3d41): comparison URL. Overall result: ✅ improvements - no action neededBenchmarking means the PR may be perf-sensitive. Consider adding rollup=never if this change is not fit for rolling up. @rustbot label: -S-waiting-on-perf -perf-regression Instruction countOur most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
Max RSS (memory usage)Results (secondary -4.2%)A less reliable metric. May be of interest, but not used to determine the overall result above.
CyclesResults (primary 2.5%, secondary 5.5%)A less reliable metric. May be of interest, but not used to determine the overall result above.
Binary sizeResults (primary -0.7%)A less reliable metric. May be of interest, but not used to determine the overall result above.
Bootstrap: 489.75s -> 488.876s (-0.18%) |
Previously,
rustc_codegen_ssa::size_of_val::size_and_align_of_dst(and its counterpart inrustc_codegen_cranelift) used a naive recursive algorithm that simultaneously computed both size and alignment at every level of struct/tuple nesting. This caused several codegen inefficiencies:size_of_valoralign_of_valcomputed both values and discarded half the work. InLayout::for_value_raw(used when droppingBox<T>,Rc<T>, andArc<T>), callingsize_of_val_rawfollowed byalign_of_val_rawemitted the entire vtable load and alignment-rounding sequence twice (Dropping boxes with unsized tails generates a bunch of duplicate IR #152773).O(n)sequential alignment-rounding steps (add,sub,neg,and) instead of folding the layout hierarchy at compile time.align(1)wrappers arounddyn Trait(such asManuallyDrop<dyn Trait>,UnsafeCell<dyn Trait>, andCell<dyn Trait>) loadedvtable.alignand roundedvtable.sizeup tovtable.aligneven thoughvtable.sizeis already guaranteed to be a multiple ofvtable.align.PlaceRef::project_fieldcalledsize_and_align_of_dstand emitted dynamic alignment-rounding IR whenever projecting to any unsized field at a non-zero offset, even for slice-tailed DST fields whose alignment (and therefore field offset) is a compile-time constant.Replace
size_and_align_of_dstwith decoupledsize_of_dst,align_of_dst, anddst_field_offsetroutines inrustc_codegen_ssaandrustc_codegen_cranelift:Decouple
align_of_dstfromsize_of_dst:[T],str),layout.align.abialready incorporates the element alignment alongside all outer fields,#[repr(align)], and#[repr(packed)], soalign_of_dstreturns it as a compile-time constant.dyn Trait-tailed DSTs, by distributivity ofmaxandminon power-of-two alignments, the alignment of an arbitrarily nested wrapper chain collapses tomax(layout.align.abi, min(vtable.align, min_pack)). Whenmin_pack <= layout.align.abi, this is staticallylayout.align.abiwith no vtable load.Distill slice-tailed DSTs into
O(1)parameters insize_of_dst:[T]/strtail outward at compile time, folding steps intoouter_offset + align_to(base + offset, align)via the identitiesalign_to(c + align_to(x, a1), a2) == align_to(align_to(c, a1) + x, a2)(whena1 <= a2),align_to(c + align_to(x, a1), a2) == align_to(c, a2) + align_to(x, a1)(whena2 < a1), and divisibility byelem_size.mul nuw nsw(omitted whenelem_size == 1), one pre-maskadd nuw nsw(omitted when the addend is 0), one bitwiseand(omitted whenelem_size % align == 0), and (only when an outer#[repr(packed)]struct places a dynamically-padded inner DST at an unaligned offset) one post-maskadd nuw nsw. Whenelem_size == 0, fold the entire DST size to a compile-time constant.isizeandisize::MAX + 1is a multiple of every valid alignment,base + offset + (align - 1) <= isize::MAX, allowing the use ofunchecked_suadd(add nuw nsw). For DSTs with a non-zero header, this lets LLVM provesize > 0and eliminate the deadsize == 0branch before__rust_deallocinBox::drop,Rc::drop, andArc::drop.Collapse
dyn Traitwrapper chains insize_of_dst:offset == 0 && align == 1wrappers (ManuallyDrop<dyn Trait>,UnsafeCell<dyn Trait>, etc.) sosize_of_valemits a singlevtable.sizeload (a 2.18x speedup at-O3).prev.align <= cur.alignand eithercur.offset == 0 && cur.pack == prev.packor both steps have statically capped alignment (pack <= align).offset == 1 && align == 1(e.g.struct S(u8, dyn Trait)),full_aligndividesvtable.alignand thus dividesvtable.size, soalign_to(1 + vtable.size, full_align)simplifies tovtable.size + full_alignwithout a bitmask.Unify unsized field offset calculation into
dst_field_offset:None(using the staticlayout.fields.offset(i)) for all slice-tailed DST fields and fordyn Traitfields whose static offset is already a multiple ofmin_pack.dyn Traitfield, exploit the fact thatoffsetis already aligned tofield.align.abi: whenoffset == field.align.abi,align_to(offset, effective_align)iseffective_alignitself; whenoffset > field.align.abi, all bits belowfield.align.abiin-offsetare already 0, so clampingvtable.alignup tofield.align.abibefore computingoffset + (-offset & (align - 1))is unnecessary.Extend MIR
InstSimplify(simplify_size_or_align_of_val):align_of_valonstrand single-field unaligned/unpacked wrapper structs and 1-tuples around[T]orstr(such asPath,OsStr, andCStr) toalign_of::<Elem>()in MIR (a 1.50x speedup onLayout::for_value::<Path>at-O3).Fixes #152773
Validation: ./x test --stage 1 tests/codegen-llvm tests/mir-opt tests/ui/packed tests/ui/extern tests/ui/dst tests/ui/layout
Agent-Authored-By: AI agent acting on Josh Liebow-Feeser's behalf