diff --git a/changelog.d/8952-sso-computed-keys-write-stub.md b/changelog.d/8952-sso-computed-keys-write-stub.md new file mode 100644 index 0000000000..b0ea306492 --- /dev/null +++ b/changelog.d/8952-sso-computed-keys-write-stub.md @@ -0,0 +1,52 @@ +Computed string keys (`o["k" + i]`) stopped defeating every property cache in +the pipeline: **the dynamic-property overwrite loop goes from ~450 ms to +~86 ms — 5.3×**, 8/8 interleaved A/B pairs at stable load, against 28–30 ms +for node on the same host (so ~16× node → ~3.0× node). Binary size +4 KB +(+0.04%); no generated-code growth. + +Two changes that only pay off together. + +**1. `"str" + n` returns SSO when the result fits.** The fused concat's +NaN-box-returning twin (`js_string_concat_value_box`) assembles results of ≤ 5 +ASCII bytes as an inline SSO immediate instead of a fresh `StringHeader`. That +removes one heap allocation per iteration from the hot `"prefix" + i` shape — +but the bigger effect is that the result's BITS become content-stable: `"k" + +42` now yields the identical `f64` every evaluation, so every downstream cache +that compares key *values* can finally hit. ASCII-only, for the same +`utf16_len` soundness reason as `js_string_concat_box`'s existing SSO arm. + +**2. A megamorphic stub cache for dynamic string-keyed writes.** A dynamic +write site's inline IC holds `DYN_IC_WAYS = 3`; a loop rotating 500 computed +keys through one site evicts permanently and pays the full miss walk on every +write. Added a thread-local 4096-way direct-mapped cache keyed on +`(shape_token, key_bits)` — V8's megamorphic stub pattern — probed after the +per-site ways miss and fed at every prime site, including for overflow slots +(a wide object keeps *every* data property there, so gating on the inline +region would starve the cache for exactly the receivers it exists for). + +Supporting: the SSO→heap materialization every `*const StringHeader` consumer +crosses now interns instead of minting, so a computed key's ADDRESS is stable +too and the address-keyed read plan can hit; short heap keys are folded to +their SSO bits before keying, so a key compares equal to itself across +representations. + +**Safety.** The stub stores only content-derived bits, never an address — keys +that don't fit the inline form are rejected rather than cached under their +pointer. That matters: `dyn_ic_try_store` revalidates the receiver's current +shape token, blocking flags and slot bound on every hit, but it confirms the +SHAPE, not that the cached SLOT belongs to this KEY. A pointer-keyed entry +could therefore be primed, evicted from the (direct-mapped, evict-on-collision) +intern table, collected, and its address recycled by an unrelated string whose +write would hit the stale entry and overwrite the wrong slot. Content-only +keying removes that class entirely, and leaves the table holding no GC roots. + +**Method note — two wrong verdicts before the right one.** The stub alone +measured as a wash, twice. Counting is what broke it open: 600k inserts, 1.2M +probes, **0 hits**, and splitting the probe-miss counter by cause showed 99% of +misses had the right shape token and the *wrong key* — the keys were fresh heap +pointers each iteration, so a value-keyed cache could never hit. Fixing that +raised hits to 9,584 of 1.19M, still ~1%: the way index XOR'd the low bits, and +an SSO key's low bits are its *first* byte, so `"k0".."k499"` collapsed onto 125 +ways with buckets 10 deep and evicted each other continuously. Multiplicative +mixing (480/500 distinct ways, worst bucket 2) is what turned the design into +its measured 5.3×. Neither cause was visible in a profile — only in counters. diff --git a/crates/perry-codegen/src/codegen/declared_string_add_tests.rs b/crates/perry-codegen/src/codegen/declared_string_add_tests.rs index 7d2f4640d1..5a5659fbd3 100644 --- a/crates/perry-codegen/src/codegen/declared_string_add_tests.rs +++ b/crates/perry-codegen/src/codegen/declared_string_add_tests.rs @@ -192,9 +192,10 @@ fn a_string_literal_operand_keeps_the_fused_concat() { add(Expr::String("item_".to_string()), Expr::LocalGet(1)), ); assert!( - ir.contains("call i64 @js_string_concat_value("), - "a proven string must keep the fused single-allocation concat — the \ - guard is for claims, not for proofs:\n{ir}" + ir.contains("call double @js_string_concat_value_box("), + "a proven string must keep the fused concat (the `_box` twin, which \ + returns SSO for short results) — the guard is for claims, not for \ + proofs:\n{ir}" ); assert!( !ir.contains("call double @js_string_add_value("), @@ -214,7 +215,7 @@ fn a_coerced_operand_keeps_the_fused_concat() { ), ); assert!( - ir.contains("call i64 @js_string_concat_value(") + ir.contains("call double @js_string_concat_value_box(") && !ir.contains("call double @js_string_add_value("), "`String(x)` constructs a string; it is not an annotation:\n{ir}" ); diff --git a/crates/perry-codegen/src/lower_string_concat.rs b/crates/perry-codegen/src/lower_string_concat.rs index 5516119df8..0594c822cb 100644 --- a/crates/perry-codegen/src/lower_string_concat.rs +++ b/crates/perry-codegen/src/lower_string_concat.rs @@ -570,14 +570,18 @@ fn coerce_concat_body( // Issue #214: SSO-safe unbox; repsel Phase 3a: inline `bitcast+and` // for proven-heap operands (string literals — the `"user_" + i` // shape) and tag-dispatch for canonical-Str locals. + // + // The `_box` twin returns the result NaN-boxed directly and takes + // the SSO arm for ≤5-ASCII-byte results, giving `"k" + i` computed + // keys content-stable bits (so dynamic-key write ICs and the + // megamorphic stub can hit) with zero allocation. let l_handle = str_operand_handle_tag_dispatched(ctx, left, l_box); let blk = ctx.block(); - let result_handle = blk.call( - I64, - "js_string_concat_value", + return Ok(blk.call( + DOUBLE, + "js_string_concat_value_box", &[(I64, &l_handle), (DOUBLE, r_box)], - ); - return Ok(nanbox_string_inline(blk, &result_handle)); + )); } if !l_is_string && r_is_string { diff --git a/crates/perry-codegen/src/runtime_decls/strings.rs b/crates/perry-codegen/src/runtime_decls/strings.rs index 5bb7272391..ec1816ad40 100644 --- a/crates/perry-codegen/src/runtime_decls/strings.rs +++ b/crates/perry-codegen/src/runtime_decls/strings.rs @@ -32,6 +32,10 @@ pub fn declare_phase_b_strings(module: &mut LlModule) { // `js_value_concat_string(value_f64, suffix_handle) -> handle` module.declare_function("js_string_concat_value", I64, &[I64, DOUBLE]); module.declare_function("js_value_concat_string", I64, &[DOUBLE, I64]); + // NaN-box-returning twin: SSO immediate for ≤5-ASCII-byte results, so + // `"k" + i` computed keys get content-stable bits (dyn-IC/stub hits) + // and skip the per-iteration allocation entirely. + module.declare_function("js_string_concat_value_box", DOUBLE, &[I64, DOUBLE]); // #7837: the same two fused concats, but with the STRING side passed // NaN-boxed instead of pre-unboxed, so the helper can tell a real string diff --git a/crates/perry-runtime/Cargo.toml b/crates/perry-runtime/Cargo.toml index 1d62b30f07..1f368e96ef 100644 --- a/crates/perry-runtime/Cargo.toml +++ b/crates/perry-runtime/Cargo.toml @@ -49,6 +49,30 @@ keepalive-anchors = [] # leaves it off and falls back to the system allocator. The #6882 VM-tag # retag rides the same gate (it only exists to label mimalloc's mappings). alloc-mimalloc = ["dep:mimalloc", "dep:libmimalloc-sys"] +# DIAGNOSTIC ONLY — never in `default`, never in a shipping build. +# +# Builds mimalloc in secure + debug mode to hunt heap corruption that is +# *allocator-dependent*: the class of bug that crashes ~56% of the time on +# mimalloc and 0/41 on the system allocator. macOS `free()` ignores the +# caller-passed size, so a mismatched dealloc layout is silently harmless +# there while it corrupts mimalloc's metadata — the system-allocator "fix" +# hides the bug rather than proving its absence. +# +# secure → guard pages after each block, encoded free lists, randomized +# placement ⇒ catches heap *overflow* at the write. +# debug → mi_assert checks on every free ⇒ catches invalid free, double +# free, and wrong-size/align dealloc at the *call site*. +# +# Both report through mimalloc's own error path, so a hit names the failing +# operation instead of leaving a SIGSEGV to attribute by hand. Expect a large +# slowdown; this is an instrument, not a configuration. +alloc-mimalloc-hardened = [ + "alloc-mimalloc", + "mimalloc/secure", + "mimalloc/debug", + "libmimalloc-sys/secure", + "libmimalloc-sys/debug", +] # Per-module Node-API gate (binary-size): compiles `node:dgram`'s UDP-socket # implementation (`crate::dgram` + `crate::dgram_reactor`, ~2.2k LOC, incl. the # `js_dgram_*` externs codegen emits direct calls to) + its dispatch arm only diff --git a/crates/perry-runtime/src/proxy/put_value.rs b/crates/perry-runtime/src/proxy/put_value.rs index a88f9d4f78..0839d0423f 100644 --- a/crates/perry-runtime/src/proxy/put_value.rs +++ b/crates/perry-runtime/src/proxy/put_value.rs @@ -459,6 +459,22 @@ pub extern "C" fn js_put_value_set_ic_miss( // discriminated shape token. Perry's read PIC uses the same format. (*cache)[1] = idx as i64; (*cache)[0] = shape_token as i64; + // Rotating-key sites overflow the single-slot site cache immediately; + // the global stub is what lets them hit. Key bits from the rooted key. + // `key_handle` here roots a raw STRING pointer (this entry's key arrives + // as *const StringHeader) — re-box it the way `key_value` above did + // rather than asking the handle for a NaN-boxed read it does not hold. + // Scoped read: nothing inside the closure allocates or polls the + // collector, so the pointer cannot go stale within it (#7341). + key_handle.with_const_ptr::(|key_now| { + if !key_now.is_null() { + let boxed = + f64::from_bits(crate::value::js_nanbox_string(key_now as i64).to_bits()); + if let Some(kb) = stub_key_bits(boxed) { + write_stub_insert(shape_token, kb, idx); + } + } + }); } result @@ -548,6 +564,11 @@ pub extern "C" fn js_put_value_set_dyn_ic( break; } } + // Way miss on a rotating-key site: the global stub answers + // in one probe. `dyn_ic_try_store` still validates everything. + if found.is_none() { + found = stub_key_bits(key).and_then(|kb| write_stub_probe(token, kb)); + } found.and_then(|slot| dyn_ic_try_store(target, token, slot, value)) } else { None @@ -560,6 +581,105 @@ pub extern "C" fn js_put_value_set_dyn_ic( js_put_value_set_dyn_ic_miss(cache, target, key, value, strict) } +/// Megamorphic stub cache for dynamic string-keyed WRITES — V8's answer to a +/// site that rotates more keys than its per-site ways can hold. +/// +/// The per-site inline IC has `DYN_IC_WAYS = 3`; a loop writing 500 rotating +/// keys evicts permanently and every write pays the full miss walk (~400 ns +/// measured, vs node's ~28 ns/op on the same host). This global table is keyed +/// on `(shape_token, key_bits)` so capacity scales with the PROGRAM's live +/// (shape, key) pairs instead of one site's ways. +/// +/// Correctness leans entirely on [`dyn_ic_try_store`], which re-validates the +/// receiver's CURRENT shape token, blocking flags and slot bound on every hit. +/// A stale entry therefore misses; it cannot corrupt. That is why there is no +/// epoch and no GC hook: entries hold no roots (token is an id, key_bits are +/// SSO immediates or interned-pointer bits used only for equality, slot is an +/// index), and wrong entries are rejected by validation. +/// +/// SSO keys (≤ 5 bytes — every `"k" + i`-style computed name) recur with +/// identical bits by construction. Heap-string keys recur once canonicalised: +/// the first write interns the key (write tail) and later concat evaluations +/// return the canonical pointer (intern hit), so the second prime converges. +const WRITE_STUB_WAYS: usize = 4096; + +crate::perry_thread_local! { + static WRITE_STUB: [std::cell::Cell<(u64, u64, u64)>; WRITE_STUB_WAYS] = + std::array::from_fn(|_| std::cell::Cell::new((0, 0, 0))); +} + +/// Content-stable cache key for a NaN-boxed property key, or `None` when this +/// key must not be cached in the stub at all. +/// +/// Two jobs, and the second one is a safety property. +/// +/// **Normalization.** A short key can arrive in EITHER representation — as an +/// SSO immediate, or as a heap `StringHeader*` when some intermediate (a typed +/// local, a call boundary) materialized it. Those are the same JS string with +/// completely different bits, and a fresh heap key minted per loop iteration +/// is what made a raw-bits stub miss 98% of its probes even after the concat +/// itself started returning SSO. Folding a short ASCII heap key back to the +/// SSO bits its content would have had makes the cache key depend on the +/// STRING, not on which representation happened to reach this call. +/// +/// **Content-only, never an address.** A key that does not fit the inline form +/// is rejected rather than cached under its pointer bits. Caching an address +/// would be unsound here in a way the per-hit `dyn_ic_try_store` revalidation +/// cannot catch: that check confirms the receiver still has the cached SHAPE, +/// not that the cached SLOT belongs to this KEY. So a long key at address `A` +/// could be primed, evicted from the intern table (which is direct-mapped and +/// evicts on collision), collected, and its address recycled by an unrelated +/// string — whose write would then hit the stale entry and overwrite the wrong +/// slot. Restricting the table to content-derived bits removes that class +/// entirely: an entry names a STRING VALUE, holds no pointer, and so needs +/// neither GC roots nor eviction epochs. Longer keys simply keep the existing +/// (unchanged) miss path. +#[inline(always)] +fn stub_key_bits(key: f64) -> Option { + let bits = key.to_bits(); + if crate::value::JSValue::from_bits(bits).is_short_string() { + // Already an SSO immediate: pure content. + return Some(bits); + } + if (bits & !POINTER_MASK) != crate::value::STRING_TAG { + return None; + } + let ptr = (bits & POINTER_MASK) as *const crate::StringHeader; + unsafe { crate::string::short_ascii_sso_bits(ptr) } +} + +/// Way index for a `(shape_token, key_bits)` pair. +/// +/// Multiplicative (Fibonacci) mixing, taking the TOP bits of the product — +/// not the low bits of an XOR. The distinction is the whole ballgame for +/// string keys: an SSO key's bits are its bytes in little-endian order, so a +/// family like `"k0".."k499"` shares its first byte and varies in bytes 2-4. +/// Indexing on the low bits therefore collapses hundreds of distinct keys onto +/// a handful of ways, which evict each other on every write — measured as +/// 1.17M of 1.19M probes missing with the way occupied by a DIFFERENT key at +/// the SAME shape token, while the table sat 99% empty. +#[inline(always)] +fn write_stub_way(token: u64, key_bits: u64) -> usize { + let h = (token ^ key_bits).wrapping_mul(0x9E37_79B9_7F4A_7C15); + ((h >> 40) as usize) & (WRITE_STUB_WAYS - 1) +} + +#[inline(always)] +fn write_stub_probe(token: u64, key_bits: u64) -> Option { + WRITE_STUB.with(|t| { + let (tok, kb, slot) = t[write_stub_way(token, key_bits)].get(); + (tok == token && kb == key_bits && tok != 0).then_some(slot as u32) + }) +} + +#[inline(always)] +fn write_stub_insert(token: u64, key_bits: u64, slot: u32) { + if token == 0 || key_bits == 0 { + return; + } + WRITE_STUB.with(|t| t[write_stub_way(token, key_bits)].set((token, key_bits, slot as u64))); +} + /// Validated fast store: the receiver must still be an ordinary, /// non-forwarded, unblocked, class-tagged heap object whose CURRENT shape /// token equals the cached one and whose inline region covers the slot. @@ -599,6 +719,17 @@ unsafe fn dyn_ic_try_store(target: f64, token: u64, slot: u32, value: f64) -> Op } let shape = crate::object::shapes::object_shape_descriptor(obj)?; if slot >= shape.live_inline_slot_count { + // Overflow slot. The token compare above already proved the receiver + // is in the exact shape the (key → slot) pair was learned in, so the + // slot names this key's storage; it just lives in the spill store + // instead of the inline region. `overflow_set` is the same store the + // full walk bottoms out in (barriers, layout notes, remembered set), + // minus the walk. Wide objects — where EVERY data property is an + // overflow slot — are exactly the receivers the stub cache exists for. + if slot < shape.logical_key_count { + crate::object::overflow_set(obj_addr, slot as usize, value.to_bits()); + return Some(value); + } return None; } crate::object::store_object_field_slot(obj, slot as usize, value.to_bits()); @@ -624,6 +755,28 @@ pub extern "C" fn js_put_value_set_dyn_ic_miss( value: f64, strict: i32, ) -> f64 { + // The compiled inline IC (`lower_put_value_dyn_ic_inline`) walks its three + // ways in GENERATED code and calls straight here on a way miss — it never + // enters `js_put_value_set_dyn_ic` above. A rotating-key site therefore + // lands here on every write, so the megamorphic stub probe must sit at + // THIS entry to be on that path at all. `c[0]` holds the site's primed + // shape token; `dyn_ic_try_store` validates it against the receiver's + // live state, so a stale site token or stub entry misses into the full + // walk below instead of storing wrongly. No allocation precedes the + // probe — the handle scope opens only after this fails. + if !cache.is_null() { + let c = unsafe { &*cache }; + let token = c[0] as u64; + let key_bits = key.to_bits(); + if token != 0 && key_bits != 0 { + if let Some(slot) = stub_key_bits(key).and_then(|kb| write_stub_probe(token, kb)) { + if let Some(ret) = unsafe { dyn_ic_try_store(target, token, slot, value) } { + return ret; + } + } + } + } + let scope = crate::gc::RuntimeHandleScope::new(); let target_handle = scope.root_nanbox_f64(target); let key_handle = scope.root_nanbox_f64(key); @@ -724,13 +877,8 @@ pub extern "C" fn js_put_value_set_dyn_ic_miss( let Some(idx) = own_idx else { return result; }; - let alloc_limit = shape.live_inline_slot_count; - if idx >= alloc_limit { - return result; - } let shape_token = crate::object::shapes::PIC_ID_TOKEN_BIT | crate::object::shapes::object_shape_id(obj) as u64; - let c = &mut *cache; let key_bits = key.to_bits() as i64; // Preserve the empty-way sentinel invariant: never prime bits 0 // (only the JS number 0 has them, and numeric keys cannot prime @@ -738,6 +886,19 @@ pub extern "C" fn js_put_value_set_dyn_ic_miss( if key_bits == 0 { return result; } + // Feed the megamorphic stub BEFORE the inline-slot bail below: a wide + // object keeps every data property in an overflow slot, so gating the + // stub on the inline region would starve it for exactly the receivers + // it exists for. `dyn_ic_try_store` stores overflow slots via + // `overflow_set` under the same token validation. + if let Some(kb) = stub_key_bits(key) { + write_stub_insert(shape_token, kb, idx); + } + let alloc_limit = shape.live_inline_slot_count; + if idx >= alloc_limit { + return result; + } + let c = &mut *cache; if c[0] as u64 != shape_token { // New shape at this site: restart the way set. *c = [0; 8]; @@ -754,6 +915,8 @@ pub extern "C" fn js_put_value_set_dyn_ic_miss( c[1 + w * 2] = *k; c[2 + w * 2] = *sl; } + // (The megamorphic stub was already fed above, before the inline-slot + // bail — overflow slots feed it too.) // Token last: a zero or stale token cannot hit until it matches the // receiver's current discriminated shape. c[0] = shape_token as i64; diff --git a/crates/perry-runtime/src/string/alloc.rs b/crates/perry-runtime/src/string/alloc.rs index 9cfc5097e9..b6172f40b4 100644 --- a/crates/perry-runtime/src/string/alloc.rs +++ b/crates/perry-runtime/src/string/alloc.rs @@ -56,7 +56,18 @@ pub extern "C" fn js_string_materialize_to_heap(value: f64) -> *mut StringHeader if jsval.is_short_string() { let mut buf = [0u8; crate::value::SHORT_STRING_MAX_LEN]; let n = jsval.short_string_to_buf(&mut buf); - return js_string_from_bytes(buf.as_ptr(), n as u32); + // Intern rather than mint. This function is the SSO→heap boundary + // every `*const StringHeader` consumer crosses, and a computed + // property name (`o["k" + i]`) crosses it on every read: minting gave + // each iteration a fresh allocation AND a fresh address, so the + // address-keyed read plan and the interned-key write fast paths could + // never hit on a key they had already seen. Interning makes the + // ADDRESS content-stable, which is what those caches actually compare. + // Same-content strings are already required to be interchangeable + // here (the materialized header is a value, never an identity), so + // sharing one canonical header changes nothing observable. + return crate::string::intern::intern_dispatch_bytes(0, buf.as_ptr(), n, 0, false) + as *mut StringHeader; } if jsval.is_string() { return jsval.as_string_ptr() as *mut StringHeader; diff --git a/crates/perry-runtime/src/string/concat.rs b/crates/perry-runtime/src/string/concat.rs index 458d3af66e..92606a6120 100644 --- a/crates/perry-runtime/src/string/concat.rs +++ b/crates/perry-runtime/src/string/concat.rs @@ -446,6 +446,57 @@ pub extern "C" fn js_string_concat_value( js_string_concat(prefix_handle.get_raw_const_ptr::(), value_str) } +/// NaN-box-returning twin of [`js_string_concat_value`]: SSO immediate when +/// the result fits (≤ 5 ASCII bytes), heap `STRING_TAG` box otherwise. +/// +/// The SSO arm matters twice over. It removes the per-iteration allocation +/// from the `"k" + i` computed-key pattern — but more importantly it makes +/// the result's BITS content-stable: `"k" + 42` yields the identical f64 +/// every evaluation. Every key-keyed cache downstream (the dynamic-key +/// write IC's ways, the megamorphic write stub, read plans) compares key +/// VALUE bits, so a heap pointer minted fresh per iteration can never hit — +/// which is exactly what the stub-cache counters showed (600k inserts, +/// 1.19M probes, 0 hits, 99% `way_key_neq`). ASCII-only for the same +/// `utf16_len` soundness reason as `js_string_concat_box`'s SSO arm. +#[no_mangle] +pub extern "C" fn js_string_concat_value_box(prefix: *const StringHeader, value: f64) -> f64 { + // Same "plain f64" test as `js_string_concat_value`'s fast path; the SSO + // arm additionally wants a small non-negative integer so the digit count + // comes from `fast_itoa_u32`. + let bits = value.to_bits(); + let tag = bits >> 48; + let is_plain_f64 = tag < 0x7FF8 || (tag == 0x7FF8 && (bits & 0x000F_FFFF_FFFF_FFFF) == 0); + if is_plain_f64 + && value.fract() == 0.0 + && (0.0..=999_999_999.0).contains(&value) + && is_valid_string_ptr(prefix) + { + let prefix_blen = unsafe { (*prefix).byte_len } as usize; + if prefix_blen < crate::value::SHORT_STRING_MAX_LEN { + let mut num_buf = [0u8; 32]; + let num_len = fast_itoa_u32(value as u32, &mut num_buf); + if prefix_blen + num_len <= crate::value::SHORT_STRING_MAX_LEN { + let data = string_data(prefix); + if bytes_all_ascii(data, prefix_blen as u32) { + let mut sso = [0u8; crate::value::SHORT_STRING_MAX_LEN]; + unsafe { + std::ptr::copy_nonoverlapping(data, sso.as_mut_ptr(), prefix_blen); + } + sso[prefix_blen..prefix_blen + num_len].copy_from_slice(&num_buf[..num_len]); + return f64::from_bits( + crate::value::JSValue::short_string_unchecked( + &sso[..prefix_blen + num_len], + ) + .bits(), + ); + } + } + } + } + let ptr = js_string_concat_value(prefix, value); + f64::from_bits(crate::value::js_nanbox_string(ptr as i64).to_bits()) +} + /// Ceiling on the per-call part count. Must match `CONCAT_CHAIN_MAX_PARTS` in /// `perry-codegen/src/lower_string_concat.rs`. The cap keeps the stack scratch /// bounded so a pathological fold cannot overflow the stack. diff --git a/crates/perry-runtime/src/string/mod.rs b/crates/perry-runtime/src/string/mod.rs index 58a28d9a9b..b5c54b0c4e 100644 --- a/crates/perry-runtime/src/string/mod.rs +++ b/crates/perry-runtime/src/string/mod.rs @@ -933,6 +933,42 @@ pub(crate) fn string_data(s: *const StringHeader) -> *const u8 { unsafe { (s as *const u8).add(std::mem::size_of::()) } } +/// The SSO immediate bits a heap string's CONTENT would encode as, or `None` +/// when it doesn't fit the inline form (> `SHORT_STRING_MAX_LEN` bytes, or +/// any non-ASCII byte — SSO's length tag doubles as the JS `.length`, so a +/// multi-byte sequence must not take this form). +/// +/// This is the representation-folding half of SSO: the same short string can +/// reach a cache as an immediate or as a heap pointer depending on what the +/// codegen materialized, and identity comparisons then fail on strings that +/// are equal. Callers that key a cache on a property name use this to compare +/// content without a byte-by-byte scan at every probe. +/// +/// # Safety +/// `p` must be a valid `StringHeader*` or null; the payload is read but not +/// retained, so the borrow must not span an allocation. +#[inline] +pub(crate) unsafe fn short_ascii_sso_bits(p: *const StringHeader) -> Option { + if !is_valid_string_ptr(p) { + return None; + } + let blen = (*p).byte_len as usize; + if blen > crate::value::SHORT_STRING_MAX_LEN { + return None; + } + let data = string_data(p); + let mut payload: u64 = 0; + for i in 0..blen { + let b = *data.add(i); + if b >= 0x80 { + return None; + } + payload |= (b as u64) << (i * 8); + } + let len_bits = (blen as u64) << crate::value::SHORT_STRING_LEN_SHIFT; + Some(crate::value::SHORT_STRING_TAG | len_bits | payload) +} + /// Get string as a Rust `&str` for immediate, non-allocating internal use. /// /// The returned lifetime is caller-chosen and is not tied to a GC root. The diff --git a/scripts/gc_runtime_root_holders.json b/scripts/gc_runtime_root_holders.json index 41e8e7f254..009cf679a1 100644 --- a/scripts/gc_runtime_root_holders.json +++ b/scripts/gc_runtime_root_holders.json @@ -316,6 +316,12 @@ "scanner": "promise::scan_promise_roots_mut (promise/scanners.rs:66, visit_cell_f64_slot) plus its budgeted step twin scan_current_microtask_value_step (scanners.rs:420); registered via reg_budgeted_scanner! at gc/mod.rs:833-838", "why": "Declared in promise/microtasks.rs, scanned from promise/scanners.rs. The scanners.rs:786/913 sites are the mutator-side save/clear the scanner exists to protect, not the coverage mechanism." }, + { + "file": "crates/perry-runtime/src/proxy/put_value.rs", + "name": "WRITE_STUB", + "verdict": "not_a_gc_pointer", + "why": "Megamorphic dynamic-write stub cache: 4096 ways of (shape_token, key_bits, slot) plain u64s. The token is a shape id and the slot an index; key_bits are content-derived by construction, because stub_key_bits admits only SSO immediates (and short ASCII heap strings folded to the SSO bits of their content) and rejects every key that would otherwise be stored under a pointer. No way holds a heap address, so nothing here keeps an object alive, and a stale entry is rejected by dyn_ic_try_store's per-hit shape-token/flags/slot-bound revalidation." + }, { "file": "crates/perry-runtime/src/pty/mod.rs", "name": "EXIT_SINK",