perf: take the keyed table read lock on hits and duplicate waiters

[aisk]

Summary

This implements step 2 of #200, using the contention baseline from #202. The keyed table shared by OnceMap and singleflight::Group moves from a single Mutex to an internal RwLock. Lookups that do not mutate the table — OnceMap::get, compute on an already-initialized entry, and a singleflight call joining an in-flight leader as a duplicate waiter — now take the read lock and run concurrently. A miss falls back to the write lock, where get_or_insert double-checks before inserting. Removals and the cleanup guards keep running under the write lock.

The baseline showed that read paths degrade worst under contention (about 37–49x at 8 threads for initialized hits) while write-heavy singleflight churn degrades least, so a read fast path targets exactly the measured pain. Unlike sharding, it also helps the same-key hotspot, and it leaves the HashTable and hasher ownership untouched. Sharding was evaluated and deferred: it cannot help same-key traffic, its benefit lands on the write-churn scenarios whose measurements are the noisiest, and it can still be layered on top of this change later if real workloads warrant it.

Correctness

The Arc::strong_count == 2 cleanup protocol is preserved. Entry Arcs are only cloned out of the table while holding the table lock, and the count check runs under the write lock, which excludes all readers, so the check can never miss a concurrent new owner. Callers that release their clone outside the lock only do so after the cell is initialized or after the entry left the table, which the existing !initialized() and Arc::ptr_eq guards already reject. No lock is held across an .await, and the read guard is always released before the write lock is requested.

New regression tests pin the races this design touches: a cancelled duplicate waiter must not remove an in-flight or initialized entry, and an abandoned first caller must not remove a replacement installed for the same key (via discard for OnceMap and forget for Group).

Breaking change: Sync now requires the hasher to be Sync

std::sync::Mutex<T> is Sync when T: Send, but RwLock<T> requires T: Sync, so OnceMap and Group are now Sync only when the hasher S is also Sync. This is inherent to any design where concurrent readers hash keys through a shared &S, not an artifact of the implementation. The default RandomState and common third-party hashers are all Sync; a hasher that is Send but not Sync still yields a Send (but no longer Sync) map. This is recorded in the CHANGELOG, and a new trait test pins the semantics with a Send + !Sync hasher. Flagging it explicitly since #200 lists expected auto traits as a compatibility constraint — if this trade-off is unacceptable, the read fast path cannot be kept.

Results

Test environment: AMD Ryzen 7 5700X (8 cores / 16 threads), Windows 11 Pro, rustc 1.96.0, divan 0.1.21, cargo bench --bench benchmarks -- contended. As in #202, t=8 is the meaningful contention point on this machine and t=32 medians are distorted by oversubscription.

Median time per operation at t=8, single Mutex baseline vs this change:

Benchmark	Baseline	This PR	Change
once_map::contended_get_hit_same_key	487 ns	63 ns	~7.7x faster
once_map::contended_get_hit_disjoint (1024)	915 ns	238 ns	~3.8x faster
once_map::contended_compute_hit_same_key	819 ns	63 ns	~13x faster
once_map::contended_compute_hit_disjoint (1024)	450 ns	228 ns	~2x faster
singleflight::contended_work_same_key	694 ns	543 ns	on par
miss/churn scenarios	0.6–2 µs	0.7–3.9 µs	within noise

The churn rows need the caveat spelled out: re-running the same build back to back swings their t=8 medians by up to 5x (the baseline itself measured 600 ns in one run and 1.78 µs in another for contended_work_disjoint_churn), and both builds land in the same band, so no regression is distinguishable from noise there. The stable cost is single-threaded: a miss now pays a read-lock probe before the write lock, which shows up as roughly +10% at t=1 on the miss/churn paths (for example contended_compute_miss_churn 113 ns → 127 ns). Initialized hits at t=1 are unchanged (get stays at 13 ns). The coalesced benches stay flat across thread counts, dominated by the leader's in-flight window as intended.

Full divan output (this PR)

Timer precision: 100 ns
benchmarks                            fastest       │ slowest       │ median        │ mean          │ samples │ iters
├─ once_map                                         │               │               │               │         │
│  ├─ contended_compute_coalesced                   │               │               │               │         │
│  │  ├─ t=1                          7.599 µs      │ 20.89 µs      │ 7.699 µs      │ 7.791 µs      │ 100     │ 100
│  │  ├─ t=2                          7.499 µs      │ 13.39 µs      │ 7.899 µs      │ 7.935 µs      │ 100     │ 100
│  │  ├─ t=8                          1.799 µs      │ 12.49 µs      │ 8.749 µs      │ 8.51 µs       │ 104     │ 104
│  │  ╰─ t=32                         899.7 ns      │ 11.59 µs      │ 7.999 µs      │ 6.306 µs      │ 128     │ 128
│  ├─ contended_compute_hit_disjoint                │               │               │               │         │
│  │  ├─ 64                                         │               │               │               │         │
│  │  │  ├─ t=1                       31.62 ns      │ 40.21 ns      │ 31.82 ns      │ 31.89 ns      │ 100     │ 51200
│  │  │  ├─ t=2                       76.35 ns      │ 95.88 ns      │ 88.07 ns      │ 87.41 ns      │ 100     │ 12800
│  │  │  ├─ t=8                       48.22 ns      │ 432.6 ns      │ 198.2 ns      │ 204.7 ns      │ 104     │ 6656
│  │  │  ╰─ t=32                      41.97 ns      │ 216.9 ns      │ 54.47 ns      │ 67.39 ns      │ 128     │ 8192
│  │  ╰─ 1024                                       │               │               │               │         │
│  │     ├─ t=1                       33.96 ns      │ 44.32 ns      │ 34.36 ns      │ 34.51 ns      │ 100     │ 51200
│  │     ├─ t=2                       80.25 ns      │ 106 ns        │ 93.54 ns      │ 93.37 ns      │ 100     │ 12800
│  │     ├─ t=8                       63.85 ns      │ 591.9 ns      │ 227.9 ns      │ 258.7 ns      │ 104     │ 6656
│  │     ╰─ t=32                      49.79 ns      │ 1.162 µs      │ 242.7 ns      │ 418.9 ns      │ 128     │ 8192
│  ├─ contended_compute_hit_same_key                │               │               │               │         │
│  │  ├─ t=1                          28.5 ns       │ 37.68 ns      │ 28.69 ns      │ 28.69 ns      │ 100     │ 51200
│  │  ├─ t=2                          69.32 ns      │ 252.9 ns      │ 102.1 ns      │ 102.8 ns      │ 100     │ 12800
│  │  ├─ t=8                          29.47 ns      │ 502.9 ns      │ 63.07 ns      │ 95.41 ns      │ 104     │ 6656
│  │  ╰─ t=32                         31.04 ns      │ 702.9 ns      │ 41.97 ns      │ 147.4 ns      │ 128     │ 4096
│  ├─ contended_compute_miss_churn                  │               │               │               │         │
│  │  ├─ 64                                         │               │               │               │         │
│  │  │  ├─ t=1                       125.5 ns      │ 129.4 ns      │ 127.1 ns      │ 126.7 ns      │ 100     │ 12800
│  │  │  ├─ t=2                       437.2 ns      │ 1.131 µs      │ 612.2 ns      │ 618.2 ns      │ 100     │ 1600
│  │  │  ├─ t=8                       199.7 ns      │ 7.899 µs      │ 799.7 ns      │ 1.415 µs      │ 104     │ 208
│  │  │  ╰─ t=32                      224.7 ns      │ 20.95 µs      │ 9.927 µs      │ 9.931 µs      │ 128     │ 2048
│  │  ╰─ 1024                                       │               │               │               │         │
│  │     ├─ t=1                       127.9 ns      │ 253.6 ns      │ 131 ns        │ 132.2 ns      │ 100     │ 12800
│  │     ├─ t=2                       406 ns        │ 749.7 ns      │ 602.9 ns      │ 602.9 ns      │ 100     │ 1600
│  │     ├─ t=8                       187.2 ns      │ 6.499 µs      │ 3.887 µs      │ 3.56 µs       │ 104     │ 832
│  │     ╰─ t=32                      174.7 ns      │ 28.39 µs      │ 349.7 ns      │ 4.316 µs      │ 128     │ 1024
│  ├─ contended_compute_mixed                       │               │               │               │         │
│  │  ├─ 64                                         │               │               │               │         │
│  │  │  ├─ t=1                       78.69 ns      │ 101.3 ns      │ 80.25 ns      │ 80.81 ns      │ 100     │ 12800
│  │  │  ├─ t=2                       312.2 ns      │ 427.9 ns      │ 387.2 ns      │ 388.1 ns      │ 100     │ 3200
│  │  │  ├─ t=8                       149.7 ns      │ 4.174 µs      │ 549.7 ns      │ 1.104 µs      │ 104     │ 416
│  │  │  ╰─ t=32                      112.2 ns      │ 9.699 µs      │ 868.5 ns      │ 1.922 µs      │ 128     │ 1024
│  │  ╰─ 1024                                       │               │               │               │         │
│  │     ├─ t=1                       79.47 ns      │ 82.6 ns       │ 81.04 ns      │ 80.77 ns      │ 100     │ 12800
│  │     ├─ t=2                       124.7 ns      │ 402.9 ns      │ 362.2 ns      │ 356 ns        │ 100     │ 3200
│  │     ├─ t=8                       149.7 ns      │ 4.199 µs      │ 1.799 µs      │ 1.871 µs      │ 104     │ 416
│  │     ╰─ t=32                      124.7 ns      │ 3.599 µs      │ 187.2 ns      │ 660.7 ns      │ 128     │ 1024
│  ├─ contended_get_hit_disjoint                    │               │               │               │         │
│  │  ├─ 64                                         │               │               │               │         │
│  │  │  ├─ t=1                       16.97 ns      │ 31.72 ns      │ 17.07 ns      │ 17.22 ns      │ 100     │ 102400
│  │  │  ├─ t=2                       78.69 ns      │ 99 ns         │ 87.29 ns      │ 87.75 ns      │ 100     │ 12800
│  │  │  ├─ t=8                       18.54 ns      │ 441.9 ns      │ 221.6 ns      │ 206.1 ns      │ 104     │ 6656
│  │  │  ╰─ t=32                      18.54 ns      │ 1.559 µs      │ 131 ns        │ 303.2 ns      │ 128     │ 8192
│  │  ╰─ 1024                                       │               │               │               │         │
│  │     ├─ t=1                       18.34 ns      │ 32.5 ns       │ 18.44 ns      │ 18.61 ns      │ 100     │ 102400
│  │     ├─ t=2                       28.69 ns      │ 102.9 ns      │ 87.29 ns      │ 86.43 ns      │ 100     │ 12800
│  │     ├─ t=8                       23.22 ns      │ 463.8 ns      │ 238 ns        │ 230 ns        │ 104     │ 6656
│  │     ╰─ t=32                      20.1 ns       │ 1.354 µs      │ 303.6 ns      │ 492.6 ns      │ 128     │ 8192
│  ╰─ contended_get_hit_same_key                    │               │               │               │         │
│     ├─ t=1                          12.97 ns      │ 13.16 ns      │ 13.07 ns      │ 13.04 ns      │ 100     │ 102400
│     ├─ t=2                          60.72 ns      │ 165.4 ns      │ 75.57 ns      │ 75.16 ns      │ 100     │ 25600
│     ├─ t=8                          13.85 ns      │ 257.6 ns      │ 63.07 ns      │ 84.39 ns      │ 104     │ 6656
│     ╰─ t=32                         68.54 ns      │ 1.394 µs      │ 686.5 ns      │ 627.6 ns      │ 128     │ 16384
╰─ singleflight                                     │               │               │               │         │
   ├─ contended_work_coalesced                      │               │               │               │         │
   │  ├─ t=1                          7.449 µs      │ 9.599 µs      │ 7.499 µs      │ 7.536 µs      │ 100     │ 200
   │  ├─ t=2                          7.499 µs      │ 9.049 µs      │ 7.649 µs      │ 7.67 µs       │ 100     │ 200
   │  ├─ t=8                          1.199 µs      │ 12.69 µs      │ 9.949 µs      │ 8.914 µs      │ 104     │ 104
   │  ╰─ t=32                         1.499 µs      │ 11.69 µs      │ 7.999 µs      │ 6.383 µs      │ 128     │ 128
   ├─ contended_work_disjoint_churn                 │               │               │               │         │
   │  ├─ t=1                          134.1 ns      │ 139.6 ns      │ 136.5 ns      │ 136.4 ns      │ 100     │ 12800
   │  ├─ t=2                          393.5 ns      │ 868.5 ns      │ 449.7 ns      │ 460 ns        │ 100     │ 1600
   │  ├─ t=8                          162.2 ns      │ 6.662 µs      │ 3.174 µs      │ 3.063 µs      │ 104     │ 832
   │  ╰─ t=32                         149.7 ns      │ 15.42 µs      │ 687.2 ns      │ 3.108 µs      │ 128     │ 512
   ╰─ contended_work_same_key                       │               │               │               │         │
      ├─ t=1                          109.1 ns      │ 110.7 ns      │ 109.9 ns      │ 109.9 ns      │ 100     │ 12800
      ├─ t=2                          293.5 ns      │ 449.7 ns      │ 393.5 ns      │ 390.5 ns      │ 100     │ 3200
      ├─ t=8                          124.7 ns      │ 3.237 µs      │ 543.5 ns      │ 805.3 ns      │ 104     │ 832
      ╰─ t=32                         174.7 ns      │ 10.79 µs      │ 2.012 µs      │ 2.704 µs      │ 128     │ 512

The baseline numbers are the ones reported in #202, measured on the same machine.

🤖 Generated with Claude Code

[aisk]

@tisonkun This PR trades an auto trait for the read fast path: OnceMap and singleflight::Group are now Sync only when the hasher S is also Sync (concurrent readers hash through a shared &S, so this requirement is inherent to any shared-reader design, and #200 lists expected auto traits as a compatibility constraint). RandomState and common hashers are unaffected. Could you make the call on whether this trade-off is acceptable? If not, I'd close this in favor of keeping the single mutex or exploring sharding instead.

[tisonkun]

I can reproduce a stable macOS/Apple Silicon regression under sustained read contention.

Environment: Apple M4 Max (14 cores), macOS/Darwin 25.3.0, rustc 1.96.0, divan 0.1.21. I ran head → base → head to control for ordering:

cargo +1.96.0 bench --workspace --all-features --bench '*' -- \
  contended_get_hit_same_key --threads 8 --sample-count 100 --sample-size 10000

• base c461305: 181.5–193.0 ns median across 3 runs
• PR dba8ecd: 1.629–1.763 µs median across 3 runs

Replacing the filter with contended_compute_hit_same_key gives 321.6–336.2 ns on base versus 2.468–3.332 µs on the PR. The default adaptive sample size was much noisier on this machine; a fixed large sample size makes the sustained-reader regression repeatable. Since get alone regresses, this is not only the extra Arc clone in compute; the std::sync::RwLock read path itself is platform-sensitive here.

This does not contradict the Windows result, but I think the optimization needs Linux/macOS coverage (or a different locking choice) before merging.

[aisk]

Fair enough. If I were to test this today, the only option would be WSL2, and that would be inaccurate anyway. Its virtualization can distort a lock contention microbenchmark by itself, and my machine is busy updating Honkai: Star Rail on top of that. Also this Claude Code session has already burned through a fair chunk of tokens. If no better approach lands here by tomorrow, I'll run the same methodology on a real Linux environment and report back.

[tisonkun]

Yep. A result of a Linux environment would be helpful since I typically deploy this code on a Linux server, although I'm using a MacBook Pro for development.

Honkai: Star Rail

Enjoy 4.5! I haven't checked it out today.

[PragmaTwice]

Adding a WSL2 data point. I benchmarked base c461305 and this PR at dba8ecd on:

• Intel x86_64 laptop CPU, 16 vCPUs
• WSL2 Linux
• rustc 1.96.0, divan 0.1.21

For each row, I ran base and PR three times in an interleaved order at t=8 and report the median of the three run medians. The hit benchmarks used fixed --sample-count 100 --sample-size 10000 settings.

Benchmark	Base	PR
once_map::contended_get_hit_same_key	688 ns	959 ns
once_map::contended_get_hit_disjoint (1024)	815 ns	1.071 µs
once_map::contended_compute_hit_same_key	1.091 µs	955 ns
once_map::contended_compute_hit_disjoint (1024)	1.346 µs	1.023 µs
singleflight::contended_work_same_key	3.034 µs	3.764 µs

Sharing these as an additional WSL2 result; the default adaptive runs were much noisier on this machine as well.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[aisk]

Bare-metal Linux results, a follow-up optimization they motivated (pushed as 6c55ca4), and a comparison with #225.

Environment: AMD Ryzen 7 5700X (8C/16T, same CPU as the Windows numbers above), NixOS 26.11, kernel 7.2.0, glibc 2.42, rustc 1.96.0. Same methodology as the M4 Max run: interleaved head → base, 3 rounds each, --threads 8 --sample-count 100 --sample-size 10000.

This PR (dba8ecd) vs base (c461305), divan medians, t=8

round	contended_get_hit_same_key base → PR	contended_compute_hit_same_key base → PR
1	478.3 ns → 475.4 ns	805.9 ns → 340.0 ns
2	555.5 ns → 296.2 ns	803.9 ns → 342.5 ns
3	568.1 ns → 302.1 ns	844.1 ns → 386.7 ns

~1.8× on get hits, ~2.3× on compute hits. Disjoint and churn scenarios were within noise.

Follow-up: 6c55ca4

On both Linux and macOS, compute hits are slower than get hits; the delta is one entry-Arc round trip made under contention just to read an already-initialized value. 6c55ca4 resolves initialized hits while the read lock is still held, so the hot path no longer touches the entry refcount.

round	contended_compute_hit_same_key dba8ecd → 6c55ca4	contended_work_same_key dba8ecd → 6c55ca4
1	349.0 ns → 264.1 ns	2.266 µs → 2.091 µs
2	360.7 ns → 293.5 ns	2.284 µs → 2.099 µs
3	418.4 ns → 266.3 ns	2.329 µs → 2.244 µs

compute hits now match get hits (~270 ns, ≈3× over base); work_same_key improves ~7%; everything else is unchanged within variance. On macOS this should narrow the compute regression toward the get one, but doesn't address the lock cost itself.

Platform split

std::sync::RwLock is SRWLOCK on Windows (cheap shared acquisition → 7.7×), futex-based on Linux (→ 1.8–3×), and queue-based on macOS, where sustained reader-reader contention is expensive on Apple Silicon (→ the ~9× regression above). Any std-only shared-reader design inherits this.

vs #225 (its t=8 numbers, relative to baseline)

scenario	this PR	#225 (Mutex shards)	#225 (parking_lot::RwLock shards)
compute_hit_same_key	3.3×	0.87×	3.1×
get_hit_same_key	2.6×	0.89×	1.3×
compute_hit_disjoint (64)	3.1×	10×	10.8×
get_hit_disjoint (64)	2.0×	10.5×	7.5×
compute_miss_churn (64)	0.71×	17×	16×
compute_mixed (64)	0.78×	16×	15.6×
work_disjoint_churn	0.74×	16×	16×
work_same_key	0.87×	1.09×	1.05×

• This PR wins only the same-key hot paths and regresses the write-heavy scenarios (an exclusive RwLock acquisition is pricier than a Mutex under contention; worse at t=32).
• perf: sharded OnceTable #225's parking_lot::RwLock variant nearly matches this PR on same-key hits while keeping the 10–20× disjoint/churn wins — sharding and a shared reader path compose. Per-shard std::sync::RwLock would likely land similarly on Linux/Windows without the new dependency, keeping the macOS caveat.
• perf: sharded OnceTable #225's get hit (638 ns) being 2× slower than its compute hit (312 ns) looks like the same issue 6c55ca4 fixes, so the optimization should transfer.
• The S: Sync requirement discussed above applies to perf: sharded OnceTable #225 as well (it also hashes outside the exclusive lock).

Overall the sharded direction looks stronger; happy to help port the reader-path pieces over, or proceed however the maintainers prefer.