Repository navigation
Conversation
select_snapshots builds an inverted index and enumerates candidate pairs to find the cheapest two-row quote. With heavily shared prefixes (agent-style traffic that reuses one long context across many requests) every posting list carries almost every candidate, so the pair loop approaches O(n^2 * pages): a single quote on a package with ~2000 free snapshots measured 9.4 s in release mode, and the quoting loop re-runs it after every eviction, wedging the scheduler (health stays 200, GPUs idle, no log output). See NVIDIA#319 for full evidence including symbolized stacks of the livelock. Truncate the candidate list entering the pair phase to a constant 128 ids. Candidates arrive in id order and ties already prefer lower ids, so the cap keeps the documented preference while bounding the worst case; quoting stays a heuristic and apply_snapshot_quote re-verifies deterministically. On the reproduced input the quote now takes ~40 ms, and 15 minutes of the wedge-triggering load (12 workers, /v1/responses, max reasoning effort, ~350 KB shared-prefix contexts, mid-stream cancels) run with no stall while decode throughput is unchanged (C16 -1.2%, C32 -0.7% vs the same load before the window).
original-el8
left a comment
There was a problem hiding this comment.
The scheduler stall in #319 needs a bounded-cost fix, but this truncation changes snapshot selection and can reject a feasible allocation. I reproduced the regression using the exact posting_starts/select_snapshots functions from this PR and from main 5fd4070, compiled with rustc. With 130 candidates, ids 0..127 costing two pages and ids 128/129 costing zero, two available pages and keep_pages=2, main selects [128] for count=1 and [128,129] for count=2; this head returns None for both. This is a functional regression, separate from the reported performance improvement. The complete runtime workload was not rerun during this review.
| const MAX_PAIR_CANDIDATES: usize = 128; | ||
| let truncated = candidates.len() > MAX_PAIR_CANDIDATES; | ||
| let candidates = if truncated { | ||
| candidates.into_iter().take(MAX_PAIR_CANDIDATES).collect::<Vec<_>>() |
There was a problem hiding this comment.
[P1] Preserve feasible snapshots beyond the first 128 IDs
Taking the first 128 IDs drops candidates before evaluating their costs and even before the count==1 fast path. IDs are only a tie-break after cost; lower IDs are not generally better choices. Reproduction: backing=[true,true], protected_refs=[0,0], ranges[0..128]=0..2, ranges[128]=ranges[129]=0..0, candidates=0..130 and keep_pages=2. Main returns Some([128]) for one row and Some([128,129]) for two rows; this PR returns None for both although those zero-cost aliases fit. Callers can consequently evict useful checkpoints or reject/delay admissible work. Bound the expensive pair search without imposing this ID-prefix allocation restriction, preserve the unaffected single-row path, and add coverage where the only feasible or cheapest candidates lie past index 127. Deterministically rechecking a returned quote cannot recover candidates that were discarded.
Fixes the scheduler livelock analyzed in #319.
Root cause (confirmed with symbolized gdb stacks + an offline reproduction):
select_snapshotsenumerates candidate pairs through an inverted page index. Under heavily shared prefixes every posting list carries almost every candidate, so the pair loop approaches O(n² · pages): on a package with ~2000 free snapshot slots a single quote measured 9.4 s in release mode (offline unit test on the exact function,select_snapshots_shared_prefix_blowupin this PR).simulate_snapshot_quotere-runs the selection after every eviction, so planning never returns — the worker thread spins insidequote_snapshots_preserving, ranks idle, GPUs idle,/healthkeeps answering 200 and nothing is logged.Fix: truncate the candidate list entering the pair phase to a constant 128 ids. Candidates arrive in id order and the tie-break already prefers lower ids, so the documented preference is kept; quoting stays a heuristic and
apply_snapshot_quotere-verifies the quote deterministically. The reproduced input now quotes in ~40 ms (222×).Validation:
/v1/responses,reasoning_effort: 100, ~350 KB shared-prefix contexts, 2–8 s mid-stream aborts) — zero stalls; the unpatched build wedges in 8 s–8 min on the same load