Skip to content

Limit pair-enumeration candidates in snapshot selection (fixes #319 livelock) - #326

Open
ouyyy18 wants to merge 1 commit into
NVIDIA:mainfrom
ouyyy18:fix/snapshot-pair-candidate-cap
Open

ouyyy18 wants to merge 1 commit into
NVIDIA:mainfrom
ouyyy18:fix/snapshot-pair-candidate-cap

Conversation

@ouyyy18

@ouyyy18 ouyyy18 commented Nov 18, 2026

Copy link
Copy Markdown

Fixes the scheduler livelock analyzed in #319.

Root cause (confirmed with symbolized gdb stacks + an offline reproduction): select_snapshots enumerates candidate pairs through an inverted page index. Under heavily shared prefixes every posting list carries almost every candidate, so the pair loop approaches O(n² · pages): on a package with ~2000 free snapshot slots a single quote measured 9.4 s in release mode (offline unit test on the exact function, select_snapshots_shared_prefix_blowup in this PR). simulate_snapshot_quote re-runs the selection after every eviction, so planning never returns — the worker thread spins inside quote_snapshots_preserving, ranks idle, GPUs idle, /health keeps answering 200 and nothing is logged.

Fix: truncate the candidate list entering the pair phase to a constant 128 ids. Candidates arrive in id order and the tie-break already prefers lower ids, so the documented preference is kept; quoting stays a heuristic and apply_snapshot_quote re-verifies the quote deterministically. The reproduced input now quotes in ~40 ms (222×).

Validation:

select_snapshots builds an inverted index and enumerates candidate
pairs to find the cheapest two-row quote. With heavily shared
prefixes (agent-style traffic that reuses one long context across
many requests) every posting list carries almost every candidate,
so the pair loop approaches O(n^2 * pages): a single quote on a
package with ~2000 free snapshots measured 9.4 s in release mode,
and the quoting loop re-runs it after every eviction, wedging the
scheduler (health stays 200, GPUs idle, no log output). See NVIDIA#319
for full evidence including symbolized stacks of the livelock.

Truncate the candidate list entering the pair phase to a constant
128 ids. Candidates arrive in id order and ties already prefer
lower ids, so the cap keeps the documented preference while
bounding the worst case; quoting stays a heuristic and
apply_snapshot_quote re-verifies deterministically. On the
reproduced input the quote now takes ~40 ms, and 15 minutes of
the wedge-triggering load (12 workers, /v1/responses, max
reasoning effort, ~350 KB shared-prefix contexts, mid-stream
cancels) run with no stall while decode throughput is unchanged
(C16 -1.2%, C32 -0.7% vs the same load before the window).

@original-el8 original-el8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scheduler stall in #319 needs a bounded-cost fix, but this truncation changes snapshot selection and can reject a feasible allocation. I reproduced the regression using the exact posting_starts/select_snapshots functions from this PR and from main 5fd4070, compiled with rustc. With 130 candidates, ids 0..127 costing two pages and ids 128/129 costing zero, two available pages and keep_pages=2, main selects [128] for count=1 and [128,129] for count=2; this head returns None for both. This is a functional regression, separate from the reported performance improvement. The complete runtime workload was not rerun during this review.

const MAX_PAIR_CANDIDATES: usize = 128;
let truncated = candidates.len() > MAX_PAIR_CANDIDATES;
let candidates = if truncated {
candidates.into_iter().take(MAX_PAIR_CANDIDATES).collect::<Vec<_>>()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve feasible snapshots beyond the first 128 IDs

Taking the first 128 IDs drops candidates before evaluating their costs and even before the count==1 fast path. IDs are only a tie-break after cost; lower IDs are not generally better choices. Reproduction: backing=[true,true], protected_refs=[0,0], ranges[0..128]=0..2, ranges[128]=ranges[129]=0..0, candidates=0..130 and keep_pages=2. Main returns Some([128]) for one row and Some([128,129]) for two rows; this PR returns None for both although those zero-cost aliases fit. Callers can consequently evict useful checkpoints or reject/delay admissible work. Bound the expensive pair search without imposing this ID-prefix allocation restriction, preserve the unaffected single-row path, and add coverage where the only feasible or cheapest candidates lie past index 127. Deterministically rechecking a returned quote cannot recover candidates that were discarded.

@original-el8 original-el8 added area:scheduling Request scheduling, fairness, batching, and speculative-width selection. bug Something isn't working priority:P1 High attention: unresolved correctness, serving failure, or model-data deletion risk. triage:review Proposed change needs code review; this does not imply merge readiness. labels Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scheduling Request scheduling, fairness, batching, and speculative-width selection. bug Something isn't working priority:P1 High attention: unresolved correctness, serving failure, or model-data deletion risk. triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants