Skip to content

Reuse checkpoint selections through scheduler commit - #329

Open
yatesdr wants to merge 1 commit into
NVIDIA:mainfrom
yatesdr:fix/helios-commit-decision-cache
Open

yatesdr wants to merge 1 commit into
NVIDIA:mainfrom
yatesdr:fix/helios-commit-decision-cache

Conversation

@yatesdr

@yatesdr yatesdr commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Problem

Under prompt load, every rank sits idle while the worker prepares the next submission. On 4x RTX PRO 6000 serving GLM-5.3-Flash with 16 returning users and a cold 20K-token prompt every 4 s, about 28% of wall time fell outside graph replay. Nsight shows the four GPUs idle together, and 96% of that time comes before any rank starts the next input upload. Copies, event waits and peer transfers are negligible in those gaps. Symbolized perf places the worker in the following chain in 74% of its samples during the gaps, against 6% elsewhere:

commit → service_participants → service_rates → prefill_work → checkpoint_selection → quote_snapshots_preserving → simulate_snapshot_quote → select_snapshots

Cause

Commit computes a service-rate normalizer for every participant. For each prefilling request it walks the prefill catalog, and each capacity runs an exact snapshot-storage simulation through checkpoint selection.

Selection already caches those decisions per request, but that cache closes before commit. The same decisions are therefore recomputed once per capacity on every dispatch, including dispatches that perform no prefill, restore or snapshot. The cost grows with the number of admitted prefilling requests, which is why it appears only under prompt load.

Change

The worker keeps the existing checkpoint selection cache open for one planning and submission attempt:

  • Nested scopes reuse an enclosing cache.
  • Commit reuses selection's decisions for its normalizers, then clears them before it changes request or storage state.
  • advance_to invalidates them when the snapshot cost expires.
  • The worker closes the cache on success, error and panic.

The begin/end contract is documented on begin_checkpoint_planning. Plans and service prices are unchanged. Nothing is added to configuration, storage policy or GPU code.

A smaller variant scoped the cache only around commit's rate loop. It leaves one simulation per request per dispatch, and measured 7.2% outside replay with a 44 s TTFT p95, against 4.4% and 37 s for this change.

Tests

  • Helios default suite: 544 passed, 4 ignored.
  • Server/runtime-core library suite: 1,436 passed, 33 ignored.
  • New checkpoint tests:
    • complete plans and service-price bits match over a loaded trace with and without the cache;
    • commit runs no storage simulation while costs stay fresh;
    • expiry invalidates the cached selections.
  • Warnings match main for default, server and test builds.
  • No unit test drives the real Worker::run loop. The runtime bracket is covered by the serving measurement below.

Measurement

Setup:

  • 4x RTX PRO 6000 Blackwell Max-Q, TP4 with decode context parallelism, GLM-5.3-Flash-NVFP4, native flare run.
  • 16 returning users at 8K, 32K, 100K and 200K contexts, 600 output tokens per turn.
  • An exact cold 20K-token prompt every 4 s, for 180 s of arrivals.
Metric Before After
Wall time outside graph replay 28.4% 4.4%
Prefill achieved 2,880 tok/s 3,569 tok/s
Returning-turn decode, median / p10 14.3 / 10.7 tok/s 17.7 / 14.3 tok/s
Returning-turn TTFT p95 66.2 s 36.8 s
Returning turns completed 49 73
Request errors 0 0

Isolated single-stream decode and cold prefill at 7K, 100K and 400K stayed within run-to-run variation, about 1–2%.

Commit computes a service-rate normalizer for every participant. For each
prefilling request it walks the prefill catalog, and each capacity runs an
exact snapshot-storage simulation through checkpoint selection. Selection
already caches those decisions per request, but the cache closed before
commit, so the same decisions were recomputed per capacity on every
dispatch. Under prompt load this kept the worker on the CPU for 20-100 ms
before each submission while every rank sat idle.

The worker now keeps the existing checkpoint selection cache open across
one planning and submission attempt. Nested scopes reuse an enclosing
cache; commit reuses selection's decisions for its normalizers and clears
them before changing request or storage state; clock advancement
invalidates them when the snapshot cost expires. The worker closes the
cache on success, error and panic. Plans and service prices are unchanged.

Tested: Helios default suite (544 passed) and server/runtime-core library
suite (1,436 passed). New tests compare complete plans and service-price
bits over a loaded trace with and without the cache, check that commit
runs no storage simulation while costs stay fresh, and cover expiry. On
4x RTX PRO 6000 serving GLM-5.3-Flash with 16 returning users and a cold
20K prompt every 4 s, time outside graph replay fell from 28.4% to 4.4%,
prefill from 2,880 to 3,569 tok/s achieved, returning-turn decode median
from 14.3 to 17.7 tok/s and TTFT p95 from 66 s to 37 s.

Design-Signed-Off-By: yatesdr; design-review=ed22
@original-el8 original-el8 added area:scheduling Request scheduling, fairness, batching, and speculative-width selection. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:review Proposed change needs code review; this does not imply merge readiness. labels Nov 18, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

This overlaps #296's narrower service-rate scope; both remain unmerged and merge cleanly into main 5fd4070. The wider cache lifetime makes invalidation at clock expiry and before state mutation central to correctness, and the added plan/service-price comparisons directly address that risk. Also evaluate the shared-prefix/cancellation reproduction in #319: #326 currently drops feasible candidates, so its reported stall fix cannot be assumed as a safe prerequisite. This is a source/triage update; I have not reproduced the GPU workload or approved this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scheduling Request scheduling, fairness, batching, and speculative-width selection. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants