Repository navigation
Conversation
Commit computes a service-rate normalizer for every participant. For each prefilling request it walks the prefill catalog, and each capacity runs an exact snapshot-storage simulation through checkpoint selection. Selection already caches those decisions per request, but the cache closed before commit, so the same decisions were recomputed per capacity on every dispatch. Under prompt load this kept the worker on the CPU for 20-100 ms before each submission while every rank sat idle. The worker now keeps the existing checkpoint selection cache open across one planning and submission attempt. Nested scopes reuse an enclosing cache; commit reuses selection's decisions for its normalizers and clears them before changing request or storage state; clock advancement invalidates them when the snapshot cost expires. The worker closes the cache on success, error and panic. Plans and service prices are unchanged. Tested: Helios default suite (544 passed) and server/runtime-core library suite (1,436 passed). New tests compare complete plans and service-price bits over a loaded trace with and without the cache, check that commit runs no storage simulation while costs stay fresh, and cover expiry. On 4x RTX PRO 6000 serving GLM-5.3-Flash with 16 returning users and a cold 20K prompt every 4 s, time outside graph replay fell from 28.4% to 4.4%, prefill from 2,880 to 3,569 tok/s achieved, returning-turn decode median from 14.3 to 17.7 tok/s and TTFT p95 from 66 s to 37 s. Design-Signed-Off-By: yatesdr; design-review=ed22
Collaborator
|
This overlaps #296's narrower service-rate scope; both remain unmerged and merge cleanly into main 5fd4070. The wider cache lifetime makes invalidation at clock expiry and before state mutation central to correctness, and the added plan/service-price comparisons directly address that risk. Also evaluate the shared-prefix/cancellation reproduction in #319: #326 currently drops feasible candidates, so its reported stall fix cannot be assumed as a safe prerequisite. This is a source/triage update; I have not reproduced the GPU workload or approved this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Under prompt load, every rank sits idle while the worker prepares the next submission. On 4x RTX PRO 6000 serving GLM-5.3-Flash with 16 returning users and a cold 20K-token prompt every 4 s, about 28% of wall time fell outside graph replay. Nsight shows the four GPUs idle together, and 96% of that time comes before any rank starts the next input upload. Copies, event waits and peer transfers are negligible in those gaps. Symbolized
perfplaces the worker in the following chain in 74% of its samples during the gaps, against 6% elsewhere:commit → service_participants → service_rates → prefill_work → checkpoint_selection → quote_snapshots_preserving → simulate_snapshot_quote → select_snapshotsCause
Commit computes a service-rate normalizer for every participant. For each prefilling request it walks the prefill catalog, and each capacity runs an exact snapshot-storage simulation through checkpoint selection.
Selection already caches those decisions per request, but that cache closes before commit. The same decisions are therefore recomputed once per capacity on every dispatch, including dispatches that perform no prefill, restore or snapshot. The cost grows with the number of admitted prefilling requests, which is why it appears only under prompt load.
Change
The worker keeps the existing checkpoint selection cache open for one planning and submission attempt:
advance_toinvalidates them when the snapshot cost expires.The begin/end contract is documented on
begin_checkpoint_planning. Plans and service prices are unchanged. Nothing is added to configuration, storage policy or GPU code.A smaller variant scoped the cache only around commit's rate loop. It leaves one simulation per request per dispatch, and measured 7.2% outside replay with a 44 s TTFT p95, against 4.4% and 37 s for this change.
Tests
mainfor default, server and test builds.Worker::runloop. The runtime bracket is covered by the serving measurement below.Measurement
Setup:
flare run.Isolated single-stream decode and cold prefill at 7K, 100K and 400K stayed within run-to-run variation, about 1–2%.