Skip to content

Publish core cache payloads on a largest-first work queue and read ahead in slice digests - #124

Open
javieratomic wants to merge 2 commits into
NVIDIA:mainfrom
javieratomic:parallel-publication
Open

javieratomic wants to merge 2 commits into
NVIDIA:mainfrom
javieratomic:parallel-publication

Conversation

@javieratomic

@javieratomic javieratomic commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Why

ignition export publishes its core into the core cache in phases: each rank's data, then the shared data. On main, cache publication checks payloads in batches of up to eight in listed order. Each batch hashes its sources and existing cache objects concurrently, then inserts the missing objects, links them and records locations one payload at a time, and the next batch waits for the slowest payload of the previous one. With an empty core cache every object is missing, so most of the copying and hashing of a phase runs on one thread. Separately, Slice::digest reads 1 MiB and then hashes it, so the hashing core waits on storage for part of every block; that digest also bounds "Receive rank export payloads" with a warm core cache, where rank zero hashes each object already in its cache, and "Assemble core payloads", which hashes contiguous payload sources.

On a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts, publishing each rank's 101.1-101.5 GB of data takes 56-62 s and the 202.8 GB of shared data takes 763 s with an empty core cache, and "Export native cores" takes 2132 s.

What changed

  1. Cache publication on a largest-first work queue. Before, install_cache_phase_payloads ran prepare_cache_payload on one scoped thread per payload of a batch and kept insertion, linking and location recording in the batch loop. Now up to eight workers take the phase's payloads from one queue, largest first, so the largest payloads, which set the phase's length, do not start after the short ones, and each worker publishes its payload completely:

    • a cached weight the export reuses is linked without hashing, as before;
    • any other payload's source is hashed; the first occurrence of an identity then resolves its cache object, then links it into staging under a lock, and later occurrences prove only their own source bytes;
    • resolving an existing object checks its length and hashes it against the identity, unless the object is the same file span just hashed (same device, inode, offset and length), which the batches hashed twice;
    • a missing object is inserted with Root::insert_payload, which hashes the bytes it writes, by the worker that hashed its source; an export that omits weights still requires each weight object to be cached already.

    A worker drops each object once it is linked, so open object handles stay bounded by the worker count, as the batch size bounded them before. Locations are recorded in listed order after every payload finishes, so the signed catalog order is unchanged. After the first error no new payload starts, and that error is returned once running workers finish; a panic in a worker counts as such an error, as a panicking verifier did before.

  2. Read-ahead in the slice digest. Before, every slice was hashed by the single-buffer 1 MiB loop, and publication sources by their own serial reader. Now slices of 32 MiB or more hash in order while four positional readers fetch interleaved 8 MiB blocks ahead, holding at most 64 MiB per digest; smaller slices keep the single-buffer loop, and publication sources use the slice digest. At most 16 digests read ahead at once in a process (64 reader threads, 1 GiB of buffers); a digest started while 16 are reading ahead uses the single-buffer loop. With three ranks and the default 8 transfer streams per peer, receive runs at most 16 digests, so every digest reads ahead; when peer ranks times streams per peer exceeds 16, digests beyond 16 use the single-buffer loop. Cache publication runs at most eight. The read-ahead buffers are outside the receive I/O budget (IoBudget), which heliopause sizes from available memory and the helios slice digest cannot see; the fixed limit bounds them instead. Cache insertion still hashes the bytes it copies, and re-verifies the installed object, with serial 1 MiB loops.

Core format, signed contents, digests, locations and catalog order are unchanged. Directory publication output is unchanged; its source digests also take the read-ahead path, as does every other Slice::digest of 32 MiB or more in the process. As before, objects inserted before a publication error stay in the cache as verified, unreferenced objects until collection, and no core entry is published.

Result

Measured on the export above, one export per variant and cache state. With an empty core cache, "Export native cores" took 1916 s with item 1 and 1692 s with both items, from 2132 s on main (−21 %). With the previous core cached, it took 1225 s and 822 s, from 1340 s (−39 %). Each rank's data published in 19-20 s instead of 56-62 s with an empty cache, and in 19 s instead of 58-62 s with a warm one. The shared data published in 500 s instead of 763 s (empty) and 112 s instead of 300 s (warm). Item 2 also shortens two phases outside publication: "Assemble core payloads" in both states (389 → 262 s, 274 → 153 s) and, with a warm cache, "Receive rank export payloads" (232 → 152 s). These are export-time measurements of one model on one host configuration, one export per variant and state. "Receive rank export payloads", which item 1 does not change, took 462 s in main's empty-cache export and 522 s with item 1 (13 % apart), so differences of that relative size are not resolved: item 1's change to the whole export (−10 % empty, −9 % warm) and item 2's empty-cache changes to shared publication (584 → 500 s, −14 %) and to the whole export (1916 → 1692 s, −12 %) are of that order. Serving was not measured.

Evidence

Full exports of DeepSeek V4.1 Flash (deepseek-ai/DeepSeek-V4.1-Flash-1017, gb10-tp3) on three DGX Spark GB10 hosts with ignition export --ranks 3 and the default 8 transfer streams, from one Python tree at d2b5c5f3, before main renamed packages to cores. Only the flare_heliopause extension module changed between runs:

  • main: built from d2b5c5f3;
  • + work queue: item 1, built from the first commit of this PR applied to d2b5c5f3;
  • + read-ahead: items 1 and 2, built from both commits applied to d2b5c5f3.

The commits were then rebased onto bbad44fe. Between the two bases, the helios files they change differ only by the package-to-core rename, and in heliopause, core assembly and rank transfer (src/export/core.rs, src/export/transfer.rs) are unchanged apart from one test; heliopause's other export changes since d2b5c5f3 (CUDA graph recording and the cached-weight handoff encoding) and the newer Python tree were not in the measured builds, so the measurement was not repeated. The measured builds did not catch a panicking cache publication worker; that code runs only when a worker panics. The exports reported "Assemble core payloads" as "Assemble package payloads" and "Export native cores" as "Export native packages"; the tables use the current names.

The kernel cache was warm in every run. The forward cache was warm in every run except main's export with an empty core cache, which compiled its forwards before the timed export phase (compilation is not part of "Export native cores"). Page caches were dropped on all hosts before each export. "Empty" runs start with an empty core cache; "warm" runs repeat the export with the previous core still cached. Each variant ran once per state, so run-to-run variation was not measured. "Receive rank export payloads" took 462-522 s with an empty cache in all three variants; item 1 does not change that phase, so the 60 s between main's run (462 s) and the run with item 1 (522 s) is variation between runs, not an effect of item 1. Main's warm export received 5.3 GB from peer ranks and found 198.6 GB cached, against 1.3 GB and 202.7 GB in the other two warm exports; its previous core came from the export that compiled its forwards.

Seconds per phase, as reported by ignition export. "Each rank's data" gives the range over the three ranks' publication phases.

Empty core cache:

Phase main + work queue + read-ahead
Receive rank export payloads 462 522 506
Assemble core payloads 389 389 262
Verify and publish each rank's data 56-62 26-28 19-20
Verify and publish shared data 763 584 500
Export native cores 2132 1916 1692

Warm core cache:

Phase main + work queue + read-ahead
Receive rank export payloads 245 232 152
Assemble core payloads 276 274 153
Verify and publish each rank's data 58-62 27-28 19
Verify and publish shared data 300 300 112
Export native cores 1340 1225 822

Tests on this branch: helios cargo test --release --features server,runtime-core --lib, with FLARE_TEST_TEMPLATE_DIR prepared by tests/frontend/prepare_templates.py: 1338 passed, 0 failed, 26 ignored (1335 at the first commit). heliopause cargo test --release: 189 passed, 0 failed, 3 ignored at both commits.

  • The existing cache publication tests in export_cache_tests.rs cover listed order and deduplication, verification of every duplicate source, read failures that publish no entry, rejection of corrupted cached objects, exports that omit weights, and reused weights; they pass unchanged.
  • same_bytes_requires_the_same_file_span checks that only the same span of the same file counts as the same bytes, and that a payload whose source is its own cache object resolves to that object without reading it again.
  • read_ahead_digest_matches_one_pass_hash and read_ahead_digest_reports_a_short_file check that the read-ahead and single-buffer digests both equal a one-pass SHA-256 of the same span and both report a file shorter than its slice as an error. digest_slots_admit_at_most_their_limit checks that the slot counter (DigestSlots) admits at most its limit and releases a slot on drop.

Not tested: cores from the measured builds were not served; the embedded-asset checks during core acquisition, directory publication with the read-ahead digest, and exports where the limit of 16 read-ahead digests applies were not measured; #157, which parallelizes Engram plane hashing in "Assemble core payloads", was not measured together with item 2. In the empty-cache exports the transfer streams stored the 203.5 GB they received through Root::store_slice with the read-ahead digest; "Receive rank export payloads" took 506 s against 522 s with item 1 alone, which one export each does not resolve.

🤖 Generated with Claude Code

@javieratomic
javieratomic force-pushed the parallel-publication branch 2 times, most recently from 0a31c5d to 3ebf498 Compare October 30, 2026 11:05
@original-el8 original-el8 added area:caching Prefix/KV reuse, cache retention, and model or payload cache behavior. area:packaging Package assembly, verification, distribution, inspection, and loading. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:partial Some reported behavior is addressed; remaining symptoms or requested scope need separate assessment. labels Nov 5, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

Merged #209 overlaps the package-publication optimization, including bounded parallel directory publication and canonical-wrapper reuse. This PR also changes source verification and shared payload-cache resolution, so the full PR is not established as redundant.

Marked triage:partial: rebase onto main and isolate the remaining source-hashing/cache-publication work, then measure those phases separately. #125 targets received-rank cache adoption and #157 targets Engram-plane assembly; neither is proven superseded merely because #209 merged. Source/history review only; the TP3 timing experiments were not rerun.

@original-el8

Copy link
Copy Markdown
Collaborator

@javieratomic GitHub reports that this PR has merge conflicts with main. A local git merge-tree check also confirms conflicts for PR head e66d772cf93e against main at c65024dba32b (2026-11-05).

Conflicting files:

  • helios/src/package/export.rs

Please update the branch against main (rebase or merge), resolve these conflicts, and rerun the affected checks.

javieratomic added a commit to javieratomic/hyperflare-inference that referenced this pull request Nov 6, 2026
A rank export receives each cache payload object as a complete companion
wrapper in a synced temporary file in the package cache. `store_slice` then
hashed the received payload, copied it into a new wrapper in cache staging,
synced the copy, and hashed the installed copy again: three full reads and
one write and sync per object, on the receiving stream's thread. In a
DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts with an empty
package cache, the receive phase on rank zero reports 203.7 GB received from
the other two ranks.

`Root::adopt_companion` links the received file itself into cache staging
when it is byte-for-byte the wrapper `store_slice` would write, removing the
copy read, the write and the sync; both hashes remain. The link step is
shared with directory publication, which already hard-links source files
that are canonical wrappers: `container::link_companion` checks the length
and leading bytes against `companion_prefix`, links the opened file through
its descriptor so that a replaced path cannot be linked instead, checks that
the link names the same inode, and hashes the linked file against the
declared identity. A mismatched identity is an error. For a file that is not
the canonical wrapper, or a link the filesystem refuses, `link_companion`
returns false, `adopt_companion` returns `None`, and the receiver stores the
slice as before. Adoption checks and links the descriptor the receiver
opened, the same one `store_slice` copies from. An already cached identity
returns the verified cached object, as `store_slice` does. Publication is
unchanged: the object is made immutable, linked under its identity and
verified once more. The receiver closes its writable descriptor before the
file is published.

Compatibility: package format, cache layout and object identities are
unchanged. Directory publication keeps its behavior, error kinds and
messages; its link step moved from `export.rs` into the shared function.
Adoption needs Linux (`linkat` through `/proc/self/fd`); on other platforms
received objects are copied as before.

Validation: helios `cargo test --release --features server,runtime-package
--lib`: 1286 passed, 0 failed, 25 ignored, with FLARE_TEST_TEMPLATE_DIR
prepared by tests/frontend/prepare_templates.py. heliopause `cargo test
--release`: 176 passed, 0 failed, 3 ignored. A helios test covers adoption
by link (same inode); a wrapper with an extra tensor and a wrapper of the
right length with different header bytes, both declined; a mismatched
identity, which is rejected; a second arrival of a cached identity, which
returns the cached object; and that cache staging is empty after the
declined, rejected and cached cases. A heliopause test checks that the
receiver's install step adopts a canonical wrapper (same inode) and copies
another framing into a new file. The existing directory publication tests
cover the shared link step.

Measurement, on a different tree: full TP3 exports on three DGX Spark GB10
hosts with the same model and export command, an empty package cache and an
empty forward compilation cache, built from `ca1f960e` plus the parallel
package publication of open pull request NVIDIA#124, without NVIDIA#209 (hashing signed
payloads once and hard-linking unchanged wrappers in directory publication,
merged as `e13ecb74`). The adoption step in that tree is the code of commit
`3978c313`: the same receive-side adoption, linking the received file by
path instead of through the shared `link_companion`. With that step and the
default 8 transfer streams, the "Receive rank export payloads" phase took
280.6 s. Without it, the phase took 457.8 s with 8 streams and 477.2 s with
`--transfer-streams 16`. "Export native packages" took 1473.2 s with that
step and 1677.4 s without it. The package exported with that step served
with `flare run` and answered a 262k-token prompt that asks for 16 values
placed through the context with all 16 correct, with identical output with
an empty and with a warm prefix cache. These figures do not measure this
commit on `972e28e4`.

Limitations: one export per configuration was measured. The payload is still
read twice after it arrives (identity check and post-installation
verification) instead of being hashed while it streams.

Design-Signed-Off-By: Claude Opus 5.5; design-review=4d3a
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
@javieratomic javieratomic changed the title Hash and publish package payloads on parallel workers Publish package cache payloads on parallel workers and read ahead in slice digests Nov 6, 2026
@javieratomic

Copy link
Copy Markdown
Contributor Author

@original-el8 Rebased onto 972e28e4 and reduced to two commits.

The PR now contains only the work that #209 does not cover:

  • Package cache publication on parallel workers. Hash signed payloads once and hard-link unchanged wrappers #209 parallelized directory publication (ignition sign), and its commit message states that cache publication is unchanged. ignition export publishes into the package cache, so on main it still publishes one payload at a time. The separate "Verify payload sources" pass is gone: each worker hashes its source once, just before resolving its cache object.
  • Read-ahead in the slice digest, for every Slice::digest of 32 MiB or more, with at most 16 digests reading ahead at once.

The canonical-wrapper reuse and parallel directory publication from #209 are not duplicated here. The description has the per-phase measurements: main, then each change added in turn, each with an empty and with a warm package cache, from one Python tree with only the extension module changed between runs.

#125 is rebased onto 972e28e4 as well. It now uses the companion_prefix from #209 instead of defining its own, and shares one link step with directory publication. git merge-tree reports no conflicts between any two of this PR, #125 and #157, or between any of them and 972e28e4. #157 and the read-ahead digest (item 2 of the description) both shorten "Assemble package payloads"; they were not measured together.

@original-el8 original-el8 added triage:review Proposed change needs code review; this does not imply merge readiness. and removed triage:partial Some reported behavior is addressed; remaining symptoms or requested scope need separate assessment. labels Nov 6, 2026
@original-el8

Copy link
Copy Markdown
Collaborator

The rebased diff now separates package-cache publication from the directory-publication work merged in #209. Main keeps the cache path sequential; this PR adds parallel cache publication and bounded read-ahead in Slice::digest. The earlier separate source-verification pass is gone.

Moved from triage:partial to triage:review. #125's received-object adoption and #157's Engram-plane hashing remain distinct work. The supplied per-phase measurements are more directly attributable to the two proposed changes; interactions with #157 and the 16-digest saturation case remain unmeasured. Review must preserve digest verification, failure atomicity, and bounded resources. No performance runs were repeated during triage.

@original-el8 original-el8 added the model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. label Nov 8, 2026
@original-el8

original-el8 commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Updated mergeability check: head 50c664a merges cleanly into main 5fd4070; the earlier export-file conflict is resolved. The rebase report distinguishes the largest-first publication queue from the batched concurrent checks already on main. #125's companion adoption and #157's plane hashing remain separate changes. Reported throughput belongs to the measured earlier base; I have not repeated the publication benchmarks or the author's rebased test suites.

javieratomic and others added 2 commits November 10, 2026 10:34
An export publishes its core into the core cache in phases: each rank's data, then the shared data. Cache publication checked payloads in batches of up to eight, in listed order: each batch hashed its sources and any existing cache objects concurrently, and then inserted the missing objects, linked them into staging and recorded locations one payload at a time. A batch could not start before the previous batch's slowest payload and its serial insertions had finished, and with an empty core cache every object is missing, so most of the copying and hashing of a phase ran on one thread. On a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts with an empty core cache, publishing each rank's 101.1-101.5 GB of data took 56-62 s and the 202.8 GB of shared data took 763 s.

Before this change, `install_cache_phase_payloads` ran `prepare_cache_payload` on one scoped thread per payload of a batch and kept insertion, linking and location recording in the batch loop. Now up to eight workers take the phase's payloads from one queue, largest first, so the largest payloads, which set the phase's length, do not start after the short ones, and each worker publishes its payload completely: a cached weight the export reuses is linked without hashing, as before; any other payload's source is hashed; the first occurrence of an identity then resolves its cache object, then links it into staging under a lock, and later occurrences prove only their own source bytes. Resolving an existing object checks its length and hashes it against the identity, unless the object is the same file span just hashed (same device, inode, offset and length), which the batches hashed twice. A missing object is inserted with `Root::insert_payload`, which hashes the bytes it writes, by the worker that hashed its source; an export that omits weights still requires each weight object to be cached already. A worker drops each object once it is linked, so open object handles stay bounded by the worker count, which the batches bounded with their size. Locations are recorded in listed order after every payload finishes, so the signed catalog order is unchanged. After the first error no new payload starts, and that error is returned once running workers finish; a panic in a worker counts as such an error (the batches also reported a panicking verifier as an error); as before, objects inserted before the error stay in the cache as verified, unreferenced objects until collection, and no core entry is published.

Measured on the export described above, one export per variant and cache state: with an empty core cache, each rank's data published in 26-28 s (from 56-62 s), the shared data in 584 s (from 763 s), and "Export native cores" took 1916 s (from 2132 s). With the previous core cached, each rank's data published in 27-28 s (from 58-62 s), the shared data took 300 s in both, and the export took 1225 s (from 1340 s).

Evidence: both variants ran from one Python tree at d2b5c5f with only the `flare_heliopause` extension module changed, built from d2b5c5f and from this commit applied to d2b5c5f, with the kernel cache warm and page caches dropped on all hosts before each export; the forward cache was warm except in the baseline's export with an empty core cache, which compiled its forwards before the timed export phase (compilation is not part of "Export native cores"). The measurements were taken on d2b5c5f, before main renamed packages to cores; there the two export phases were named "Assemble package payloads" and "Export native packages". Between d2b5c5f and bbad44f the helios files this commit changes differ only by that rename, and in heliopause, core assembly and rank transfer (`src/export/core.rs`, `src/export/transfer.rs`) are unchanged apart from one test; heliopause's other export changes since d2b5c5f (CUDA graph recording and the cached-weight handoff encoding) and the newer Python tree were not in the measured builds, so the measurement was not repeated. The measured build did not catch a panicking worker; that code runs only when a worker panics. Each figure is a single export, so run-to-run variation was not measured; "Receive rank export payloads", which this commit does not change, took 462 s and 522 s in the two exports with an empty cache. The baseline's warm export received 5.3 GB from peer ranks and found 198.6 GB cached, against 1.3 GB and 202.7 GB for this commit, probably because its previous core came from the export that compiled its forwards. helios `cargo test --release --features server,runtime-core --lib`: 1335 passed, 0 failed, 26 ignored. heliopause `cargo test --release`: 189 passed, 0 failed, 3 ignored. The existing cache publication tests in `export_cache_tests.rs` cover listed order and deduplication, verification of every duplicate source, read failures that publish no entry, rejection of corrupted cached objects, exports that omit weights, and reused weights; they pass unchanged. A new test, `same_bytes_requires_the_same_file_span`, checks that only the same span of the same file counts as the same bytes, and that a payload whose source is its own cache object resolves to that object without reading it again. Cores from the measured builds were not served.

Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
Several core phases of an export spend most of their time hashing large payloads: cache publication hashes every source and every existing cache object, "Receive rank export payloads" hashes each object already in rank zero's cache instead of receiving it, and "Assemble core payloads" hashes contiguous payload sources. Each of these read 1 MiB and then hashed it, so the hashing core waited on storage for part of every block: `Slice::digest` hashed cache objects, already-cached objects in receive and contiguous assembly sources, and a separate loop (`hash_reader`) hashed publication sources.

Before this change every slice was hashed by that single-buffer loop, and publication sources were hashed by their own serial reader (`hash_reader`). Now slices of 32 MiB or more hash in order while four positional readers fetch interleaved 8 MiB blocks ahead, holding at most 64 MiB per digest; smaller slices keep the single-buffer loop, and publication sources use the slice digest. At most 16 digests read ahead at once in a process, which bounds read-ahead to 64 reader threads and 1 GiB of buffers; a digest started while 16 are reading ahead uses the single-buffer loop. The limit applies to every `Slice::digest` caller in the process, including `Root::store_slice` and the embedded-asset checks when a core is acquired. In receive, rank zero checks the objects it already has cached on one thread per peer rank, one object at a time, before that peer's transfer streams start, and each transfer stream hashes the objects it receives as it stores them (`Root::store_slice`); both run while the model compiled for the export is still in memory. With three ranks and the default 8 streams per peer, receive runs at most 16 digests, so every digest reads ahead; when peer ranks times streams per peer exceeds 16, digests beyond 16 use the single-buffer loop. Cache publication runs at most eight. The read-ahead buffers are outside the receive I/O budget (`IoBudget`), which heliopause sizes from available memory and the helios slice digest cannot see; the fixed limit bounds them instead. Cache insertion still hashes the bytes it copies, and re-verifies the installed object, with serial 1 MiB loops. Digest values are unchanged, and a short file is still reported as an error.

Measured on a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts, on top of the work-queue cache publication, one export per variant and cache state: with an empty core cache, each rank's data published in 19-20 s (from 26-28 s), the shared data in 500 s (from 584 s), "Assemble core payloads" took 262 s (from 389 s), and "Export native cores" 1692 s (from 1916 s). With the previous core cached, each rank's data published in 19 s (from 27-28 s), the shared data in 112 s (from 300 s), "Receive rank export payloads" took 152 s (from 232 s), "Assemble core payloads" 153 s (from 274 s), and the export 822 s (from 1225 s).

Evidence: both variants ran from one Python tree at d2b5c5f with only the `flare_heliopause` extension module changed, built from the previous commit and from this one, both applied to d2b5c5f, with the kernel and forward caches warm, the default 8 transfer streams, and page caches dropped on all hosts before each export. Each figure is a single export, so run-to-run variation was not measured. The measurements were taken on d2b5c5f, before main renamed packages to cores; there the two export phases were named "Assemble package payloads" and "Export native packages". Between d2b5c5f and bbad44f the helios files this commit changes differ only by that rename, and in heliopause, core assembly and rank transfer (`src/export/core.rs`, `src/export/transfer.rs`) are unchanged apart from one test; heliopause's other export changes since d2b5c5f (CUDA graph recording and the cached-weight handoff encoding) and the newer Python tree were not in the measured builds, so the measurement was not repeated. In this configuration receive runs at most 16 digests and cache publication at most eight, so no digest falls back to the single-buffer loop; in the empty-cache exports the transfer streams stored the 203.5 GB they received through `Root::store_slice`, so those digests read ahead; "Receive rank export payloads" took 506 s against 522 s without read-ahead, which one export each does not resolve (the same phase took 462 s and 522 s in two exports with identical receive code). Directory publication, the embedded-asset checks during core acquisition, and exports where more than 16 digests run at once were not measured. helios `cargo test --release --features server,runtime-core --lib`: 1338 passed, 0 failed, 26 ignored. heliopause `cargo test --release`: 189 passed, 0 failed, 3 ignored. Two tests, `read_ahead_digest_matches_one_pass_hash` and `read_ahead_digest_reports_a_short_file`, check that the read-ahead and single-buffer digests both equal a one-pass SHA-256 of the same span and both report a file shorter than its slice as an error. A third, `digest_slots_admit_at_most_their_limit`, checks that the slot counter (`DigestSlots`) admits at most its limit and releases a slot on drop. Cores from the measured builds were not served.

Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
@javieratomic javieratomic changed the title Publish package cache payloads on parallel workers and read ahead in slice digests Publish core cache payloads on a largest-first work queue and read ahead in slice digests Nov 10, 2026
@javieratomic

Copy link
Copy Markdown
Contributor Author

@original-el8 Rebased onto bbad44fe; the conflict in helios/src/package/export.rs (now helios/src/core/export.rs) came from the batched concurrent cache-payload checks that af0907d9 added to main.

The first commit now builds on that version instead of replacing it: the batches of eight become a largest-first queue of up to eight workers, and each worker also inserts and links its missing object, so with an empty core cache insertion no longer runs one payload at a time after each batch. The second commit (read-ahead in Slice::digest, at most 16 digests reading ahead at once) is unchanged apart from the rebase.

Checks on the rebased branch: helios cargo test --release --features server,runtime-core --lib passed 1338 with 26 ignored (1335 at the first commit); heliopause cargo test --release passed 189 with 3 ignored at both commits.

The description has new per-phase export measurements against main's batched publication at d2b5c5f3, then each commit added in turn, with an empty and a warm core cache. "Export native cores" went from 2132 s to 1692 s with an empty cache and from 1340 s to 822 s with a warm one. Between d2b5c5f3 and bbad44fe the helios files this PR changes differ only by the package-to-core rename, and heliopause's core assembly and transfer code is unchanged apart from one test. Each variant ran once per cache state, so run-to-run variation was not measured.

javieratomic added a commit to javieratomic/hyperflare-inference that referenced this pull request Nov 10, 2026
A rank export receives each cache payload object as a complete companion
wrapper in a synced temporary file in the core cache. `store_slice` then
hashes the received payload, copies it into a new wrapper in cache
staging, syncs the copy, and hashes the installed copy again: three full
reads and one write and sync per object, on the receiving stream's
thread. When the received file already is the wrapper `store_slice`
would write, the copy, its write and its sync repeat work whose result
is already on disk. In a DeepSeek V4.1 Flash TP3 export on three DGX
Spark GB10 hosts with an empty core cache, rank zero receives 203.5 GB
from the other two ranks this way.

Before this change the receiver always called `store_slice`. Now it
first calls the new `Root::adopt_companion`, which links the received
file itself into cache staging when it is byte-for-byte that wrapper;
both hashes remain. The link step is shared with directory publication,
which already hard-linked source files that are canonical wrappers from
its own code in `export.rs`; that code moves into
`container::link_companion`. It checks the length and leading bytes
against `companion_prefix`, links the opened file through its descriptor
so that a replaced path cannot be linked instead, checks that the link
names the same inode, and hashes the linked file against the declared
identity; a mismatched identity is an error. For a file that is not the
canonical wrapper, or a link the filesystem refuses, `link_companion`
returns false, `adopt_companion` returns `None`, and the receiver stores
the slice as before. Adoption checks and links the descriptor the
receiver opened, the same one `store_slice` copies from. An already
cached identity returns the verified cached object, as `store_slice`
does. Publication is unchanged: the object is made immutable, linked
under its identity and verified once more, and the receiver closes its
writable descriptor before the file is published. Core format, cache
layout and object identities are unchanged, and directory publication
keeps its behavior, error kinds and messages. Adoption needs Linux
(`linkat` through `/proc/self/fd`); on other platforms received objects
are copied as before.

Expected result: with an empty core cache, "Receive rank export
payloads" no longer copies, writes and syncs each received object. On an
earlier tree, with an earlier version of this adoption, that phase took
280.6 s instead of 457.8 s; this commit itself was not measured on its
current base.

Evidence: helios `cargo test --release --features server,runtime-core
--lib`: 1335 passed, 0 failed, 26 ignored, with FLARE_TEST_TEMPLATE_DIR
prepared by tests/frontend/prepare_templates.py. heliopause `cargo test
--release`: 190 passed, 0 failed, 3 ignored. A helios test covers
adoption by link (same inode); a wrapper with an extra tensor and a
wrapper of the right length with different header bytes, both declined;
a mismatched identity, which is rejected; a second arrival of a cached
identity, which returns the cached object; and that cache staging is
empty after the declined, rejected and cached cases. A heliopause test
checks that the receiver's install step adopts a canonical wrapper (same
inode) and copies another framing into a new file. The existing
directory publication tests cover the shared link step. The 203.5 GB is
the byte count reported by "Receive rank export payloads" in an export
on `d2b5c5f3` without this change, with an empty core cache; there that
phase took 462.3 s. The measurement above used full TP3 exports with the
same model and export command, an empty core cache (then named the
package cache) and an empty forward compilation cache, built from
`ca1f960e` plus an earlier version of NVIDIA#124 that published cache payloads
concurrently and without the directory publication changes merged as
`e13ecb74`; its adoption step linked the received file by path instead
of through `link_companion`. There, with the default 8 transfer streams,
"Receive rank export payloads" took 280.6 s with adoption and 457.8 s
without it (477.2 s without it and `--transfer-streams 16`), and the
whole export phase took 1473.2 s against 1677.4 s; the package exported
with adoption served with `flare run` and answered a 262k-token prompt
that asks for 16 values placed through the context with all 16 correct,
with identical output with an empty and with a warm prefix cache. One
export per configuration was measured. The payload is still read twice
after it arrives (identity check and post-installation verification)
instead of being hashed while it streams.

Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
javieratomic added a commit to javieratomic/hyperflare-inference that referenced this pull request Nov 10, 2026
Name the upstream project and repository after their rename, point every
link at NVIDIA/hyperflare-inference, note that NVIDIA#124 and NVIDIA#125 were rebased
onto upstream's package-to-core rename, add the NVIDIA#124 re-measurement, and
say where the prefill-scratch prototype's branches are.

Design-Signed-Off-By: Claude Opus 5.5; design-review=1c01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
javieratomic added a commit to javieratomic/hyperflare-inference that referenced this pull request Nov 10, 2026
When a core is assembled, `core_engram` resolves each Engram plane's
payload source in turn, and `payload_source` hashes each plane, or
copies and hashes it when it spans several files, with one synchronous
reader. On DeepSeek V4.1 Flash the two weight planes are about 98 GB
each, so "Assemble core payloads" spends most of its time hashing one
plane at a time while one reader at a time cannot keep the NVMe device
busy.

Before this change `core_engram` validated each table and resolved its
planes one after another. Now it first validates every table and lists
its planes (shared weights and scales, or each shard's weights and
scales) with the same checks as before; the new `plane_sources` then
resolves those planes with `payload_source` on up to eight workers,
largest first, stops scheduling at the first error and returns it once
running planes finish. The sources come back in declared order, and the
tables, shards and installed payloads are built from them in the same
order as before, so the published layout does not change.

Measured result, on an earlier base: on a TP3 export on three DGX Spark
GB10 hosts, "Assemble package payloads" (now "Assemble core payloads")
took 169.1 s instead of 262.0 s.

Evidence: helios `cargo test --release --features server,runtime-core
--lib`: 1334 passed, 0 failed, 26 ignored. heliopause `cargo test
--release`: 190 passed, 0 failed, 3 ignored, including
`core_engram_preserves_shared_and_partitioned_plane_layouts`. The new
test `plane_sources_keep_declared_order_and_report_failures` resolves
three planes of different sizes, one of them spanning two source files,
checks that they come back in declared order with the right identities
and sizes, and checks that a plane beyond the payload fails. The
measurement used full exports with the same model and recipe flags, one
per configuration, on builds based on `a161b17f` that also contained
earlier versions of NVIDIA#122, NVIDIA#124 and NVIDIA#125 and the local checkpoint reading
proposed in NVIDIA#126, none of which change this phase in those versions; the
export completed and committed the core, including the data of all three
ranks. This model has four planes, so four readers run at once; the
other two planes are about 3 GB each, so most of the remaining time is
the two weight planes, each still hashed by one reader. The change was
not measured on `bbad44fe`, nor together with the read-ahead digest of
NVIDIA#124 (`50c664a3`), which also shortens this phase.

Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:caching Prefix/KV reuse, cache retention, and model or payload cache behavior. area:packaging Package assembly, verification, distribution, inspection, and loading. model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants