Repository navigation
Publish core cache payloads on a largest-first work queue and read ahead in slice digests - #124
javieratomic wants to merge 2 commits into
Conversation
0a31c5d to
3ebf498
Compare
ac69333 to
a1e6acd
Compare
|
Merged #209 overlaps the package-publication optimization, including bounded parallel directory publication and canonical-wrapper reuse. This PR also changes source verification and shared payload-cache resolution, so the full PR is not established as redundant. Marked |
|
@javieratomic GitHub reports that this PR has merge conflicts with Conflicting files:
Please update the branch against |
A rank export receives each cache payload object as a complete companion wrapper in a synced temporary file in the package cache. `store_slice` then hashed the received payload, copied it into a new wrapper in cache staging, synced the copy, and hashed the installed copy again: three full reads and one write and sync per object, on the receiving stream's thread. In a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts with an empty package cache, the receive phase on rank zero reports 203.7 GB received from the other two ranks. `Root::adopt_companion` links the received file itself into cache staging when it is byte-for-byte the wrapper `store_slice` would write, removing the copy read, the write and the sync; both hashes remain. The link step is shared with directory publication, which already hard-links source files that are canonical wrappers: `container::link_companion` checks the length and leading bytes against `companion_prefix`, links the opened file through its descriptor so that a replaced path cannot be linked instead, checks that the link names the same inode, and hashes the linked file against the declared identity. A mismatched identity is an error. For a file that is not the canonical wrapper, or a link the filesystem refuses, `link_companion` returns false, `adopt_companion` returns `None`, and the receiver stores the slice as before. Adoption checks and links the descriptor the receiver opened, the same one `store_slice` copies from. An already cached identity returns the verified cached object, as `store_slice` does. Publication is unchanged: the object is made immutable, linked under its identity and verified once more. The receiver closes its writable descriptor before the file is published. Compatibility: package format, cache layout and object identities are unchanged. Directory publication keeps its behavior, error kinds and messages; its link step moved from `export.rs` into the shared function. Adoption needs Linux (`linkat` through `/proc/self/fd`); on other platforms received objects are copied as before. Validation: helios `cargo test --release --features server,runtime-package --lib`: 1286 passed, 0 failed, 25 ignored, with FLARE_TEST_TEMPLATE_DIR prepared by tests/frontend/prepare_templates.py. heliopause `cargo test --release`: 176 passed, 0 failed, 3 ignored. A helios test covers adoption by link (same inode); a wrapper with an extra tensor and a wrapper of the right length with different header bytes, both declined; a mismatched identity, which is rejected; a second arrival of a cached identity, which returns the cached object; and that cache staging is empty after the declined, rejected and cached cases. A heliopause test checks that the receiver's install step adopts a canonical wrapper (same inode) and copies another framing into a new file. The existing directory publication tests cover the shared link step. Measurement, on a different tree: full TP3 exports on three DGX Spark GB10 hosts with the same model and export command, an empty package cache and an empty forward compilation cache, built from `ca1f960e` plus the parallel package publication of open pull request NVIDIA#124, without NVIDIA#209 (hashing signed payloads once and hard-linking unchanged wrappers in directory publication, merged as `e13ecb74`). The adoption step in that tree is the code of commit `3978c313`: the same receive-side adoption, linking the received file by path instead of through the shared `link_companion`. With that step and the default 8 transfer streams, the "Receive rank export payloads" phase took 280.6 s. Without it, the phase took 457.8 s with 8 streams and 477.2 s with `--transfer-streams 16`. "Export native packages" took 1473.2 s with that step and 1677.4 s without it. The package exported with that step served with `flare run` and answered a 262k-token prompt that asks for 16 values placed through the context with all 16 correct, with identical output with an empty and with a warm prefix cache. These figures do not measure this commit on `972e28e4`. Limitations: one export per configuration was measured. The payload is still read twice after it arrives (identity check and post-installation verification) instead of being hashed while it streams. Design-Signed-Off-By: Claude Opus 5.5; design-review=4d3a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
e66d772 to
20d9ac7
Compare
|
@original-el8 Rebased onto The PR now contains only the work that #209 does not cover:
The canonical-wrapper reuse and parallel directory publication from #209 are not duplicated here. The description has the per-phase measurements: main, then each change added in turn, each with an empty and with a warm package cache, from one Python tree with only the extension module changed between runs. #125 is rebased onto |
|
The rebased diff now separates package-cache publication from the directory-publication work merged in #209. Main keeps the cache path sequential; this PR adds parallel cache publication and bounded read-ahead in Moved from |
|
Updated mergeability check: head 50c664a merges cleanly into main 5fd4070; the earlier export-file conflict is resolved. The rebase report distinguishes the largest-first publication queue from the batched concurrent checks already on main. #125's companion adoption and #157's plane hashing remain separate changes. Reported throughput belongs to the measured earlier base; I have not repeated the publication benchmarks or the author's rebased test suites. |
An export publishes its core into the core cache in phases: each rank's data, then the shared data. Cache publication checked payloads in batches of up to eight, in listed order: each batch hashed its sources and any existing cache objects concurrently, and then inserted the missing objects, linked them into staging and recorded locations one payload at a time. A batch could not start before the previous batch's slowest payload and its serial insertions had finished, and with an empty core cache every object is missing, so most of the copying and hashing of a phase ran on one thread. On a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts with an empty core cache, publishing each rank's 101.1-101.5 GB of data took 56-62 s and the 202.8 GB of shared data took 763 s. Before this change, `install_cache_phase_payloads` ran `prepare_cache_payload` on one scoped thread per payload of a batch and kept insertion, linking and location recording in the batch loop. Now up to eight workers take the phase's payloads from one queue, largest first, so the largest payloads, which set the phase's length, do not start after the short ones, and each worker publishes its payload completely: a cached weight the export reuses is linked without hashing, as before; any other payload's source is hashed; the first occurrence of an identity then resolves its cache object, then links it into staging under a lock, and later occurrences prove only their own source bytes. Resolving an existing object checks its length and hashes it against the identity, unless the object is the same file span just hashed (same device, inode, offset and length), which the batches hashed twice. A missing object is inserted with `Root::insert_payload`, which hashes the bytes it writes, by the worker that hashed its source; an export that omits weights still requires each weight object to be cached already. A worker drops each object once it is linked, so open object handles stay bounded by the worker count, which the batches bounded with their size. Locations are recorded in listed order after every payload finishes, so the signed catalog order is unchanged. After the first error no new payload starts, and that error is returned once running workers finish; a panic in a worker counts as such an error (the batches also reported a panicking verifier as an error); as before, objects inserted before the error stay in the cache as verified, unreferenced objects until collection, and no core entry is published. Measured on the export described above, one export per variant and cache state: with an empty core cache, each rank's data published in 26-28 s (from 56-62 s), the shared data in 584 s (from 763 s), and "Export native cores" took 1916 s (from 2132 s). With the previous core cached, each rank's data published in 27-28 s (from 58-62 s), the shared data took 300 s in both, and the export took 1225 s (from 1340 s). Evidence: both variants ran from one Python tree at d2b5c5f with only the `flare_heliopause` extension module changed, built from d2b5c5f and from this commit applied to d2b5c5f, with the kernel cache warm and page caches dropped on all hosts before each export; the forward cache was warm except in the baseline's export with an empty core cache, which compiled its forwards before the timed export phase (compilation is not part of "Export native cores"). The measurements were taken on d2b5c5f, before main renamed packages to cores; there the two export phases were named "Assemble package payloads" and "Export native packages". Between d2b5c5f and bbad44f the helios files this commit changes differ only by that rename, and in heliopause, core assembly and rank transfer (`src/export/core.rs`, `src/export/transfer.rs`) are unchanged apart from one test; heliopause's other export changes since d2b5c5f (CUDA graph recording and the cached-weight handoff encoding) and the newer Python tree were not in the measured builds, so the measurement was not repeated. The measured build did not catch a panicking worker; that code runs only when a worker panics. Each figure is a single export, so run-to-run variation was not measured; "Receive rank export payloads", which this commit does not change, took 462 s and 522 s in the two exports with an empty cache. The baseline's warm export received 5.3 GB from peer ranks and found 198.6 GB cached, against 1.3 GB and 202.7 GB for this commit, probably because its previous core came from the export that compiled its forwards. helios `cargo test --release --features server,runtime-core --lib`: 1335 passed, 0 failed, 26 ignored. heliopause `cargo test --release`: 189 passed, 0 failed, 3 ignored. The existing cache publication tests in `export_cache_tests.rs` cover listed order and deduplication, verification of every duplicate source, read failures that publish no entry, rejection of corrupted cached objects, exports that omit weights, and reused weights; they pass unchanged. A new test, `same_bytes_requires_the_same_file_span`, checks that only the same span of the same file counts as the same bytes, and that a payload whose source is its own cache object resolves to that object without reading it again. Cores from the measured builds were not served. Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
Several core phases of an export spend most of their time hashing large payloads: cache publication hashes every source and every existing cache object, "Receive rank export payloads" hashes each object already in rank zero's cache instead of receiving it, and "Assemble core payloads" hashes contiguous payload sources. Each of these read 1 MiB and then hashed it, so the hashing core waited on storage for part of every block: `Slice::digest` hashed cache objects, already-cached objects in receive and contiguous assembly sources, and a separate loop (`hash_reader`) hashed publication sources. Before this change every slice was hashed by that single-buffer loop, and publication sources were hashed by their own serial reader (`hash_reader`). Now slices of 32 MiB or more hash in order while four positional readers fetch interleaved 8 MiB blocks ahead, holding at most 64 MiB per digest; smaller slices keep the single-buffer loop, and publication sources use the slice digest. At most 16 digests read ahead at once in a process, which bounds read-ahead to 64 reader threads and 1 GiB of buffers; a digest started while 16 are reading ahead uses the single-buffer loop. The limit applies to every `Slice::digest` caller in the process, including `Root::store_slice` and the embedded-asset checks when a core is acquired. In receive, rank zero checks the objects it already has cached on one thread per peer rank, one object at a time, before that peer's transfer streams start, and each transfer stream hashes the objects it receives as it stores them (`Root::store_slice`); both run while the model compiled for the export is still in memory. With three ranks and the default 8 streams per peer, receive runs at most 16 digests, so every digest reads ahead; when peer ranks times streams per peer exceeds 16, digests beyond 16 use the single-buffer loop. Cache publication runs at most eight. The read-ahead buffers are outside the receive I/O budget (`IoBudget`), which heliopause sizes from available memory and the helios slice digest cannot see; the fixed limit bounds them instead. Cache insertion still hashes the bytes it copies, and re-verifies the installed object, with serial 1 MiB loops. Digest values are unchanged, and a short file is still reported as an error. Measured on a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts, on top of the work-queue cache publication, one export per variant and cache state: with an empty core cache, each rank's data published in 19-20 s (from 26-28 s), the shared data in 500 s (from 584 s), "Assemble core payloads" took 262 s (from 389 s), and "Export native cores" 1692 s (from 1916 s). With the previous core cached, each rank's data published in 19 s (from 27-28 s), the shared data in 112 s (from 300 s), "Receive rank export payloads" took 152 s (from 232 s), "Assemble core payloads" 153 s (from 274 s), and the export 822 s (from 1225 s). Evidence: both variants ran from one Python tree at d2b5c5f with only the `flare_heliopause` extension module changed, built from the previous commit and from this one, both applied to d2b5c5f, with the kernel and forward caches warm, the default 8 transfer streams, and page caches dropped on all hosts before each export. Each figure is a single export, so run-to-run variation was not measured. The measurements were taken on d2b5c5f, before main renamed packages to cores; there the two export phases were named "Assemble package payloads" and "Export native packages". Between d2b5c5f and bbad44f the helios files this commit changes differ only by that rename, and in heliopause, core assembly and rank transfer (`src/export/core.rs`, `src/export/transfer.rs`) are unchanged apart from one test; heliopause's other export changes since d2b5c5f (CUDA graph recording and the cached-weight handoff encoding) and the newer Python tree were not in the measured builds, so the measurement was not repeated. In this configuration receive runs at most 16 digests and cache publication at most eight, so no digest falls back to the single-buffer loop; in the empty-cache exports the transfer streams stored the 203.5 GB they received through `Root::store_slice`, so those digests read ahead; "Receive rank export payloads" took 506 s against 522 s without read-ahead, which one export each does not resolve (the same phase took 462 s and 522 s in two exports with identical receive code). Directory publication, the embedded-asset checks during core acquisition, and exports where more than 16 digests run at once were not measured. helios `cargo test --release --features server,runtime-core --lib`: 1338 passed, 0 failed, 26 ignored. heliopause `cargo test --release`: 189 passed, 0 failed, 3 ignored. Two tests, `read_ahead_digest_matches_one_pass_hash` and `read_ahead_digest_reports_a_short_file`, check that the read-ahead and single-buffer digests both equal a one-pass SHA-256 of the same span and both report a file shorter than its slice as an error. A third, `digest_slots_admit_at_most_their_limit`, checks that the slot counter (`DigestSlots`) admits at most its limit and releases a slot on drop. Cores from the measured builds were not served. Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
20d9ac7 to
50c664a
Compare
|
@original-el8 Rebased onto The first commit now builds on that version instead of replacing it: the batches of eight become a largest-first queue of up to eight workers, and each worker also inserts and links its missing object, so with an empty core cache insertion no longer runs one payload at a time after each batch. The second commit (read-ahead in Checks on the rebased branch: helios The description has new per-phase export measurements against main's batched publication at |
A rank export receives each cache payload object as a complete companion wrapper in a synced temporary file in the core cache. `store_slice` then hashes the received payload, copies it into a new wrapper in cache staging, syncs the copy, and hashes the installed copy again: three full reads and one write and sync per object, on the receiving stream's thread. When the received file already is the wrapper `store_slice` would write, the copy, its write and its sync repeat work whose result is already on disk. In a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts with an empty core cache, rank zero receives 203.5 GB from the other two ranks this way. Before this change the receiver always called `store_slice`. Now it first calls the new `Root::adopt_companion`, which links the received file itself into cache staging when it is byte-for-byte that wrapper; both hashes remain. The link step is shared with directory publication, which already hard-linked source files that are canonical wrappers from its own code in `export.rs`; that code moves into `container::link_companion`. It checks the length and leading bytes against `companion_prefix`, links the opened file through its descriptor so that a replaced path cannot be linked instead, checks that the link names the same inode, and hashes the linked file against the declared identity; a mismatched identity is an error. For a file that is not the canonical wrapper, or a link the filesystem refuses, `link_companion` returns false, `adopt_companion` returns `None`, and the receiver stores the slice as before. Adoption checks and links the descriptor the receiver opened, the same one `store_slice` copies from. An already cached identity returns the verified cached object, as `store_slice` does. Publication is unchanged: the object is made immutable, linked under its identity and verified once more, and the receiver closes its writable descriptor before the file is published. Core format, cache layout and object identities are unchanged, and directory publication keeps its behavior, error kinds and messages. Adoption needs Linux (`linkat` through `/proc/self/fd`); on other platforms received objects are copied as before. Expected result: with an empty core cache, "Receive rank export payloads" no longer copies, writes and syncs each received object. On an earlier tree, with an earlier version of this adoption, that phase took 280.6 s instead of 457.8 s; this commit itself was not measured on its current base. Evidence: helios `cargo test --release --features server,runtime-core --lib`: 1335 passed, 0 failed, 26 ignored, with FLARE_TEST_TEMPLATE_DIR prepared by tests/frontend/prepare_templates.py. heliopause `cargo test --release`: 190 passed, 0 failed, 3 ignored. A helios test covers adoption by link (same inode); a wrapper with an extra tensor and a wrapper of the right length with different header bytes, both declined; a mismatched identity, which is rejected; a second arrival of a cached identity, which returns the cached object; and that cache staging is empty after the declined, rejected and cached cases. A heliopause test checks that the receiver's install step adopts a canonical wrapper (same inode) and copies another framing into a new file. The existing directory publication tests cover the shared link step. The 203.5 GB is the byte count reported by "Receive rank export payloads" in an export on `d2b5c5f3` without this change, with an empty core cache; there that phase took 462.3 s. The measurement above used full TP3 exports with the same model and export command, an empty core cache (then named the package cache) and an empty forward compilation cache, built from `ca1f960e` plus an earlier version of NVIDIA#124 that published cache payloads concurrently and without the directory publication changes merged as `e13ecb74`; its adoption step linked the received file by path instead of through `link_companion`. There, with the default 8 transfer streams, "Receive rank export payloads" took 280.6 s with adoption and 457.8 s without it (477.2 s without it and `--transfer-streams 16`), and the whole export phase took 1473.2 s against 1677.4 s; the package exported with adoption served with `flare run` and answered a 262k-token prompt that asks for 16 values placed through the context with all 16 correct, with identical output with an empty and with a warm prefix cache. One export per configuration was measured. The payload is still read twice after it arrives (identity check and post-installation verification) instead of being hashed while it streams. Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
Name the upstream project and repository after their rename, point every link at NVIDIA/hyperflare-inference, note that NVIDIA#124 and NVIDIA#125 were rebased onto upstream's package-to-core rename, add the NVIDIA#124 re-measurement, and say where the prefill-scratch prototype's branches are. Design-Signed-Off-By: Claude Opus 5.5; design-review=1c01 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
When a core is assembled, `core_engram` resolves each Engram plane's payload source in turn, and `payload_source` hashes each plane, or copies and hashes it when it spans several files, with one synchronous reader. On DeepSeek V4.1 Flash the two weight planes are about 98 GB each, so "Assemble core payloads" spends most of its time hashing one plane at a time while one reader at a time cannot keep the NVMe device busy. Before this change `core_engram` validated each table and resolved its planes one after another. Now it first validates every table and lists its planes (shared weights and scales, or each shard's weights and scales) with the same checks as before; the new `plane_sources` then resolves those planes with `payload_source` on up to eight workers, largest first, stops scheduling at the first error and returns it once running planes finish. The sources come back in declared order, and the tables, shards and installed payloads are built from them in the same order as before, so the published layout does not change. Measured result, on an earlier base: on a TP3 export on three DGX Spark GB10 hosts, "Assemble package payloads" (now "Assemble core payloads") took 169.1 s instead of 262.0 s. Evidence: helios `cargo test --release --features server,runtime-core --lib`: 1334 passed, 0 failed, 26 ignored. heliopause `cargo test --release`: 190 passed, 0 failed, 3 ignored, including `core_engram_preserves_shared_and_partitioned_plane_layouts`. The new test `plane_sources_keep_declared_order_and_report_failures` resolves three planes of different sizes, one of them spanning two source files, checks that they come back in declared order with the right identities and sizes, and checks that a plane beyond the payload fails. The measurement used full exports with the same model and recipe flags, one per configuration, on builds based on `a161b17f` that also contained earlier versions of NVIDIA#122, NVIDIA#124 and NVIDIA#125 and the local checkpoint reading proposed in NVIDIA#126, none of which change this phase in those versions; the export completed and committed the core, including the data of all three ranks. This model has four planes, so four readers run at once; the other two planes are about 3 GB each, so most of the remaining time is the two weight planes, each still hashed by one reader. The change was not measured on `bbad44fe`, nor together with the read-ahead digest of NVIDIA#124 (`50c664a3`), which also shortens this phase. Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
Why
ignition exportpublishes its core into the core cache in phases: each rank's data, then the shared data. On main, cache publication checks payloads in batches of up to eight in listed order. Each batch hashes its sources and existing cache objects concurrently, then inserts the missing objects, links them and records locations one payload at a time, and the next batch waits for the slowest payload of the previous one. With an empty core cache every object is missing, so most of the copying and hashing of a phase runs on one thread. Separately,Slice::digestreads 1 MiB and then hashes it, so the hashing core waits on storage for part of every block; that digest also bounds "Receive rank export payloads" with a warm core cache, where rank zero hashes each object already in its cache, and "Assemble core payloads", which hashes contiguous payload sources.On a DeepSeek V4.1 Flash TP3 export on three DGX Spark GB10 hosts, publishing each rank's 101.1-101.5 GB of data takes 56-62 s and the 202.8 GB of shared data takes 763 s with an empty core cache, and "Export native cores" takes 2132 s.
What changed
Cache publication on a largest-first work queue. Before,
install_cache_phase_payloadsranprepare_cache_payloadon one scoped thread per payload of a batch and kept insertion, linking and location recording in the batch loop. Now up to eight workers take the phase's payloads from one queue, largest first, so the largest payloads, which set the phase's length, do not start after the short ones, and each worker publishes its payload completely:Root::insert_payload, which hashes the bytes it writes, by the worker that hashed its source; an export that omits weights still requires each weight object to be cached already.A worker drops each object once it is linked, so open object handles stay bounded by the worker count, as the batch size bounded them before. Locations are recorded in listed order after every payload finishes, so the signed catalog order is unchanged. After the first error no new payload starts, and that error is returned once running workers finish; a panic in a worker counts as such an error, as a panicking verifier did before.
Read-ahead in the slice digest. Before, every slice was hashed by the single-buffer 1 MiB loop, and publication sources by their own serial reader. Now slices of 32 MiB or more hash in order while four positional readers fetch interleaved 8 MiB blocks ahead, holding at most 64 MiB per digest; smaller slices keep the single-buffer loop, and publication sources use the slice digest. At most 16 digests read ahead at once in a process (64 reader threads, 1 GiB of buffers); a digest started while 16 are reading ahead uses the single-buffer loop. With three ranks and the default 8 transfer streams per peer, receive runs at most 16 digests, so every digest reads ahead; when peer ranks times streams per peer exceeds 16, digests beyond 16 use the single-buffer loop. Cache publication runs at most eight. The read-ahead buffers are outside the receive I/O budget (
IoBudget), which heliopause sizes from available memory and the helios slice digest cannot see; the fixed limit bounds them instead. Cache insertion still hashes the bytes it copies, and re-verifies the installed object, with serial 1 MiB loops.Core format, signed contents, digests, locations and catalog order are unchanged. Directory publication output is unchanged; its source digests also take the read-ahead path, as does every other
Slice::digestof 32 MiB or more in the process. As before, objects inserted before a publication error stay in the cache as verified, unreferenced objects until collection, and no core entry is published.Result
Measured on the export above, one export per variant and cache state. With an empty core cache, "Export native cores" took 1916 s with item 1 and 1692 s with both items, from 2132 s on main (−21 %). With the previous core cached, it took 1225 s and 822 s, from 1340 s (−39 %). Each rank's data published in 19-20 s instead of 56-62 s with an empty cache, and in 19 s instead of 58-62 s with a warm one. The shared data published in 500 s instead of 763 s (empty) and 112 s instead of 300 s (warm). Item 2 also shortens two phases outside publication: "Assemble core payloads" in both states (389 → 262 s, 274 → 153 s) and, with a warm cache, "Receive rank export payloads" (232 → 152 s). These are export-time measurements of one model on one host configuration, one export per variant and state. "Receive rank export payloads", which item 1 does not change, took 462 s in main's empty-cache export and 522 s with item 1 (13 % apart), so differences of that relative size are not resolved: item 1's change to the whole export (−10 % empty, −9 % warm) and item 2's empty-cache changes to shared publication (584 → 500 s, −14 %) and to the whole export (1916 → 1692 s, −12 %) are of that order. Serving was not measured.
Evidence
Full exports of DeepSeek V4.1 Flash (
deepseek-ai/DeepSeek-V4.1-Flash-1017,gb10-tp3) on three DGX Spark GB10 hosts withignition export --ranks 3and the default 8 transfer streams, from one Python tree atd2b5c5f3, before main renamed packages to cores. Only theflare_heliopauseextension module changed between runs:d2b5c5f3;d2b5c5f3;d2b5c5f3.The commits were then rebased onto
bbad44fe. Between the two bases, the helios files they change differ only by the package-to-core rename, and in heliopause, core assembly and rank transfer (src/export/core.rs,src/export/transfer.rs) are unchanged apart from one test; heliopause's other export changes sinced2b5c5f3(CUDA graph recording and the cached-weight handoff encoding) and the newer Python tree were not in the measured builds, so the measurement was not repeated. The measured builds did not catch a panicking cache publication worker; that code runs only when a worker panics. The exports reported "Assemble core payloads" as "Assemble package payloads" and "Export native cores" as "Export native packages"; the tables use the current names.The kernel cache was warm in every run. The forward cache was warm in every run except main's export with an empty core cache, which compiled its forwards before the timed export phase (compilation is not part of "Export native cores"). Page caches were dropped on all hosts before each export. "Empty" runs start with an empty core cache; "warm" runs repeat the export with the previous core still cached. Each variant ran once per state, so run-to-run variation was not measured. "Receive rank export payloads" took 462-522 s with an empty cache in all three variants; item 1 does not change that phase, so the 60 s between main's run (462 s) and the run with item 1 (522 s) is variation between runs, not an effect of item 1. Main's warm export received 5.3 GB from peer ranks and found 198.6 GB cached, against 1.3 GB and 202.7 GB in the other two warm exports; its previous core came from the export that compiled its forwards.
Seconds per phase, as reported by
ignition export. "Each rank's data" gives the range over the three ranks' publication phases.Empty core cache:
Warm core cache:
Tests on this branch: helios
cargo test --release --features server,runtime-core --lib, withFLARE_TEST_TEMPLATE_DIRprepared bytests/frontend/prepare_templates.py: 1338 passed, 0 failed, 26 ignored (1335 at the first commit). heliopausecargo test --release: 189 passed, 0 failed, 3 ignored at both commits.export_cache_tests.rscover listed order and deduplication, verification of every duplicate source, read failures that publish no entry, rejection of corrupted cached objects, exports that omit weights, and reused weights; they pass unchanged.same_bytes_requires_the_same_file_spanchecks that only the same span of the same file counts as the same bytes, and that a payload whose source is its own cache object resolves to that object without reading it again.read_ahead_digest_matches_one_pass_hashandread_ahead_digest_reports_a_short_filecheck that the read-ahead and single-buffer digests both equal a one-pass SHA-256 of the same span and both report a file shorter than its slice as an error.digest_slots_admit_at_most_their_limitchecks that the slot counter (DigestSlots) admits at most its limit and releases a slot on drop.Not tested: cores from the measured builds were not served; the embedded-asset checks during core acquisition, directory publication with the read-ahead digest, and exports where the limit of 16 read-ahead digests applies were not measured; #157, which parallelizes Engram plane hashing in "Assemble core payloads", was not measured together with item 2. In the empty-cache exports the transfer streams stored the 203.5 GB they received through
Root::store_slicewith the read-ahead digest; "Receive rank export payloads" took 506 s against 522 s with item 1 alone, which one export each does not resolve.🤖 Generated with Claude Code