Repository navigation
Hash Engram planes on parallel readers when assembling a core - #157
Open
javieratomic wants to merge 1 commit into
Open
javieratomic wants to merge 1 commit into
javieratomic wants to merge 1 commit into
Conversation
javieratomic
force-pushed
the
parallel-engram-digest
branch
from
October 30, 2026 15:02
9e52ec4 to
cae3c74
Compare
When a core is assembled, `core_engram` resolves each Engram plane's payload source in turn, and `payload_source` hashes each plane, or copies and hashes it when it spans several files, with one synchronous reader. On DeepSeek V4.1 Flash the two weight planes are about 98 GB each, so "Assemble core payloads" spends most of its time hashing one plane at a time while one reader at a time cannot keep the NVMe device busy. Before this change `core_engram` validated each table and resolved its planes one after another. Now it first validates every table and lists its planes (shared weights and scales, or each shard's weights and scales) with the same checks as before; the new `plane_sources` then resolves those planes with `payload_source` on up to eight workers, largest first, stops scheduling at the first error and returns it once running planes finish. The sources come back in declared order, and the tables, shards and installed payloads are built from them in the same order as before, so the published layout does not change. Measured result, on an earlier base: on a TP3 export on three DGX Spark GB10 hosts, "Assemble package payloads" (now "Assemble core payloads") took 169.1 s instead of 262.0 s. Evidence: helios `cargo test --release --features server,runtime-core --lib`: 1334 passed, 0 failed, 26 ignored. heliopause `cargo test --release`: 190 passed, 0 failed, 3 ignored, including `core_engram_preserves_shared_and_partitioned_plane_layouts`. The new test `plane_sources_keep_declared_order_and_report_failures` resolves three planes of different sizes, one of them spanning two source files, checks that they come back in declared order with the right identities and sizes, and checks that a plane beyond the payload fails. The measurement used full exports with the same model and recipe flags, one per configuration, on builds based on `a161b17f` that also contained earlier versions of NVIDIA#122, NVIDIA#124 and NVIDIA#125 and the local checkpoint reading proposed in NVIDIA#126, none of which change this phase in those versions; the export completed and committed the core, including the data of all three ranks. This model has four planes, so four readers run at once; the other two planes are about 3 GB each, so most of the remaining time is the two weight planes, each still hashed by one reader. The change was not measured on `bbad44fe`, nor together with the read-ahead digest of NVIDIA#124 (`50c664a3`), which also shortens this phase. Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
javieratomic
force-pushed
the
parallel-engram-digest
branch
from
November 10, 2026 10:20
cae3c74 to
293334a
Compare
Contributor
Author
|
Rebased onto |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When a core is assembled,
core_engramresolves each Engram plane's payload source in turn, andpayload_sourcehashes each plane, or copies and hashes it when it spans several files, with one synchronous reader. On DeepSeek V4.1 Flash the two weight planes are about 98 GB each, so "Assemble core payloads" spends most of its time hashing one plane at a time while one reader at a time cannot keep the NVMe device busy.What changed
Before,
core_engramvalidated each table and resolved its planes one after another. Now:core_engramfirst validates every table and lists its planes (shared weights and scales, or each shard's weights and scales), with the same checks as before.plane_sourcesresolves those planes withpayload_sourceon up to eight workers, largest first. The first error stops scheduling and is returned once running planes finish.Result
Measured on an earlier base (see Evidence): on a TP3 export on three DGX Spark GB10 hosts, "Assemble package payloads" (now "Assemble core payloads") took 169.1 s instead of 262.0 s. This PR has not been measured on
bbad44fe.Evidence
Tests on this branch:
cargo test --release --features server,runtime-core --lib: 1334 passed, 0 failed, 26 ignored.cargo test --release: 190 passed, 0 failed, 3 ignored, includingcore_engram_preserves_shared_and_partitioned_plane_layouts.plane_sources_keep_declared_order_and_report_failuresresolves three planes of different sizes, one of them spanning two source files, checks that they come back in declared order with the right identities and sizes, and checks that a plane beyond the payload fails.Measurement: full exports on three GB10 hosts, same model and recipe flags, one per configuration, on builds based on
a161b17fthat also contained earlier versions of #122, #124 and #125 and the local checkpoint reading proposed in #126, none of which change this phase in those versions. Phase names are as reported then.The export completed and committed the core, including the data of all three ranks.
Limitations:
50c664a3adds read-ahead toSlice::digest, whichpayload_sourceuses for contiguous planes, so it also shortens this phase; the two were not measured together. The 262 s that Publish core cache payloads on a largest-first work queue and read ahead in slice digests #124 reports for this phase comes from a different tree and is not comparable with the 262.0 s baseline here.🤖 Generated with Claude Code