Skip to content

Hash Engram planes on parallel readers when assembling a core - #157

Open
javieratomic wants to merge 1 commit into
NVIDIA:mainfrom
javieratomic:parallel-engram-digest
Open

javieratomic wants to merge 1 commit into
NVIDIA:mainfrom
javieratomic:parallel-engram-digest

Conversation

@javieratomic

@javieratomic javieratomic commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Why

When a core is assembled, core_engram resolves each Engram plane's payload source in turn, and payload_source hashes each plane, or copies and hashes it when it spans several files, with one synchronous reader. On DeepSeek V4.1 Flash the two weight planes are about 98 GB each, so "Assemble core payloads" spends most of its time hashing one plane at a time while one reader at a time cannot keep the NVMe device busy.

What changed

Before, core_engram validated each table and resolved its planes one after another. Now:

  • core_engram first validates every table and lists its planes (shared weights and scales, or each shard's weights and scales), with the same checks as before.
  • A new plane_sources resolves those planes with payload_source on up to eight workers, largest first. The first error stops scheduling and is returned once running planes finish.
  • The sources come back in declared order, and the tables, shards and installed payloads are built from them in the same order as before. The published layout does not change.

Result

Measured on an earlier base (see Evidence): on a TP3 export on three DGX Spark GB10 hosts, "Assemble package payloads" (now "Assemble core payloads") took 169.1 s instead of 262.0 s. This PR has not been measured on bbad44fe.

Evidence

Tests on this branch:

  • helios cargo test --release --features server,runtime-core --lib: 1334 passed, 0 failed, 26 ignored.
  • heliopause cargo test --release: 190 passed, 0 failed, 3 ignored, including core_engram_preserves_shared_and_partitioned_plane_layouts.
  • The new test plane_sources_keep_declared_order_and_report_failures resolves three planes of different sizes, one of them spanning two source files, checks that they come back in declared order with the right identities and sizes, and checks that a plane beyond the payload fails.

Measurement: full exports on three GB10 hosts, same model and recipe flags, one per configuration, on builds based on a161b17f that also contained earlier versions of #122, #124 and #125 and the local checkpoint reading proposed in #126, none of which change this phase in those versions. Phase names are as reported then.

without this change with this change
Assemble package payloads 262.0 s 169.1 s

The export completed and committed the core, including the data of all three ranks.

Limitations:

🤖 Generated with Claude Code

@javieratomic
javieratomic force-pushed the parallel-engram-digest branch from 9e52ec4 to cae3c74 Compare October 30, 2026 15:02
@original-el8 original-el8 added area:packaging Package assembly, verification, distribution, inspection, and loading. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:review Proposed change needs code review; this does not imply merge readiness. labels Nov 5, 2026
@original-el8 original-el8 added the model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. label Nov 8, 2026
When a core is assembled, `core_engram` resolves each Engram plane's
payload source in turn, and `payload_source` hashes each plane, or
copies and hashes it when it spans several files, with one synchronous
reader. On DeepSeek V4.1 Flash the two weight planes are about 98 GB
each, so "Assemble core payloads" spends most of its time hashing one
plane at a time while one reader at a time cannot keep the NVMe device
busy.

Before this change `core_engram` validated each table and resolved its
planes one after another. Now it first validates every table and lists
its planes (shared weights and scales, or each shard's weights and
scales) with the same checks as before; the new `plane_sources` then
resolves those planes with `payload_source` on up to eight workers,
largest first, stops scheduling at the first error and returns it once
running planes finish. The sources come back in declared order, and the
tables, shards and installed payloads are built from them in the same
order as before, so the published layout does not change.

Measured result, on an earlier base: on a TP3 export on three DGX Spark
GB10 hosts, "Assemble package payloads" (now "Assemble core payloads")
took 169.1 s instead of 262.0 s.

Evidence: helios `cargo test --release --features server,runtime-core
--lib`: 1334 passed, 0 failed, 26 ignored. heliopause `cargo test
--release`: 190 passed, 0 failed, 3 ignored, including
`core_engram_preserves_shared_and_partitioned_plane_layouts`. The new
test `plane_sources_keep_declared_order_and_report_failures` resolves
three planes of different sizes, one of them spanning two source files,
checks that they come back in declared order with the right identities
and sizes, and checks that a plane beyond the payload fails. The
measurement used full exports with the same model and recipe flags, one
per configuration, on builds based on `a161b17f` that also contained
earlier versions of NVIDIA#122, NVIDIA#124 and NVIDIA#125 and the local checkpoint reading
proposed in NVIDIA#126, none of which change this phase in those versions; the
export completed and committed the core, including the data of all three
ranks. This model has four planes, so four readers run at once; the
other two planes are about 3 GB each, so most of the remaining time is
the two weight planes, each still hashed by one reader. The change was
not measured on `bbad44fe`, nor together with the read-ahead digest of
NVIDIA#124 (`50c664a3`), which also shortens this phase.

Design-Signed-Off-By: Claude Opus 5.5; design-review=cfcf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Javier López <59578422+javieratomic@users.noreply.github.com>
@javieratomic
javieratomic force-pushed the parallel-engram-digest branch from cae3c74 to 293334a Compare November 10, 2026 10:20
@javieratomic javieratomic changed the title Hash Engram planes on parallel readers when assembling a package Hash Engram planes on parallel readers when assembling a core Nov 10, 2026
@javieratomic

Copy link
Copy Markdown
Contributor Author

Rebased onto bbad44fe. The code is unchanged apart from upstream's package-to-core rename in heliopause/src/export/core.rs, which caused the conflict, and the branch merges cleanly with the heads of #124 (50c664a3) and #125 (8c97421a). On the rebased branch: helios cargo test --release --features server,runtime-core --lib: 1334 passed, 0 failed, 26 ignored; heliopause cargo test --release: 190 passed, 0 failed, 3 ignored. The measurement in the description is still the earlier one on a161b17f-based builds; this PR has not been measured on bbad44fe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:packaging Package assembly, verification, distribution, inspection, and loading. model:deepseek-v4.1 Affected recipe or documented reproducer: deepseek-v4.1. Impact may be broader. performance Performance, latency, throughput, or memory-efficiency investigation or improvement. triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants