Skip to content

Report conversion phases while ignition convert verifies and writes a core - #223

Open
kyonjova wants to merge 1 commit into
NVIDIA:mainfrom
kyonjova:convert-progress
Open

kyonjova wants to merge 1 commit into
NVIDIA:mainfrom
kyonjova:convert-progress

Conversation

@kyonjova

@kyonjova kyonjova commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

ignition convert printed nothing until it finished. Converting a two-host core of about 178 GB to packed form hashes every payload and copies every byte, so the command stayed silent for minutes with no sign of whether it was verifying, writing, or stuck, while ignition export reports each of its native steps as a phase row with elapsed time and byte counts. Conversion also read the stored payloads twice before writing: opening the verified core hashed every payload, and pack and unpack then hashed the same payloads again. Reporting verification as its own phase requires the hashing to happen in one place, and the second pass checked nothing the first had not.

What changed

Native pack and unpack gain pack_with and unpack_with, which take a StartupProgress and report four phases:

  • Verify core validates the metadata structure, the commitment digests, and the presence of every selected payload, and finishes with the detail metadata and commitments. It authenticates no signature: conversion needs no key, and the loader authenticates signatures before serving.
  • Verify stored data hashes each located payload once against its signed identity and counts payload bytes. Before this change the payloads were hashed twice.
  • Write packed core or Write core companions counts every byte written to the destination. A forwarding writer reports each write of the primary container and of each companion. The expected total comes from the layout planner that Writer::finish itself uses, so it includes the header length field, the padded header, and the alignment padding as well as the payloads.
  • The last phase publishes the destination.

The convert binding passes the standard error row reporter that export uses. pack, unpack, and their selected variants keep their signatures and report silently; the cache, combine, and promote paths are unchanged. In unit-test builds a progress observation also records the final snapshot of each completed phase, which the tests read; other builds retain the latest snapshot only. docs/cli-design.md states what each phase checks and counts. The core format, the metadata, and the bytes the container writer emits are unchanged.

Result

A terminal shows a redrawing bar for each phase and a redirected log gets a progress line every ten seconds. A finished write phase reports a total, a completed count, and a detail that all equal the size of the files it produced. Verification reads the stored payloads once instead of twice: on a DGX Spark a core of 180.5 GB verified in 47 s, where a core of 178.1 GB took 45 s and 47 s in the two passes of an earlier revision.

Evidence

  • x86 host without a GPU: cargo test --locked --lib --features runtime-core,server core::conversion:: passes its 52 tests on this branch rebased onto main at 5fd4070.

    • One test packs and unpacks a signed fixture under an observed reporter. It asserts each phase's label and detail, that the stored-data total and completed count equal the located payload bytes, and that each write phase's total, completed count, and detail equal the size on disk of the packed file or of the unpacked directory.
    • A second test repeats the write assertions for tensors whose lengths force alignment padding.
    • A third confirms that the reporting forms write the same bytes as the silent forms.
  • A container written from five tensors with misaligned lengths, and four companion headers, hashed identically before and after the writer moved to the shared planner.

  • DGX Spark (GB10): a build that contains this change converted a two-rank GLM 5.3 Flash export to packed form and printed:

    [    OK    ] Verify core.  metadata and commitments                           0.0s
    [    OK    ] Verify stored data.  180523902823 source bytes verified         47.2s
    [    OK    ] Write packed core.  180528104976 bytes written                 146.7s
    [    OK    ] Publish packed core.  <output path>                              6.9s
    Payload form: packed; bytes written: 180528104976
    
  • The earlier two-pass timing comes from a different core, a two-rank MiMo V2.6 Flash RL export of 178,122,320,431 source bytes converted by a revision of this change that still hashed twice (45 s and 47 s). The comparison is between cores of similar size, not a repeat of one conversion.

  • Not run on hardware: unpack to directory form at that size, and conversion of a core whose signature is invalid.

🤖 Generated with Claude Code

@original-el8 original-el8 added area:packaging Package assembly, verification, distribution, inspection, and loading. enhancement New feature or request triage:review Proposed change needs code review; this does not imply merge readiness. labels Nov 8, 2026
kyonjova added a commit to kyonjova/flare-local-inference that referenced this pull request Nov 10, 2026
Upstream renamed the repository to NVIDIA/hyperflare-inference, split the
CLI into flare-cli/ over a flare/ library, and renamed the runtime package
to the runtime core throughout (helios::package to helios::core,
runtime_package.py to runtime_core.py, package_storage to core_storage),
alongside Kimi K3.1 preparation work and a FlashInfer pin move. Keeping the
integration branch on main is what lets the Spark checkouts keep testing the
open PRs against current code.

Nine files conflicted. docs/roce-ring.md, helios/src/execution/roce.rs: keep
the ABI 11 runtime (roce-abi-11) in upstream's core wording. helios/src/core/
conversion.rs and heliopause/src/export/conversion.rs: keep the convert
progress reporting (NVIDIA#223) on the renamed module and crate paths; the primary
name filter already matched upstream's new primary_payload helper.
ignition/preparation/runtime_core.py: keep the pre-export low-floor listing
on the renamed RuntimeCore. The MiMo and DeepSeek V4 recipes, bootstraps,
GLM's reseal report, and tests/recipes/test_deepseek4_hosts.py take upstream's
core_storage parameter name through our multi-host code paths;
tests/preparation/test_low_window.py imports from runtime_core and CoreStorage.

Validation on this box: flare-cli type-checks; Helios core::conversion tests
pass (50); heliopause export tests pass (172); the Python suites for the
touched recipes, low window, toolchain and bmm pass apart from the Qwen, Kimi
and Muse metadata cases that fail identically on unmodified main. Not run on
GB10; the Sparks need uv sync and a flare-cli rebuild after pulling.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Design-Signed-Off-By: Claude Fable 5.1 (Claude Code); design-review=cfcf
Signed-off-by: Kyonjova <kyonjova@pm.me>
kyonjova added a commit to kyonjova/flare-local-inference that referenced this pull request Nov 10, 2026
The convert progress rows still said Verify package after the core rename;
the phase and its documentation now say Verify core, matching the rebased
NVIDIA#223.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Design-Signed-Off-By: Claude Fable 5.1 (Claude Code); design-review=cfcf
Signed-off-by: Kyonjova <kyonjova@pm.me>

@original-el8 original-el8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This addresses a concrete usability problem and merges cleanly into main at 91b2e85. The existing conversion validation, staging, and publication paths remain in place. Two reporting corrections are needed: the successful verification phase claims signature authentication that conversion does not perform, and the write counters mix logical payload bytes with physical container bytes.

Validation at 6cb5d88: all 50 core::conversion tests passed with cargo test --locked --lib --features runtime-core,server core::conversion::. An additional local fixture test flipped one byte of the core signature, confirmed that protection::verify rejected it, then successfully packed and unpacked it while both operations printed [ OK ] Verify core. signatures. That same test reproduced the byte-count discrepancies noted inline. This is a misleading new status message, not a newly introduced authentication bypass: keyless conversion already intentionally preserves signatures without authenticating them.

The added round-trip test checks output bytes but supplies StartupProgress::silent() to both paths; please also cover the observable status text and progress accounting. I did not repeat the reported 178 GB GPU/package-serving validation, and the tests above ran on the PR head rather than the merged tree.

Comment thread helios/src/core/conversion.rs Outdated
}
let payloads = verified.payloads()?;
payloads.verify()?;
phase.finish("signatures");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Describe the checks actually performed, rather than reporting verified signatures

Verified::open validates metadata, body commitments, and payload closure; it does not authenticate signatures. In a local fixture I flipped one byte of core_signature and confirmed protection::verify rejected the envelope, but pack_with and unpack_with both succeeded and printed [ OK ] Verify core. signatures. This new success message can therefore tell an operator that an invalid signature passed verification. Use an accurate completion detail such as metadata and commitments here and at the corresponding unpack site, and correct the PR description. Preserve the existing keyless conversion contract; this does not require adding publisher trust or signature authentication to conversion.

let mut writer = Writer::new(
Reporting::new(file, |bytes| {
written.set(written.get() + bytes);
phase.advance_bytes(bytes);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Use the same byte accounting for totals and progress increments

The expected total sums tensor payload lengths, but Reporting counts every underlying write, including the safetensors header and alignment padding. With the existing fixture, the logical total was 1,218 bytes while the packed writer advanced by 24,605 bytes, so completed can exceed total and the bar can reach 100% prematurely. Unpack also mixes conventions: companions advance only slice.len(), while the primary writer includes container overhead; its final report said 16,627 bytes written for files totaling 24,819 bytes. Choose one consistent measure for total, increments, and the final detail in both operations, and exercise the actual reported counters in a test.

… core

`ignition convert` printed nothing until it finished. Converting a two-host
core of about 178 GB to packed form hashes every payload and copies every
byte, so the command stayed silent for minutes with no sign of whether it
was verifying, writing, or stuck, while `ignition export` reports each of
its native steps as a phase row with elapsed time and byte counts.
Conversion also read the stored payloads twice before writing: opening the
verified core hashed every payload, and pack and unpack then hashed the
same payloads again. Reporting verification as its own phase requires the
hashing to happen in one place, and the second pass checked nothing the
first had not.

Native pack and unpack gain `pack_with` and `unpack_with`, which take a
`StartupProgress` and report four phases. `Verify core` validates the
metadata structure, the commitment digests, and the presence of every
selected payload, and finishes with the detail `metadata and commitments`.
It authenticates no signature: conversion needs no key, and the loader
authenticates signatures before serving. `Verify stored data` hashes each
located payload once against its signed identity and counts payload bytes.
`Write packed core` and `Write core companions` count every byte written
to the destination. A forwarding writer reports each write of the primary
container and of each companion, and the expected total comes from the
layout planner that `Writer::finish` itself uses, so the total includes
the header length field, the padded header, and the alignment padding as
well as the payloads. The last phase publishes the destination. The
convert binding passes the standard error row reporter that export uses.
`pack`, `unpack`, and their selected variants keep their signatures and
report silently; the cache, combine, and promote paths are unchanged. In
unit-test builds a progress observation also records the final snapshot of
each completed phase; other builds retain the latest snapshot only.
`docs/cli-design.md` states what each phase checks and counts. The core
format, the metadata, and the bytes the container writer emits are
unchanged.

A terminal shows a redrawing bar for each phase and a redirected log gets
a progress line every ten seconds. A finished write phase reports a total,
a completed count, and a detail that all equal the size of the files it
produced. Verification reads the stored payloads once instead of twice: on
a DGX Spark a core of 180.5 GB verified in 47 s, where a core of
178.1 GB took 45 s and 47 s in the two passes of an earlier revision.

On an x86 host without a GPU,
`cargo test --locked --lib --features runtime-core,server core::conversion::`
passes its 52 tests. One packs and unpacks a signed fixture under an
observed reporter and asserts each phase's label and detail, that the
stored-data total and completed count equal the located payload bytes, and
that each write phase's total, completed count, and detail equal the size
on disk of the packed file or of the unpacked directory. A second repeats
the write assertions for tensors whose lengths force alignment padding. A
third confirms that the reporting forms write the same bytes as the silent
forms. A container written from five tensors with misaligned lengths, and
four companion headers, hashed identically before and after the writer
moved to the shared planner.

On a DGX Spark (GB10), a build that contains this change converted a
two-rank GLM 5.3 Flash export to packed form. `Verify core` finished in
under 0.1 s with the detail `metadata and commitments`, `Verify stored
data` hashed 180,523,902,823 source bytes in 47.2 s, `Write packed core`
wrote 180,528,104,976 bytes in 146.7 s, and publication took 6.9 s. The
command's summary line reported the same 180,528,104,976 bytes. The
earlier two-pass timing comes from a different core, a two-rank MiMo V2.6
Flash RL export of 178,122,320,431 source bytes, so the comparison is
between cores of similar size and not a repeat of one conversion. Unpack
to directory form at that size, and conversion of a core whose signature
is invalid, have not been run on hardware.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Design-Signed-Off-By: Claude Fable 5.2 (Claude Code); design-review=ed22
Signed-off-by: Kyonjova <kyonjova@pm.me>
@kyonjova kyonjova changed the title Report conversion phases while ignition convert verifies and writes a package Report conversion phases while ignition convert verifies and writes a core Nov 18, 2026
@kyonjova

Copy link
Copy Markdown
Contributor Author

Both corrections are in 81f949a, rebased onto main at 5fd4070.

[P2] Verification detail. Pack and unpack now finish Verify core with metadata and commitments, and docs/cli-design.md states that conversion authenticates no signature. Keyless conversion is unchanged. The description no longer says signatures are verified.

[P3] Byte accounting. Both write phases now count physical bytes throughout. Companions report every write, as the primary already did, and the expected total comes from the layout planner that Writer::finish uses, so it includes the header length field, the padded header, and the alignment padding. Verify stored data keeps payload bytes for its total, increments, and detail.

Tests. A test packs and unpacks the signed fixture under an observed reporter and asserts each phase's label, detail, total, and completed count against the located payload bytes and the sizes on disk. A second covers tensors that force alignment padding. Unit-test builds of StartupObservation record each completed phase for this; other builds are unchanged.

One further change to review: conversion hashed the stored payloads twice, once while opening the verified core and once before writing, which is why the earlier Spark run showed 45 s under Verify core. They are now hashed once, in Verify stored data. The same payload set is hashed before any byte is written.

On a DGX Spark, converting a two-rank GLM 5.3 Flash export at this revision took under 0.1 s in Verify core, 47.2 s to hash 180,523,902,823 source bytes, and 146.7 s to write 180,528,104,976 bytes; the description has the rows.

Not run: unpack to directory form at that size, and a conversion of a core with an invalid signature.

@original-el8 original-el8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 81f949a against main at 5fd4070. Both findings from my previous review are resolved, and I found no remaining blocker. The branch merges cleanly.

Pack and unpack now report metadata and commitments without claiming signature authentication. Write totals, increments, and final details consistently include container headers and alignment padding. Removing the duplicate hash pass preserves verification of the same stored payload set before staging or writing begins.

Validation on the PR head:

  • cargo test --locked --manifest-path helios/Cargo.toml --lib --features runtime-core,server core::: 424 passed, 8 ignored.
  • The same configuration with startup::: 5 passed.
  • Two additional local regression tests passed. One corrupts the core signature, confirms that protection::verify rejects it, and packs/unpacks while asserting the accurate verification detail and physical write counts: 24,605 packed bytes and 24,819 unpacked bytes. The other compares the shared writer byte-for-byte with main's writer across multiple tensor counts and alignment boundaries, including companion headers.

I did not repeat the reported 180 GB conversion or package-serving validation. The additional regression tests were local review checks and are not committed to this branch.

Approving; this supersedes my earlier request for changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:packaging Package assembly, verification, distribution, inspection, and loading. enhancement New feature or request triage:review Proposed change needs code review; this does not imply merge readiness.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants