Split out of Wave B PR-B4 final review (PR #325, issue #304).
Gap
The written path supports var_fields ride-alongs on with_seqs("variant-windows")
output — _FlatVariantWindows.fields carries start/ilen/dosage/INFO columns, and
skills/genvarloader/SKILL.md documents reading them as win.fields["AF"].
Streaming does not. StreamingDataset.with_settings(var_fields=[...]) raises
NotImplementedError for any with_seqs kind other than "variants" (PR-B3a's Phase-1
guard), so streaming is stricter than the written path here.
Consequence: dead-but-harmless code already merged
PR-B4 Task 8 wired the Rust plumbing on the assumption the guard would be lifted, so the
following can never carry data today:
Svar1StreamEngine::generate_variant_windows's gather_call_bufs call
(src/ffi/stream_engine.rs:577) and RecordBackend's gather_info_out
(src/record_stream/engine.rs:485) — both always produce an empty info_out
- both engines'
next_batch_variant_windows info_out marshaling loops
(stream_engine.rs:1098, engine.rs:918) — they run, but always iterate zero items
Harmless, but it is untested code on a parity-critical path.
To close
- Lift the
with_settings(var_fields=...) guard for kind == "variant-windows" (keep it
for all other kinds).
- Actually pack
info_out in _iter_batches's variant-windows branch — it is currently
silently dropped even though the engines return it.
- Extend
_RESERVED_VAR_FIELD_NAMES with the windows-dict keys (ref_window,
alt_window, and their _offsets variants). The variant-windows output is marshaled as
a plain dict, so a user-requested var_field named e.g. ref_window would silently
clobber a token buffer.
- Byte-identical parity vs the written
_FlatVariantWindows.fields oracle, across all
three backends and both win_ref_mode/win_alt_mode settings — mirroring
tests/dataset/test_streaming_variants_parity.py::test_streaming_variant_windows_matches_written.
- Docs:
docs/source/dataset.md + skills/genvarloader/SKILL.md currently document the
streaming restriction; drop it when the guard lifts.
Relates to #304, #277.
Split out of Wave B PR-B4 final review (PR #325, issue #304).
Gap
The written path supports
var_fieldsride-alongs onwith_seqs("variant-windows")output —
_FlatVariantWindows.fieldscarriesstart/ilen/dosage/INFO columns, andskills/genvarloader/SKILL.mddocuments reading them aswin.fields["AF"].Streaming does not.
StreamingDataset.with_settings(var_fields=[...])raisesNotImplementedErrorfor anywith_seqskind other than"variants"(PR-B3a's Phase-1guard), so streaming is stricter than the written path here.
Consequence: dead-but-harmless code already merged
PR-B4 Task 8 wired the Rust plumbing on the assumption the guard would be lifted, so the
following can never carry data today:
Svar1StreamEngine::generate_variant_windows'sgather_call_bufscall(
src/ffi/stream_engine.rs:577) andRecordBackend'sgather_info_out(
src/record_stream/engine.rs:485) — both always produce an emptyinfo_outnext_batch_variant_windowsinfo_outmarshaling loops(
stream_engine.rs:1098,engine.rs:918) — they run, but always iterate zero itemsHarmless, but it is untested code on a parity-critical path.
To close
with_settings(var_fields=...)guard forkind == "variant-windows"(keep itfor all other kinds).
info_outin_iter_batches's variant-windows branch — it is currentlysilently dropped even though the engines return it.
_RESERVED_VAR_FIELD_NAMESwith the windows-dict keys (ref_window,alt_window, and their_offsetsvariants). The variant-windows output is marshaled asa plain
dict, so a user-requestedvar_fieldnamed e.g.ref_windowwould silentlyclobber a token buffer.
_FlatVariantWindows.fieldsoracle, across allthree backends and both
win_ref_mode/win_alt_modesettings — mirroringtests/dataset/test_streaming_variants_parity.py::test_streaming_variant_windows_matches_written.docs/source/dataset.md+skills/genvarloader/SKILL.mdcurrently document thestreaming restriction; drop it when the guard lifts.
Relates to #304, #277.