Skip to content

streaming: var_fields ride-alongs for with_seqs("variant-windows") #328

Description

@d-laub

Split out of Wave B PR-B4 final review (PR #325, issue #304).

Gap

The written path supports var_fields ride-alongs on with_seqs("variant-windows")
output — _FlatVariantWindows.fields carries start/ilen/dosage/INFO columns, and
skills/genvarloader/SKILL.md documents reading them as win.fields["AF"].

Streaming does not. StreamingDataset.with_settings(var_fields=[...]) raises
NotImplementedError for any with_seqs kind other than "variants" (PR-B3a's Phase-1
guard), so streaming is stricter than the written path here.

Consequence: dead-but-harmless code already merged

PR-B4 Task 8 wired the Rust plumbing on the assumption the guard would be lifted, so the
following can never carry data today:

  • Svar1StreamEngine::generate_variant_windows's gather_call_bufs call
    (src/ffi/stream_engine.rs:577) and RecordBackend's gather_info_out
    (src/record_stream/engine.rs:485) — both always produce an empty info_out
  • both engines' next_batch_variant_windows info_out marshaling loops
    (stream_engine.rs:1098, engine.rs:918) — they run, but always iterate zero items

Harmless, but it is untested code on a parity-critical path.

To close

  1. Lift the with_settings(var_fields=...) guard for kind == "variant-windows" (keep it
    for all other kinds).
  2. Actually pack info_out in _iter_batches's variant-windows branch — it is currently
    silently dropped even though the engines return it.
  3. Extend _RESERVED_VAR_FIELD_NAMES with the windows-dict keys (ref_window,
    alt_window, and their _offsets variants). The variant-windows output is marshaled as
    a plain dict, so a user-requested var_field named e.g. ref_window would silently
    clobber a token buffer.
  4. Byte-identical parity vs the written _FlatVariantWindows.fields oracle, across all
    three backends and both win_ref_mode/win_alt_mode settings — mirroring
    tests/dataset/test_streaming_variants_parity.py::test_streaming_variant_windows_matches_written.
  5. Docs: docs/source/dataset.md + skills/genvarloader/SKILL.md currently document the
    streaming restriction; drop it when the guard lifts.

Relates to #304, #277.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions