fix(write): bound SVAR2 genotype-writing memory with max_mem

[d-laub]

Closes #333.

Note

Unblocked. genoray 3.4.0 is published, pixi.lock has been regenerated against it, and all 8 CI checks pass.

Three defects, only one named in the issue

1. genoray find_ranges was O(regions × total_variants). A SearchTree plus a v_ends vector were rebuilt for every (region, column) pair, so a batch swept the packed store once per region instead of once. At the reported scale (3,964 regions × 414,830 samples × ploidy 2) that is 6.6e9 tree builds — the caller sat at 0% for hours before being OOM-killed. Fixed upstream by hoisting per-column state into a VkColumnIndex, inverting the loop to column-outer, and parallelising with rayon.
2. _svar2_region_max_ends decoded ALL samples for the whole contig, ignoring the caller's sample selection, purely to extend chromEnd. Deleted. genoray now returns per-region packed max-end keys from the same range sweep, so no decode pass happens at all.
3. _write_from_svar2 received no memory budget and materialised a whole contig's ranges at once (~245 GiB at 414k samples). It now consumes genoray's chunked _find_ranges_chunked stream: per-sample-slice chunks sized from max_mem, written straight into the memmaps through a transposed view and flushed per chunk.

Also: a disk preflight that logs and warns on the ~98 GiB permanent range cache, fractional within-contig progress (a single-contig BED no longer sits at 0% for the whole run), and corrected docs — format.md previously called that cache "small".

Effect

At the #333 scale with max_mem="4g": bytes_per_sample = 253,696 → 8,465 samples/chunk → 49 chunks of ~2.15 GB. The 245 GiB peak is gone.

Max-end keys are reduced packed, not unpacked

SVAR1 parity picks the highest-position overlapping variant, ties broken by the larger end — not the largest end overall. So the reduction unit across chunks is the packed (pos << 21) | (1 + deletion_len) key, reduced with an elementwise np.maximum and unpacked exactly once at the end. Reducing unpacked ends would let a lower-position deletion that reaches further win, silently corrupting chromEnd.

Testing

• Full tree: 1104 passed, 4 xfailed, 56 skipped, 0 failures. ruff check / ruff format --check clean; pyrefly check 0 errors; Sphinx docs build succeeds (45 warnings, all pre-existing myst.xref_missing, none on a touched line).
• test_write_svar2_chunked_matches_unchunked forces genuine multi-chunk writing (max_mem=256 → 1 sample/chunk → 2 chunks) and asserts the output is byte-identical to a single-chunk write.
• The pre-existing test_write_svar2_max_ends_matches_svar1 SVAR1 ground-truth oracle still passes through the new packed-key path.

Before merging

Both former blockers are resolved:

1. Release genoray 3.4.0. Published; perf(query): make find_ranges O(total_variants) and add a chunked API d-laub/genoray#146 and Upgrade genoray dep #147 are merged upstream.
2. Regenerate pixi.lock. Done in build(deps): require genoray>=3.4.0 for the chunked find_ranges API — genoray 3.3.1 → 3.4.0 across all 10 environments, with no other resolver churn. That commit was pushed with the pixi-lock and pyrefly-check pre-push hooks enabled (they were skipped on the earlier push only because pixi lock cannot resolve an unpublished version), and both passed.

CI is green on all 8 checks: pre-commit hooks, pre-commit.ci - pr, pytest on py310/py311/py312/py313, coverage (py312), and polars join order audit (py312).

Follow-ups filed

• Shrink the SVAR2 on-disk range cache below 16 bytes/entry #341 — shrink the on-disk SVAR2 range cache below 16 bytes/entry (the real fix for the 98 GiB).
• _write_from_svar (SVAR1) ignores max_mem #342 — _write_from_svar (SVAR1) also ignores max_mem.
• svar2_slice::region_hits rebuilds a VkColumnIndex per (region, column) d-laub/genoray#145 — svar2_slice::region_hits rebuilt a VkColumnIndex per (region, column); pre-existing and read-path only. Filed during this work and since fixed upstream by perf(query): build each SVAR2 search tree once per channel, not once per region d-laub/genoray#149.

🤖 Generated with Claude Code