Follow-up from #333.
gvl.write with a .svar2 source writes two permanent (regions, samples, ploidy, 2) int64 caches under genotypes/svar2_ranges/:
2 x regions x samples x ploidy x 2 x 8 bytes
That is ~98 GiB for ~4,000 regions over 414,830 diploid samples — now documented in docs/source/format.md and skills/genvarloader/SKILL.md, and logged with a free-disk warning by _svar2_preflight at write time.
#333 bounded the memory used to produce this cache, but not the cache itself. The obvious shrink is to store start: int64 + len: int32 (12 B/entry) instead of (start, end) as two int64s (16 B/entry) — a 25% cut — or narrower still if the offsets can be made relative.
This is an on-disk format change, so it needs:
- read-path changes in the SVAR2 dataset-open path,
- a format version bump and compat handling for existing datasets,
docs/source/format.md updated with the new layout and formula.
Deferred from #333 deliberately: that PR was scoped to stopping the OOM.
Follow-up from #333.
gvl.writewith a.svar2source writes two permanent(regions, samples, ploidy, 2)int64 caches undergenotypes/svar2_ranges/:That is ~98 GiB for ~4,000 regions over 414,830 diploid samples — now documented in
docs/source/format.mdandskills/genvarloader/SKILL.md, and logged with a free-disk warning by_svar2_preflightat write time.#333 bounded the memory used to produce this cache, but not the cache itself. The obvious shrink is to store
start: int64+len: int32(12 B/entry) instead of(start, end)as two int64s (16 B/entry) — a 25% cut — or narrower still if the offsets can be made relative.This is an on-disk format change, so it needs:
docs/source/format.mdupdated with the new layout and formula.Deferred from #333 deliberately: that PR was scoped to stopping the OOM.