docs(spec): #334 dataset concatenation design

[d-laub]

Design doc for gvl.concat (issue #334). Spec only — no implementation. Draft pending review of the design before an implementation plan is written.

Summary

#334 asks two things; they get different answers.

Q1 (reuse variant indexing across cohorts sharing a variant space) needs no feature. gvl.write(path, bed, "parent.pgen", samples=cohort_A_samples) already works — write() calls variants.set_samples(samples) (_write.py:307), genoray builds and caches the PGEN index once, and gvl hardlinks it into each dataset. Pre-splitting the PGEN with plink2 --keep is exactly what forces the per-cohort index rebuild the issue reports. This is a docs fix.

Q2 (merging datasets) is the real feature, serving the issue's other two use cases: region-sharded parallel preprocessing, and incremental sample addition.

Design highlights

• One primitive, gvl.concat(path, datasets, axis="regions"|"samples"). Both axes reduce to a single run-coalesced gather over the (R, S[, P]) grid; axis only selects a provenance map.
• Stores split into two kinds by offset semantics: cumulative-into-owned-payload (PGEN/VCF genotypes, intervals) vs absolute-ranges-into-external-store (.svar, .svar2). Both are bulk streaming moves — the .svar/.svar2 range arrays are not small (16–32 GB at R=1000, S=500k, P=2).
• Single-threaded, buffered IO on both sides, 16 MiB chunks, destination-ordered. No np.memmap in the bulk data path.
• All inputs must share one variant source; merging distinct variant tables is out of scope.

Measured, not assumed

Benchmarked on both filesystems, 512 MiB × 3 reps, with posix_fadvise(POSIX_FADV_DONTNEED) evicting sources before every read so reads are genuinely cold. carter-cn-03, Slurm job 13336789.

NFSv3:

pattern	rep1	rep2	rep3
buffered read → buffered write	85.6	102.0	93.0 MB/s
memmap read → buffered write	16.9	19.7	14.4 MB/s
buffered read → memmap write	13.8	15.7	12.6 MB/s
memmap read → memmap write	13.6	14.1	14.4 MB/s

Local XFS (/dev/md0 on /tmp): all four patterns within noise of ~100 MB/s.

So: on NFS any memmap in the path costs ~5–6× (4 KiB fault RPCs instead of the mount's 1 MiB rsize/wsize); on local disk it makes no difference. Buffered IO is the safe unconditional choice.

An earlier revision of this spec reported memmap reads at 316 MB/s and concluded they were fastest. That run never evicted the page cache and its sources had just been written, so it measured RAM. Corrected in the second commit — the finding is the opposite.

Scope limit stated in the spec: these are sequential-streaming numbers and say nothing about the random fancy-indexed access in gvl's read path.

variants.arrow stays a hardlink

The SIGBUS concern that motivated switching to a copy does not apply:

• gvl never mmaps variants.arrow — _haps.py:121 and _haps.py:185 both pass memory_map=False.
• genoray rewrites the PGEN index (_pgen.py:1125) and VCF index (_vcf.py:1095) via atomic_write_path — temp + os.replace, which never touches the old inode.

So the fix is the missing integrity guard, not the aliasing: concat records a bounded blake2b fingerprint and verifies it at open, mirroring the existing svar_link/svar2_link precedent. The spec also documents why fallible-hardlink, catching SIGBUS, and NFS-detection were each rejected.

Related issues filed

• gvl.write hardlinks variants.arrow into the dataset with no integrity guard #337 — gvl.write hardlinks variants.arrow with no integrity guard
• perf: bulk dataset IO should use buffered read/write, not np.memmap — ~5-6x slower on NFSv3 #338 — bulk dataset IO should use buffered read/write, not np.memmap (~5–6× on NFSv3)

Both are boy-scout findings from this design work, out of scope for the eventual implementation PR.

Relates to #334.

🤖 Generated with Claude Code