feat(concat): add gvl.concat for merging datasets along regions or samples

docs/source/api.md

@@ -9,6 +9,8 @@
 
 .. autofunction:: update
 
+.. autofunction:: concat
+
 .. autofunction:: get_splice_bed
 
 .. autofunction:: read_bedlike

docs/source/faq.md

@@ -82,6 +82,15 @@ Both are sparse columnar variant archives from [`genoray`](https://github.com/mc
 
 One documented difference in raw output: for a pure deletion, `with_seqs("variants")` on a `.svar` dataset reports the VCF anchor base as ALT (e.g. `b"G"` for `GTA>G`), while a `.svar2` dataset reports the atomized empty ALT (`b""`) — a genoray `.svar2` format convention, not a bug. Reconstructed haplotypes are unaffected; only `RaggedVariants.alt` differs (and `FlatVariantWindows.alt`/`.alt_window` for `"variant-windows"`), and only for pure-deletion records. `ref_window` is byte-identical between the two backends.
 
+## Can I build a dataset in parallel shards and merge them?
+
+Yes. Split your BED across jobs, `gvl.write` one dataset per shard, then merge with
+[`gvl.concat(out, shards, axis="regions")`](api.md#genvarloader.concat). All shards must be
+written from the same variant source (same PGEN/VCF table, or the same `.svar`/`.svar2` store)
+and have identical samples in identical order. See "Merging datasets" in the
+[write guide](write.md) for the cost model and the `axis="samples"` alternative (merging disjoint
+cohorts over shared regions).
+
 ## How can I get personalized protein/spliced RNA sequences?
 
 Write a dataset from an exon-level BED containing transcript and exon-order columns,

docs/source/format.md

@@ -42,6 +42,7 @@ below for the on-disk size of this cache; it is not small at cohort scale.
 | `version` | `SemanticVersion \| None` | Package version that wrote this dataset. Drives format dispatch. |
 | `svar_link` | `SvarLink \| None` | Back-reference to a source `.svar`, when present. |
 | `svar2_link` | `Svar2Link \| None` | Back-reference to a source `.svar2`, when present. |
+| `variants_fingerprint` | `Fingerprint \| None` | Bounded content fingerprint of `genotypes/variants.arrow` (blake2b over the first 1 MiB + total size). Set only when [`gvl.concat`](api.md#genvarloader.concat) hardlinks `variants.arrow` from a PGEN/VCF-backed input; `None` for datasets written directly by `gvl.write`. `Dataset.open` verifies it against the on-disk file and raises `ValueError` on mismatch (e.g. the source variant index was rewritten out-of-band); absent (`None`) skips the check. |
 
 `SvarLink`:
 
@@ -185,6 +186,7 @@ See the `genvarloader` skill's `.svar2` section for the full narrative and `var_
 | `0.18.0` | Variant coordinates switched to 1-based. |
 | `0.25.0` | `metadata.json` gains `svar_link`; old `genotypes/link.svar` symlink layout deprecated. `Metadata.version` typed as `SemanticVersion` (on-disk JSON unchanged). |
 | `0.37.0` | `metadata.json` gains `svar2_link`; `.svar2` accepted as a `gvl.write` variant source, cached under `genotypes/svar2_ranges/` and read via a read-bound, all-Rust path. |
+| `0.42.0 (unreleased)` | `metadata.json` gains `variants_fingerprint`, set when [`gvl.concat`](api.md#genvarloader.concat) hardlinks `variants.arrow` from an input dataset; `Dataset.open` verifies it and raises on mismatch. |
 
 > **Upgrading legacy datasets.** A dataset written before `0.25.0` that was built from an
 > `.svar` will still open (with a `DeprecationWarning`). Run

docs/source/write.md

@@ -108,3 +108,51 @@ gvl.write(
 Both formats store a back-reference in the dataset's `metadata.json` instead of duplicating per-variant arrays, so the source store must remain accessible when the dataset is later opened with [`gvl.Dataset.open()`](api.md#genvarloader.Dataset.open) (override its location with `svar=`/`svar2=` if it has moved).
 
 `.svar2` additionally produces a write-time cache under `<path>/genotypes/svar2_ranges/` and reads back through an all-Rust, read-bound path with no interval-search-tree build and no dense-union rebuild per read — see [the FAQ](faq.md) for the read-path and on-disk-size tradeoffs, and [the format reference](format.md) for the on-disk layout, including the size formula. This cache is **not small at cohort scale**. `gvl.write` honours `max_mem` when writing `.svar2` genotype ranges, bounding the RAM used while producing the cache; the permanent range cache itself is governed by disk space, not `max_mem` (see the format reference). `.svar2` currently has a Phase-1 scope: a handful of output combinations (`annotated` haplotypes, `min_af`/`max_af`, spliced variant-window/track outputs, etc.) aren't wired yet and raise `NotImplementedError` — see the `genvarloader` skill or the format reference for the full list. Haplotype and `variants` output support splicing and `var_filter="exonic"`.
+
+## Reusing a variant index across cohorts
+
+If several cohorts are sample subsets of one parent PGEN, do **not** pre-split the
+PGEN with `plink2 --keep`. Point `gvl.write` at the parent and pass `samples=`:
+
+```python
+gvl.write("cohort_A.gvl", bed, "parent.pgen", samples=cohort_A_samples)
+gvl.write("cohort_B.gvl", bed, "parent.pgen", samples=cohort_B_samples)
+```
+
+genoray caches the parent PGEN's variant index on disk (`.pvar.gvi`, keyed by the `.pvar`'s
+mtime); `samples=` only subsets at the genotype-extraction level and never touches that index, so
+a second `gvl.write` against the same parent path reuses the cached index instead of rescanning
+the PVAR — and gvl hardlinks that cached index file into each dataset's
+`genotypes/variants.arrow`. Splitting the PGEN first (`plink2 --keep`) produces a distinct `.pvar`
+per cohort, which forces the expensive per-cohort index rebuild.
+
+## Merging datasets
+
+[`gvl.concat()`](api.md#genvarloader.concat) merges datasets that were written from **one shared
+variant source** (the same PGEN/VCF variant table, or the same `.svar`/`.svar2` store — merging
+datasets built from different variant sources raises), along either axis:
+
+```python
+gvl.concat("merged.gvl", ["chr1.gvl", "chr2.gvl"], axis="regions")
+gvl.concat("merged.gvl", ["cohortA.gvl", "cohortB.gvl"], axis="samples")
+```
+
+`axis="regions"` requires identical samples in identical order across every input, and
+concatenates their regions. `axis="samples"` requires identical regions across every input and
+disjoint sample sets, and merges the samples into sorted order. Both axes merge tracks and
+annotation tracks alongside the genotypes, and require at least two input datasets.
+
+`gvl.concat` streams data at the byte level rather than re-deriving anything from the source
+variants, so its cost is dominated by I/O rather than computation. As an order-of-magnitude
+expectation (not a measured figure — this hasn't been benchmarked), it moves roughly the full
+size of the merged dataset at sequential-IO speed, so a merge of a very large dataset (e.g. on the
+order of a terabyte) could plausibly take hours rather than minutes. It's worth doing against the
+alternative of re-extracting genotypes from scratch, not as a routine step.
+
+For contig-sharded `.svar2` workflows there is a cheaper path at the variant-store layer:
+`genoray.SparseVar2.concat(output, sources, mode="copy")` merges disjoint-contig `.svar2` stores
+that share identical samples/ploidy/fields, then a single `gvl.write` call over the merged store
+only needs to populate `.svar2`'s small per-`(region, sample, ploid)` cache ranges rather than
+touch the bulk variant data. This is a different operation from `gvl.concat`, which merges already
+*written* `.gvl` dataset directories — `SparseVar2.concat` merges the upstream `.svar2` stores
+before `gvl.write` ever runs.