feat: declarative edge uuid_fields so edge ids stop drifting

CHANGELOG.md

@@ -2,6 +2,31 @@
 
 All notable changes to this project are documented in this file.
 
+## 16.0.0 - 2026-08-25
+
+### Breaking Changes
+- **Edge ids are canonicalized before hashing, so every existing edge id changes once.** The derivation sorted only *top-level* keys: nested values reached the digest through `Value::to_string`, which under serde_json's `preserve_order` emits insertion order. Every edge carries a `sources` array, so a polars struct-field reordering — or a Biolink release that shifts a nested slot — silently re-minted ids across a whole graph while nothing about the assertion had changed. Canonicalization now recurses (object keys sorted at every depth; array order preserved, because it is semantic), which makes ids robust to that churn at the cost of one migration. Relatedly, `false` is now hashed as `"false"` instead of being dropped along with its key: `strip_nulls` deliberately keeps `false` (`negated: false` is a meaningful Biolink value), so dropping it made `{subject, negated: false}` and `{subject}` — two distinct records — derive the *same* id. No shipped graph is affected by that second fix (the 1.27M-edge MultiomicsKG 3.0.0 build contains zero `false` values); the canonicalization fix moves every id. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
+  **Migration:** readers pinning Tablassert edge ids must re-key against the rebuilt graph. There is no mapping from old id to new — the old value was a hash of a serialization detail. This is a one-time event: adopt `uuid_fields` (below) in the same rebuild and ids stop moving for attribute changes thereafter.
+
+### Added
+- **`uuid_fields`: a graph config declares which edge fields constitute edge identity.** Edge ids were UUIDv3 hashes of the *entire* emitted record, so every field was an identity field — a corrected `p_value`, a bumped `subject_nlp_level` inside `supporting_text`, a changed `sources[].source_record_urls`, or a reordered source row (the `row:<N>` inside `has_supporting_studies`) each minted a brand-new id, and downstream Translator consumers saw a new edge rather than the same edge with updated attributes. `Graph` gained an optional `uuid_fields` list; when set, only those top-level edge keys feed the hash, and everything else is free to change. Entries canonicalize onto their allow-listed spelling exactly as annotations do, and a list that cannot be a key — empty, repeating, containing `id`, or naming a field no edge emits — is rejected at config time as `uuid-bad-fields`. Leaving `uuid_fields` unset preserves the whole-record hash, so the feature is strictly opt-in. Measured on MultiomicsKG 3.0.0 (1,265,355 edges): re-analysing `p_value` and `effect_size` on every row moved **930,081 ids (73%)** under the whole-record hash and **none at all** under a declared `uuid_fields`. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
+- **The UUID namespace derives from the graph's infores when `uuid_fields` is declared.** Full-record hashing gave cross-graph uniqueness by accident: two graphs asserting the same triple from the same publication were separated by their differing `sources` and `supporting_text`. A narrow field set removes that accident, so the domain moves from the `TABLASSERT` constant onto `rig.source_info.infores_id`, making the separation structural instead. A new optional `uuid_domain` overrides it for the opposite case — graphs that must deliberately *share* an id space, such as a KG compiled in shards or renamed across versions while keeping its published ids. Both default to the historic constant when `uuid_fields` is unset. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
+- **A `duplicate-edge-ids` assertion in `--qc` study output.** `study_kgx` checked `duplicate-node-ids` but had no edge equivalent, even though KGX requires edge ids to be unique. The symmetric check now streams the final edges file and reports duplicates independently of the writer, catching an id collision from any source. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
+### Fixed
+- **Two edges deriving one id abort the build instead of shipping a duplicate.** Dedup keyed on the full canonical record bytes, which was safe only because the id was a pure function of those bytes. That equivalence breaks the moment the hash covers a subset, and it was already imperfect: `stable_json_bytes` never sorted despite its name, so two logically identical records arriving with different key order derived one id, produced different bytes, and *both* shipped. Edges now dedup on the derived id itself — an exact repeat collapses as before, while two genuinely different edges claiming one id raise `uuid-fields-not-a-key` with the id, the fields that differ (recovered by re-reading the partial output, on the failure path only), and the declared field list. The nodes path is unchanged: node ids are CURIEs, not derived hashes. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
+### Performance
+- **Edge dedup holds 24 bytes per edge instead of a full copy of every record: 2469 MB -> 100 MB peak on a 1.27M-edge graph.** `record_if_new` retained the complete bytes of every unique record to make suppression byte-exact — roughly 800 bytes per edge, and ~2.4 GB resident on MultiomicsKG 3.0.0 purely for the dedup set. The edge path now keys a `FxHashMap<[u8; 16], u64>` on the raw UUID bytes with an xxh64 of the id-free record as the value: a measured **25x** reduction that also removes the per-record heap allocation, and on the same graph took the dedup pass from 477s to 37s (the old path's allocation churn dominated; the margin will be smaller on machines under less memory pressure). The content hash is computed *before* the id is inserted, so no record is cloned to strip it back out, and a declared `uuid_fields` list additionally cuts the number of keys the hasher visits per edge. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
+### Changed
+- **`docs/api/utils.md` now describes the derivation that actually runs.** The page documented a `"\t".join(values)` encoding removed some releases ago (the join has been length-prefixed for collision-safety) and claimed edge ids covered only "subject, predicate, object, qualifiers, and publication" — which was never true, and is precisely what `uuid_fields` now makes achievable. Both are corrected, and the canonicalization, namespacing, and uniqueness rules are documented alongside. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
+- **Golden UUID vectors pin the derivation.** No test asserted a concrete UUID value, so any change to the encoding could land silently. Three vectors now fix the full-record, declared-field, and raw-parts derivations; if one moves, edge ids in every published graph moved with it. ([#122](https://github.com/SkyeAv/Tablassert/pull/122))
+
 ## 15.1.0 - 2026-08-25
 
 ### Added

docs/api/utils.md

@@ -54,7 +54,7 @@ Domain string used to create the namespace UUID. The default domain used interna
 
 **`values: list[str]`**
 
-The values to incorporate into the UUID. Empty/None entries are filtered out, the rest are joined with tabs (`"\t"`) and hashed within the domain namespace.
+The values to incorporate into the UUID. Empty entries are dropped; each surviving value is length-prefixed as `<byte-length>:<value>` and concatenated, then hashed within the domain namespace.
 
 ### Return Value
 
@@ -68,38 +68,70 @@ Returns a string representation of a UUID v3: `"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
 domain_uuid = uuid3(UUID("00000000-0000-0000-0000-000000000000"), domain)
 ```
 
-**Step 2:** Join the (filtered) values with tabs and hash within the namespace:
+**Step 2:** Length-prefix each value and hash the concatenation within that namespace:
 
 ```python
-return str(uuid3(domain_uuid, "\t".join(values)))
+joined = "".join(f"{len(v.encode())}:{v}" for v in values)
+return str(uuid3(domain_uuid, joined))
 ```
 
+The length prefix is what makes the encoding **injective**. A plain separator join is ambiguous
+whenever a value contains the separator: `["a", "x\tb", "y"]` and `["a", "x", "b", "y"]` both join
+to `"a\tx\tb\ty"`, so two different inputs would derive the same UUID.
+
 ### Deterministic Behavior
 
 Same inputs always produce the same UUID; different inputs (or different domains) produce different UUIDs.
 
-### Use Case: KGX Edge IDs
+## Edge IDs
 
-Tablassert uses this function to generate reproducible edge identifiers from the edge's subject, predicate, object, qualifiers, and publication:
+Edge ids are **not** built by calling `namespace_uuid()` from Python. They are assigned inside the
+Rust deduper (`dedup_ndjson`) as each edge is written, from the record itself.
 
-```python
-from tablassert.rs import namespace_uuid
+### Which fields feed the id
+
+By default, **every field** of the emitted edge. That makes the id maximally sensitive: a corrected
+`p_value`, a new `supporting_text` entry, a reordered source row, or a Biolink release that renames
+a slot all mint a brand-new id, and downstream consumers see a new edge rather than an updated one.
 
-edge_id = namespace_uuid(
-  "TABLASSERT",
-  ["HGNC:11998", "biolink:associated_with", "MONDO:0005148", "PMC11708054"],
-)
-# e.g. "2cfea591-0f8f-33af-a7df-03da531d3359"
+A graph config can instead declare which fields constitute edge *identity*:
+
+```yaml
+# graph.yaml
+uuid_fields: [subject, predicate, object, publications, has_supporting_studies]
 ```
 
-**Benefits:**
-- **Reproducible:** the same edge always gets the same ID across runs
-- **Collision-resistant:** MD5 hashing makes collisions extremely unlikely
-- **Traceable:** the ID incorporates the edge components (subject, predicate, object, provenance)
+Only those fields then feed the hash, so attribute-only changes leave the id alone. See
+[Graph Configuration](../configuration/graph.md#stable-edge-ids) for how to choose a field set.
+
+### Canonicalization
+
+Before hashing, the record is normalized so that only *meaning* reaches the digest:
+
+- object keys are sorted, recursively, so insertion order never changes the id;
+- array order is preserved, because it is semantic;
+- each key and its value are fed as **separate** parts, so the key/value boundary cannot shift to
+  create a collision (`{"a": "b=c"}` and `{"a=b": "c"}` stay distinct);
+- `null` and empty values drop out, key included; `false` is hashed as `"false"`, because
+  `negated: false` is a meaningful Biolink value.
+
+### Namespace
+
+The domain defaults to `"TABLASSERT"`. When `uuid_fields` is declared it becomes the graph's
+`rig.source_info.infores_id`, so two graphs asserting the same triple can never mint the same id —
+the uniqueness that full-record hashing provided by accident becomes structural. `uuid_domain`
+overrides it for graphs that must deliberately share an id space.
+
+### Uniqueness
+
+Edges deduplicate on their derived id, so an output file can never carry the same id twice. An
+exact repeat collapses; two genuinely different edges deriving one id abort the build with
+`uuid-fields-not-a-key`, naming the fields that would disambiguate them.
 
 ### KGX Compliance
 
-NCATS Translator KGX requires edge IDs to be globally unique and, where possible, deterministic. `namespace_uuid()` satisfies both: UUID v3 with domain namespacing yields unique, reproducible identifiers.
+NCATS Translator KGX requires edge IDs to be globally unique and, where possible, deterministic.
+UUID v3 with domain namespacing satisfies both.
 
 ## Next Steps
 

docs/configuration/graph.md

@@ -27,6 +27,107 @@ QC auditing and verbose logging are controlled at build time via the `build-kg -
 
 The legacy top-level RIG fields (`description`, `contributions`, `ui_explanation`, `infores`) are **rejected** with a migration pointer; they now live under `rig:`.
 
+### Optional Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `uuid_fields` | List[str] | Edge fields that constitute edge identity. Only these feed the derived edge `id` (see [Stable edge ids](#stable-edge-ids)) |
+| `uuid_domain` | String | Explicit UUID namespace. Defaults to `rig.source_info.infores_id` when `uuid_fields` is set, `TABLASSERT` otherwise |
+
+## Stable edge ids
+
+Every edge gets a deterministic `id`: a UUID v3 derived from the edge itself. By default it is
+derived from the **whole record**, which makes it maximally brittle — a corrected `p_value`, a new
+`supporting_text` entry, a reordered source row, or a Biolink release that renames a slot all mint a
+brand-new id. Downstream Translator consumers then see a new edge where they should see the same
+edge with updated attributes.
+
+`uuid_fields` fixes that by naming the fields that actually identify an edge:
+
+```yaml
+uuid_fields: [subject, predicate, object, publications, has_supporting_studies]
+```
+
+Everything else is then free to change without moving the id.
+
+### Choosing a field set
+
+Start from what identifies an assertion, and what each entry buys you:
+
+- **`subject` / `predicate` / `object`** — the assertion itself.
+- **`publications`** — the evidence it rests on.
+- **`has_supporting_studies`** — carries `has_study_results[].id` (`row:<N>`), the **row
+  discriminator**. Include it whenever one table contributes several rows that share a subject,
+  predicate, and object. It also carries the `study_*` metadata, but those come from per-section
+  config and are far more stable than `p_value` or `effect_size`.
+
+Add qualifiers (`object_direction_qualifier`, `anatomical_context_qualifier`, …) when they
+*distinguish* assertions rather than merely describe them.
+
+Leave out anything that is an observation *about* the edge rather than the edge's identity:
+`p_value`, `effect_size`, `effect_type`, `original_subject` / `original_object`, `supporting_text`,
+`sources`, `category`, `knowledge_level`, `agent_type`.
+
+**Then build, and let the failure tell you what is missing.** That list is a starting point, not an
+answer — whether it is a key depends on your data, and the only way to find out is to run it. On a
+real 1.27M-edge graph the set above left 2,476 collisions (0.2% of edges): pairs whose subject,
+predicate, object, publication and row were identical, differing only in the NLP level recorded in
+`supporting_text` because two raw strings had resolved onto the same CURIE. Adding
+`supporting_text` made it a key.
+
+Expect to iterate once or twice. Each failure names the id, the fields that differ, and one
+offending edge, so each round is mechanical.
+
+### What it buys
+
+On that same 1.27M-edge graph, re-analysing the statistics (new `p_value` and `effect_size` on
+every row) and rebuilding:
+
+| | edge ids that changed |
+|---|---|
+| no `uuid_fields` (whole-record hash) | 930,081 of 1,265,355 — **73%** |
+| `uuid_fields` declared | 0 of 1,265,355 — **none** |
+
+### The field set must be a key
+
+Narrowing what feeds the hash means two different edges can derive the same id. Tablassert refuses
+to ship duplicate edge ids, so that is a build failure, not a silent collapse:
+
+```
+uuid-fields-not-a-key: declared uuid_fields are not a key for this graph.
+  id 83ade536-9b07-34ec-a2f9-abf0fb5b6a2f is claimed by 2 different edges.
+  they differ in: effect_size, p_value
+  declared uuid_fields: subject, predicate, object
+  offending edge: subject=A predicate=r object=B
+Add a discriminating field to `uuid_fields` (...).
+```
+
+The fix is whatever the message names: add the qualifier that separates them,
+`has_supporting_studies` for the source row, or the statistic that genuinely differs. Two rows with
+an identical subject, predicate, and object that differ only in `p_value` are exactly this case —
+and were previously producing two ids for what config claimed was one assertion.
+
+An exact duplicate is *not* a violation: identical edges collapse, as they always have.
+
+### Namespacing
+
+Because a narrow field set no longer distinguishes graphs by accident, declaring `uuid_fields`
+moves the UUID namespace onto the graph's own `rig.source_info.infores_id`. Two graphs asserting
+the same triple from the same publication then still derive different ids, structurally.
+
+Set `uuid_domain` only when graphs must deliberately **share** an id space — a KG compiled in
+shards, or one renamed across versions that has to keep its published ids:
+
+```yaml
+uuid_domain: infores:multiomicskg
+```
+
+### Migration
+
+Adding `uuid_fields` to an existing graph **changes every edge id in it, once**. That is the cost of
+switching identity models; ids are stable from then on. Plan it as a deliberate version bump and
+tell your consumers.
+
 ### The `rig:` section
 
 The `rig:` section carries every human-authored RIG fact. Its shape mirrors the released [RIG schema](https://github.com/biolink/resource-ingest-guide-schema), so the generated `.RIG.yaml` is always schema-shaped. The generator derives only mechanical facts from the build (generated artifact file entries, observed edge/node type summaries) and **validates the complete document before writing anything**, so a build never leaves behind an invalid or incomplete RIG.

rust/src/json.rs

@@ -65,10 +65,40 @@ pub fn strip_nulls(value: &Value) -> Value {
     }
 }
 
-pub fn stable_json_bytes(value: &Value) -> serde_json::Result<Vec<u8>> {
+/// Serialize in the record's own key order.  This is the EMITTED form -- what actually
+/// gets written to the NDJSON -- so it must not reorder anything.  It is deliberately not
+/// a canonical form: use `canonical_json_bytes` when comparing two records for equality.
+pub fn emitted_json_bytes(value: &Value) -> serde_json::Result<Vec<u8>> {
     serde_json::to_vec(value)
 }
 
+/// Serialize with every object's keys sorted, recursively.
+///
+/// `serde_json` is built with `preserve_order`, so plain `to_vec` leaks insertion order:
+/// two logically identical records that arrived with different key order produce
+/// different bytes.  The edge deduper compares records for equality, so it needs a form
+/// where "same content" means "same bytes"; array order is preserved because it is
+/// semantic.
+pub fn canonical_json_bytes(value: &Value) -> serde_json::Result<Vec<u8>> {
+    serde_json::to_vec(&canonical_value(value))
+}
+
+fn canonical_value(value: &Value) -> Value {
+    match value {
+        Value::Object(entries) => {
+            let mut keys: Vec<&String> = entries.keys().collect();
+            keys.sort_unstable();
+            let mut sorted: Map<String, Value> = Map::with_capacity(entries.len());
+            for key in keys {
+                sorted.insert(key.clone(), canonical_value(&entries[key]));
+            }
+            Value::Object(sorted)
+        }
+        Value::Array(items) => Value::Array(items.iter().map(canonical_value).collect()),
+        _ => value.clone(),
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::strip_nulls;