feat: declarative edge uuid_fields so edge ids stop drifting

[SkyeAv]

Summary

Let a graph declare which edge fields constitute edge identity, so edge ids stop drifting on attribute-only changes — and make edge dedup key on the derived id, which guarantees uniqueness and cuts the dedup set from 2469 MB to 100 MB on a 1.27M-edge build.

Changes

• What was changed

  uuid_fields (opt-in). Graph gains an optional uuid_fields list naming the fields that identify an edge; only those feed the derived id. Unset preserves today's whole-record hash, so the feature changes nothing for existing configs. A list that cannot be a key — empty, repeating, containing id, or naming a field no edge emits — is rejected at config time as uuid-bad-fields.

  Namespacing. Narrowing the hash inputs removes the accidental cross-graph uniqueness full-record hashing provided, so the UUID domain moves onto rig.source_info.infores_id when uuid_fields is set. New optional uuid_domain overrides it for graphs that must deliberately share an id space (sharded builds, a renamed KG keeping its published ids).

  Three correctness fixes that this design requires:

  • Nested values leaked key order into the hash. Only top-level keys were sorted; nested values reached the digest via Value::to_string, which under serde_json's preserve_order emits insertion order — so a polars struct-field reordering silently re-minted ids graph-wide. Canonicalization now recurses; array order is preserved because it is semantic.
  • false was dropped along with its key while strip_nulls deliberately keeps it, so {subject, negated: false} and {subject} — two distinct records — derived the same id.
  • Dedup keyed on full record bytes, safe only while the id was a pure function of those bytes. stable_json_bytes never sorted despite its name, so two logically identical records arriving with different key order derived one id, produced different bytes, and both shipped.

  Edges now dedup on the derived id. An exact repeat collapses; two genuinely different edges claiming one id raise uuid-fields-not-a-key with the id, the fields that differ, and one offending edge. Nodes are unchanged (their ids are CURIEs, not derived hashes).

  Also: a duplicate-edge-ids assertion in --qc study output (the symmetric partner to duplicate-node-ids, which had no edge equivalent).

• Why this change is needed

  Every field was an identity field. A corrected p_value, a bumped subject_nlp_level inside supporting_text, a changed source_record_urls, or a reordered source row each minted a brand-new id — so downstream Translator consumers saw a new edge where they should have seen the same edge with updated attributes.

• How it affects the project

  Measured on MultiomicsKG 3.0.0 (1,265,355 edges), re-analysing p_value and effect_size on every row:

  	edge ids that changed
  no uuid_fields (today)	930,081 of 1,265,355 — 73%
  uuid_fields declared	0

  Dedup peak RSS on the same graph: 2469 MB → 100 MB (25x), and the pass went from 477s to 37s — the old path's allocation churn dominated, so that margin will be smaller on machines under less memory pressure.

Testing

• Commands run and results

  • ruff check: passed · ruff format --check: 78 files already formatted · pyright: 0 errors
  • pytest: 1105 passed, 15 skipped (94% coverage)
  • cargo test: 89 + 10 + 14 passed, 0 failed · cargo fmt --check: passed · cargo clippy --all-targets -- -D warnings: clean

  Note: make check was run stage by stage rather than as one target — this machine's rustup toolchain ships a broken bundled linker wrapper (a dangling /nix/store path), so every cargo invocation needed RUSTFLAGS="-C link-self-contained=-linker -C link-arg=-fuse-ld=bfd". Unrelated to this change; CI is unaffected.

• Unit tests added/updated

  Rust: golden vectors pinning three concrete UUIDs (nothing pinned an actual value before, so the encoding could change silently); nested key-order stability; array order still mattering; false staying distinguishable; declared fields ignoring undeclared edits while tracking declared ones; declared-field order not mattering; cross-domain separation; the not-a-key abort and its diagnostic; exact repeats still collapsing; key-order-only duplicates no longer shipping two lines.

  Python: Graph validation of uuid_fields (casing, empty, duplicate, id, unknown field), uuid_namespace across all four unset/set combinations, dedup_stream plumbing, and the duplicate-edge-ids study assertion.

• Integration tests added/updated — none; covered by the real-data run below.

• Manual testing performed

  Ran the deduper over the published 1.27M-edge MultiomicsKG 3.0.0 edges file with ids stripped back out, in four configurations. Output was 1,265,355 edges with 1,265,355 unique ids in every case.

  Worth flagging: the field set the docs suggest as a starting point is not a key for this graph — 2,476 collisions (0.2%), pairs whose subject, predicate, object, publication and row were identical and differed only in the NLP level in supporting_text, because two raw strings had resolved onto the same CURIE. The guard caught all of them. Adding supporting_text made it a key. The docs now say to build and let the failure name what is missing, rather than presenting a field list as an answer.

Related Issues

• None

Checklist

• Tests pass
• Code style compliant
• Documentation updated
• No breaking changes

Breaking output change. Recursive canonicalization changes every existing edge id, once: every edge carries a sources array whose sorted key order differs from its insertion order. There is no old-id → new-id mapping; the old value was a hash of a serialization detail.

Migration: readers pinning Tablassert edge ids must re-key against the rebuilt graph. Adopt uuid_fields in the same rebuild and ids stop moving for attribute changes thereafter. The false fix moves nothing in practice (MultiomicsKG 3.0.0 contains zero false values).

Release is deliberately deferred

pyproject.toml stays at 15.1.0 in this PR. A version change landing on main is what triggers tag-version.yml and pipy.yml, so merging this does not tag or publish anything. The ## 16.0.0 CHANGELOG entry is written and ready; cut the release as a separate deliberate chore(release): 16.0.0 commit bumping pyproject.toml and CITATION.cff in lockstep.

Sizing is unambiguous when you do: this is a major — the edge id is a public contract and every one of them moves.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NuGRZ2fn3gE5zkUrv56Jqh

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5488cda0-d319-47ca-a0fc-73a3dba367e0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}