[bjacobso]

Why

WorldVM's programmable-operations demo runs Triplex core in memory, in the browser. Committing a large account was the bottleneck. On main it costs 230–550 µs and about 5.2 KB of memory per fact. Loading 10,000 workers (160k facts) in one transaction runs out of a 4 GB heap. In chunks it takes 41 s.

This PR measures that path, makes ordinary writes faster, and then prototypes the git-shaped design:

• build an immutable, indexed, content-addressed segment in bulk;
• attach it in one small transaction;
• publish by moving a named ref to a commit position.

Headline numbers (160,000 facts)

Node 24, 4 GB heap, 4 vCPU Linux VM. "Read" is 25 match calls (every attribute) at one validAt. "Memory" is the V8 heap plus array buffers after GC. Full tables for 16k, 80k and 160k facts are in test/stress/README.md; reproduce them with pnpm --filter triplex-stress benchmark:memory.

Path	Build (can run off-thread)	Commit	First read at ref	Read	Memory/fact
main, one transact		out of memory
main, 4,000-fact transacts		40.7 s	186 ms	1,495 ms	5.2 KB
this PR, one transact		9.7 s	172 ms	1,699 ms	5.3 KB
this PR, 4,000-fact transacts		6.8 s	147 ms	1,545 ms	5.0 KB
Import.run, summary journal + ref		2.8 s	105 ms	1,450 ms	2.5 KB
Segments, built and attached + ref	650 ms	34 ms	7.5 ms	93 ms	80 B
Prebuilt segment image + ref		13 ms	7.0 ms	85 ms	79 B

At 16k facts, segments build in 100 ms, attach in 13 ms and read everything in 17 ms. Memory per fact is about 65 times lower, and reads are about 16 times faster, because segment reads binary-search typed arrays instead of decoding META records.

Commits

Step 1: ordinary writes, read cuts, refs

1. Benchmark with a main baseline.
2. recordedPosition in the public TemporalBasis. A read at a commit position sees exactly the transactions committed at or before it, which stays exact when commits share a millisecond. Paged reads clamp it to the latest commit, and derivation runs persist it.
3. Batched KV transaction writes. No semantic change: one assertBatch per transaction (flushed before anything that reads), string-keyed bulk buffering, and an incrementally merged sorted key index. This is the 41 s → 6.8 s row.
4. PositionRef (@triplex-build/triplex/operational). A named pointer to a committed position. Each move is one journaled transaction with compare-and-retract, an optional expected old position, and an optional manifest. Moves past the latest commit are rejected. basis() resolves it once into { recordedPosition, validAt }.
5. journal: "summary". One :_tx/summary (counts, attributes, content digest) instead of one :_tx/change per fact.
6. Import.run. Chunked commits that yield to the host between chunks, then publish by moving the ref.

Step 2: segments

7. Faster SHA-256 (2.7×, about 55 MB/s) plus an incremental Sha256.create() hasher.
8. Segment. A pure, worker-safe build of a sorted dictionary, typed-array columns in EAVT order, and permutation indexes (AEVT, AVET, VAET, entity type). The ID is ContentId("triplex/segment", canonical JSON of the facts). encode and decode produce a single buffer whose columns are viewed in place.
9. Attach and merge-on-read behind the unchanged Triples contract, plus Import.run(..., { store: "segments" }), tests and docs/segments.md.
10. Benchmark modes for segments and prebuilt images.

Follow-ups

11. kv: answer entity-type matches at a temporal basis from the TYPE index #46 cherry-picked (entity-type matches at a temporal basis read the TYPE index). Segments join that path through their type index, with the same visibility check. A segment fact retracted later stays visible at an earlier basis; a new test fails without this.
12. Sorted-key merges use binary search when a commit adds few keys to a large partition, copying the runs between them without comparing. A read after a small write at 160k facts was spending about 35 ms per touched index on string comparisons.
13. transactionsForEntity scopes attach records to the entity. An attach record lists only that entity's facts from the segment, so history pages stay small however large the segment is. transaction(txId) still derives every change.

Semantics and tests

• Segment reads. match, entity, entities, entityPage, get, history, dependencyState and Datalog see attached segments merged with transacted facts:

  • at recorded instants and positions before and after the attach;
  • at valid times either side of closed intervals;
  • by attribute, attribute+value, ref target and entity type;
  • in joins with transacted facts, with Datalog binding the attaching transaction.

  Datalog's ?tx binds the attaching transaction.

• Retracting a segment fact works and stays in history:

  • it is visible before the retraction's position, hidden after;
  • history shows it retracted, and the retraction's journal entry lists it;
  • retracting it twice is a no-op.

• Journal.

  • transactions() returns one attach record (summary with segments: [id], digest equal to the segment ID, no changes).
  • transaction(txId) derives all per-fact changes from the segment.
  • transactionsForEntity merges attachments containing the entity with indexed transactions, including paging.
  • Command IDs are idempotent across attaches.

• Refs and imports (both store: "transactions" and "segments"):

  • readers at the ref see nothing partial, checked after every chunk of a second generation;
  • moving the ref is exactly one transaction;
  • concurrent and stale moves conflict;
  • reading at the previous ref position is unchanged.

• Content addressing.

  • The same facts in any order give the same ID.
  • The ID equals ContentId.hash over CanonicalJson of the facts (cross-checked against the generic encoder).
  • decode(bytes, { verify: true }) rejects tampered images.
  • Randomized lookups match a naive filter on every access path.

• Gating. Attaching on SQLite fails with WriteError and commits nothing. Reserved namespaces are rejected.

The step 1 cases are in the shared conformance corpus, so they run on in-memory KV, SQLite, Cloudflare (Node harness and workerd) and PostgreSQL in CI. Segment tests are in-memory only, as the feature is.

Checks: pnpm check is green, and every commit typechecks and passes the core suite on its own.

Not run locally:

• pnpm test:postgres:integration (no Docker here; CI runs it);
• pnpm pack:check;
• pnpm test:e2e.

Timeouts and flakes. I raised the whole-corpus conformance tests' timeouts to 30 s, as the testkit already does: the corpus grew, and those tests crossed Vitest's 5 s default under pnpm check's parallel load. One run hit the existing timing-sensitive Reactor "stays linear" test under load; it passes alone and on rerun.

Design trade-offs

Details are in docs/segments.md.

• Segment format. Columns over a sorted dictionary make comparisons integer comparisons, and keep a fact at about 60 bytes of columns plus its share of the dictionary. Indexes are permutations, so they're derivable and rebuildable. The binary image is the columns 8-byte aligned, so attaching a shipped dataset costs a decode of the dictionary JSON and no sorting.
• Recorded time comes from the attach, not the segment. The same segment attached twice is two attachments with different positions and triple IDs. Valid time stays data.
• Merge-on-read cost is O(attachments × log n) plus the matching facts. Segment facts are decoded per read (no cache, so memory stays flat). Chunked imports create one attachment per chunk; repacking (not built) would merge them while keeping each fact's attachment as a column.
• Journal shape. The attach envelope keeps actor, command, correlation, causation and position, plus a summary naming the segment.
  • Per-fact changes and the entity index are derived from the segment on demand rather than stored.
  • What's lost: the journal alone can't rebuild the facts (the body must survive), and Datalog over :_tx/change or :_tx/changed-entity doesn't see attached facts.
  • Derivation relevance uses the summary's attributes; emission and snapshots use the derived changes.
• Ref semantics.
  • Resolve once, read many: a per-read { ref } field could straddle a move.
  • Positions are a total commit order, so a published position is a consistent cut, and it includes other writers' commits up to that point.
  • Moving backwards is a rollback.
• What a position basis exposes. Commit positions were already public, as TransactionResult.position and the feed cursor. A position past the latest commit reads the latest state, so only committed positions are stable; refs only ever hold committed ones.
• Other backends.
  • FoundationDB/KV: chunked body keys plus the same attachment record and overlays.
  • SQL: a segments table, with the attach expanding into triples rows by bulk insert while keeping the attach-shaped journal.
  • Both are designed only; they currently fail with WriteError.

What didn't pan out, or isn't done

• Making transact itself cheap enough. Batching gave 6× on time, but memory stays about 5 KB per fact with the full journal (2.5 KB with the summary journal). Per-fact string keys, serialized META and the decoded-datom cache are the floor of the KV emulation. Segments are the answer for bulk data, not a faster transact.
• Content hashing dominates segment build time (about half of the 650 ms). crypto.subtle.digest would be far faster but is asynchronous, and build is synchronous by design. An async build variant is the obvious next step.
• Not built:
  • a worker helper (the build is worker-safe, but hosts own the plumbing);
  • repacking;
  • segment storage for FoundationDB/SQL;
  • constraint enforcement on attaches (rejected explicitly).
• The heap figures in my first PR description undercounted. Typed-array backing stores sit outside heapUsed, so all rows now count array buffers. That raised main from 4.6 to 5.2 KB per fact.
• PR kv: answer entity-type matches at a temporal basis from the TYPE index #46 is cherry-picked here, and in ConfigStore: read only what each operation needs #48 (ConfigStore targeted reads). Whichever merges first, the others rebase cleanly.
• WorldVM pins Effect 4.0.0-rc.112 and won't consume this until its Effect upgrade lands.

🤖 Generated with Claude Code