Repository navigation
Multi-tab Electric: a deleted row returns after a peer reset and deploys #2281
Description
Summary
The multi-tab Electric oracle found a history where a deleted row returns. After every tab catches up, durable storage and both tabs still contain row 1. The model expects only row 2.
The failure reproduces on origin/main at 4c39add and on the head of #2262 (f276b4a). The oracle and the behavior it checks came in with #2265. #2262 does not cause it.
Law
A row that the server deletes stays deleted in durable storage and in every tab after all tabs catch up. A peer reset and a later deploy do not bring back a deleted row.
Counterexample
Oracle: packages/browser-db-sqlite-persistence/tests/multi-tab-electric-oracle.test.ts, test keeps reserved commits lawful across generated multi-tab histories (random or replayed), property browser.multi-tab-reserved-history.
{ seed: -1115798760, path: "1:4:3:3:3:3:3:3:3:3:3", endOnFailure: true }
Counterexample: [{"tabCount":2,"steps":[
{"kind":"server change","key":1,"pick":0},
{"kind":"advance","tab":1,"pick":0},
{"kind":"server change","key":1,"pick":0},
{"kind":"advance","tab":0,"pick":0},
{"kind":"peer reset"},
{"kind":"server change","key":1,"pick":0},
{"kind":"deploy"},
{"kind":"deploy"},
{"kind":"server change","key":1,"pick":0},
{"kind":"server change","key":1,"pick":0}]}]
Shrunk 10 time(s)
Observation after every tab caught up:
actual {"durableRows":[{"id":1,"label":"one"},{"id":2,"label":"two"}],
"tabs":[{"tab":"A","status":"ready","rows":[{"id":1,...},{"id":2,...}],"sourceError":true},
{"tab":"B","status":"ready","rows":[{"id":1,...},{"id":2,...}],"sourceError":true}]}
expected {"durableRows":[{"id":2,"label":"two"}],
"tabs":[{"tab":"A","status":"ready","rows":[{"id":2,...}],"sourceError":true},
{"tab":"B","status":"ready","rows":[{"id":2,...}],"sourceError":true}]}
The CI job that found it: Test (sqlite-adapters) on #2262, run 40723478437.
Replay
Run this in packages/browser-db-sqlite-persistence:
TANSTACK_DB_ORACLE_PROPERTY=browser.multi-tab-reserved-history \
TANSTACK_DB_ORACLE_SEED=-1115798760 \
TANSTACK_DB_ORACLE_PATH="1:4:3:3:3:3:3:3:3:3:3" \
npx vitest --run tests/multi-tab-electric-oracle.test.ts -t "generated multi-tab"Related
- Electric resume check puts a lagging tab in error when a peer deleted the row it updates #2267 is a different symptom: a lagging resumed tab goes into error. This issue is a deleted row that returns.
- The fixed-seed campaign and the pinned storm histories pass. Only the random campaign reached this history.
Next steps
- Diagnose which step reintroduces row 1: the peer reset, the deploys, or the reserved-commit replay.
- Record the trace in
docs/contributing/oracle-coverage.mdunder the multi-tab Electric owner until a fix lands. - Fix oracle-first: keep this history as a replay, show RED on the parent, GREEN after the fix, and a hostile mutant that fails.
Activity
test(persistence): pin two known gaps that random campaigns found
KyleAMathews commentedon Oct 28, 2026
Two more random-campaign counterexamples reproduce on
main(45eeef8). PR #2231 pins both withit.failsreplays that run through each oracle's own driver and checks (commit b027969).1. Multi-tab Electric: a stored row is lost after the tabs stop
This is the same property as this issue,
browser.multi-tab-reserved-history. It is a different history, and the symptom is the opposite: a row that should be stored is missing, where in this issue a deleted row comes back. The history has no peer reset and no deploy.TANSTACK_DB_ORACLE_PROPERTY=browser.multi-tab-reserved-history \ TANSTACK_DB_ORACLE_SEED=-1660880085 \ TANSTACK_DB_ORACLE_PATH=55:0:1:4:4:4:1:1:1:1:3:4:4:4:4 \ pnpm --filter @tanstack/browser-db-sqlite-persistence exec vitest --run tests/multi-tab-electric-oracle.test.tsTwo tabs: server change k1, tab 0 catches up, server change k1, tab 0 catches up, tab 1 catches up, tab 0 catches up, tab 1 stops, tab 0 catches up, tab 0 stops. After every tab catches up:
2. Persisted sync metadata session: a reload keeps a seed's metadata
The property is
sync-metadata-session.persisted-peers-full, seed654774679. It failed CI on #2231. This is a different subsystem from the multi-tab case and is recorded here only to keep track of it.TANSTACK_DB_ORACLE_PROPERTY=sync-metadata-session.persisted-peers-full \ TANSTACK_DB_ORACLE_SEED=654774679 \ pnpm --filter @tanstack/db-sqlite-persistence-core exec vitest --run tests/sync-metadata-session-persisted.test.tsThe shrunk prefix is
begin seed(1,bare) insertOverSeed(1) begin reinsert(1) commit peer(2,reload). It breaks the law that a source reads its own row metadata (metadata.row.get):Neither case shares a proven root cause with this issue. The random campaigns can still draw both histories until they are fixed.
fix(db): keep a live query running when a lazy demand fails during a …