Serialize database destruction with concurrent opens - #787
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces robust database lifecycle management for RocksDB JS bindings. It implements a timed-wait mechanism (lifecycleWaitSeconds) for open, destroy, and shutdown operations to prevent concurrent lifecycle conflicts. It also introduces a "quarantine" state for database paths when a native close, flush, compaction, or physical directory cleanup fails, preventing subsequent opens until the cleanup is retried via destroy() or shutdown(). Additionally, it ensures that in-flight operations (like backups and checkpoints) are safely awaited before destruction, and that thread-affine N-API references are cleaned up safely. There are no review comments, so I have no feedback to provide.
📊 Benchmark Resultsget-sync.bench.tsgetSync() > random keys - small key size (100 records)
getSync() > sequential keys - small key size (100 records)
ranges.bench.tsgetRange() > small range (100 records, 50 range)
realistic-load.bench.tsRealistic write load with workers > write variable records with transaction log
transaction-log.bench.tsTransaction log > read 100 iterators while write log with 100 byte records
Transaction log > read one entry from random position from log with 1000 100 byte records
worker-put-sync.bench.tsputSync() > random keys - small key size (100 records, 10 workers)
worker-transaction-log.bench.tsTransaction log with workers > write log with 100 byte records
Results from commit 347546f |
|
Reviewed Re-review of the one new commit since
Also checked and cleared: the dropped Verification: full suite 55 files, 773 passed / 1 skipped / 0 failed; targeted destroy + ranges 64/64 including the new One merge-ordering note, not a defect in this PR: — |
…GetCount against concurrent close finishClose() cleared compactCancelRequested right after the operationsInFlight drain, but an async compact() releases its OperationGuard at setup handoff and is not awaited until the closables sweep — so it can still be running after the drain returns, and clearing the token there left it able to stall teardown (and every concurrent open on the path) indefinitely. Keep the token armed for finishClose()'s whole duration instead, and have the close-time compact-on-close pass opt out via a new compactRange() `cancellable` param rather than relying on the shared flag being cleared. Transaction::GetCount now takes an OperationGuard and checks isClosing() before scanning: without it, finishClose()'s drain can return immediately and the closables sweep can roll back the transaction while the count scan is parked between rows, reading freed memory. Carries the in-progress PR #787 lifecycle repair plan describing the fuller atomic-admission fix these two changes are a first slice of. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CvCKcGKG7vyhRts1Mv4Wh
f24ef7a to
5b459e4
Compare
…GetCount against concurrent close finishClose() cleared compactCancelRequested right after the operationsInFlight drain, but an async compact() releases its OperationGuard at setup handoff and is not awaited until the closables sweep — so it can still be running after the drain returns, and clearing the token there left it able to stall teardown (and every concurrent open on the path) indefinitely. Keep the token armed for finishClose()'s whole duration instead, and have the close-time compact-on-close pass opt out via a new compactRange() `cancellable` param rather than relying on the shared flag being cleared. Transaction::GetCount now takes an OperationGuard and checks isClosing() before scanning: without it, finishClose()'s drain can return immediately and the closables sweep can roll back the transaction while the count scan is parked between rows, reading freed memory. Carries the in-progress PR #787 lifecycle repair plan describing the fuller atomic-admission fix these two changes are a first slice of. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CvCKcGKG7vyhRts1Mv4Wh
5b459e4 to
542b058
Compare
|
Reviewed Re-review of the one new commit since
Also re-verified at object-code level:
— |
…GetCount against concurrent close finishClose() cleared compactCancelRequested right after the operationsInFlight drain, but an async compact() releases its OperationGuard at setup handoff and is not awaited until the closables sweep — so it can still be running after the drain returns, and clearing the token there left it able to stall teardown (and every concurrent open on the path) indefinitely. Keep the token armed for finishClose()'s whole duration instead, and have the close-time compact-on-close pass opt out via a new compactRange() `cancellable` param rather than relying on the shared flag being cleared. Transaction::GetCount now takes an OperationGuard and checks isClosing() before scanning: without it, finishClose()'s drain can return immediately and the closables sweep can roll back the transaction while the count scan is parked between rows, reading freed memory. Carries the in-progress PR #787 lifecycle repair plan describing the fuller atomic-admission fix these two changes are a first slice of. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CvCKcGKG7vyhRts1Mv4Wh
542b058 to
3cdd9f9
Compare
…GetCount against concurrent close finishClose() cleared compactCancelRequested right after the operationsInFlight drain, but an async compact() releases its OperationGuard at setup handoff and is not awaited until the closables sweep — so it can still be running after the drain returns, and clearing the token there left it able to stall teardown (and every concurrent open on the path) indefinitely. Keep the token armed for finishClose()'s whole duration instead, and have the close-time compact-on-close pass opt out via a new compactRange() `cancellable` param rather than relying on the shared flag being cleared. Transaction::GetCount now takes an OperationGuard and checks isClosing() before scanning: without it, finishClose()'s drain can return immediately and the closables sweep can roll back the transaction while the count scan is parked between rows, reading freed memory. Carries the in-progress PR #787 lifecycle repair plan describing the fuller atomic-admission fix these two changes are a first slice of. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CvCKcGKG7vyhRts1Mv4Wh
107b216 to
ab5cd96
Compare
cb1kenobi
left a comment
There was a problem hiding this comment.
Please rebase with main and resolve the merge conflicts.
…GetCount against concurrent close finishClose() cleared compactCancelRequested right after the operationsInFlight drain, but an async compact() releases its OperationGuard at setup handoff and is not awaited until the closables sweep — so it can still be running after the drain returns, and clearing the token there left it able to stall teardown (and every concurrent open on the path) indefinitely. Keep the token armed for finishClose()'s whole duration instead, and have the close-time compact-on-close pass opt out via a new compactRange() `cancellable` param rather than relying on the shared flag being cleared. Transaction::GetCount now takes an OperationGuard and checks isClosing() before scanning: without it, finishClose()'s drain can return immediately and the closables sweep can roll back the transaction while the count scan is parked between rows, reading freed memory. Carries the in-progress PR #787 lifecycle repair plan describing the fuller atomic-admission fix these two changes are a first slice of. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CvCKcGKG7vyhRts1Mv4Wh
Rebase of PR #787 onto latest main. main independently landed a resolved-identity/per-entry destroy-vs-open gate (secondary/read-only aware) while this branch was in flight, so the two designs are merged here rather than replayed commit-by-commit: - Keep main's DBKey{identityPath, readOnly, secondaryPath} cross-key gate as the identity/correctness foundation instead of this PR's original path-only destroyingPaths set. - Layer this PR's close/destroy quarantine on top: a failed close or destroy leaves the registry entry with closeError set instead of erasing or leaking it, so shutdown()/destroy() can retry and registryStatus()/`database:closeFailed` surface the failure. destroyingPaths returns as a path-level gate solely for the window between "registry entries erased" and "physical files deleted" -- that deletion runs without databasesMutex held (it's I/O), so the registry alone can't gate a concurrent open once its entries are gone. - Bound every lifecycle wait (open/destroy/shutdown) by a new lifecycleWaitSeconds setting (default 30s) instead of waiting unboundedly, consistent with this codebase's existing ROCKSDB_JS_PARK_TIMEOUT_MS precedent. - Add DBRegistry::Teardown(), run from the module's last env-cleanup hook after Shutdown(): a descriptor quarantined at process exit must not survive to the registry singleton's static destructor, which runs after RocksDB's own statics and aborts the process. - Move handle attach/adoption inside DBRegistry::OpenDB() under the same lock as its lifecycle waits (was: caller-side, after the lock released) -- closes a real race in main's current code where a concurrent destroy() could tear down a descriptor between OpenDB() returning and the caller's separate attach() call. - Add close-time compaction cancellation (two tokens: descriptor-wide for synchronous callers, per-handle for async, per invariant 6) and make async-work admission/cancellation atomic under one mutex so a registration can never land after a close has already observed zero in-flight work. - Serialize iterator calls against a foreign forced close and add a destroy()-tolerant readOnly-aware destroy() precondition on the JS side (Store::readOnly, checked even for a never-opened handle). AGENTS.md invariants 17, 19 (renumbered from main's 18), 21-23 document the merged design; invariant 6 documents the quarantine and compaction-cancellation additions in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
d082b42 to
e728a97
Compare
Rebase of PR #787 onto latest main. main independently landed a resolved-identity/per-entry destroy-vs-open gate (secondary/read-only aware) while this branch was in flight, so the two designs are merged here rather than replayed commit-by-commit: - Keep main's DBKey{identityPath, readOnly, secondaryPath} cross-key gate as the identity/correctness foundation instead of this PR's original path-only destroyingPaths set. - Layer this PR's close/destroy quarantine on top: a failed close or destroy leaves the registry entry with closeError set instead of erasing or leaking it, so shutdown()/destroy() can retry and registryStatus()/`database:closeFailed` surface the failure. destroyingPaths returns as a path-level gate solely for the window between "registry entries erased" and "physical files deleted" -- that deletion runs without databasesMutex held (it's I/O), so the registry alone can't gate a concurrent open once its entries are gone. - Bound every lifecycle wait (open/destroy/shutdown) by a new lifecycleWaitSeconds setting (default 30s) instead of waiting unboundedly, consistent with this codebase's existing ROCKSDB_JS_PARK_TIMEOUT_MS precedent. - Add DBRegistry::Teardown(), run from the module's last env-cleanup hook after Shutdown(): a descriptor quarantined at process exit must not survive to the registry singleton's static destructor, which runs after RocksDB's own statics and aborts the process. - Move handle attach/adoption inside DBRegistry::OpenDB() under the same lock as its lifecycle waits (was: caller-side, after the lock released) -- closes a real race in main's current code where a concurrent destroy() could tear down a descriptor between OpenDB() returning and the caller's separate attach() call. - Add close-time compaction cancellation (two tokens: descriptor-wide for synchronous callers, per-handle for async, per invariant 6) and make async-work admission/cancellation atomic under one mutex so a registration can never land after a close has already observed zero in-flight work. - Serialize iterator calls against a foreign forced close and add a destroy()-tolerant readOnly-aware destroy() precondition on the JS side (Store::readOnly, checked even for a never-opened handle). AGENTS.md invariants 17, 19 (renumbered from main's 18), 21-23 document the merged design; invariant 6 documents the quarantine and compaction-cancellation additions in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
e728a97 to
42b42aa
Compare
Rebase of PR #787 onto latest main. main independently landed a resolved-identity/per-entry destroy-vs-open gate (secondary/read-only aware) while this branch was in flight, so the two designs are merged here rather than replayed commit-by-commit: - Keep main's DBKey{identityPath, readOnly, secondaryPath} cross-key gate as the identity/correctness foundation instead of this PR's original path-only destroyingPaths set. - Layer this PR's close/destroy quarantine on top: a failed close or destroy leaves the registry entry with closeError set instead of erasing or leaking it, so shutdown()/destroy() can retry and registryStatus()/`database:closeFailed` surface the failure. destroyingPaths returns as a path-level gate solely for the window between "registry entries erased" and "physical files deleted" -- that deletion runs without databasesMutex held (it's I/O), so the registry alone can't gate a concurrent open once its entries are gone. - Bound every lifecycle wait (open/destroy/shutdown) by a new lifecycleWaitSeconds setting (default 30s) instead of waiting unboundedly, consistent with this codebase's existing ROCKSDB_JS_PARK_TIMEOUT_MS precedent. - Add DBRegistry::Teardown(), run from the module's last env-cleanup hook after Shutdown(): a descriptor quarantined at process exit must not survive to the registry singleton's static destructor, which runs after RocksDB's own statics and aborts the process. - Move handle attach/adoption inside DBRegistry::OpenDB() under the same lock as its lifecycle waits (was: caller-side, after the lock released) -- closes a real race in main's current code where a concurrent destroy() could tear down a descriptor between OpenDB() returning and the caller's separate attach() call. - Add close-time compaction cancellation (two tokens: descriptor-wide for synchronous callers, per-handle for async, per invariant 6) and make async-work admission/cancellation atomic under one mutex so a registration can never land after a close has already observed zero in-flight work. - Serialize iterator calls against a foreign forced close and add a destroy()-tolerant readOnly-aware destroy() precondition on the JS side (Store::readOnly, checked even for a never-opened handle). AGENTS.md invariants 17, 19 (renumbered from main's 18), 21-23 document the merged design; invariant 6 documents the quarantine and compaction-cancellation additions in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
c9cecc1 to
215ee4d
Compare
Rebase of PR #787 onto latest main. main independently landed a resolved-identity/per-entry destroy-vs-open gate (secondary/read-only aware) while this branch was in flight, so the two designs are merged here rather than replayed commit-by-commit: - Keep main's DBKey{identityPath, readOnly, secondaryPath} cross-key gate as the identity/correctness foundation instead of this PR's original path-only destroyingPaths set. - Layer this PR's close/destroy quarantine on top: a failed close or destroy leaves the registry entry with closeError set instead of erasing or leaking it, so shutdown()/destroy() can retry and registryStatus()/`database:closeFailed` surface the failure. destroyingPaths returns as a path-level gate solely for the window between "registry entries erased" and "physical files deleted" -- that deletion runs without databasesMutex held (it's I/O), so the registry alone can't gate a concurrent open once its entries are gone. - Bound every lifecycle wait (open/destroy/shutdown) by a new lifecycleWaitSeconds setting (default 30s) instead of waiting unboundedly, consistent with this codebase's existing ROCKSDB_JS_PARK_TIMEOUT_MS precedent. - Add DBRegistry::Teardown(), run from the module's last env-cleanup hook after Shutdown(): a descriptor quarantined at process exit must not survive to the registry singleton's static destructor, which runs after RocksDB's own statics and aborts the process. - Move handle attach/adoption inside DBRegistry::OpenDB() under the same lock as its lifecycle waits (was: caller-side, after the lock released) -- closes a real race in main's current code where a concurrent destroy() could tear down a descriptor between OpenDB() returning and the caller's separate attach() call. - Add close-time compaction cancellation (two tokens: descriptor-wide for synchronous callers, per-handle for async, per invariant 6) and make async-work admission/cancellation atomic under one mutex so a registration can never land after a close has already observed zero in-flight work. - Serialize iterator calls against a foreign forced close and add a destroy()-tolerant readOnly-aware destroy() precondition on the JS side (Store::readOnly, checked even for a never-opened handle). AGENTS.md invariants 17, 19 (renumbered from main's 18), 21-23 document the merged design; invariant 6 documents the quarantine and compaction-cancellation additions in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
215ee4d to
5e8803d
Compare
Rebase of PR #787 onto latest main. main independently landed a resolved-identity/per-entry destroy-vs-open gate (secondary/read-only aware) while this branch was in flight, so the two designs are merged here rather than replayed commit-by-commit: - Keep main's DBKey{identityPath, readOnly, secondaryPath} cross-key gate as the identity/correctness foundation instead of this PR's original path-only destroyingPaths set. - Layer this PR's close/destroy quarantine on top: a failed close or destroy leaves the registry entry with closeError set instead of erasing or leaking it, so shutdown()/destroy() can retry and registryStatus()/`database:closeFailed` surface the failure. destroyingPaths returns as a path-level gate solely for the window between "registry entries erased" and "physical files deleted" -- that deletion runs without databasesMutex held (it's I/O), so the registry alone can't gate a concurrent open once its entries are gone. - Bound every lifecycle wait (open/destroy/shutdown) by a new lifecycleWaitSeconds setting (default 30s) instead of waiting unboundedly, consistent with this codebase's existing ROCKSDB_JS_PARK_TIMEOUT_MS precedent. - Add DBRegistry::Teardown(), run from the module's last env-cleanup hook after Shutdown(): a descriptor quarantined at process exit must not survive to the registry singleton's static destructor, which runs after RocksDB's own statics and aborts the process. - Move handle attach/adoption inside DBRegistry::OpenDB() under the same lock as its lifecycle waits (was: caller-side, after the lock released) -- closes a real race in main's current code where a concurrent destroy() could tear down a descriptor between OpenDB() returning and the caller's separate attach() call. - Add close-time compaction cancellation (two tokens: descriptor-wide for synchronous callers, per-handle for async, per invariant 6) and make async-work admission/cancellation atomic under one mutex so a registration can never land after a close has already observed zero in-flight work. - Serialize iterator calls against a foreign forced close and add a destroy()-tolerant readOnly-aware destroy() precondition on the JS side (Store::readOnly, checked even for a never-opened handle). AGENTS.md invariants 17, 19 (renumbered from main's 18), 21-23 document the merged design; invariant 6 documents the quarantine and compaction-cancellation additions in place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…on, update docs Test coverage for the rebased lifecycle work: destroy/close quarantine and retry, bounded lifecycle waits, the destroy-vs-open path gate, compaction cancellation on all four close paths (self, foreign close, destroy, shutdown), async-work admission mutual exclusion, and iterator/count races against a concurrent forced close. test/secondary.test.ts's "settle an in-flight catch-up when the database is torn down" is updated for the merged design: destroy() now waits out and tolerates a lingering async-state reference the same way an in-flight checkpoint/backup already does (see test/checkpoint.test.ts and test/backup-stream.test.ts), rather than failing on it the way the old refcount-based design did -- there was no principled reason for catchUpWithPrimary to be the one operation treated differently here, and the two now agree. README.md documents the quarantine/retry contract for close(), destroy(), shutdown(), and registryStatus(); AGENTS.md documents the new env vars. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ng, destroy flush waste - CloseDB and the iterator finalizer detached from `closables` before close()/Reset(), leaving a concurrent destroy()/shutdown() sweep unable to see (and wait for) a handle/iterator still draining async work or mid-Next() -- a real use-after-free window on the shared rocksdb::DB. Detach after close() returns instead; closeMutex/iteratorMutex already serialize a foreign close arriving in that window. - DBHandle::close() released `logRefs` napi_refs gated on a std::thread::id equality check, the same recycled-pthread-id hazard invariant 18 already fixed for TransactionHandle. Add DBRegistry::ReleaseLogRefsByEnv, wired into the env cleanup hook like CloseTransactionsByEnv, so a dying env's logRefs are emptied while it is still alive -- before its thread id could ever be reused against the stale guard. - OpenDB's "still closing"/"retry in progress" waits parked on one matching descriptor's condition variable but re-scanned the whole path in their predicate, so two descriptors closing on one path (e.g. a writable and a secondary) could leave an opener asleep for the full lifecycleWaitSeconds even after the path was free. Track the specific selected entry instead. - finishClose(destroying=true) still ran a full flush + close-time compaction before the files are unlinked -- wasted I/O, and with allow_write_stall's default a destroy() of a stalled database could hang indefinitely while holding the destroyingPaths gate. Skip both when destroying. - Add a fixture covering a lifecycleWaitSeconds timeout actually firing and the path recovering afterward (previously only config validation was tested). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01122SfiNfCtiLgWvQTD6wZ7
…eal two-descriptor regression test - fork-lifecycle-timeout.mts raced the shutdown retry claim against the main thread's open() with no barrier, so under load the open could win and throw the wrong error instead of timing out -- a flake, not a proof. Expose `closeRetrying` on registryStatus() and poll for it before racing the open. - fork-compact-cancel-destroy.mts lost its discriminating power once destroy() skips compactOnClose (previous commit): with nothing left to block on compactMutex, an early vs. late cancellation arm became timing-indistinguishable. Drive it through shutdown() instead, which still runs compactOnClose. Verified: reverting the early arm now makes the fixture fail again (7.7s vs the ~500ms bound), confirming this restores the fixture's purpose. - Added a genuine two-descriptor regression test for the OpenDB condition-variable/predicate fix (db_registry.cpp:603/:655): a writable and a read-only descriptor quarantine, then retry concurrently under one shutdown() call, with an opener racing in. Verified against the pre-fix predicate: 4/4 runs stalled to the full 8s deadline instead of the expected ~3s, confirming this catches the regression the prior test explicitly could not. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01122SfiNfCtiLgWvQTD6wZ7
shutdown() is process-wide, not scoped to one path, and its worker call could still be re-scanning for anything left to close after the racing open() call returned but before it posted shutdownResult -- a handle opened in that window is not safe to read from or hold onto (shutdown() could sweep and force-close it too). Keep the timing-only open (the actual regression proof) but defer the data-preservation check to a fresh open, strictly after the worker's shutdown() call and the worker itself are both confirmed done. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01122SfiNfCtiLgWvQTD6wZ7
Review follow-up on the `shutdown()`-throws thread: the behavior is kept
(db.close() throws the identical quarantine error, and three of the four
throw sites are lifecycle timeouts that `database:closeFailed` cannot
carry), but the only place it was written down was the README. The
exported symbol now carries the same contract, including why a
`process.on('exit')` listener has to wrap the call: a throw from an exit
listener skips every listener registered after it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eams
Independent pre-push review round 1 on the rebased head (codex + gemini +
harper-domain) found five real items:
- `DBHandle::open()` now drops the previous lifecycle's `logRefs`. The
owner-thread guard that makes a foreign close napi-safe (AGENTS.md
invariant 18) also means a cross-env `destroy()`/`shutdown()` leaves the
cache populated, so a reopened handle handed `useLog()` back a
`TransactionLog` whose store `weak_ptr` pointed at the unregistered store
of the closed lifecycle. Only `addEntry` re-resolves, so every read
accessor reported an empty log — `getLogFileSize()` returned 0 instead of
31 in the new fixture, which fails 1/1 without the fix.
- `registryStatus()`'s destroy-cleanup tombstone branch omitted
`transactionDetails`, which `RegistryStatusDB` declares non-optional; a
monitor reading `entry.transactionDetails.length` threw on exactly the
entry shape that only appears when a physical destroy failed.
- The three test-delay seams this PR added (`ROCKSDB_JS_BACKUP_DELAY_MS`,
`ROCKSDB_JS_DESTROY_DELAY_MS`, `ROCKSDB_JS_CLOSE_RETRY_DELAY_MS`) still
called `::getenv` from a libuv worker / arbitrary teardown thread. They
are snapshotted in `initializeTestSeams()` now, like the fault flags
beside them.
- `fork-shutdown-retry.mts` raced the worker's retry claim: the worker posts
before calling `shutdown()`, so the parent's open could hit the still
quarantined entry. It now polls `closeRetrying` first and re-reads data
from a fresh handle after the worker is done — the two barriers the
two-descriptor fixture already had.
- `fork-compact-cancel-async.mts` had no `'message'` listener attached
across `await outcome`, so a `{destroyed:true}` posted in that window was
dropped and the fixture would hang to its timeout.
Declined, with reasons in the PR body: making `isClosing()` a relaxed load
(the reviewer's own estimate is that it is dwarfed by `iterator->Next()`,
and the flag is read under mutexes elsewhere), and the aggregate
comment-narration nit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI turned red on macOS (Bun + Deno) with four quarantine tests failing on
`/private/var/...` vs `/var/...`: a tombstoned `registryStatus()` entry and
every `database:closeFailed` event reported the registry key's resolved
identity, so a caller matching either against the path it opened did not
recognize it. That is exactly what AGENTS.md invariant 19 forbids
("returning only the resolved identity breaks callers that match paths
against the spelling they supplied") and what `registryStatus()` already
does correctly for a live descriptor.
- `emitCloseFailures()` reports `descriptor->path`.
- `DBRegistryEntry::reportedPath` remembers the opening caller's spelling so
a destroy-cleanup tombstone — which has no descriptor left to ask — can
still report it, in `registryStatus()` and in its own emit.
Not caused by the rebase (the pre-rebase head carried identical code and was
green on the older macos-26-arm64 runner image); the new image's TMPDIR made
the latent mismatch reachable. `test/destroy.test.ts` now covers it on every
platform with an explicit symlink instead of depending on macOS's `/var`
link — verified to fail with `expected undefined to be true`, the same
assertion macOS CI produced, when either report site is reverted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tombstone Round 3 of the independent pre-push review (codex) found the one gap in the previous commit: a second `destroy()` after a failed physical cleanup starts with `reportedPath = identityPath` and its scan skipped the existing tombstone, because that entry has no descriptor. So `registryStatus().path` kept reporting the opened spelling while the retry's `database:closeFailed` reverted to the resolved identity — the two disagreeing is the same defect one step later. The scan now falls back to a matching entry's remembered spelling, still preferring a live descriptor's. The symlink test asserts the event path on the retry as well, and fails with `expected [ …(2) ] to match object [ …(2) ]` without the fallback. Also dropped from that round, verified rather than assumed: Gemini's blocker on `checkpoint.cpp` leaking `operationsInFlight` when admission or queueing fails — `CheckpointInFlightClaim` is the RAII equivalent of `BackupInFlightClaim` and `handedOff` is only set after both succeed (`src/binding/database/checkpoint.cpp:69-76,114-115,206`). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI's remaining SIGSEGV (`fork-destroy-open.mts`, one Deno/Bun job per run since the rebase) is a real native use-after-free on the JS thread, not a flake. Reproduced locally at 2/60 by running that fixture 6-way concurrently, and gdb put it exactly here: __strlen_avx2 v8::String::NewFromUtf8 napi_set_named_property rocksdb_js::DBRegistry::RegistryStatus `registryStatus()` holds `databasesMutex`, which covers the registry map but not a descriptor's own maps. It walked `descriptor->columns` — guarded by `columnsMutex` — while a cross-env `destroy()`'s `finishClose()` cleared that map from the worker thread, so `name.c_str()` pointed into a freed map node and `napi_set_named_property()` `strlen()`ed it. `locks.size()` was read unguarded the same way (a count, so a torn read rather than a fault). The column summary is now snapshotted under `columnsMutex` (plus the per-CF `userSharedBuffersMutex` for its buffer count) and the JS values are built after releasing it — holding it across the N-API calls would risk a finalizer re-entering the same non-recursive mutex on this thread, which is why `transactions` already had this shape under `txnsMutex`. `locks.size()` is read under `locksMutex`. Neither adds a lock-order inversion: `OpenDB` and `CollectWriteBufferManagerInventory` already establish `databasesMutex → columnsMutex`, `getUserSharedBuffer` is a leaf, and no `locksMutex` region reaches the registry. Pre-existing on `main` (its `registryStatus()` walks `columns` unguarded too); this PR is what makes it reachable, because `destroy()` now force-closes every descriptor and its own fixtures poll `registryStatus()` across that window. `test/fixtures/fork-registry-status-column-race.mts` makes it deterministic rather than leaving it to CI luck: a worker churns `dropSync()` against the shared descriptor while the main thread polls `registryStatus()`, with the new `ROCKSDB_JS_REGISTRY_STATUS_COLUMNS_DELAY_MS` seam parking the walk per column family so an erase lands inside it. Column names run past libstdc++'s 15-char small-string buffer so the erase frees a separate heap allocation (an SSO name usually survives the free intact and hides the bug). 5/5 abort without the fix, 3/3 clean with it; the concurrent `fork-destroy-open.mts` stress went 2/60 → 0/132. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TMokk8DsJyGwpHz4Lmts85
…iming Round 5 of the independent pre-push review (codex, seconded by the domain adjudicator) caught `test/fixtures/fork-registry-status-destroy-race.mts` shipping unreferenced: it was the first attempt at the `registryStatus()` column-walk regression test and does not reproduce (0/7 against the unfixed walk), because a single close-time `columns.clear()` rarely lands inside a walk and a small-string column name usually survives the free intact. The drop-churn fixture that replaced it does reproduce deterministically (5/5), and it covers the same thing — the fix is the snapshot in the walk, not anything per-mutator, so reverting that snapshot fails the churn fixture too. Also reworded the tombstone branch's comment, which claimed it fills "every non-optional field of RegistryStatusDB": `userSharedBuffers` is declared non-optional and set by no branch, and `columnFamilies` is typed `string[]` against an object. Both predate this PR and are noted rather than fixed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TMokk8DsJyGwpHz4Lmts85
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
5e8803d to
8784294
Compare
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
cb1kenobi
left a comment
There was a problem hiding this comment.
This was a big one, but it looks great! I love the OperationGuard!
Summary
Rebases this PR onto latest
main. While this was in flight,mainindependently landed an equivalent-but-different fix for the same destroy-vs-open race (2c2066aa,84157bf9): a resolved-identity, per-entry gate (DBKey{path, readOnly, secondaryPath}) instead of this PR's original path-onlydestroyingPathsset. Per Kris's direction, this rebase adopts main's gate rather than porting the PR's original design: the first commit skips the PR's now-duplicate commit and adapts every dependent commit onto main's per-entry/cross-key conditions, and the second carries forward the PR's quarantine/retry/compaction-cancellation work and its test fixtures on top of that.destroyingPathsitself is re-derived (not ported) as a separate, lock-free-compatible gate for the window between "registry entries erased" and "physical files deleted" — physical deletion runs withoutdatabasesMutexheld (it's I/O-bound), so main's design alone left that window ungated; this was found by test failures against the carried-forward fixtures, not assumed from the PR.This is the rocksdb-js root-cause fix for Harper PR #2169: it moves a lifecycle invariant Harper worked around in JavaScript (locking root opens) into the native registry, makes teardown failures observable/recoverable via quarantine +
database:closeFailed, and adds cancellation tokens so a manual compaction cannot block teardown indefinitely.Second rebase: onto main's WriteBufferManager stall watchdog (#824)
mainmoved again (7213b98a, "Make a WriteBufferManager write stall observable") and this branch wasCONFLICTING. Four conflicts, all resolved keeping both sides:binding.cppShutdown— main's watchdogbegin…Shutdown()/join…Watchdog()bracket now wraps this PR'stry/catchGlobalEvents::Shutdown()afterDBRegistry::Shutdown()(a quarantining close emitsdatabase:closeFailed, which needs its listeners to still exist —test/fixtures/fork-shutdown-failure.mtsasserts the event), and the watchdog join must run before the throw, or a failedshutdown()leaves the 1 Hz thread alive. Same combination in the last-env cleanup hook.backup.cpp— both hunks were pure PR-side additions (BackupInFlightClaim, thenapi_cancelledin-flight decrement) that git could not place.db_registry.h— this PR'sCloseResult CloseDB(...)return type alongside main'sCollectWriteBufferManagerInventory.AGENTS.md— invariant renumbering. Main's WBM-stall invariant becomes 21; this PR's three become 22/23/24, with the two internal cross-references (see invariant 21→ 22,see invariant 22→ 23) updated. Main'sdroppedColumns.clear()infinishClose()and its two-argumentColumnFamilyDescriptorconstructor were carried into this PR's rewrittenfinishClose(bool destroying)/OpenDB().Fixed by the rebase's own independent pre-push review
Two full rounds against the rebased head (codex + gemini + harper-domain; Cursor legs are disabled on any diff that edits
AGENTS.md). Round 1 found five real items, all fixed; round 2 verified each against HEAD and produced no new actionable findings.ownerThreadIdguard that makes a foreign close napi-safe also means a cross-envdestroy()/shutdown()can no longer clearlogRefs, so a reopened handle handeduseLog()back aTransactionLogwhoseTransactionLogHandle::storeweak_ptrpointed at the unregistered store of the closed lifecycle. OnlyaddEntryre-resolves; every read accessor reported an empty log.DBHandle::open()now releases the cache on the owning thread beforeDBRegistry::OpenDB.test/fixtures/fork-foreign-close-log-cache.mtsholds the staleTransactionLogalive across the foreignshutdown()(the cache entry is a weaknapi_ref, so letting it be collected would mask the bug) and asserts log size, queried entry count, and object identity — it reports0 bytes, expected 31without the fix, 3/3 clean with it.transactionDetails—RegistryStatusDBdeclares it non-optional, so a monitor readingentry.transactionDetails.lengththrew on exactly the entry shape that only appears when a physical destroy failed, i.e. when the diagnostic is needed.::getenvoff the JS thread —ROCKSDB_JS_BACKUP_DELAY_MS(libuv worker),ROCKSDB_JS_DESTROY_DELAY_MSandROCKSDB_JS_CLOSE_RETRY_DELAY_MS(any teardown thread), all added by this PR, now snapshotted ininitializeTestSeams()like the fault flags beside them. AGENTS.md's entries for the three say so.fork-shutdown-retry.mtsraced the retry claim — the worker posts before callingshutdown(), so the parent's open could reach the still quarantined entry and fail with "previous close failed" instead of measuring the wait; and a handle opened while the process-wideshutdown()loop is still scanning can be force-closed before the data assertion reads it. Both barriers the two-descriptor fixture already had.fork-compact-cancel-async.mtscould drop the destroy result —await outcomeyields to the event loop with no'message'listener attached, so a{ destroyed: true }posted in that window was delivered to a zero-listener emitter and dropped, hanging the fixture to its runner timeout. The promise is now claimed before the await. The sync sibling is not affected (compactSync()blocks the loop, so the queued message is only delivered after the next listener attaches).Red CI caught a real path-identity defect
The first rebase push turned macOS CI red (Bun + Deno) with four quarantine tests failing on
/private/var/...vs/var/.... Root cause, not a test problem: a destroy-cleanup tombstone inregistryStatus()and everydatabase:closeFailedevent reported the registry key's resolved identity, so a caller matching either against the path it opened did not recognize it. That is exactly what AGENTS.md invariant 19 forbids ("returning only the resolved identity breaks callers that match paths against the spelling they supplied") and whatregistryStatus()already did correctly for a live descriptor.emitCloseFailures()reportsdescriptor->path.DBRegistryEntry::reportedPathremembers the opening caller's spelling, because a tombstone has no descriptor left to ask.DestroyDBcaptures it from the first claimed descriptor underdatabasesMutex, and a retry of a failed destroy — which finds only the tombstone — falls back to the spelling that entry already remembered (the round-3 review finding).Not caused by the rebase: the pre-rebase head carried identical code and was green on the older
macos-26-arm64runner image (20260728→20260831), whoseTMPDIRmade the latent mismatch reachable.test/destroy.test.tsnow covers it on every platform with an explicit symlink rather than depending on macOS's/varlink, and asserts both the initial failure and the retry. Each report site was reverted individually to confirm the test fails — withexpected undefined to be true, the same assertion macOS CI produced, andexpected [ …(2) ] to match object [ …(2) ]for the retry.A CI "flake" that was a real use-after-free
The first two rebase pushes each had exactly one job die with
SIGSEGVinfork-destroy-open.mts— a different runtime each time (Bun/ubuntu, then Deno/ubuntu), which reads like flake. It is not. Running that fixture six-way concurrently reproduced it locally at 2/60, and gdb named the frame:databasesMutexcovers the registry map, not a descriptor's own maps.registryStatus()walkeddescriptor->columns— guarded bycolumnsMutex— while a cross-envdestroy()'sfinishClose()cleared that map from the worker thread, soname.c_str()pointed into a freed map node andnapi_set_named_property()strlen()ed it.locks.size()was read the same way (a count, so a torn read rather than a fault). The column summary is now snapshotted undercolumnsMutex(plus the per-CFuserSharedBuffersMutexfor its buffer count) with the N-API values built after releasing it — holding it across those calls would risk a finalizer re-entering the same non-recursive mutex on this thread, which is whytransactionsalready had this shape undertxnsMutex.locks.size()is read underlocksMutex.No lock-order inversion:
OpenDBandCollectWriteBufferManagerInventoryalready establishdatabasesMutex → columnsMutex,getUserSharedBufferis a leaf, and nolocksMutexregion reaches the registry. Pre-existing onmain(itsregistryStatus()walkscolumnsunguarded too); this PR is what makes it reachable, becausedestroy()now force-closes every descriptor and the PR's own fixtures pollregistryStatus()across that window.test/fixtures/fork-registry-status-column-race.mtsmakes it deterministic instead of leaving it to CI luck: a worker churnsdropSync()against the shared descriptor while the main thread pollsregistryStatus(), with the newROCKSDB_JS_REGISTRY_STATUS_COLUMNS_DELAY_MSseam parking the walk per column family so an erase lands inside it. The column names run past libstdc++'s 15-char small-string buffer so the erase frees a separate heap allocation — an SSO name usually survives the free intact and hides the bug, which is why the first attempt at this fixture (driving the close-timecolumns.clear()instead) reproduced 0/7 and was deleted rather than shipped unreferenced. 5/5 abort without the fix, 3/3 clean with it, and the concurrentfork-destroy-open.mtsstress went 2/60 → 0/132. Reverting the snapshot fails the churn fixture, so the close-time path is covered by the same net.Review-thread adjudication
All three previously-unresolved threads were re-checked against the current code; none is left unanswered, and the one new thread is ruled on below.
shutdown()can abort process exit and skip later cleanup (new,binding.cpp) — claim verified, prescribed fix overruled. Details under## For the human reviewer.db_registry.cpp) — unchanged ruling: real, pre-existing (registry identity has always been the raw path string), and canonicalizing changes a user-visible identity contract. Wants its own PR.transaction.cpp) — unchangedruling: the window is real but is not specific to quarantine, and the suggested fix (reject admission when
descriptor->isClosing()) is what AGENTS.md invariant 18 exists to prevent. The fix that closes it — running the closables sweep before the flush infinishClose()— reorders the most delicate path here and wants its own change.Third rebase: onto main's tsdown bump
mainmoved again (dependabot's tsdown bump, merged as531af655). No conflicts — the only delta between the previous head's merge-base andorigin/mainwaspackage.json/pnpm-lock.yaml, andgit rebase origin/mainreplayed all 11 commits clean. Build,pnpm test:native(194/194), and the full Vitest suite (935 passed / 9 skipped) all pass post-rebase;pnpm fmt:check/lint/type-checkare clean.Independent pre-push review (round 25, full — a force-push is never an ancestor of the prior review): codex (graded) and Gemini both ran; the Harper-domain adjudicator hit its time budget and was killed (
SIGKILL, timeout) before it could rank/filter, so I adjudicated the raw output myself against the code:benchmark/setup.ts's teardownresolve()-then-throwsilently reports success: it conflates the serialization gate (activeBenchmark/promise, used only to sequence between benchmarks) with the current benchmark's own result (theasync setup()call's returned promise, which the trailingthrowgenuinely rejects —throws: trueis set exactly so vitest surfaces it). Not applied.db_registry.cpp:1029holdsdatabasesMutexacrossnapi_create_string_utf8calls inRegistryStatus(), and if V8 GC runs aDBHandlefinalizer synchronously mid-allocation, that finalizer'sDBRegistry::CloseDBwould re-lock the same non-recursive mutex on the same thread — checks out as a real hazard class (it's exactly what invariant 6'scolumnsMutex/txnsMutex/locksMutexnarrow-scoping exists to avoid for the other locks in this same function), but the outerdatabasesMutexhold is byte-for-byte unchanged fromorigin/main(confirmed viagit show origin/main:src/binding/database/db_registry.cpp) — it predates this PR and every prior round. Left alone as out-of-scope for a rebase; flagged separately for its own issue.For the human reviewer
shutdown()now throws — the review claim is correct, and I kept the behavior. Your call to overrule me. The bot's facts check out, and I verified the Node semantics directly: a throw from aprocess.on('exit')listener skips everyexitlistener registered after it (unconditionally), and flips the exit code to 1 unless anuncaughtExceptionhandler is installed. Harper core calls it exactly that way —resources/RocksTransactionLogStore.ts:19,process.on('exit', () => shutdown()), bare, registered at module load — so on a close-time flush failure at exit it would skip laterexitlisteners, including ones registered by application components. Onmain,shutdown()genuinely never throws (descriptor->close()isvoidand the close-time flush status is discarded), so this is a behavior change, not a clarification.Three reasons I did not adopt "keep
shutdown()non-throwing":db.close()already throws the identical error (database.cpp:211, with the "Call shutdown() to retry close, or destroy() to delete the database" suffix). Makingshutdown()silent would have the two close entry points disagree about whether a failed close is an error.DBRegistry::Shutdown()'s four throw sites are lifecycle timeouts, not closefailures:
shutdownMutexcontention, the per-descriptor drain, and the destroy-in-flight wait.database:closeFailedcannot carry those — no descriptor failed. A blanket non-throwingshutdown()would return normally while databases are still open or files still being deleted, which is a worse silent failure than the one being avoided.database:closeFailed(no listener anywhere in the repo), so today the throw is the only channel that reaches it. Event-only reporting would make a close-time flush failure fully invisible there.What I did instead is make the contract explicit where a caller will see it: the README already documented the throw and the
try/catchexit-listener pattern, and the exportedshutdownnow carries the same JSDoc. The harper-side guard is still missing and is a one-line change atresources/RocksTransactionLogStore.ts:19; it needs to land with or before this, and it is outside this repo. If you would rather not ask that of consumers, the alternative is a return value plus a separate throwing API, and I'd want your direction before building it.Carried from the original PR body, still open (unchanged by either rebase):
../symlink alias can still bypass the destroy/open gate (narrower than it reads — RocksDB's ownLOCKfile blocks a second read-write open through an alias; a read-only alias during destroy is the live hazard). Wants its own PR — canonicalizing changes a user-visible identity contract (registryStatus().path, lock/backup file paths,TransactionLogStoreRegistrykeys).finishClose()) reorders the most delicate path here and wants its own change.--moduleRefCount == 0; a fresh env loading the module concurrently isn't coordinated against a concurrentShutdown()/Teardown(). Pre-existing.DBIteratorHandle::Next()'s unconditional per-rowiteratorMutexlock/unlock on the hottest read path. Estimated low single digits percent overhead against a several-hundred-ns per-row N-API cost — real, but the obvious fix (a relaxed-load gate) is insufficient (a closer could still free the iterator between the load and the mutex), and a correct reader/closer handshake plus apnpm benchrange-scan comparison is more than a rebase should take on.Declined this round, with reasons:
isClosing()as amemory_order_relaxedload (raised as a major by Gemini, downgraded to a nit by the adjudicator in both rounds, and independently scored a nit by Codex). Its own estimate is that theLDARis dwarfed byiterator->Next(), andgetKeysCount()is not a request hot path. The flag is also read under mutexes in the registry paths, so changing the default ordering of a widely-used accessor is not a local change.db_registry.cpp,db_descriptor.{h,cpp},async.h,db_handle.cpp,closable.h,binding.cppthat narrate implementation history or address the reviewer). The technical content is accurate; churning ten comment blocks across a 69-file lifecycle diff adds review surface for no behavior change. Better as a follow-up sweep over the whole file set at once.RegistryStatusDB.userSharedBuffersis declared non-optional but no branch ofregistryStatus()sets a top-level property of that name (it is per-column-family, which is what the README documents) — soentry.userSharedBuffers > 0silently readsundefined > 0. Verified pre-existing onmain; same forcolumnFamiliesbeing typedstring[]against an object. Wants its own change.TransactionLogretained across a foreign close and never reopened still reports an empty log to its holder. Unchanged frommain(which deleted the weak ref on close, leaving any retained object equally stale), and the open-time invalidation above fixes the reopen path, which is the reachable one.Remaining CI failure, not from this PR
test/txn-close-commit-uaf.test.tsaborts on Deno/macOS only. That is the pre-existing worker-env teardown abort the test itself names and already mitigates —const retry = process.versions.deno && process.platform === 'darwin' ? 1 : 0with a comment pointing at #746 — and it exhausted that single retry. It appeared on the same job before the path-spelling and use-after-free commits, and nothing in this PR touches that repro's path.Verification
pnpm check(type-check + lint + fmt) — clean.pnpm test:native— 194/194 passed.pnpm test(Vitest) — 933 passed, 9 skipped (expected Deno/GC-related skips), 0 failed.test/destroy.test.tsalone — 31/31, including the new foreign-close log-cache fixture.test/destroy.test.tsalone — 32/32, run 5× to check the new symlink test for flakiness.--full, forced by the rebase — the prior review is no longer an ancestor) found the five items above. Round 2 (delta) confirmed all five fixed against HEAD and surfaced no new actionable finding; its three surviving items are the two declined nits plus the pre-existinguserSharedBufferstype mismatch, and it recorded a correction to round 1 (which had claimed the tombstone branch omitteduserSharedBufferstoo — never a top-level field). Round 3, on the path-spelling fix, found the tombstone-retry gap, fixed above. Round 4 converged: the graded leg reported zero findings.RegistryStatusDBfields the tombstone branch fills (reworded); round 6 converged, with every item previously adjudicated.checkpoint.cppleakingoperationsInFlightwhen admission/queueing fails (CheckpointInFlightClaimis the RAII equivalent ofBackupInFlightClaim, andhandedOffis set only after both succeed), andbenchmark/setup.tsmaking concurrent workers share one database path (they already did before this PR — the declaration moved within the same scope chain, and sharing one database across workers is the point of those benchmarks);checkpoint.cpp'sCheckpointInFlightClaimagain;ReleaseLogRefsByEnv"permanently leaking"napi_refs for an erased descriptor (they are env-owned and reclaimed at env teardown, and once the entry is erased no foreignclose()can reach the handle throughclosables, which is the hazard invariant 18 exists for); and, for the third time, a per-call__cxa_guard_acquireon the seam flags (astd::atomic<int>with aconstexprconstructor is constant-initialized, so no guard is emitted — confirmed by inspecting the generated assembly in an earlier round).Refs #787
🤖 Generated with Claude Code
Final maintenance pass — 2026-09-09
This note supersedes earlier rebase, review-coverage, and verification statements above. The branch is rebased cleanly onto
17fceeef; PR headac31bdeacontains the final review-feedback fix.registryStatus()thread was valid: carrying ashared_ptr<DBDescriptor>beyonddatabasesMutexcould make a racing last-handle close see an extra owner and skip its purge with no release-side retry.RegistryStatusEntrynow contains only copied values, captured under the registry and owning child locks, and all N-API construction happens after unlocking. The audited registry-to-child ordering is documented beside each mutex, and invariant 6 records the ownership-pin failure mode.isClosing()violates invariant 18 and the safer closables-before-flush reorder needs a separate change; andshutdown()error reporting is a consumer-compatibility choice rather than a mechanical review fix.Changes
DBSettings::Configvalidates the new lifecycle wait bound stored in db_settings.h; database.ts documents close-failure events and carries read-only intent into destroy, while store.ts distinguishes closing from closed sync operations. db_options.h only updates the shifted invariant reference.Verification
pnpm check— clean (type-check, lint, formatting);git diff --check— clean.pnpm test:native— 194/194 passed.pnpm test— 956 passed, 9 skipped, 0 failed.87842947with a retained registry entry (followed by a teardownSIGSEGV) and passes on this head; the strengthened overlap assertion also passes.Framing-Verdict: chosen-approach-sound(Gemini). Final full review produced no new actionablefinding: Gemini repeated an adjudicated benchmark false positive and the existing comment nit; Claude failed at startup, Cursor was policy-pruned for the
AGENTS.mdedit, and the domain leg timed out. These are review-infrastructure coverage gaps, not test failures.Complexity: complicated
Review-Coverage: authored=codex; ran=gemini; blocked=claude(exit-1),domain(timeout); declined=cursor-grok,cursor-composer; rounds=8 @ ac31bde
Human-Review-Need: 4 @ ac31bde