Skip to content

Refuse a better-sqlite3 build that aborts Node 24 when it frees a statement (#719) - #720

Merged
irparent merged 1 commit into
mainfrom
fix/native-statement-teardown
Sep 28, 2026
Merged

irparent merged 1 commit into
mainfrom
fix/native-statement-teardown

Conversation

@irparent

@irparent irparent commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Closes #719.

The crash

test (24, node) in run 36453179190 aborted inside the spawned server: Assertion failed: (env) != nullptr in node::RemoveEnvironmentCleanupHook, called from better-sqlite3's Statement::~Statement during a garbage collection, on Node 24.21.0.

  • Cause (upstream). Node 24.19.0 backported a header-only change to node::ObjectWrap: the destructor now removes an environment cleanup hook. It did not backport the runtime's global hook list, which lets that removal work during a garbage collection. See nodejs/node#65446. The fix (nodejs/node#65943) is on v24.x-staging and in no release up to 24.21.0. The same abort is reported at WiseLibs/better-sqlite3#1515.
  • Why CI hit it once. The prebuilt better-sqlite3 12.11.1 binaries were built on 2026-06-15 against older headers, and none of them contain the call (checked for linux-x64, win32-x64 and darwin-arm64). In the failing job, npm ci took 1m24s against 4s in the sibling jobs: the prebuild download failed and node-gyp compiled the addon against 24.21.0's headers. A user whose download fails gets the same binary.
  • Frequency. 1 of the 62 failed jobs in the last 400 CI runs (2026-09-21 to 2026-09-28). It is the only one of those jobs whose install compiled the addon.

Characterised in a node:24.21.0 container, better-sqlite3 12.11.1

Statements freed, db open Freed after db.close() Exit paths (exit, natural end, SIGTERM→close→exit, uncaught throw)
Built from source aborted 5/5 aborted 5/5 0/10 each
Prebuilt 0/5 0/5 not run

Finalizing statements before close() does not help, and a statement cache that eventually drops its statements does not either. Any Database, Statement or iterator object that V8 collects triggers it, including the in-memory probe the seam opens. The one fix inside Iris is to not load that binary on a runtime that aborts on it.

Upgrading to 13.x (N-API, not affected) is not safe yet: #308 records spawned-process crashes with 13.0.3.

Changes

  • src/storage/driver.ts: before loading better-sqlite3, nativeAbortsOnCollect finds the better_sqlite3.node that require('better-sqlite3') would load, the same way its bindings loader does, and looks for the RemoveEnvironmentCleanupHook import. No better-sqlite3 source calls it, so its presence means the file was compiled against the new header. runtimeKeepsAddonHooks covers 26.4.0+ and no 24.x yet. When the binary carries the call and the runtime lacks the list, the default falls back to node:sqlite with one warning and a reason. IRIS_SQLITE_DRIVER=native refuses and names npm rebuild better-sqlite3. The scan reads about 2 MB once per process (4 ms measured).
  • src/index.ts: in stdio mode with no HTTP server, the end of stdin runs the same shutdown as SIGINT/SIGTERM. Shutdown is idempotent. Measured on a store with 80,000 unindexed traces: the process used to exit 1.5 s after stdin closed, still building the search index. Now it exits in 32 ms with "Shutdown complete". With the dashboard up, stdin ending does not stop the process, as before.
  • New CI job native addon built from source (os, Node): Node 24 on ubuntu, macOS and Windows, and Node 22 (the control: its headers never changed) on ubuntu and macOS. Windows × Node 22 is excluded because the node-gyp in Node 22's npm 10 cannot identify the runner's Visual Studio 18 (unknown version "undefined" found at "C:\Program Files\Microsoft Visual Studio\18\Enterprise"), so nothing compiles there and npm drops the optional module. The job installs with npm_config_build_from_source=true, checks that the addon was compiled on the runner (and on Node 24, that the call is in it), then runs:
    • tests/integration/native-addon-collect.test.ts: runs the real binary in a child under collection and requires aborted === predicted. On the ordinary test jobs the prebuilt binary survives and is predicted to. When a Node 24 release ships the fix, this test fails with the message to add the version.
    • tests/integration/native-teardown-stdio.test.ts: 6 real server processes over stdio, 25 rounds each of log_trace, evaluate_output and a search, then stdin closed. Each must exit 0 with no signal and no native assertion, log "Shutdown complete", and use the driver Iris predicts. The server runs with --max-semi-space-size=1 so V8 collects inside each short session. With the check turned off, on the compiled binary in the container, this test failed 5 runs of 5 (the server died mid-session). Without the small young generation, the server did not abort in any of 5 runs. That flag is what makes the test catch the bug every time instead of occasionally.
    • tests/unit/storage/driver.test.ts: the version table, marked and unmarked binaries, missing files, the fallback and the refusal, and a check that the inspected file is the one require loads.
    • --self-test, which must name the driver and the reason.
  • README, docs/architecture.md, docs/api-reference.md: the fallback now also covers this build.

Checks

  • Local, Node 24.11.0 on Windows: lint, typecheck and build clean. claims:check, claims:check-hardcoded, llms:check, mcp-json:check, changelog:check, version:check and proof -- --check all pass.
  • Captured tests, rebased on main e78a205: vitest root 3,742 passed, 0 failed (main: 3,734). Dashboard 400 passed, 0 failed. .claims.json totalCombined 4,182.
  • In the container (Node 24.21.0, compiled addon): the new stdio and collect tests pass with the check. The collect test logged aborted (exit SIGABRT); Iris predicted abort.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
website Ready Ready Preview Sep 28, 2026 10:04pm UTC

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Iris gate — 1 of 2 tripped --fail-on detector_veto

iris-eval ingest: 3 stored, 1 tripped --fail-on detector_veto (2 of 3 evaluated in dataset "release-gate")

Trace Basis Rules Evidence
b194dd4244a741022ecb0f7c95176831 detector_veto no_pii no_pii: AWS Access Key (output 45–65)
Verdict basis Traces
detector_veto 2
clean 1

Unjudged questions: task_completed (3), tool_use_correct (3) — a trace that did not carry what a rule needs.

tests/fixtures/ci-gate/traces.ndjson · 3 evaluated · dataset release-gate: 2 in the gate · exit 1 · what the bases mean

@github-actions

Copy link
Copy Markdown

Iris gate — 1 stored, nothing tripped --fail-on any

iris-eval ingest: 1 stored, 0 tripped --fail-on any

Verdict basis Traces
clean 1

Unjudged questions: task_completed (1), tool_use_correct (1) — a trace that did not carry what a rule needs.

tests/fixtures/ci-gate/clean.ndjson · 1 evaluated · exit 0 · what the bases mean

…tement (#719)

Node 24.19.0 changed the header-only node::ObjectWrap: its destructor now
calls RemoveEnvironmentCleanupHook, and no 24.x runtime so far has the
global hook list that makes that safe during a garbage collection
(nodejs/node#65446). A better-sqlite3 compiled against that header, as npm
does when the prebuilt download fails, aborts the process the first time
V8 collects one of its statements, open or closed. The prebuilt 12.11.1
binaries predate the header and are safe.

The driver seam now reads the binary better-sqlite3 would load before
loading it. When it carries the call and the runtime lacks the fix, Iris
holds the store with node:sqlite and says why; IRIS_SQLITE_DRIVER=native
refuses and names npm rebuild better-sqlite3.

The end of stdin now shuts a stdio server down in order, stopping an
in-flight search-index build instead of running it for a client that left.

A new CI job compiles the addon from source on Node 24 (Linux, macOS,
Windows) and Node 22 (Linux, macOS), and requires the abort to happen
exactly when Iris predicts it, and six stdio sessions of the real server to end cleanly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@irparent
irparent force-pushed the fix/native-statement-teardown branch from a0673e5 to 8c2237a Compare September 28, 2026 22:03
@irparent
irparent merged commit a2b3153 into main Sep 28, 2026
64 checks passed
@irparent
irparent deleted the fix/native-statement-teardown branch September 28, 2026 22:12
irparent added a commit that referenced this pull request Sep 28, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
irparent added a commit that referenced this pull request Sep 28, 2026
…where the job's binary would abort Node (#720)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
irparent added a commit that referenced this pull request Sep 28, 2026
…695 erased in steps; checkpoints off the event loop (#727)

* perf: the MCP connection opens before retention runs, and nothing in the background holds the event loop (#695)

- The boot retention sweep starts after the transport connects and deletes
  in steps of about 50 ms, each its own transaction. A large sweep deletes
  with FTS5 secure-delete off and then merges the index to one segment in
  steps; the merge owed is recorded in the same transaction and finished by
  the next start if the server closes first.
- A retired search index is erased a bounded number of rows per step
  (defensive mode off for those DELETEs only), with a one-statement
  fallback for a connection that refuses the writes.
- WAL checkpoints run on a worker thread with its own connection; the
  server's connection checkpoints by itself until it is up and if it fails.
- The index build logs when it starts and finishes; health reports
  search state and progress as a share; --self-test reports the configured
  database's index.
- A stall guard (tests/stall) runs alone in CI on both drivers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf: sweep steps turn automerge off; a refused worker checkpoint is not retried on the event loop; changelog and truthbase

- A merge-mode sweep step turns FTS5 automerge off with secure-delete and
  restores both, so no segment merge lands inside a delete step; the owed
  merge does that work in its own steps.
- checkpoint() with the worker running no longer falls back to a TRUNCATE
  on the server's connection, which waited for readers inside the event
  loop; it is best effort, as before.
- CHANGELOG [Unreleased]: the sweep after connect, #695, checkpoints on a
  worker, health/self-test search status, with measured numbers.
- .claims.json recaptured.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* claims: test counts after merging main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* claims: test counts after merging main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* checkpoint worker: a TRUNCATE never waits for a reader, so delete_trace does not wait for a search (#716); the adapter retries while the reader reads

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* tests: FTS5 integrity-check on both indexes after the stepped erase; defensive mode restored when the work throws, on both drivers

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* claims: test counts

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* claims: test counts after merging main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* stall guard: one retry per case, for the runner's own rare stalls; a stall the code causes repeats

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* delete_trace's erasure stays on the server's connection, as #716 has it; sweeps and purge use the checkpoint worker

A delete's log is a few pages, and its answer must not wait behind the
worker's periodic checkpoint: on a loaded runner a delete during a search
took 122 ms through the worker.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* stall guard: read the main thread's CPU time per gap, not the wall clock, so a runner's descheduling is not read as a statement holding the loop

On the same commit the node job saw 296 and 414 ms wall-clock gaps in
steps that held the thread for under 100 ms. The code before the steps
still fails: 2,125 ms and 280 ms of this thread's CPU at 10,000 traces.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* claims: test counts after merging main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* checkpoint worker: #716's lifecycle; started at the first write, replaced after a crash, never tried again after a failed start, and left to end on its own at close

The thread used to start at open and be terminated right after its
connection closed. A macOS run died once with SIGBUS in the search worker
suite, where each store now had two native threads; the search worker's
pattern never terminates a thread that is ending by itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* claims: test counts

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* claims: test counts after merging main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* tests: the defensive-mode test on better-sqlite3 asserts the refusal where the job's binary would abort Node (#720)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 8c2237a8 Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server aborts on Node 24 when a better-sqlite3 built from source frees a statement

1 participant