Skip to content

Worker-exit test: Iris's threads start with no --import, as an installed server's do - #831

Merged
irparent merged 2 commits into
mainfrom
test/worker-exit-dist-runs-as-installed
Oct 4, 2026
Merged

irparent merged 2 commits into
mainfrom
test/worker-exit-dist-runs-as-installed

Conversation

@irparent

@irparent irparent commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

What this changes

The worker-exit job's Windows, Node 24 cell has failed twice on main. Each time, one process in the "iris dist native checkpoint-kill" scenario exited with an access violation (#804). This finds the cause. It is a Node.js bug, and the test reached it through a configuration an installed server never runs.

The crash is in Node, not in Iris or SQLite

Eight crash dumps, symbolized against Node 24.21.0's published symbols, fault at the same instruction. The fault comes as the thread is disposed of (node_worker.cc, ~WorkerThreadData):

  • The isolate's heap is torn down.
  • The garbage collector's last sweep runs the finalizer of a node::contextify::ContextifyScript.
  • That finalizer writes to an address that is no longer valid.

This is the use-after-free in nodejs/node#65778:

The thread's environment has already been freed when this happens, and better-sqlite3 closes a thread's connections as its environment is freed. The databases left behind by the eleven processes that crashed during this work all pass PRAGMA integrity_check.

The test put tsx's loader in the threads; a server does not

The fixture ran as node --import tsx iris-workers.ts, and a worker thread inherits its process's --import (checked on Node 22.23.3 and 24.21.0). So in the "dist" cells, which are meant to start Iris's threads as an installed server does, every thread also started with tsx's loader.

On Windows with Node 24.21.0, checkpoint-kill from the build, 24 processes at a time, with six more processes writing and syncing 4 MB files to load the disk:

how the process started processes crashed
--import tsx, as the job did 120 6
no --import, as a server does 480 0
  • 120 of the 480 ran interleaved with the 120 above; 360 more ran after.
  • Without the extra load, the crash is rare on this machine: 3 in 596 processes, on Node 24.11.0 and 24.21.0. All three came while a full test run was also writing to the disk. CI's runners met it in 2 of 15 runs.
  • The "src" cells did not crash either way, in 120 processes each. Their threads start through sourceThread, which registers tsx inside the thread.

The change

  • stress.cjs starts iris-workers.ts with no --import. Node strips the file's types itself; a Node that does not do this by default gets --experimental-strip-types.
  • iris-workers.ts registers tsx on its own main thread, and only for the "src" cells. The "dist" cells' threads now load checkpoint-worker.js and search-worker.js with no loader, as the fixture's comment always said they did.
  • source-thread.ts: a comment said a thread does not take --import. It does; the comment now says why the thread still registers tsx itself.
  • worker-exit.yml: a note on the above.
  • CHANGELOG.md and the website's copy of it: the 0.20.0 entry said the crash came "when close() ended a checkpoint thread inside a statement". It now names the cause. The release page carries only that entry's lead, so it needs no change.

No product code changes.

Checks

  • All four endings pass from the sources and from the build, with Node's own type stripping:
    • Node 24.21.0, both drivers;
    • Node 22.23.3, node:sqlite (this machine's better-sqlite3 is built for Node 24).
  • lint, typecheck and typecheck:tests pass.
  • The job's full stress step (WORKER_EXIT_SEARCH=1 node stress.cjs), run locally on Windows with Node 24.21.0: all 30 rows clean, 10 of 10 processes each. That covers every raw ending on both drivers, and Iris's two workers from the sources and from the build.
  • The tests that read the changelog pass, and changelog:check confirms the website's copy matches.

🤖 Generated with Claude Code

…led server's do

The fixture ran as `node --import tsx`, and a worker thread inherits its
process's --import, so the "dist" cells started every thread with tsx's
loader. On Node 24 such a thread can crash the process as it ends, in
Node's own teardown of the thread (nodejs/node#65778, fixed in 26.10.0,
backport to 24.x pending). An installed server's threads have no loader.

The harness now starts the fixture with no --import (Node strips its
types itself) and the fixture registers tsx on its main thread only, for
the "src" cells.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
website Ready Ready Preview Oct 4, 2026 3:47pm UTC

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Iris gate — 1 of 2 tripped --fail-on detector_veto

iris-eval ingest: 3 stored, 1 tripped --fail-on detector_veto (2 of 3 evaluated in dataset "release-gate")

Trace Verdict Basis Rules, classes or missing inputs Evidence
93f5920c0f94ec14f5e2c3a899940dc5 failed detector_veto + risk_over_loss no_pii, pii_leak, credential_leak no_pii: AWS Access Key (output 45–65)
Verdict basis Traces
detector_veto 2
clean 1

Unjudged questions: task_completed (3), tool_use_correct (3) — a trace that did not carry what a rule needs.

tests/fixtures/ci-gate/traces.ndjson · 3 evaluated · dataset release-gate: 2 in the gate · exit 1 · what the bases mean

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

Iris gate — 1 stored, nothing tripped --fail-on any

iris-eval ingest: 1 stored, 0 tripped --fail-on any

Verdict basis Traces
clean 1

Unjudged questions: task_completed (1), tool_use_correct (1) — a trace that did not carry what a rule needs.

tests/fixtures/ci-gate/clean.ndjson · 1 evaluated · exit 0 · what the bases mean

It is a Node.js bug in a thread's teardown (nodejs/node#65778), which
the job reached by starting its threads with tsx's loader; it is not
close() ending a checkpoint thread inside a statement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@irparent
irparent merged commit cc01669 into main Oct 4, 2026
80 checks passed
@irparent
irparent deleted the test/worker-exit-dist-runs-as-installed branch October 4, 2026 16:27

This branch was successfully deployed

1 active deployment
Preview — d4a4655c Deployed Oct 4, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant