StructuredBatch skips a checkpoint another connection's write refuses

[turbolytics]

What this changes

The bluesky demo's first deploy of v2026.09.14 exited minutes after starting:

ERROR sqlflow.run core/turbine.go:907 error reinitializing handler
  {"error": "checkpoint after truncate: Invalid Argument: TransactionContext Error: Cannot CHECKPOINT:
   there are other write transactions active. Try using FORCE CHECKPOINT to wait until all active
   transactions are finished"}

handlers.StructuredBatch checkpoints after every batch to reclaim the rows its truncate left (#247). DuckDB refuses a checkpoint while another connection holds an uncommitted UPDATE or DDL. #281 moved the window to connections of its own: its watermark save is an UPDATE, and the sqlcommand sink drops and creates sqlflow_sink_batch. A batch that re-initialised while either was open failed, and a failed batch stops the process.

• Fix. Init skips a refused checkpoint, and the next batch's checkpoint reclaims what it left. Any other checkpoint error still fails the batch.
• Signal. Each skip logs at debug and counts in handler_checkpoints_skipped_total. A write held open for longer than a batch shows as a rising count, not as memory growth nobody can explain.
• Invariant. pipeline.batch.independent_of_window_io, enforced: a window's sink write or close, however long, never fails the consume loop. It applies to manager.watermark, and the manager harness proves it with the structured handler. Every statement on the pipeline's DuckDB connection holds the shared … #283's rule, every statement on the pipeline's connection holds its lock, stopped reaching the window when it moved, and no row said so. A dropped check now leaves an enforced cell uncovered, and make coverage-check fails.

If this is wrong, a windowed StructuredBatch pipeline still exits when a close coincides with a batch. Or it skips checkpoints indefinitely and grows the way #247 did, with the counter rising.

Verification

• go test -short -race ./...: exit 0. go vet ./... is clean.
• uv run --locked pytest tests/tooling -q: 205 passed.
• make coverage-page && git status --short docs/coverage: clean after the commit. The page gains the invariant row, ✅ u for manager.watermark, and the status file gains its line.
• uv run --locked pytest tests/release -q: not run. No flag, command or build stamp changed. The new series is asserted by TestExportedSeriesNames.
• make soak against turbolytics/sql-flow:checkpoint-busy-test, built from this branch at 5deb256, before d025729 restored the deferred unlock in initHandler: PASS, verdict below. The soak drives handlers.InferredMemBatch, so it measures the consume loop's initHandler change, not the skip.

What refuses a checkpoint. A probe against DuckDB 1.5.2: a statement on a second connection, left uncommitted, then CHECKPOINT on the first.

Open on the other connection	CHECKPOINT
Nothing, or a read	ok
INSERT, including a long-running one	ok
DELETE of committed rows	ok
UPDATE of committed rows	refused
CREATE TABLE, DROP TABLE	refused

Checkpoints refused in 3 seconds of the loop's shape (insert, truncate, checkpoint) beside each workload:

Beside the loop	Refused
Nothing	0 of 3,571
A manager connection only reading	0 of 2,937
UPDATE then COMMIT in a loop	2,242 of 3,992
The watermark manager, recording sink	223 of 3,171
The watermark manager, sqlcommand sink	2,721 of 4,298

Tests that fail without the fix:

• The harness check pipeline.batch.independent_of_window_io first holds the window's flush open and runs three batches. Then it holds a close after its delete and watermark UPDATE, before the commit, and runs three more. Against main:

  --- FAIL: TestManagerWatermark_Conformance/pipeline.batch.independent_of_window_io
      manager.go:106: batch 1 of 3, run while the window held its uncommitted delete and watermark, failed:
      checkpoint after truncate: ... Cannot CHECKPOINT: there are other write transactions active ...
  

  With the fix both holds pass, and both closes land once released.
• TestStructuredInit_SkipsACheckpointAnotherWriterHolds holds an uncommitted UPDATE on a second connection for 50 batches. Init returns no error and CheckpointSkipped() is true. Storage grows from 0 to 4,422 KiB; one batch after the writer commits, it is back to 0 KiB.
• TestCoreConsumeLoop_CountsSkippedCheckpoints: a handler that reports skips increments the counter, and one that does not records nothing.

Soak verdict

window        minutes 3-10, 105,160,945 messages
native        22.6 -> 21.2 MiB
go retained   23.1 -> 25.1 MiB
native slope  -0.123 MiB/min
go slope      +0.286 MiB/min
per message   -0.014 B/msg (threshold 1.0)

PASS  memory is flat over 105,160,945 messages

A first run printed PASS over a dead process and does not count. Docker's disk filled at minute 7, the Kafka broker exited with No space left on device, and the analyzer read the missing samples from minutes 7 to 10 as 0.0 MiB. The run above had every sample, the container logged no errors, and the broker stayed up.

End to end

The bluesky demo's worker config on the same image, against compose Postgres, capped at --cpus 0.5 like the Render plan, from 14:30 to 15:47 UTC:

• 76 consecutive minutes published, 14:30 through 15:45, no gap, no ERROR line.
• handler_checkpoints_skipped_total 1. That collision is the one that stopped the demo on v2026.09.14; here the batch continued.

Notes for the reviewer

• Skip, not wait. FORCE CHECKPOINT aborts the other transactions, so the window's close would fail. A retry loop would stall the consume loop behind a sink's network write. The skip costs one batch of reclamation.
• Matching the message. DuckDB gives the refusal no code of its own, so checkpointRefused matches the message text.
• A second test hook. HoldCommit wraps the manager's connection so its commit waits. NewWatermark takes the transaction from the connection, so the harness can stop a close at that point without a hook in the manager.
• Conflicts. The CHANGELOG adds ## Unreleased above ## v2026.09.14, as Windows: reemit publishes the late rows alone, so the Postgres example drops #288 does. Whichever merges second keeps both entries.