Skip to content

Process contract: a bounded drain that exits 15, and a four-state health endpoint - #273

Merged
turbolytics merged 14 commits into
mainfrom
feat/process-contract
Sep 13, 2026
Merged

Process contract: a bounded drain that exits 15, and a four-state health endpoint#273
turbolytics merged 14 commits into
mainfrom
feat/process-contract

Conversation

@turbolytics

@turbolytics turbolytics commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Closes #161. Rebased on #279, which fixed the offset commit after a failed batch that the review of this PR found on main.

Design: docs/superpowers/specs/2026-09-12-process-contract-design.md
Plan: docs/superpowers/plans/2026-09-12-process-contract.md

What changes

  • Drain deadline. pipeline.drain_deadline_seconds, default 30, bounds the whole shutdown after SIGTERM. One core.DrainBudget is shared by the batch in flight, the drain of what is still buffered, both state syncs and every manager's final poll. When it passes, the process exits 15 with system.lifecycle.drain_incomplete. Retryable: nothing unwritten was committed, and the next start replays it.
  • A SIGTERM during a flush lets the flush finish. Every batch runs on a context that survives the cancel until the drain deadline. Before, the cancel aborted the flush at once and the process exited with the sink's error.
  • Managers. A cancel during a regular poll still runs the final poll. A final poll that fails is the run's error, with the drain code when the deadline ended it. Before, it was logged and the process exited 0.
  • Health. /healthz reports starting, healthy, degraded or failed with a reason. failed answers 503 and the rest answer 200. The old ok and stuck bodies become healthy and failed, and their HTTP codes do not change. Retries are tracked per sink as role/type.
  • Validate. A warning when the retry ladders of every laddered sink, summed, are longer than the drain deadline.
  • Invariants. lifecycle.drain.bounded loses tracked_by: #161 and is enforced. manager.drain.bounded is added and enforced.

Decisions made during implementation

  • The drain and retry defaults and the list of retrying sink types moved into internal/config. validate compares them and must not link DuckDB.
  • system.lifecycle.internal is added beside drain_incomplete, because the registry requires a catch-all in every domain.
  • Rollbacks run on context.WithoutCancel. The DuckDB driver ignores the context, so this changes nothing there; it guards a transaction that honours it, and a test with one covers it.
  • The harness's hanging sink has a watchdog, so an unbounded drain fails the invariant instead of hanging the suite.

Adversarial review

An adversarial review of the first version found three critical and three important defects. Two of the critical ones were the offset bug on main, fixed in #279. The rest are fixed here: the flush aborted by a cancel, the manager's final poll skipped or its failure ignored, the health wiring untested, the retry key shared between sinks of one type, the validate branches untested, one failure counted three times, and a rollback "fix" that DuckDB never needed described as one.

Verified

  • go test -short -race ./..., go vet, gofmt, pytest tests/tooling: clean.
  • Mutation: with the turbine's drain reverted to an unbounded context, both pipeline subjects fail lifecycle.drain.bounded in 5 seconds.
  • Branch binary on local Kafka, stateful window example: /healthz healthy with idle-tick resets; SIGTERM exits 0 with 300 of 300 rows and offset 299.
  • Branch binary with ClickHouse paused mid-drain and drain_deadline_seconds: 2: validate warned beforehand; SIGTERM exited 15 after 2 seconds; 0 rows written.

Coverage status files follow from this PR's CI report artifacts.

@turbolytics turbolytics changed the title Process contract: design and plan for #161 Process contract: a bounded drain that exits 15, and a four-state health endpoint Sep 13, 2026
@turbolytics
turbolytics marked this pull request as ready for review September 13, 2026 14:58
@turbolytics
turbolytics marked this pull request as draft September 13, 2026 19:17
@turbolytics
turbolytics marked this pull request as ready for review September 13, 2026 20:32
@turbolytics
turbolytics merged commit 600468b into main Sep 13, 2026
5 checks passed
turbolytics added a commit that referenced this pull request Sep 13, 2026
…ates

CHANGELOG.md: main now keeps an Unreleased section. serve's entries join
its Added list, and serve's limits sit under Known limits, rather than
under a v1.3.0 heading of their own. The version is named when it is
tagged.

internal/errs/errs_test.go: both sides appended a test at the end of the
file. Both are kept.

internal/validate/validate.go merged without a conflict and with a bug:
#273's checkDrainDeadline ran on every file, so a serve file reported
pipeline.drain_deadline: pass for a pipeline it does not have. A check
that could not apply must not read as a pass. Each file now gets its own
checks: serve rules for a serve file, the drain deadline for a pipeline.
TestValidateServe_AValidServeFilePasses now asserts no pipeline check
reports on a serve file; it failed on the merged code and passes now.

docs/coverage/matrix.md merged as text into a page that no longer matched
its status files, and test_the_committed_page_is_current failed. make
coverage-page regenerated it: 39 features, cli.serve covered at unit and
release, 0 gaps.

go vet ./..., go test -short -race ./... and pytest tests/tooling pass on
the merge.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Process contract: documented exit codes, drain deadline, health endpoint

1 participant