sinks: a value the driver cannot encode fails once, as a user error - #269
Merged
Conversation
A ClickHouse value the driver refuses in Append was returned uncoded, so the retry ladder retried it and reported system.sink.unreachable (#233). user.sink.encode_failed names the fault: a payload value the user's SQL shaped, terminal, exit 10. type_unsupported stays for a column whose Arrow type the sink cannot convert. That is a schema property and fails for every row. encode_failed is one value in a supported column. An operator reading type unsupported for a badly formatted string looks at the wrong thing. The registry is append-only; codes.golden gains one line. What breaks if this is wrong: a supervisor restarts a pipeline into the same bad value forever, because the exit code said retryable.
retryable listed three codes it would not retry and retried everything else. user.sink.type_unsupported was not on the list, so a column the ClickHouse sink could not convert was re-encoded four times and then reported as system.sink.unreachable. Nothing was unreachable, and the exit code told a supervisor to restart into the same failure (#233). The list leaked more widely than that one code. Walking the registry shows user.config.not_found took the ladder too. Any user-class error now makes one attempt, and so does write_failed. Unreachable and uncoded errors are retried as before; a driver timeout arrives without a code and is exactly what the ladder exists for. TestSinkRetry_NoUserCodeIsRetried walks the registry, so a user code added later cannot fall back into the ladder. TestSinkRetry_UncodedErrorRunsTheWholeLadder pins the other direction. What breaks if this is wrong: a transient fault some sink coded as user stops being retried. No sink does that today, and the tests would show it.
batch.Append failed on an ISO 8601 timestamp bound for a DateTime column and the sink returned the error with no code. The ladder retried it four times and reported system.sink.unreachable for a server that was up (#233). Append validates and buffers in memory. clickhouse-go v2.48.0's native and HTTP batches both call block.Append and return; neither touches the connection until Send. The value cannot succeed on a later attempt, so it is wrapped with user.sink.encode_failed at the driver boundary. The driver's message, which names the column and quotes the value, survives the wrap. PrepareBatch and Send keep sinkError, which is where bytes cross the network and where unreachable is a real answer. The live ClickHouse tests ran rather than skipped. What breaks if this is wrong: Append had a failure mode a retry could fix, and that batch is now reported terminal after one attempt.
The README did not document sink.retry at all, and nothing stated which failures the ladder retries. An operator reading system.sink.unreachable after four attempts on a bad value had no way to learn the rule (#233). Sink retries is a new README subsection: the config block, the deadline's relation to the flush interval, and a table of what is retried by error code. The retryable doc comment carries the same table and names the section. The sink.error.classifies invariant names the user-fault class. Every value is checked against the code rather than restated: the four codes against codes.golden, exits 10 and 12 against exit.go, the metric name against the Metrics table, the defaults against policy.go. The spec and plan for this change land here too. What breaks if this is wrong: the README and the code disagree about what is retried, and the operator trusts the README.
make coverage-page renders each invariant's claim into matrix.md. The sink.error.classifies claim changed in the previous commit, so the page was stale and the coverage-check gate would fail on it. The row's status is unchanged. It was missing for every sink before this branch, because the invariant is verified_by harness and no conformance test covers it yet; the unit tests added here do not flip a harness cell. What breaks if this is wrong: the stale-page gate fails in CI.
turbolytics
added a commit
that referenced
this pull request
Sep 12, 2026
turbolytics
added a commit
that referenced
this pull request
Sep 12, 2026
…ormance subject (#270) * Managers: a refused poll stops the process, and the manager is a conformance subject Fixes #267. The tumbling window manager logged a failed poll and polled again. A window the destination rejected was collected, written and refused every tick for as long as the process lived, with the container reporting healthy throughout. Observed against the live Bluesky firehose: three minutes, a failure every ten seconds, zero rows written, container Up. Start now returns the first failed poll, and run treats a manager's error as the pipeline's: it cancels the consume loop, which drains as it would on SIGTERM, and the process exits with the manager's error. The sink ran its retry ladder before the error reached the manager, so what arrives is final, and a retry in place repeats one the ladder already exhausted. The rows stay in the state table, because Poll deletes only after the sink accepted them, so a restart republishes the window. The manager is its own kind in the registry. It is not a sink and not a handler: it collects, flushes and deletes, and the delete is its commit. So its invariants are the consume loop's checkpoint claims restated for that commit, plus the liveness a poll loop owes on its own: - manager.delete.after_flush: the state table still holds every closed window when Flush runs, and none once the poll returns. - manager.delete.nothing_on_failure: a failed flush deletes nothing. - manager.publish.eventually: the loop publishes a closed window on its own. - manager.failure.exits: a refused poll stops the loop after one attempt. internal/conformance drives a ManagerSubject the way it drives a PipelineSubject: the harness owns a recording sink, fails its flush, and judges the event order and the state table. The tumbling manager supplies build, seed and count. All four are enforced. The status file for the new integration follows from CI's reports. * coverage: the manager's status file, from CI's reports * coverage: regenerate the page after rebasing on #269
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A value the sink's driver could not encode was retried the full backoff ladder and then reported as
system.sink.unreachable(#233). Nothing was unreachable. The destination was up, the fault was one value, and the exit code told a supervisor to restart into the same failure.Two layers caused it:
Appenderror with no code.retryablelisted three codes it would not retry and retried everything else.user.sink.type_unsupportedwas not on the list. Walking the registry showsuser.config.not_foundtook the ladder too.The fix:
user.sink.encode_failed, names a value the sink's client refuses before anything reaches the network. It is classuserand exits 10.retryabledecides by class. Anyusercode andsystem.sink.write_failedmake one attempt.system.sink.unreachableand uncoded errors are retried as before, because a driver timeout arrives without a code.Appenderror with the new code.PrepareBatchandSendkeepsinkError, because that is where bytes cross the network.retryabledoc comment carries the same table and names the section.What breaks if this is wrong: a transient fault that some sink coded as
userstops being retried. No sink does that today.TestSinkRetry_NoUserCodeIsRetriedwalks the registry, so a user code added later cannot fall back into the ladder.TestSinkRetry_UncodedErrorRunsTheWholeLadderpins the other direction.Closes #233.
Verification
go test -short -race ./...passes with no data races. The only output is pre-existing macOSLC_DYSYMTABlinker warnings.uv run --locked pytest tests/tooling -q: 205 passed.make coverage-page && git status --short docs/coverageis clean, after committing the regeneratedmatrix.md, which renders the edited invariant claim.uv run --locked pytest tests/release -qwas not run. No CLI surface, flag, Dockerfile, or build-stamped value changed.make soakwas not run, because it cannot observe this change.scripts/soak.shrunsdev/config/soak/inferred.noop.ymlby design, and the noop sink is not wrapped in the retry ladder. No line in this diff executes under the soak. Both changed paths run only when a flush fails, so the happy path allocates as before.The live ClickHouse tests ran rather than skipped.
TestSinkClickhouse_InsertsRows,InsertsArraysandStringTemporalsAreNotShiftedByHostZonepassed against a local server.Acceptance: the issue's own repro
The config from #233, run against a local ClickHouse
DateTimecolumn, with one message{"label":"isoz","ts":"2026-09-01T12:00:00Z"}:The run logged no retry lines, and the log records
error.classasuser. The issue reported the same repro on v1.0.6 as four attempts andsystem.sink.unreachable, which maps to exit 12.Notes for the reviewer
user.sink.encode_failedis separate fromuser.sink.type_unsupported. Type unsupported is a column whose Arrow type the sink cannot convert. That is a schema property and fails every row. Encode failed is one value in a supported column. An operator reading "type unsupported" for a badly formatted string would look at the wrong thing.AppendTablecall, and its conversion errors cannot be told from I/O errors without string matching. A false "permanent" on a transient Iceberg fault would drop a batch, which is worse than wasted attempts.temporalFromStringis a type-matrix change with an open Python-parity question, left as a follow-up.sink.error.classifiescoverage row stays missing for every sink. The invariant isverified_by: harness, and unit tests do not fill a harness cell. That was already true before this branch.docs/superpowers/.