Skip to content

sinks: a value the driver cannot encode fails once, as a user error - #269

Merged
turbolytics merged 5 commits into
mainfrom
fix/sink-encode-failed
Sep 12, 2026
Merged

sinks: a value the driver cannot encode fails once, as a user error#269
turbolytics merged 5 commits into
mainfrom
fix/sink-encode-failed

Conversation

@turbolytics

Copy link
Copy Markdown
Owner

What this changes

A value the sink's driver could not encode was retried the full backoff ladder and then reported as system.sink.unreachable (#233). Nothing was unreachable. The destination was up, the fault was one value, and the exit code told a supervisor to restart into the same failure.

Two layers caused it:

  • The ClickHouse sink returned the driver's Append error with no code.
  • retryable listed three codes it would not retry and retried everything else. user.sink.type_unsupported was not on the list. Walking the registry shows user.config.not_found took the ladder too.

The fix:

  • A new code, user.sink.encode_failed, names a value the sink's client refuses before anything reaches the network. It is class user and exits 10.
  • retryable decides by class. Any user code and system.sink.write_failed make one attempt. system.sink.unreachable and uncoded errors are retried as before, because a driver timeout arrives without a code.
  • The ClickHouse sink wraps the Append error with the new code. PrepareBatch and Send keep sinkError, because that is where bytes cross the network.
  • The README gains a "Sink retries" subsection. It documents the retry block and states the rule as a table keyed by error code. The retryable doc comment carries the same table and names the section.

What breaks if this is wrong: a transient fault that some sink coded as user stops being retried. No sink does that today. TestSinkRetry_NoUserCodeIsRetried walks the registry, so a user code added later cannot fall back into the ladder. TestSinkRetry_UncodedErrorRunsTheWholeLadder pins the other direction.

Closes #233.

Verification

  • go test -short -race ./... passes with no data races. The only output is pre-existing macOS LC_DYSYMTAB linker warnings.
  • uv run --locked pytest tests/tooling -q: 205 passed.
  • make coverage-page && git status --short docs/coverage is clean, after committing the regenerated matrix.md, which renders the edited invariant claim.
  • uv run --locked pytest tests/release -q was not run. No CLI surface, flag, Dockerfile, or build-stamped value changed.
  • make soak was not run, because it cannot observe this change. scripts/soak.sh runs dev/config/soak/inferred.noop.yml by design, and the noop sink is not wrapped in the retry ladder. No line in this diff executes under the soak. Both changed paths run only when a flush fails, so the happy path allocates as before.

The live ClickHouse tests ran rather than skipped. TestSinkClickhouse_InsertsRows, InsertsArrays and StringTemporalsAreNotShiftedByHostZone passed against a local server.

Acceptance: the issue's own repro

The config from #233, run against a local ClickHouse DateTime column, with one message {"label":"isoz","ts":"2026-09-01T12:00:00Z"}:

$ ./bin/sqlflow run iso-probe.yml --max-msgs 1
Error: [user.sink.encode_failed] clickhouse sink: encode row: clickhouse [AppendRow]: dt_plain parsing time "2026-09-01T12:00:00Z" as "2006-01-02 15:04:05": cannot parse "T12:00:00Z" as " "
$ echo $?
10

The run logged no retry lines, and the log records error.class as user. The issue reported the same repro on v1.0.6 as four attempts and system.sink.unreachable, which maps to exit 12.

Notes for the reviewer

  • user.sink.encode_failed is separate from user.sink.type_unsupported. Type unsupported is a column whose Arrow type the sink cannot convert. That is a schema property and fails every row. Encode failed is one value in a supported column. An operator reading "type unsupported" for a badly formatted string would look at the wrong thing.
  • Iceberg is unchanged and still retries an encode failure. iceberg-go converts and writes in one AppendTable call, and its conversion errors cannot be told from I/O errors without string matching. A false "permanent" on a transient Iceberg fault would drop a batch, which is worse than wasted attempts.
  • The issue's ISO 8601 timestamp still fails. It now fails fast and clearly. Accepting RFC 3339 in temporalFromString is a type-matrix change with an open Python-parity question, left as a follow-up.
  • The sink.error.classifies coverage row stays missing for every sink. The invariant is verified_by: harness, and unit tests do not fill a harness cell. That was already true before this branch.
  • The spec and plan are under docs/superpowers/.

A ClickHouse value the driver refuses in Append was returned uncoded, so
the retry ladder retried it and reported system.sink.unreachable (#233).
user.sink.encode_failed names the fault: a payload value the user's SQL
shaped, terminal, exit 10.

type_unsupported stays for a column whose Arrow type the sink cannot
convert. That is a schema property and fails for every row. encode_failed
is one value in a supported column. An operator reading type unsupported
for a badly formatted string looks at the wrong thing.

The registry is append-only; codes.golden gains one line.

What breaks if this is wrong: a supervisor restarts a pipeline into the
same bad value forever, because the exit code said retryable.
retryable listed three codes it would not retry and retried everything
else. user.sink.type_unsupported was not on the list, so a column the
ClickHouse sink could not convert was re-encoded four times and then
reported as system.sink.unreachable. Nothing was unreachable, and the
exit code told a supervisor to restart into the same failure (#233).

The list leaked more widely than that one code. Walking the registry
shows user.config.not_found took the ladder too.

Any user-class error now makes one attempt, and so does write_failed.
Unreachable and uncoded errors are retried as before; a driver timeout
arrives without a code and is exactly what the ladder exists for.

TestSinkRetry_NoUserCodeIsRetried walks the registry, so a user code
added later cannot fall back into the ladder.
TestSinkRetry_UncodedErrorRunsTheWholeLadder pins the other direction.

What breaks if this is wrong: a transient fault some sink coded as user
stops being retried. No sink does that today, and the tests would show it.
batch.Append failed on an ISO 8601 timestamp bound for a DateTime column
and the sink returned the error with no code. The ladder retried it four
times and reported system.sink.unreachable for a server that was up (#233).

Append validates and buffers in memory. clickhouse-go v2.48.0's native and
HTTP batches both call block.Append and return; neither touches the
connection until Send. The value cannot succeed on a later attempt, so it
is wrapped with user.sink.encode_failed at the driver boundary. The
driver's message, which names the column and quotes the value, survives
the wrap.

PrepareBatch and Send keep sinkError, which is where bytes cross the
network and where unreachable is a real answer.

The live ClickHouse tests ran rather than skipped.

What breaks if this is wrong: Append had a failure mode a retry could fix,
and that batch is now reported terminal after one attempt.
The README did not document sink.retry at all, and nothing stated which
failures the ladder retries. An operator reading system.sink.unreachable
after four attempts on a bad value had no way to learn the rule (#233).

Sink retries is a new README subsection: the config block, the deadline's
relation to the flush interval, and a table of what is retried by error
code. The retryable doc comment carries the same table and names the
section. The sink.error.classifies invariant names the user-fault class.

Every value is checked against the code rather than restated: the four
codes against codes.golden, exits 10 and 12 against exit.go, the metric
name against the Metrics table, the defaults against policy.go.

The spec and plan for this change land here too.

What breaks if this is wrong: the README and the code disagree about what
is retried, and the operator trusts the README.
make coverage-page renders each invariant's claim into matrix.md. The
sink.error.classifies claim changed in the previous commit, so the page
was stale and the coverage-check gate would fail on it.

The row's status is unchanged. It was missing for every sink before this
branch, because the invariant is verified_by harness and no conformance
test covers it yet; the unit tests added here do not flip a harness cell.

What breaks if this is wrong: the stale-page gate fails in CI.
@turbolytics
turbolytics merged commit 4b40a5c into main Sep 12, 2026
5 checks passed
turbolytics added a commit that referenced this pull request Sep 12, 2026
…ormance subject (#270)

* Managers: a refused poll stops the process, and the manager is a conformance subject

Fixes #267.

The tumbling window manager logged a failed poll and polled again. A window
the destination rejected was collected, written and refused every tick for
as long as the process lived, with the container reporting healthy
throughout. Observed against the live Bluesky firehose: three minutes, a
failure every ten seconds, zero rows written, container Up.

Start now returns the first failed poll, and run treats a manager's error
as the pipeline's: it cancels the consume loop, which drains as it would on
SIGTERM, and the process exits with the manager's error. The sink ran its
retry ladder before the error reached the manager, so what arrives is
final, and a retry in place repeats one the ladder already exhausted. The
rows stay in the state table, because Poll deletes only after the sink
accepted them, so a restart republishes the window.

The manager is its own kind in the registry. It is not a sink and not a
handler: it collects, flushes and deletes, and the delete is its commit. So
its invariants are the consume loop's checkpoint claims restated for that
commit, plus the liveness a poll loop owes on its own:

- manager.delete.after_flush: the state table still holds every closed
  window when Flush runs, and none once the poll returns.
- manager.delete.nothing_on_failure: a failed flush deletes nothing.
- manager.publish.eventually: the loop publishes a closed window on its own.
- manager.failure.exits: a refused poll stops the loop after one attempt.

internal/conformance drives a ManagerSubject the way it drives a
PipelineSubject: the harness owns a recording sink, fails its flush, and
judges the event order and the state table. The tumbling manager supplies
build, seed and count. All four are enforced.

The status file for the new integration follows from CI's reports.

* coverage: the manager's status file, from CI's reports

* coverage: regenerate the page after rebasing on #269
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A value the driver cannot encode is retried, then reported as sink.unreachable

1 participant