Windows: reemit publishes the late rows alone, so the Postgres example drops - #288
Merged
Merged
Conversation
…e drops bluesky.postgres.windowed.yml paired late_rows: reemit with ON CONFLICT (bucket, lang) DO UPDATE SET posts = EXCLUDED.posts. A close deletes the bucket's rows, so reemit runs emit_sql over the late rows alone, and the upsert replaced a published minute's count with theirs. Against the v2026.09.14 image, a webhook source and Postgres 18: five posts closed a minute, one late post arrived for it, and the row went from 5 to 1. The same run under drop kept 5, with window_late_rows_total at 1. The example now declares drop. The README, the config schema, the example comments, ParseLatePolicy's message and validate's reemit warning said reemit publishes the bucket again for a sink that upserts, which is the pairing that loses the count. They now say reemit publishes emit_sql over the late rows, for a sink that adds them to the bucket it holds. TestManagerWindow_ReemitPublishesTheLateRowsAlone pins the behaviour the docs now describe: a bucket closes with a sum of 5, a late row arrives, and the second flush carries 1. With the close's delete removed, so reemit republishes the whole bucket, the test fails with 6. If this is wrong, the docs steer a user with a replacing upsert away from reemit when reemit would have kept the count, or the example undercounts by every late post it drops.
turbolytics
force-pushed
the
fix/window-reemit-example
branch
from
September 14, 2026 11:03
87c2937 to
58e5bd3
Compare
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
bluesky.postgres.windowed.ymlpairslate_rows: reemitwith an upsert that replaces the count:A close deletes the bucket's rows, so
reemitrunsemit_sqlover the late rows alone. The upsert then replaces a published minute's count with the late rows' count. One late post replaces the minute's count with 1.late_rows: drop.ParseLatePolicy's message andvalidate'sreemitwarning saidreemitpublishes the bucket again, for a sink that upserts. That is the pairing that loses the count. They now sayreemitpublishesemit_sqlover the late rows, for a sink that adds them to the bucket it holds.TestManagerWindow_ReemitPublishesTheLateRowsAlonepins the behaviour the docs describe.If this is wrong, the docs steer a user with a replacing upsert away from
reemitwhen it would have kept the count.Verification
go test -short -race ./...: pass, noFAILlines.go vet ./...is clean.uv run --locked pytest tests/tooling -q: 205 passed.make coverage-page && git status --short docs/coverage: clean.uv run --locked pytest tests/release -q: not run. No flag, CLI command or build stamp changed; the schema changed one description.make soak: not run.watermark.gochanged a comment and an error string, and nothing on the consume loop, a handler, a sink or the allocation path changed.The late-row claim, against the published
turbolytics/sql-flow:v2026.09.14, a webhook source, and a replacing upsert into Postgres 18. Five posts closed a minute, then one late post arrived for it:late_rowswindow_late_rows_totalreemitdropThe new test fails when
reemitrepublishes the whole bucket. With the close's delete removed:Soak verdict
Not run; see above.
Notes for the reviewer
reemit"is today's behaviour, for sinks that upsert". The old predicates overwrote the same way: a late row matched the collect on the next poll and replaced the count. The spec is a record of the design, so this PR leaves it as written.SET posts = t.posts + EXCLUDED.posts, makesreemitcomplete but counts a republished close twice. No sink makesreemitexact today; the README says so by pointing at Guarantees.## Unreleased, above the heading changelog: name the v2026.09.14 section #286 namesv2026.09.14. Whichever merges second keeps both.drop: pipeline: declare the window for sql-flow v2026.09.14 sql-flow-bluesky-demo#4.