Skip to content

Windows: reemit publishes the late rows alone, so the Postgres example drops - #288

Merged
turbolytics merged 1 commit into
mainfrom
fix/window-reemit-example
Sep 14, 2026
Merged

turbolytics merged 1 commit into
mainfrom
fix/window-reemit-example

Conversation

@turbolytics

@turbolytics turbolytics commented Sep 14, 2026 •

Copy link
Copy Markdown
Owner

What this changes

bluesky.postgres.windowed.yml pairs late_rows: reemit with an upsert that replaces the count:

ON CONFLICT (bucket, lang) DO UPDATE SET posts = EXCLUDED.posts

A close deletes the bucket's rows, so reemit runs emit_sql over the late rows alone. The upsert then replaces a published minute's count with the late rows' count. One late post replaces the minute's count with 1.

  • The example declares late_rows: drop.
  • The README, the config schema, the example comments, ParseLatePolicy's message and validate's reemit warning said reemit publishes the bucket again, for a sink that upserts. That is the pairing that loses the count. They now say reemit publishes emit_sql over the late rows, for a sink that adds them to the bucket it holds.
  • TestManagerWindow_ReemitPublishesTheLateRowsAlone pins the behaviour the docs describe.

If this is wrong, the docs steer a user with a replacing upsert away from reemit when it would have kept the count.

Verification

  • go test -short -race ./...: pass, no FAIL lines. go vet ./... is clean.
  • uv run --locked pytest tests/tooling -q: 205 passed.
  • make coverage-page && git status --short docs/coverage: clean.
  • uv run --locked pytest tests/release -q: not run. No flag, CLI command or build stamp changed; the schema changed one description.
  • make soak: not run. watermark.go changed a comment and an error string, and nothing on the consume loop, a handler, a sink or the allocation path changed.

The late-row claim, against the published turbolytics/sql-flow:v2026.09.14, a webhook source, and a replacing upsert into Postgres 18. Five posts closed a minute, then one late post arrived for it:

late_rows Count after the close Count after the late post window_late_rows_total
reemit 5 1 1
drop 5 5 1

The new test fails when reemit republishes the whole bucket. With the close's delete removed:

--- FAIL: TestManagerWindow_ReemitPublishesTheLateRowsAlone (0.03s)
    watermark_test.go:223: [][]string{[]string{"5"}, []string{"6"}} != [][]string{[]string{"5"}, []string{"1"}}

Soak verdict

Not run; see above.

Notes for the reviewer

…e drops

bluesky.postgres.windowed.yml paired late_rows: reemit with
ON CONFLICT (bucket, lang) DO UPDATE SET posts = EXCLUDED.posts. A close
deletes the bucket's rows, so reemit runs emit_sql over the late rows
alone, and the upsert replaced a published minute's count with theirs.
Against the v2026.09.14 image, a webhook source and Postgres 18: five posts
closed a minute, one late post arrived for it, and the row went from 5 to
1. The same run under drop kept 5, with window_late_rows_total at 1.

The example now declares drop. The README, the config schema, the example
comments, ParseLatePolicy's message and validate's reemit warning said
reemit publishes the bucket again for a sink that upserts, which is the
pairing that loses the count. They now say reemit publishes emit_sql over
the late rows, for a sink that adds them to the bucket it holds.

TestManagerWindow_ReemitPublishesTheLateRowsAlone pins the behaviour the
docs now describe: a bucket closes with a sum of 5, a late row arrives,
and the second flush carries 1. With the close's delete removed, so reemit
republishes the whole bucket, the test fails with 6.

If this is wrong, the docs steer a user with a replacing upsert away from
reemit when reemit would have kept the count, or the example undercounts
by every late post it drops.
@turbolytics
turbolytics force-pushed the fix/window-reemit-example branch from 87c2937 to 58e5bd3 Compare September 14, 2026 11:03
@turbolytics
turbolytics merged commit e069fc2 into main Sep 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant