Repository navigation
fix(test): isolate live outbox + launch tests from shared-volume state - #119
Merged
Merged
Conversation
Live PG tests asserted against the FULL result set of a tenant-scoped claim, and against first-write-wins launch inserts, using tenant and key literals shared by every run ever pointed at the volume. ClaimUnpublished returns ORDER BY created_at ASC LIMIT n under RLS, so with a shared tenant each run's window is filled by the OLDER rows earlier runs left behind; the rows the run had just seeded sort last and fall outside LIMIT. workflow_launches keys on a GLOBAL investigation_id primary key and RecordLaunch is ON CONFLICT DO NOTHING, so a fixed investigation_id converged after the very first run and "first record must insert" could never hold again. Both only ever held against a pristine volume. Isolation is by unique identity, not cleanup: migrations 005 and 010 revoke DELETE from claimops_app (and UPDATE on workflow_launches), so a test cannot remove what it wrote and must not try -- the volume is shared and its contents are evidence. Give each test a pid-scoped tenant and per-run IDs, the idiom hitl_test.go and hitl_expiry_test.go already use, and compare the whole claim result by equality rather than by membership on the rows the test owns. Adds TestOutbox_ClaimScopeExcludesPresentForeignRows: under each foreign tenant's own scope it first proves the excluded rows are present and due, so the assertion that our claim excludes them cannot pass vacuously. Cross-tenant and round-trip legs are also pinned to exact non-empty sets, and B now owns a row so "B did not see A" cannot hold by finding nothing.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
Live PG tests scoped themselves by literals shared by every run ever pointed at the volume, then asserted on the full result set.
ClaimUnpublishedis RLS-scoped to the ctx tenant and returnsORDER BY created_at ASC LIMIT n. With a static tenant (ob-t2-tenant,ob-t3-tenant-a) each run's window was occupied by the older rows earlier runs left behind — the rows the run had just seeded sorted last and fell outsideLIMIT. Membership assertions on that set held only against a pristine volume. (Volume held 89 due rows underob-t2-tenant, 46 underob-t3-tenant-a; the 10 oldest are exactly the stale set in the failure map.)workflow_launcheskeys on a globalinvestigation_idPK andRecordLaunchisON CONFLICT DO NOTHING, so the hard-codedinv-…61001/2/3converged after the first ever run. These two failures reproduce on pristinemain— pre-existing, not caused by the outbox fix, but blocking the same gate.Not RLS leaking: every returned row was the right tenant. The rows were foreign in time (earlier runs), not foreign in tenant.
Isolation approach
Unique identity per run, not cleanup — matching
hitl_test.go/hitl_expiry_test.go, which already doclaims.TenantID("tnt-" + string(uniqueClaimID(...))). Cleanup is not an option: migrations 005/010 revokeDELETEfromclaimops_app(andUPDATEonworkflow_launches), so a test cannot remove what it wrote. Verified live:has_table_privilege('claimops_app','outbox_events','DELETE') = false.uniqueOutboxTenant— pid-scoped tenant per test, so a claim's visible set is exactly the rows that run seeded.uniqueLiveInvID/uniqueLiveTenant—inv-+ 32 hex chars (pid + counter), the shapeinvest.ValidateIDrequires.idsOfnever filters, so a leaked row cannot be hidden by narrowing the slice).Before → after
TestOutbox_ClaimMarkRoundTripob-ok/ob-failabsentTestOutbox_CrossTenantInvisibletenant A cannot claim own eventTestLaunchLive_RecordGet_Idempotentmain—clean volume requiredTestLaunchLive_EnsureLaunched_EndToEndmain—launched=falseProof the assertions still bite
Mutation-tested — every case fails on the intended assertion:
claimable = 14, want 12leaked foreign row ob-fz-…,claimed 4 rows, want exactly 2MarkFailedbackoff pushed to the futurereclaimed = [], want exactly [ob-fail-…]MarkPublishedskippedpublished ob-ok-… still claimabletenant B claimed A's event ob-x-a-…New
TestOutbox_ClaimScopeExcludesPresentForeignRowsseeds 16 rows across 3 tenants, then first asserts under each foreign tenant's own scope that its rows are present and due — so the exclusion cannot pass vacuously over an empty result. Claims uselimit 50, so the verdict is independent ofcreated_atordering: any leak is caught wherever it lands.Gates
gofmt -lclean ·go build ./...·go vet ./...· fullgo test ./...with PG enabled 41/41 ·go test -raceon both touched packages ·uv run pytest tests/unit27 passed ·ruff check .clean ·git diff main --stat -- eval/ tests/ workflows/empty · 2 files changed, both*_test.go; no production code, no migrations.DB: no DELETE/TRUNCATE/DDL; PostgreSQL left up. All 433 pre-existing
outbox_eventsrows byte-identical (0 lost or altered); all 4 pre-existingworkflow_launchesrows intact. Additions are only per-run test rows (tnt-ob-*, per-runinv-*) plus 3 rows my own RED runs left in the legacy static tenants — inert, nothing reads them, and undeletable by design.🤖 Generated with Claude Code