Skip to content

fix(test): isolate live outbox + launch tests from shared-volume state - #119

Merged
Aparnap2 merged 1 commit into
mainfrom
fix/outbox-test-isolation
Oct 6, 2026
Merged

Aparnap2 merged 1 commit into
mainfrom
fix/outbox-test-isolation

Conversation

@Aparnap2

@Aparnap2 Aparnap2 commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Root cause

Live PG tests scoped themselves by literals shared by every run ever pointed at the volume, then asserted on the full result set.

  • ClaimUnpublished is RLS-scoped to the ctx tenant and returns ORDER BY created_at ASC LIMIT n. With a static tenant (ob-t2-tenant, ob-t3-tenant-a) each run's window was occupied by the older rows earlier runs left behind — the rows the run had just seeded sorted last and fell outside LIMIT. Membership assertions on that set held only against a pristine volume. (Volume held 89 due rows under ob-t2-tenant, 46 under ob-t3-tenant-a; the 10 oldest are exactly the stale set in the failure map.)
  • Same class, different table: workflow_launches keys on a global investigation_id PK and RecordLaunch is ON CONFLICT DO NOTHING, so the hard-coded inv-…61001/2/3 converged after the first ever run. These two failures reproduce on pristine main — pre-existing, not caused by the outbox fix, but blocking the same gate.

Not RLS leaking: every returned row was the right tenant. The rows were foreign in time (earlier runs), not foreign in tenant.

Isolation approach

Unique identity per run, not cleanup — matching hitl_test.go / hitl_expiry_test.go, which already do claims.TenantID("tnt-" + string(uniqueClaimID(...))). Cleanup is not an option: migrations 005/010 revoke DELETE from claimops_app (and UPDATE on workflow_launches), so a test cannot remove what it wrote. Verified live: has_table_privilege('claimops_app','outbox_events','DELETE') = false.

  • uniqueOutboxTenant — pid-scoped tenant per test, so a claim's visible set is exactly the rows that run seeded.
  • uniqueLiveInvID / uniqueLiveTenant — inv- + 32 hex chars (pid + counter), the shape invest.ValidateID requires.
  • Assertions moved from membership on owned IDs to equality on the whole unfiltered result (idsOf never filters, so a leaked row cannot be hidden by narrowing the slice).

Before → after

before after
TestOutbox_ClaimMarkRoundTrip FAIL — claimed 10 stale rows, fresh ob-ok/ob-fail absent PASS
TestOutbox_CrossTenantInvisible FAIL — tenant A cannot claim own event PASS
TestLaunchLive_RecordGet_Idempotent FAIL on main — clean volume required PASS
TestLaunchLive_EnsureLaunched_EndToEnd FAIL on main — launched=false PASS
full suite, PG enabled 39/41 packages 41/41, x3 consecutive runs

Proof the assertions still bite

Mutation-tested — every case fails on the intended assertion:

mutation assertion that fired
foreign scope collapsed onto own scope non-vacuity precondition: claimable = 14, want 12
same, precondition rigged to pass claim equality: leaked foreign row ob-fz-…, claimed 4 rows, want exactly 2
MarkFailed backoff pushed to the future reclaimed = [], want exactly [ob-fail-…]
MarkPublished skipped published ob-ok-… still claimable
tenant B collapsed onto A tenant B claimed A's event ob-x-a-…

New TestOutbox_ClaimScopeExcludesPresentForeignRows seeds 16 rows across 3 tenants, then first asserts under each foreign tenant's own scope that its rows are present and due — so the exclusion cannot pass vacuously over an empty result. Claims use limit 50, so the verdict is independent of created_at ordering: any leak is caught wherever it lands.

Gates

gofmt -l clean · go build ./... · go vet ./... · full go test ./... with PG enabled 41/41 · go test -race on both touched packages · uv run pytest tests/unit 27 passed · ruff check . clean · git diff main --stat -- eval/ tests/ workflows/ empty · 2 files changed, both *_test.go; no production code, no migrations.

DB: no DELETE/TRUNCATE/DDL; PostgreSQL left up. All 433 pre-existing outbox_events rows byte-identical (0 lost or altered); all 4 pre-existing workflow_launches rows intact. Additions are only per-run test rows (tnt-ob-*, per-run inv-*) plus 3 rows my own RED runs left in the legacy static tenants — inert, nothing reads them, and undeletable by design.

🤖 Generated with Claude Code

Live PG tests asserted against the FULL result set of a tenant-scoped claim,
and against first-write-wins launch inserts, using tenant and key literals
shared by every run ever pointed at the volume.

ClaimUnpublished returns ORDER BY created_at ASC LIMIT n under RLS, so with a
shared tenant each run's window is filled by the OLDER rows earlier runs left
behind; the rows the run had just seeded sort last and fall outside LIMIT.
workflow_launches keys on a GLOBAL investigation_id primary key and
RecordLaunch is ON CONFLICT DO NOTHING, so a fixed investigation_id
converged after the very first run and "first record must insert" could never
hold again. Both only ever held against a pristine volume.

Isolation is by unique identity, not cleanup: migrations 005 and 010 revoke
DELETE from claimops_app (and UPDATE on workflow_launches), so a test cannot
remove what it wrote and must not try -- the volume is shared and its contents
are evidence. Give each test a pid-scoped tenant and per-run IDs, the idiom
hitl_test.go and hitl_expiry_test.go already use, and compare the whole claim
result by equality rather than by membership on the rows the test owns.

Adds TestOutbox_ClaimScopeExcludesPresentForeignRows: under each foreign
tenant's own scope it first proves the excluded rows are present and due, so
the assertion that our claim excludes them cannot pass vacuously. Cross-tenant
and round-trip legs are also pinned to exact non-empty sets, and B now owns a
row so "B did not see A" cannot hold by finding nothing.
@coderabbitai

coderabbitai Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b662ef30-86df-4ccc-9f97-17875430b1ab
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Aparnap2
Aparnap2 marked this pull request as ready for review October 6, 2026 03:19
@Aparnap2
Aparnap2 merged commit 1dbc76e into main Oct 6, 2026
5 checks passed
@Aparnap2
Aparnap2 deleted the fix/outbox-test-isolation branch October 6, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant