fix(trust): add durable retry isolation for terminal analysis

[sayed710]

Persistent corrupt endings previously retried on every trust-worker scan, and overlapping workers could repeat expensive analysis. This change gives each terminal item durable scheduling and fenced ownership per consumer while preserving successful receipts and forward/reverse discovery of unrelated work.

Migration 0048 adds retry state keyed by consumer/game/sequence, with known-consumer, count, finite-time and paired-lease constraints. Short PostgreSQL transactions claim one eligible item, record one failure per owned attempt, and atomically write receipts with retry cleanup. Five-minute leases renew every minute; renewal loss aborts analysis. Failure n waits min(21,600,000 ms, 120,000 ms × 2^min(n−1, 8)), capped at six hours indefinitely. Claims and crashes do not count as failures. Decode corruption uses this same recoverable path; complete ending validation follows the game authority.

Both trust report-writing paths carry the actual lease and cancellation signal. Their short transaction validates consumer/game scope, locks event then retry state, checks the unexpired token and receipt predicate, and rechecks before commit. Delayed stale writes, expiry during writing and cancellation roll back. Report transactions have five-second lock, statement and idle limits; no analysis runs under these locks. Existing unleased repository callers and domain interfaces retain their contract.

Helm uses Recreate for first-upgrade legacy/new separation. Apply migration 0048 before upgraded workers start and drain separately deployed legacy workers. Leases still protect accidental overlap of upgraded workers. No new flags, moderation policy, sanctions, engine search limits, public API or telemetry subsystem. Logs expose only coarse phase and closed-whitelist diagnostic codes, never arbitrary payload/exception text. A permanently stalled existing database operation can still delay graceful shutdown; process termination leaves a finite recoverable lease. ADR-0155 and append-only M15 Increment 85 document these limits.

Validation on the combined main tree:

• Build, lint, all eight guards, 312 script tests, and 3,886 tests across all 19 hermetic workspaces.
• Real PostgreSQL persistence 201, API 77, two populated backup/restore drills, and 13 final restored inbox/report-fence tests.
• Nine pinned Stockfish/Fairy-Stockfish smoke tests and 86 gateway tests, including actual trust-worker startup, poison readiness, safe logs and shutdown.
• Helm lint, five schema scenarios and 140 snapshots; backend Playwright 229 with four workers and zero retries. Executed suites had zero skips; external-secret schema validation retains CI's expected missing-CRD skip.
• All twelve required mutations plus three report-write mutations compiled and were killed; sources restored byte-identically. Tests cover genuine separate-connection races, a gated MVCC receipt race, delayed stale report writes, independent cancellation/expiry rollback, count saturation, crash recovery and poison fairness. Compiled real-PG RED tests preceded both decoder corrections.

Main advanced during implementation and was merged normally to 2dd6d4d, preserving Increment 84. Strict exact-head Codex self-review is the task-authorized fallback because Claude and Gemini are genuinely quota-unavailable. Final exact-head CI, Qodo, Greptile and unresolved-thread evidence follows their fresh runs. The owner performs the merge manually.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: ee213c54-2da8-4bf8-b216-5dafeeb88e00

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qodo-code-review]

PR Summary by Qodo

Add durable, fenced retries for trust terminal analysis

🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

AI Description

• Prevent overlapping trust workers from repeating analysis with per-consumer, renewable leases.
• Defer failed or corrupt endings with durable backoff while preserving receipts and discovery of
 other work.
• Document migration and rollout requirements; test races, recovery, cancellation, and backup
 restoration.

Diagram

sequenceDiagram
    actor Wake as Scan wakeup
    participant R as Trust reconciler
    participant I as Fenced inbox
    participant D as PostgreSQL
    participant A as Trust analyzer
    Wake->>R: Start eligible scan
    R->>I: Claim next ending
    I->>D: Check receipt and due lease
    D-->>I: Fenced claim
    I-->>R: Ending and token
    R->>A: Analyze with abort signal
    loop During analysis
        R->>I: Renew lease
        I->>D: Verify token and expiry
    end
    alt Analysis succeeds
        R->>I: Acknowledge token
        I->>D: Write receipt and clear retry
    else Analysis fails
        R->>I: Record owned failure
        I->>D: Set next retry deadline
    else Ownership is lost
        R->>A: Abort analysis
    end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Move retries to a separate job queue

• ➕ Could provide built-in delayed delivery and worker coordination.
• ➖ Introduces another durable work source and a consistency boundary with committed events and receipts.

2. Lease pages of endings

• ➕ Would reduce claim transactions for large backlogs.
• ➖ Could expire leases before queued engine analysis begins and impede fair discovery of unrelated work.

Recommendation: Keep one-item PostgreSQL claims alongside the existing event-and-receipt inbox. They provide durable, per-consumer coordination without a second work source or prematurely leasing an analysis backlog.

Files changed (30) +1373 / -240

Enhancement (1) +2 / -0

index.tsExport retry primitives and in-memory inbox +2/-0

Export retry primitives and in-memory inbox

• Makes the scheduling helpers and deterministic inbox available to persistence consumers and tests.

packages/persistence/src/index.ts

Bug fix (12) +331 / -130

engine.tsPass lease cancellation to both engine searches +4/-0

Pass lease cancellation to both engine searches

• The evaluator checks cancellation before analysis and passes its signal to MultiPV and played-move searches.

packages/anti-cheat/src/engine.ts

analysis-service.tsCarry ownership cancellation through anti-cheat analysis +6/-4

Carry ownership cancellation through anti-cheat analysis

• Accepts an optional abort signal, checks it around the source load, and forwards it to the evaluator.

packages/api/src/anti-cheat/analysis-service.ts

engine-provider.tsConnect anti-cheat cancellation to the evaluator +1/-1

Connect anti-cheat cancellation to the evaluator

• Forwards the analysis service's signal when constructing an engine-backed evaluator.

packages/api/src/anti-cheat/engine-provider.ts

source.tsPrevent replay after anti-cheat ownership loss +7/-8

Prevent replay after anti-cheat ownership loss

• Checks cancellation before and after loading events. Replay failures now log a fixed error class rather than exception text that could contain stored payload.

packages/api/src/anti-cheat/source.ts

analysis-service.tsCheck lease ownership around bot analysis loads +4/-2

Check lease ownership around bot analysis loads

• Accepts an optional signal and checks it before and after the finished-game source load.

packages/api/src/bot-detection/analysis-service.ts

source.tsDiscard late bot source reads after cancellation +4/-2

Discard late bot source reads after cancellation

• Checks the signal on either side of event loading so a lost lease cannot begin replay after a delayed read.

packages/api/src/bot-detection/source.ts

terminal-event-reconciler.tsClaim, renew, and settle terminal work under fenced leases +71/-36

Claim, renew, and settle terminal work under fenced leases

• Replaces pending-page processing with bounded one-item forward and reverse claims. It renews ownership during analysis, aborts on lease loss, records owned failures or successful receipts, and emits coarse error metadata.

packages/api/src/terminal-event-reconciler.ts

trust-analyzers.tsWire cancellable consumers and payload-safe logging +12/-10

Wire cancellable consumers and payload-safe logging

• Passes lease signals into bot and anti-cheat services. Worker failure logs now contain structured retry details without arbitrary exception text.

packages/api/src/trust-analyzers.ts

0048_terminal_event_retries.sqlAdd constrained per-consumer retry and lease state +19/-0

Add constrained per-consumer retry and lease state

• Creates terminal retry rows keyed to committed events and known consumers, with bounded failure state, finite deadlines, paired lease fields, and a partial due-work index.

packages/persistence/migrations/0048_terminal_event_retries.sql

event-store.tsDefine the fenced terminal-inbox contract +31/-12

Define the fenced terminal-inbox contract

• Introduces consumer, position, lease, claim, and failure types. Replaces pending-page methods with one-item claims and token-checked renewal, acknowledgement, and failure operations.

packages/persistence/src/event-store.ts

terminal-event-inbox.tsImplement transactional claims and fenced completion +145/-55

Implement transactional claims and fenced completion

• Uses short PostgreSQL transactions for due-filtered claims, renewal, failure scheduling, and atomic receipt-and-retry cleanup. It also rejects malformed terminal endings before dispatch so corrupt aborts enter backoff rather than receiving a success receipt.

packages/persistence/src/pg/terminal-event-inbox.ts

terminal-event-retry.tsCentralize lease and bounded backoff rules +27/-0

Centralize lease and bounded backoff rules

• Defines lease, renewal, timeout, and retry intervals alongside deterministic exponential delay and safe deadline validation.

packages/persistence/src/terminal-event-retry.ts

Tests (10) +780 / -105

engine.test.tsVerify cancellation across engine searches +16/-0

Verify cancellation across engine searches

• Tests that both search requests receive the same signal and that an aborted lease prevents another evaluation.

packages/anti-cheat/test/engine.test.ts

anti-cheat-analysis-service.test.tsGuard replay logs against stored FEN disclosure +4/-3

Guard replay logs against stored FEN disclosure

• Uses a corrupt stored-FEN sentinel to verify that replay errors retain an operator signal without exposing payload text.

packages/api/test/anti-cheat-analysis-service.test.ts

terminal-event-reconciler.test.tsExercise backoff, fairness, renewal, and shutdown +252/-85

Exercise backoff, fairness, renewal, and shutdown

• Replaces pending-page fixtures with a fenced in-memory inbox. Tests restart and wakeup behavior, overlapping workers, poison fairness, reverse rediscovery, cancellation, storage outages, and stop races.

packages/api/test/terminal-event-reconciler.test.ts

terminal-recovery.integration.test.tsVerify durable terminal recovery with PostgreSQL +20/-17

Verify durable terminal recovery with PostgreSQL

• Updates recovery tests for claims and known consumers. Checks single-owner processing, restart-respecting backoff, and reverse discovery of due corrupt work without a receipt.

packages/api/test/terminal-recovery.integration.test.ts

trust-analysis-cancellation.test.tsReject late source reads after lease loss +47/-0

Reject late source reads after lease loss

• New tests verify that both trust consumers avoid loads when already cancelled and avoid replay or evaluator construction when a delayed load returns after cancellation.

packages/api/test/trust-analysis-cancellation.test.ts

trust-analyzers.integration.test.tsAssert overlapping workers avoid duplicate loads +7/-0

Assert overlapping workers avoid duplicate loads

• Counts event-store loads during an overlapping rollout to establish that leases prevent repeated analysis, not just duplicate reports.

packages/api/test/trust-analyzers.integration.test.ts

terminal-event-inbox.integration.test.tsProve PostgreSQL fencing, races, and migration safety +313/-0

Prove PostgreSQL fencing, races, and migration safety

• Uses real connections to test competing claims, independent consumers, expiry, failure counting, receipt atomicity, and a gated snapshot race. It also tests corruption, malformed aborts, and an upgrade from migration 0047.

packages/persistence/test/terminal-event-inbox.integration.test.ts

terminal-event-inbox.test.tsTest retry arithmetic and in-memory ownership +48/-0

Test retry arithmetic and in-memory ownership

• Checks backoff growth and bounds plus due times, independent consumers, expiry, stale fencing, and receipt suppression.

packages/persistence/test/terminal-event-inbox.test.ts

backup-restore-drill.integration.test.mjsVerify retry state survives backup and restore +62/-0

Verify retry state survives backup and restore

• Adds a populated logical backup/restore drill covering deadlines, active leases, and receipts, then checks due retry and expired-lease recovery in the restored database.

scripts/test/backup-restore-drill.integration.test.mjs

trust-worker-entrypoint.integration.test.tsCheck poison recovery in the deployed worker +11/-0

Check poison recovery in the deployed worker

• Runs the worker with corrupt terminal work and verifies durable backoff, continued readiness, no success receipt, payload-safe logs, and clean shutdown.

services/gateway/test/trust-worker-entrypoint.integration.test.ts

Documentation (6) +174 / -5

trust-worker.yamlClarify fenced rollout and migration prerequisite +4/-3

Clarify fenced rollout and migration prerequisite

• Deployment comments describe lease-protected overlap and crash recovery. They require migration 0048 and draining workers that do not participate in claims before the first upgrade.

deploy/helm/gambit/templates/trust-worker.yaml

values.yamlDocument first-upgrade trust-worker precautions +2/-1

Document first-upgrade trust-worker precautions

• Values comments replace the former overlap assumption with fenced leases and the requirement to drain pre-0048 workers.

deploy/helm/gambit/values.yaml

PROJECT_STATE.mdRecord Increment 85 implementation and validation +24/-1

Record Increment 85 implementation and validation

• Adds an append-only account of the retry contract, rollout, tests, limitations, and subsequent malformed-ending decoder correction.

docs/PROJECT_STATE.md

0144-committed-terminal-event-recovery.mdLink terminal recovery to its reliability extension +4/-0

Link terminal recovery to its reliability extension

• Points readers to ADR-0155 and notes that due-filtered, one-item claims retain the existing scan budgets.

docs/adr/0144-committed-terminal-event-recovery.md

0152-moderation-and-trust-operations.mdClose the documented poison-retry follow-up +7/-0

Close the documented poison-retry follow-up

• Identifies durable scheduling and fenced leases as the reliability follow-up and notes that legacy workers must be drained.

docs/adr/0152-moderation-and-trust-operations.md

0155-trust-terminal-retry-isolation.mdSpecify the trust retry and ownership contract +133/-0

Specify the trust retry and ownership contract

• Documents schema, transactional claims, fencing, backoff, cancellation, scan fairness, safe logging, and upgrade requirements. It also records validation evidence and the stalled-read shutdown limitation.

docs/adr/0155-trust-terminal-retry-isolation.md

Other (1) +86 / -0

in-memory-terminal-event-inbox.tsMirror fenced inbox behavior for deterministic use +86/-0

Mirror fenced inbox behavior for deterministic use

• Adds an in-memory implementation with independent consumer receipts, due filtering, expiring leases, token fencing, retry scheduling, and receipt cleanup.

packages/persistence/src/in-memory-terminal-event-inbox.ts

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Lost leases can overwrite trust reports ✓ Resolved

Description

AntiCheatAnalysisService.analyzeAndStore passes the new abort signal to loading and engine
evaluation but not to the downstream report write; bot analysis likewise writes reports without an
ownership check. If renewal fails while saveBatch is in flight, the expired attempt can commit an
upsert after a replacement worker has written its report, even though the reconciler then refuses to
acknowledge the expired attempt.

Code

packages/api/src/anti-cheat/analysis-service.ts[29]

+    const service = new AntiCheatService(this.makeEvaluator(g.variant, opts.signal), this.repository);

Relevance

●●● Strong

Fencing analysis ownership does not protect unconditional report upserts after lease loss; stale
results can overwrite replacements.

PR-#8

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The signal is passed to the evaluator, but the service that persists its result receives no signal
or lease. Both PostgreSQL repositories use unconditional conflict updates, and the reconciler checks
for cancellation only after the consumer returns.

packages/api/src/anti-cheat/analysis-service.ts[20-35]
packages/api/src/bot-detection/analysis-service.ts[16-25]
packages/anti-cheat/src/service.ts[29-45]
packages/persistence/src/pg/anti-cheat.ts[39-68]
packages/persistence/src/pg/bot-reports.ts[39-68]
packages/api/src/terminal-event-reconciler.ts[121-133]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A cancelled or expired analysis attempt can still commit its report after another worker claims the ending. Checking the abort signal after the write does not protect the report from a stale upsert.

## Fix Focus Areas
- packages/api/src/anti-cheat/analysis-service.ts[20-30]
- packages/api/src/bot-detection/analysis-service.ts[16-21]
- packages/persistence/src/pg/anti-cheat.ts[39-68]
- packages/persistence/src/pg/bot-reports.ts[39-68]

## Recommended Fix
Carry lease ownership through both report-writing paths. Before committing a report upsert, verify the current, unexpired fencing token in the write transaction so an expired attempt cannot overwrite a replacement's report; retain abort checks to avoid unnecessary work.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Trust-worker failure logs no longer say why work failed ✓ Resolved

Description

The new onError hook in startTrustAnalyzers throws away the error argument and logs only a
coarse errorClass. Scan failures reach it from this.report('', error) with no metadata and are
logged as scan-store-error with an empty gameId, and the startup-scan handler drops its error too.
Real causes such as a missing terminal_event_retries table, the new 5s statement/lock timeout, an
engine crash or a report-repository constraint violation therefore produce identical log lines with
no error name, code or stack.

Code

packages/api/src/trust-analyzers.ts[R72-76]

+    onError: (gameId: string, _error: unknown, metadata?: TerminalReconcilerErrorMetadata) => logger.error('Trust analysis failed; durable work remains recoverable', {
+      gameId, consumer: metadata?.consumer ?? null, seq: metadata?.seq ?? null,
+      failures: metadata?.retry?.failures ?? null, nextRetryAt: metadata?.retry?.nextRetryAt ?? null,
+      errorClass: metadata?.errorClass ?? 'scan-store-error',
    }),

Relevance

●●● Strong

Historical reviews accept preserving sanitized operational error details; dropping startup and
database causes reduces actionable observability.

PR-#12
PR-#63

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

_error is never read, and the reconciler's interval and pubsub scans call report('', error)
without metadata, so every claim or DB failure collapses to scan-store-error. The removed code
logged error.stack ?? error.message. Payload text can be kept out of logs while still recording
safe, non-payload fields such as error.name, the pg code (e.g. 42P01 or 57014) or a stack
trimmed of its message.

packages/api/src/trust-analyzers.ts[72-76]
packages/api/src/trust-analyzers.ts[98-100]
packages/api/src/terminal-event-reconciler.ts[40-48]
packages/api/src/terminal-event-reconciler.ts[156-163]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Trust-worker error logs throw away the error object, so operators cannot tell a missing migration, a statement timeout, an engine crash or a repository error apart.

## Fix Focus Areas
- packages/api/src/trust-analyzers.ts[72-76]
- packages/api/src/trust-analyzers.ts[98-100]
- packages/api/src/terminal-event-reconciler.ts[156-163]

## Recommended Fix
Add a helper that pulls out only fields that cannot contain payload text: `error.name`, `error.constructor.name`, the pg `code`, `routine` and `severity`, and an `AbortError` flag. Include these in the `onError`, startup-scan and default `console.error` log records. Keep `message` and stack text excluded.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. First chart upgrade still overlaps legacy and new workers ✓ Resolved

Description

The chart still uses RollingUpdate with maxSurge 1, so on the first upgrade a legacy worker and an
upgraded worker run at the same time. The new comments only tell operators to drain legacy workers;
nothing enforces it. Legacy workers ignore leases and acknowledge without deleting retry rows, so
expensive analysis can run twice and leased retry rows can be left behind next to receipts.

Code

deploy/helm/gambit/templates/trust-worker.yaml[R18-20]

+  upgraded workers use fenced per-consumer leases to prevent duplicate analysis;
+  expired leases recover work after a crash. Drain legacy workers when first
+  upgrading to migration 0048 (ADR-0155), since they do not participate in claims.

Relevance

●●● Strong

The PR explicitly requires draining legacy workers, but RollingUpdate still permits unsafe overlap
without enforcement.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The template keeps strategy RollingUpdate with maxSurge: 1, which guarantees overlap. The
wait-for-api init container only checks API readiness, which an old API pod satisfies before 0048 is
applied, as the new comment itself says.

deploy/helm/gambit/templates/trust-worker.yaml[42-46]
deploy/helm/gambit/templates/trust-worker.yaml[62-64]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The first upgrade to migration 0048 depends on operators manually draining legacy workers, but the chart's rolling strategy overlaps them by default.

## Fix Focus Areas
- deploy/helm/gambit/templates/trust-worker.yaml[15-64]

## Recommended Fix
Pick one: use the `Recreate` strategy for the trust worker (it is a singleton, and leases now make overlap unnecessary), or add an init check that waits until `schema_migrations` contains version 48 before the worker starts.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

Review mode: 🧠 Deep: This is a high-risk, bug-dense cross-cutting change spanning PostgreSQL migration and concurrency fencing, retry semantics, cancellation, deployment ordering, and multiple independent worker paths.

Tip of the day

💡 Did you know, you can copy the agent prompt from any finding and feed it to your IDE agent

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[greptile-apps]

Confidence Score: 5/5

[Critical risk] Adds durable retry state and changes worker deployment strategy.

The PR appears safe to merge; no outstanding Greptile finding or new blocking failure was identified.

Summary

The PR adds durable, per-consumer retry scheduling and fenced ownership for terminal trust analysis. Since the previous review, it switches the trust-worker Deployment to Recreate, fences both report-writing paths, tightens terminal-ending validation, and limits logged failure details to safe diagnostic codes. All three previous Greptile findings are resolved.

_{Reviews (2) · Last reviewed commit: "fix(trust): fence report writes and enfo..."}

[sayed710]

@greptile review

[sayed710]

PR #88 final exact-head handoff — 2026-10-03

PR: #88 — OPEN, unmerged; owner merges manually.

Initial base: f9144a2.
Refreshed base/current main: 2dd6d4d.
Final HEAD: 18f0104.
M15 Increment 85; migration 0048; ADR-0155. Main was merged normally, preserving history and Increment 84. PROJECT_STATE history is append-only and prior text matches after newline normalization.

Durable contract

terminal_event_retries is keyed by consumer/game/sequence. It stores failure count, finite retry deadline and paired nullable UUID lease token/expiry, with known-consumer, count, time and event-FK constraints. Successful receipts remain authoritative. No payload/error persistence, dead letter or backfill.

Each consumer independently claims one eligible item in a short transaction using SKIP LOCKED plus a conditional upsert and a fresh receipt predicate. Ownership uses a random token and a five-minute lease. Renewal runs every minute with a ten-second renewal timeout; lease loss aborts analysis. Claiming, renewal and crashes do not increment failures. Failure n schedules min(21,600,000 ms, 120,000 ms × 2^min(n−1,8)); count saturates at 2,147,483,647, retry continues indefinitely at the six-hour cap. Success atomically writes the receipt and deletes owned retry state; stale failure, renewal and acknowledgement are fenced out. Expired crashed work can be reclaimed after restart.

Both report repositories validate consumer/game scope and check the current unexpired token and absence of receipt under event/retry locks, then recheck ownership/expiry and cancellation before commit. Delayed stale writes, expiry during writing and cancellation roll back. Analysis runs outside database locks; fenced report SQL uses five-second transaction-local limits.

Forward scans retain a 1,000-item window; reverse scans retain 100 and restart after exhaustion. Poison items are skipped until due while healthy later work progresses; reverse rediscovery prevents older due work starving behind sustained new endings. Malformed and contradictory endings use durable decode-failure retry without receipts.

Helm trust-worker uses Recreate for upgrade separation. Apply migration 0048 before upgraded workers start; separately deployed legacy workers must be drained. Recreate does not guarantee absence of manually created/deleted-pod overlap; upgraded-worker leases protect such overlap. Logs retain coarse phase and closed-whitelist diagnostic codes without arbitrary exception/payload text.

Verification

• Build/lint, all eight guards, 312 script tests, 3,886 hermetic tests across 19 workspaces.
• PostgreSQL persistence 201, API 77, two populated backup/restore drills (13 structural checks each), and final restored inbox/report-fence suite 13. Separate-pool claim races, consumer independence, crash/renewal/fencing, gated MVCC receipt race, acknowledgement rollback, retry cap/saturation, upgrade integrity, ending matrix, delayed stale reports and cancellation/expiry rollback verified.
• Pinned real-engine tests 9 (Stockfish/Fairy-Stockfish), gateway 86 including actual trust-worker poison/readiness/log/shutdown behavior.
• Helm lint, five schema scenarios and 140 snapshots; Playwright 229, four workers, zero retries. Executed suites had zero skips; external-secret schema checks retain the expected unavailable-CRD skip.
• Twelve required and three additional report-fence mutations: all 15 compiled and were killed. Source hashes match restoration evidence. Genuine compiled PostgreSQL RED evidence preceded decoder fixes.
• Strict Codex exact-head self-review completed with no remaining actionable findings, using the authorized fallback for quota-unavailable Claude/Gemini. This is not an independent Claude/Gemini review.

Final external gates

All applicable mandatory checks PASS at final HEAD, including Node 22/24, PostgreSQL, real engines, gateway, Helm, Playwright/Lighthouse and pin parity. Production image build is path-gated and SKIPPED, not applicable.
CI: https://github.com/sayed710/rocky/actions/runs/37072575936
Pin parity: https://github.com/sayed710/rocky/actions/runs/37072575887

Qodo exact-head footer identifies 18f0104; Bugs 0, Rules 0. The PR-88-filtered portal lists only three historical resolved findings and no open actionable or requirement-gap finding (no separate gap counter exposed).
#88 (comment)

Greptile exact-head review identifies 18f0104, confidence 5/5, all prior findings resolved and no outstanding/new blocking or nonblocking actionable findings.
#88 (comment)

CodeRabbit skipped review for repository eligibility; supplementary only. All six review threads resolved; unresolved 0.

Fresh fetch proves local HEAD = remote branch HEAD = PR HEAD = 18f0104. Local/remote divergence 0 0; current-main behind/ahead 0 3; worktree clean. Final gate/comment/thread JSON and restoration evidence are in this ignored evidence directory.

Task-owned validation/PostgreSQL/Redis containers, anonymous volumes and network removed; filtered inventories are empty. Shared Docker daemon was left running.

Deliberate limits

No moderation/sanction/policy change, engine-limit change, public API/export/PGN change, new flags, telemetry subsystem or dead letter queue. Existing permanently stalled database I/O can delay graceful shutdown; abort does not physically cancel all general database requests. Process termination leaves a finite recoverable lease. Merge remains owner-only. Canonical Vault project destination is ambiguous, so execution write-back is Daily-only.