Repository navigation
fix(gateway): recover dead-lettered request logs - #1037
sbalabanov wants to merge 7 commits into
Conversation
Summary: A terminal request log that exhausts primary retries can leave the public status permanently stale. Replay the gateway-owned final log backlog through the existing materializer, retain malformed payloads and envelopes, and clean up both consumers on partial startup or shutdown. Test Plan: 1. Seed a terminal log in log_dlq before the gateway starts, while its receipt summary is unavailable. 2. Observe recovery retrying without acknowledgment, then restore the summary without publishing another event. 3. TestFinalLogBacklogRecovery verifies the repaired terminal summary, URI mapping, List receipt, and retained history. TestLogConsumers_RetainFailedRecovery, TestServeGateway_ConsumerCleanup, and TestConsumer_ProcessDelivery_InvalidTenantMetadata cover retention and lifecycle failures. Revert Plan: Revert this change and leave the log_dlq backlog intact.
|
[independent-review] Should fix — shutdown failures can be hidden (95% confidence). In Smallest correction: return the stop error instead of expected cancellation when cleanup fails; still join genuine listener/server failures and keep stopping both consumers. Extend the existing cleanup table with cancellation + stop error; assert both consumers stop, the error contains All four affected package tests, including fresh Docker integration, passed independently. The lifecycle/policy refactoring already removes duplicated branching; no additional safe complexity reduction was established. Generated with the repository's |
Keep expected signal cancellation from masking a failed consumer stop. Preserve joined cleanup errors for listener and server failures, and continue stopping both consumers. Test Plan: Extend TestServeGateway_ConsumerCleanup with cancellation plus a stop failure; verify both consumers stop, the stop error survives, and context.Canceled does not. Revert Plan: Revert this follow-up commit.
|
[agent-response SQC-010] Addressed #1037 (comment) in commit 2669cfb. A consumer-stop failure now replaces expected signal cancellation; genuine listener/server errors still join cleanup failures, and both consumers are stopped. Extended TestServeGateway_ConsumerCleanup with cancellation plus a stop failure, verifying stopErr is retained and context.Canceled is absent. The focused Bazel package test and formatting/lint/tidy/Gazelle checks passed locally. The PR remains a draft; skipped code-CI checks are not counted as passes. |
Summary: Keep invalid envelope identities durable for subscriptions with unlimited retries and no further dead-letter queue. The pre-controller guard uses the registry's actual subscription policy to Nack these deliveries instead of Reject, while finite primary subscriptions keep their existing behavior. Test Plan: TestConsumer_ProcessDelivery_InvalidTenantMetadata covers final retention, finite primary rejection, and settlement failures. The consumer, SQL queue, and existing consumer integration Bazel targets verify the extracted change independently. Revert Plan: Revert the dependent gateway recovery change before this policy; leave retained final-queue rows intact.
Merge the independent consumer-envelope retention prerequisite into the gateway recovery branch without rewriting its published commits. The gateway change stays byte-for-byte unchanged on top of the prerequisite and current main. Test Plan: The consumer, gateway log controller, request materializer, server lifecycle, and existing gateway integration Bazel targets verify the combined tree, including retained-backlog recovery with no new log event. Revert Plan: Revert gateway recovery before its envelope-retention prerequisite; leave retained log rows intact.
Summary: Remove the added final-log backlog integration case and the startup seeding, import, and build dependency used only by it. Restore the pre-existing gateway integration suite unchanged; production recovery wiring and the other tests stay intact. Test Plan: The remaining gateway log-controller, request-materializer, server lifecycle, and gateway integration targets pass. The gateway service binary also builds. Revert Plan: Revert this test-only commit to restore the dedicated backlog regression.
Summary: Record the single always-retry primary alternative and why recovery stays separate: failed logs leave normal traffic and the SQL primary acknowledgment/cleanup checkpoint can advance. Test Plan: The gateway binary builds, and the existing server and SQL queue unit targets pass. No behavior or tests changed. Revert Plan: Revert this comment-only commit.
Summary: Keep gateway recovery independent of the separate platform comment. Restore main's existing logged-rejection behavior directly, without merging the revised platform PR tip, and clarify the invalid-envelope exception in the gateway docs. Gateway wiring, lifecycle, and prior test removal are unchanged. Test Plan: Existing consumer, gateway log-controller, materializer, server, and gateway integration targets pass. Platform source/tests match main exactly, and the gateway diff has no platform files. Revert Plan: Preserve logged rejection for invalid metadata and keep the gateway recovery change independent.
TL;DR: Recover dead-lettered logs so public request status no longer stays permanently stale.
Summary
Requests could finish while their public status stayed stuck: logs in
log_dlqhad no reader, so unlimited retries could not help. Add a gateway reader that saves those logs through the normal path and keeps trying when saving fails. Started readers are cleaned up after a failed startup and during shutdown.Stack
#1039 currently uses this branch as its review base.
Repro
log_dlq.Test Plan
Verified with the remaining tests:
The dedicated backlog regression has been removed; this PR no longer adds integration coverage of
log_dlqrecovery.Revert Plan
Revert this change and leave the log_dlq backlog intact.
Issues