Skip to content

fix(submitqueue): preserve conclude dead-letter outcomes - #1041

Draft
sbalabanov wants to merge 2 commits into
wt/sqc009from
wt/sqc014
Draft

sbalabanov wants to merge 2 commits into
wt/sqc009from
wt/sqc014

Conversation

@sbalabanov

@sbalabanov sbalabanov commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: Conclusion retries finish request status updates without changing the recorded batch outcome.

Summary

A batch can finish while some of its requests still lack their final status updates. This change finishes those updates when conclusion retries, keeping successful work successful and preserving the original failed or cancelled result. If the batch is still running, records are missing, or the message is invalid, keep the message for another attempt instead of guessing an outcome.

Repro

  1. Start with a finished Succeeded batch containing two requests, then interrupt conclude after one final status update succeeds and the next status-log write fails.
  2. Let the failed message reach conclude_dlq, then retry it after status-log writes work again.
  3. Before: the message is discarded and missing updates stay missing. After: finish those updates as Landed, without turning successful work into an error.

Test Plan

  • Successful, failed, and cancelled batches keep their matching request outcomes; later infrastructure errors do not replace the original result.
  • A retry finishes missing request logs and leaves completed request states unchanged; it need not keep original timestamps or diagnostics. The partial-update test uses a Failed batch.
  • The final queue keeps unfinished, missing, or invalid work available for another attempt.

Revert Plan

Pause conclude final processing before reverting. Retain final queue records
and request logs; do not reinterpret stored terminal outcomes as failures.

Issues

Summary:
Share primary conclusion's terminal member reconciliation with a dedicated
conclude final handler. Preserve the batch outcome, repair terminal membership,
and retain non-terminal, missing, malformed, or partially completed work.

Test Plan:
TestConcludeFinalBinding runs the actual pipeline final consumer and verifies
log replay on success and Nack retention of incomplete obligations.
TestDLQConcludeController_TerminalOutcomes checks all three outcome mappings;
TestDLQConcludeController_ReplaysPartialFanout proves persisted diagnostics
survive a member log failure and replay without overwriting another winner.

Revert Plan:
Pause conclude final processing before reverting. Retain final queue records
and request logs; do not reinterpret stored terminal outcomes as failures.
Keep partial fan-out coverage focused on request identity, terminal status,
version, and publication order. Retain existing no-extra-write and divergent
winner expectations without requiring diagnostic or timestamp fidelity.
Clarify the same status-repair contract in conclusion's comments and docs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants