Repository navigation
Hedge stalled Firestore message writes from Kick webhooks - #1755
Merged
Merged
Conversation
A Twitch chat message was dropped with "failed to write message to firestore: context deadline exceeded". createMessage retried transient failures within a 15s budget but had no per-attempt deadline, so one Create RPC that stalled consumed the whole budget and no retry ran. chatmessage.Create runs a message document Create and, since Creates are idempotent (an existing document counts as written), starts another attempt beside a pending one after 2s, or shortly after a transient failure, up to three attempts. The first to land wins; duplicate is reported only when a single attempt found the document already there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EzdqAWrbTMdtJutmK41coy
A Kick chat.message.sent delivery failed with a 500 after its message Create returned DeadlineExceeded: the handler made a single attempt in its 10s persistence budget, so one stalled RPC consumed all of it. Message documents are now written through chatmessage.Create, which starts another attempt beside a stalled or transiently failed one and takes the first to land. Redeliveries still count as duplicates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EzdqAWrbTMdtJutmK41coy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Error group
CMPH0OLm48j3xgE:The Kick handler makes one
CreateMessageattempt inside its 10s persistence budget. That budget is already detached from the request, so it survives Kick hanging up. A single stalled RPC used all 10s, the handler answered 500, and Kick had already given up (request_canceled: true).Fix
Message documents are now written through
chatmessage.Createfrom #1754. If an attempt stalls for 2s, or fails transiently,Createstarts another one beside it, up to 3 attempts, and the first to land wins. This is safe because a messageCreateis idempotent: an existing document counts as written.The metrics outcome comes from
Create'sduplicateresult, so Kick redeliveries still count asDuplicate. A write that lands after hedging counts asCreated. The 10s budget, its detachment from request cancellation, and theMergeChannelpath are unchanged.Tests
TestKickStalledMessageWriteIsHedgedreproduces the production failure: the first message write stalls and a second would succeed. Before this change it fails withstatus=500 attempts=1 after 10s. Now the handler returns 200 after about 2s and stores the message.go test -race -count=2 ./internal/webhookpasses.go test ./...passes except theinternal/youtubelive tests, which need youtube.com (blocked in the sandbox where this was developed).Generated by Claude Code