[kevmo314]

Problem

Error group CJ-W7Oau86r1eg:

failed to write message to firestore: context deadline exceeded
  handler.(*TwitchAgent).write  handler/twitch.go:124

createMessage retried transient Firestore errors within a 15s budget, but it had no per-attempt deadline. One Create RPC that stalled used up the whole 15s, no retry ever ran, and the chat message was dropped. The plain context deadline exceeded (not a gRPC status) is the outer budget expiring.

Fix

This adds chatmessage.Create, a hedged writer for channels/{id}/messages documents. It lives in internal/chatmessage, which already holds what is shared by every writer of those documents.

• A message Create is idempotent: if the document already exists, it counts as written. So instead of waiting out a pending attempt, Create starts another one beside it after HedgeAfter (2s), or 100–200ms after a transient failure. It allows up to MaxAttempts (3), and the first attempt to land wins. Remaining attempts are canceled on return.
• If a hedge finds AlreadyExists, that may be the stalled attempt's own write. So duplicate is reported only when a single attempt ran, which keeps the chatmetrics duplicate counts honest.
• Permanent errors (e.g. PermissionDenied) return immediately. When the budget expires, the result is the last transient error, or the context error if there was none.

handler.createMessage keeps its 15s budget and now delegates to chatmessage.Create. write reads duplicate from it instead of sniffing the error inside the closure.

A follow-up stacked PR applies the same writer to the Kick webhook (CMPH0OLm48j3xgE).

Tests

• TestCreateMessageSurvivesStalledAttempt reproduces the production failure: the first attempt stalls and a second would succeed. On main it fails with calls=1 err=context deadline exceeded after 15s. Here it succeeds after about 2s.
• chatmessage.TestCreate (under synctest) covers:
  • written and already-written documents
  • a permanent failure
  • a transient failure that is retried
  • a stalled attempt that gets hedged
  • a hedge that sees the stalled attempt's own write (not counted as a duplicate)
  • transient failures until attempts run out
  • every attempt stalling until the budget expires
• go test -race -count=3 passes for internal/chatmessage and internal/handler. go test ./... passes except the internal/youtube live tests, which need youtube.com (blocked in the sandbox where this was developed).

---

Generated by Claude Code