Repository navigation
Detect and recover from Slack listeners that silently disconnect - #37
Conversation
…r good A Slack Socket Mode loop that exits on a fatal error (invalid_auth, token_revoked, ...) emits no event, and the listener discarded the return value, so the bot went deaf while its Channel kept reporting Connected=True until channelsd was restarted. Listeners can now report a lost transport via Deps.TransportLost. channelsd marks the Channel Connected=False (reason TransportLost), which the monitoring watchers already publish to the monitoring Channel, and restarts the listener with per-Channel exponential backoff (5s doubling to 5m). Failed listener starts share the same backoff instead of retrying on every 5s tick.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
|
||
| // noteListenerFailure records a failed start or a lost transport for key and | ||
| // returns how long until the next attempt is allowed. | ||
| func (m *channelManager) noteListenerFailure(key string) time.Duration { |
There was a problem hiding this comment.
Nothing ever clears a key's restart backoff: there's no Reset, DeleteEntry or GC call. A successful start leaves the entry in place, and flowcontrol only drops it after 2×listenerRestartMax (10m) with no failures.
What an operator sees: a Slack token gets revoked, the listener fails a few times, and the backoff climbs to 5m. They fix the token, or delete and recreate the Channel under the same name (same key). The Channel then stays detached with Connected=False for up to 5 more minutes. Before this change, it would have reattached on the next 5s tick. That makes the fix look like it didn't work. The same thing happens if the Channel goes invalid and becomes valid again.
Suggestion:
- Call
m.restarts.Reset(key)instopListener, so a detached or recreated Channel starts fresh. - Clear the backoff once a listener has been up long enough to count as healthy, for example about a minute after a successful start. Resetting right at
Startwould remove the protection against a token that flips between valid and invalid. - Optionally, call
restarts.GC()from the reconcile loop so entries for deleted Channels don't pile up. Each entry is tiny, so this is minor.
There was a problem hiding this comment.
Fixed in 1fd963d. The backoff now resets when the inputs change (Channel UID/generation or Secret resourceVersion, so a fixed token or recreated Channel retries immediately), when the Channel detaches (deleted or invalid, including never-started ones, which stopListener alone would miss), and after a minute of healthy uptime. I kept the reset out of stopListener itself since startListener uses it to replace a dead listener, which would wipe the backoff on every restart; no GC() needed since detach now drops entries.
…er applies Nothing ever cleared a key's restart backoff, so a Channel whose token was fixed, or that was deleted and recreated under the same name, could sit detached for up to 5m after the fix. The backoff now resets when the listener's inputs change (Channel UID, generation, or credentials Secret resourceVersion), when the Channel detaches (deleted, or invalid and not viable — including Channels whose Start never succeeded and so were never registered), and when a transport is lost after at least a minute of healthy uptime.
Problem
A Slack bot can stop receiving messages while its Channel still reports
Connected=True. When Slack Socket Mode hits an error it treats as fatal (for example a transientinvalid_authduring a Slack degradation), it stops reconnecting without emitting an event, and the listener discarded that signal. The bot stayed deaf until channelsd was restarted, and nothing alerted anyone.Changes
Connected=False(reasonTransportLost) with the cause. The existing monitoring watchers already publish that to the monitoring Channel, so an agent going down is now visible there, and its recovery is too.Testing
Unit tests cover reporting versus normal shutdown, marking the Channel down and restarting it, a failure during start, and the backoff schedule.