fix(changes): close the change feed's silent-failure paths - #94
Merged
Conversation
A resume buffer opens exactly one ScopedStack.subscribe(), carrying the filter of whichever connection minted it, so every connection that shares a buffer shares that filter. Two things could put connections that mean different things onto one buffer. resumeBufferKey() built its key with a positional JSON.stringify over an array, where an undefined element renders as null. That collapsed "no parentId filter" onto `parentId: null` — root records only, a filter the wire format defines. Whichever connected first then decided what the other received: an unfiltered connection silently never heard about child records, with no reset frame to announce the gap, or a roots-only connection received changes outside its filter. Both are contract violations, and the silent one is the failure wire-format.md § Change feed calls untrustworthy. Keying on an object fixes it directly: JSON.stringify drops an undefined property but not an undefined array element, so absent is now encoded as an absent key. acquire() also registered a buffer before awaiting its subscription, so a subscription that rejected left a buffer in the map that nothing was feeding and nothing would retry — every later connection on that key got ready followed by permanent silence. The buffer now lands in the map only once its subscription resolves, with in-flight attempts tracked separately so two connections racing on one key still open one subscription and share however it settles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FH88yDv3Fk2RmLsU7DU4Rz
The streamSSE callback had no try/finally and passed no onError, so a throw anywhere in it — the backlog replay awaits ScopedStack.get() per entry against the adapter — skipped clearInterval on both timers, unsubscribe(), and unregister(). That leaked a live core subscription, a listener on the buffer, and a refcount that never returned to zero, which meant the buffer was never evicted. Hono catches the throw itself, so nothing surfaced beyond a bare console.error. Cleanup now runs in a finally. The throw is caught rather than left to hono, which answers one by writing the raw error message to the client as an error frame — internal detail on a stream any anonymous caller can open. A closed connection is already a repair the client knows how to make, and the failure is logged with its request id like every other. Three smaller things in the same path: - The session re-check's lookupToken() had no rejection handler, so an unreachable token store would raise an unhandled rejection rather than closing the stream. It now closes, which is the answer the client already handles via a 401 on reconnect. - The owner-token comparison here was a plain ===, where authMiddleware deliberately uses timingSafeEqual. safeCompare is exported and reused so the same secret is compared the same way in both places. - Keepalives went out as raw writes outside FrameGate, and the backlog replay kept running permission checks after the gate had tripped. Both now go through the gate, so "in-flight frames are bounded" holds for the whole stream rather than for record frames alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FH88yDv3Fk2RmLsU7DU4Rz
`WellknownRouteOptions` carried two flags, `changeFeedResume` and `changeFeedRecords`, whose only effect was to print a different literal in the discovery response. Neither was a deployer lever, and `createApp()` passed neither, so production was consistent by two defaults happening to line up rather than by construction. `changeFeedRecords` was the worse of the two: there is no route-side counterpart at all. `parseIncludeRecord()` honors `?include=record` unconditionally, so the flag could only ever make discovery advertise something this server does not do. That is not cosmetic — a client is entitled to act on discovery without asking again, since `APIAdapter.subscribeChanges()` against a server advertising no feed throws locally without sending a request. What the flags bought was one conformance fixture, discovery-advertises-a-feed-that-neither-resumes-nor-includes-records, which describes a conformant server with neither capability. This server is not that server, and satisfying a fixture about a different implementation is not worth an option that lets this one misreport itself. The fixture is skipped with that reason, using the SKIPPED set assertCoverage() already takes and the createRecord block already uses. Both fields are now literals. `ChangeRouteOptions.resume` stays: unlike these, it drives real spec-defined behavior — `ready` with no `seq`, then `reset` with reason `not_supported` — that a behavioral fixture exercises and that needs a way to be reached. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FH88yDv3Fk2RmLsU7DU4Rz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six bugs found reviewing the change-feed work from #81–#84. Five are in the resume layer #84 added, one is in discovery, and all of them share a failure mode: the server stops telling a client the truth, and the client is never told that it stopped.
Bounding the resume buffers — the other half of that review — is deliberately not here. It needs a config surface and a decision about where the ceiling goes, and it shouldn't hold up fixes that don't. It follows in #95.
What's fixed
1. Two different filters could share one resume buffer.
resumeBufferKey()built its key with a positionalJSON.stringifyover an array, where anundefinedelement renders asnull. That collapsed "noparentIdfilter" ontoparentId: null— root records only, a filter the wire format defines.A buffer opens exactly one
ScopedStack.subscribe(), carrying the filter of whichever connection minted it, so whichever connected first decided what the other received:?parentId=nullGET /changesreset, no errorGET /changes?parentId=nullBoth violate the contract. The silent one is what
wire-format.md § Change feedcalls the behavior that makes a feed untrustworthy: a client that can't tell it missed something can't repair it either. Keying on an object fixes it directly —JSON.stringifydrops an undefined property but not an undefined array element, so absent is now encoded as an absent key, which no present value can imitate.2. A failed subscription permanently poisoned a buffer key.
acquire()registered the buffer before awaitingsubscribeToStack(), so a rejection left a buffer in the map that nothing was feeding and nothing would retry. Every later connection on that key gotreadyand then permanent silence. The buffer now lands in the map only once its subscription resolves, with in-flight attempts tracked separately so two connections racing on one key still open one subscription and share however it settles.3. A throw in the stream body skipped every cleanup. The
streamSSEcallback had notry/finallyand passed noonError, so a throw — the backlog replay awaitsScopedStack.get()per entry against the adapter — skipped bothclearIntervals,unsubscribe()andunregister(). That leaked a live core subscription, a listener on the buffer, and a refcount that never returned to zero, so the buffer was never evicted. Hono catches the throw itself, so nothing surfaced beyond a bareconsole.error.Cleanup now runs in a
finally, and the throw is caught rather than left to hono — which answers one by writing the raw error message to the client as anerrorframe, on a stream any anonymous caller can open. It's logged with its request id like every other failure.4. The session re-check could take the process down.
lookupToken()had no rejection handler, so an unreachable token store raised an unhandled rejection instead of closing the stream. It now closes, which is the answer the client already handles via a401on reconnect.5. Discovery could contradict the route it describes.
WellknownRouteOptionscarriedchangeFeedResumeandchangeFeedRecords, whose only effect was to print a different literal in the discovery response. Neither was a deployer lever, andcreateApp()passed neither — production was consistent because two defaults happened to line up, not by construction.changeFeedRecordswas the worse of the two: there is no route-side counterpart at all.parseIncludeRecord()honors?include=recordunconditionally, so that flag could only ever advertise something this server doesn't do. Not cosmetic — a client is entitled to act on discovery without asking again, sinceAPIAdapter.subscribeChanges()against a server advertising no feed throws locally without sending a request.What the flags bought was one conformance fixture,
discovery-advertises-a-feed-that-neither-resumes-nor-includes-records, which describes a conformant server with neither capability. This server isn't that server, and satisfying a fixture about a different implementation isn't worth an option that lets this one misreport itself. Both fields are now literals, and the fixture is skipped with that reason — using theSKIPPEDsetassertCoverage()already takes and thecreateRecordblock already uses.ChangeRouteOptions.resumestays. Unlike those two it drives real spec-defined behavior —readywith noseq, thenresetwith reasonnot_supported— that a behavioral fixture exercises and that needs a way to be reached.6. Smaller consistency fixes. The owner-token comparison here was a plain
===whereauthMiddlewaredeliberately usestimingSafeEqual—safeCompareis now exported and reused, so the same secret is compared the same way in both places. Keepalives went out as raw writes outsideFrameGate, and the backlog replay kept running permission checks after the gate had tripped; both now go through the gate, so "in-flight frames are bounded" holds for the whole stream rather than forrecordframes alone.Verification
Each fix has a regression test, and each was confirmed to fail against the code before it — the two end-to-end filter tests and the keying test fail on the old
resumeBufferKey, and both poisoning tests fail on the oldacquire().pnpm typecheck,pnpm lint,pnpm format:checkandpnpm testall pass. 416 tests, from 409 onmain— net of two deleted with the discovery flags.Review note
Most of
src/routes/changes.ts's diff is re-indentation from wrapping the body intry.git diff -wshows the real change.🤖 Generated with Claude Code
https://claude.ai/code/session_01FH88yDv3Fk2RmLsU7DU4Rz