Skip to content

fix(realtime): surface terminal startup rejection reasons - #399

Open
chaser-kong wants to merge 2 commits into
mainfrom
fix/realtime-connection-rejection
Open

chaser-kong wants to merge 2 commits into
mainfrom
fix/realtime-connection-rejection

Conversation

@chaser-kong

@chaser-kong chaser-kong commented Oct 15, 2026 •

Copy link
Copy Markdown

Change

A terminal realtime startup rejection could return from the native connect call without a usable connection, leaving a subscription to fail later with a readiness timeout. A failed native connect also left its transport running.

Wait for connection readiness or a terminal disconnect, preserving the reason and optional numeric code in the exported RealtimeConnectionError. Bound startup to ten seconds and dispose failed or cancelled attempts without replacing their original error. Retryable native interruptions remain recoverable. Public documentation and type-consumer coverage show how to read error.code without casting.

Verification

Tested Python revision: 55e38ff.

  • Regression tests reproduced the original failure before the fix.
  • Ruff lint/format, mypy, basedpyright, policy, dependency audit, and deterministic OpenAPI check passed.
  • Full suite: 2,682 tests passed; 100% line and branch coverage.
  • Wheel/sdist isolated installation and documented synthetic-HTTP quickstarts passed; all four optional-dependency/type package environments passed.
  • Real native Centrifuge/WebSocket loopback rejected with 3507: permission denied promptly and left no connected transport.
  • Hosting contract tooling: 90 tests passed; go test ./scripts/ci passed. Installed wheel discovered all 59 canonical scenarios in the isolated Docker runner. Discovery and loopback checks are not live staging acceptance.
  • CI follow-up: added delayed native readiness, unbound retryable-disconnect, cancelled cleanup, and ten-second startup deadline regressions. All 2,682 tests pass with 100% line/branch coverage.
  • The affected realtime module mutation check passes locally: 171 mutants killed, 284 rejected by the native type checker, zero survivors or failures. All five mutations that survived in the earlier CI run are now caught. Full SDK mutation testing remains in PR CI.

Cross-language impact

Classification: public facade, shared behavior, native behavior, and public examples. No wire/OpenAPI snapshot change.

  • Requirement: SDK-REALTIME-005 — terminal startup rejection preserves the disconnect reason and numeric code, reports disconnected, and releases its transport.
  • JavaScript: fix(realtime): surface terminal connection rejection reasons volcano-sdk-js#329 (a80904b). Ruby: fix(realtime): preserve terminal disconnect reasons and codes volcano-sdk-ruby#352 (0d9b9f9). Hosting: https://github.com/Kong/volcano-hosting/pull/1925 (0e2fc6439).
  • Canonical Gherkin and all three bindings are maintained together on Hosting's matching fix/realtime-connection-rejection branch. There are no SDK-owned active/staged copies.
  • Equivalent public examples are updated in each SDK's docs/realtime.md.
  • Python exposes a typed exception from subscription startup; JavaScript has explicit connect() and Ruby uses Realtime::ClosedError. No new application API is copied between languages.
  • Merge the SDK implementations before validating the coordinated Hosting scenario against SDK main revisions. Live staging acceptance remains pending Staging Validation; no cloud fixtures were provisioned locally.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@tkkhq tkkhq left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The inline notes are non-blocking checks for shutdown timing and retryable recovery. Per the repository rollout guidance, validate the merged SDK main revision through the coordinated Hosting staging scenario; the PR reports live acceptance as pending.

except BaseException:
self.connection_session_lineage = None
self.connection_access_token = None
await self._discard_failed_connection(connection)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If transport shutdown is slow after startup times out, the caller receives the exception after more than ten seconds: cleanup is awaited outside the timeout scope. Non-blocking: add a delayed-close test to establish the caller-visible bound, since the current deadline test uses immediate cleanup.

None
):
events = ClientEvents(lambda _context: None)
await events.on_disconnected(SimpleNamespace(code=3001, reason="transport closed"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handler has no connection binding, so the test exercises the default false predicate rather than the production native-state check. Non-blocking: add a bound connection case that emits a retryable disconnect while connecting, then connects successfully, to protect recovery within the startup deadline.

@chaser-kong
chaser-kong marked this pull request as ready for review October 15, 2026 21:59
@chaser-kong
chaser-kong requested a review from a team as a code owner October 15, 2026 21:59
@github-actions

Copy link
Copy Markdown

AI review: no P0-P2 findings

Reviewed 55e38ff. Run log

Makes realtime startup wait for connection readiness or a terminal disconnect (bounded to 10s), surfaces terminal rejections as a new exported RealtimeConnectionError carrying reason and numeric code, and disposes failed or cancelled native connections. The logic is sound and well tested; remaining concerns are low-severity: cleanup that swallows cancellation, the connection lock held through the readiness wait, a hard-coded deadline that forces a 10-second unit test, and the new error sitting outside the VolcanoError hierarchy.

⚪ P3

  • src/volcano_sdk/_realtime_connection.py:551 Failed-connection cleanup swallows an external cancellation and can leave disconnect half-done
  • src/volcano_sdk/_realtime_connection.py:525 Readiness wait holds connection_lock up to 10s, stalling disconnect() during a retrying first connect
  • src/volcano_sdk/_tests/test_realtime_connection_boundaries.py:189 Hard-coded 10s startup deadline forces a unit test that waits 10 real seconds
  • src/volcano_sdk/errors.py:72 New public RealtimeConnectionError sits outside the VolcanoError hierarchy

Not checked: The AI review read the diff, the code, and the PR's text; it ran no tests or builds and read nothing outside this repository.

  • That centrifuge-python 0.6.0 sets state to "disconnected" before calling on_disconnected, and only on terminal transitions; the fix relies on this and tests only use fakes.
  • That the new docs claim, that auth-only public keys need realtime.connect and realtime.subscribe scopes, matches Hosting's key-scope behavior.
  • That the coordinated SDK-REALTIME-005 Hosting scenario passes against live staging with this SDK revision.
  • Verify native Centrifuge retryable startup recovery and WebSocket disposal after failure or cancellation through the coordinated SDK-REALTIME-005 acceptance scenario.

Human review: needed

  • Is replacing terminal Channel.subscribe() readiness-timeout failures with RealtimeConnectionError compatible with existing callers' exception handling (src/volcano_sdk/_realtime_connection.py and src…

The AI review found no P0-P2 problems at 55e38ff, so the code needs no second review for bugs. Decide the points above: approve to accept them, or request changes.

Suggested reviewer: swkeever (most commits to _realtime_connection.py, _realtime_transport.py, errors.py and docs/realtime.md). Any collaborator with write access other than the author can approve.

connection: _native.VolcanoCentrifugeConnection,
) -> None:
# Cleanup must not replace the original connection failure/cancellation.
with suppress(Exception, asyncio.CancelledError):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Failed-connection cleanup swallows an external cancellation and can leave disconnect half-done

_discard_failed_connection wraps connection.disconnect() in suppress(Exception, asyncio.CancelledError). Suppose the task is cancelled while this cleanup runs, for example by app shutdown or a TaskGroup sibling failing after a RealtimeConnectionError or TimeoutError. The cancellation is discarded and the task ends with the original error instead of CancelledError. The native disconnect() is also interrupted midway, so the transport this PR means to release can stay open. Elsewhere the module finishes native stops under asyncio.shield and re-raises the cancellation afterwards (unsubscribe_native). Using the same pattern here would keep the original error when cleanup fails with an Exception, while still finishing the disconnect and propagating cancellation.

Found by claude-opus-5-5

self.connection_access_token = session.access_token
try:
await connection.connect()
async with asyncio.timeout(10):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Readiness wait holds connection_lock up to 10s, stalling disconnect() during a retrying first connect

Before this change, if the first native attempt failed with a retryable error (for example, connection refused), native connect() scheduled a reconnect and returned quickly. That released connection_lock, and the subscription's readiness wait ran outside the lock. Now _connect_locked holds the lock through events.wait_ready() for up to 10 seconds. A concurrent client.realtime.disconnect(), channel.unsubscribe() or remove_channel(), such as a shutdown handler, now blocks until the deadline passes instead of aborting the startup. A fix would be to let disconnect() cancel or supersede an in-flight startup, or to wait for readiness without holding the lock.

Found by claude-opus-5-5

try:
# The outer deadline distinguishes the SDK's ten-second timeout from
# an absent deadline or one extended beyond its documented bound.
async with asyncio.timeout(10.75):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: Hard-coded 10s startup deadline forces a unit test that waits 10 real seconds

asyncio.timeout(10) is written inline in _connect_locked, so test_connection_startup_times_out_within_its_ten_second_deadline really sleeps about 10 seconds. That adds 10 seconds to every suite run. Under mutmut it also runs for any _connect_locked/ClientEvents mutant that faster tests don't kill. Moving the deadline into a named constant (for example in _realtime_messages, next to the other realtime limits) would let the test patch it to a small value and assert separately that the constant is 10. That keeps coverage of the documented bound without the wall-clock cost. Separately, a timeout surfaces as a bare TimeoutError() with no message, and docs/realtime.md doesn't say which exception callers should catch for it.

Found by claude-opus-5-5

Comment thread src/volcano_sdk/errors.py
"""The request failed before receiving an API response."""


class RealtimeConnectionError(RuntimeError):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ P3: New public RealtimeConnectionError sits outside the VolcanoError hierarchy

Every other class in errors.py ("Typed errors raised by the Volcano SDK") derives from VolcanoError, and applications that catch VolcanoError as their SDK-wide handler won't catch this one. Its code: int | None also conflicts with VolcanoError.code: str | None, so it can't be re-parented later without changing the attribute's type. Subclassing RuntimeError keeps existing except RuntimeError handlers working, which is a reasonable tradeoff. Since the class becomes public API on release, it would help to either record this decision or rename the attribute (for example disconnect_code) so the error could also inherit from VolcanoError.

Found by claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants