Skip to content

feat(durable): send, list, and subscribe to durable messages - #343

Merged
subnetmarco merged 25 commits into
mainfrom
feat/durable-messages
Oct 16, 2026
Merged

subnetmarco merged 25 commits into
mainfrom
feat/durable-messages

Conversation

@subnetmarco

@subnetmarco subnetmarco commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Change

A Ruby application can send messages to a running durable execution, list them, and follow what it publishes. Server support is in volcano-hosting#1826. Ruby has no durable authoring API, so waiting and publishing stay in JavaScript and Python.

  • client.durable.messages.send_message(function, execution_id, name, data, client_token:, idempotency_key:). It's named send_message so it doesn't shadow Object#send.
  • client.durable.messages.list(...) takes direction:, name:, after: and limit:. Passing project_id: uses the project route.
  • client.durable.messages.subscribe(...) { |message| ... } follows the execution's realtime channel and lists from the last seq after every reconnect, so each published message arrives once, in order.
  • Durable#start returns the client_token of the execution it creates.
  • The realtime subscribe request carries a bounded data: field for the client token; the adapter stays internal.
  • docs/functions.md and the README are updated. The OpenAPI client is regenerated.
  • Purely additive.

Verification

bundle exec rake quality:

  • quality:audit and quality:generated
  • quality:lint: 410 files, no offenses
  • quality:types: RBS validate and both Steep targets, no errors
  • quality:spec: 2,122 examples, 0 failures, with 100% line coverage (5,684/5,684) and 100% branch coverage (1,579/1,579)
  • quality:defects and quality:package, which builds the gem and smoke-tests it in an isolated install

Cross-language impact

Classify this change: public facade / wire contract / shared behavior

  • Requirement IDs and affected behavior: SDK-DURABLE-011 to 020. Ruby binds the send, list and subscribe steps; the authoring side runs in the shared JavaScript fixture.
  • JavaScript / Python / Ruby companion PRs, or why each is unaffected: volcano-sdk-js#319 and volcano-sdk-python#391.
  • Canonical scenarios, bindings and active/staged copies: tests/sdk-contract/features/durable-messages.feature, with Ruby step definitions and support, in volcano-hosting#1827.
  • Public examples updated in each affected language: docs/functions.md and the README here; JavaScript and Python in their PRs; docs/public/functions/durable-messages.md in Hosting.
  • Native checks and shared acceptance evidence (with tested revisions): native checks above on b615912. Shared acceptance hasn't run yet; it runs from volcano-hosting#1827 once the three SDK PRs merge and volcano-hosting#1826 is on staging.

subnetmarco and others added 2 commits October 13, 2026 04:02
Bundles the hosting contract that adds durable messages and the execution client token, and regenerates the client.

Co-authored-by: Cursor <cursoragent@cursor.com>
Adds client.durable.messages: send_message and list on the application and project routes, and subscribe, which follows an execution's realtime channel and reads back what it missed after every reconnect. Durable#start returns the client_token of the execution it creates.

Co-authored-by: Cursor <cursoragent@cursor.com>

@subnetmarco subnetmarco left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Client-only, and the send/list/subscribe surface matches JS and Python, including the conflict codes. One issue: DurableMessageStream uses the same seq <= last_seq drop as the other SDKs, so a broadcast the server loses or delivers out of order is never delivered on an open subscription. The server-side fix is on #1826; once that payload change lands, this stream needs the matching catch-up.

A broadcast now carries prev_seq, the seq of the execution's previous
publish. A subscription that has not delivered it reads the list before
handing on the new message, so a lost or reordered broadcast is no longer
skipped. A catch-up page that delivers nothing new also ends the read.

Co-authored-by: Cursor <cursoragent@cursor.com>
@subnetmarco

Copy link
Copy Markdown
Member Author

Done in 0e82287, matching the server change on Kong/volcano-hosting#1826.

  • Each broadcast now carries prev_seq, the seq of the execution's previous publish. When prev_seq is past the last delivered seq, DurableMessageStream#live holds the message and starts a background catch-up, then releases the message in order. A failed catch-up is reported through on_error and the held message is still delivered. A missing or non-integer prev_seq reads as 0, so an older server behaves as before.
  • read_pages now stops at a page that delivered nothing new, not only at an empty one. A page that claims has_more but repeats what was already delivered would otherwise be requested again unchanged.
  • New specs cover the gap, a late broadcast, non-integer prev_seq, the failed catch-up, and the page that delivers nothing new. All three fail against the previous stream. quality:lint, quality:types, quality:spec (100% line and branch coverage) and quality:defects pass locally.
  • The README now says that a client with only a client token or the service key gets AuthenticationError from subscribe.

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	lib/volcano/generated/.openapi-generator/FILES
#	lib/volcano/generated/README.md
#	lib/volcano/generated/lib/volcano-generated.rb
#	lib/volcano/generated/lib/volcano-generated/api/system_api.rb
#	lib/volcano/generated/lib/volcano-generated/models/capability.rb
#	openapi/openapi.yaml
#	spec/volcano_generation_spec.rb
@subnetmarco
subnetmarco marked this pull request as ready for review October 14, 2026 23:37
@subnetmarco
subnetmarco requested a review from a team as a code owner October 14, 2026 23:37
@github-actions
github-actions Bot requested a review from swkeever October 14, 2026 23:43
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

AI review: no P0-P2 findings

Reviewed 6add532. Run log

Adds client.durable.messages (send_message, list, subscribe) on a new internal durable realtime channel type, exposes client_token on DurableExecution, and regenerates the OpenAPI client. The hand-written stream, catch-up, session-binding and realtime lifecycle code is carefully built and extensively tested, and no defects were found; the main risk is the larger change to shared realtime connection behavior and reliance on an unverified Hosting contract.

Not checked: The AI review read the diff, the code, and the PR's text; it ran no tests or builds and read nothing outside this repository.

  • Whether the realtime server accepts the durable:<execution_id> channel, the subscribe data.client_token field, prev_seq on broadcasts and the temporary error flag (Kong/volcano-hosting#1826).
  • Shared acceptance scenarios in Kong/volcano-hosting#1827 have not yet run against staging.
  • Reconnect, dropped-publication and task-cancellation behavior against a real server, which only running the code shows.
  • That async 2.46 leaves transient children running when their parent task is stopped and raises Async::Cancel from Task#stop, which the worker and cancel paths rely on.
  • Verify the shared scenarios in Kong/volcano-hosting#1827 against server support in #1826, including credential enforcement, prev_seq broadcasts, and reconnect ordering.

Human review: needed

  • Should a subscribe stopped after its frame is written, on any channel type, now close the shared WebSocket and reconnect every channel (lib/volcano/realtime/protocol_recovery.rb)?
  • Should an unconfirmed or interrupted durable leave drop the whole realtime connection rather than only that channel (lib/volcano/realtime/lifecycle.rb)?
  • Is the messages contract (send_message name, subscribe replaying from seq 0, endless 30s read-back retries) what the JS/Python SDKs and Hosting agreed?
  • Should possession of an execution's client_token grant message access to anonymous callers and other users, as specified in openapi/openapi.yaml and implemented in lib/volcano/durable_messages.rb?

The AI review found no P0-P2 problems at 6add532, so the code needs no second review for bugs. Decide the points above: approve to accept them, or request changes.

Suggested reviewer: swkeever (wrote most of the realtime, client and transport code this builds on). Any collaborator with write access other than the author can approve.

Comment thread lib/volcano/durable_message_stream.rb Outdated
Comment thread docs/functions.md Outdated
Comment thread lib/volcano/durable_message_stream.rb Outdated
Comment thread lib/volcano/durable_message_stream.rb Outdated
Comment thread lib/volcano/realtime.rb
Comment thread lib/volcano/durable_messages.rb Outdated
Comment thread docs/functions.md Outdated
Comment thread docs/functions.md Outdated
…ch-up

A broadcast now carries the seq of the publish before it. Live messages
are held while a catch-up reads, and behind any message still waiting on
its predecessor, so a missed publish can no longer be skipped. A failed or
short catch-up is retried with backoff from half a second to 30 seconds,
messages the connection dropped for lack of room are read back, and
removing or disconnecting the channel ends the subscription.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread lib/volcano/durable_message_stream.rb
Comment thread lib/volcano/durable_message_stream.rb
Comment thread lib/volcano/durable_messages.rb Outdated
Comment thread lib/volcano/durable_message_catch_ups.rb Outdated
…e catch-ups, and read off the reactor

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread lib/volcano/durable_message_stream.rb Outdated
Comment thread lib/volcano/durable_message_stream.rb Outdated
Comment thread lib/volcano/durable_message_stream.rb Outdated
Bind each subscription to the session that opened it, as Postgres
delivery does. Signing out, signing in again, or setting another session
closes the stream before its next read or delivery and raises
SessionChangedError; a token refresh keeps it.

Co-authored-by: Cursor <cursoragent@cursor.com>
subnetmarco and others added 2 commits October 15, 2026 07:05
A typed call without one always raised ArgumentError at runtime.

Co-authored-by: Cursor <cursoragent@cursor.com>
idempotency_key and client_token travel as headers, so a non-ASCII or control character now raises an ArgumentError naming the field before the request is built.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread docs/functions.md Outdated
Comment thread lib/volcano/durable_message_catch_ups.rb
Comment thread lib/volcano/durable_messages.rb
Comment thread docs/functions.md Outdated
Comment thread docs/functions.md
…ivering status

Co-authored-by: Cursor <cursoragent@cursor.com>

@subnetmarco subnetmarco left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second pass at 6eb894f. The gap fix matches JS and Python:

  • live holds a message whose prev_seq is ahead, catches up in the background, and releases the held messages in order.
  • read_pages stops at a page that delivers nothing new.

I re-read durable_messages.rb, the session binding, durable_channel.rb and the realtime lifecycle changes against hosting#1826. Nothing blocking:

  • On the AI review's question about leave_durable_channel: dropping the connection when a leave isn't confirmed seems right to me.

    • A leave only fails when the socket is already in trouble, so a reconnect is usually coming anyway.
    • Dropping guarantees the server-side subscription is gone after a session change.
    • The durable channel is removed from @channels first, so the reconnect doesn't bring it back.
  • Each SDK handles a resubscribe the server refuses differently:

    • Ruby: restore_subscriptions reports the error, and the channel stays restoration_needed?. Since reconnect_needed? checks that, the loop retries it about every 20 s and reports to on_error each time. Meanwhile the subscription delivers nothing and never ends.
    • JS logs once and removes the channel.
    • Python goes silent (see Kong/volcano-sdk-python#391).

    A refused durable subscribe is usually permanent: the visibility changed, or the execution is gone after retention. So ending the subscription and reporting once, like JS, would be better here. Not blocking, but worth aligning across the three.

Co-authored-by: Cursor <cursoragent@cursor.com>
@subnetmarco

Copy link
Copy Markdown
Member Author

Done in 5986c5f. A durable resubscribe the server refuses now ends the subscription and reports once, like JS.

  • Realtime::ServerError#temporary? now reads the flag the server sends with an error, and redaction keeps it. Centrifuge marks internal server error (code 100) temporary, which is what hosting returns while an authorization lookup fails. permission denied (103), its answer for an execution that's gone or hidden, isn't temporary.
  • When a reconnect's subscribe is refused and the error isn't temporary, the durable channel tells its follower. The follower unsubscribes, so the stream closes and the channel leaves @channels, and reconnect_needed? stops counting it. The refusal reaches on_error once, as permission denied; the durable message subscription has ended, with the server's code. The execution can then be subscribed to again. A temporary refusal is retried as before. Other channel types are unchanged.
  • New specs cover both cases. The permanent one fails against the old code with 104 subscribes and as many reports within 10 ms. The protocol and redaction specs cover the new flag.
  • The subscribe doc comment, the README and docs/functions.md now list it among the things that end a subscription.

Rubocop, both Steep targets, the full suite (2173 examples, 100% branch coverage) and check-defects pass.

Comment thread lib/volcano/durable_messages.rb
… its answer

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread lib/volcano/realtime/lifecycle.rb
…fore its answer

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread lib/volcano/realtime/lifecycle.rb
…iting for the lock

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread lib/volcano/durable_message_stream.rb
Comment thread README.md
subnetmarco and others added 3 commits October 15, 2026 21:01
A publication the stream cannot read is still one the execution published. It now starts a catch-up, as a dropped one does, instead of being skipped until the next gap.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sync the contract with Hosting: send, publish and message data accept and return an explicit null.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread lib/volcano/realtime/protocol_recovery.rb
subnetmarco and others added 2 commits October 16, 2026 08:00
… its frame is written

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@subnetmarco
subnetmarco added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 4a9e309 Oct 16, 2026
14 of 15 checks passed
@subnetmarco
subnetmarco deleted the feat/durable-messages branch October 16, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant