diff --git a/src/data/nav/aitransport.ts b/src/data/nav/aitransport.ts index e558d692b5..f742b2994c 100644 --- a/src/data/nav/aitransport.ts +++ b/src/data/nav/aitransport.ts @@ -9,13 +9,22 @@ export default { }, content: [ { - name: 'Getting started', + name: 'Why AI Transport?', pages: [ { - name: 'About AI Transport', + name: 'Overview', link: '/docs/ai-transport', index: true, }, + { + name: 'HTTP streaming and AI', + link: '/docs/ai-transport/why/http-streaming-and-ai', + }, + ], + }, + { + name: 'Getting started', + pages: [ { name: 'By SDK', pages: [ @@ -47,20 +56,6 @@ export default { }, ], }, - { - name: 'Why AI Transport?', - pages: [ - { - name: 'Overview', - link: '/docs/ai-transport/why', - index: true, - }, - { - name: 'HTTP streaming and AI', - link: '/docs/ai-transport/why/http-streaming-and-ai', - }, - ], - }, { name: 'Concepts', pages: [ diff --git a/src/pages/docs/ai-transport/concepts/index.mdx b/src/pages/docs/ai-transport/concepts/index.mdx index e6005499d2..6302073746 100644 --- a/src/pages/docs/ai-transport/concepts/index.mdx +++ b/src/pages/docs/ai-transport/concepts/index.mdx @@ -38,6 +38,6 @@ You do not need this section before you start building. Pick a [getting started ## Read next + +Most AI frameworks send [one HTTP request per turn](/docs/ai-transport/why/http-streaming-and-ai) and stream the response back over it. The response lasts exactly as long as that request, and only the client that made the request can read it. + +| Scenario | Direct HTTP streaming | AI Transport session | +| --- | --- | --- | +| The connection drops mid-response | The response is lost, the model keeps generating tokens that are discarded, and the user sees an error and starts again. | The agent keeps publishing into the session. The client reconnects and [resumes from the last token it received](/docs/ai-transport/features/reconnection-and-recovery). | +| A phone opens the same conversation | The stream belongs to the client that opened it, so a second device cannot read it. | Every device [attaches to the same session](/docs/ai-transport/features/multi-device) and sees the conversation as it streams. | +| The user reloads the page mid-answer | The reload discards the request and the partial answer with it. | The client reattaches and reads [history](/docs/ai-transport/features/history) and the in-flight response from the session, where the answer is a single message that grows as tokens arrive. | +| The user presses stop | Closing the connection is the only signal available, and the agent cannot tell it apart from a network drop. | The client publishes a [cancel signal](/docs/ai-transport/features/cancellation). The agent's current [run](/docs/ai-transport/concepts/runs) ends and the session stays open. | +| A second message arrives before the first answer finishes | A second request opens a second stream, and the client interleaves two responses or drops one. | Each turn is [its own run](/docs/ai-transport/features/concurrent-turns) on the same session, with its own stream and cancel handle. | +| The agent process restarts mid-answer | The outbound stream ends with the process, and the client is left displaying a truncated response. | Where the agent runs inside a workflow engine, the retried step publishes into [the same session](/docs/ai-transport/features/durable-execution). | + +A dropped connection loses a response whether the application has one user or a million. + +Solving these individually means building a buffer so a stream can resume, a database for conversation state, and a queue or a second WebSocket for the client to signal on. It also means a reconciliation step that merges stored state with the live stream whenever a client joins. None of it is specific to the AI product, and [HTTP streaming and AI](/docs/ai-transport/why/http-streaming-and-ai) goes through each limitation and the workaround code it requires. + +## What the session adds beyond delivery + +A [session](/docs/ai-transport/concepts/sessions) runs on an Ably channel, with a conversation model on top of it. The rest of Ably's channel features work on that channel, and the SDK builds a [conversation tree](/docs/ai-transport/concepts/conversation-tree) from its message log, which is what makes branching and editing possible. + +Each of the following is part of the session: + +| What you get | How it works | +| --- | --- | +| [Branching, edit, and regenerate](/docs/ai-transport/features/branching) | Editing a message or regenerating a response forks the conversation instead of overwriting it. The tree keeps every branch, and a view selects one path through it per client. | +| [Agent and client presence](/docs/ai-transport/features/agent-presence) | An agent reports whether it is thinking, streaming, idle, or offline, and clients that enter presence appear alongside it. An agent can also watch presence and stop work when nobody is connected to read the answer. | +| [Shared live state](/docs/ai-transport/features/liveobjects) | The agent reacts to what the user is doing, such as the record they have selected, without polling or extra tool calls. Client and agent read and write the same state over the session. | +| [Interruption and steering](/docs/ai-transport/features/interruption-and-steering) | A client sends a follow-up into the run that is already streaming, or cancels it and re-prompts. | +| [Human-in-the-loop approval](/docs/ai-transport/features/human-in-the-loop) | A run suspends until a tool call is approved. The request waits in the session, so any user can answer it from any device, minutes or hours later. | +| [Chain of thought](/docs/ai-transport/features/chain-of-thought) | Reasoning arrives as a separate stream within the same run, so a UI can render it beside the response. | + +Each of these needs a second transport alongside the stream if you build it on direct HTTP streaming. + +## Keep the stack you have + +In the agent, pipe the stream you already build into a run on the session instead of returning it as the HTTP response body: + + +```javascript +// Agent-side, in place of `return result.toUIMessageStreamResponse()`: +import * as Ably from 'ably'; +import { streamText, convertToModelMessages } from 'ai'; +import { anthropic } from '@ai-sdk/anthropic'; +import { Invocation } from '@ably/ai-transport'; +import { createAgentSession } from '@ably/ai-transport/vercel'; + +const ably = new Ably.Realtime({ key: process.env.ABLY_API_KEY }); + +export async function POST(req) { + const invocation = Invocation.fromJSON(await req.json()); + const session = createAgentSession({ client: ably, channelName: invocation.sessionName }); + await session.connect(); + const run = session.createRun(invocation, { signal: req.signal }); + + try { + // Page the view back through history to rebuild the prompt. Do this before + // run.start(), or it waits for the triggering input to arrive live. + while (run.view.hasOlder()) await run.view.loadOlder(); + const messages = run.view.getMessages().map(({ message }) => message); + + await run.start(); + const result = streamText({ + model: anthropic('claude-sonnet-4-20250514'), + messages: await convertToModelMessages(messages), + abortSignal: run.abortSignal, + }); + const streamResult = await run.pipe(result.toUIMessageStream()); + await run.end({ reason: streamResult.reason }); + } finally { + await session.end(); + } + + return Response.json({ runId: run.runId, invocationId: run.invocationId }); +} +``` + + +The route returns the run's identifiers. The client reads the answer from the session. Where an answer takes longer than your function's timeout, [durable execution](/docs/ai-transport/features/durable-execution) moves the run into a workflow engine and the route returns before it finishes. + +In the browser, a client attaches to that same session by name: + + +```javascript +// Client-side. +import * as Ably from 'ably'; +import { createClientSession } from '@ably/ai-transport/vercel'; + +const ably = new Ably.Realtime({ authUrl: '/auth' }); +const session = createClientSession({ client: ably, channelName: 'conversations:42' }); + +await session.connect(); +``` + + +Prompts, tool definitions, model calls, and rendering are untouched. AI Transport implements Vercel AI SDK's [`ChatTransport` interface](/docs/ai-transport/frameworks/vercel-ai-sdk-ui), so `useChat` accepts it as its transport. + +Two one-time setup steps apply. Browser clients connect with [token authentication](/docs/ai-transport/getting-started/authentication) rather than an API key, and the namespace your conversations live on needs [one channel rule](/docs/ai-transport/getting-started/channel-rules) enabled, because AI Transport streams tokens by appending to a message. -## Get started +The SDK is JavaScript and TypeScript, with React hooks for the client. The [roadmap](/docs/ai-transport/roadmap) covers other languages. + +## Limits and dependencies + +These apply to every application built on AI Transport: + +| Limit or dependency | Detail | +| --- | --- | +| Retention limits what the session holds | The session holds the conversation for as long as [channel retention](/docs/storage-history/storage) covers it, a window you set per namespace. Beyond that window you keep your own database. Persist each completed run, and the union of those runs reconstructs the conversation. [Database hydration](/docs/ai-transport/features/database-hydration) joins the stored history to the live session with no gaps and no duplicates. | +| Hydrated history is linear | Messages you have persisted yourself cannot be edited or regenerated, so branching applies to the part of the conversation the session still holds. | +| Session content is visible to subscribers | Every message reaches every subscriber whose token capability allows it, tool inputs and outputs included. Scope [capabilities](/docs/ai-transport/getting-started/authentication) per namespace so a token only grants access to the sessions that client should read. | +| A network hop and a third-party dependency | Conversations go through Ably instead of your own infrastructure, so every publish makes a round trip. [Append rollup](/docs/ai-transport/features/token-streaming#rollup) batches tokens, so that round trip happens once per batch rather than once per token. | +| A pre-1.0 SDK | Minor releases can still change the API. | + +[Going to production](/docs/ai-transport/going-to-production) covers limits, retention, monitoring, auth hardening, and pricing. + +## How a session is built + +Every session runs on an [Ably channel](/docs/channels): a durable, ordered, append-only log that any client or agent attaches to by name. Messages outlive the connection, device, or process that published them, and they have a total order. A client that drops reattaches and resumes without gaps or duplicates. + +Any client or agent publishes, which puts cancel and steering on the same path as tokens. The SDK layers a conversation on top of that log: + +- A [codec](/docs/ai-transport/internals/codec-architecture) maps your framework's event types onto channel messages. Tokens stream by [appending to a single message](/docs/messages/updates-deletes#append), so a client arriving late reads one assembled response. +- The transport does not constrain what publishes into it, so a [custom codec](/docs/ai-transport/internals/codec-architecture#write-a-custom-codec) adds support for a framework with no bundled adapter. +- [Client and agent sessions](/docs/ai-transport/concepts/sessions#connect) own attachment, the [run lifecycle](/docs/ai-transport/concepts/runs), and cancel routing. +- React hooks cover streaming, pagination, and [branch navigation](/docs/ai-transport/concepts/conversation-tree#views) in the UI. + +## When you do not need a durable session + +A single-turn chatbot does not need a durable session. If a user asks one question and never returns to the conversation, direct HTTP streaming is enough. + +## Platform guarantees + +Ordering, persistence, replication, and [regional failover](/docs/platform/architecture/fault-tolerance) are guarantees of the Ably platform, which is [designed for 99.999% global service availability](/docs/platform/architecture). They apply to AI Transport in the same way as to every other Ably product. + +Ably is [SOC 2 Type II certified and HIPAA compliant](https://ably.com/security-and-compliance), and operates a bug bounty program. + +## Start building + +Choose where to go next: {[ { - title: 'Getting started', - description: 'Build a working app with Vercel AI SDK or the Core SDK in a few minutes.', + title: 'Get started', + description: 'Build a working chat application with the Vercel AI SDK.', image: 'icon-tech-javascript', link: '/docs/ai-transport/getting-started/vercel-ai-sdk', }, - { - title: 'Frameworks', - description: 'See how AI Transport composes with the AI framework you already use.', - image: 'icon-tech-javascript', - link: '/docs/ai-transport/frameworks/vercel-ai-sdk-ui', - }, { title: 'Features', - description: 'Browse what AI Transport does once you have a session running.', - image: 'icon-tech-javascript', + description: 'Streaming, branching, presence, cancellation, and the rest.', + image: 'icon-product-ai-transport', link: '/docs/ai-transport/features/token-streaming', }, { - title: 'Why AI Transport', - description: 'Understand the production problems that direct HTTP streaming does not solve.', - image: 'icon-tech-javascript', - link: '/docs/ai-transport/why', + title: 'Concepts', + description: 'Sessions, runs, and the conversation tree behind the SDK.', + image: 'icon-gui-resources', + link: '/docs/ai-transport/concepts', + }, + { + title: 'Going to production', + description: 'The checklist to work through before you ship.', + image: 'icon-product-platform', + link: '/docs/ai-transport/going-to-production', }, ]} - -## Read next - -- [Concepts](/docs/ai-transport/concepts): sessions, runs, and the conversation tree. -- [Going to production](/docs/ai-transport/going-to-production): the production checklist for shipping AI Transport. diff --git a/src/pages/docs/ai-transport/roadmap.mdx b/src/pages/docs/ai-transport/roadmap.mdx index 7ce58f6794..914f96169a 100644 --- a/src/pages/docs/ai-transport/roadmap.mdx +++ b/src/pages/docs/ai-transport/roadmap.mdx @@ -23,7 +23,7 @@ Available today. | Drop-in framework integration | A drop-in transport for the [Vercel AI SDK](/docs/ai-transport/getting-started/vercel-ai-sdk), plus the [Core SDK](/docs/ai-transport/getting-started/core-sdk) for everything else. | | Shared live state and presence | [Agent presence](/docs/ai-transport/features/agent-presence) and [shared session state](/docs/ai-transport/features/liveobjects) exposed directly through the SDK. | | [Durable execution](/docs/ai-transport/features/durable-execution) | Pair durable sessions with a workflow engine such as [Temporal](/docs/ai-transport/frameworks/temporal) or [Vercel WDK](/docs/ai-transport/frameworks/vercel-wdk), so a mid-flight process crash retries the failed step cleanly instead of stranding the turn. | -| Enterprise-ready platform | SOC 2 Type II and HIPAA, on [Ably's realtime platform](/docs/ai-transport/why#why-ably). | +| Enterprise-ready platform | SOC 2 Type II and HIPAA, on [Ably's realtime platform](/docs/ai-transport#why-ably). | ## Now diff --git a/src/pages/docs/ai-transport/why/http-streaming-and-ai.mdx b/src/pages/docs/ai-transport/why/http-streaming-and-ai.mdx index 04defde16d..7ce89822f2 100644 --- a/src/pages/docs/ai-transport/why/http-streaming-and-ai.mdx +++ b/src/pages/docs/ai-transport/why/http-streaming-and-ai.mdx @@ -105,12 +105,11 @@ All of this infrastructure is the transport layer being reinvented around the li A durable session replaces the ephemeral HTTP stream with a persistent, shared medium that any client or agent connects to. The properties the infrastructure above tries to assemble (persistence, ordering, multi-subscriber fan-out, bidirectional publishing, presence) are properties the session already has. -AI Transport implements durable [sessions](/docs/ai-transport/concepts/sessions) on Ably channels. Read [how a durable session solves each of these problems](/docs/ai-transport/why), or [get started with the Vercel AI SDK](/docs/ai-transport/getting-started/vercel-ai-sdk). +AI Transport implements durable [sessions](/docs/ai-transport/concepts/sessions) on Ably channels. Read [how a durable session solves each of these problems](/docs/ai-transport), or [get started with the Vercel AI SDK](/docs/ai-transport/getting-started/vercel-ai-sdk). ![Diagram showing how AI Transport concepts compose around the session, with connections attaching from outside and the conversation tree, runs, invocations, codecs, authentication, and infrastructure relating to it](../../../../images/content/diagrams/ait-concepts-overview.png) ## Read next - -The client and the agent are coupled by a single HTTP request and response for the lifetime of the interaction. With simple HTTP streaming: - -- Streams cannot resume: when the connection drops (network switch, page refresh, laptop lid closes), the response is gone. The agent keeps generating tokens; there is nowhere to deliver them. -- Sessions do not span devices: the stream exists only for the client that opened it. A second tab or a phone has no way in. -- No way back to the agent: HTTP streams are server-to-client. The only upstream signal a client has is to close the connection, which is indistinguishable from a disconnect. -- Agents cannot recover from their own restarts: a serverless agent that restarts mid-stream loses its outbound stream. The client sees a dead response. -- No stateful features: presence, shared mutable state, and multi-participant observation do not exist over a one-shot HTTP request. - -[HTTP streaming and AI](/docs/ai-transport/why/http-streaming-and-ai) works through each of these issues in detail. - -## Durable sessions change the model - -A durable session drops in between your agent framework and your users. It is persistent, shared, and stateful, and it handles reconnection, ordering, multi-device sync, presence, and failover so you don't need to build them. - -![Diagram showing clients and agents attached to a single Ably AI Transport session, with durable-conversation, streaming, multi-device, tool calling, reconnection, history, and presence as the features the session provides](../../../../images/content/diagrams/ait-overview.png) - -On the client, the minimal code to adopt it: - - -```javascript -import * as Ably from 'ably'; -import { createClientSession } from '@ably/ai-transport/vercel'; - -const ably = new Ably.Realtime({ authUrl: '/api/auth/token' }); -const session = createClientSession({ - client: ably, - channelName: 'chat-123', -}); - -await session.connect(); -``` - - -A durable session provides: - -- Durable streaming: [token streams](/docs/ai-transport/features/token-streaming) persist, accumulate, and resume. Reconnecting clients [receive assembled state](/docs/ai-transport/features/reconnection-and-recovery) rather than a replay of every token. -- Session continuity: the session follows the user rather than the connection, across devices, users, and tabs. Users switch devices; the [session continues with full state](/docs/ai-transport/features/multi-device). Agents [hand off to humans](/docs/ai-transport/features/human-in-the-loop) without losing context. -- Visibility and control: the session is bidirectional. [Cancel, interrupt, and steer](/docs/ai-transport/features/interruption-and-steering) mid-response. [Push to users](/docs/ai-transport/features/push-notifications) when they are offline. Know which [agents are online](/docs/ai-transport/features/agent-presence). The session also holds state of its own. - -How direct HTTP and a durable session compare: - -| Feature | Direct HTTP | Durable session | -| --- | --- | --- | -| Resume after disconnect | Build from scratch: buffer, order, sequence-number, and add a resume endpoint. | Automatic. Client reconnects and picks up where it left off. | -| Multi-device sync | Not possible without custom infrastructure. | Any device subscribes to the same session. | -| Cancel mid-stream | Close the connection (and lose the ability to resume). | Publish a cancel signal. Stream and session survive. | -| Steer or interrupt | Requires a separate back channel. | Signal the agent through the session. | -| Multi-agent visibility | Route all updates through a single HTTP orchestrator. | Each agent publishes directly to the session. | - -## How AI Transport implements this - -The sessions in AI Transport are built on [Ably channels](/docs/channels), allowing: - -- Any client or agent to connect by specifying a channel name. -- Messages outlive any single connection, device, or agent process. -- Events arrive at subscribers in publish order, even across disconnects. -- A client that drops reconnects and picks up where it left off. -- Any participant publishes. Cancel, steer, and interrupt all happen through the same session. -- Multiple participants subscribe; every participant sees every event. - -No participant is special. A client that drops and reconnects, a serverless agent that spins up for one run and terminates, a second client joining from another device, and an orchestrator delegating to sub-agents all interact with the same session in the same way. - -The SDK provides: - -- A [codec layer](/docs/ai-transport/internals/codec-architecture) that bridges your framework's event types and Ably's message primitives, with type-safe input and output messages and [message-append](/docs/messages/updates-deletes#append) accumulation. -- A [conversation tree](/docs/ai-transport/concepts/conversation-tree) that materialises session state into a branching structure with views for pagination and branch navigation. -- [Client and agent sessions](/docs/ai-transport/concepts/sessions#connect) that own connecting to the session, the [run lifecycle](/docs/ai-transport/concepts/runs), and cancel routing. -- React hooks for building UIs with streaming, pagination, and branch navigation. -- Adapters that drop into existing frameworks. AI Transport plugs directly into Vercel AI SDK, Temporal, and Vercel WDK. - -## When you don't need this - -Quick, single-turn chatbots do not need this. AI Transport is for experiences that are long-lived, agentic, and interactive, where sessions span conversations, devices, and time. If your users start and finish in one request, direct HTTP streaming is the simpler choice. -AI Transport is also not the right tool for conversations between people: for human-to-human chat such as group messaging or human-staffed support, use the [Chat SDK](/docs/chat). - -## Why Ably - -AI Transport runs on infrastructure that has been delivering realtime experiences at scale for over a decade: - -- Trillions of realtime transactions monthly. -- Billions of devices reached. -- Seven years of zero global downtime. -- Global edge network, multi-region, SOC 2 Type II, HIPAA-compliant. - -The hard problems of running stateful infrastructure (ordering, persistence, replication, presence, failover) are already solved. AI Transport inherits all of it. Ably also runs a bug bounty programme with independent security researchers, and [security and compliance](https://ably.com/security-and-compliance) lists the current certifications. - -## Read next