perf(chat): reduce preparation latency and use Hyperdrive

docs/tanchat-send-latency.md

@@ -0,0 +1,43 @@
+# TanChat send latency, October 4, 2026
+
+## What changed
+
+Ordinary messages no longer load the entire MCP connection inventory or skill catalog before calling the model. The model can discover tools and skills when it needs them. Explicit tool selections still load their inventory, and selected skill and plugin versions remain pinned.
+
+Independent admission checks run together and settle before their database context is released. Bot lifecycle, workspace context, memory preferences, and thread source use fewer database statements. Each authorization snapshot still checks its exact scope against current database state. Post-preparation checks, reset epochs, receipt matching, and runtime tool authorization remain in place.
+
+Usage receipts, reservations, and counters publish in one statement under the existing transaction and locks. Duplicate sends, concurrent cap enforcement, scheduled usage, funded reservations, and admin policy retain their existing behavior.
+
+Automatic Kody memory and account guidance remain enabled and overlap with account preferences and thread source reads. Their latency is not represented by the no-credential benchmark below.
+
+Native PostgreSQL clients allow five connections, matching the existing Workers limit, so independent reads can actually overlap. Copy activation now explicitly waits for its activity publication before checking it. That ordering cannot depend on all queries sharing one socket.
+
+## Controlled measurements
+
+The baseline is commit 1bbafeec, the changes merged through PR 1339. Both versions run the actual conversation admission and preparation code against disposable PostgreSQL. The provider is synthetic. Each version runs three messages, the first includes cold client setup.
+
+| Database connection                              | Baseline, milliseconds | Candidate, milliseconds |
+| ------------------------------------------------ | ---------------------- | ----------------------- |
+| Delayed TCP proxy, nominal 80 ms round trip      | 5406, 5140, 5148       | 2610, 2210, 2207        |
+| Delayed proxy, five connections in both versions | 5358, 4984, 4980       | 2610, 2210, 2207        |
+| Same-machine PostgreSQL, no proxy delay          | Not measured           | 60, 16, 13              |
+
+Warm preparation in the delayed fixture fell from about 5.14 seconds to 2.21 seconds, about 57 percent. The proxy delays chunks, so protocol startup can incur more than one nominal round trip. The native baseline uses one connection and the candidate uses five. Workers already allowed five. With both versions using five connections, warm preparation fell from 4.98 seconds to 2.21 seconds, about 56 percent. Neither comparison predicts the production improvement.
+
+These numbers measure the beginning of durable admission to provider dispatch. They exclude HTTP authentication, browser rendering, real model time to first token, and real Kody network reads. They do not establish instant live responses.
+
+## Remaining production work
+
+The production configuration now binds HYPERDRIVE to the cache-disabled tanstack-com pool, 6ec1513b87bb47e9897e5502727b0388, created after user approval on October 4. The host already supports HYPERDRIVE.connectionString and otherwise uses DATABASE_URL. A new pool must target this site's database, not the existing unrelated dashboard demo pool.
+
+Cloudflare confirmed query caching is disabled and the origin connection limit is twenty. Local development omits the production binding and continues using its optional DATABASE_URL. The binding takes effect on the live site when this configuration is deployed. Local startup was verified with CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID explicitly empty, and the temporary test server was stopped. No deployment was performed during pool provisioning.
+
+Hyperdrive pools connections to reduce TCP, TLS, and authentication setup. It does not remove every SQL round trip or relocate existing Durable Objects. Region placement needs a measurement against the actual database and conversation objects before changing hints.
+
+Sources: [Hyperdrive connection pooling](https://developers.cloudflare.com/hyperdrive/concepts/how-hyperdrive-works/) and [Durable Object location](https://developers.cloudflare.com/durable-objects/reference/data-location/).
+
+## Verification
+
+The runtime suite exercises real PostgreSQL, including access revocation, reset races, retries, copy activation, quota concurrency, funded usage, response preference persistence, and skill discovery through the actual SDK loop. The lazy MCP test also uses the SDK loop with a synthetic transport.
+
+The local Assistant page rendered from the QA checkout on localhost:3000. No real provider messages were sent during this pass. Full pnpm test passed: all five TypeScript configurations, lint with zero errors and 105 existing warnings, 537 site tests, 2,274 chat tests, and 20 desktop tests. Five optional tests were skipped. The complete PostgreSQL runtime suite passed 268 tests in 35 files. The final metadata scope suite passed 10 tests, including four additional thread-scope rejection cases.

harness-tests/core/assistant-mcp-lazy.test.ts

@@ -0,0 +1,69 @@
+import { expect, it, vi } from 'vitest'
+import { chat, maxIterations } from '@tanstack/ai'
+import { createCloudflareText } from '@tanstack/ai-cloudflare'
+import { assistantMcpTools } from '../../src/chat/server/assistant-mcp-tools'
+import type { McpConnection } from '../../src/chat/server/mcp'
+
+it('loads current inventory only when the SDK invokes discovery', async () => {
+  const connections: McpConnection[] = []
+  const loadConnections = vi.fn(async () => {
+    connections.splice(0, connections.length, {
+      id: 'current',
+      label: 'Current service',
+      url: 'https://example.invalid/mcp',
+    })
+  })
+  const tools = assistantMcpTools({
+    connections,
+    loadConnections,
+    catalog: vi.fn(),
+    propose: vi.fn(),
+    read: vi.fn(),
+  })
+  expect(loadConnections).not.toHaveBeenCalled()
+  let pass = 0
+  const fetch = vi.fn(async () => {
+    const first = pass++ === 0
+    const delta = first
+      ? {
+          role: 'assistant',
+          tool_calls: [
+            {
+              index: 0,
+              id: 'inventory',
+              type: 'function',
+              function: { name: 'list_connected_tools', arguments: '{}' },
+            },
+          ],
+        }
+      : { role: 'assistant', content: 'Finished.' }
+    return new Response(
+      'data: ' +
+        JSON.stringify({
+          id: 'synthetic',
+          choices: [
+            { index: 0, delta, finish_reason: first ? 'tool_calls' : 'stop' },
+          ],
+        }) +
+        '\n\ndata: [DONE]\n\n',
+      { headers: { 'Content-Type': 'text/event-stream' } },
+    )
+  })
+  const adapter = createCloudflareText('synthetic', {
+    accountId: 'synthetic',
+    apiKey: 'synthetic',
+    fetch,
+  })
+  const results: unknown[] = []
+  for await (const chunk of chat({
+    adapter,
+    tools,
+    messages: [{ role: 'user', content: 'List my connected services.' }],
+    agentLoopStrategy: maxIterations(2),
+  })) {
+    if (chunk.type === 'TOOL_CALL_RESULT') results.push(chunk.content)
+  }
+  expect(loadConnections).toHaveBeenCalledOnce()
+  expect(JSON.stringify(results)).toContain('Current service')
+  expect(fetch).toHaveBeenCalledTimes(2)
+})

harness-tests/pending-runtime/assistant-discovery-runtime.test.ts

@@ -43,7 +43,7 @@ const system = (request: any) =>
     .map((message: any) => message.content)
     .join('\n')
 
-it('exposes authorized skill metadata on the first request and loads the full skill only after reading', async () => {
+it('discovers skill metadata on demand and loads full instructions only after reading', async () => {
   const h = await conversationHarness()
   const skillId = crypto.randomUUID()
   await new Skills(identity).command({
@@ -60,9 +60,11 @@ it('exposes authorized skill metadata on the first request and loads the full sk
   const requests: any[] = []
   const run = vi.fn(async (_model, request) => {
     requests.push(structuredClone(request))
-    return requests.length === 1
-      ? reply('read_skill', { skillId, version: 1 })
-      : reply()
+    if (requests.length === 1)
+      return reply('list_skills', { query: 'test-review' })
+    if (requests.length === 2)
+      return reply('read_skill', { skillId, version: 1 })
+    return reply()
   })
   Object.assign(h.env, { AI: { run } })
   await h.c.begin({
@@ -72,13 +74,20 @@ it('exposes authorized skill metadata on the first request and loads the full sk
   })
   await h.settle()
   expect((await h.c.snapshot()).error).toBeUndefined()
-  expect(requests).toHaveLength(2)
-  expect(system(requests[0])).toContain(skillId)
-  expect(system(requests[0])).toContain('Review tests for useful coverage.')
+  expect(requests).toHaveLength(3)
+  expect(system(requests[0])).not.toContain(skillId)
+  expect(names(requests[0])).toContain('list_skills')
+  expect(JSON.stringify(requests[1].messages)).toContain(skillId)
+  expect(JSON.stringify(requests[1].messages)).toContain(
+    'Review tests for useful coverage.',
+  )
   expect(system(requests[0])).not.toContain(
     'INSTRUCTIONS_LOADED_ONLY_AFTER_READ',
   )
-  expect(system(requests[1])).toContain('INSTRUCTIONS_LOADED_ONLY_AFTER_READ')
+  expect(system(requests[1])).not.toContain(
+    'INSTRUCTIONS_LOADED_ONLY_AFTER_READ',
+  )
+  expect(system(requests[2])).toContain('INSTRUCTIONS_LOADED_ONLY_AFTER_READ')
   expect((await h.c.snapshot()).assistantTask?.loadedSkills).toEqual([
     { skillId, version: 1 },
   ])

harness-tests/pending-runtime/conversation-metadata.test.ts

@@ -0,0 +1,96 @@
+import { expect, it } from 'vitest'
+import { conversationHarness } from './fixtures/conversation-runtime'
+import { conversationThreadContext } from '../../src/chat/server/conversation-threads'
+import { Memories } from '../../src/chat/server/memory'
+import {
+  readConversationLifecycle,
+  readConversationRunContext,
+} from '../../src/chat/server/conversation-database'
+
+it('reads fresh run context and ordered saved actions from a single database snapshot', async () => {
+  const h = await conversationHarness()
+  const input = h.input('metadata')
+  const identity = { workspaceId: 'w', botId: 'b', userId: input.userId }
+  await h.db`INSERT INTO chat_recipes(id,workspace_id,title,description,code,created_at) VALUES('second','w','Second','','',2),('first','w','First','','',1)`
+  const result = await readConversationRunContext(identity)
+  expect(result.bot).toMatchObject({ id: 'b', workspace_id: 'w', name: 'Bot' })
+  expect(typeof result.bot.created_at).toBe('number')
+  expect(result.recipes).toEqual([
+    expect.objectContaining({ id: 'first' }),
+    expect.objectContaining({ id: 'second' }),
+  ])
+  expect((await readConversationRunContext(identity, false)).recipes).toEqual(
+    [],
+  )
+  await h.db`DELETE FROM chat_memberships WHERE workspace_id='w'`
+  await expect(readConversationRunContext(identity)).rejects.toThrow(
+    'Workspace access is unavailable.',
+  )
+})
+
+it('returns lifecycle nulls and fresh archive state without losing the missing-bot distinction', async () => {
+  const h = await conversationHarness()
+  expect(await readConversationLifecycle('missing', 'missing')).toEqual({
+    bot: null,
+    thread: null,
+  })
+  expect(await readConversationLifecycle('b', 'main-conversation')).toEqual({
+    bot: { archived_at: null, deleted_at: null },
+    thread: null,
+  })
+  await h.db`UPDATE chat_bots SET archived_at=to_timestamp(100) WHERE id='b'`
+  expect(
+    await readConversationLifecycle('b', 'main-conversation'),
+  ).toMatchObject({ bot: { archived_at: 100000 } })
+})
+
+it.each(['membership', 'owner', 'workspace', 'deleted'])(
+  'does not expose memory preferences after %s access changes',
+  async (change) => {
+    const h = await conversationHarness()
+    const userId = h.input('metadata').userId
+    const scope = {
+      workspaceId: 'w',
+      conversationId: 'main-conversation',
+      userId,
+    }
+    const memories = new Memories(scope)
+    expect(await memories.preferences()).toEqual({
+      enabled: false,
+      revision: 0,
+    })
+    await h.db`INSERT INTO chat_memory_preferences(conversation_id,enabled,revision,updated_at) VALUES('main-conversation',true,7,1)`
+    expect(await memories.preferences()).toEqual({ enabled: true, revision: 7 })
+    if (change === 'membership')
+      await h.db`DELETE FROM chat_memberships WHERE workspace_id='w'`
+    if (change === 'owner')
+      scope.userId = '00000000-0000-4000-8000-000000000002'
+    if (change === 'workspace') scope.workspaceId = 'foreign'
+    if (change === 'deleted')
+      await h.db`UPDATE chat_bots SET deleted_at=now() WHERE id='b'`
+    await expect(memories.preferences()).rejects.toThrow(
+      'Conversation access is unavailable.',
+    )
+  },
+)
+
+it.each(['membership', 'owner', 'workspace', 'bot'])(
+  'rejects thread context after %s scope changes',
+  async (change) => {
+    const h = await conversationHarness()
+    const identity = {
+      workspaceId: 'w',
+      botId: 'b',
+      conversationId: 'main-conversation',
+      userId: h.input('metadata').userId,
+    }
+    expect(await conversationThreadContext(identity)).toBeUndefined()
+    if (change === 'membership')
+      await h.db`DELETE FROM chat_memberships WHERE workspace_id='w'`
+    if (change === 'owner')
+      identity.userId = '00000000-0000-4000-8000-000000000002'
+    if (change === 'workspace') identity.workspaceId = 'foreign'
+    if (change === 'bot') identity.botId = 'foreign'
+    await expect(conversationThreadContext(identity)).rejects.toThrow()
+  },
+)

harness-tests/pending-runtime/conversation-preparation.test.ts

@@ -7,28 +7,18 @@ import { SkillCatalog } from '../../src/chat/server/skill-catalog'
 
 afterEach(() => vi.restoreAllMocks())
 
-it('reads skill metadata while MCP inventory is pending and waits for both before calling the model', async () => {
+it('dispatches a plain response without reading MCP inventory or the skill directory', async () => {
   const h = await conversationHarness()
-  let releaseInventory = () => {}
-  const inventoryGate = new Promise<void>((resolve) => {
-    releaseInventory = resolve
-  })
-  let releaseDirectory = () => {}
-  const directoryGate = new Promise<void>((resolve) => {
-    releaseDirectory = resolve
-  })
   const inventory = vi
     .spyOn(connections, 'connectedMcpServers')
-    .mockImplementation(async () => {
-      await inventoryGate
-      return []
-    })
+    .mockRejectedValue(
+      new Error('Unused inventory must not block a plain response'),
+    )
   const directory = vi
     .spyOn(SkillCatalog.prototype, 'list')
-    .mockImplementation(async () => {
-      await directoryGate
-      return { items: [] }
-    })
+    .mockRejectedValue(
+      new Error('Unused directory must not block a plain response'),
+    )
   h.env.AI.run.mockImplementation(
     async () =>
       new Response(
@@ -51,52 +41,18 @@ it('reads skill metadata while MCP inventory is pending and waits for both befor
         { headers: { 'Content-Type': 'text/event-stream' } },
       ),
   )
-  try {
-    await h.c.begin({ ...h.input('parallel-preparation'), fixture: false })
-    await vi.waitFor(() => expect(inventory).toHaveBeenCalledOnce())
-    await vi.waitFor(() => expect(directory).toHaveBeenCalledOnce())
-    expect(h.env.AI.run).not.toHaveBeenCalled()
-    releaseInventory()
-    await Promise.resolve()
-    expect(h.env.AI.run).not.toHaveBeenCalled()
-    releaseDirectory()
-    await h.settle()
-    expect(h.env.AI.run).toHaveBeenCalledOnce()
-  } finally {
-    releaseInventory()
-    releaseDirectory()
-    await h.settle()
-  }
-})
-
-it('settles the directory read before reporting an inventory failure and never calls the model', async () => {
-  const h = await conversationHarness()
-  let releaseDirectory = () => {}
-  const directoryGate = new Promise<void>((resolve) => {
-    releaseDirectory = resolve
-  })
-  vi.spyOn(connections, 'connectedMcpServers').mockRejectedValue(
-    new Error('Synthetic inventory failure'),
-  )
-  const directory = vi
-    .spyOn(SkillCatalog.prototype, 'list')
-    .mockImplementation(async () => {
-      await directoryGate
-      return { items: [] }
-    })
-  try {
-    await h.c.begin({ ...h.input('failed-preparation'), fixture: false })
-    await vi.waitFor(() => expect(directory).toHaveBeenCalledOnce())
-    expect((await h.c.snapshot()).status).toBe('running')
-    expect(h.env.AI.run).not.toHaveBeenCalled()
-    releaseDirectory()
-    await h.settle()
-    expect((await h.c.snapshot()).error).toBe('Synthetic inventory failure')
-    expect(h.env.AI.run).not.toHaveBeenCalled()
-  } finally {
-    releaseDirectory()
-    await h.settle()
-  }
+  await h.c.begin({ ...h.input('deferred-preparation'), fixture: false })
+  await h.settle()
+  expect(h.env.AI.run).toHaveBeenCalledOnce()
+  expect(inventory).not.toHaveBeenCalled()
+  expect(directory).not.toHaveBeenCalled()
+  expect(
+    (await h.c.snapshot()).messages.some((message) =>
+      message.parts.some(
+        (part) => part.type === 'text' && part.content === 'Ready.',
+      ),
+    ),
+  ).toBe(true)
 })
 
 it('reauthorizes membership inside send admission without requiring a prior identity RPC', async () => {

harness-tests/pending-runtime/plugin-reference-runtime.test.ts

@@ -259,7 +259,7 @@ it('pins a selected old package before dispatch and discovers its exact old skil
     state.assistantTask?.loadedPlugins,
   )
   expect(h.network).not.toHaveBeenCalled()
-  expect(discovery).toHaveBeenCalled()
+  expect(discovery).not.toHaveBeenCalled()
   discovery.mockRestore()
 })
 

src/chat/server/assistant-instructions.ts

@@ -299,6 +299,12 @@ export function buildAssistantInstructions(
         16000,
       ),
     )
+  if (!options.skillDirectory && enabled.has('list_skills'))
+    sections.push({
+      id: 'core.skill-discovery',
+      version: '1',
+      text: 'Skill metadata loads on demand. When a skill may help the request, search with list_skills, then read its exact id and version with read_skill before following it. An unloaded catalog does not mean no skills exist. Explicitly selected skills and their versions remain part of this request.',
+    })
   if (options.workflowInputs?.length && enabled.has('read_stored_result'))
     sections.push(
       dynamicSection(