Skip to content

Commit 17ae3bc

Browse files
authored
feat(mcp): let agent tools return MCP content blocks (#451)
1 parent f5b0b33 commit 17ae3bc

17 files changed

Lines changed: 349 additions & 9 deletions

File tree

‎docs/content/1.guide/15.agent-native.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,29 @@ ctx.agent.registerTool({
189189
})
190190
```
191191

192+
## Returning images and other content
193+
194+
The MCP adapter sends a tool result as one text block that holds the result as JSON. To send other MCP content, such as a screenshot next to its description, attach content blocks to the result with `withMcpContent`:
195+
196+
```ts
197+
import { defineRpcFunction, withMcpContent } from 'devframe'
198+
199+
export const captureComponent = defineRpcFunction({
200+
name: 'my-plugin:capture-component',
201+
type: 'query',
202+
agent: { description: 'Take a screenshot of one component. Call it when the layout of the component matters.' },
203+
handler: async () => {
204+
const { markdown, png } = await capture()
205+
return withMcpContent({ markdown }, [
206+
{ type: 'text', text: markdown },
207+
{ type: 'image', data: png, mimeType: 'image/png' },
208+
])
209+
},
210+
})
211+
```
212+
213+
The adapter sends the blocks in place of the JSON text block. A block is `text`, `image` or `audio`, and `data` is base64. The blocks are stored under a hidden key, so the returned value does not change: the SPA gets `{ markdown }` over RPC, without the image data. If the tool has an output schema, `structuredContent` is also that plain value. Invalid blocks make the call fail with [`DF0081`](/errors/DF0081).
214+
192215
## Structured errors
193216

194217
A coded diagnostic thrown from a handler crosses the MCP boundary as JSON:

‎docs/content/6.errors/DF0081.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
title: 'DF0081: Invalid MCP Content Blocks'
3+
description: 'Tool "{name}" returned invalid MCP content blocks: {reason}'
4+
---
5+
6+
## Message
7+
8+
> Tool "`{name}`" returned invalid MCP content blocks: `{reason}`
9+
10+
## Cause
11+
12+
A tool result carries content blocks from `withMcpContent`, but the blocks are not a non-empty array of valid blocks. A valid block is `{ type: 'text', text }`, `{ type: 'image', data, mimeType }` or `{ type: 'audio', data, mimeType }`, and each field is a string. The MCP adapter does not send invalid blocks. It returns this error as an `isError` result of the `tools/call` request.
13+
14+
## Example
15+
16+
```ts
17+
import { withMcpContent } from 'devframe'
18+
19+
ctx.agent.registerTool({
20+
id: 'my-plugin:snapshot',
21+
description: 'Take a snapshot.',
22+
// The image block has no `mimeType`.
23+
handler: () => withMcpContent({}, [{ type: 'image', data: png }]),
24+
})
25+
```
26+
27+
## Fix
28+
29+
Give `withMcpContent` at least one block, and give each block all of its fields: `text` for a text block, base64 `data` and a `mimeType` (for example `image/png`) for an image or audio block.
30+
31+
## Source
32+
33+
- [`packages/agentic/src/mcp/build-server.ts`](https://github.com/devframes/devframe/blob/main/packages/agentic/src/mcp/build-server.ts): `mcpContentOf()` throws this when the content blocks of a `tools/call` result are invalid.

‎docs/content/6.errors/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ Emitted by `devframe`: the framework-neutral host, RPC, streaming, assets, servi
8686
| [DF0077](/errors/DF0077) | error | In-Page Channel Function Not Registered |
8787
| [DF0078](/errors/DF0078) | warn | Agent Surface Without @devframes/agentic |
8888
| [DF0079](/errors/DF0079) | error | MCP Enabled Without @devframes/agentic |
89+
| [DF0081](/errors/DF0081) | error | Invalid MCP Content Blocks |
8990

9091
## Hub: context & lifecycle (DF80xx)
9192

‎packages/agentic/src/mcp/__tests__/mcp-http.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
import type { StartedServer } from 'devframe/internal'
22
import type { DevframeDefinition, McpRouteOptions } from 'devframe/types'
33
import { Client, StreamableHTTPClientTransport } from '@modelcontextprotocol/client'
4+
import { withMcpContent } from 'devframe'
45
import { createDevServer } from 'devframe/adapters/dev'
56
import { afterEach, describe, expect, it } from 'vitest'
67

78
const TOKEN = 'a-high-entropy-test-bearer-token'
9+
/** A 1x1 PNG. */
10+
const PNG_BASE64 = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFBQIAX8jx0gAAAABJRU5ErkJggg=='
811

912
function defineTestDef(overrides?: Partial<DevframeDefinition>): DevframeDefinition {
1013
return {
@@ -95,6 +98,38 @@ describe('mcp adapter (streamable http route)', () => {
9598
}
9699
})
97100

101+
it('sends the content blocks attached with withMcpContent', async () => {
102+
const started = await boot(true, defineTestDef({
103+
setup(ctx) {
104+
ctx.agent.registerTool({
105+
id: 'snapshot',
106+
description: 'Take a snapshot.',
107+
safety: 'read',
108+
handler: () => withMcpContent({ markdown: 'A red pixel.' }, [
109+
{ type: 'text', text: 'A red pixel.' },
110+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
111+
]),
112+
})
113+
},
114+
}))
115+
const client = new Client(
116+
{ name: 'test-client', version: '0.0.0' },
117+
{ versionNegotiation: { mode: 'auto' } },
118+
)
119+
try {
120+
await client.connect(originTransport(started))
121+
const result = await client.callTool({ name: 'snapshot', arguments: {} })
122+
expect(result.isError).toBeFalsy()
123+
expect(result.content).toEqual([
124+
{ type: 'text', text: 'A red pixel.' },
125+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
126+
])
127+
}
128+
finally {
129+
await client.close()
130+
}
131+
})
132+
98133
it('answers a bare GET with 405 (no session lifecycle)', async () => {
99134
const started = await boot()
100135
// Stateless serving has no session stream to open, so the SDK answers a

‎packages/agentic/src/mcp/__tests__/mcp-server.test.ts‎

Lines changed: 115 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,27 @@
1-
import type { DevframeHost } from 'devframe/types'
1+
import type { StandardSchemaV1 } from '@standard-schema/spec'
2+
import type { DevframeHost, McpContentBlock } from 'devframe/types'
23
import { Client, InMemoryTransport } from '@modelcontextprotocol/client'
4+
import { withMcpContent } from 'devframe'
35
import { createHostContext } from 'devframe/node'
46
import * as v from 'valibot'
57
import { describe, expect, it } from 'vitest'
68
import { buildMcpServerFromContext } from '../build-server'
79

10+
/** A 1x1 PNG. */
11+
const PNG_BASE64 = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8DwHwAFBQIAX8jx0gAAAABJRU5ErkJggg=='
12+
13+
/** A Standard Schema that accepts any value and converts to `jsonSchema`. */
14+
function objectSchema(jsonSchema: Record<string, unknown>): StandardSchemaV1 {
15+
return {
16+
'~standard': {
17+
version: 1,
18+
vendor: 'test',
19+
validate: (value: unknown) => ({ value }),
20+
jsonSchema: { input: () => jsonSchema, output: () => jsonSchema },
21+
} as StandardSchemaV1['~standard'],
22+
}
23+
}
24+
825
function nullHost(): DevframeHost {
926
return {
1027
mountStatic: () => { /* no-op */ },
@@ -187,6 +204,103 @@ describe('mcp adapter (in-memory)', () => {
187204
}
188205
})
189206

207+
it('sends the content blocks attached with withMcpContent in place of the JSON text', async () => {
208+
const { ctx, client, cleanup } = await bootPair()
209+
try {
210+
ctx.agent.registerTool({
211+
id: 'snapshot',
212+
description: 'Take a snapshot.',
213+
handler: () => withMcpContent({ markdown: 'A red pixel.' }, [
214+
{ type: 'text', text: 'A red pixel.' },
215+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
216+
{ type: 'audio', data: 'AAAA', mimeType: 'audio/wav' },
217+
]),
218+
})
219+
220+
const result = await client.callTool({ name: 'snapshot', arguments: {} })
221+
expect(result.isError).toBeFalsy()
222+
expect(result.content).toEqual([
223+
{ type: 'text', text: 'A red pixel.' },
224+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
225+
{ type: 'audio', data: 'AAAA', mimeType: 'audio/wav' },
226+
])
227+
expect(result.structuredContent).toBeUndefined()
228+
}
229+
finally {
230+
await cleanup()
231+
}
232+
})
233+
234+
it('keeps the plain result as structured content for an rpc-backed tool with an output schema', async () => {
235+
const { ctx, client, cleanup } = await bootPair()
236+
try {
237+
ctx.rpc.register({
238+
name: 'snapshot',
239+
type: 'query',
240+
jsonSerializable: true,
241+
args: [],
242+
returns: objectSchema({ type: 'object', properties: { markdown: { type: 'string' } }, required: ['markdown'] }),
243+
agent: { description: 'Take a snapshot.' },
244+
handler: () => withMcpContent({ markdown: 'A red pixel.' }, [
245+
{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' },
246+
]),
247+
} as never)
248+
249+
const listed = await client.listTools()
250+
const tool = listed.tools.find(t => t.name.endsWith('snapshot'))!
251+
expect(tool.outputSchema).toBeDefined()
252+
253+
const result = await client.callTool({ name: tool.name, arguments: {} })
254+
expect(result.isError).toBeFalsy()
255+
expect(result.content).toEqual([{ type: 'image', data: PNG_BASE64, mimeType: 'image/png' }])
256+
expect(result.structuredContent).toEqual({ markdown: 'A red pixel.' })
257+
}
258+
finally {
259+
await cleanup()
260+
}
261+
})
262+
263+
it('returns DF0081 when the attached content blocks are invalid', async () => {
264+
const { ctx, client, cleanup } = await bootPair()
265+
try {
266+
ctx.agent.registerTool({
267+
id: 'broken',
268+
description: 'Returns a bad block.',
269+
handler: () => withMcpContent({}, [{ type: 'image', data: PNG_BASE64 } as never]),
270+
})
271+
272+
const result = await client.callTool({ name: 'broken', arguments: {} })
273+
expect(result.isError).toBe(true)
274+
const content = result.content as Array<{ type: string, text: string }>
275+
expect(content[0]!.text).toContain('DF0081')
276+
}
277+
finally {
278+
await cleanup()
279+
}
280+
})
281+
282+
it('returns DF0081 when the attached content blocks have holes', async () => {
283+
const { ctx, client, cleanup } = await bootPair()
284+
try {
285+
// eslint-disable-next-line unicorn/no-new-array
286+
const sparse = new Array(2) as McpContentBlock[]
287+
sparse[1] = { type: 'text', text: 'second' }
288+
ctx.agent.registerTool({
289+
id: 'sparse',
290+
description: 'Returns a sparse block array.',
291+
handler: () => withMcpContent({}, sparse),
292+
})
293+
294+
const result = await client.callTool({ name: 'sparse', arguments: {} })
295+
expect(result.isError).toBe(true)
296+
const content = result.content as Array<{ type: string, text: string }>
297+
expect(content[0]!.text).toContain('DF0081')
298+
}
299+
finally {
300+
await cleanup()
301+
}
302+
})
303+
190304
it('calls an rpc-backed tool whose return schema has no native converter', async () => {
191305
const { ctx, client, cleanup } = await bootPair()
192306
try {

‎packages/agentic/src/mcp/build-server.ts‎

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
import type { Tool } from '@modelcontextprotocol/server'
22
import type { StandardSchemaV1 } from '@standard-schema/spec'
33
import type { RpcFunctionDefinitionAnyWithContext } from 'devframe/rpc'
4-
import type { AgentTool, CreateMcpServerOptions, DevframeDefinition, DevframeHost, DevframeNodeContext, McpServerHandle } from 'devframe/types'
4+
import type { AgentTool, CreateMcpServerOptions, DevframeDefinition, DevframeHost, DevframeNodeContext, McpContentBlock, McpServerHandle } from 'devframe/types'
55
import { homedir } from 'node:os'
66
import process from 'node:process'
77
import { Server } from '@modelcontextprotocol/server'
88
import { DEVFRAME_EVENTS } from 'devframe/constants'
9-
import { argsToJsonSchema, diagnostics, formatMcpError, returnToJsonSchema, stringifyForMcp } from 'devframe/internal'
9+
import { argsToJsonSchema, diagnostics, formatMcpError, getMcpContent, returnToJsonSchema, stringifyForMcp } from 'devframe/internal'
1010
import { createHostContext } from 'devframe/node'
1111
import { toAgentToolName } from 'devframe/utils/agent-tool-name'
1212
import { join } from 'pathe'
@@ -277,13 +277,12 @@ function registerToolHandlers(
277277
? usableOutputSchema(tool.outputSchema ?? computeOutputSchema(tool, ctx))
278278
: undefined
279279
const result = await ctx.agent.invoke(tool?.id ?? name, args ?? {})
280+
// Blocks attached with `withMcpContent` replace the default JSON text
281+
// block. `structuredContent` stays the plain result either way, so a
282+
// tool with an output schema still matches it.
283+
const content = mcpContentOf(name, result) ?? [{ type: 'text', text: stringifyForMcp(result) }]
280284
return {
281-
content: [
282-
{
283-
type: 'text',
284-
text: stringifyForMcp(result),
285-
},
286-
],
285+
content,
287286
...(outputSchema ? { structuredContent: result as Record<string, unknown> } : {}),
288287
}
289288
}
@@ -372,6 +371,29 @@ function registerResourceHandlers(
372371
})
373372
}
374373

374+
/**
375+
* The content blocks a tool result carries through `withMcpContent`, checked
376+
* and copied field by field, or `undefined` when it carries none. Invalid
377+
* blocks throw DF0081, which the `tools/call` handler turns into an
378+
* `isError` result.
379+
*/
380+
function mcpContentOf(name: string, result: unknown): McpContentBlock[] | undefined {
381+
const blocks = getMcpContent(result)
382+
if (blocks === undefined)
383+
return undefined
384+
if (!Array.isArray(blocks) || blocks.length === 0)
385+
throw diagnostics.DF0081({ name, reason: 'expected a non-empty array' })
386+
// `Array.from` visits the holes of a sparse array as `undefined`, so they fail the check below.
387+
return Array.from(blocks, (block: unknown, index): McpContentBlock => {
388+
const b = (block ?? {}) as Record<string, unknown>
389+
if (b.type === 'text' && typeof b.text === 'string')
390+
return { type: 'text', text: b.text }
391+
if ((b.type === 'image' || b.type === 'audio') && typeof b.data === 'string' && typeof b.mimeType === 'string')
392+
return { type: b.type, data: b.data, mimeType: b.mimeType }
393+
throw diagnostics.DF0081({ name, reason: `block ${index} is not a valid text, image or audio block` })
394+
})
395+
}
396+
375397
/**
376398
* MCP constrains a tool's `outputSchema` to a JSON Schema of `type:
377399
* "object"`; clients (the SDK included) reject anything else. Non-object
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
import { describe, expect, it } from 'vitest'
2+
import { getMcpContent, withMcpContent } from '../mcp-content'
3+
4+
describe('withMcpContent', () => {
5+
const blocks = [
6+
{ type: 'text', text: 'A red pixel.' },
7+
{ type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png' },
8+
] as const
9+
10+
it('attaches the blocks without changing the value', () => {
11+
const value = { markdown: 'A red pixel.' }
12+
const result = withMcpContent(value, blocks)
13+
expect(result).toBe(value)
14+
expect(getMcpContent(result)).toBe(blocks)
15+
expect(Object.keys(result)).toEqual(['markdown'])
16+
expect(JSON.stringify(result)).toBe('{"markdown":"A red pixel."}')
17+
expect(structuredClone(result)).toEqual({ markdown: 'A red pixel.' })
18+
expect(getMcpContent(structuredClone(result))).toBeUndefined()
19+
})
20+
21+
it('reads nothing from values without blocks', () => {
22+
expect(getMcpContent({ markdown: 'x' })).toBeUndefined()
23+
expect(getMcpContent('text')).toBeUndefined()
24+
expect(getMcpContent(null)).toBeUndefined()
25+
expect(getMcpContent(undefined)).toBeUndefined()
26+
})
27+
})
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
import type { McpContentBlock } from '../types/mcp'
2+
3+
/**
4+
* Registry symbol for the content blocks attached by {@link withMcpContent}.
5+
* `Symbol.for` keeps the key the same across duplicate copies of devframe.
6+
*/
7+
const MCP_CONTENT = Symbol.for('devframe:mcp-content')
8+
9+
/**
10+
* Attach MCP content blocks to a tool result, for example an image next to
11+
* its text. The MCP adapter sends these blocks as the `content` of the
12+
* `tools/call` result, in place of the default single JSON text block.
13+
*
14+
* The blocks are stored under a non-enumerable symbol key, so `value` itself
15+
* does not change: the RPC transport to the SPA, `returns` validation, and the
16+
* `structuredContent` of a tool with an output schema all see the plain value
17+
* and never the (often large) block data.
18+
*
19+
* `value` must be an object that is not frozen. Returns `value`.
20+
*/
21+
export function withMcpContent<T extends object>(value: T, content: readonly McpContentBlock[]): T {
22+
Object.defineProperty(value, MCP_CONTENT, {
23+
value: content,
24+
enumerable: false,
25+
configurable: true,
26+
writable: true,
27+
})
28+
return value
29+
}
30+
31+
/**
32+
* Read the blocks attached by {@link withMcpContent}, or `undefined` when the
33+
* value carries none. The value is not validated here; the MCP adapter checks
34+
* each block before it sends it.
35+
*
36+
* @internal
37+
*/
38+
export function getMcpContent(value: unknown): unknown {
39+
if (value === null || typeof value !== 'object')
40+
return undefined
41+
return (value as Record<symbol, unknown>)[MCP_CONTENT]
42+
}

‎packages/devframe/src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,6 @@
22
// `defineRpcFunction`) as values, plus every public type. `devframe/types`
33
// remains available as the type-only subpath, but `devframe` is the canonical
44
// import for both values and types.
5+
export { withMcpContent } from './agent/mcp-content'
56
export * from './define'
67
export type * from './types'

0 commit comments

Comments
 (0)