Was it a bad idea to stream the text generation process using Context.report_progress()?

[lottopotato]

Question

I built a chatbot server using VLLM's AsyncLLMEngine and an MCP Streamable HTTP server via its Python SDK.

To stream the token generation process from the AsyncLLMEngine generator to the MCP client, I used context.report_progress as shown below.

...
foundation_model = AsyncLLMEngine.from_engine_args(
    AsyncEngineArgs(
        ...
    )
)


...

@server.tool()
async def chat(q:str, ctx: Context):

    messages = get_messages(q)
    generator = foundation_model.generate(messages, ...)
    
    async for request_output in generator:
        current_text = request_output.outputs[0].text
        next_text = current_text[len(answer):]
        if ctx is not None and isinstance(ctx, Context):
            await ctx.report_progress(i, None, next_text)
        answer = current_text
        i += 1
    

Is this usage appropriate for streaming the token generation process?

Is there another way?

Additional Context

No response

[maxisbey]

Yes, that works, and it's a reasonable use of the API when you control the client: ctx.report_progress(progress, total, message) emits notifications/progress while the tool is still running, and message is free-form text, so token deltas fit — see the Progress guide. Two caveats: progress must strictly increase on every report (your counter does that), and it's a silent no-op unless the client opts in per call by passing progress_callback= to call_tool, so a client that isn't listening gets nothing until the full result arrives. Progress is a status side-channel rather than part of the tool result — the tool still has to return the complete text at the end, and clients aren't obliged to render message. On "another way": the spec has no partial/streaming tool results yet, so there's no portable token-streaming alternative; #472 has the spec-side proposals to follow. Your pattern boiled down against the v2 API, with the client side included:

Example

import anyio
from mcp import Client
from mcp.server import MCPServer
from mcp.server.mcpserver import Context

mcp = MCPServer("chat")


@mcp.tool()
async def chat(q: str, ctx: Context) -> str:
    answer = ""
    for i, delta in enumerate(["Hel", "lo ", "wor", "ld"], start=1):
        answer += delta
        await ctx.report_progress(i, None, delta)  # progress must strictly increase
    return answer  # the result still carries the full text


async def on_progress(progress: float, total: float | None, message: str | None) -> None:
    print("delta:", message)  # fires while the tool is still running


async def main() -> None:
    async with Client(mcp) as client:
        result = await client.call_tool("chat", {"q": "hi"}, progress_callback=on_progress)
    print(result.content[0].text)


anyio.run(main)

Closing this as part of a wider backlog cleanup around the v2 release. If it's still relevant, feel free to reopen.

_{AI Disclaimer}