Repository navigation
fix(adk): fail the task when the model stops at max_tokens instead of completing with a cut answer - #2896
fix(adk): fail the task when the model stops at max_tokens instead of completing with a cut answer#2896fiunchinho wants to merge 6 commits into
Conversation
… completing with a cut answer A model that stops at its output limit returns an answer cut mid-sentence. The A2A task ended as completed, so a client - a person's front end, or a parent agent calling this one through the remote agent tool - read the partial text as the whole answer with no notice. The turn's last model response is now watched for genai.FinishReasonMaxTokens, which every model adapter maps its own stop reason onto, and the terminal task status becomes failed with a message naming the limit. The streamed artifacts are left as they are, so the partial answer stays in the caller's hands. Signed-off-by: Jose Armesto <github@armesto.net>
Signed-off-by: Jose Armesto <github@armesto.net>
| } | ||
| return nil | ||
| }, | ||
| AfterExecuteCallback: func(ctx adka2a.ExecutorContext, finalEvent *a2atype.TaskStatusUpdateEvent, _ error) error { |
There was a problem hiding this comment.
#2890 also rewrites this callback, conflicts on merge
There was a problem hiding this comment.
Thanks. #2890 has merged, so I merged main into this branch (c14d593): the observe call sits at the top of the AfterEventCallback that now ends in transformStructuredOutput, the new AfterExecuteCallback is unchanged, and the test follows the new NewKAgentExecutor error return. The two paths do not overlap: a structured-output turn that stops at max_tokens already fails inside transformStructuredOutput, and failTruncatedAnswer leaves a non-completed terminal status alone, so there is no double report. build, vet, the package tests (incl. TestKAgentExecutorFailsCompletedStructuredOutputWithoutResult) and make lint pass on the merged branch.
…-fail-task-on-max-tokens NewKAgentExecutor now returns an error too; the output-limit test checks it. Signed-off-by: Jose Armesto <github@armesto.net>
|
This pull request has been marked as stale because of no activity in the last 15 days. It will be closed in the next 5 days unless it is tagged "no stalebot" or other activity occurs. |
Signed-off-by: Jose Armesto <github@armesto.net>
| return transformStructuredOutput(output, rootName, event, processed) | ||
| }, | ||
| AfterExecuteCallback: func(ctx adka2a.ExecutorContext, finalEvent *a2atype.TaskStatusUpdateEvent, _ error) error { | ||
| outputLimitWatchFrom(ctx).failTruncatedAnswer(finalEvent) |
There was a problem hiding this comment.
remote tool reads artifacts first, does parent see this?
There was a problem hiding this comment.
you are right, it did not. I pushed a new commit, a failed task now yields the status message as error and the artifacts as result, so the parent reads both.
…-max-tokens The terminal status callback now both fails a truncated answer and stamps the task usage (kagent-dev#2461). Signed-off-by: Jose Armesto <github@armesto.net>
…s output The remote agent tool read a failed task's artifacts before its status message, which is right for a completed task and wrong for a failed one that kept its output: a parent agent got the cut answer as the error and never the reason the task failed. A failed task now yields the status message as the error and the artifacts as the result, in the Go tool and its Python twin. A failure without a status message still reads its text from the artifacts, and one with neither still names the agent. The Python tool returned a bare string for a failure, which ADK hands to the model as a plain result; it now returns the error/result/subagent_session_id dict the Go tool and its own HITL failure branch already use. The import block of output_limit_test.go follows gci, which main now lints with. Signed-off-by: Jose Armesto <github@armesto.net>
Fixes #2895
What
When the turn's last model response ends with
genai.FinishReasonMaxTokens, the A2A task now ends inTASK_STATE_FAILEDwith a status message naming what happened:The number comes from the response's usage: a model that stopped at the limit was billed for exactly that many output tokens. When usage does not carry it, the sentence is sent without a number rather than with a guess. The artifacts already streamed are untouched, so the caller keeps the partial answer and reads the failure beside it.
The remote agent tool (Go and Python) now reports a failed task to the parent agent the same way: the status message is the
error, and the artifacts the task kept are theresult. Before, the tool read the artifacts first for a failed task too, so a parent would have seen the cut answer as the error text and never the reason.Why
A model that stops at its output limit returns an answer cut mid-sentence, and the A2A task reported it as completed. The finish reason was already on the
model.LLMResponse—anthropicStopReasonToGenaimapsstop_reason: max_tokensontogenai.FinishReasonMaxTokens, and the OpenAI chat, OpenAI Responses and Ollama adapters map their own "length" reasons onto the same value — but nothing downstream acted on it, so it was dropped at the executor boundary.Nothing after that point could tell a cut answer from a whole one: not a chat front end showing the reply, and not a parent agent calling this one through the remote A2A agent tool, which will happily summarise or act on half an answer. The issue has the details of a turn where this happened (an Anthropic
ModelConfigwithoutmaxTokens, so the adapter default of 8192, and a thinking model that spent most of the budget before it began writing).The check keys on the provider-neutral
genai.FinishReasonMaxTokens, not on any provider's own type, so every adapter is covered.Design
completedis a statement about the work; for a cut answer it is a false one. A flag in metadata is something every client would have to learn to read, and the clients that most need the signal — agents calling agents — are the least likely to read it.AfterExecuteCallbackrewrites the terminal status.ExecutorConfig.AfterEventCallbacksees the ADK*session.Eventcarrying the finish reason and records how the turn's last model response ended;ExecutorConfig.AfterExecuteCallbacksees the terminalTaskStatusUpdateEventbefore it is yielded and turnscompletedintofailed. The status message part carriesadka2a.ToA2AMetaKey("is_error_message"), the same metadata ADK sets on the failures it reports itself, so clients render it the way they already render an agent failure. Both callbacks are configured once per executor and shared by concurrent executions, so the per-turn state travels in the request context.completedis left alone, so a failure ADK already reports (a function call broken by the same truncation, say) or an input-required pause is neither overwritten nor reported twice.AfterEventCallbackwas tried first and rejected. ADK v2.3.0 does turn it into aTaskStateFailedstatus and keeps the artifacts (Executor.process→makeTaskFailedEvent→writeFinalTaskStatus), but it prefixes the message withprocessor failed:, which reads as an internal fault rather than a model limit, and it aborts the run at that event instead of letting the turn end. Rewriting the terminal status keeps the message exact and the event order unchanged.processResultingo/adk/pkg/tools/remote_a2a_tool.goand its Python twin in_remote_a2a_tool.pypreferred the artifacts for every terminal state, which is right for a completed task and wrong for a failed one that kept its output: the parent got the cut answer as the error and never the reason. A failed task now yieldserrorfrom the status message andresultfrom the artifacts, so the parent reads both. A failed task without a status message still reads its text from the artifacts, and one with neither still names the agent, as before. The Python tool returned a bare string for a failure, which ADK hands to the model as a plain result; it now returns the sameerror/result/subagent_session_iddict as the Go tool, the shape its HITL failure branch already used.Tests and commands
New tests in
go/adk/pkg/a2a/output_limit_test.go, each driving a fake agent throughNewKAgentExecutorin the style of the existing executor tests:TestKAgentExecutorFailsTheTaskWhenTheModelStopsAtTheOutputLimit— a partial chunk and a final response withFinishReasonMaxTokensand usage: the task fails, the message namesoutput limitandmax_tokens=8192, and the streamed artifact still holds the partial answer. The fixture also setsThoughtsTokenCount, which is a breakdown of the billed output and must not be added to it (the OpenAI Responses adapter fills both fields).TestKAgentExecutorOmitsTheLimitWhenUsageDoesNotNameIt— the same without usage: the task fails and the message carries no number.TestKAgentExecutorCompletesTheTaskWhenTheLastResponseStopsNormally— a partial chunk carrying the limit followed by aFinishReasonStopresponse: the task still completes with no status message.For the remote agent tool,
TestProcessResult_FailedTaskReportsTheReasonBesideThePartialAnsweringo/adk/pkg/tools/remote_a2a_tool_test.goand three cases inpython/packages/kagent-adk/tests/unittests/test_remote_a2a_tool.pycover a failure with a status message and artifacts, with a message alone, with artifacts alone, and with neither.From
go/:From
python/:uv run pytest packages/kagent-adk/tests/unittests/test_remote_a2a_tool.py(32 passed),ruff format --diffandruff checkclean.With the rewrite disabled, the first two executor tests fail with
terminal state = "TASK_STATE_COMPLETED"— the reported behaviour.Known gap
A response truncated before it emitted any content part — a turn whose whole budget went to thinking — produces no artifact event, so
AfterEventCallbacknever fires for it and the watch does not see it. That turn arrives empty rather than cut. Covering it would need a runner plugin (OnEventCallback), which seemed more machinery than this fix warrants; happy to add it if you would rather have the complete coverage.