Repository navigation
Investigation trace: reasoning can store host output unbounded #50
Description
Found while reviewing #45 (merged as c2dda49). Neither blocked the merge.
- The trace's reasoning can store host output, with no size limit. Before Keep each model turn's raw trace in the stream and the record #45 the investigation record kept the agent's steps but never tool results.
spec.trace[].reasoningnow keeps the model'sreasoning_contentverbatim, and a model's reasoning often quotes the host output it just read (log lines,dfoutput, config fragments). So host output reaches the record indirectly, and nothing bounds its length (tool text to the model is capped atTOOL_TEXT_LIMIT, tool names atTOOL_NAME_LIMIT, the ids now atREPLY_ID_LIMIT). Decide whether the record caps reasoning (and the turn text) at a limit like the others, and whether the "names and hashes only" rule from the secrets guide should apply to it - An invalid tool-call argument string is kept raw in the trace while the step stores
{}. When the model writes arguments that aren't valid JSON, the step records{}buttrace.toolCalls[].argumentskeeps the raw string. That is arguably the point of a raw trace, but the two parts of one record then disagree; worth one line in the schema description saying which is authoritative
Code: sdk/src/cloudfall/investigation.py (_turn, _call_text), config/schemas/v1/agent-investigation.schema.json.
Activity
shipmill-romamo commentedon Oct 9, 2026
Triage: waits on your decision below. Filed from the review of #45; both points are real on
main(c2dda49), and the first one changes what the investigation record stores (config/schemas/v1/agent-investigation.schema.json), so it goes through the design gate before anyone builds it.Plan once decided: cap
trace[].reasoningand the turn text in_turnlike the other limits, withmaxLengthin the schema so a longer value is refused rather than stored; add one line to thetoolCalls[].argumentsschema description saying the step's parsed{}is what ran and the raw string is only the model's text.shipmill-romamo commentedon Oct 9, 2026
@romamo Decision needed: should the investigation record cap the model's reasoning it keeps in
spec.trace, now that the reasoning can quote host output?TOOL_TEXT_LIMIT(recommended): the record stays bounded and keeps the part of the reasoning the hosted demo shows; a very long reasoning is cut, with the cut markedReply here with a number or your own answer; shipmill takes this up on the tick after your reply.
romamo commentedon Oct 9, 2026
1 — cap the reasoning kept in
spec.traceat 8000 characters per turn, likeTOOL_TEXT_LIMIT, with the cut marked(decision by @romamo, given in chat; posted by Claude on his behalf)
shipmill-romamo commentedon Oct 9, 2026
Triage: implement, taking up the answer above (option 1), recorded as D-4 in #51.
Plan:
_turncaps each turn's reasoning kept inspec.traceat 8000 characters (theTOOL_TEXT_LIMITvalue), with the cut marked, and the schema gets a matchingmaxLengthso a longer value is refused rather than stored. The turn text gets the same cap. Point 2: one line in thetoolCalls[].argumentsschema description saying the step's parsed arguments are what ran and the raw string is only the model's text.#44 also changes
investigation.pyand the same schema; the same implementer builds both on separate branches.shipmill-romamo commentedon Oct 9, 2026
PR: #52. Each turn's reasoning and text in the trace keep at most 8000 characters (
TRACE_TEXT_LIMIT), cut from the front with the same[first N characters cut]mark as the tool-output cap, so the model's conclusion stays; the schema hasmaxLength: 8000on both. The stream'sturnevent is capped the same way, so stream and record agree. ThetoolCalls[].argumentsdescription now says the parsed arguments are what ran. Older records still load: nothing re-validates stored investigation records.romamo commentedon Oct 9, 2026
Decision by @romamo, given in chat; posted by Claude on his behalf.
1. Cap the reasoning kept in
spec.traceat 8000 characters per turn, likeTOOL_TEXT_LIMIT, with the cut marked.shipmill-romamo commentedon Oct 9, 2026
Released in v0.6.0rc9.