Skip to content

Cap the reasoning and text the investigation trace keeps - #52

Merged
shipmill-romamo[bot] merged 1 commit into
mainfrom
fix/50-cap-trace-reasoning
Oct 9, 2026
Merged

shipmill-romamo[bot] merged 1 commit into
mainfrom
fix/50-cap-trace-reasoning

Conversation

@shipmill-romamo

Copy link
Copy Markdown
Contributor

Summary

Since #45 the investigation record's spec.trace[].reasoning kept the model's reasoning_content verbatim, with no bound, and a model's reasoning often quotes the host output it just read. This caps it as D-4 settled.

  • _turn in sdk/src/cloudfall/investigation.py keeps at most TRACE_TEXT_LIMIT (= TOOL_TEXT_LIMIT, 8000) characters of each turn's reasoning and of its text. A longer value keeps its end after the same [first N characters cut] mark the tool-text cap writes (host_text now shares the mark helper). The mark counts against the limit, so the stored value never exceeds 8000 characters, mark included. Characters are counted, not bytes
  • The stream's turn event carries the same Turn, so it is capped the same way: the stream and the record stay equal
  • config/schemas/v1/agent-investigation.schema.json: maxLength: 8000 on trace[].text and trace[].reasoning, with descriptions, and point 2 of the issue: toolCalls[].arguments says the step's parsed arguments are what ran and the raw string is only the model's text
  • The record's answer (what the finding is read from) is not capped; only the trace's copy of the turn text is

Stored records are not re-validated on read: InvestigationStore only validates on save, cloudfall why does not read investigations, and evaluations/nemotron-incident/run.py reads them as plain JSON. A record written before with a longer reasoning stays readable everywhere it is read today.

Test plan

  • New test_a_long_reasoning_is_cut_to_the_limit_and_marked: a 10,532-character Cyrillic reasoning and a 9,017-character turn text are cut to at most 8000 characters with the mark, keep their ends, and the saved record passes the schema; the record's answer stays whole
  • New test_the_schema_refuses_a_trace_longer_than_the_limit: an 8001-character reasoning fails the schema
  • uv run ruff check ., uv run mypy, uv run pytest (438 passed), uv run cloudfall config validate --project config/examples, cloudfall-engine inventory render all pass locally
  • The Ansible syntax checks and ansible-lint could not run in the local session (Ansible refuses non-blocking stdio there); this PR touches no Ansible file, and CI runs them

Fixes #50

🤖 Generated with Claude Code

A model's reasoning often quotes the host output it just read, and since
PR #45 the trace kept it verbatim with no bound. Each turn's reasoning and
text now keep at most 8000 characters (TOOL_TEXT_LIMIT), their end after
a [first N characters cut] mark that counts against the limit, and the
schema bounds both with maxLength. The stream's turn event carries the
same Turn, so it is capped alike. The schema also says the step's parsed
arguments are what ran, and toolCalls[].arguments only the model's text.

Fixes #50

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo
shipmill-romamo Bot merged commit 10f2254 into main Oct 9, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the fix/50-cap-trace-reasoning branch October 9, 2026 14:43
shipmill-romamo Bot added a commit that referenced this pull request Oct 9, 2026
…aracters

spec.answer and trace[].toolCalls[].arguments are the model's text and can
quote host output, yet the record kept both unbounded after #52 capped the
trace's reasoning and text (D-4).

Both now keep at most 8000 characters, cut mark included, keeping the start
and ending with a [last N characters cut] mark (D-6). Only the stored copy
is cut: the finding is read from the whole answer and the tool runs on the
whole arguments. The schema bounds both fields with maxLength 8000.

Fixes #57

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigation trace: reasoning can store host output unbounded

0 participants