Skip to content

Cap the investigation's answer and raw tool-call arguments at 8000 characters - #62

Merged
shipmill-romamo[bot] merged 1 commit into
mainfrom
fix/57-cap-answer-and-arguments
Oct 9, 2026
Merged

shipmill-romamo[bot] merged 1 commit into
mainfrom
fix/57-cap-answer-and-arguments

Conversation

@shipmill-romamo

Copy link
Copy Markdown
Contributor

Summary

The investigation record kept spec.answer and spec.trace[].toolCalls[].arguments unbounded after #52 capped the trace's reasoning and text (D-4). Both are the model's text and can quote host output. This caps them as D-6 (#59) settles.

  • sdk/src/cloudfall/investigation.py: new ANSWER_LIMIT and ARGUMENTS_LIMIT (both TOOL_TEXT_LIMIT, 8000) and _head_text, the mirror of D-4's _trace_text. It keeps the start and ends with a \n[last N characters cut] mark, D-4's [first N characters cut]\n turned round. As in D-4, the mark counts against the limit, so a stored value is never longer than 8000 characters, mark included (a cut value can be one to three under). Characters are counted, not bytes
  • Only the stored copy is cut:
    • the finding (and so status: answered) is still read from the model's whole answer, so a finding past the cut still counts
    • the raw arguments are cut in _call_text, which builds only the trace's ToolCallText. _run_call still parses the endpoint's own argument string, so the tool runs on the whole arguments and nothing parses the cut copy as JSON
  • The stream's turn events carry the same Turn, and the last investigation event is the saved record's as_document(), so what agent investigate prints matches the record
  • config/schemas/v1/agent-investigation.schema.json: maxLength: 8000 on spec.answer and trace[].toolCalls[].arguments, with descriptions, the same bound D-4 put on text and reasoning

Records already on disk: handled as #52 did. Stored investigation records are not re-validated on read, still true on main: InvestigationStore validates only on save, nothing in the SDK or engine loads investigations/, and evaluations/nemotron-incident/run.py reads them as plain JSON and uses only finding.proposed. A record written before with a longer answer or arguments stays readable everywhere it is read today; only revalidating it against the new schema would fail.

No schema lock or snapshot test covers this schema; none needed updating. One existing assertion changed: test_a_long_reasoning_is_cut_to_the_limit_and_marked asserted the record's answer stayed whole (that was #52's design); it now asserts the start is kept with the mark.

Out of scope, unchanged: spec.steps[].arguments (the parsed object that ran) and finding.rootCause/why stay unbounded.

Test plan

  • test_a_long_answer_keeps_its_start_and_its_finding[8000|8001|50000]: exactly 8000 is kept whole; 8001 and 50,000 (Cyrillic) keep the start and the mark, the finding is still read, and the saved record passes the schema
  • test_a_finding_past_the_cut_is_still_read_from_the_whole_answer
  • test_the_cli_streams_the_answer_as_the_record_keeps_it: the stream's record event and the file on disk have the same cut answer
  • test_long_raw_arguments_keep_their_start_in_the_trace_only[8000|8001|50000]: the tool receives the whole arguments, steps[].arguments is whole, the trace's raw string is cut with the mark
  • test_the_schema_refuses_an_answer_or_arguments_past_the_limit[answer|arguments]: 8001 characters fail the schema
  • uv run ruff check ., uv run mypy, uv run pytest (full suite, 476 passed), uv run cloudfall config validate --project config/examples: all pass

Fixes #57

🤖 Generated with Claude Code

…aracters

spec.answer and trace[].toolCalls[].arguments are the model's text and can
quote host output, yet the record kept both unbounded after #52 capped the
trace's reasoning and text (D-4).

Both now keep at most 8000 characters, cut mark included, keeping the start
and ending with a [last N characters cut] mark (D-6). Only the stored copy
is cut: the finding is read from the whole answer and the tool runs on the
whole arguments. The schema bounds both fields with maxLength 8000.

Fixes #57

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo
shipmill-romamo Bot force-pushed the fix/57-cap-answer-and-arguments branch from 1654b27 to 4524679 Compare October 9, 2026 16:25
@shipmill-romamo
shipmill-romamo Bot merged commit 33d2c4c into main Oct 9, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the fix/57-cap-answer-and-arguments branch October 9, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigation record: answer and raw tool-call arguments are still unbounded

0 participants