Skip to content

Investigation record: steps[].arguments and the finding's rootCause and why are still unbounded #63

Closed

Description

@shipmill-romamo

What happens

Found while implementing #57 (#62). D-4 and D-6 cap the reasoning, text, answer and raw tool-call arguments in the investigation record, but three fields still have no length limit in config/schemas/v1/agent-investigation.schema.json or in sdk/src/cloudfall/investigation.py:

  1. spec.steps[].arguments: the parsed arguments of the operation that ran, which can hold the same long text as the raw toolCalls[].arguments that D-6 caps
  2. finding.rootCause
  3. finding.why

All three come from the model's output, which can quote host output (the reason given for D-4 and D-6).

Question

Should these get a cap too, and if so which part should be kept? steps[].arguments is structured, so a cut may not fit there; refusing may be the better rule.

Activity

  1. shipmill-romamo commented on Oct 9, 2026

    @shipmill-romamo
    ContributorAuthor

    Triage: implement, once the rule is settled. It waits on your decision in the next comment. The gap is confirmed on #62's branch: steps[].arguments, finding.rootCause and finding.why have no maxLength. This change bounds stored state and a published schema, so it goes through the design gate. D-4 and D-6 don't cover these fields.

  2. shipmill-romamo commented on Oct 9, 2026

    @shipmill-romamo
    ContributorAuthor

    @romamo Decision needed: how should the investigation record bound steps[].arguments, finding.rootCause and finding.why?

    1. Cap rootCause and why at 8000 characters, keeping the start and marking the cut as D-6 does. Refuse to record a step whose serialized arguments exceed 8000 characters, since structured arguments can't be cut and stay meaningful (recommended)
    2. Cap all three at 8000 characters, keeping the start. For steps[].arguments, that means storing a cut string in place of the object
    3. Leave them unbounded. rootCause and why are short by the prompt's format, and steps[].arguments is what really ran, so it should be kept whole

    Reply here with a number or your own answer. shipmill takes this up on the tick after your reply.

  3. romamo commented on Oct 9, 2026

    @romamo
    Collaborator

    1 — cap rootCause and why at 8000 characters, keeping the start and marking the cut as D-6 does; refuse to record a step whose serialized arguments exceed 8000 characters

    (decision by @romamo, given in chat; posted by Claude on his behalf)

  4. shipmill-romamo commented on Oct 9, 2026

    @shipmill-romamo
    ContributorAuthor

    Triage: implement, taking up @romamo's answer (option 1).

    Design:

    • finding.rootCause and finding.why: keep at most 8000 characters, keeping the start and marking the cut as D-6 does (_head_text); the schema gets maxLength on both
    • spec.steps[].arguments: a call whose parsed arguments serialize to more than 8000 characters is refused before it runs, with outcome invalid and an error telling the model the limit. The step is recorded with empty arguments {}, as a call with non-JSON arguments already is, so steps[] only ever holds arguments that ran whole. The trace's raw toolCalls[].arguments keeps its D-6 cut copy, so what the model sent stays visible
    • JSON Schema can't bound an object's serialized size, so the step rule is enforced in code and tested there

    Recorded as D-8 in a docs PR before the implementer starts.

  5. removed
    needs-decisionA shipmill session asked a question here; waits for a reply
    on Oct 9, 2026
  6. shipmill-romamo commented on Oct 9, 2026

    @shipmill-romamo
    ContributorAuthor

    Triage: PRs for this issue:

    Trade-off: a saved record from before this change with a longer finding line no longer validates against the schema, as with D-6.

  7. added a commit that references this issue on Oct 9, 2026
  8. shipmill-romamo commented on Oct 9, 2026

    @shipmill-romamo
    ContributorAuthor

    Released in v0.6.0rc11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions