Skip to content

agent investigate: a model failure partway through a run leaves no investigation record #29

Closed

Description

@shipmill-romamo

What happens

cloudfall agent investigate (PR #28) writes its AgentInvestigation record only when the loop ends normally or runs out of turns (exit 93). When the model endpoint fails partway through a run (exit 92 MODEL_UNAVAILABLE), no record is written, although the decisions the earlier turns proposed are already recorded.

Reproduce

  1. Start a local OpenAI-compatible server that answers the first chat request with a tool call proposing a mutating operation, and the second with HTTP 500
  2. cloudfall agent investigate --alert "disk full" --base-url http://127.0.0.1:<port>/v1 --model test
  3. Exit 92; decisions list shows the proposed decision, investigations/ has no record of the run that proposed it

Expected

The run is recorded on a 92 too, with the calls made so far and an outcome saying the model became unavailable, so every decision an agent proposed traces back to its investigation.

Found while reviewing #28.

Activity

  1. shipmill-romamo commented on Oct 8, 2026

    @shipmill-romamo
    ContributorAuthor

    Triage: implement.

    Confirmed on main (cb367d9, just after #28 merged): agent_investigate in sdk/src/cloudfall/app.py saves the record only after investigate() returns, so an InvestigationError or a model-unavailable exit raised mid-loop leaves investigations/ empty while the decisions proposed in earlier turns stay recorded.

    Design (a stored-state change, additive to a schema that has only shipped in 0.6.0rc5):

  2. shipmill-romamo commented on Oct 8, 2026

    @shipmill-romamo
    ContributorAuthor

    PR #32 implements the design above: a model failure after the run started saves the investigation as model-unavailable (steps, turns and tokens so far, empty answer, no finding), then exits 92 with the record's id in error.context.investigation.

    One choice beyond the plan: when the very first request fails (endpoint unreachable), a record with turns: 1 and no steps is written, since the request was sent. Refusals before any request (exit 2) still write nothing. The record stores no error text, because an endpoint's error body can echo the request's key.

  3. shipmill-romamo commented on Oct 8, 2026

    @shipmill-romamo
    ContributorAuthor

    Released in v0.6.0rc6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions