Skip to content

Keep each model turn's raw trace in the stream and the record - #45

Merged
shipmill-romamo[bot] merged 2 commits into
mainfrom
feat/agent-trace
Oct 9, 2026
Merged

shipmill-romamo[bot] merged 2 commits into
mainfrom
feat/agent-trace

Conversation

@romamo

@romamo romamo commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #39 (base feat/agent-stream); retarget to main once #39 merges.

What

  • A Turn carries the raw trace of one model turn: the reasoning when the model returns one (Nemotron's reasoning_content), each tool call exactly as written (name and raw argument string), the endpoint's response id (chatcmpl-…) and x-request-id, the milliseconds the turn took, and its prompt, completion and reasoning tokens
  • ChatTransport returns a ModelReply (body and request id), so the x-request-id header survives
  • The stream's turn event has it under trace; the investigation record keeps every turn in spec.trace (optional in the schema, so older records stay valid)

Why

The hosted demo read as a simulation; it now shows what the model actually returned. And the record can answer what the model thought when it proposed a change, not only what it did.

Proven

Live, Super against the replayed incident: four turns, each with its reasoning ("We need diagnose why postgres down. Likely disk full? Let's run disk report on hz1."), raw calls, chatcmpl-… and Nebius request ids, 0.9 to 1.4 s per turn, reasoning tokens 7 to 95.

Checks

  • uv run pytest: 406 passed (new: the trace lands in the turn and the record)
  • uv run ruff check ., uv run mypy: clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv

A turn now carries the model's reasoning (Nemotron's reasoning_content),
the tool calls as written, the response id and x-request-id, its time and
its prompt, completion and reasoning tokens. The transport returns a
ModelReply so the request id survives. The record's spec.trace is
optional, so records written before stay valid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
romamo added a commit that referenced this pull request Oct 9, 2026
The page now shows what actually came back: the command the server ran,
each turn's reasoning, tool calls as the model wrote them, response and
Nebius request ids, time and tokens, the full Ansible output the model
saw, which recorded run answered each call (file, time, sha256), the raw
approval logs, and a download of the CLI's exact stream. CLI lines pass
through untouched; the demo's own lines carry _demo. Reads treaty rc38's
stream; pinned to the trace commit (#45).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
Base automatically changed from feat/agent-stream to main October 9, 2026 08:13
The schema caps responseId and requestId at 256 characters, but the turn
copied both from the endpoint as sent. An over-long id made the record
fail schema validation on save with a raw jsonschema error, after the
run's decisions were already written, so the investigation was lost.
Keep at most 256 characters, as the record already does for tool names.

The CLI stream test now also checks the turn event's trace over the real
transport, with no x-request-id header and no API key in the stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo
shipmill-romamo Bot merged commit c2dda49 into main Oct 9, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the feat/agent-trace branch October 9, 2026 09:05
romamo added a commit that referenced this pull request Oct 9, 2026
The page now shows what actually came back: the command the server ran,
each turn's reasoning, tool calls as the model wrote them, response and
Nebius request ids, time and tokens, the full Ansible output the model
saw, which recorded run answered each call (file, time, sha256), the raw
approval logs, and a download of the CLI's exact stream. CLI lines pass
through untouched; the demo's own lines carry _demo. Reads treaty rc38's
stream; pinned to the trace commit (#45).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant