Skip to content

Hosted demo: a live Nemotron on the recorded incident - #40

Merged
shipmill-romamo[bot] merged 6 commits into
mainfrom
demo/hosted
Oct 9, 2026
Merged

shipmill-romamo[bot] merged 6 commits into
mainfrom
demo/hosted

Conversation

@romamo

@romamo romamo commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Pinned to main (9e51dc5) now that #39 (stream) and #45 (raw trace) are merged; rebased on main. Deployed to h5 as https://demo.cloudfall.dev from an earlier commit of this branch (romamo/myfleet#7); redeploy from main once this lands.

What

  • demo/: Starlette app for demo.cloudfall.dev. A visitor picks one of four Nemotron models; /api/runs runs cloudfall agent investigate --replay on the disk-full incident and forwards the CLI's stream line by line, untouched, adding only _demo lines (the exact command, and which recorded run answered each call: file, time, sha256); /api/approve plays back the recorded approval of the fix with its raw execution and verify logs, refuses the destructive wipe, and says when no approval was recorded
  • The page shows raw traces: each turn's reasoning_content, the tool calls as the model wrote them, response and Nebius x-request-id, time and tokens; the full Ansible output the model saw; the record; a download of the exact NDJSON stream
  • Limits from the environment: runs per visitor per hour, concurrent runs, tokens per day; run directories are removed after an hour
  • demo/incident/ copies examples/disk-full-incident (the demo deploys from its own directory); a test fails when it drifts
  • examples/disk-full-incident/recordings/disk-full-approved: a real approved and verified fix from a fresh cx23, no addresses
  • The example's shop-logrotate is idempotent now (the first recorded approval came back NOT_VERIFIED, correctly)
  • CI: a Demo step syncs, type-checks and tests demo/

Checks

  • demo: 7 passed, mypy strict clean; ruff clean
  • In headless Chrome over DevTools, a live Super run rendered four turns with reasoning, ids and provenance, two proposals, and the finding, with no console errors

🤖 Generated with Claude Code

https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv

romamo and others added 4 commits October 9, 2026 22:19
demo/ is a small Starlette app. A visitor picks one of four Nemotron
models; the server runs agent investigate --replay against the disk-full
incident and streams its events to the page. Approving the fix plays back
the approval recorded on a real host; approving the wipe is refused.
Limits per visitor, per run and per day.

The example's logrotate fix is idempotent now, so its verify run changes
nothing and the recorded approval verifies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
The page now shows what actually came back: the command the server ran,
each turn's reasoning, tool calls as the model wrote them, response and
Nebius request ids, time and tokens, the full Ansible output the model
saw, which recorded run answered each call (file, time, sha256), the raw
approval logs, and a download of the CLI's exact stream. CLI lines pass
through untouched; the demo's own lines carry _demo. Reads treaty rc38's
stream; pinned to the trace commit (#45).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APTdZZw8Jw9PQF4TD55wMv
@romamo
romamo marked this pull request as ready for review October 9, 2026 19:20
shipmill-romamo Bot and others added 2 commits October 9, 2026 22:28
filecmp.dircmp is shallow by default: two files of the same size and mtime
count as equal without being read, so a same-length edit to the copy could
pass the drift test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The budget counted only the stream's last event. A visitor who closes the
page mid-run kills the CLI before that event, so the turns the model had
already answered never counted and abandoned runs were free.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shipmill-romamo

Copy link
Copy Markdown
Contributor

Review: merge after two small fixes, pushed as 930d891 on top of 9d3d9d0. The root checks and the Demo step pass locally (root pytest 517, demo pytest 9).

Fixed, each with a test that fails on the old code:

  • d654af1 demo/tests/test_demo.py: the drift test used filecmp.dircmp, which is shallow by default and trusts size and mtime. A same-length edit to demo/incident/ passed. It now compares contents (shallow=False)
  • 930d891 demo/src/cloudfall_demo/server.py: the daily token budget counted only the stream's last investigation event. When a visitor closes the page, the CLI is killed before that event, so an abandoned run cost nothing. Each turn event now spends its tokens as it arrives. Behaviour change: abandoned runs now count against the daily cap

Checked and clean: the model comes from a fixed list; the API key comes from the environment and never reaches argv, a _demo line, the stream or logs; /api/approve stays inside its recordings and runs nothing; the recordings hold no addresses, keys or real hostnames; the shop-logrotate change is idempotent and the copy matches.

Left as follow-ups, none blocking a demo with a daily token cap:

  • server.py:350: the visitor is keyed on CF-Connecting-IP / X-Real-IP as sent. A client that reaches h5 without going through Cloudflare can rotate the header and get past 6 runs an hour, and the per-visitor table never shrinks. Fixing it means trusting the header only from Cloudflare's ranges or the local nginx, which is a deployment choice
  • server.py:127,147: a body that isn't JSON, or isn't an object, returns 500 instead of 400
  • A visitor leaving force-kills the CLI, so the run isn't recorded as stopped (D-3); queued runs beyond the 4 slots wait in an unbounded queue; limits are per process; old run folders are removed only when a new run starts

@shipmill-romamo
shipmill-romamo Bot merged commit 06c8767 into main Oct 9, 2026
1 check passed
@shipmill-romamo
shipmill-romamo Bot deleted the demo/hosted branch October 9, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant