Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cmd/agentop/cmd_export.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,13 @@ Usage:
agentop export --session ID one session, repeatable or comma-separated
agentop export --endpoint URL ask a specific proxy rather than the local one

Each line is one archived event reduced to ~30 content-free scalars — timings, token
Each line is one archived event reduced to 36 content-free scalars — timings, token
counts, hosts, models, outcomes — never message content, completions or tool arguments,
which is what makes it shareable where the full events are not. It is not anonymous,
though: hosts, models, session ids and timestamps remain, and together they describe who
talked to what and when. A month of traffic is about 12MB. The reading rules that keep a
consumer from misinterpreting the fields (zero means "not recorded", never a real zero;
order by "at" then "seq") are in Cortex's docs/session-export.md.
order by "at" then "seq") are in Cortex's docs/session-export-reference.md.

The export reads the durable archive, not memory: a session with nothing flushed yet
does not appear, and the live tail is /v1/events, not this.
Expand Down
91 changes: 86 additions & 5 deletions core/sessionexport/doc_example_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,33 @@ import (
"encoding/json"
"os"
"path/filepath"
"reflect"
"regexp"
"strconv"
"strings"
"testing"
)

// TestDocExampleRecordMatchesWire pins that the record example in docs/session-export.md
// TestDocExampleRecordMatchesWire pins that the record example in docs/session-export-reference.md
// strict-decodes into Record. The doc once showed promptTokens and completionTokens —
// fields Record deliberately has not, the very double count its own reading rules warn
// about — so every jq written against the example came back null (#1830 review). The
// strict decode is the pin: a field the projection does not emit fails here before it
// ships again, the way sessionapi's TestDocExamplePromptContextMatchesWire holds its doc.
func TestDocExampleRecordMatchesWire(t *testing.T) {
raw, err := os.ReadFile(filepath.Join("..", "..", "docs", "session-export.md"))
raw, err := os.ReadFile(filepath.Join("..", "..", "docs", "session-export-reference.md"))
if err != nil {
t.Fatalf("read docs/session-export.md: %v", err)
t.Fatalf("read docs/session-export-reference.md: %v", err)
}
doc := string(raw)
start := strings.Index(doc, "```json\n")
if start < 0 {
t.Fatal("docs/session-export.md carries no json example")
t.Fatal("docs/session-export-reference.md carries no json example")
}
scope := doc[start+len("```json\n"):]
end := strings.Index(scope, "```")
if end < 0 {
t.Fatal("docs/session-export.md's json example is never closed")
t.Fatal("docs/session-export-reference.md's json example is never closed")
}
example := scope[:end]

Expand All @@ -50,3 +53,81 @@ func TestDocExampleRecordMatchesWire(t *testing.T) {
t.Errorf("example carries no real token figures: %+v", r)
}
}

// readGuide returns docs/session-export-guide.md, which states two counts about this
// package as prose.
func readGuide(t *testing.T) string {
t.Helper()
raw, err := os.ReadFile(filepath.Join("..", "..", "docs", "session-export-guide.md"))
if err != nil {
t.Fatalf("read docs/session-export-guide.md: %v", err)
}
return string(raw)
}

// docCount pulls the one number a pattern captures out of a document.
func docCount(t *testing.T, doc, pattern string) int {
t.Helper()
m := regexp.MustCompile(pattern).FindStringSubmatch(doc)
if m == nil {
t.Fatalf("no count matches %q any more — if the sentence was reworded, "+
"reword this pattern with it", pattern)
}
n, err := strconv.Atoi(m[1])
if err != nil {
t.Fatalf("count %q is not a number: %v", m[1], err)
}
return n
}

// TestDocGuideFieldCountMatchesRecord pins the guide's field count to Record.
//
// The guide's "What an export contains" section tables every field, so a reader can decide
// whether an export is shareable before running one. That table is a hand-maintained copy
// of this struct, and the prose above it states the count. A field added to Record made the
// sentence quietly wrong, in the direction that matters: a reader would trust a list that
// no longer described the export (#1931 review).
//
// Neither existing pin covers this. TestDocExampleRecordMatchesWire pins the REFERENCE
// page's json example, and TestRecord_EveryFieldIsPopulated pins that FromEvent sets every
// field. Neither counts them.
// Three places state the field count: the guide's table, the reference's one-line

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit — Two small things in this pin. (1) The doc comment's last paragraph is glued to "Neither counts them." with no // blank line, so godoc renders it as one run-on paragraph. (2) The test pins the count, but not the names. A rename (ttfbMs → ttfb) or a swap (one field added, one removed) keeps 36 and passes, and the table is then wrong. You already read the guide here. Pulling the backticked names out of the What a record carries table and comparing them, as a set, with the json tags from reflect would pin the table itself, not just its row count. It is roughly the same amount of code you just wrote. Optional, but it would make the "verified against the struct" claim permanent.

// summary, and `agentop export --help`. Before #1931 the guide said 36 while the other
// two said "about 30", which is how a hand-maintained count drifts.
func TestDocGuideFieldCountMatchesRecord(t *testing.T) {
want := reflect.TypeOf(Record{}).NumField()
for _, c := range []struct {
where, path, pattern string
}{
{"docs/session-export-guide.md", filepath.Join("..", "..", "docs", "session-export-guide.md"),
`A record holds (\d+) fields`},
{"docs/session-export-reference.md", filepath.Join("..", "..", "docs", "session-export-reference.md"),
`an allowlist of (\d+)\s+scalars`},
{"cmd/agentop/cmd_export.go", filepath.Join("..", "..", "cmd", "agentop", "cmd_export.go"),
`reduced to (\d+) content-free scalars`},
} {
raw, err := os.ReadFile(c.path)
if err != nil {
t.Fatalf("read %s: %v", c.where, err)
}
if got := docCount(t, string(raw), c.pattern); got != want {
t.Errorf("%s says %d fields, Record has %d — add the new field to the guide's "+
"\"What a record carries\" table, or to \"What a record never carries\" if it "+
"must not be exported, and fix all three counts", c.where, got, want)
}
}
}

// TestDocGuideSecretCountMatchesAllSecrets pins the guide's planted-secret count.
//
// The guide cites the count as the evidence for the privacy contract. Understating it is
// the harmful direction: a 25th secret planted here without a doc edit leaves the guide
// claiming a weaker guarantee than the one the test holds, and the number is the reason a
// reader believes the claim at all (#1931 review).
func TestDocGuideSecretCountMatchesAllSecrets(t *testing.T) {
want := len(allSecrets())
if got := docCount(t, readGuide(t), `puts (\d+) distinct secrets`); got != want {
t.Errorf("docs/session-export-guide.md says the test plants %d secrets, allSecrets() has %d — "+
"update the count in \"What a record never carries\"", got, want)
}
}
6 changes: 3 additions & 3 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ holds the repo-level pieces; this file is the index.
| Run an agent on a local model, or switch inference servers | [`laptop-local-model.md`](laptop-local-model.md) |
| Use the `agentop` TUI | [`cmd/agentop/README.md`](../cmd/agentop/README.md) |
| Add your agent's sessions from before Cortex | [`agentop import`](../cmd/agentop/README.md#importing-agent-history-agentop-import) |
| Export sessions to files | [`session-dump.md`](session-dump.md) |
| Export the archive for offline analysis (content-free records) | [`session-export.md`](session-export.md) |
| Walk through exporting and analysing it, with the traps | [`session-export-guide.md`](session-export-guide.md) |
| **Without content:** measure traffic — tokens, latency, errors. Start here | [`session-export-guide.md`](session-export-guide.md) |
| **Without content:** look up what a record field means | [`session-export-reference.md`](session-export-reference.md) |
| **With content:** keep prompts, completions and tool results as files | [`session-dump.md`](session-dump.md) |

## Kubernetes and architecture

Expand Down
2 changes: 1 addition & 1 deletion docs/session-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ through DNS rebinding. The stats server applies the same check.
| `DELETE /v1/sessions` | `application/json` | Clear every session: the store, and the session archive where one runs. The cost ledger is kept |
| `DELETE /v1/sessions/{id}` | `application/json` | Erase one session: the store's entry and, where the archive runs, its files (see below) |
| `GET /v1/events` | `text/event-stream` | SSE stream of new events; `?session=<id>` filters to one session. Heartbeat every 30s |
| `GET /v1/export` | `application/x-ndjson` | The session archive as content-free records, one per event — the export for offline analysis; `?session=<id>` and `?since=<duration>` select. A walk that dies past its first record resets the connection rather than ending the body cleanly, so a truncated export reaches a client as an error ([`session-export.md`](session-export.md)) |
| `GET /v1/export` | `application/x-ndjson` | The session archive as content-free records, one per event — the export for offline analysis; `?session=<id>` and `?since=<duration>` select. A walk that dies past its first record resets the connection rather than ending the body cleanly, so a truncated export reaches a client as an error ([`session-export-reference.md`](session-export-reference.md)) |
| `GET /v1/pipeline` | `application/json` | Active pipeline composition, per plugin: name, direction, position, capabilities, runtime `config`. `onError` is sent only when it is not the default — in practice `observe`, since an `on_error: off` plugin is not built. A plugin under observe runs and records, but its rejections, body writes and redirects are dropped (header writes still apply), so its `config` says what it would do. agentop renders this as the Pipeline pane (`P`) |
| `GET /v1/plugins` | `application/json` | Catalog of every registered plugin, active or not. 404 when the binary built its session API without `WithCatalog`. agentop renders this as the Catalog pane (`C`) |
| `GET /v1/usage` | `application/json` | Windowed usage aggregation — the aggregate behind `agentop cost`; parameters and semantics in [`pricing.md`](pricing.md#reading-usage-over-http) |
Expand Down
2 changes: 1 addition & 1 deletion docs/session-dump.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ holds at the moment it runs. You must run it before the thing you want is gone.
laptop the session archive closes that gap. In a cluster the gap is still open.

A dump is the *content* — prompts, completions, tool results. When what you want is the
*measurements* over a month of traffic, [`GET /v1/export`](session-export.md) writes the
*measurements* over a month of traffic, [`GET /v1/export`](session-export-reference.md) writes the
archive as content-free records instead: no prompts, completions or tool results, which
is what makes it shareable where a dump is not. It is not anonymous, though — hosts,
models, session ids and timestamps remain, and together they describe who talked to what
Expand Down
Loading
Loading