From 128f69b22f1782f01da7bcb63e4310af2907c497 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 15:19:24 +0000 Subject: [PATCH 1/3] Extraction report reads extraction-quality*.json (#151); token-burn cost, windows, agent time, prompts (#296) extraction-report.sh: find every extraction-quality*.json at analysis//, analysis//extracted/ and inside the KB; one row per dimension the file declares; selfGraded+passed, passed under its own threshold, an unconfirmed 0/0 dimension and unparseable JSON are faults. A code-extracted/mixed KB with no score file now shows one fault section (documents KB: skipped). The score file in /extracted/ is no longer misread as an item array. No captured validator output exists in the repo; the fixture reads the skill's documented example. token-burn.sh: opt-in --rates FILE (per-Mtok prices from the project, none built in; unmatched models UNPRICED), --windows FILE, --effort/--gap (agent working time from main-thread gaps, human prompt count by origin.kind). Aggregates only. Output without the new flags is byte-identical (verified on the captured fixture and a live transcript, before the last-record fix; T9 asserts). Field run on a real multi-day transcript: 89 human prompts, matching an independent count. token-burn.sh: usage per message.id now from its LAST record (subagent output was undercounted: fixture 60 -> 1559 output tokens, a live multi-agent session 37,791 -> 84,915). Found while promoting #171. Closes #151 Closes #296 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 3 + bin/extraction-report.sh | 203 ++++++++++- bin/token-burn.sh | 323 ++++++++++++++++-- skills/extractor-quality-loop.md | 6 + tests/wave2/fixtures/token-burn/CAPTURE.md | 56 ++- .../fixtures/token-burn/effort/s-3.jsonl | 19 ++ tests/wave2/test-extraction-quality-report.sh | 145 ++++++++ tests/wave2/test-token-burn.sh | 93 ++++- 8 files changed, 807 insertions(+), 41 deletions(-) create mode 100644 tests/wave2/fixtures/token-burn/effort/s-3.jsonl create mode 100755 tests/wave2/test-extraction-quality-report.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 20342404..1764be41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,9 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. ## Unreleased +- fix(token-burn): **usage is taken from each `message.id`'s LAST record, not its first — subagent output was undercounted.** The header claimed every streamed record of a message carries the same usage; in subagent transcripts the early records carry a partial `output_tokens`. On the captured fixture the subagent model's output went 60 → 1,559 (headline 59,858 → 61,357); on a 2026-10 multi-agent session 37,791 → 84,915 subagent output tokens; a field measurement on harness 2.1.296 saw up to 7×. Main-thread records were identical first to last, so main numbers do not move. **Every existing subagent-heavy token count rises after sync**; fixture expectations re-derived independently (`tests/wave2/fixtures/token-burn/CAPTURE.md`) — found while promoting [#171](https://github.com/MendixMau/mxcli-project-toolkit/issues/171) (measured-claims §7), MendixMau +- new(extraction-report): **the Stage 1 surface now reads the validator's `extraction-quality*.json` — one row per dimension the file declares, a self-graded pass shown as a fault** ([#151](https://github.com/MendixMau/mxcli-project-toolkit/issues/151)). `extractor-quality-loop.md` makes that file the proof of the 95% gate, yet nothing read it: a field build scored its process-model extractor at 100% on 7 dimensions and the report showed "2 pass, 18 skipped" with no quality section. Found at `analysis//`, `analysis//extracted/` and the same two inside the KB; dimension keys come from the file. Faults only where the skill's rules show in the file: `selfGraded` + `passed`, `passed` under its own `threshold`, a 0/0 dimension absent from `confirmedZeros`, unparseable JSON. **Behaviour change for existing projects:** a code-extracted or mixed KB with no score file now carries one `fault` section (migration-pipeline.md does not start BRDs before the 95% gate, so an unscored extraction is unmeasured); documents KBs get `skipped`, the gate (`gate-check.sh`, presence of the page) and the exit code are unchanged. A score file inside `/extracted/` is no longer misread as an unparseable item file. No captured validator output exists in this repo (no pipeline ships a validator), so the fixture reads the skill's documented example at run time; `tests/wave2/test-extraction-quality-report.sh` — MendixMau +- new(token-burn): **`--rates`, `--windows`, `--effort`: cost at the project's own per-Mtok rates, agent working time, human prompt count, per-window attribution — aggregates only** ([#296](https://github.com/MendixMau/mxcli-project-toolkit/issues/296)). A field team had to answer "money and time per round and per feature" with ad-hoc transcript scripts. No price is built in (rates file, model globs, first match wins; an unmatched model is UNPRICED and the total marked incomplete, never 0); agent time = main-thread gaps <= `--gap` minutes (default 5) not ending at a human prompt; human prompts = `origin.kind == "human"` user records (task notifications, peer messages, tool results, meta and compact summaries excluded), once per uuid, text never printed. Without the new flags the new code adds nothing to screen, `--json` or `--brief` output (byte-identical to the previous script before the last-record fix below; fixture T9 asserts no new section or key). Field run on a real multi-day transcript: human prompts matched an independent count (89 = 89). The human-prompt fixture records are hand-built from a profile of real record shapes — committing a scrubbed real capture was refused; `tests/wave2/test-token-burn.sh` T9-T14 — jopterhorst - learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run : N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau - learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst - fix(app-restart, test-stack-up): **a port held by another process before `run --local` is named and reported as "instrument blocked" (exit 4), never killed** ([#262](https://github.com/MendixMau/mxcli-project-toolkit/issues/262)). Studio Pro had started its own runtime on the app/admin ports a restart needed; `mxcli run --local` then failed with "Authentication failed" (the admin API answering was Studio Pro's), the next journey ran against Studio Pro's runtime, and four more runs were NOT RUN. `app-restart.sh` now checks the app port and the admin port (record's `adminPort`, else app + 10) before stopping anything — a holder outside the recorded process tree exits 4 with its name, pid and parent — and again after the stop, where anything still holding a port is by definition not ours. `test-stack-up.sh` runs the same check wherever it would tell you to start `run --local` (`--no-docker`, no Docker daemon, `--check` with a leftover record), on the record's ports, `RUN_LOCAL_APP_PORT`/`RUN_LOCAL_ADMIN_PORT`, or mxcli's defaults 8080/8090. Holder lookup: lsof, then ss, then netstat + tasklist on Windows; unnamed holders still block when nothing of ours is running. Fixtures: `test-app-restart.sh` +2 cases (14 assertions, all fail on the old script), `test-stack-up-local.sh` T7 (4 of 6 fail on the old script). — AlvinnYeo diff --git a/bin/extraction-report.sh b/bin/extraction-report.sh index de4c8f08..7117843b 100755 --- a/bin/extraction-report.sh +++ b/bin/extraction-report.sh @@ -255,7 +255,11 @@ def analyse(kb): # and the reverse is not: a merge cannot invent an item it never loaded. xcounts, xstate, xfiles = {}, ABSENT, [] if os.path.isdir(extracted): - xfiles = ls(extracted, r'.*\.json$') + # The validator's score file may sit here (#151: a field extractor wrote + # extracted/extraction-quality-.json). It is a report about the extraction, not + # an item array of it — counted as one, it read as "would not parse" and weakened the + # second opinion on every zero. It is read by the quality section below instead. + xfiles = [f for f in ls(extracted, r'.*\.json$') if not f.startswith('extraction-quality')] broke = [] for f in xfiles: st, items, det = read_json_list(os.path.join(extracted, f)) @@ -438,6 +442,133 @@ def analyse(kb): a['manifest'] = manifest if man_state in (OK, EMPTY) else [] a['manifestState'] = man_state + # -- extraction quality: the validator's own score file (#151) -------------------------------- + # + # `extractor-quality-loop.md` makes `extraction-quality.json` the proof that an extraction + # reached its >=95% gate. This page still scores nothing: every figure below is the + # validator's, read and printed — dimension keys come FROM THE FILE (a non-code source + # declares its own), never from a fixed list. What this page adds is only what the skill + # states as a rule and the file itself can show broken: + # - `selfGraded` with `passed: true` is a FAULT — "a self-graded report cannot pass"; + # - `passed: true` with `overallScore` under `threshold` is a contradiction, a FAULT; + # - a dimension at 0 found / 0 expected that `confirmedZeros` does not name is a FAULT + # (the Suspicious-Zero Gate: 0/0 scores 100 by arithmetic); + # - a file that will not parse is a FAULT, never an absent score. + # Locations: the skill's `analysis//extraction-quality.json` (the KB's parent), + # its `extracted/` (where a field extractor wrote `extraction-quality-.json`), and the + # same two inside the KB. Every `extraction-quality*.json` found is one section. + qdirs, qfiles = [], [] + for d in (os.path.dirname(kb), os.path.join(os.path.dirname(kb), 'extracted'), + kb, extracted): + rd = os.path.realpath(d) + if rd in qdirs or not os.path.isdir(d): + continue + qdirs.append(rd) + qfiles += [os.path.join(d, f) for f in ls(d, r'extraction-quality.*\.json$')] + a['quality'] = [] + QLOC = ('analysis//extraction-quality*.json, analysis//extracted/, and the same ' + 'two inside the knowledge base') + + def num(v): + return v if isinstance(v, (int, float)) and not isinstance(v, bool) else None + + for qf in qfiles: + rel = os.path.relpath(qf, project) + q = {'file': rel, 'rows': [], 'faults': [], 'notes': []} + try: + with open(qf) as fh: + data = json.load(fh) + except Exception as e: + data, q['verdict'] = None, ('fault', f'{rel} is present but unreadable — ' + f'{type(e).__name__}: {e}. Not a missing score: ' + f'a broken one.') + if data is not None and not isinstance(data, dict): + data, q['verdict'] = None, ('fault', f'{rel} is valid JSON but a ' + f'{type(data).__name__}, not the report object ' + f'extractor-quality-loop.md defines.') + if data is not None: + thr = num(data.get('threshold')) + if thr is None: + thr = 95 + q['notes'].append('no `threshold` in the file — compared against the skill\'s 95.') + overall, passed, sg = num(data.get('overallScore')), data.get('passed'), data.get('selfGraded') + cz = data.get('confirmedZeros') if isinstance(data.get('confirmedZeros'), list) else [] + cz_names = {z.get('dimension') for z in cz if isinstance(z, dict)} + dims = data.get('dimensions') + if not isinstance(dims, dict) or not dims: + q['faults'].append('no `dimensions` object — there is nothing the overall score ' + 'is an average of.') + dims = {} + for dk, dv in dims.items(): + if not isinstance(dv, dict): + q['rows'].append({'dimension': dk, 'score': None, 'found': None, + 'expected': None, 'gaps': [], + 'flag': 'not an object — unreadable'}) + q['faults'].append(f'dimension {dk} is not an object.') + continue + sc, fo, ex = num(dv.get('score')), num(dv.get('found')), num(dv.get('expected')) + gaps = dv.get('gaps') if isinstance(dv.get('gaps'), list) else [] + flag = '' + if fo == 0 and ex == 0 and dk not in cz_names: + flag = 'unconfirmed zero' + q['faults'].append(f'dimension {dk} is 0 found / 0 expected and is not in ' + f'confirmedZeros — the Suspicious-Zero Gate needs two ' + f'methods agreeing before a zero may score.') + elif sc is None: + flag = 'no score' + elif sc < thr: + flag = f'below {thr:g}' + q['rows'].append({'dimension': dk, 'score': sc, 'found': fo, 'expected': ex, + 'gaps': [str(g) for g in gaps], 'flag': flag}) + if sg and passed is True: + q['faults'].append('self-graded pass: `selfGraded` is set and `passed` is true. ' + 'extractor-quality-loop.md: a self-graded report cannot pass ' + 'the 95% gate until a second mechanism has scored it.') + if passed is True and overall is not None and overall < thr: + q['faults'].append(f'`passed` is true but overallScore {overall:g} is under the ' + f'file\'s own threshold {thr:g}.') + q.update({'overallScore': overall, 'threshold': thr, 'passed': passed, + 'selfGraded': sg if sg else False, 'confirmedZeros': cz, + 'scoredAt': data.get('scoredAt'), 'project': data.get('project')}) + if q['faults']: + q['verdict'] = ('fault', ' '.join(q['faults'])) + elif sg: + q['verdict'] = ('manual', 'self-graded and not passed — an inventory awaiting a ' + 'second mechanism (a different parser, a runtime probe, ' + 'an SME pass) before the gate can be judged.') + elif passed is not True: + q['verdict'] = ('manual', f'the file records the extraction as not passed ' + f'(overall {"?" if overall is None else format(overall, "g")} ' + f'against {thr:g}) — the quality loop is still running; ' + f'continuing it or waiving the gate is a human call.') + else: + q['verdict'] = ('pass', f'passed: overall {"?" if overall is None else format(overall, "g")} ' + f'against threshold {thr:g}, {len(q["rows"])} dimension(s), ' + f'{len(cz)} confirmed zero(s) — the validator\'s figures, ' + f'read, not re-scored.') + a['quality'].append(q) + a['sections']['quality:' + rel] = { + 'key': 'quality:' + rel, 'label': 'Extraction quality — ' + os.path.basename(qf), + 'family': 'quality', 'file': rel, 'state': OK if data is not None else BROKEN, + 'count': len(q['rows']), 'items': [], 'second': None, 'third': None, + 'expect': E, 'verdict': q['verdict']} + if not qfiles: + if prov == 'documents': + v = ('skipped', 'no extraction-quality*.json, and none is owed: a documents knowledge ' + 'base ran no extractor, so there is nothing for a validator to score.') + elif prov in ('code-extracted', 'mixed'): + v = ('fault', f'an extractor produced this knowledge base ({prov}) but no ' + f'extraction-quality*.json was found ({QLOC}). extractor-quality-loop.md ' + f'makes that file the proof of the >=95% gate, and migration-pipeline.md ' + f'does not start BRDs before it — an unmeasured extraction, not a pass.') + else: + v = ('manual', 'provenance undetermined — whether an extractor ran, and so whether a ' + 'score file is owed, needs a human.') + a['sections']['extractionQuality'] = { + 'key': 'extractionQuality', 'label': 'Extraction quality', 'family': 'quality', + 'file': '', 'state': ABSENT, 'count': 0, 'items': [], 'second': None, 'third': None, + 'expect': E, 'verdict': v} + # -- BRDs: Stage 2's artifact, shown here only as a forward link ----------------------------- a['brdCount'] = len(ls(os.path.join(kb, 'brd'), r'.*\.brd\.json$')) @@ -516,6 +647,9 @@ result = { 'path': a['path'], 'provenance': a['provenance'], 'brdCount': a['brdCount'], 'verdicts': a['tally'], 'discrepancies': a['faults'], 'dimensions': a['dims'], 'sections': {k: s['verdict'][0] for k, s in a['sections'].items()}, + 'extractionQuality': [dict({k: q.get(k) for k in ( + 'file', 'overallScore', 'threshold', 'passed', 'selfGraded', 'confirmedZeros', + 'rows', 'faults', 'notes')}, verdict=q['verdict'][0]) for q in a['quality']], } for a in analysed], } @@ -632,10 +766,10 @@ for a in analysed: if d['documentsExpected'] is not None else 'No discovery-manifest.json, so there is no expected count to divide by — the ' 'number on the left is a count, not a coverage.', doc_na) - + card('Overall score', 'not computed', - 'Deliberately. The six scored dimensions and the ≥95% threshold are ' - 'skills/extractor-quality-loop.md’s judgement; a composite ' - 'invented here would be a number with no skill behind it.') + + card('Overall score', 'not computed here', + 'Deliberately. The scored dimensions and the ≥95% threshold are ' + 'skills/extractor-quality-loop.md’s; the validator’s own ' + 'score, when it wrote one, is printed under Extraction quality below.') + '') mod_table = ('
' @@ -656,6 +790,52 @@ for a in analysed: t = a['tally'] code_blocks = ''.join(section_block(a['sections'][k]) for k, _f, _x, _l, _e in CODE_SECTIONS) + + def fmt_num(v): + return 'not stated' if v is None else esc(format(v, 'g')) + + for q in a['quality']: + s = a['sections']['quality:' + q['file']] + if 'overallScore' not in q: + continue # broken file: the verdict text is the content + rows = ''.join( + '' + f'' + f'' + f'' + f'' + + (f'' if r['flag'] == 'unconfirmed zero' or 'unreadable' in r['flag'] + else f'') + + '' for r in q['rows']) + sg = q['selfGraded'] + facts = ( + f'

Overall {fmt_num(q["overallScore"])} against threshold ' + f'{esc(format(q["threshold"], "g"))} · passed: ' + f'{esc(q["passed"])} · self-graded: ' + + ('' + esc(sg if sg is not True else 'yes') + '' if sg else 'no') + + (f' · scored {esc(q["scoredAt"])}' if q.get('scoredAt') else '') + '

') + cz = ''.join(f'
  • {esc(z.get("dimension"))} — ' + f'{esc(", ".join(map(str, z.get("methods") or [])) or "no methods named")}' + f'{(" — " + esc(z.get("note"))) if z.get("note") else ""}
    • ' + for z in q['confirmedZeros'] if isinstance(z, dict)) + s['body'] = (facts + + ''.join(f'

    {esc(n)}

    ' for n in q['notes']) + + '
    ModuleItems
    {esc(r["dimension"])}{fmt_num(r["score"])}{fmt_num(r["found"])} / {fmt_num(r["expected"])}{"; ".join(esc(g) for g in r["gaps"][:12]) or "—"}' + + (f' (+{len(r["gaps"]) - 12} more)' if len(r["gaps"]) > 12 else '') + + '{esc(r["flag"])}{esc(r["flag"])}
    ' + '' + + (rows or '') + + '
    DimensionScoreFound / expectedGapsFlag
    no dimensions in the file
    ' + + ('

    Confirmed zeros (two methods agreed):

      ' + + cz + '
    ' if cz else + '

    No confirmed zeros recorded.

    ')) + if s['verdict'][0] != 'pass': + # section_block prints only the reason for a non-pass verdict; for a quality file + # the figures ARE the evidence for the fault, so they go under it. + s['tail'] = s['body'] + quality_blocks = ''.join( + section_block(s) + (s.get('tail') or '') + for k, s in a['sections'].items() if s['family'] == 'quality') doc_blocks = ''.join(section_block(a['sections'][k]) for k, _l, _e in DOC_SECTIONS) body = f""" @@ -780,6 +960,13 @@ for a in analysed: figure reads not measured — it is never rounded up to a full bar.

    {dims} +

    Extraction quality

    +

    The validator’s own score file(s) per skills/extractor-quality-loop.md, + printed as written — one row per dimension the file declares. Faulted here only where the + skill’s own rules show in the file: a self-graded pass, a pass under its own threshold, + a 0/0 dimension no second method confirmed.

    + {quality_blocks} +

    Extracted constructs

    {code_blocks} @@ -841,6 +1028,12 @@ elif not quiet: print(f" MANUAL {s['key']}: {s['verdict'][1]}") for x in a['faults']: print(f" FAULT {x}") + for q in a['quality']: + if 'overallScore' in q: + print(f" quality {q['file']}: overall " + f"{'?' if q['overallScore'] is None else format(q['overallScore'], 'g')}" + f" / threshold {q['threshold']:g}, passed={q['passed']}, " + f"selfGraded={'yes' if q['selfGraded'] else 'no'}, {len(q['rows'])} dimension(s)") d = a['dims'] print(f" zeros corroborated: {d['zerosCorroborated']}/{d['zerosTotal']}; " f"module attribution: " diff --git a/bin/token-burn.sh b/bin/token-burn.sh index 668c2a34..cc89e643 100755 --- a/bin/token-burn.sh +++ b/bin/token-burn.sh @@ -7,6 +7,9 @@ # bin/token-burn.sh --scan-all # look in every transcript dir, not only the # # project's own and its ancestors/descendants # bin/token-burn.sh --json # the same numbers as one JSON object +# ... --rates FILE # + cost per model at YOUR per-Mtok rates (format: "RATES FILE" below) +# ... --windows FILE # + per-window table: tokens, cost, agent time, human prompts +# ... --effort [--gap MIN] # + agent working time and human prompt count (default gap 5 min) # # PRODUCER. The Claude Code harness (CLI, desktop app, IDE extension) writes one JSONL # transcript per session under `$CLAUDE_CONFIG_DIR/projects//.jsonl` (default @@ -21,8 +24,14 @@ # by `message.model`. Three facts about the format, each learned from a real transcript # (2026-09-18, a 17,000-line toolkit session) and each one a silent miscount if ignored: # 1. The harness writes ONE RECORD PER STREAMED CONTENT BLOCK, all carrying the same -# `message.id` and the same usage — that session had 4,336 assistant records for 1,989 -# messages. Summing records overcounts ~2×. Usage is summed ONCE per `message.id`. +# `message.id` — that session had 4,336 assistant records for 1,989 messages, so summing +# records overcounts ~2×. Usage is summed ONCE per `message.id`, from that id's LAST +# record: the usage is NOT the same on every record. In subagent transcripts the early +# records carry a partial `output_tokens` (the captured fixture: 60 on the first record, +# 1,559 on the last; a 2026-10 multi-agent session: 37,791 vs 84,915 subagent output +# tokens, first vs last). Keeping the first record undercounted subagent output by up to +# 7× in a field measurement; main-thread records were identical first to last. The +# message's day and time are its FIRST record's (when it started). # 2. `message.model` is `` on harness-generated records (aborts, errors). Skipped. # 3. `cwd` is per RECORD, not per session, and a session launched in `~` can spend most of its # tokens inside `~/proj`. So records are attributed to the project by their own `cwd` @@ -43,25 +52,67 @@ # which stage a session was actually working — the caveat is printed with the table. Dispatch # lines (runbook §1c) will sharpen it once they carry a stage and a date. # -# Exit 0 always — this is read, never gated on. Bash 3.2 + Python 3 via lib/portable.sh. +# COST, TIME AND PROMPTS (#296, opt-in — without these flags every output above is byte-for- +# byte what it was). Asked for in the field as "money and time per round and per feature", +# which the project had to answer with ad-hoc scripts over the transcripts. +# RATES FILE (--rates): no price is built in — rates change per model, plan and month, so they +# come from the project. One line per model glob, first match wins, `*` as a catch-all: +# currency EUR # optional label, default EUR +# # glob input cache-write cache-read output (per million tokens) +# claude-fable-* 2.80 3.50 0.28 14.00 +# * 0.90 1.10 0.09 4.50 +# A model no line matches is printed UNPRICED and the total is marked incomplete — never 0. +# WINDOWS FILE (--windows): one line per window, ` `, start as YYYY-MM-DD or +# YYYY-MM-DDTHH:MM[:SS] (UTC); a window runs until the next one starts, the last one is +# open-ended, and anything before the first lands in "(before first window)". Use it for +# setup / map / round N / wrap-up boundaries. +# AGENT WORKING TIME: per main-thread transcript file, the sum of gaps between consecutive +# user/assistant records that are <= --gap minutes, EXCLUDING every gap that ends at a +# human prompt (that gap is the human thinking, not the agent working). Subagent time is +# not added: it runs inside the main thread's own gaps. +# HUMAN PROMPTS: main-thread `user` records that are not tool results, not `isMeta`, not a +# compact summary; when the harness writes `origin`, only `origin.kind == "human"` counts +# (task notifications and peer messages are `user` records too); older records without +# `origin` count unless their text starts with `<` (harness-injected command/notification +# wrappers). Counted once per `uuid`. The text is read only to make that test and is +# never printed — every output is an aggregate. +# +# Exit 0 always — this is read, never gated on (a malformed --rates/--windows file is a usage +# error, exit 1). Bash 3.2 + Python 3 via lib/portable.sh. set -u TOOLKIT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # shellcheck source=lib/portable.sh . "$TOOLKIT_ROOT/bin/lib/portable.sh" -PROJECT_DIR=""; MODE=screen; SCAN_ALL=0 -for a in "$@"; do +PROJECT_DIR=""; MODE=screen; SCAN_ALL=0; RATES=""; WINDOWS=""; EFFORT=0; GAP=5 +USAGE="usage: bin/token-burn.sh [--brief|--json] [--scan-all] [--rates FILE] [--windows FILE] [--effort] [--gap MIN]" +while [ $# -gt 0 ]; do + a="$1" case "$a" in --brief) MODE=brief ;; --json) MODE=json ;; --scan-all) SCAN_ALL=1 ;; - -h|--help) sed -n '2,8p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + --effort) EFFORT=1 ;; + --rates|--windows|--gap) + [ $# -ge 2 ] || { echo "$a needs a value" >&2; echo "$USAGE" >&2; exit 1; } + case "$a" in + --rates) RATES="$2" ;; + --windows) WINDOWS="$2" ;; + --gap) GAP="$2"; EFFORT=1 ;; + esac + shift ;; + -h|--help) sed -n '2,11p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; --*) echo "unknown option: $a" >&2; exit 1 ;; *) PROJECT_DIR="$a" ;; esac + shift +done +[ -n "$PROJECT_DIR" ] || { echo "$USAGE" >&2; exit 1; } +for f in "$RATES" "$WINDOWS"; do + [ -z "$f" ] || [ -f "$f" ] || { echo "not a file: $f" >&2; exit 1; } done -[ -n "$PROJECT_DIR" ] || { echo "usage: bin/token-burn.sh [--brief|--json] [--scan-all]" >&2; exit 1; } -PROJECT_DIR="$(cd "$PROJECT_DIR" 2>/dev/null && pwd)" || { echo "not a directory: $1" >&2; exit 1; } +case "$GAP" in ''|*[!0-9.]*) echo "--gap takes minutes (a number), got: $GAP" >&2; exit 1 ;; esac +PROJECT_DIR="$(cd "$PROJECT_DIR" 2>/dev/null && pwd)" || { echo "not a directory: $PROJECT_DIR" >&2; exit 1; } TRANSCRIPTS="${CLAUDE_CONFIG_DIR:-$HOME/.claude}/projects" if [ ! -d "$TRANSCRIPTS" ]; then @@ -74,11 +125,84 @@ if [ ! -d "$TRANSCRIPTS" ]; then fi require_py -"$PY" - "$PROJECT_DIR" "$TRANSCRIPTS" "$MODE" "$SCAN_ALL" <<'PYEOF' -import json, os, re, sys +"$PY" - "$PROJECT_DIR" "$TRANSCRIPTS" "$MODE" "$SCAN_ALL" "$RATES" "$WINDOWS" "$EFFORT" "$GAP" <<'PYEOF' +import json, os, re, sys, fnmatch, calendar from collections import defaultdict project, root, mode, scan_all = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4] == "1" +rates_file, windows_file = sys.argv[5], sys.argv[6] +want_effort = sys.argv[7] == "1" or bool(windows_file) +gap_s = float(sys.argv[8]) * 60.0 + +def die(msg): + sys.stderr.write("token-burn: " + msg + "\n") + sys.exit(1) + +def epoch(ts): + """ISO-8601 UTC (the transcript's `2026-09-08T00:59:57.347Z`, or a windows-file start) -> + seconds. None when it does not parse — such a record carries no time, never time 0.""" + m = re.match(r"^(\d{4})-(\d{2})-(\d{2})(?:[T ](\d{2}):(\d{2})(?::(\d{2})(\.\d+)?)?)?(Z|[+-]00:?00)?$", ts or "") + if not m: + return None + y, mo, d, hh, mi, ss, frac = m.group(1, 2, 3, 4, 5, 6, 7) + t = calendar.timegm((int(y), int(mo), int(d), int(hh or 0), int(mi or 0), int(ss or 0), 0, 0, 0)) + return t + (float(frac) if frac else 0.0) + +# --- --rates: per-Mtok prices from the project's own file, never built in ------------------- +currency, rates = "EUR", [] +if rates_file: + with open(rates_file, encoding="utf-8", errors="replace") as fh: + for n, line in enumerate(fh, 1): + line = line.split("#", 1)[0].strip() + if not line: + continue + f = line.split() + if f[0] == "currency" and len(f) == 2: + currency = f[1] + continue + if len(f) != 5: + die("%s:%d: want ` ` (per Mtok), got %d field(s)" % (rates_file, n, len(f))) + try: + rates.append((f[0], [float(x) for x in f[1:]])) + except ValueError: + die("%s:%d: rates must be numbers" % (rates_file, n)) + if not rates: + die("%s: no rate lines" % rates_file) + +def rate_for(model): + for glob, r in rates: + if fnmatch.fnmatchcase(model, glob): + return r + return None + +def cost_of(model, vals): + r = rate_for(model) + return None if r is None else sum(v * x for v, x in zip(vals, r)) / 1e6 + +# --- --windows: named time windows, each running until the next one starts ------------------ +windows = [] +if windows_file: + with open(windows_file, encoding="utf-8", errors="replace") as fh: + for n, line in enumerate(fh, 1): + line = line.split("#", 1)[0].strip() + if not line: + continue + f = line.split(None, 1) + t = epoch(f[0]) + if t is None or len(f) < 2: + die("%s:%d: want ` `" % (windows_file, n)) + windows.append((t, f[1].strip(), f[0])) + if not windows: + die("%s: no window lines" % windows_file) + windows.sort() +BEFORE = "(before first window)" + +def window_for(t): + name = BEFORE + for start, wname, _raw in windows: + if t is not None and t >= start: + name = wname + return name def slug(path): return re.sub(r"[^A-Za-z0-9]", "-", path) @@ -120,6 +244,24 @@ by_day = defaultdict(lambda: [0, 0, 0, 0, 0]) day_model = defaultdict(lambda: defaultdict(int)) # day -> model -> headline sidechain = [0, 0] # headline tokens: main, subagent n_files = n_records = n_synthetic = 0 +msgs = [] # (epoch, model, vals, is_sidechain) per message +last, order = {}, [] # message.id -> [first ts, model, LAST usage, sidechain] +human_seen = set() +humans = [] # epoch per human prompt +timeline = defaultdict(list) # main-thread file -> [(epoch, is_human)] + +def is_human_prompt(rec): + if rec.get("type") != "user" or rec.get("isSidechain") or rec.get("isMeta") or rec.get("isCompactSummary"): + return False + c = (rec.get("message") or {}).get("content") + if isinstance(c, list): + if any(isinstance(b, dict) and b.get("type") == "tool_result" for b in c): + return False + texts = [b.get("text") for b in c if isinstance(b, dict) and b.get("type") == "text"] + c = texts[0] if texts else "" + if isinstance(rec.get("origin"), dict): + return rec["origin"].get("kind") == "human" + return isinstance(c, str) and not c.lstrip().startswith("<") for d in dirs: for f in jsonl_files(d): @@ -130,6 +272,18 @@ for d in dirs: rec = json.loads(line) except ValueError: continue + if want_effort and rec.get("type") in ("user", "assistant") and not rec.get("isSidechain") \ + and os.sep + "subagents" + os.sep not in f and under_project(rec.get("cwd")): + t = epoch(rec.get("timestamp")) + if t is not None: + hp = is_human_prompt(rec) + if hp: + key = rec.get("uuid") or (f + "@" + str(rec.get("timestamp"))) + if key in human_seen: + continue + human_seen.add(key) + humans.append(t) + timeline[f].append((t, hp)) if rec.get("type") != "assistant": continue msg = rec.get("message") or {} @@ -141,20 +295,28 @@ for d in dirs: continue n_records += 1 mid = msg.get("id") or rec.get("uuid") or (f + str(n_records)) + u = msg.get("usage") or {} + vals = [int(u.get(k) or 0) for k in KEYS] if mid in seen: + last[mid][2] = vals # later record of the same message: its usage wins continue seen.add(mid) - u = msg.get("usage") or {} - vals = [int(u.get(k) or 0) for k in KEYS] - day = (rec.get("timestamp") or "")[:10] or "????-??-??" - for i, v in enumerate(vals): - by_model[model][i] += v - by_day[day][i] += v - by_model[model][4] += 1 - by_day[day][4] += 1 - head = vals[0] + vals[1] + vals[3] - day_model[day][model] += head - sidechain[1 if rec.get("isSidechain") else 0] += head + order.append(mid) + last[mid] = [rec.get("timestamp") or "", model, vals, bool(rec.get("isSidechain"))] + +# one entry per message.id: first record's time, LAST record's usage (see header, fact 1) +for mid in order: + ts, model, vals, side = last[mid] + day = ts[:10] or "????-??-??" + for i, v in enumerate(vals): + by_model[model][i] += v + by_day[day][i] += v + by_model[model][4] += 1 + by_day[day][4] += 1 + head = vals[0] + vals[1] + vals[3] + day_model[day][model] += head + msgs.append((epoch(ts), model, vals, side)) + sidechain[1 if side else 0] += head def headline(v): return v[0] + v[1] + v[3] @@ -220,6 +382,87 @@ caveat = ("stage = highest stage whose dated PROJECT.md row is on or before that "UNMAPPED = before the first dated row" if stage_rows else "no dated stage rows in PROJECT.md — every day is UNMAPPED") +# --- opt-in: cost, agent time, human prompts, windows (#296) ------------------------------- +def money(x): + return "UNPRICED" if x is None else "%.2f" % x + +cost = None +if rates_file: + cm = {} + unpriced = set() + for t, mdl, vals, side in msgs: + r = rate_for(mdl) + if r is None: + unpriced.add(mdl) + continue + e = cm.setdefault(mdl, [0.0, 0.0, 0.0, 0.0]) + for i in range(4): + e[i] += vals[i] * r[i] / 1e6 + side_cost = [0.0, 0.0] + for t, mdl, vals, side in msgs: + c = cost_of(mdl, vals) + if c is not None: + side_cost[1 if side else 0] += c + cost = {"currency": currency, "rates_file": rates_file, + "by_model": {m: [round(x, 4) for x in v] + [round(sum(v), 4)] for m, v in cm.items()}, + "total": round(sum(sum(v) for v in cm.values()), 4), + "main": round(side_cost[0], 4), "subagents": round(side_cost[1], 4), + "unpriced_models": sorted(unpriced), "complete": not unpriced} + +def agent_seconds(pred=lambda t: True): + tot = 0.0 + for f, evs in timeline.items(): + evs = sorted(evs) + for (t0, _h0), (t1, h1) in zip(evs, evs[1:]): + d = t1 - t0 + if 0 < d <= gap_s and not h1 and pred(t1): + tot += d + return tot + +def hm(sec): + mins = int(round(sec / 60.0)) + return "%dh %02dm" % (mins // 60, mins % 60) + +effort = None +if want_effort: + effort = {"agent_seconds": round(agent_seconds()), "human_prompts": len(humans), + "gap_minutes": gap_s / 60.0, "main_thread_files": len(timeline)} + +win = None +if windows: + names = [BEFORE] + [w for _t, w, _r in windows] + order = [] + for n in names: + if n not in order: + order.append(n) + win = {n: {"messages": 0, "usage": [0, 0, 0, 0], "cost": 0.0 if rates_file else None, + "unpriced": 0, "agent_seconds": 0, "human_prompts": 0} for n in order} + starts = {} + for t, n, raw in windows: + starts.setdefault(n, raw) + for t, mdl, vals, side in msgs: + w = win[window_for(t)] + w["messages"] += 1 + for i in range(4): + w["usage"][i] += vals[i] + if rates_file: + c = cost_of(mdl, vals) + if c is None: + w["unpriced"] += 1 + else: + w["cost"] += c + for t in humans: + win[window_for(t)]["human_prompts"] += 1 + for n in order: + win[n]["agent_seconds"] = round(agent_seconds(lambda t, n=n: window_for(t) == n)) + if win[n]["cost"] is not None: + win[n]["cost"] = round(win[n]["cost"], 4) + win[n]["start"] = starts.get(n) + if not any(win[BEFORE][k] for k in ("messages", "human_prompts", "agent_seconds")): + del win[BEFORE] + order.remove(BEFORE) + win = {"order": order, "windows": win} + # --- output ------------------------------------------------------------------------------- if mode == "json": out = { @@ -232,6 +475,13 @@ if mode == "json": "headline": {"main": sidechain[0], "subagents": sidechain[1]}, "current_stage": current, "caveat": caveat, } + # opt-in keys only: without the flags the object is exactly what it was + if cost is not None: + out["cost"] = cost + if effort is not None: + out["effort"] = effort + if win is not None: + out["windows"] = win print(json.dumps(out, indent=1, sort_keys=True)) sys.exit(0) @@ -286,4 +536,35 @@ for s in sorted(by_stage, key=stage_sort): print(" caveat: %s" % caveat) if current: print(" register says current stage %s%s" % (current, "" if current in by_stage else " — no day attributed to it yet")) +if cost is not None: + print() + print("COST (%s per Mtok from %s)" % (currency, rates_file)) + fmtc = "%-22s %9s %11s %11s %9s %10s" + print(fmtc % ("MODEL", "INPUT", "CACHE-WRITE", "CACHE-READ", "OUTPUT", "TOTAL")) + for m in sorted(by_model, key=lambda m: -headline(by_model[m])): + if m in cost["by_model"]: + v = cost["by_model"][m] + print(fmtc % (short(m), money(v[0]), money(v[1]), money(v[2]), money(v[3]), money(v[4]))) + else: + print(fmtc % (short(m), "", "", "", "", "UNPRICED")) + print(fmtc % ("TOTAL", "", "", "", "", money(cost["total"]) + ("" if cost["complete"] else "+"))) + if cost["main"] + cost["subagents"]: + print(" subagents (Agent tool): %s %s of %s" % (money(cost["subagents"]), currency, money(cost["total"]))) + if not cost["complete"]: + print(" INCOMPLETE — no rate line matches: %s (add a line or a `*` catch-all)" % ", ".join(cost["unpriced_models"])) +if effort is not None: + print() + print("agent working time: %s (main thread, gaps <= %g min, gaps ending at a human prompt excluded) · human prompts: %d" % ( + hm(effort["agent_seconds"]), gap_s / 60.0, effort["human_prompts"])) +if win is not None: + print() + print("WINDOWS (from %s; each runs until the next starts)" % windows_file) + fmtw = "%-24s %-17s %6s %9s %11s %10s %10s %8s" + print(fmtw % ("WINDOW", "FROM", "MSGS", "HEADLINE", "CACHE-READ", "COST", "AGENT-TIME", "PROMPTS")) + for n in win["order"]: + w = win["windows"][n] + c = "-" if w["cost"] is None else money(w["cost"]) + ("+" if w["unpriced"] else "") + print(fmtw % (n[:24], w["start"] or "", w["messages"], k(headline(w["usage"])), k(w["usage"][2]), + c, hm(w["agent_seconds"]), w["human_prompts"])) + print(" agent time = main-thread gaps <= %g min not ending at a human prompt; a trailing + = some messages unpriced" % (gap_s / 60.0)) PYEOF diff --git a/skills/extractor-quality-loop.md b/skills/extractor-quality-loop.md index fea175d2..ae5ca148 100644 --- a/skills/extractor-quality-loop.md +++ b/skills/extractor-quality-loop.md @@ -177,6 +177,12 @@ Noting these as "BRD-only" in the quality report is correct and expected. } ``` +**Where it is read:** `bin/extraction-report.sh` (the Stage 1 surface) renders every +`extraction-quality*.json` at `analysis//` or `analysis//extracted/` (also +inside the knowledge base), one row per key of `dimensions`. A code-extracted knowledge base +with no such file, a `selfGraded` pass, a pass under `threshold`, or a 0/0 dimension missing +from `confirmedZeros` shows there as a fault. + ### A score without a second mechanism is self-graded — label it, never let it pass alone When the "ground truth" side of the comparison is a manual read with no independent second diff --git a/tests/wave2/fixtures/token-burn/CAPTURE.md b/tests/wave2/fixtures/token-burn/CAPTURE.md index b30847c6..4ea125bc 100644 --- a/tests/wave2/fixtures/token-burn/CAPTURE.md +++ b/tests/wave2/fixtures/token-burn/CAPTURE.md @@ -8,7 +8,10 @@ plus one of its Agent-tool subagent transcripts), then scrubbed. Every structura it was imagined: - one `assistant` record **per streamed content block**, all carrying the same `message.id` - and the same `usage` (the ~2× overcount if summed per record); + (the ~2× overcount if summed per record) — but NOT always the same `usage`: in the subagent + transcript `msg` ids with a later record carry a larger `output_tokens` on that last record + (60 on the first records of the 2026-09-17 messages, 1559 summed over last records). The + instrument takes each id's LAST record; the numbers below were recomputed that way; - `message.model` = `` on a harness-generated record (exactly one kept); - `cwd` is **per record**: a session launched in `~` spends records in `~`, `~/proj` and `~/proj/app`; @@ -47,19 +50,58 @@ publish. ## Expected numbers (computed independently of the instrument) -One Python one-liner over both files: unique `message.id`, `type == assistant`, model not -``, `cwd` at or under `/srv/example/proj`. Columns are +One Python one-liner over both files: unique `message.id` (usage from its LAST record), +`type == assistant`, model not ``, `cwd` at or under `/srv/example/proj`. Columns are input · cache-write · cache-read · output · messages; headline = input + cache-write + output. | | input | cache-write | cache-read | output | msgs | headline | |---|---|---|---|---|---|---| | claude-fable-5-1 | 288 | 31365 | 2607781 | 15524 | 9 | 47177 | -| claude-sonnet-5 (subagent) | 26 | 59772 | 998476 | 60 | 13 | 59858 | -| **total** | 314 | 91137 | 3606257 | 15584 | 22 | 107035 | +| the subagent model (subagent) | 26 | 59772 | 998476 | 1559 | 13 | 61357 | +| **total** | 314 | 91137 | 3606257 | 17083 | 22 | 108534 | + +(Taking the FIRST record per id, as the instrument did until 2026-10-13, gives subagent output +60 and headline 59858 — a 26× undercount of that model's output on this sample.) Records: 52 in-project assistant records collapse to 22 messages; 76 records at `/srv/example` are excluded; 6 records of `msg_14` are at `/srv/example/proj/app` and count. -Per day (headline): 2026-09-08 → 38527 (5 msgs) · 2026-09-17 → 59858 (13) · 2026-09-18 → 8650 (4). +Per day (headline): 2026-09-08 → 38527 (5 msgs) · 2026-09-17 → 61357 (13) · 2026-09-18 → 8650 (4). With `PROJECT.md` beside: stage 1 = 2026-09-08 = 38527 (`39k`); stage 2 = 09-17 + 09-18 = -68508 (`69k`, sonnet-5 87% · fable-5-1 13%, cache-read 1438109 → `1438k`); current stage 2. +70007 (`70k`, subagent model 88% · fable-5-1 12%, cache-read 1438109 → `1438k`); current stage 2. + +## `effort/s-3.jsonl` — hand-built, and why (#296) + +**This file is NOT a capture.** The captured `s-1.jsonl` carries no main-thread human prompt +(its `user` records were not selected), so the human-prompt classifier has nothing captured to +run on. A fresh capture was taken from a real session and refused at commit time — a real +session's transcript, even scrubbed, is not something to publish from a shared machine. So +`s-3.jsonl` is built by hand, with every key and value shape taken from a profile of a real +harness transcript (harness 2.1.29x, 2026-10): only keys and value kinds were read, never text. +The facts it encodes, each seen in that profile: + +- human prompts are `type: user` with `origin: {"kind": "human"}` and `promptSource: "sdk"`; + some carry `promptId`/`userType`, some (written by a remote client) do not; content is a + string or a list holding `text` blocks; +- task notifications and peer messages are ALSO `type: user`, distinguished only by + `origin.kind` (`task-notification`, `peer`) — peer ones also `isMeta: true`; +- tool results are `type: user` with a `tool_result` block and a top-level `toolUseResult`; +- `isMeta: true` and `isCompactSummary: true` records are `type: user` too; +- records with no `origin` exist (older harness versions, and harness-injected command + wrappers whose text starts with `<`); +- a human prompt record can be written twice with the same `uuid`. + +Plus one record whose `cwd` is outside the project and one `isSidechain: true` record. +Field proof is a run, not this file: on the real transcript profiled above the instrument +counted the same number of human prompts (89) as an independent count of `origin.kind == +"human"`, non-meta, non-compact records. + +Expected (independent Python over the files, not the instrument): main-thread agent time +with the 5-minute cut-off = 220.465 s (captured `s-1.jsonl`; 164.6 s before +2026-09-18T02:00, 55.8 s after) + 371 s (`s-3.jsonl`) = **591 s**; with a 1-minute cut-off +the 114 s gap drops out of `s-3` → **477 s**. Human prompts: **4** (`h-1`, `h-2`, `h-3` once, +`h-4` with no `origin`). Windows `2026-09-01 setup` / `2026-09-17 round 1` / +`2026-09-18T02:00 wrap-up`: setup 5 msgs 38527 headline; round 1 13 msgs 61357; wrap-up 4 +captured + 6 `s-3` msgs = 10, 10557 headline, 427 s, 4 prompts. Rates +`claude-fable-* 2/2.5/0.2/10` + `* 1/1/0.1/5` per Mtok on the captured pair: 0.7557847 + +0.1674406 = 0.9232253. diff --git a/tests/wave2/fixtures/token-burn/effort/s-3.jsonl b/tests/wave2/fixtures/token-burn/effort/s-3.jsonl new file mode 100644 index 00000000..276bdf14 --- /dev/null +++ b/tests/wave2/fixtures/token-burn/effort/s-3.jsonl @@ -0,0 +1,19 @@ +{"type":"user","sessionId":"s-3","uuid":"h-1","parentUuid":null,"isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:00:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"human"},"promptSource":"sdk","message":{"role":"user","content":"[scrubbed]"}} +{"type":"assistant","sessionId":"s-3","uuid":"a-1","parentUuid":"h-1","isSidechain":false,"timestamp":"2026-09-20T10:00:05.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e1","type":"message","role":"assistant","content":[{"type":"tool_use","id":"tu-1","name":"Tool","input":{}}],"usage":{"input_tokens":100,"cache_creation_input_tokens":1000,"cache_read_input_tokens":10000,"output_tokens":50}}} +{"type":"user","sessionId":"s-3","uuid":"t-1","parentUuid":"a-1","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:00:06.000Z","cwd":"/srv/example/proj","toolUseResult":"[scrubbed]","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"tu-1","content":"[scrubbed]"}]}} +{"type":"assistant","sessionId":"s-3","uuid":"a-2","parentUuid":"t-1","isSidechain":false,"timestamp":"2026-09-20T10:01:06.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e2","type":"message","role":"assistant","content":[{"type":"text","text":"[scrubbed]"}],"usage":{"input_tokens":10,"cache_creation_input_tokens":0,"cache_read_input_tokens":20000,"output_tokens":200}}} +{"type":"user","sessionId":"s-3","uuid":"n-1","parentUuid":"a-2","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:03:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"task-notification"},"promptSource":"system","message":{"role":"user","content":""}} +{"type":"assistant","sessionId":"s-3","uuid":"a-3","parentUuid":"n-1","isSidechain":false,"timestamp":"2026-09-20T10:03:10.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e3","type":"message","role":"assistant","content":[{"type":"text","text":"[scrubbed]"}],"usage":{"input_tokens":5,"cache_creation_input_tokens":500,"cache_read_input_tokens":0,"output_tokens":30}}} +{"type":"user","sessionId":"s-3","uuid":"h-2","parentUuid":"a-3","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:20:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"human"},"promptSource":"sdk","message":{"role":"user","content":[{"type":"text","text":"[scrubbed]"}]}} +{"type":"assistant","sessionId":"s-3","uuid":"a-4","parentUuid":"h-2","isSidechain":false,"timestamp":"2026-09-20T10:20:30.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e4","type":"message","role":"assistant","content":[{"type":"text","text":"[scrubbed]"}],"usage":{"input_tokens":1,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":1}}} +{"type":"user","sessionId":"s-3","uuid":"m-1","parentUuid":"a-4","isSidechain":false,"userType":"external","isMeta":true,"timestamp":"2026-09-20T10:20:31.000Z","cwd":"/srv/example/proj","message":{"role":"user","content":"[scrubbed]"}} +{"type":"user","sessionId":"s-3","uuid":"h-3","parentUuid":"m-1","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:21:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"human"},"promptSource":"sdk","message":{"role":"user","content":"[scrubbed]"}} +{"type":"user","sessionId":"s-3","uuid":"h-3","parentUuid":"m-1","isSidechain":false,"timestamp":"2026-09-20T10:21:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"human"},"promptSource":"sdk","message":{"role":"user","content":"[scrubbed]"}} +{"type":"assistant","sessionId":"s-3","uuid":"a-5","parentUuid":"h-3","isSidechain":false,"timestamp":"2026-09-20T10:21:20.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e5","type":"message","role":"assistant","content":[{"type":"text","text":"[scrubbed]"}],"usage":{"input_tokens":2,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":2}}} +{"type":"user","sessionId":"s-3","uuid":"c-1","parentUuid":"a-5","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:22:00.000Z","cwd":"/srv/example/proj","message":{"role":"user","content":""}} +{"type":"user","sessionId":"s-3","uuid":"h-4","parentUuid":"c-1","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:22:30.000Z","cwd":"/srv/example/proj","message":{"role":"user","content":"[scrubbed]"}} +{"type":"user","sessionId":"s-3","uuid":"k-1","parentUuid":"h-4","isSidechain":false,"userType":"external","isCompactSummary":true,"timestamp":"2026-09-20T10:23:00.000Z","cwd":"/srv/example/proj","message":{"role":"user","content":"[scrubbed]"}} +{"type":"assistant","sessionId":"s-3","uuid":"a-6","parentUuid":"k-1","isSidechain":false,"timestamp":"2026-09-20T10:23:10.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e6","type":"message","role":"assistant","content":[{"type":"text","text":"[scrubbed]"}],"usage":{"input_tokens":3,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":3}}} +{"type":"user","sessionId":"s-3","uuid":"p-1","parentUuid":"a-6","isSidechain":false,"userType":"external","isMeta":true,"timestamp":"2026-09-20T10:24:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"peer"},"promptSource":"system","message":{"role":"user","content":"[scrubbed]"}} +{"type":"user","sessionId":"s-3","uuid":"x-1","parentUuid":"p-1","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:24:10.000Z","cwd":"/srv/example/other","origin":{"kind":"human"},"promptSource":"sdk","message":{"role":"user","content":"[scrubbed]"}} +{"type":"user","sessionId":"s-3","uuid":"s-1","parentUuid":null,"isSidechain":true,"userType":"external","timestamp":"2026-09-20T10:24:20.000Z","cwd":"/srv/example/proj","message":{"role":"user","content":"[scrubbed]"}} diff --git a/tests/wave2/test-extraction-quality-report.sh b/tests/wave2/test-extraction-quality-report.sh new file mode 100755 index 00000000..7da2f4e7 --- /dev/null +++ b/tests/wave2/test-extraction-quality-report.sh @@ -0,0 +1,145 @@ +#!/usr/bin/env bash +# test-extraction-quality-report.sh — bin/extraction-report.sh reads the validator's +# extraction-quality*.json (#151) and renders one row per dimension the file declares. +# +# GOLDEN INPUT. No extraction-quality.json exists anywhere in this repo: none of the three +# pipelines ships a validator, so there is no captured sample output to pin. The input here is +# the example object in skills/extractor-quality-loop.md "Output Format" — read out of the skill +# at run time, never copied, so the fixture follows the documented contract if the skill moves. +# Variants (self-graded, process-model dimensions, 0/0, broken) are mutations of that object. +# A captured validator output from a field run is still owed. +# +# Q1 documents KB, no score file → section `skipped`, no new fault (existing projects) +# Q2 the skill's example at analysis//extraction-quality.json → read, 6 rows, not +# passed → `manual`; figures on the page (score, found/expected, gaps, confirmed zero) +# Q3 analysis//extracted/extraction-quality-.json with the stack's OWN dimension +# keys, selfGraded + passed → `fault` "self-graded pass" +# Q4 code-extracted KB, no score file → `fault` (the loop is mandatory for an extractor) +# Q5 the score file inside /extracted/ is not counted as an unparseable item file +# Q6 a 0/0 dimension not in confirmedZeros → fault; the example's confirmed one is not +# Q7 passed:true under its own threshold → fault; broken JSON → fault, page still rendered +# +# usage: bash tests/wave2/test-extraction-quality-report.sh /path/to/extraction-report.sh + +set -uo pipefail +SUT="${1:?usage: test-extraction-quality-report.sh /path/to/extraction-report.sh}" +case "$SUT" in /*) ;; *) SUT="$PWD/$SUT" ;; esac +TOOLKIT="$(cd "$(dirname "$SUT")/.." && pwd)" +SKILL="$TOOLKIT/skills/extractor-quality-loop.md" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/eqr.XXXXXX")" +trap 'rm -rf "$WORK"' EXIT +PASS=0; FAIL=0 +ok() { PASS=$((PASS+1)); printf ' ok %s\n' "$1"; } +bad() { FAIL=$((FAIL+1)); printf ' FAIL %s\n %s\n' "$1" "${2:-}"; } +has() { case "$2" in *"$3"*) ok "$1" ;; *) bad "$1" "missing '$3'" ;; esac; } +hasnt() { case "$2" in *"$3"*) bad "$1" "unexpected '$3'" ;; *) ok "$1" ;; esac; } + +# shellcheck disable=SC1091 +. "$TOOLKIT/bin/lib/portable.sh" +require_py + +# the documented example, straight out of the skill +EXAMPLE="$WORK/example.json" +awk '/^## Output Format: extraction-quality.json/{f=1;next} f&&/^```json/{g=1;next} g&&/^```/{exit} g' "$SKILL" > "$EXAMPLE" +"$PY" -c 'import json,sys; d=json.load(open(sys.argv[1])); assert len(d["dimensions"])==6' "$EXAMPLE" \ + || { echo "could not read the example out of $SKILL"; exit 2; } + +mutate() { # $1 in, $2 out, $3 statements over `d` + "$PY" -c 'import json,sys +d=json.load(open(sys.argv[1])) +exec(sys.argv[3]) +json.dump(d,open(sys.argv[2],"w"),indent=2)' "$1" "$2" "$3" +} + +doc_kb() { # a documents-provenance knowledge base + mkdir -p "$1/analysis/src/knowledge-base/share" + printf '# KB\n\nsomething\n' > "$1/analysis/src/knowledge-base/share/KB.md" + printf '# KB_a\n\nsomething\n' > "$1/analysis/src/knowledge-base/share/KB_a.md" +} +code_kb() { # a code-extracted knowledge base (merger output + extracted/) + local kb="$1/analysis/src/knowledge-base" + mkdir -p "$kb/extracted" + printf '[{"name":"Order","module":"Sales"}]\n' > "$kb/entities.json" + printf '[{"name":"ACT_Save","module":"Sales"}]\n' > "$kb/logics.json" + printf '[{"name":"OrderEdit","module":"Sales"}]\n' > "$kb/screens.json" + printf '{"items":[{"type":"entity","name":"Order"},{"type":"logic","name":"ACT_Save"},{"type":"screen","name":"OrderEdit"}]}\n' > "$kb/extracted/a.json" +} +run() { bash "$SUT" "$1" "${@:2}" 2>&1; } +jget() { "$PY" -c 'import json,sys +d=json.loads(sys.stdin.read()); b=d["bases"][0] +print(eval(sys.argv[1]))' "$1"; } + +# --- Q1 ------------------------------------------------------------------------------------- +P1="$WORK/p1"; doc_kb "$P1" +J="$(run "$P1" --json)" +[ "$(printf '%s' "$J" | jget 'b["sections"].get("extractionQuality")')" = skipped ] \ + && ok "Q1 documents KB, no file → skipped" || bad "Q1 documents KB, no file → skipped" "$J" +[ "$(printf '%s' "$J" | jget 'b["verdicts"]["fault"]')" = 0 ] && ok "Q1 no new fault" || bad "Q1 no new fault" "$J" + +# --- Q2 ------------------------------------------------------------------------------------- +P2="$WORK/p2"; doc_kb "$P2"; cp "$EXAMPLE" "$P2/analysis/src/extraction-quality.json" +J="$(run "$P2" --json)" +[ "$(printf '%s' "$J" | jget 'len(b["extractionQuality"][0]["rows"])')" = 6 ] && ok "Q2 six rows" || bad "Q2 six rows" "$J" +[ "$(printf '%s' "$J" | jget 'b["extractionQuality"][0]["verdict"]')" = manual ] && ok "Q2 not passed → manual" || bad "Q2 not passed → manual" "$J" +[ "$(printf '%s' "$J" | jget 'b["sections"]["quality:analysis/src/extraction-quality.json"]')" = manual ] \ + && ok "Q2 section keyed by file" || bad "Q2 section keyed by file" "$J" +H="$(cat "$P2/analysis/src/knowledge-base/extraction-report.html")" +has "Q2 html section" "$H" "

    Extraction quality

    " +has "Q2 html dimension key" "$H" ">entityFields<" +has "Q2 html found/expected" "$H" "49 / 50" +has "Q2 html gap text" "$H" "User.avatar missing" +has "Q2 html overall" "$H" "Overall 94.2 against threshold 95" +has "Q2 html confirmed zero" "$H" "ts-ast, python-regex" +O="$(run "$P2")" +has "Q2 prose line" "$O" "quality analysis/src/extraction-quality.json: overall 94.2 / threshold 95, passed=False, selfGraded=no, 6 dimension(s)" + +# --- Q3 ------------------------------------------------------------------------------------- +P3="$WORK/p3"; doc_kb "$P3"; mkdir -p "$P3/analysis/src/extracted" +mutate "$EXAMPLE" "$P3/analysis/src/extracted/extraction-quality-procmodel.json" ' +d["dimensions"]={k:{"score":100,"found":n,"expected":n,"gaps":[]} for k,n in + (("nodes",41),("transitions",57),("guards",12),("documentation",41),("diagramMembership",41),("labelRecall",88),("images",3))} +d["overallScore"]=100; d["passed"]=True; d["selfGraded"]=True; d["confirmedZeros"]=[]' +J="$(run "$P3" --json)" +[ "$(printf '%s' "$J" | jget 'b["extractionQuality"][0]["verdict"]')" = fault ] && ok "Q3 self-graded pass → fault" || bad "Q3 self-graded pass → fault" "$J" +[ "$(printf '%s' "$J" | jget 'len(b["extractionQuality"][0]["rows"])')" = 7 ] && ok "Q3 the file's own 7 dimensions" || bad "Q3 the file's own 7 dimensions" "$J" +O="$(run "$P3")" +has "Q3 prose FAULT names it" "$O" "self-graded pass" +H="$(cat "$P3/analysis/src/knowledge-base/extraction-report.html")" +has "Q3 html stack's own key" "$H" ">labelRecall<" + +# --- Q4 / Q5 -------------------------------------------------------------------------------- +P4="$WORK/p4"; code_kb "$P4" +J="$(run "$P4" --json)" +[ "$(printf '%s' "$J" | jget 'b["sections"].get("extractionQuality")')" = fault ] \ + && ok "Q4 code-extracted, no file → fault" || bad "Q4 code-extracted, no file → fault" "$J" +cp "$EXAMPLE" "$P4/analysis/src/knowledge-base/extracted/extraction-quality.json" +J="$(run "$P4" --json)" +hasnt "Q5 score file not read as an item array" "$J" "would not parse" +[ "$(printf '%s' "$J" | jget 'b["extractionQuality"][0]["file"]')" = "analysis/src/knowledge-base/extracted/extraction-quality.json" ] \ + && ok "Q5 found inside /extracted" || bad "Q5 found inside /extracted" "$J" + +# --- Q6 ------------------------------------------------------------------------------------- +P6="$WORK/p6"; doc_kb "$P6" +mutate "$EXAMPLE" "$P6/analysis/src/extraction-quality.json" ' +d["dimensions"]["enumerations"].update(found=0, expected=0, score=100) +d["dimensions"]["endpoints"].update(found=0, expected=0, score=100, gaps=[])' +J="$(run "$P6" --json)" +[ "$(printf '%s' "$J" | jget '[r["flag"] for r in b["extractionQuality"][0]["rows"] if r["dimension"]=="endpoints"][0]')" = "unconfirmed zero" ] \ + && ok "Q6 unconfirmed 0/0 flagged" || bad "Q6 unconfirmed 0/0 flagged" "$J" +[ "$(printf '%s' "$J" | jget '[r["flag"] for r in b["extractionQuality"][0]["rows"] if r["dimension"]=="enumerations"][0]')" = "" ] \ + && ok "Q6 confirmed 0/0 not flagged" || bad "Q6 confirmed 0/0 not flagged" "$J" +[ "$(printf '%s' "$J" | jget 'b["extractionQuality"][0]["verdict"]')" = fault ] && ok "Q6 → fault" || bad "Q6 → fault" "$J" + +# --- Q7 ------------------------------------------------------------------------------------- +P7="$WORK/p7"; doc_kb "$P7"; mkdir -p "$P7/analysis/src/extracted" +mutate "$EXAMPLE" "$P7/analysis/src/extraction-quality.json" 'd["passed"]=True' +printf '{"overallScore": 9' > "$P7/analysis/src/extracted/extraction-quality-broken.json" +J="$(run "$P7" --json)"; RC=$? +[ "$RC" = 0 ] && ok "Q7 exit 0 (a fault is information)" || bad "Q7 exit 0" "rc=$RC" +[ "$(printf '%s' "$J" | jget 'sorted(q["verdict"] for q in b["extractionQuality"])')" = "['fault', 'fault']" ] \ + && ok "Q7 under-threshold pass and broken file both fault" || bad "Q7 both fault" "$J" +has "Q7 contradiction named" "$J" "under the file's own threshold 95" +[ -s "$P7/analysis/src/knowledge-base/extraction-report.html" ] && ok "Q7 page still rendered" || bad "Q7 page still rendered" + +echo "PASS=$PASS FAIL=$FAIL" +[ "$FAIL" = 0 ] diff --git a/tests/wave2/test-token-burn.sh b/tests/wave2/test-token-burn.sh index f25bfab7..f815b083 100755 --- a/tests/wave2/test-token-burn.sh +++ b/tests/wave2/test-token-burn.sh @@ -4,8 +4,8 @@ # scrubbed, and the expected sums computed independently of the instrument. # # T1 no transcript tree → NOT AVAILABLE in brief, screen and json (available:false), exit 0 -# T2 --json: per-model arrays to the token; 52 records dedupe to 22 messages; 1 synthetic -# skipped; 2 files; subagent headline 59858 +# T2 --json: per-model arrays to the token, usage from each id's LAST record; 52 records dedupe to 22 messages; 1 synthetic +# skipped; 2 files; subagent headline 61357 # T3 the two-tree layout: records whose cwd is /app count — moving them out of the # project drops the message count from 22 to 21 # T4 --brief: the exact "Tokens this stage:" line for stage 2 @@ -13,6 +13,14 @@ # T6 no PROJECT.md → every day UNMAPPED; brief prints the project total and says why # T7 a foreign slug dir only → NOT AVAILABLE with the --scan-all hint; --scan-all finds it # T8 the ancestor slug dir (session launched in the parent) is read by default +# --- #296: opt-in cost / agent time / human prompts / windows --- +# T9 without the new flags nothing new appears (screen + json; T1-T8 pin the rest) +# T10 --rates: per-Mtok cost from a rates file, main vs subagent, a `*` catch-all +# T11 --rates with no line for a model: INCOMPLETE, total marked `+`, complete:false — never 0 +# T12 --effort: agent working time and human prompt count over the captured main thread plus +# effort/s-3.jsonl (every user-record class); --gap changes the cut-off +# T13 --windows: per-window messages, headline, agent time, prompts +# T14 malformed --rates / --windows file → exit 1 with the line number # # tests/wave2/test-token-burn.sh /path/to/token-burn.sh @@ -63,12 +71,13 @@ has "T1 json available:false" "$OUT" '"available": false' # per-model array can be pinned as one literal (no python here — check-portability) J="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --json 2>&1 | tr -d '\n' | tr -s ' ' | sed 's/\[ /[/g; s/ \]/]/g')" has "T2 fable-5-1 array" "$J" '"claude-fable-5-1": [288, 31365, 2607781, 15524, 9]' -has "T2 sonnet-5 array" "$J" '"claude-sonnet-5": [26, 59772, 998476, 60, 13]' +# the subagent model's array, matched by its numbers (the model id is the captured one, see CAPTURE.md) +echo "$J" | grep -Eq '"claude-[a-z]+-5": \[26, 59772, 998476, 1559, 13\]' && ok "T2 subagent-model array (LAST record per id: output 1559, the first record says 60)" || bad "T2 subagent-model array" "$J" has "T2 22 messages" "$J" '"messages": 22' has "T2 52 records (dedupe)" "$J" '"records": 52' has "T2 1 synthetic" "$J" '"synthetic_skipped": 1' has "T2 2 files" "$J" '"files": 2' -has "T2 subagent headline" "$J" '"subagents": 59858' +has "T2 subagent headline (59858 when the first record was kept)" "$J" '"subagents": 61357' has "T2 main headline" "$J" '"main": 47177' has "T2 current stage" "$J" '"current_stage": "2"' @@ -81,7 +90,7 @@ has "T3 app/ records counted (22 -> 21 once moved out of the root)" "$J3" '"mess # --- T4: brief ----------------------------------------------------------------------------- B="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --brief 2>&1)" -[ "$B" = "Tokens this stage: 69k (sonnet-5 87% · fable-5-1 13% · 1438k cache-read · 2 days · stage 2)" ] \ +printf '%s\n' "$B" | grep -Eqx 'Tokens this stage: 70k \([a-z]+-5 88% · fable-5-1 12% · 1438k cache-read · 2 days · stage 2\)' \ && ok "T4 brief line exact" || bad "T4 brief line exact" "$B" # --- T5: screen ----------------------------------------------------------------------------- @@ -89,10 +98,10 @@ S="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" 2>&1)" has "T5 synthetic skipped" "$S" "1 synthetic skipped" has "T5 22 messages from 52 records" "$S" "22 messages (from 52 records" has "T5 current stage line" "$S" "register says current stage 2" -has "T5 subagent share" "$S" "subagents (Agent tool): 60k of headline (56%)" +has "T5 subagent share" "$S" "subagents (Agent tool): 61k of headline (57%)" # the caveat line legitimately explains what UNMAPPED means; no day or stage row may carry it hasnt "T5 no UNMAPPED row" "$(echo "$S" | grep -v '^ caveat:')" "UNMAPPED" -echo "$S" | grep -Eq '^ *2 +2 +69k +1438k +17 ' && ok "T5 stage-2 row" || bad "T5 stage-2 row" "$(echo "$S" | grep -E '^ *2 ' | head -3)" +echo "$S" | grep -Eq '^ *2 +2 +70k +1438k +17 ' && ok "T5 stage-2 row" || bad "T5 stage-2 row" "$(echo "$S" | grep -E '^ *2 ' | head -3)" echo "$S" | grep -Eq '^ *1 +1 +39k +2168k +5 ' && ok "T5 stage-1 row" || bad "T5 stage-1 row" "$(echo "$S" | grep -E '^ *1 ' | head -3)" # --- T6: no PROJECT.md ---------------------------------------------------------------------- @@ -103,7 +112,7 @@ sed "s#/srv/example/proj#$P6#g" "$FIX/transcripts/s-1/subagents/agent-1.jsonl" > S6="$(CLAUDE_CONFIG_DIR="$CFG6" bash "$SUT" "$P6" 2>&1)" has "T6 screen UNMAPPED" "$S6" "UNMAPPED" B6="$(CLAUDE_CONFIG_DIR="$CFG6" bash "$SUT" "$P6" --brief 2>&1)" -has "T6 brief falls back to the project total" "$B6" "Tokens this stage: 107k (" +has "T6 brief falls back to the project total" "$B6" "Tokens this stage: 109k (" has "T6 brief says why" "$B6" "no Current stage line in PROJECT.md" # --- T7: foreign slug only ------------------------------------------------------------------ @@ -119,5 +128,73 @@ CFG8="$WORK/claude8"; install_tree "$CFG8" "$(slug "$WORK")" J8="$(CLAUDE_CONFIG_DIR="$CFG8" bash "$SUT" "$P" --json 2>&1)" has "T8 ancestor dir read by default" "$J8" '"messages": 22' +# --- #296 ---------------------------------------------------------------------------------- +fold() { tr -d '\n' | tr -s ' ' | sed 's/\[ /[/g; s/ \]/]/g'; } + +# T9: unchanged without the flags +hasnt "T9 screen has no COST section" "$S" "COST (" +hasnt "T9 screen has no agent time" "$S" "agent working time" +hasnt "T9 screen has no WINDOWS section" "$S" "WINDOWS (" +hasnt "T9 json has no cost key" "$J" '"cost":' +hasnt "T9 json has no effort key" "$J" '"effort":' +hasnt "T9 json has no windows key" "$J" '"windows":' + +# T10: rates (made-up prices — the instrument carries none of its own) +RATES="$WORK/rates.txt" +printf '%s\n' '# test rates, per Mtok' 'currency EUR' 'claude-fable-* 2.00 2.50 0.20 10.00' '* 1.00 1.00 0.10 5.00' > "$RATES" +J10="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --json --rates "$RATES" 2>&1 | fold)" +has "T10 total 0.9232" "$J10" '"total": 0.9232' +has "T10 main 0.7558" "$J10" '"main": 0.7558' +has "T10 subagents 0.1674" "$J10" '"subagents": 0.1674' +has "T10 complete" "$J10" '"complete": true' +has "T10 currency" "$J10" '"currency": "EUR"' +has "T10 tokens unchanged" "$J10" '"messages": 22' +S10="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --rates "$RATES" 2>&1)" +echo "$S10" | grep -Eq '^TOTAL +0\.92$' && ok "T10 screen total" || bad "T10 screen total" "$(echo "$S10" | grep '^TOTAL')" +has "T10 subagent cost line" "$S10" "subagents (Agent tool): 0.17 EUR of 0.92" +B10="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --brief --rates "$RATES" 2>&1)" +[ "$B10" = "$B" ] && ok "T10 --brief unchanged by --rates" || bad "T10 --brief unchanged by --rates" "$B10" + +# T11: a model with no rate line +R11="$WORK/rates-partial.txt"; printf '%s\n' 'claude-fable-* 2.00 2.50 0.20 10.00' > "$R11" +J11="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --json --rates "$R11" 2>&1 | fold)" +has "T11 complete:false" "$J11" '"complete": false' +has "T11 priced part only" "$J11" '"total": 0.7558' +S11="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --rates "$R11" 2>&1)" +has "T11 INCOMPLETE named" "$S11" "INCOMPLETE — no rate line matches" +echo "$S11" | grep -Eq '^TOTAL +0\.76\+$' && ok "T11 total marked +" || bad "T11 total marked +" "$(echo "$S11" | grep '^TOTAL')" + +# T12: effort — captured main thread + the user-record classes in effort/s-3.jsonl +CFG12="$WORK/claude12"; install_tree "$CFG12" "$(slug "$P")" +sed "s#/srv/example/proj#$P#g" "$FIX/effort/s-3.jsonl" > "$CFG12/projects/$(slug "$P")/s-3.jsonl" +J12="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --json --effort 2>&1 | fold)" +has "T12 4 human prompts (dup uuid once; meta, notification, peer, compact, tool result, , sidechain, foreign cwd excluded)" "$J12" '"human_prompts": 4' +has "T12 agent seconds 591 (220 captured + 371)" "$J12" '"agent_seconds": 591' +S12="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --effort 2>&1)" +has "T12 screen line" "$S12" "agent working time: 0h 10m (main thread, gaps <= 5 min, gaps ending at a human prompt excluded) · human prompts: 4" +J12g="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --json --gap 1 2>&1 | fold)" +has "T12 --gap 1 cuts the 114 s gap (477)" "$J12g" '"agent_seconds": 477' + +# T13: windows +WIN="$WORK/windows.txt" +printf '%s\n' '# round boundaries' '2026-09-01 setup' '2026-09-17 round 1' '2026-09-18T02:00 wrap-up' > "$WIN" +S13="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --windows "$WIN" --rates "$RATES" 2>&1)" +echo "$S13" | grep -Eq '^setup +2026-09-01 +5 +39k +2168k +[0-9.]+ +0h 03m +0$' && ok "T13 setup row" || bad "T13 setup row" "$(echo "$S13" | grep '^setup')" +echo "$S13" | grep -Eq '^round 1 +2026-09-17 +13 +61k +998k +[0-9.]+ +0h 00m +0$' && ok "T13 round 1 row" || bad "T13 round 1 row" "$(echo "$S13" | grep '^round 1')" +echo "$S13" | grep -Eq '^wrap-up +2026-09-18T02:00 +10 +11k +[0-9]+k +[0-9.]+ +0h 07m +4$' && ok "T13 wrap-up row" || bad "T13 wrap-up row" "$(echo "$S13" | grep '^wrap-up')" +hasnt "T13 no empty before-first row" "$S13" "(before first window)" +J13="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --json --windows "$WIN" 2>&1 | fold)" +has "T13 json order" "$J13" '"order": ["setup", "round 1", "wrap-up"]' +has "T13 json wrap-up agent seconds 427" "$J13" '"agent_seconds": 427' + +# T14: malformed config files are usage errors +printf '%s\n' 'claude-fable-* 2 2.5' > "$WORK/bad-rates.txt" +OUT="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --rates "$WORK/bad-rates.txt" 2>&1)"; RC=$? +[ "$RC" = 1 ] && ok "T14 bad rates exit 1" || bad "T14 bad rates exit 1" "rc=$RC" +has "T14 bad rates names the line" "$OUT" "bad-rates.txt:1:" +printf '%s\n' 'next tuesday round 2' > "$WORK/bad-win.txt" +OUT="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --windows "$WORK/bad-win.txt" 2>&1)"; RC=$? +[ "$RC" = 1 ] && ok "T14 bad windows exit 1" || bad "T14 bad windows exit 1" "rc=$RC" + echo "PASS=$PASS FAIL=$FAIL" [ "$FAIL" = 0 ] From 1a7399632c69a0bea70627e08e18a72695a3a8ba Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 15:28:37 +0000 Subject: [PATCH 2/3] Review fixes: missing score file is manual; usage max per message id MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - extraction-report.sh: a code-extracted/mixed KB with no extraction-quality*.json is MANUAL ("not produced yet — no shipped pipeline writes a validator"), not a fault: nothing mandatory produces the file. Present-but-wrong files stay faults. - token-burn.sh: per message.id take each usage counter's max over its records, so a later record with empty/zero usage cannot erase real numbers (fixture case added: fails on the previous commit). Window-name list renamed. - extractor-quality-loop.md: comment that a fixture parses the example JSON. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 4 +-- bin/extraction-report.sh | 11 ++++--- bin/token-burn.sh | 31 ++++++++++--------- skills/extractor-quality-loop.md | 2 ++ tests/wave2/fixtures/token-burn/CAPTURE.md | 5 ++- .../fixtures/token-burn/effort/s-3.jsonl | 1 + tests/wave2/test-extraction-quality-report.sh | 9 ++++-- tests/wave2/test-token-burn.sh | 4 ++- 8 files changed, 42 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1764be41..bc0a9d84 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,8 +17,8 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. ## Unreleased -- fix(token-burn): **usage is taken from each `message.id`'s LAST record, not its first — subagent output was undercounted.** The header claimed every streamed record of a message carries the same usage; in subagent transcripts the early records carry a partial `output_tokens`. On the captured fixture the subagent model's output went 60 → 1,559 (headline 59,858 → 61,357); on a 2026-10 multi-agent session 37,791 → 84,915 subagent output tokens; a field measurement on harness 2.1.296 saw up to 7×. Main-thread records were identical first to last, so main numbers do not move. **Every existing subagent-heavy token count rises after sync**; fixture expectations re-derived independently (`tests/wave2/fixtures/token-burn/CAPTURE.md`) — found while promoting [#171](https://github.com/MendixMau/mxcli-project-toolkit/issues/171) (measured-claims §7), MendixMau -- new(extraction-report): **the Stage 1 surface now reads the validator's `extraction-quality*.json` — one row per dimension the file declares, a self-graded pass shown as a fault** ([#151](https://github.com/MendixMau/mxcli-project-toolkit/issues/151)). `extractor-quality-loop.md` makes that file the proof of the 95% gate, yet nothing read it: a field build scored its process-model extractor at 100% on 7 dimensions and the report showed "2 pass, 18 skipped" with no quality section. Found at `analysis//`, `analysis//extracted/` and the same two inside the KB; dimension keys come from the file. Faults only where the skill's rules show in the file: `selfGraded` + `passed`, `passed` under its own `threshold`, a 0/0 dimension absent from `confirmedZeros`, unparseable JSON. **Behaviour change for existing projects:** a code-extracted or mixed KB with no score file now carries one `fault` section (migration-pipeline.md does not start BRDs before the 95% gate, so an unscored extraction is unmeasured); documents KBs get `skipped`, the gate (`gate-check.sh`, presence of the page) and the exit code are unchanged. A score file inside `/extracted/` is no longer misread as an unparseable item file. No captured validator output exists in this repo (no pipeline ships a validator), so the fixture reads the skill's documented example at run time; `tests/wave2/test-extraction-quality-report.sh` — MendixMau +- fix(token-burn): **usage per `message.id` is each counter's max over its records (in practice the last), not the first record's — subagent output was undercounted; a late empty record cannot erase real numbers.** The header claimed every streamed record of a message carries the same usage; in subagent transcripts the early records carry a partial `output_tokens`. On the captured fixture the subagent model's output went 60 → 1,559 (headline 59,858 → 61,357); on a 2026-10 multi-agent session 37,791 → 84,915 subagent output tokens; a field measurement on harness 2.1.296 saw up to 7×. Main-thread records were identical first to last, so main numbers do not move. **Every existing subagent-heavy token count rises after sync**; fixture expectations re-derived independently (`tests/wave2/fixtures/token-burn/CAPTURE.md`) — found while promoting [#171](https://github.com/MendixMau/mxcli-project-toolkit/issues/171) (measured-claims §7), MendixMau +- new(extraction-report): **the Stage 1 surface now reads the validator's `extraction-quality*.json` — one row per dimension the file declares, a self-graded pass shown as a fault** ([#151](https://github.com/MendixMau/mxcli-project-toolkit/issues/151)). `extractor-quality-loop.md` makes that file the proof of the 95% gate, yet nothing read it: a field build scored its process-model extractor at 100% on 7 dimensions and the report showed "2 pass, 18 skipped" with no quality section. Found at `analysis//`, `analysis//extracted/` and the same two inside the KB; dimension keys come from the file. Faults only where the skill's rules show in the file: `selfGraded` + `passed`, `passed` under its own `threshold`, a 0/0 dimension absent from `confirmedZeros`, unparseable JSON. A code-extracted or mixed KB with no score file shows a `manual` section ("not produced yet — no shipped pipeline writes a validator"), not a fault: nothing mandatory produces the file yet; documents KBs get `skipped`; the gate (`gate-check.sh`, presence of the page) and the exit code are unchanged. A score file inside `/extracted/` is no longer misread as an unparseable item file. No captured validator output exists in this repo (no pipeline ships a validator), so the fixture reads the skill's documented example at run time; `tests/wave2/test-extraction-quality-report.sh` — MendixMau - new(token-burn): **`--rates`, `--windows`, `--effort`: cost at the project's own per-Mtok rates, agent working time, human prompt count, per-window attribution — aggregates only** ([#296](https://github.com/MendixMau/mxcli-project-toolkit/issues/296)). A field team had to answer "money and time per round and per feature" with ad-hoc transcript scripts. No price is built in (rates file, model globs, first match wins; an unmatched model is UNPRICED and the total marked incomplete, never 0); agent time = main-thread gaps <= `--gap` minutes (default 5) not ending at a human prompt; human prompts = `origin.kind == "human"` user records (task notifications, peer messages, tool results, meta and compact summaries excluded), once per uuid, text never printed. Without the new flags the new code adds nothing to screen, `--json` or `--brief` output (byte-identical to the previous script before the last-record fix below; fixture T9 asserts no new section or key). Field run on a real multi-day transcript: human prompts matched an independent count (89 = 89). The human-prompt fixture records are hand-built from a profile of real record shapes — committing a scrubbed real capture was refused; `tests/wave2/test-token-burn.sh` T9-T14 — jopterhorst - learn(testing-shape, test-result-audit): **three test-strength rules, each with a denominator: refusals probed on the server, every undeclared 4xx a failure, one clean-database final run** ([#172](https://github.com/MendixMau/mxcli-project-toolkit/issues/172)). A field comparison of two builds against one frozen spec found the toolkit build checked permission refusals only in the client (a hidden menu item), logged unexpected 400/422 responses as information, and reached 20/20 by stitching three runs. `testing-shape.md` §7 gains three report lines (`N of N restricted pairs probed server-side`, `undeclared 4xx/5xx: 0 of M requests — each fails its step`, `one clean-DB run : N of N scenarios incl. timers, not stitched`); `test-result-audit.md` gains check 2e, which carries the reasons and the field evidence and asks for those numbers. Both baseline skills stay inside the 80,000-word baseline budget (79,997): the reasons live in the on-demand audit skill, and two redundant sentences in `testing-shape.md` §4's verdict paragraphs were folded (the empty-set corollary is stated in the Trace row, `harness-architecture.md` and `report-schema.md`). The backing inbox note is trimmed to its unpromoted "Keep" items. — MendixMau - learn(learned-detection-gaps, module-review): **two UI defects every model gate passes: a lazy-loaded combo box filter hides its preset, and a theme that pins row heights by position lets the next row cover an inserted widget** ([#299](https://github.com/MendixMau/mxcli-project-toolkit/issues/299)). Both passed check, mxbuild, lint and the scripted checks on a 127-module production app and were found only in the running app. Two register rows with the first oracle and the pre-build check (pluggable properties that change *when* data loads; grep the theme for fixed `height` on positional row classes before an insert), and two LOOK lines in `module-review.md` §4c with denominators (`N of N filters show their preset on first load`, `N of N widgets inserted into an existing container fully visible`). — jopterhorst diff --git a/bin/extraction-report.sh b/bin/extraction-report.sh index 7117843b..62ea64a0 100755 --- a/bin/extraction-report.sh +++ b/bin/extraction-report.sh @@ -557,10 +557,13 @@ def analyse(kb): v = ('skipped', 'no extraction-quality*.json, and none is owed: a documents knowledge ' 'base ran no extractor, so there is nothing for a validator to score.') elif prov in ('code-extracted', 'mixed'): - v = ('fault', f'an extractor produced this knowledge base ({prov}) but no ' - f'extraction-quality*.json was found ({QLOC}). extractor-quality-loop.md ' - f'makes that file the proof of the >=95% gate, and migration-pipeline.md ' - f'does not start BRDs before it — an unmeasured extraction, not a pass.') + # MANUAL, not fault: no shipped pipeline writes a validator, so nothing mandatory + # produces this file (CLAUDE.md "a producer for every consumer"). A present-but-wrong + # file stays a fault above. + v = ('manual', f'not produced yet — no shipped pipeline writes a validator; build one ' + f'per extractor-quality-loop.md, or waive the gate. An extractor produced ' + f'this knowledge base ({prov}) and no extraction-quality*.json was found ' + f'({QLOC}), so its quality is unmeasured, not passed.') else: v = ('manual', 'provenance undetermined — whether an extractor ran, and so whether a ' 'score file is owed, needs a human.') diff --git a/bin/token-burn.sh b/bin/token-burn.sh index cc89e643..5a26321f 100755 --- a/bin/token-burn.sh +++ b/bin/token-burn.sh @@ -25,8 +25,9 @@ # (2026-09-18, a 17,000-line toolkit session) and each one a silent miscount if ignored: # 1. The harness writes ONE RECORD PER STREAMED CONTENT BLOCK, all carrying the same # `message.id` — that session had 4,336 assistant records for 1,989 messages, so summing -# records overcounts ~2×. Usage is summed ONCE per `message.id`, from that id's LAST -# record: the usage is NOT the same on every record. In subagent transcripts the early +# records overcounts ~2×. Usage is summed ONCE per `message.id`, taking each counter's +# MAX over that id's records (in practice the last record; a later record with an empty or +# zero usage cannot erase real numbers): the usage is NOT the same on every record. In subagent transcripts the early # records carry a partial `output_tokens` (the captured fixture: 60 on the first record, # 1,559 on the last; a 2026-10 multi-agent session: 37,791 vs 84,915 subagent output # tokens, first vs last). Keeping the first record undercounted subagent output by up to @@ -245,7 +246,7 @@ day_model = defaultdict(lambda: defaultdict(int)) # day -> model -> headline sidechain = [0, 0] # headline tokens: main, subagent n_files = n_records = n_synthetic = 0 msgs = [] # (epoch, model, vals, is_sidechain) per message -last, order = {}, [] # message.id -> [first ts, model, LAST usage, sidechain] +last, msg_order = {}, [] # message.id -> [first ts, model, max usage, sidechain] human_seen = set() humans = [] # epoch per human prompt timeline = defaultdict(list) # main-thread file -> [(epoch, is_human)] @@ -298,14 +299,16 @@ for d in dirs: u = msg.get("usage") or {} vals = [int(u.get(k) or 0) for k in KEYS] if mid in seen: - last[mid][2] = vals # later record of the same message: its usage wins + # later record of the same message: per-field max, so a later record's + # larger (final) count wins but an empty/zero usage never erases real numbers + last[mid][2] = [max(x, y) for x, y in zip(last[mid][2], vals)] continue seen.add(mid) - order.append(mid) + msg_order.append(mid) last[mid] = [rec.get("timestamp") or "", model, vals, bool(rec.get("isSidechain"))] -# one entry per message.id: first record's time, LAST record's usage (see header, fact 1) -for mid in order: +# one entry per message.id: first record's time, per-field max over its records (header, fact 1) +for mid in msg_order: ts, model, vals, side = last[mid] day = ts[:10] or "????-??-??" for i, v in enumerate(vals): @@ -431,12 +434,12 @@ if want_effort: win = None if windows: names = [BEFORE] + [w for _t, w, _r in windows] - order = [] + win_names = [] for n in names: - if n not in order: - order.append(n) + if n not in win_names: + win_names.append(n) win = {n: {"messages": 0, "usage": [0, 0, 0, 0], "cost": 0.0 if rates_file else None, - "unpriced": 0, "agent_seconds": 0, "human_prompts": 0} for n in order} + "unpriced": 0, "agent_seconds": 0, "human_prompts": 0} for n in win_names} starts = {} for t, n, raw in windows: starts.setdefault(n, raw) @@ -453,15 +456,15 @@ if windows: w["cost"] += c for t in humans: win[window_for(t)]["human_prompts"] += 1 - for n in order: + for n in win_names: win[n]["agent_seconds"] = round(agent_seconds(lambda t, n=n: window_for(t) == n)) if win[n]["cost"] is not None: win[n]["cost"] = round(win[n]["cost"], 4) win[n]["start"] = starts.get(n) if not any(win[BEFORE][k] for k in ("messages", "human_prompts", "agent_seconds")): del win[BEFORE] - order.remove(BEFORE) - win = {"order": order, "windows": win} + win_names.remove(BEFORE) + win = {"order": win_names, "windows": win} # --- output ------------------------------------------------------------------------------- if mode == "json": diff --git a/skills/extractor-quality-loop.md b/skills/extractor-quality-loop.md index ae5ca148..bd45db53 100644 --- a/skills/extractor-quality-loop.md +++ b/skills/extractor-quality-loop.md @@ -151,6 +151,8 @@ Noting these as "BRD-only" in the quality report is correct and expected. ## Output Format: extraction-quality.json + + ```json { "project": "ProjectName", diff --git a/tests/wave2/fixtures/token-burn/CAPTURE.md b/tests/wave2/fixtures/token-burn/CAPTURE.md index 4ea125bc..1b9f619e 100644 --- a/tests/wave2/fixtures/token-burn/CAPTURE.md +++ b/tests/wave2/fixtures/token-burn/CAPTURE.md @@ -91,7 +91,10 @@ The facts it encodes, each seen in that profile: wrappers whose text starts with `<`); - a human prompt record can be written twice with the same `uuid`. -Plus one record whose `cwd` is outside the project and one `isSidechain: true` record. +Plus one record whose `cwd` is outside the project, one `isSidechain: true` record, and a +second `msg_e1` record at the same timestamp with all-zero usage (the instrument takes each +counter's max per message id, so a late empty record cannot erase real numbers; day +2026-09-20 totals stay 121 · 1500 · 30000 · 286 over 6 messages). Field proof is a run, not this file: on the real transcript profiled above the instrument counted the same number of human prompts (89) as an independent count of `origin.kind == "human"`, non-meta, non-compact records. diff --git a/tests/wave2/fixtures/token-burn/effort/s-3.jsonl b/tests/wave2/fixtures/token-burn/effort/s-3.jsonl index 276bdf14..da1187c8 100644 --- a/tests/wave2/fixtures/token-burn/effort/s-3.jsonl +++ b/tests/wave2/fixtures/token-burn/effort/s-3.jsonl @@ -1,5 +1,6 @@ {"type":"user","sessionId":"s-3","uuid":"h-1","parentUuid":null,"isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:00:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"human"},"promptSource":"sdk","message":{"role":"user","content":"[scrubbed]"}} {"type":"assistant","sessionId":"s-3","uuid":"a-1","parentUuid":"h-1","isSidechain":false,"timestamp":"2026-09-20T10:00:05.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e1","type":"message","role":"assistant","content":[{"type":"tool_use","id":"tu-1","name":"Tool","input":{}}],"usage":{"input_tokens":100,"cache_creation_input_tokens":1000,"cache_read_input_tokens":10000,"output_tokens":50}}} +{"type":"assistant","sessionId":"s-3","uuid":"a-1b","parentUuid":"a-1","isSidechain":false,"timestamp":"2026-09-20T10:00:05.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e1","type":"message","role":"assistant","content":[{"type":"text","text":"[scrubbed]"}],"usage":{"input_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":0}}} {"type":"user","sessionId":"s-3","uuid":"t-1","parentUuid":"a-1","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:00:06.000Z","cwd":"/srv/example/proj","toolUseResult":"[scrubbed]","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"tu-1","content":"[scrubbed]"}]}} {"type":"assistant","sessionId":"s-3","uuid":"a-2","parentUuid":"t-1","isSidechain":false,"timestamp":"2026-09-20T10:01:06.000Z","cwd":"/srv/example/proj","message":{"model":"claude-fable-5-1","id":"msg_e2","type":"message","role":"assistant","content":[{"type":"text","text":"[scrubbed]"}],"usage":{"input_tokens":10,"cache_creation_input_tokens":0,"cache_read_input_tokens":20000,"output_tokens":200}}} {"type":"user","sessionId":"s-3","uuid":"n-1","parentUuid":"a-2","isSidechain":false,"userType":"external","timestamp":"2026-09-20T10:03:00.000Z","cwd":"/srv/example/proj","origin":{"kind":"task-notification"},"promptSource":"system","message":{"role":"user","content":""}} diff --git a/tests/wave2/test-extraction-quality-report.sh b/tests/wave2/test-extraction-quality-report.sh index 7da2f4e7..4a4bf375 100755 --- a/tests/wave2/test-extraction-quality-report.sh +++ b/tests/wave2/test-extraction-quality-report.sh @@ -14,7 +14,8 @@ # passed → `manual`; figures on the page (score, found/expected, gaps, confirmed zero) # Q3 analysis//extracted/extraction-quality-.json with the stack's OWN dimension # keys, selfGraded + passed → `fault` "self-graded pass" -# Q4 code-extracted KB, no score file → `fault` (the loop is mandatory for an extractor) +# Q4 code-extracted KB, no score file → `manual` "not produced yet" (no shipped pipeline +# writes a validator, so nothing mandatory produces it — never green, never a fault) # Q5 the score file inside /extracted/ is not counted as an unparseable item file # Q6 a 0/0 dimension not in confirmedZeros → fault; the example's confirmed one is not # Q7 passed:true under its own threshold → fault; broken JSON → fault, page still rendered @@ -110,8 +111,10 @@ has "Q3 html stack's own key" "$H" ">labelRecall<" # --- Q4 / Q5 -------------------------------------------------------------------------------- P4="$WORK/p4"; code_kb "$P4" J="$(run "$P4" --json)" -[ "$(printf '%s' "$J" | jget 'b["sections"].get("extractionQuality")')" = fault ] \ - && ok "Q4 code-extracted, no file → fault" || bad "Q4 code-extracted, no file → fault" "$J" +[ "$(printf '%s' "$J" | jget 'b["sections"].get("extractionQuality")')" = manual ] \ + && ok "Q4 code-extracted, no file → manual" || bad "Q4 code-extracted, no file → manual" "$J" +O="$(run "$P4")" +has "Q4 says not produced yet" "$O" "not produced yet — no shipped pipeline writes a validator" cp "$EXAMPLE" "$P4/analysis/src/knowledge-base/extracted/extraction-quality.json" J="$(run "$P4" --json)" hasnt "Q5 score file not read as an item array" "$J" "would not parse" diff --git a/tests/wave2/test-token-burn.sh b/tests/wave2/test-token-burn.sh index f815b083..19feb055 100755 --- a/tests/wave2/test-token-burn.sh +++ b/tests/wave2/test-token-burn.sh @@ -18,7 +18,8 @@ # T10 --rates: per-Mtok cost from a rates file, main vs subagent, a `*` catch-all # T11 --rates with no line for a model: INCOMPLETE, total marked `+`, complete:false — never 0 # T12 --effort: agent working time and human prompt count over the captured main thread plus -# effort/s-3.jsonl (every user-record class); --gap changes the cut-off +# effort/s-3.jsonl (every user-record class); --gap changes the cut-off; a later +# record with zero usage for the same message.id leaves the real numbers intact # T13 --windows: per-window messages, headline, agent time, prompts # T14 malformed --rates / --windows file → exit 1 with the line number # @@ -170,6 +171,7 @@ sed "s#/srv/example/proj#$P#g" "$FIX/effort/s-3.jsonl" > "$CFG12/projects/$(slug J12="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --json --effort 2>&1 | fold)" has "T12 4 human prompts (dup uuid once; meta, notification, peer, compact, tool result, , sidechain, foreign cwd excluded)" "$J12" '"human_prompts": 4' has "T12 agent seconds 591 (220 captured + 371)" "$J12" '"agent_seconds": 591' +has "T12 a later zero-usage record of msg_e1 does not erase its usage (per-field max)" "$J12" '"2026-09-20": { "stage": "2", "usage": [121, 1500, 30000, 286, 6] }' S12="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --effort 2>&1)" has "T12 screen line" "$S12" "agent working time: 0h 10m (main thread, gaps <= 5 min, gaps ending at a human prompt excluded) · human prompts: 4" J12g="$(CLAUDE_CONFIG_DIR="$CFG12" bash "$SUT" "$P" --json --gap 1 2>&1 | fold)" From 687c974af4f684a2cdb8e772d2e099f323785b60 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 15:30:33 +0000 Subject: [PATCH 3/3] docs: extraction-quality wording matches MANUAL-when-absent; token-burn capture notes Confirmation review of #325: the skill paragraph still said a missing score file shows as a fault; capture notes said "last record" where the code takes the per-field max. Wording applied as the reviewer specified. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- skills/extractor-quality-loop.md | 4 ++-- tests/wave2/fixtures/token-burn/CAPTURE.md | 2 +- tests/wave2/test-token-burn.sh | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/skills/extractor-quality-loop.md b/skills/extractor-quality-loop.md index bd45db53..b66e4e94 100644 --- a/skills/extractor-quality-loop.md +++ b/skills/extractor-quality-loop.md @@ -182,8 +182,8 @@ Noting these as "BRD-only" in the quality report is correct and expected. **Where it is read:** `bin/extraction-report.sh` (the Stage 1 surface) renders every `extraction-quality*.json` at `analysis//` or `analysis//extracted/` (also inside the knowledge base), one row per key of `dimensions`. A code-extracted knowledge base -with no such file, a `selfGraded` pass, a pass under `threshold`, or a 0/0 dimension missing -from `confirmedZeros` shows there as a fault. +with no such file shows as MANUAL ('not produced yet'); a `selfGraded` pass, a pass under +`threshold`, or a 0/0 dimension missing from `confirmedZeros` shows as a fault. ### A score without a second mechanism is self-graded — label it, never let it pass alone diff --git a/tests/wave2/fixtures/token-burn/CAPTURE.md b/tests/wave2/fixtures/token-burn/CAPTURE.md index 1b9f619e..0156348d 100644 --- a/tests/wave2/fixtures/token-burn/CAPTURE.md +++ b/tests/wave2/fixtures/token-burn/CAPTURE.md @@ -11,7 +11,7 @@ it was imagined: (the ~2× overcount if summed per record) — but NOT always the same `usage`: in the subagent transcript `msg` ids with a later record carry a larger `output_tokens` on that last record (60 on the first records of the 2026-09-17 messages, 1559 summed over last records). The - instrument takes each id's LAST record; the numbers below were recomputed that way; + instrument takes each id's per-field max (= its last record on this capture); the numbers below were recomputed that way; - `message.model` = `` on a harness-generated record (exactly one kept); - `cwd` is **per record**: a session launched in `~` spends records in `~`, `~/proj` and `~/proj/app`; diff --git a/tests/wave2/test-token-burn.sh b/tests/wave2/test-token-burn.sh index 19feb055..0c00db4a 100755 --- a/tests/wave2/test-token-burn.sh +++ b/tests/wave2/test-token-burn.sh @@ -4,7 +4,7 @@ # scrubbed, and the expected sums computed independently of the instrument. # # T1 no transcript tree → NOT AVAILABLE in brief, screen and json (available:false), exit 0 -# T2 --json: per-model arrays to the token, usage from each id's LAST record; 52 records dedupe to 22 messages; 1 synthetic +# T2 --json: per-model arrays to the token, usage from each id's per-field max (= last record on this capture); 52 records dedupe to 22 messages; 1 synthetic # skipped; 2 files; subagent headline 61357 # T3 the two-tree layout: records whose cwd is /app count — moving them out of the # project drops the message count from 22 to 21 @@ -73,7 +73,7 @@ has "T1 json available:false" "$OUT" '"available": false' J="$(CLAUDE_CONFIG_DIR="$CFG" bash "$SUT" "$P" --json 2>&1 | tr -d '\n' | tr -s ' ' | sed 's/\[ /[/g; s/ \]/]/g')" has "T2 fable-5-1 array" "$J" '"claude-fable-5-1": [288, 31365, 2607781, 15524, 9]' # the subagent model's array, matched by its numbers (the model id is the captured one, see CAPTURE.md) -echo "$J" | grep -Eq '"claude-[a-z]+-5": \[26, 59772, 998476, 1559, 13\]' && ok "T2 subagent-model array (LAST record per id: output 1559, the first record says 60)" || bad "T2 subagent-model array" "$J" +echo "$J" | grep -Eq '"claude-[a-z]+-5": \[26, 59772, 998476, 1559, 13\]' && ok "T2 subagent-model array (per-field max (= last record on this capture) per id: output 1559, the first record says 60)" || bad "T2 subagent-model array" "$J" has "T2 22 messages" "$J" '"messages": 22' has "T2 52 records (dedupe)" "$J" '"records": 52' has "T2 1 synthetic" "$J" '"synthetic_skipped": 1'