docs(gfql): publish benchmark numbers from a provenance-carrying source of truth

[lmeyerov]

Review-only. Do not self-merge.

The problem

Benchmark numbers reach these docs by hand transcription. That is the same failure mode that produced a fabricated competitor column earlier in this campaign: a number copied by hand becomes authoritative, and then gets re-copied after the board underneath it moves.

An audit of every published performance figure found both outcomes had already happened — including two claims that are not merely unverifiable but refuted by a matched re-run:

Published claim	Matched, row-validated measurement
performance.rst: OLAP "q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200×)"	q8 @100k: GFQL-polars 5.06 ms vs Kuzu 9.76 ms = 1.93×	REFUTED
performance.rst: OLAP "q9 is 14.2×"	q9 @100k: 66.61 ms vs 84.06 ms = 1.26×	REFUTED
performance.rst: LDBC-vs-Neo4j table (5 rows)	all four rows that map onto a query id the current lane emits (seed-lookup, message-content, message-creator, one-hop-expand) differ materially from the current board on both sides — e.g. seed-lookup 106.1 ms → 28.08 ms, Neo4j 143.7 ms → 155.32 ms	STALE
performance.rst: LDBC recent-replies row	maps onto no query id the current lane emits at all, so it cannot be re-derived — only re-measured	UNVERIFIABLE

The Kuzu 1,004 ms figure is the historical board that pyg-bench's own RESULTS.md records as "did not reproduce and cannot be reproduced" — a competitor handicap. It was still on the docs.

Most of the rest could not be confirmed or refuted: the reproducers wrote to /tmp (or only printed), recorded no commit/host/timestamp, and their artifacts were never committed and no longer exist. A number whose provenance cannot be established is unpublishable, not presumed correct.

The mechanism

Docs no longer restate a number; they reference it.

• docs/source/_data/gfql_benchmarks.json — the single source of truth. Every cell names the run that produced it: measurement date, host, perf lock held, quiet host, rep protocol, the frozen pygraphistry commit, the pyg-bench commit, runtime image, dataset, competitor + version, raw artifact path. It is generated by scripts/export_docs_numbers.py in pyg-bench#109 (merged) from artifacts committed there — so regenerating the docs' numbers needs no GPU box and no benchmark re-run.
• docs/source/_ext/gfql_bench.py — a small Sphinx extension providing :bench:key, `:bench-diag:`key, .. bench-provenance:: and .. bench-disclosures::.
• bin/check_bench_numbers.py — the same gate without Sphinx, plus two checks a docs build cannot make.
• docs/test_bench_numbers.py — 8 negative tests pinning the rejections.

Why an extension rather than substitutions or an include

The docs build already runs -W --keep-going -n, so a warning is fatal — but neither rst_prolog substitutions nor a generated .. include:: can enforce requirement 2 (numbers must carry their disclosures). A substitution renders a bare value with no idea who used it; an include cannot stop a page from restating a figure in prose. The extension is ~300 lines, has no dependency beyond Sphinx itself, and is the smallest thing that can refuse a non-quotable cell at the point of use and require that page to render its caveats. The generated JSON keeps requirement 4 (regenerate offline) intact regardless.

What breaks the build

Failure	Caught by
a key the source of truth does not contain	Sphinx + standalone
a run older than policy.max_age_days — checked for every run, not just re-read pages, so an incremental build cannot serve an aged-out number	Sphinx + standalone
a cell that is not board-quotable (comparison_allowed: false, or a non-ok status) published as a bare number — it must use :bench-diag:, which labels it diagnostic-only	Sphinx + standalone
a page that references a cell without rendering its .. bench-provenance:: / .. bench-disclosures::	Sphinx + standalone
engine commit drift — commits touching graphistry/compute landed since a published run was measured	standalone only (needs git history)
a hand-typed literal on a managed page — a number typed by hand is by construction not a reference to the source of truth	standalone only

The disclosure rule is the load-bearing one. A bare ratio stripped of its asterisk launders the caveat, which is worse than manual transcription.

Demonstrated, not asserted

All five failure modes were run against a real sphinx-build over this branch's performance.rst and against the standalone gate:

key that does not exist       -> BenchNumberError, sphinx exit 2
run aged past the policy      -> BenchNumberError, sphinx exit 2 (also on an incremental build)
diagnostic-only cell as :bench: -> BenchNumberError, sphinx exit 2
disclosures block deleted     -> BenchNumberError, sphinx exit 2
hand-typed "0.124 ms"         -> bin/check_bench_numbers.py exit 1
commit drift policy tightened -> bin/check_bench_numbers.py exit 1 ("10 commits ... have landed since 84be35fb was measured")

What is published now

gfql/performance.rst carries the one board that is fully traceable: the matched, row-validated prrao87/graph-benchmark q1–q9 Cypher suite at 20k and 100k persons — GFQL-pandas vs GFQL-polars vs Kuzu 0.11.3 — measured on a quiet, perf-locked dgx-spark with position-balanced slots, every per-slot artifact committed in pyg-bench, and every GFQL cell's result rows validated against Kuzu's before publication. Board-quotability is derived from the artifacts, not declared: a row mismatch suppresses the ratio entirely.

Both directions are shown, including that GFQL-polars loses six of nine cells on the small graph and that GFQL-pandas loses to Kuzu on most cells at both sizes.

What was withdrawn

Removed rather than restated, on unrecoverable provenance: the four-engine Orkut table, the 0.58.0 tag-sweep tables (performance.rst and indexing.rst), the seeded-index synthetic and vs-Kuzu/Neo4j tables, the prepared-Kuzu figures, the LadybugDB head-to-head (whose competitor column was an uncited hardcoded constant in the benchmark script, not a measurement), the CPU-crossover ratios, the filter→PageRank results and their charts (the chart generator reads a results directory that has never existed in any commit), and every downstream echo (~38x, 9-28x vs Kuzu/Neo4j, 43X+, 10-50x, 100X+, 10X+).

benchmark_graphframes.rst is kept intact — its raw results are committed at docs/source/gfql/_static/graphframes/results.json and every headline cell round-trips against it. Bringing it under the gate is a follow-up.

This is a large removal. It is deliberate: removing an unverifiable number is a correct outcome, not a regression.

CI

.github/workflows/ci.yml is not touched (pending workflow OAuth scope). The patch — path-filter entries plus a Gate published benchmark numbers step in test-docs, with fetch-depth: 0 for the commit-drift check — is written up and reported separately. Note the Sphinx build in docs/ci.sh already enforces the reference/provenance/disclosure/freshness rules independently, so test-docs gains the gate even without the patch.

Verification

• bin/lint.sh clean; ruff clean on the new files.
• mypy --strict clean on all three new modules (no Any, no cast, no getattr/setattr, no bare Dict[str, Any]).
• rstcheck clean on every edited .rst (the four new roles/directives registered in docs/.rstcheck.cfg).
• pytest docs/test_bench_numbers.py — 8 passed.
• Full sphinx-build read phase over the real docs/source: 0 gfql-bench warnings, 0 errors from this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_015YsqAZQLbqjSDrYSFz2GoB

[lmeyerov]

Superseded by the split you asked for. Closing this in favour of two PRs:

• docs(gfql): delete refuted and unverifiable benchmark numbers, and gate the rest #1813 — docs/bench-numbers-gate-and-deletions (base master). The urgent, mechanical half: the gate machinery plus the deletion of every refuted and unverifiable number, including the q8 "200x" that measures 1.93x and the q9 "14.2x" that measures 1.26x, the hardcoded LADYBUG = {...} competitor column, and the unregenerable filter->PageRank charts. No editorial rewriting, no restructuring, no voice changes — reviewable in a few minutes.
• docs(gfql): publish the traceable q1-q9 board and rewrite the prose around it #1815 — docs/bench-numbers-editorial (stacked on docs(gfql): delete refuted and unverifiable benchmark numbers, and gate the rest #1813). The editorial half: publishing the traceable q1-q9 board with its lane named precisely, and rewriting the surviving prose. Expect iterations; that is the point of separating it.

Nothing from this branch is lost — #1813 carries the machinery and the deletions verbatim where they needed no judgement, and #1815 carries the board. Both are green-pending and unmerged.