Skip to content

docs(changelog): withdraw the "q8 stays a win over Kuzu" claim (#1825) - #1829

Merged
lmeyerov merged 7 commits into
masterfrom
docs/withdraw-q8-win-claim-1825
Jul 28, 2026
Merged

docs(changelog): withdraw the "q8 stays a win over Kuzu" claim (#1825)#1829
lmeyerov merged 7 commits into
masterfrom
docs/withdraw-q8-win-claim-1825

Conversation

@lmeyerov

Copy link
Copy Markdown
Contributor

Master carries a benchmark claim that my own re-measurement refutes. Correcting it in place rather than leaving it to round-trip as fact.

#1823's CHANGELOG entry closes with:

The cells this lane is not called for are unchanged: q5 and q8 are ties at both scales in both runs, and q8 stays a win over Kuzu.

The first half is a real measurement of what that lane does, and it stands. The second half is a board-position claim, and it does not survive re-measurement.

Why it fails

q8's headline number is a cross-call memoization artifact. The two-hop equal-domain degree counts are cached onto the caller's Plottable, keyed by id(), so the figure quoted on the board is a warm repeat call:

arm 20k 100k
warm (memo hit) 2.02 ms 5.09 ms
cold (memo delattr'd) 13.18 ms 49.68 ms
fresh Plottable 14.60 ms 53.02 ms
bind_only control 0.02 ms 0.02 ms

The bind_only arm is the one that makes this conclusive: re-binding costs 0.02–0.04 ms, so the gap between warm and cold is the memo, not setup. A one-shot q8 loses to embedded Kuzu by 3.1–5.2× at 20k and 2.8–6.0× at 100k.

The mechanism is itself filed as #1825: keying a cache by id() on an object the caller still owns also returns a stale answer after an in-place edge or node mutation, on all engines. So the cell is not merely optimistic — what produces it is a bug.

Scope

Reconciled, not deleted, per the standing rule that reconciling beats deleting:

Related

Two sibling instances of the same claim were caught before merge and are already handled: #1816 carried it in its own entry (removed, now points at #1825), and #1818 ships graphbench.{20k,100k}.q8.polars_vs_kuzu as board_quotable: true with an empty disclosures list — that one is deferred pending a regenerated board.

lmeyerov and others added 4 commits July 28, 2026 03:21
#1823's entry closed with "q5 and q8 are ties at both scales in both runs,
and q8 stays a win over Kuzu". The first half is a real measurement of what
that lane does. The second half is a board-position claim that does not
survive re-measurement, and it is now merged on master where it round-trips
as fact.

q8's headline number is a CROSS-CALL MEMOIZATION artifact. The two-hop
equal-domain degree counts are cached onto the CALLER's Plottable keyed by
id(), so the figure quoted on the board is a warm repeat call:

  warm        2.02 ms @20k   5.09 ms @100k
  cold (memo delattr'd)  13.18        49.68
  fresh Plottable        14.60        53.02
  bind_only control       0.02         0.02   <- rules out re-binding

A ONE-SHOT q8 therefore loses to embedded Kuzu by 3.1-5.2x at 20k and
2.8-6.0x at 100k. The memo itself is filed as #1825, because keying a cache
by id() on the caller's object also returns a stale answer after an in-place
edge or node mutation -- so the cell is not merely optimistic, the mechanism
behind it is a bug.

Reconciled rather than deleted: the withdrawn sentence is quoted in place so
the correction is traceable from the claim, and the measurement #1823 actually
reports -- that its lane leaves q8 unchanged -- is explicitly preserved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
…a too

#1818 landed the pyg-bench docs-numbers contract and its data file, which
ships graphbench.{20k,100k}.q8.polars and .q8.polars_vs_kuzu as
board_quotable=true with an empty disclosures list. That is the same claim
this PR withdraws from the CHANGELOG, in a second location, so it is
corrected the same way rather than left to be found later.

The correction is expressed through the contract's own mechanism, not around
it. Two invariants define the shape:

  quotable_implies_comparable      board_quotable implies comparison_allowed
  caveat_travels_with_the_number   status != 'ok' or comparison_allowed is
                                   false implies disclosures is non-empty

So the four cells become comparison_allowed=false (a warm-memo figure is not
a fair one-shot comparison), board_quotable=false (forced by the first
invariant), and carry a disclosure recording the warm/cold/fresh arms, the
bind_only control, and the one-shot loss factors. Re-checked all 119 cells
against cell_required, both quotable invariants, caveat_travels_with_the_number
and cell_names_a_known_run: zero violations. Diff is exactly those 4 cells;
kuzu and pandas cells are untouched.

This is an INTERIM correction on the consumer side. The durable fix is to
regenerate the run in pyg-bench and re-copy the artifact (#1827) -- these
figures are from graphbench-q1q9-*-20260726 and also predate #1828's q4
result, so the whole graphbench block wants a fresh run, not four hand-edits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
Only CHANGELOG.md changed on both sides. Resolved structurally rather than
textually: master's file taken whole, this branch's entries re-inserted under
the same '###' section they occupied on the branch. Every other branch file
carries through unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
My previous merge of this branch was built by a helper that assumes a branch
ADDS a CHANGELOG entry and re-inserts it under its section. This branch does
not add one -- it REWRITES #1823's existing entry -- so the helper produced the
entry TWICE: master's uncorrected line plus the corrected one, which left the
withdrawn claim still present in the file the PR was meant to fix.

Rebuilt from master with the correction applied as an in-place replacement,
asserting both that the anchor occurs exactly once on master before the edit
and that the entry occurs exactly once after it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
@lmeyerov
lmeyerov changed the base branch from master to perf/gfql-h3-two-hop-count-lazy July 28, 2026 15:12
@lmeyerov
lmeyerov force-pushed the perf/gfql-h3-two-hop-count-lazy branch from d395790 to 315d3cc Compare July 28, 2026 15:56
@lmeyerov
lmeyerov force-pushed the docs/withdraw-q8-win-claim-1825 branch from e311a86 to 46e2548 Compare July 28, 2026 15:56
lmeyerov and others added 3 commits July 28, 2026 08:58
Docs sits at the TOP so a change-request here cannot block the code underneath.

CHANGELOG hand-resolved: this branch REPLACES #1823's entry rather than adding
one, so neither a textual 3-way nor the add-only fallback applies. Built as the
parent file with the q8 withdrawal applied in place, asserting the anchor occurs
exactly once before the edit and the entry exactly once after.

Rebuilt from this branch's clean pre-stack head: an earlier stacking attempt
uploaded child blobs over already-merged ones, which propagated a dropped
POLARS_TEST_FILES entry up the whole stack and turned test-gfql-core red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
The stacking upload rewrote the tree entry as mode 100644, so the
test-polars lanes died with 'Permission denied' (exit 126) before
running a single test.
The data contract refuses a unit=x cell with comparison_allowed=false --
"a ratio over figures never established as comparable" -- and it is right
to. Marking the q8 polars-vs-kuzu ratio non-quotable left the number
sitting in the published data for someone to quote. Deleted instead. The
raw per-engine q8 figures stay, caveated and non-comparable.

Also withdraws the matching prose: gfql/performance.rst still asserted
"GFQL wins q8", which is the exact claim this PR exists to retract.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
@lmeyerov
lmeyerov changed the base branch from perf/gfql-h3-two-hop-count-lazy to master July 28, 2026 16:50
@lmeyerov
lmeyerov merged commit 978937f into master Jul 28, 2026
75 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant