docs(changelog): withdraw the "q8 stays a win over Kuzu" claim (#1825) - #1829
Merged
Conversation
#1823's entry closed with "q5 and q8 are ties at both scales in both runs, and q8 stays a win over Kuzu". The first half is a real measurement of what that lane does. The second half is a board-position claim that does not survive re-measurement, and it is now merged on master where it round-trips as fact. q8's headline number is a CROSS-CALL MEMOIZATION artifact. The two-hop equal-domain degree counts are cached onto the CALLER's Plottable keyed by id(), so the figure quoted on the board is a warm repeat call: warm 2.02 ms @20k 5.09 ms @100k cold (memo delattr'd) 13.18 49.68 fresh Plottable 14.60 53.02 bind_only control 0.02 0.02 <- rules out re-binding A ONE-SHOT q8 therefore loses to embedded Kuzu by 3.1-5.2x at 20k and 2.8-6.0x at 100k. The memo itself is filed as #1825, because keying a cache by id() on the caller's object also returns a stale answer after an in-place edge or node mutation -- so the cell is not merely optimistic, the mechanism behind it is a bug. Reconciled rather than deleted: the withdrawn sentence is quoted in place so the correction is traceable from the claim, and the measurement #1823 actually reports -- that its lane leaves q8 unchanged -- is explicitly preserved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
…a too #1818 landed the pyg-bench docs-numbers contract and its data file, which ships graphbench.{20k,100k}.q8.polars and .q8.polars_vs_kuzu as board_quotable=true with an empty disclosures list. That is the same claim this PR withdraws from the CHANGELOG, in a second location, so it is corrected the same way rather than left to be found later. The correction is expressed through the contract's own mechanism, not around it. Two invariants define the shape: quotable_implies_comparable board_quotable implies comparison_allowed caveat_travels_with_the_number status != 'ok' or comparison_allowed is false implies disclosures is non-empty So the four cells become comparison_allowed=false (a warm-memo figure is not a fair one-shot comparison), board_quotable=false (forced by the first invariant), and carry a disclosure recording the warm/cold/fresh arms, the bind_only control, and the one-shot loss factors. Re-checked all 119 cells against cell_required, both quotable invariants, caveat_travels_with_the_number and cell_names_a_known_run: zero violations. Diff is exactly those 4 cells; kuzu and pandas cells are untouched. This is an INTERIM correction on the consumer side. The durable fix is to regenerate the run in pyg-bench and re-copy the artifact (#1827) -- these figures are from graphbench-q1q9-*-20260726 and also predate #1828's q4 result, so the whole graphbench block wants a fresh run, not four hand-edits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
Only CHANGELOG.md changed on both sides. Resolved structurally rather than textually: master's file taken whole, this branch's entries re-inserted under the same '###' section they occupied on the branch. Every other branch file carries through unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
My previous merge of this branch was built by a helper that assumes a branch ADDS a CHANGELOG entry and re-inserts it under its section. This branch does not add one -- it REWRITES #1823's existing entry -- so the helper produced the entry TWICE: master's uncorrected line plus the corrected one, which left the withdrawn claim still present in the file the PR was meant to fix. Rebuilt from master with the correction applied as an in-place replacement, asserting both that the anchor occurs exactly once on master before the edit and that the entry occurs exactly once after it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
lmeyerov
force-pushed
the
perf/gfql-h3-two-hop-count-lazy
branch
from
July 28, 2026 15:56
d395790 to
315d3cc
Compare
lmeyerov
force-pushed
the
docs/withdraw-q8-win-claim-1825
branch
from
July 28, 2026 15:56
e311a86 to
46e2548
Compare
Docs sits at the TOP so a change-request here cannot block the code underneath. CHANGELOG hand-resolved: this branch REPLACES #1823's entry rather than adding one, so neither a textual 3-way nor the add-only fallback applies. Built as the parent file with the q8 withdrawal applied in place, asserting the anchor occurs exactly once before the edit and the entry exactly once after. Rebuilt from this branch's clean pre-stack head: an earlier stacking attempt uploaded child blobs over already-merged ones, which propagated a dropped POLARS_TEST_FILES entry up the whole stack and turned test-gfql-core red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
The stacking upload rewrote the tree entry as mode 100644, so the test-polars lanes died with 'Permission denied' (exit 126) before running a single test.
The data contract refuses a unit=x cell with comparison_allowed=false -- "a ratio over figures never established as comparable" -- and it is right to. Marking the q8 polars-vs-kuzu ratio non-quotable left the number sitting in the published data for someone to quote. Deleted instead. The raw per-engine q8 figures stay, caveated and non-comparable. Also withdraws the matching prose: gfql/performance.rst still asserted "GFQL wins q8", which is the exact claim this PR exists to retract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Master carries a benchmark claim that my own re-measurement refutes. Correcting it in place rather than leaving it to round-trip as fact.
#1823's CHANGELOG entry closes with:
The first half is a real measurement of what that lane does, and it stands. The second half is a board-position claim, and it does not survive re-measurement.
Why it fails
q8's headline number is a cross-call memoization artifact. The two-hop equal-domain degree counts are cached onto the caller's
Plottable, keyed byid(), so the figure quoted on the board is a warm repeat call:delattr'd)Plottablebind_onlycontrolThe
bind_onlyarm is the one that makes this conclusive: re-binding costs 0.02–0.04 ms, so the gap between warm and cold is the memo, not setup. A one-shot q8 loses to embedded Kuzu by 3.1–5.2× at 20k and 2.8–6.0× at 100k.The mechanism is itself filed as #1825: keying a cache by
id()on an object the caller still owns also returns a stale answer after an in-place edge or node mutation, on all engines. So the cell is not merely optimistic — what produces it is a bug.Scope
Reconciled, not deleted, per the standing rule that reconciling beats deleting:
Related
Two sibling instances of the same claim were caught before merge and are already handled: #1816 carried it in its own entry (removed, now points at #1825), and #1818 ships
graphbench.{20k,100k}.q8.polars_vs_kuzuasboard_quotable: truewith an emptydisclosureslist — that one is deferred pending a regenerated board.