Add the 1844th performance column, and what two runs of one commit say about the timings - #1099
Merged
Merged
Conversation
…y about the timings The column for 2.4.0. Seventy-five commits from the 1769th, carrying the rule-set exchange, Hensel lifting along an evaluation homomorphism, and four wrong answers found by writing rules out as data. **The Kernel Benchmark was run twice on this same commit, and the two runs are 30-57% apart on every one of eighteen benchmarks.** Allocation over the same pair agrees to 0.03% -- `SimplifyEasy` 128,098 B against 128,096 B. That is the thousandfold difference `PerformanceGate` already argues from, measured here on the workflow this file's columns come from. So a timing difference between two of these columns is evidence only above about 50%. The 1769th's write-up reads -7%, -10%, -15% as improvements and concludes "every row is faster or flat"; none of those is distinguishable from the runner. Only its `SolveEasy` row survives, and it survives because allocation moved 2.3x alongside the 2.5x -- which is the rule that write-up itself states. The table takes run 2 and run 1 is recorded in the prose, so the choice is visible. **And the exchange cost 13% of `SimplifyEasy`, which this file cannot measure but a desktop can.** 82,676 ns at the 1769th, 148,223 after the conversions, 94,904 after the node-type index, 93,732 at the 1843rd -- one machine, both arms, standard deviation under half a per cent. Every conversion was measured against the commit in front of it and each came back free or better; nothing was measured against the start. Counted rather than guessed: 4,914 rule-set invocations on this input at both commits, so the work shape is identical and this is per-operation overhead -- about 1.3 ns per invocation, 14 ns per rule attempt. `InequalityEquality` is 26% of it, `InvertNegativePowers` 15% from a single rule. Three ways of making the dispatch cheaper were measured and none moved it; about 40% is unattributed. The lever is the invocation count, not the dispatch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bjumi5K7fg8yx6UK1mZTQd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The performance column 2.4.0 needs, and a measurement that changes how the columns above should
be read.
The Kernel Benchmark was run twice on this same commit
Nothing changed between them.
Allocation over the same pair:
SimplifyEasy128,098 B against 128,096 B,SolveHard1,446,783,288 B against 1,446,779,520 B — 0.03%. A thousandfold difference in
reproducibility, which is precisely the argument
PerformanceGatealready makes for gatingallocation and not time. It had not been measured on this workflow before.
So a timing difference between two columns of this file is evidence only above ~50%. The
1769th's write-up reads −7%, −10%, −15% as improvements and concludes "every row is faster or
flat"; none of those is distinguishable from the runner. Only its
SolveEasyrow survives —2.5x with allocation moving 2.3x alongside — and it survives by the rule that write-up itself
states.
The table takes run 2; run 1 is in the prose so the choice is visible rather than quiet.
The exchange cost 13% of
SimplifyEasyReal, and measured where this file cannot measure — one desktop, both arms built and run the
same way, BenchmarkDotNet StdDev under 0.5%:
SimplifyEasyEvery conversion was measured against the commit in front of it and every one came back free or
better. Nothing was measured against the start. The index recovered most of what the exchange
spent; most is not all.
Counted rather than guessed: 4,914 rule-set invocations at both commits — the work shape is
identical, so this is per-operation overhead, ~1.3 ns per invocation and ~14 ns per rule attempt.
InequalityEqualityis 26% of it;InvertNegativePowers, a set with one rule, is 15%. Threeattempts at making dispatch cheaper (one-slot cache, eight-slot cache, array index instead of the
dictionary) were each measured and none moved it. About 40% is still unattributed and this
says so.
The lever, if wanted, is the invocation count rather than the dispatch — 4,914 for one
Simplify, of which 2,202 are the canonical order on every node.Docs only; no code changes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Bjumi5K7fg8yx6UK1mZTQd