Repository navigation
Re-baseline every measured number onto one verified run - #3
Merged
Merged
Conversation
Re-running benchmarks/benchmark.py on the same A6000 put the 20,000 and 50,000 rows about 10% below what the README claimed (30x and 29x measured against 33x and 31x). Small, and in the flattering direction, which is the direction that matters. Both tables now carry the numbers this run produced. The scaling exponents in docs/implementation.md did not reproduce at all -- n^2.01 and n^1.13 against a measured n^1.63 and n^1.39. The argument was sound; the fit range was the problem, and it was never written down. Fitting over all n mixes in the small-n region where the GPU is idle most of the time and the fit reports launch overhead rather than the algorithm. Over n >= 50,000, where both implementations are past their fixed overheads, the exponents are n^1.94 and n^1.31, so the ratio decays as n^-0.64 and predicts a 2.42x fall in speedup from 50k to 200k. The measured fall is 2.42x. The range is now stated in the text so the fit can be repeated. The d = 128 table needed no correction beyond rounding: measured 14/15/12/16/17 against a claimed 15/15/13/16/17, and the ratio really is flat -- it scales as n^0.05 across that range. The claim that ~15x is the steady-state hardware win holds. The one thing I could not verify is the assertion that both sides scale as n^1.9 at d = 128, since I measured the ratio and not the two curves separately, so the text now says what was measured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-running the benchmark put the 20k and 50k README rows ~10% below what was claimed, in the flattering direction. Both tables now carry the numbers this run produced.
The scaling exponents in implementation.md did not reproduce because the fit range was never stated. Fitted over n >= 50,000, where both implementations are past their fixed overheads: cuLOF n^1.94, scikit-learn n^1.31, so the ratio decays as n^-0.64 and predicts a 2.42x fall in speedup from 50k to 200k. Measured fall: 2.42x.
The d=128 table verified as-is (measured 14/15/12/16/17 vs claimed 15/15/13/16/17), and the ratio really is flat at n^0.05.