Feature/optimize logging repr - #3
Open
gthelen wants to merge 2 commits into
Open
Conversation
RevRange.__repr__ was previously running 'git describe' for the include
commit and every excluded commit, as well as 'git rev-list --bisect-all'
to count the commits.
On large repositories like the Linux kernel, 'git describe' takes ~1s
per commit. As bisection progresses and the number of excludes grows,
this made __repr__ extremely slow (taking dozens of seconds per call).
Furthermore, because of f-string logging (e.g. logger.debug(f"...")),
these __repr__ calls were evaluated on every step even when verbose
logging was disabled.
This caused git-brisect to effectively hang when running with many
threads and a large commit range.
Benchmarked overhead of __repr__ (on this repo vs projection):
- Case 1 (1 exclude):
- Old __repr__: 0.0300s
- New __repr__: 0.00000407s (~7,300x speedup)
- Case 2 (10 excludes):
- Old __repr__: 0.1158s
- New __repr__: 0.00000475s (~24,400x speedup)
Projections for Large Repo (describe = 1.0s, rev-list = 0.5s):
- 1 exclude: 2.5s -> ~0.00s (~2.5M x speedup)
- 10 excludes: 11.5s -> ~0.00s (~11.5M x speedup)
- 20 excludes: 21.5s -> ~0.00s (~21.5M x speedup)
Optimized __repr__ to simply print the hashes of the include and exclude
commits, which is instantaneous and avoids all git calls. Also handles
empty exclude lists cleanly and sorts excludes for deterministic output.
Before & After Examples of Logging:
Suppose we are bisecting a range with 1 good commit (include) and bad
commits (exclude), and it has 100 commits in between.
Before this change:
DEBUG: Considering range RevRange([v5.11 ^v5.10] 100 commits)
DEBUG: Canceling RevRange([v5.11-rc5~4 ^v5.10] 40 commits),
remaining: RevRange([v5.11 ^v5.11-rc5~4] 60 commits)
After this change:
DEBUG: Considering range RevRange(v5.11 ^v5.10)
DEBUG: Canceling RevRange(1111111111111111111111111111111111111111
^v5.10), remaining: RevRange(v5.11 ^1111111111111111111111
111111111111111111 ^22222222222222222222222222222222222222
22 ^v5.10)
(where 11111111... is midpoint commit hash, 22222222... is merge base)
Signed-off-by: Greg Thelen <gthelen@google.com>
Avoid calling 'describe(rev)' (which runs 'git describe') in the main thread during 'enqueue'. This debug log was evaluated always because of f-string, blocking the main thread per enqueued job. With 256 threads, this causes significant startup delays. Benchmarked overhead (with DEBUG logging disabled): - Old way (f-string with describe): ~10.3ms per call (in this repo) - New way (lazy logging with raw rev): ~0.28us per call - Speedup: ~36,800x Projections for 256 enqueues (e.g., 256 threads at startup): - Small Repo (describe = 10.8ms): 2.76s -> 0.00007s - Large Repo (describe = 1.0s): 256s (~4.27 mins) -> 0.00007s Replaced with lazy logging using the raw revision hash. Logging message examples: - Old: DEBUG Enqueued v1.0.0-12-gd83148a, new queue depth 5 - New: DEBUG Enqueued d83148a30308108a6681399923e3a7d6be6d6343, new queue depth 5 Signed-off-by: Greg Thelen <gthelen@google.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Minor adjustments to make logging faster.