Repository navigation
2026 11 11 tjr reference repos - #463
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A relative pre-existing-repos path is opened from the temporary clone directory, causing valid mappings to be missed and reference-repository reuse to be disabled for the rest of the run.
1 open issue
What changed in this PR
This PR reduces SCM cloning costs by using partial clones and optionally borrowing objects from pre-existing local repositories.
Changes:
- Adds blob filtering and skips checkout during cloning.
- Adds a JSON mapping from library names to local reference repositories.
- Adds clone-behavior tests and updates the mocked clone command.
| File | Description |
|---|---|
tests/scm.py |
Tests partial clones, reference reuse, and fallback behavior. |
tests/functionality_utilities.py |
Updates the clone mock to match the new flags. |
components/scmprovider.py |
Implements partial cloning and optional reference-repository lookup. |
Review effort: Balanced
| # A JSON file mapping library names to local clones of their upstream | ||
| # repositories (e.g. ones baked into the Docker image), which clones can | ||
| # borrow objects from instead of downloading them. | ||
| self.pre_existing_repos_file = config.get('pre-existing-repos') |
tomrittervg
force-pushed
the
2026-11-11-tjr-reference-repos
branch
from
November 11, 2026 14:59
c8f2f02 to
4c72134
Compare
Comment on lines
+161
to
+163
| repos = {} | ||
| for name, path in mapping.items(): | ||
| if os.path.isabs(path): |
| # --reference-if-able takes whatever objects it can from the local copy, and | ||
| # falls back to a normal clone if the copy isn't there. | ||
| reference = ["--reference-if-able", pre_existing_repo] if pre_existing_repo else [] | ||
| self.run(["git", "clone"] + reference + ["--filter=blob:none", "--no-checkout", repo_url, "."]) |
tomrittervg
force-pushed
the
2026-11-11-tjr-reference-repos
branch
from
November 11, 2026 21:58
4c72134 to
047c1e0
Compare
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Invalid cache schemas can abort update checks instead of falling back, while explicit branch checkouts download missing blobs and materialize files, defeating the partial-clone optimization.
3 open issues
Review effort: Balanced
tomrittervg
force-pushed
the
2026-11-11-tjr-reference-repos
branch
from
November 12, 2026 04:29
047c1e0 to
c190498
Compare
Updatebot only reads commits and trees from an upstream clone, never file contents, so make it a blobless clone with no checkout. For irregexp's V8 that's 366 MB instead of 1.7 GB, and a 104 second clone in production should take a few seconds. A host that doesn't support partial clones ignores the filter.
… is one. The Docker image already has clones of V8 and PDF.js, which mach vendor updates to the revision being vendored before we clone the same repository again to list its commits. The new 'pre-existing-repos' setting under 'SCM' names a JSON file mapping library names to such copies, and we clone with --reference-if-able so that git downloads only what the copy lacks. A missing copy falls back to a normal clone, and an unreadable or malformed mapping file is a warning. Paths must be absolute, because we read the file and use its paths from inside the clone directory; a relative one is ignored and reported to Sentry. (Bug 2091623, #385)
…g Mercurial. GitHub is moving ubuntu-latest to Ubuntu 26.04 (actions/runner-images#14748), and since then installing Mercurial fails. Pin the version so that the OS CI runs on only changes when we choose. apt-get update also keeps the install from depending on the package lists the runner image was built with.
tomrittervg
force-pushed
the
2026-11-11-tjr-reference-repos
branch
from
November 12, 2026 04:37
c190498 to
ba1b260
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.