Cache Git clones and fetches #403

Activity

1. andrew

   added

   enhancementNew feature or requestNew feature or request

   on Oct 3, 2026
2. pinguinfuss commented on Oct 11, 2026

   

   pinguinfuss

   on Oct 11, 2026

   Contributor

   More actions

   Here's how I'd approach this. I'd like to settle the questions at the end before writing code.

   The proxy gets a read-only /git/{name}/{path} route, configured like the other named upstreams. Each repo is a local bare clone served by git http-backend. Whenever a client asks for the refs, the proxy checks the upstream's branches and tags and fetches only if something changed, so a clone right after a push sees the new commit. Clients only need an insteadOf:

   git config --global url."https://proxy.example.com/git/github/".insteadOf https://github.com/
   

   upstream:
     git:
       github: "https://github.com"
       wikimedia:
         url: "https://gitlab.wikimedia.org"
         repos: ["mediawiki/*"]   # optional allowlist
   git:
     cache_dir: /data/git         # local disk, not object storage
     max_size: 50GB               # own budget, separate from storage.max_size
     refresh_wait: 5s             # then serve the cached copy if the upstream is slow
     stale_max_age: 24h           # never serve a copy older than this

   I'd split it into these PRs:

   1. Handler and cache. The route, the ref check with concurrent requests sharing one fetch, and git in the image (about 14 MB). A request for a commit the cache doesn't have yet, like the SHA fetch actions/checkout does, triggers a fetch instead of failing. New repos are cloned into a temp directory and moved into place, so nobody gets a half-built clone. Upstream calls use upstream.auth and don't follow redirects.
   2. Metrics and UI. Git reports through the existing cache and upstream metrics as ecosystem="git". Each cached repo gets a row in a new git_repositories table with size, last fetch, last error and access counts, which the UI lists and eviction works from.
   3. Eviction. Least recently used repos go once git.max_size is reached, never while a clone is reading them. A per-repo size cap keeps one huge repo from filling the disk, and git maintenance runs on our schedule instead of in the background.
   4. PURLs, mirror, SBOM. proxy mirror and SBOM ingestion accept pkg:github/…, pkg:bitbucket/… and pkg:generic/…?vcs_url=git+https://… at a tag or commit, and warm submodules too. proxy mirror runs as its own process, so this PR adds a lock per repo across processes; that would also let replicas share a cache_dir later.
   5. Later: offline use. Keep commits that a force-push drops, keep recently used repos warm, an offline mode, and copying a cache_dir into an air-gapped site.

   Git LFS is out of scope; it has its own API and would need its own cache.

   This follows git-cache-proxy closely: http-backend, a lock per repo, and refreshing on every clone instead of on a TTL. Most of the differences come from running it as a shared service: only configured upstreams can be reached, and there's a decision about what to serve when the upstream is down.

   @andrew, a few questions:

   1. Config: a name → URL map at /git/{name}/ like the other upstreams, or a hosts entry like the one you suggested for /generic/sources/{host}/?
   2. With github.com configured, anyone who can reach the proxy can make it clone any public repo. Should repos be required for public forges, or are the size limits enough?
   3. When the upstream is down, serve the cached copy up to stale_max_age, or fail the clone like git-cache-proxy does?
   4. Private repos: one proxy token plus a mandatory repo list, or pass the client's credentials through (as the Conan handler does) and check them on every clone? The second rules out offline use for private repos.

   @montehurd, since this builds on your git-cache-proxy: if you have thoughts, ideas or things you'd do differently, I'd be glad to hear them.
3. andrew commented on Oct 12, 2026

   

   andrew

   on Oct 12, 2026

   ContributorAuthor

   More actions

   I'd go with named upstreams at /git/{name}/, using the config shape you proposed. Git-specific settings fit there, and it matches the existing named upstreams.

   For public repositories, repos can be optional: configuring a public forge can allow any public repository on that forge. Basic disk limits need to work in the first usable release, including a per-repository cap.

   During an upstream outage, serve the cached copy up to stale_max_age, measured from the last successful upstream check. Authentication failures should fail the request rather than fall back to cached private content.

   For private repositories, use configured upstream credentials with a mandatory repository allowlist. The docs need to say that everyone with access to this proxy route can read those repositories. Per-client permissions would need a separate authentication design.

   I'd bring basic disk limits and protection against deleting a repository during an active read into the initial handler PR. Metrics and UI can follow, along with the later mirror and SBOM support.

   One implementation detail to settle in that first PR: a spawned git process won't use the Go HTTP transport's authentication or network restrictions. Please show how its upstream requests preserve the existing protections, including credential scoping and address checks. Disabling redirects alone does not cover the initial connection.
4. montehurd commented on Oct 14, 2026

   

   montehurd

   on Oct 14, 2026

   Contributor

   More actions

   Thanks for pulling this together, @pinguinfuss, and for building on git-cache-proxy. The design looks right to me, and andrew's answers cover the questions I'd have raised. A few things from git-cache-proxy that might save you time:

   Its fetch. Rereading it for this, I found a bug, now fixed: git clone --bare sets no fetch refspec, so its later git fetch only wrote FETCH_HEAD, and the cached branches never moved after the first clone. Its --no-tags also left tags out. Explicit refspecs fix both:

   git fetch --prune origin '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'
   

   Mirroring every ref. Gerrit advertises every patch set as a ref. For mediawiki/core, asking for all refs returns about 358,000 of them (24 MB, 3 s), against 800 (48 KB, 1 s) for branches and tags. So --mirror would be painful, and the per-clone ref check should be a protocol v2 ls-refs with ref-prefix refs/heads/ and refs/tags/, which Gerrit filters on the server. Keeping only branches and tags in the cache also keeps what http-backend advertises to clients small.

   Concurrent clones. Writing tests for that fix turned up a second bug, also fixed: every request rewrote the same git config, so concurrent requests raced on its lock and most failed with a 502 before reaching the per-repo lock. Even with that fixed, the lock makes concurrent clones fetch one after another. Sharing one in-flight refresh, as you plan, is better.

   Spawned git and safehttp. One way to meet andrew's requirement is to never let git reach the network. Point the spawned git fetch at a loopback endpoint inside the proxy that relays info/refs and git-upload-pack to the upstream through the existing safehttp client. Address checks, redirect handling, upstream.auth scoping, the access log and the upstream metrics then apply as they do now, and git never sees a credential. Serving clients is already offline, since http-backend only reads the local repo.

   Timeouts. The server's 5-minute WriteTimeout covers the whole response. On a cold clone of a big repo the client waits for the upstream clone first, so it could be cut off, and so could a slow client. git-cache-proxy needed 500 s FastCGI timeouts for that. http.ResponseController can extend the deadline for the git route.

   I'm glad to review the PRs and test them against Gerrit repos like mediawiki/core from our Cloud VPS setup. If you'd like to split the work, I'm happy to take a piece, or the whole thing if you'd rather hand it off.
5. montehurd

   added 4 commits that reference this issue on Oct 15, 2026

   Keep a cached repository when a commit fetched by ID is too large

   ...

   a919b8f

   Touch only the Git cache's own directories at startup

   ...

   04fd5a4

   Wait for Git cache refreshes to finish on Close

   ...

   85a5900

   Serve Git repositories from a local cache

   ...

   e515f37
6. montehurd commented on Oct 15, 2026

   

   montehurd

   on Oct 15, 2026

   Contributor

   More actions

   @pinguinfuss @andrew, following up on my offer above: I've put together an implementation of the first PR along the lines settled here, so there's something concrete to look at.

   • Diff against main: main...montehurd:proxy:git-http-cache
   • Description and CI (Ubuntu, macOS, Windows): Serve Git repositories from a local cache montehurd/proxy#18

   It covers the handler and cache, with the per-repository and total size limits and no deleting a repository while it's being read. On andrew's question about spawned git: it never reaches the network itself. Its fetches go through a loopback relay that uses the proxy's upstream client, so address checks, upstream.auth scoping and the access log apply, and git never sees a credential. It also fetches commits by SHA the cache doesn't have, such as the PR heads actions/checkout asks for.

   @pinguinfuss, I don't want to step on your plans. I'm happy to open this as the PR, hand it to you to build on, or fold parts into yours, whichever you'd prefer.
7. andrew commented on Oct 15, 2026

   

   andrew

   on Oct 15, 2026

   ContributorAuthor

   More actions

   @montehurd thanks, please open it as a PR against git-pkgs/proxy so the review happens here. @pinguinfuss, you're welcome to review it or build the follow-up PRs on top of it, whichever suits you.
8. montehurd

   added a commit that references this issue on Oct 15, 2026

   Document setting the Git rewrite through environment variables

   ...

   6eba336