Serve Git repositories from a local cache

[montehurd]

Serve Git repositories read-only at /git/{name}/{repo} from a local cache of bare repositories, the first part of git-pkgs#403.

upstream:
  git:
    github: "https://github.com"
    wikimedia:
      url: "https://gerrit.wikimedia.org/r"
      repos: ["mediawiki/**"]  # optional allowlist; /** matches nested paths
git:
  cache_dir: /data/git
  max_size: 50GB        # default: no limit
  max_repo_size: 5GB    # default
  refresh_wait: 5s      # default
  stale_max_age: 24h    # default

git config --global url."https://proxy.example.com/git/github/".insteadOf https://github.com/

Freshness

Every info/refs request asks the upstream for its branches, tags and HEAD (protocol v2 ls-refs with ref-prefix, so Gerrit filters out refs/changes/*; a v0 advertisement is filtered here) and fetches only when they differ from the cache. That is one GET and one POST upstream per clone, shared by requests that arrive while it runs, and it gives up after two minutes so a stalled upstream cannot hold a refresh. Fetches use --atomic, so a client served while one runs sees all refs old or all new, and http.lowSpeedTime abandons a transfer that receives nothing for 10 minutes. A new repository is cloned into cache_dir/.tmp and renamed into place. HEAD follows the upstream's default branch.

If the upstream fails or is slower than refresh_wait, a copy checked within stale_max_age is served while the refresh carries on. 401, 403, 404 and redirects are never answered from the cache, as discussed in git-pkgs#403. A repository outside repos answers 404, like a missing one.

A commit a client asks for by ID that is on no cached branch or tag, such as the pull request head actions/checkout fetches, is fetched from the upstream into a hidden refs/proxy/wanted/{time}/{id} ref (uploadpack.hideRefs), so it is neither advertised nor garbage collected. The ref goes once a cached branch or tag includes the commit, or after 7 days. Concurrent requests for it share one fetch, and a commit the upstream could not provide, or could not be reached for, is not asked for again for a minute, so a client retrying a bad ID cannot hold the repository's fetch lock. When up to 4 commits are asked for together and the fetch fails, each is fetched on its own, so a good one is not refused with a bad one: at most 5 fetches under the repository's lock. Larger batches, and any batch the upstream never answered, as in an outage, are refused for the minute without being retried one by one. A stalled upstream costs each of those fetches up to the 10 minute lowSpeedTime. Fetching by ID is off for upstreams that upstream.auth has credentials for (see below).

Cached repositories set uploadpack.allowAnySHA1InWant, so any commit the cache holds is served by ID, including one a force-push left behind, as the upstream would. A commit present in the cache is served without a reachability check, which is the trade. One race comes with it: a commit on no ref can be pruned by an auto gc inside a concurrent fetch while a client asks for it, and that client's fetch fails and retries.

Spawned git and the upstream protections

git never reaches the network. Each fetch gets a route on a loopback relay inside the proxy (random port, random per-fetch ID mapped to one upstream repository URL, only info/refs?service=git-upload-pack and POST git-upload-pack), and the relay forwards with the same upstream client as every other request. So:

• address checks (allow_loopback, allow_private_hosts) apply to every connection, including the first;
• upstream.auth is applied by URL scope, and the credential is never passed to git;
• redirects are not followed, and none is passed back to git;
• the access log records the requests.

The git processes get an environment of their own, with no system or global config, credential helpers, proxy variables or GIT_* from the proxy, GIT_ALLOW_PROTOCOL=http and http.followRedirects=false. Tests check that a client refusing loopback addresses keeps git from reaching a loopback upstream, both for the ref check and for git's own fetch, and that a redirect target sees no request.

An upstream that upstream.auth has credentials for must list repos, since everyone who can reach the proxy can read what it serves; the docs say so.

Serving

internal/handler/git.go parses and validates the request, applies the repos allowlist and maps errors to statuses; internal/gitcache keeps the repositories. Clients are served by git upload-pack --stateless-rpc run directly, as git http-backend would run it, rather than through CGI. Alpine's git package ships without http-backend, and running upload-pack directly gives context cancellation when a client goes away, plain handling of gzip request bodies, and control over its environment. It needs git 2.31 or newer, for GIT_CONFIG_COUNT and fetch --atomic. git-upload-pack requests run in full duplex, since upload-pack writes its response while its stdin is still attached to the request body, and an HTTP/1 server otherwise discards and closes the rest of the body once the response is flushed. Pushing and the dumb protocol are refused. The server's 5 minute write timeout and 30 second read timeout are extended for these requests: a cold clone of a large repository keeps the client waiting before the first byte, and the rest of a request body is read only after any fetch it needs.

Disk

Repositories live at {cache_dir}/{name}-{URL hash}/{repo}.git, so an operator can see what is cached before the UI lists it, and repointing an upstream starts a fresh cache rather than serving the old host's repositories. At startup only the configured upstreams' directories are loaded, and those of their old URLs are removed and logged. Nothing else in cache_dir is touched, so pointing it at a directory that also holds the artifact cache or the database is safe; directories of upstreams no longer configured are logged for the operator to remove. Path segments are validated (no empty, . or .. segments, no control characters or backslashes, and only the last segment may end in .git, so no repository's directory sits inside another's).

• max_repo_size: the relay counts the bytes of each fetch and cuts it off at the remaining budget, so an oversized repository is not downloaded in full first. Every fetched pack is kept as a pack (transfer.unpackLimit=1): index-pack only moves a complete pack into place, where a small pack would otherwise be unpacked object by object and a cut-off would leave, say, a commit without its blobs. A new repository that does not fit is refused for an hour. When a refresh of a cached one does not fit, it is served as it is and dropped once idle, so the next request clones it again with only its branches and tags; commits fetched by ID cannot leave it stuck. A repository right at the limit re-clones each time it grows past it, which is what the limit means. The drop can fall between a client's info/refs and git-upload-pack requests, so the second is served from the fresh clone; the advertisement and the pack then come from different copies, and a want for a ref that moved upstream in between fails and the client retries. A commit fetched by ID that does not fit is refused for an hour and the repository kept, so a large commit, even one named deliberately through a fork, cannot force re-clones.
• max_size: least recently used repositories are evicted after each fetch that takes the cache past it, and at startup. A repository is never evicted while a request or refresh is using it; it is renamed aside under the cache lock, so a later request clones it again.
• Last use and last successful check are kept as file timestamps, so they survive restarts.
• Git's automatic gc stays on, as on any Git server, but runs inside the fetch rather than detached, so sizes are measured after it and no git process outlives its fetch.
• On shutdown, Close cancels the refreshes in progress and waits for them, so none is still writing to the cache when the process exits, and refuses new ones. Fetches by ID run on request goroutines, which the HTTP server's shutdown drains first.

git is added to the image (about 13 MB).

Metrics

Git reports through the existing metrics under ecosystem="git": cache hits and misses per info/refs, upstream fetch durations and errors (not_found, refused, too_large, fetch_failed), and evictions.

Tested

Unit and integration tests run real git clients against an upstream served by git http-backend: protocols v0, v1 and v2, shallow clones, fetching into an existing clone, a cache rewound behind the upstream's back, pruning, default branch changes, ls-refs asking only for branch and tag prefixes, eight concurrent cold clones sharing one fetch, eight warm clones served while a fetch writes, stale serving and its limits, stalled ref checks and stalled transfers, credentials, address checks, redirects, size caps, eviction, gzip and chunked bodies (including a clone of 2000 tags with a small http.postBuffer), a request body read past the server's read timeout, and fetch by ID. Fetch by ID is also tested for unpinning once a branch includes the commit or it ages out, for a repository filled past max_repo_size being cloned again, for not retrying a failed commit within a minute, for staying off with credentials, for a commit too large to fetch leaving the repository and no partial objects behind, for a good commit asked for with a bad one still being fetched, and the cache directory for following the upstream URL, with only a repointed upstream's old directory removed at startup and other data in cache_dir left alone. The key checks were mutation tested.

A black-box script adapted from git-cache-proxy's suite passes against both Gerrit (mediawiki/skins/Vector) and GitHub (git-pkgs/purl), including fetching a patch set and a pull request head by ID. For mediawiki/core, the first info/refs took 5 minutes 5 seconds while the proxy cloned it (796 MB), past the old write timeout, and later checks took under a second.

Not in this PR

• The UI, with the git_repositories table.
• proxy mirror and SBOM support, and a lock shared between processes, so cache_dir must belong to one process for now.
• A cap on concurrent upload-pack processes, and bitmaps or scheduled maintenance to speed up repeated full clones. Pack generation is CPU bound: eight concurrent warm clones of mediawiki/skins/Vector took 4.7 s, against 1.7 s for one.
• Per-client permissions, and Git LFS.

Worth deciding

• Fetch by ID with credentials. GitHub serves a commit from any repository in a fork network through any other. With credentials and repos: ["acme/a"], a fetch by ID through acme/a could pull commits from a private fork outside the allowlist. The options are to document it, to turn fetch by ID off where upstream.auth applies, or to make it opt-in. This PR turns it off where credentials apply; for public hosts it gives no access a client lacks, since anyone can fetch those commits from the host directly.

• 403 means refused. Following Cache Git clones and fetches git-pkgs/proxy#403, a 403 is never answered from the cache. A host that rate-limits with 403 rather than 429 would fail clones during that window instead of serving the cached copy.

• GitHub answers 401 for a repository that doesn't exist, so a misspelled name there returns 502 "upstream refused the request" rather than 404.

Refs git-pkgs#403

[montehurd]

Superseded by git-pkgs#492.