Feature request: retry transient upstream TLS failures with bounded backoff
Would you be open to adding automatic retries with bounded backoff for transient failures during upstream git clone --mirror and git fetch? I implemented this in my fork and would be happy to submit a PR if the approach fits the project.
What I encountered
I had git-cache-proxy 0.1.11 in Docker inside WSL2, serving self-hosted Windows GitHub Actions runners. GitHub is the upstream.
A release job intermittently failed while reading a BinSync repository through the proxy:
BinSync candidate build failed: remote: upstream fetch failed
fatal: unable to access 'http://HZVM:8080/HLND2T/CS2_VibeSignatures_binsync_14180_client.dll/':
The requested URL returned error: 502
The proxy container logged the underlying failure during the same incident:
fatal: unable to access 'https://github.com/HLND2T/CS2_VibeSignatures_binsync_14180_client.dll/':
TLS connect error: error:0A000126:SSL routines::unexpected eof while reading
WARN ensure_fresh failed
repo=HLND2T/CS2_VibeSignatures_binsync_14180_client.dll
error=git fetch failed for HLND2T/CS2_VibeSignatures_binsync_14180_client.dll
The repository existed, being public, and already had a cached mirror. Other repositories showed the same TLS EOF diagnostic in historical container logs. Subsequent direct git ls-remote requests from the container succeeded.
My network also uses Fake-IP-based transparent proxying, thus cannot attribute the TLS interruption specifically to GitHub or git-cache-proxy. This request is about tolerating transient upstream connection failures.
References:
Why a brief interruption becomes a job failure
From reviewing 0.1.11, an expired mirror TTL causes an upstream fetch before refs are advertised. If that fetch fails, the request immediately receives 502 upstream fetch failed, even when a mirror exists.
There is no application-level retry around the upstream Git operation. A single transient TLS failure can therefore fail the entire CI job.
Implementation in my fork
I added a shared retry helper for upstream clone/fetch:
- At most four attempts total, with 1, 2, and 4 seconds of backoff.
- Retry recognized transient diagnostics, including TLS EOF, connection resets/timeouts, DNS resolution failures, and HTTP
408, 429, 500, 502, 503, and 504.
- Fail immediately for authentication failures, missing repositories, certificate validation errors, local filesystem errors, and unrecognized failures.
- Set the Git subprocess locale to
C for consistent diagnostic classification.
- Capture a bounded stderr tail and log the operation, attempt, delay, and error category without logging raw upstream stderr.
- Keep the existing per-repository lock across the retry sequence.
- Clean failed clone staging directories before retrying; preserve existing mirrors during fetch retries.
- Update the freshness timestamp only after success.
- Continue returning 502 if retries are exhausted; do not fall back to stale refs.
This does not change LFS or local upload-pack behavior, and does not introduce a transfer timeout for large repositories. The seven-second limit applies to added backoff, not total transfer time.
Implementation: hzqst/git-cache-proxy@6cdc8f9
Validation and deployment experience
I added tests for recovery after transient failures, retry exhaustion, permanent failures, staging cleanup, freshness behavior, concurrent request coalescing, and large stderr output.
- All 65 tests passed.
- Formatting and Clippy checks passed.
- Total line coverage was 96.73%.
- The built Docker image successfully completed real clone/fetch requests after injecting a first-attempt TLS EOF and HTTP 503.
- After deployment, cloning the previously failing BinSync repository through the production proxy succeeded.
Since deploying this retry-enabled version, I have not encountered the HTTP 502 problem again. This is my operational observation so far, rather than proof that the underlying network interruptions have stopped.
Would you accept a PR along these lines? would appreciate your preference on whether the retry count and backoff should remain fixed or be configurable.
Feature request: retry transient upstream TLS failures with bounded backoff
Would you be open to adding automatic retries with bounded backoff for transient failures during upstream
git clone --mirrorandgit fetch? I implemented this in my fork and would be happy to submit a PR if the approach fits the project.What I encountered
I had git-cache-proxy
0.1.11in Docker inside WSL2, serving self-hosted Windows GitHub Actions runners. GitHub is the upstream.A release job intermittently failed while reading a BinSync repository through the proxy:
The proxy container logged the underlying failure during the same incident:
The repository existed, being public, and already had a cached mirror. Other repositories showed the same TLS EOF diagnostic in historical container logs. Subsequent direct
git ls-remoterequests from the container succeeded.My network also uses Fake-IP-based transparent proxying, thus cannot attribute the TLS interruption specifically to GitHub or git-cache-proxy. This request is about tolerating transient upstream connection failures.
References:
Why a brief interruption becomes a job failure
From reviewing
0.1.11, an expired mirror TTL causes an upstream fetch before refs are advertised. If that fetch fails, the request immediately receives502 upstream fetch failed, even when a mirror exists.There is no application-level retry around the upstream Git operation. A single transient TLS failure can therefore fail the entire CI job.
Implementation in my fork
I added a shared retry helper for upstream clone/fetch:
408,429,500,502,503, and504.Cfor consistent diagnostic classification.This does not change LFS or local upload-pack behavior, and does not introduce a transfer timeout for large repositories. The seven-second limit applies to added backoff, not total transfer time.
Implementation: hzqst/git-cache-proxy@6cdc8f9
Validation and deployment experience
I added tests for recovery after transient failures, retry exhaustion, permanent failures, staging cleanup, freshness behavior, concurrent request coalescing, and large stderr output.
Since deploying this retry-enabled version, I have not encountered the HTTP 502 problem again. This is my operational observation so far, rather than proof that the underlying network interruptions have stopped.
Would you accept a PR along these lines? would appreciate your preference on whether the retry count and backoff should remain fixed or be configurable.