Skip to content

ci(setup-nix): disable HTTP/2 for Nix downloads - #5688

Merged
matthewgrossman merged 4 commits into
mainfrom
ci/cachix-download-flake/matthewgrossman
Nov 18, 2026
Merged

matthewgrossman merged 4 commits into
mainfrom
ci/cachix-download-flake/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Summary

Set http2 = false for the CI Nix daemon to reduce a recurring Nix download flake (HTTP error 416, then no substituter that can build it).

  • How often it happens: over the last 4 weeks, 53 job attempts hit the 416. About 1.3% of jobs that download from a binary cache had to resume at least one download. It is getting worse: 9 jobs hit the 416 on 11-16 and 14 on 11-17, against 0–4 a day before. The retry-once wrappers in setup-nix and e2e-suites usually absorb it. When both tries hit it, the job fails, as in job 134410447281 today.
  • The real fix is upstream in Nix: libstore: finish already-completed downloads after HTTP 416 NixOS/nix#16722 makes Nix treat this 416 as a completed download. It is open, and Backport NixOS/nix#16722: completed NAR download fails with HTTP 416 on resume DeterminateSystems/nix-src#750 asks Determinate to backport it. On cache.nixos.org HTTP 416 NixOS/infra#1106, Evan (@elezar) had already diagnosed the same root cause from our CI. Now we only need to wait for the fix to land in a Determinate release, then bump determinate-nix-action.
  • Why this helps until then: every 416 in the window began with an HTTP/2-only send error. Turning HTTP/2 off should remove the trigger and reduce flakes.
  • It doesn't make anything worse: an A/B test shows the same download speed with and without HTTP/2, and no new failures.
  • We'll know quickly if it works: about 40 HTTP/2-type download errors a day show up across CI today. If they disappear within a day of normal traffic, the change works.

Related Issue

No issue; this fixes a flaky CI failure. Example: job 134410447281 failed both of its nix build attempts this way.

The failure sequence:

  1. curl reports "Failed sending data to the peer" after a NAR has fully arrived.
  2. Nix resumes the download at an offset equal to the file size.
  3. Cachix correctly answers 416 Range Not Satisfiable.
  4. Nix treats the 416 as fatal and fails the substitution.

Over the last 4 weeks, 53 job attempts hit this. In all 58 failed downloads, the resume offset equals the NAR size. The retry-once wrappers in setup-nix and e2e-suites absorbed all but one of those attempts, but the rate rose to 9–14 a day on 11-16 and 11-17.

Changes

  • Add extra-conf: http2 = false to DeterminateSystems/determinate-nix-action in .github/actions/setup-nix/action.yml.

Why this should help:

  • Every resume at EOF in the 4 weeks of logs started with CURLE_SEND_ERROR ("Failed sending data to the peer").
  • During an HTTP/1.1 download, the client sends nothing, so that error can't happen there. Over HTTP/2, the client keeps sending flow-control frames (WINDOW_UPDATE) while it reads, including after the last bytes arrive.
  • One failed HTTP/2 connection also fails every NAR multiplexed on it: 73% of resumes come in same-second bursts.
  • With separate HTTP/1.1 connections, a failure stays confined to one transfer.

Upstream

Nix needs to fix this. The fix is proposed but not merged or released:

Once a Determinate Nix release includes the fix, we can bump determinate-nix-action and decide whether to keep this setting.

Testing

  • A/B experiment in run 44199998177: 24 ubuntu-24.04 jobs each fetched the full openshell-test-runner closure (581 paths, 2.7 GiB) into fresh stores, alternating http2 = true and false. That's 120 fetches per arm, with no failures in either arm.
    • The median fetch took 33 s in both arms (p90: 40 s vs 41 s), so there's no throughput cost.
    • nix config show reported http2 = false, and the HTTP/1.1 arm had no HTTP/2 framing errors.
    • EOF 416s occur in only about 0.6% of fetches, so this sample is too small to show them dropping. Production will show it quickly: about 40 HTTP/2-type resumes a day today, and 29% of Branch E2E runs see at least one. If they stop within a day of normal CI traffic, the change works. If 416s persist, revert.
  • Every setup-nix job in that experiment ran with this change and passed.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

A failed HTTP/2 connection fails every NAR multiplexed on it. Nix resumes NARs that had fully arrived at EOF, gets HTTP 416, and fails the substitution (NixOS/nix#16722). Separate HTTP/1.1 connections confine a failure to one transfer.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 18, 2026
@matthewgrossman
matthewgrossman added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit 9f3c956 Nov 18, 2026
95 checks passed
@matthewgrossman
matthewgrossman deleted the ci/cachix-download-flake/matthewgrossman branch November 18, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants