Repository navigation
Go coverage check intermittently runs the GHCR-auth refresh tests out of time #5862
Description
What happens
The organization's required ✅ Validate Go Project workflow runs this repository's whole Go test suite twice: once plainly (🧪 Test) and once with -race -coverprofile (📊 Code Coverage). Neither passes -timeout, so each test package gets Go's default 10-minute budget.
scripts/tests/refresh-flux-ghcr-auth uses most of that budget, so the coverage check fails intermittently:
| Run | Job | Package time |
|---|---|---|
#5853 at 081cbeae |
🧪 Test | 362s |
#5853 at 081cbeae |
📊 Code Coverage | 470s (passed) |
#5853 at 3ddf5e61 |
📊 Code Coverage | 592s (failed: the last running case hit the deadline derived from the package budget) |
Locally (10 CPUs) the same package takes 254s under -race.
Why
- The package runs the 7,000-line
refresh-flux-ghcr-auth.shhelper about 600 times across 338 tests. One helper run costs about 2 CPU-seconds, mostly spent starting processes (Bash subshells and the fake commands). - The race detector doesn't slow this package directly: its fakes already run from an uninstrumented binary.
- What slows it is CPU contention. In the coverage job it shares a 4-CPU runner with race-instrumented packages such as
validate-eks-ci-role-policy(75s plain, 363–441s with-race) andmigrate-longhorn-v2. The whole job is CPU-bound, so splitting the package would not help.
Options
- Cut how often the full helper runs: for example, test the table-driven message classifiers (such as
TestAmbiguousRuntimePullFailureDoesNotBootstrap, 9 full runs) at the function level, keeping one full end-to-end run per behaviour. - Make each helper run cheaper, for example with a dedicated minimal fake-command binary instead of re-executing the test binary.
- Pass an explicit
-timeoutin the organization workflow's test and coverage jobs, as this repository's own regression job already does. This changes devantler-tech/.github and every Go repository that uses the workflow.
Done when
-
scripts/tests/refresh-flux-ghcr-authfinishes in well under half of its package budget in the coverage job, measured on the organization runner. - Several consecutive coverage runs pass without a rerun.
🤖 Generated with Claude Code
Metadata
Metadata
Assignees
Labels
No labels
Type
Fields
Priority
None yet
Effort
None yet
Projects
- Status
Activity
devantler commentedon Nov 4, 2026
Measurements, 4 November 2026
How often it fails: 2 of the 4 coverage runs on #5853 hit the deadline (
3ddf5e61: 592s;d1c9ed44: 593s) and 2 passed (470s and one under budget). Every case still waiting failed withfixture command did not finish within its test budget: context deadline exceeded.Where it does not bite: the organization workflow skips its test and coverage jobs on
merge_group("these checks already passed on the PR"), so this cannot evict a queued pull request or trigger a production heal. The cost is a re-run of the coverage job on pull requests.Cost of the package (local, 10 CPUs, plain
go test -parallel 8): 312s wall, 1,494 CPU-seconds (623 user, 871 system). Time is a long tail: the slowest 25 of 337 tests account for 27% of summed test time, and the slowest takes 33s.Inside one helper run (
TestForcedRefreshRejectsFailedAPIEvidence/empty-acknowledgement, traced): 1.35s wall, of whichjq(called by the helper itself)kubectlcurlyqThe cost is process creation, not computation. A trivial Go binary starts in about 3.8ms against the fake's 4.6ms, so option 2 (a dedicated fake binary) would save only about 4% of the package. The
jqcalls are inscripts/refresh-flux-ghcr-auth.sh, so reducing them is a change to the production helper.Ruled out:
.go-test-execution.jsonaccepts onlyversionandparallelism, and its-parallelapplies to every package. More slots would also take CPU from the serial race packages that share the runner (validate-eks-ci-role-policyalready takes 446s under-race).TestMaindoes not work:go testkills the binary at its own deadline plus one minute. Bound Go test parallelism in required instrumented runs .github#865 also records that changing the test deadline is not the intended fix.What remains: fewer full helper runs, for example table cases that run the whole helper only to check how it classifies a message, or fewer
jqprocesses per run in the helper. Either needs about a 40% cut to keep the coverage job comfortably inside its budget.🤖 Generated with Claude Code
chore: agent-claim #5862 nonce=6be7f6ce848c26355e2ca68a67f0e603
devantler commentedon Nov 9, 2026
Measurements after #5937 (eight Go test workers), 9 November 2026
refresh-flux-ghcr-auth2227f097test-parallelism: 8)3f6ae1bb(job,GO_TEST_PARALLELISM: 8)Both runs pass, but both sit inside the 470–593s range recorded here before the change. The bound on workers did not cut the package time. The first Done-when item (well under half of the 600s budget, so under 300s) is still unmet. The second (several consecutive passes) has two passes so far, which is not enough to call it.
What is left is unchanged from the 4 November analysis, and both options need roughly a 40% cut:
TestAmbiguousRuntimePullFailureDoesNotBootstraprun the whole 7,257-line helper once per message (9 runs) only to check how it classifies that message. Testing those cases at the function level needs the helper to be sourceable. Today it runs its main flow at the top level, with noBASH_SOURCEguard.jqprocesses in each helper run. These are about 35% of one traced run (0.47s of 1.35s), and changing them changes the production helper.Neither change is started. Each touches either the production GHCR-auth helper or how its 338 tests run, so each needs its own reviewed slice. This issue stays open.
devantler commentedon Nov 9, 2026
Where the time goes, 9 November 2026: option 1 alone cannot reach the target
I measured the package at
main(dc4d2520) locally:go test -count=1 -parallel 8 -json, 10 CPUs, no-race. The goal was to check whether option 1 (fewer whole-helper runs for table cases) can deliver the ~40% cut this issue needs.TestTalosFailureAfterSafeFanoutKeepsRootAuthUnchanged, no subtests)What this means. Table cases that run the whole helper only to check how it classifies a message cover about 6% of the package's test time. Moving every one of them to function-level tests would still leave the coverage job near today's 455–485s. The cost is spread across hundreds of single-scenario end-to-end tests, and each of those exercises a different safety path that should keep its end-to-end run.
Decision. Option 1 is not the main fix, so I am not opening a pull request for it. Two levers remain that can reach ~40%:
jqprocesses per helper run, about 35% of one run's wall time. Many are separate reads of one state file (for example.status.phaseand then.status.containerStatusesfromruntime_probe_state_file). Merging those into onejqcall per file would remove most of them. This changes the production helper, so it needs its own reviewed slice with behaviour-preserving tests. It is now the recommended next step.-timeoutin the organization workflow). Bound Go test parallelism in required instrumented runs .github#865 already records that changing the deadline is not the intended fix, so it stays a fallback.The Done-when criteria are unchanged.
chore: agent-claim #5862 nonce=9900717f998ed7c4c94d7389b812f616
devantler commentedon Nov 9, 2026
Three more coverage passes with eight workers, 9 November 2026
These are the required 📊 Code Coverage runs on pull requests since the 06:21 UTC note. Each one logs
GO_TEST_PARALLELISM: 8.refresh-flux-ghcr-authpackage timec3a19589830f3c54e78ffa53That makes five passes in a row with eight workers, counting the two runs in the earlier table, and no failures. I read every pull-request CI run since #5937 merged at 04:58 UTC (33 runs); in the rest the job was skipped.
This does not meet the first Done-when item. The slowest pass took 578s, which is 96% of the 600s package budget. The spread (445–578s) is the same as before the change, so the passes say more about runner load than about margin. The fix still needs the ~40% cut described above.