Repository navigation
Reduce CI critical-path time while preserving validation #5495
Description
🤖 Generated by the Agentic Engineer
Problem
Recent successful merge-queue runs take 28–29 minutes, delaying delivery and recovery while each change waits for the next deployment.
Baseline
Runs 40639023721, 40641879084 and 40644713318 took 1,732, 1,684 and 1,751 seconds from first job start to last completion. Deployment took 989–1,044 seconds. The publication-generator validation job took 557–709 seconds, and repository guards took 528–608 seconds. These are measurements, not a demonstrated optimization.
Acceptance
- Measure individual validation helpers and the dependency path before choosing a change. Identify redundant work or independent, isolated fixtures that can run concurrently with bounded resource use.
- Preserve every required check, negative control, trust boundary and deployment health check. Failures, timeouts and incomplete background work must still fail the required aggregate.
- Keep logs attributable to each helper and retain an equivalent serial path for comparison and diagnosis.
- Demonstrate a material improvement on comparable runs, report validation and total critical-path time separately, then verify the protected default-branch run and production recovery.
- Use existing execution capacity; changing paid capacity or financial settings is outside this change.
Part of #2043
Reacted by coderabbitai
Metadata
Metadata
Assignees
Labels
No labels
Type
Fields
No fields configured for Performance.
Projects
- Status
Activity
coderabbitai commentedon Oct 28, 2026
🔗 Related PRs
#5366 - perf(ci): partition complete offline regressions [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
chore: agent-claim #5495 nonce=8e45752493a8c6e6880be2aa6113b347
devantler commentedon Oct 28, 2026
#5510 merged as
db46da4e3c71a26c7310775b0e9d4f4a5204425c. The protected run and post-merge Validate Main passed, including the production convergence check.The bounded publication runner improved the matched local nine-fixture execution from 742s serial to 406s with two isolated workers. In the protected run, the publication validation job took 451s versus baseline job times of 557–709s; its actual fixture runner took 390s. All nine fixtures completed successfully.
The overall readiness result is not yet an improvement. A fresh analysis of the actual workflow dependencies and complete native job timestamps puts the required-gate path at 2,742s, versus 1,657–1,762s for the same ready-to-required-completion measure in the three baseline runs. Its longest observed path was change detection → Helm post-renderer validation → production deployment → required aggregate: 1,594s ready-to-start waiting, 1,147s execution, plus a one-second transition. The post-renderer job alone waited 865s before its 192s execution. Deployment waiting can also include the protected concurrency lock; these timestamps do not establish a hosting-capacity cause.
The later queue-membership observer extends the full run to 3,107s but intentionally does not block the required gate or merging. It is excluded from the next readiness optimization target. This issue stays open: preserve all validators and deployment checks while reducing avoidable waiting or repeated execution within existing capacity.
chore: agent-claim #5495 nonce=86e6545272332256da35dd8656ee442d
chore: agent-claim #5495 nonce=453530848e0c147a5695a550c20cd4cf
16 remaining items
chore: agent-claim #5495 nonce=953cdc2ebdc26d511a406d6dc7329f08
devantler commentedon Nov 2, 2026
Hosted timing after #5700, all successful merge-group runs on
main:The original baseline was 1,684–1,751 s in total, with 989–1,044 s of deployment. These four are lower, but they span different source and deployment changes, so they do not show that #5700 caused it. Run 41367259048 also lost about 100 s to later job starts than its siblings, and #5693 changed the Kyverno rate limit just before it.
The guard chain is no longer the last gate. In 41362850036 the jobs finish at 592 s (Publication Generator), 502 s (Complete Go Regressions) and 413 s (Repository Guards), and deployment starts at 597 s.
Next slice: #5710. The publication fixtures total about 864 s on a four-vCPU runner limited to two workers. Locally, all nine passed with four workers in 278 s, against 406 s recorded for two workers on #5510. The longest fixture is about 300 s, and the Go job then becomes the gate, so I expect about 90 s off the critical path. That is an estimate until hosted runs show it.
Deployment time has no single dominant step in 41367259048: its largest gaps are about 60–75 s each. This issue stays open for comparable runs after #5710.
devantler commentedon Nov 2, 2026
Hosted timing after #5710 raised the publication fixtures to four workers. Two successful merge-group runs so far:
The original baseline was 1,684–1,751 s in total, 557–709 s for the generator and 989–1,044 s of deployment. Both runs are below all three ranges.
This does not close the issue yet. Two samples are few, the two generator times differ by 118 s, and the deploy steps ran different changes. Between these two, run 41385643976 (#5712's group) failed in the same generator job, with no stated reason; #5716 tracks it. Until that failure has a known cause, it counts against four workers as a possible cost, and I am not treating it as a flake.
chore: agent-claim #5495 nonce=f6571b61a7ff829082300a862b94bb4f
devantler commentedon Nov 2, 2026
A new adverse sample is CI run 41433154968, Publication Generator, at
787c934b066780e0155fb6b3940972f7fd6f56bc. The approved-revisions fixture failed while moving a release candidate; the existing output remained unchanged, and the shared lookup reported a broken pipe. Its per-call refusal log was discarded, so the exact reason for that fixture failure is still unproven.I separately reproduced a defect in the real shared lookup under
pipefail: looking up the first of 20,001 synthetic rows returns the correct answer and exit 141. The reader exits after its first match while its producer still has data. I am preparing a complete-input reader with first-match, missing-key and numeric-looking-key controls, plus retained diagnostics for the failed generator case. This is evidence for that lookup defect, not proof that it caused the earlier #5716 sample or that four workers are safe under every failure mode. Worker limits and required failure gates remain unchanged.devantler commentedon Nov 2, 2026
The measurement workflow has shipped in #5727. Its first main invocation passed all four arms at the same source: Go 4/8 workers took 556.95/520.90 seconds, and publication 2/4 workers took 600.16/354.39 seconds. All nine publication fixtures passed in each arm. The delivery receipt records CPU cost, exact named outcome reconciliation and the stdout-record gaps. One pair does not establish whole-pipeline causality or retire the adverse samples.
The lookup repair I mentioned earlier is on hold: #5739 is already in the protected queue, and the related shared library has another delivery owner. I removed my empty, unmodified repair worktree rather than creating a competing implementation. The failed #5750 publication fixture is retained and has not been blindly rerun.
The main Go profile points to the credential-refresh regression package as the remaining long pole: 466 seconds with four workers and 434 seconds with eight; the next slowest package took 53/63 seconds. Profiling that fixture's process and execution costs is a stronger next candidate than increasing worker limits again. No validation or capacity change is made by this observation. This issue stays open for reliability and outcome verification.
devantler commentedon Nov 2, 2026
Profile of the slowest validator, and why runner wait is the bigger target. Nothing was changed by this reading.
The Go long pole has no sleeps to remove. I traced one case of the
refresh-flux-ghcr-authpackage locally, which is macOS rather than the CI runner, so the proportions are indicative only.kubectlrenew_sync_leaseTestAfterReleaseRegressionDetectsEachAblatedProductionFencepins that check, so it is a safety control and I did not propose removing it. Merging thejqanddatecalls in that function would save about 8% of one job, which is not worth changing a fence-critical script.Waiting for a runner costs far more than this job.
This is one run during a busy period. It shows waiting can dominate, not how often it does. The one wave source I have sized is now monorepo#4775: about 1,800 jobs a day from submodule bumps.
Next action here: read the queue's wait against its run time after #4775 lands, before tuning any more validators.
devantler commentedon Nov 3, 2026
The dependency grouping in monorepo#4775 has shipped. A first read of two successful overnight protected runs shows detection starting 5 and 16 seconds after run creation, versus 983 seconds in the earlier busy-period sample.
These are different source revisions and off-peak samples, so they do not prove a causal speedup from grouping or replace the planned November 9 observation. Failed deploy samples remain adverse evidence. The complete Go step is still about 10½ minutes; the latest native output attributes 493.878s to the existing credential-refresh regression package. No safety check, worker limit or execution capacity changed in this follow-up.
The next measurement should distinguish runner acquisition, prerequisite/concurrency waiting and execution before selecting another optimization. This issue remains open for comparable delivery and reliability evidence.
chore: agent-claim #5495 nonce=4b5a0110686fbeccff799f8e554ff289
chore: agent-claim #5495 nonce=7904521eadca3eab9100b91f6a0aa325
devantler commentedon Nov 3, 2026
The performance changes are consolidated in #5841 to require one protected queue deployment. The isolated bridge step was observed at 13s after duplicate removal versus 408s before; the two complete Go partitions passed natively with test-step times of 325s and 193s. A companion unpartitioned run took 691s. Runner conditions varied, so these are validation-step observations, not a controlled end-to-end throughput result.
Delivery remains unfinished: the combined head is in fresh CI and review has not been requested while its required checks are unsettled. GitHub's current primary-language classification admits the organization Go workflow, exposing broader repository-scan failures tracked in #5843. #5846 supplies a locally validated first compatibility repair; the existing #5785 owns lint/dead-code fixes and is already in the protected queue. #5838 was closed only because its complete change is now in #5841.
Keep this issue open until the required baseline is repaired, current-head CI/review pass, the protected merge and default-branch verification finish, and comparable whole-queue measurements demonstrate the intended outcome.