Repository navigation
Production deploys time out in the data-product rollout check #5805
Description
🤖 Generated by the Agentic Engineer
Evidence
Every production deploy since 03:36 UTC on 3 November has failed at the data-product rollout check, and so has the Heal Prod re-deploy of main that follows each failure. Each failure removes the PR at the head of the merge queue. The queue holds 17 more PRs.
| Run | PR deployed | Step | Receipt |
|---|---|---|---|
| 41573251181 | #5775 | Deploy to Prod | deadline_exceeded, phase public_contract, role publisher_library, request_timeout, curl exit 28 |
| same run | main |
Heal Prod | deadline_exceeded, phase public_contract, role publisher_script, request_timeout, curl exit 28 |
| 41576071642 | #5750 | Deploy to Prod | deadline_exceeded, phase public_contract, role publisher_script, request_timeout, curl exit 28 |
- The last two good deploys (runs 41567665892 and 41561757715, at 02:41 and 01:45 UTC) finished this check in 45 and 66 seconds. Each failure above used its whole 300-second budget.
- None of the changes merged after the last good deploy touch routing, gateways or data products (fix(storage): bind historical backups across validation heartbeats #5794 and fix(storage): report known Talos allocation phases #5803 change storage code only). The Heal Prod re-deploy of
mainfails the same way, so the deployed PRs are not the cause. - The cluster side looks healthy: the data-product pods are running with no restarts, and node load is low.
- Public fetches of the same data-product pages from outside the cluster, between 04:45 and 04:55 UTC, all returned 200 but were slow and uneven: 0.3 to 6 seconds to the first byte, and one request to another public page took 18 seconds. Connection and TLS setup stayed under 0.05 seconds, so the delay comes after the edge, on the way to or inside the cluster.
Problem
- The public path to the cluster has become slow, on and off. That alone pushes a check that normally takes about a minute past its 5-minute limit.
- The receipt cannot say where the time went. It names only the last fetch that ran out of time. It does not say how long the readiness wait took, how many times the rollout was re-checked, or how long each public fetch took. So a single stalled response and many slow ones look the same.
- One stalled response can use up the whole budget. Each fetch may run until the step's deadline, so one hung connection fails the deploy even when a second try would have worked.
Related: #5170 covered dropped connections. This failure is a different one: requests are slow, not refused.
Hypothesis
If the receipt records time spent in each phase, and each public fetch has its own time limit with one retry on a timeout, the next failure will show whether the cluster or the public path is slow, and one stalled response will no longer evict an unrelated PR.
Success signal
- Short term: a deploy of
mainpasses the check again (a Heal Prod run or the next merge-group deploy). - Within two weeks, until 17 November: no merge-group eviction whose receipt shows
request_timeoutwithout a phase timing that explains it.
Acceptance criteria
- Find why public fetches slowed from about 03:30 UTC on 3 November, and fix it or record it as an outside cause.
- A failing receipt reports, as numbers only, the seconds spent waiting for readiness and in the public checks, and how many times the rollout was re-checked.
- A single public fetch has its own time limit, well below the step budget, and a timeout is retried once within that budget.
- Tests cover a slow fetch that succeeds on retry and one that keeps timing out.
Rough size: small (one script and its tests), plus the investigation.
Reacted by coderabbitai
Metadata
Metadata
Assignees
Labels
No labels
Type
Fields
Priority
None yet
Effort
None yet
Projects
- Status
Activity
coderabbitai commentedon Nov 3, 2026
🔗 Related PRs
#5598 - fix(observability): collect contract probe failure metrics [closed]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
devantler commentedon Nov 3, 2026
Two more failures since this was filed:
deadline_exceeded,publisher_script,request_timeout, curl exit 28. That removed fix(security): stage the initial runtime detection baseline #5750 from the queue after about 8 hours there.mainin the same run failed differently:public_contract_incomplete, rolepublisher_index, reasonhttp_status, HTTP 520, at 05:03 UTC. The edge returns 520 when its own connection to the cluster fails. So the public path is now dropping some requests as well as slowing them down.So the public path to the cluster is failing on and off, both as slow answers and as dropped requests. The check fails on either kind. I am preparing a change that gives each public request its own time limit and tries once more on a timeout or an edge 520–524, as it already does for a dropped connection. That makes the check tolerate a brief failure while still reporting it in
publicRetries. It does not fix the slow public path, which stays open here.chore: agent-claim #5805 nonce=39cc07da354e8d98efc6acf16554ddda
devantler commentedon Nov 3, 2026
The next deploy passed: #5776's merge-group deploy in run 41579796675 succeeded at about 05:15 UTC, and Heal Prod was not needed. So the public-path failures come and go; they have not stopped deploys completely.
devantler commentedon Nov 3, 2026
#5807 was queued at 05:35 UTC at its Codex-reviewed head
fc4b22c5bd(position 17). #5750 stays out of the queue, as its blocker record says, until #5807 merges and its deploy receipt shows the public contract passing. The root cause of the slow public path is still open here.