Repository navigation
Prevent intermittent policy-validation timeouts during reconciliation #5539
Description
🤖 Generated by the Agentic Engineer
Expected Behavior
Normal platform reconciliation validates existing policies within the admission deadline and completes without webhook timeout alerts. Preserve all policy validation, authorization and failure behavior.
Actual Behavior
Policy validation intermittently exceeds its deadline during reconciliation. Later retries complete, but the failures produce warning alerts and delay convergence. Discovery requests showed client-side waits around the failure; their contribution to the failed admission request is still unverified.
Steps to Replicate
- Capture a bounded reconciliation window with policy admission responses and discovery-client timing.
- Bind each failed admission request to its discovery work, separating client-side waits from API processing and response writing.
- Reproduce the confirmed cause with controlled inputs before changing the owning implementation.
Acceptance Criteria
- Establish the failed request's actual critical path and reproduce the cause with a regression that fails before the repair.
- Fix the owning implementation while preserving admission scope, authorization, fail-closed behavior and existing deadlines. Do not suppress the alert or increase request limits without measured justification.
- Verify complete current-head CI and review, protected delivery, and repeated normal reconciliations without this timeout.
Part of #2043
Reacted by coderabbitai
Activity
coderabbitai commentedon Oct 28, 2026
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
chore: agent-claim #5539 nonce=1c3e9fce2a9f7879967792b2635929de
chore: agent-claim #5539 nonce=05b55f2974f245acd977d08d59822277
devantler commentedon Nov 1, 2026
This recurred during ordinary reconciliation on 1 November. Policy validation exceeded its deadline at 06:38:13 and 06:38:41 UTC, delaying infrastructure and a dependent deployment. Infrastructure recovered at 06:41:55 and the dependent deployment at 06:43:00 without intervention.
The admission service remained ready with no container restarts. Bounded logs captured client-side discovery waits of about one second across the same failure window. Those discovery log records carry no admission-request identifier, and the captured policy-validation records provide no matching request duration or timeout response. This is correlation, not attribution: neither discovery throttling nor resource pressure is established as the cause.
The next useful proof is a failed request's critical path, binding its discovery work and response timing to that request. Keep the regression and repair criteria open until that proof exists; the single successful retry does not establish sustained reliability. Full observations remain in private operator evidence. No production setting, validation deadline or policy enforcement was changed.
devantler commentedon Nov 1, 2026
The timeout family recurred on 1 November. Coroot's bounded API-server messages show failed policy-validation calls at 15:25:13, 15:27:54 and 15:30:12 UTC; cancellation and handler-timeout messages also occurred at 16:02:53 and 16:04:40 UTC. These are actual event timestamps, not the older sample shown in the grouped pattern.
The subsequent read-only health snapshot has all nine nodes Ready and the API servers, controller managers and admission replicas running without restarts. This narrows the immediate recovery check; it does not clear the failed requests or establish sustained reliability.
The bounded current admission logs contain policy-validation notices but no failed-request duration or matching timeout response. Their byte bound also excludes the earlier 15:25 window. The root cause remains unassigned: this observation does not prove discovery throttling, resource pressure, a network failure or an implementation defect. The next repair still needs a request-bound critical-path capture and a regression. No monitoring filter, admission scope, fail-closed behavior, deadline, production configuration or capacity was changed.
chore: agent-claim #5539 nonce=3feccd14ca19b259a76e9fd6a2959117
devantler commentedon Nov 1, 2026
A new interruption was observed on 1 November at 17:43 UTC: policy validation exceeded its deadline, delaying infrastructure reconciliation and two dependents. A later declared live-health check returned healthy. A separately owned recovery workflow also completed during that interval, so this observation does not attribute the recovery to an automatic retry.
The timestamp-bounded admission capture now includes a response-write timeout at 17:43:51 UTC, with the same admission-request identifier on its validation notice. This is a different policy update from the failed reconciliation reported at 17:43:25, so it corroborates the timeout family without identifying that failed request. Both admission replicas were ready with zero restarts.
The capture also contains approximately one-second client-side discovery waits, but those records have no admission-request identifier. Neither a request start nor its complete duration was captured. Discovery delay, transport delay and resource pressure remain unassigned causes; a request-bound critical-path capture is still required before choosing a repair. Detailed identifiers and raw diagnostics remain in private operator evidence. This run changed no production setting, deadline or policy enforcement.
Deployment impact is now confirmed: merge-group run 41331622603 stopped in reconciliation at 17:43:55 UTC on this same policy-validation deadline, and #5681 was removed from the merge queue at 17:46:21 with
failed_checks. Its existing automatic heal was still restoring main and awaiting leftover verification at collection. The run has not proved a safe re-entry condition; do not interpret the healthy point-in-time read as a completed heal or clearance to requeue.chore: agent-claim #5539 nonce=f183509be632f6b7957127bde801002c
devantler commentedon Nov 2, 2026
Consolidating duplicate #5538 into this issue. Both describe intermittent policy-validation deadlines during ordinary reconciliation, subsequent recovery, and the same unproved root cause. This issue has the newer recurrence and deployment-impact evidence; #5538 preserves the earlier 28–29 October observations as historical evidence.
The retained work includes every earlier criterion: capture the failed request’s critical path and distinguish processing, discovery and transport delays; reproduce the attributed cause with a regression; deliver a justified repair through full validation, substantive review and protected merge; then verify repeated ordinary reconciliation and dependent readiness. Preserve admission enforcement and deadlines. A healthy point-in-time snapshot or one successful retry does not satisfy the reliability outcome.
This consolidation does not establish a fix or assign a cause. #5539 remains open.