Repository navigation
Policy validation timeouts interrupt infrastructure reconciliation #5538
Description
🤖 Generated by the Agentic Engineer
Expected Behavior
Ordinary production reconciliation completes with policy validation enforced. A healthy admission service must not intermittently prevent infrastructure and its dependent deployments from reconciling.
Actual Behavior
On 28 October 2026, the declared live-health check changed from healthy to unhealthy during an ordinary reconciliation. Infrastructure failed because policy validation exceeded its deadline, and a dependent deployment reported that its prerequisite was not ready. Bounded admission diagnostics independently recorded two response-write timeouts while admission readiness remained healthy.
The existing controller retried without an intervention. Reconciliation subsequently progressed, and the declared live-health check returned healthy. This is a recovered reconciliation interruption; no user-facing outage, sustained recovery, or root cause has been established. Raw runtime diagnostics remain in private operator evidence.
Steps to Replicate
- Observe the next ordinary production reconciliation through the existing live-health check; do not inject a failure or force a production change.
- If the same timeout recurs, correlate the failing reconciliation phase with bounded admission latency, controller and resource observations.
- Distinguish request-processing delay, transport failure and resource pressure before selecting a repair.
Acceptance Criteria
- Attribute the captured timeout with bounded, non-secret evidence, or record what additional observation is needed to distinguish the remaining causes.
- Deliver any justified root-cause repair through the normal tested, reviewed and protected source path. Preserve admission enforcement and required checks.
- Verify subsequent ordinary reconciliation and dependent readiness; distinguish a single recovered sample from sustained reliability.
Additional Context
Part of #2043. This interrupts deployment convergence and warrants diagnosis, while the observed automatic recovery limits the immediate impact. Diagnosis is required before estimating a source repair. No workload, policy, credential, timeout setting or production configuration was changed during this investigation.
Activity
chore: agent-claim #5538 nonce=9c23fadcd38fad99462c8d04e5ff4ff9
chore: agent-claim #5538 nonce=8c10a04f094fb65dba8e912bd9145d12
devantler commentedon Oct 29, 2026
The timeout recurred during ordinary reconciliation on 29 October: infrastructure reported policy-validation failure at 08:08:06 UTC, then completed successfully at 08:11:24 UTC without intervention. The declared live-health check is being verified separately; this single recovery does not establish sustained reliability.
Both admission replicas remained available with no restarts, and both service endpoints were ready. The bounded admission log sample independently contains a response-write timeout at 08:08:34 UTC on a different policy update. That corroborates admission-path timeouts in the same window; it does not identify the failed reconciliation request or distinguish processing delay from transport delay. Point-in-time resource usage does not establish the absence of earlier pressure.
Request-linked latency and transport evidence is still needed before selecting a repair. Root cause remains unverified, so this issue stays open. No policy enforcement, timeout, production configuration or workload was changed.
devantler commentedon Nov 2, 2026
Closing this duplicate in favor of #5539, which tracks the same intermittent policy-validation timeout and includes the later recurrences and confirmed deployment impact. Its consolidation note preserves all diagnosis, regression, protected-delivery and repeated-reconciliation criteria from this issue. The earlier observations remain here as linked evidence.
This is a duplicate disposition, not a claim that the timeout is fixed. The actual repair and sustained verification remain open in #5539.