Repository navigation
Restore Email availability after repeated startup failures #5412
Description
🤖 Generated by the Agentic Engineer
Description
Email became unavailable on 27 October. Read-only checks confirm repeated container restarts after failed health probes, and the deployed health route returns HTTP 503. The running image is present; this is distinct from the earlier image-verification failure. The cause has not been established.
Steps to Reproduce
- Run the portfolio's read-only platform health check.
- Inspect the Email workload's readiness, termination history and probe events.
- Check the deployed health route and bounded current/previous container logs.
Expected Behavior
The deployed Email service remains ready and serves its health and administration routes. A startup or runtime fault provides safe diagnostic evidence without exposing mailbox contents or credentials.
Actual Behavior
The workload repeatedly fails startup or liveness probes and remains unavailable. Current and previous container log reads return no content. Runtime observations and access limitations are retained in private operator evidence.
Acceptance Criteria
- Establish the failing phase and root cause from bounded, non-secret diagnostics; distinguish process, listener, storage and routing failures.
- Deliver the root-cause repair through current-head validation, substantive review, protected merge and deployment. Do not mask the failure by weakening probes or broadening runtime access.
- Observe stable readiness without further probe-driven restarts and successful deployed health and authenticated administration navigation.
- Preserve existing mailbox data, credentials, receiving configuration and rollback. Keep migration, delivery, encryption and recovery criteria on their existing issues.
Additional Context
Related to #5030 and #5387. No production mutation or credential read was performed during discovery. Size: Medium; diagnosis is required before selecting the repair.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
coderabbitai commentedon Oct 27, 2026
🔗 Related PRs
#5416 - fix(email): expose focused startup diagnostics [merged]
#5444 - fix(email): restore health checks and recover the stalled rollout [merged]
#5335 - fix(mail): separate recovery from serving management [merged]
#5351 - feat(mail): activate Stalwart administration and Homepage entry [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
chore: agent-claim #5412 nonce=5bea8f3c82715fd9463297f95e5467e2
devantler commentedon Oct 27, 2026
The outage now has a confirmed default-branch deployment impact. CD run 40446622865, on main
a56a2cf3b4a10ce5352f19e9a8e9da106373b00f, passed its validation jobs but failed production reconciliation at the Email readiness condition. A separate read-only live check reports that same failing condition. The deployment failure is therefore linked to this incident; it is not evidence of a separate validation defect.The startup cause remains unestablished. Diagnostics #5416 and read-only observation #5418 are in progress, and their current writers remain active. Recovery still requires the root-cause repair, stable deployed readiness and authenticated administration; no retry, probe relaxation or production mutation was performed by this investigation.
devantler commentedon Oct 27, 2026
The merged read-only observer now supplies a narrower startup result. At reviewed main
592feccb239fa74bc70ae236c6fe8b92f050dfb7, run 40450456702 returnedMAIL_STARTUP_SAMPLE=LOCAL_HEALTHY_KUBERNETES_UNREADY(exit 1). The earlier run 40449932878 returnedINCONCLUSIVE(exit 3); it established no startup phase.The second result is an observed problem, rather than a failure to build or start the observer. Under the reviewed observer contract, its single bound sample passed the local readiness check while Kubernetes readiness remained false. This narrows the next investigation to the difference between local health and Kubernetes probe behavior, including their paths, target and timing. It does not establish a networking cause, sustained availability or authenticated administration.
#5412 remains open. The current diagnostics and observation writers remain active; this investigation dispatched no additional workflow and changed no production resource or probe.
devantler commentedon Oct 27, 2026
The reviewed read-only production observation at main
592feccb239fa74bc70ae236c6fe8b92f050dfb7returnedLOCAL_HEALTHY_KUBERNETES_UNREADY. Runtime identity, public startup input checks, datastore-lock identity and loopback readiness passed with stable runtime/configuration bindings before and after; Kubernetes readiness remained false. A later read confirmed continued startup restarts, so this was not sustained recovery.The named probe port matches the management listener port. The next investigation is the exact startup HTTP request and its path/timing compared with successful loopback readiness. The observation establishes a mismatch, not its root cause. The earlier run 40449932878 was inconclusive because its bounded window fell inside restart backoff. No recovery is claimed for #5412.
devantler commentedon Oct 27, 2026
The reviewed probe/access observer now has a complete bound production result. Run 40463260157, first attempt on merged main
18f24412ee545be1fa235102085cada9aef8fb56, returned:MAIL_PROBE_PATHS=ALL_LOCAL_TARGETS_HTTP_200MAIL_ACCESS_STATE=OTHER_ACTIVE_STORED_BLOCKSMAIL_KUBERNETES_READY=FALSEThis is an observed problem (exit 1), not an incomplete sample. The fixed one-second health requests passed on loopback and the bound Pod address; the bounded administrative query completed with complete address inventories and stable runtime, node, public-input and credential-input bindings. Active stored blocks exist, but the host-address candidate does not have an effective block match under the recorded allow/expiry rules.
The next investigation must bind the failed kubelet connection to its actual source address and compare that with the application rules. Local target success does not reproduce the kubelet network path, and stored rules do not attest the application's in-memory cache. No block was cleared, allow rule added, probe relaxed or production configuration changed. #5412 remains open for root-cause repair, sustained readiness and authenticated external administration.
Earlier runs 40461825944 and 40462635726 were inconclusive; their sampling windows fell between successive container lifetimes. The successful collection was launched from an observed new runtime window.
devantler commentedon Oct 27, 2026
The extended read-only observer has a complete bound production result. Run 40471085937, first attempt at reviewed main
8a91c6e5a99da1a3dfb9a79977eaa41c3491f63c, returned:MAIL_PROBE_PATHS=ALL_LOCAL_TARGETS_HTTP_200MAIL_ACCESS_STATE=OTHER_ACTIVE_STORED_BLOCKSMAIL_CILIUM_ACCESS_STATE=STORED_CILIUM_BLOCK_MATCHMAIL_KUBERNETES_READY=FALSEThe fixed local health requests passed, while the bound Cilium candidate matched an active stored block with no matching allow override. Pod/runtime, Node, CiliumNode, targets, public configuration and credential-input bindings remained stable through collection. This is an observed problem (exit 1). The candidate matches the separately observed node routing choice; neither metadata nor stored rules alone attest an actual socket source or the application's in-memory cache.
The next repair decision needs the matching rule's origin and scope. No access rule, probe, network policy, workload or mailbox data was changed by this observation. #5412 remains open for root-cause repair, sustained readiness and authenticated external administration. Main validation passed for this observer revision; the separate release publication authorization failure is under investigation and is not cleared by that validation.
19 remaining items
chore: agent-claim #5412 nonce=3f1e937031667da141949783f4a89413
devantler commentedon Oct 31, 2026
Fresh bounded public sample, 2026-10-31 ~15:56 UTC: the administration entry returned a gateway error in 7 of 12 requests, and the declared readiness route in 3 of 4. The serving workload was ready with zero restarts for 43 hours. So the intermittent failure reported on 10-29 is unchanged. The last protected observation on record (run 40950753695) reports all local targets healthy, with a retained permanent block matching a shared cluster network address.
The only pending route on file is the coordinated client-identity activation in #5448: the adapter bundle, the provider and receiver transport settings, and the physical-peer fence. None of it is active. That activation touches the shared gateway path, so it needs its own staged rollout rather than a one-line bundle swap: selecting the adapter alone would present every client under one shared address, which the acceptance on #5448 rules out as a substitute for proven client identity.
This issue blocks several unrelated parked PRs through its old health condition. Those are being re-read individually, because the aggregate platform health check does not cover this path.
chore: agent-claim #5412 nonce=523fb41c9ebb3e204e9723fafb5f324a
devantler commentedon Nov 1, 2026
Fresh bounded observation after #5652 deployed and merged as
dbd0cd4d1a10896d293a07107eeb28e2f944fa64: the administration entry returned HTTP 503 in 7 of 12 separate requests. An earlier four-request sample returned only HTTP 200, so that smaller sample did not establish recovery. The intermittent availability incident remains open.Provider verification and the separate live physical-peer refusal check passed. They do not prove Email's original-client path or resolve this incident; the coordinated client-identity and recovery work remains on #5448. The previously approved #5586 mailbox restore trial remains unrun on this live-health hold.
devantler commentedon Nov 1, 2026
A fresh bounded sample after #5685 reached production returned HTTP 503 on all 12 administration-entry requests. The serving workload was ready, on the expected published image, with zero restarts. This is an adverse public-path observation; process readiness does not establish availability.
The availability hold remains on #5412 and on the previously approved #5586 mailbox restore trial. #5696 is still in the protected queue for the shared Gateway transition; its queue position and a successful source deployment are not acceptance evidence for Email's client-identity path.
devantler commentedon Nov 1, 2026
Fresh bounded read-only administration-entry check on 1 November at approximately 22:56 UTC: all 12 HTTPS GET samples returned HTTP 503, with zero curl/transport failures. The destination was resolved from the current scoped production HTTPRoute, not guessed from an earlier address. No private account, mail record, credential or administration operation was accessed, and no TLS check was bypassed.
Generic platform workload health was separately OK. This observation does not clear #5411’s exact availability hold or establish recovered customer-facing traffic. #5696 has a live local Claude holder; no competing repair, takeover, requeue, cancellation or production mutation was performed.
A further scoped read narrows the diagnosis: the current route is accepted and its references resolve, with a ready backend endpoint, while the actual anonymous response reports an upstream connection failure. This does not establish the network/root cause or clear the failing visitor path. No configuration, traffic or credentials were changed. The existing active recovery owner retains the work.
devantler commentedon Nov 2, 2026
Email's administration entry answered 200 on all 21 samples between about 04:36 and 04:45 UTC on 2 November, after #5711 deployed. All 12 samples returned 503 at 22:56 UTC on 1 November and again at 01:45 UTC, on the shared-address path. #5448 has the details.
The hold on the #5586 restore trial stays for now. This is about nine minutes of samples from one client, and this path has given false comfort before: an earlier sample of 4 returned only 200, then 7 of the next 12 returned 503. Sign-in has not been exercised either.
Release condition for that hold: samples spread over at least 6 hours, with no 503 and no restart of either container. A later run reads this no earlier than 10:45 UTC today and records the result here.
devantler commentedon Nov 2, 2026
Fresh bounded anonymous administration-entry samples returned HTTP 200 on all 24 GETs at 08:36–08:37 UTC and all 12 GETs at 09:02–09:03 UTC on 2 November, split between HTTP/1.1 and HTTP/2. No transport failures occurred. A separate read-only stability observation shows both serving components ready, with zero restarts and unchanged running start times since 04:16:59 UTC.
These are additional bounded samples, not continuous availability proof. The previously recorded six-hour observation condition remains in force: reassess no earlier than 10:45 UTC today. Earlier 503 cohorts remain adverse evidence. Authenticated sign-in/navigation is still unverified; the approved isolated restore remains held on this acceptance gate.
devantler commentedon Nov 2, 2026
The hold on the #5586 restore trial stays until at least 11:10 UTC. The samples meet the release condition as written, but the adapter's own log shows a failure the samples missed.
The 05:08 burst: the mail server dropped or reset the adapter's connections for about 6 seconds, then recovered without a restart. The mail container logged nothing about it. The cause is not established: this log does not record the request path or client, and no Claude session touched this address at that time. Codex sessions were not checked. A burst like this could equally be a client loading many files at once or an automated scan, so neither is assumed.
Why this keeps the hold: the condition named 503s and restarts because those were the failures seen so far. Holding the trial for a six-hour clean window and then ignoring a failure inside it would defeat its purpose. This adapter log is a continuous record and is stronger than spaced samples, so the same test is applied from the end of the burst.
New release condition: no upstream failures in this log for six hours after 05:08:55 UTC, no restart of either container and no 503 in a fresh sample. A later run reads this no earlier than 11:10 UTC and records the result here. A repeat of the burst keeps the hold and makes its cause the next thing to find.
Sign-in was not exercised in this read. No configuration, traffic or credentials were changed.
devantler commentedon Nov 2, 2026
The recorded six-hour condition for releasing the isolated #5586 restore trial is met. The earlier 21/21 HTTP 200 samples ended around 04:45 UTC on 2 November. Subsequent paired HTTP/1.1 and HTTP/2 cohorts returned 24/24 at 08:36–08:37, 12/12 at 09:02–09:03, 12/12 at 10:09, and 12/12 at 10:50, with no 503 or transport failure. Read-only serving identity captures at 09:03, 10:10 and 10:50 are unchanged; both containers remain ready with zero restarts since their 04:16 start.
Release is limited to the already-approved isolated mailbox restore trial. It does not accept authenticated administration, independent client identity, encrypted serving storage or Email delivery. This issue remains open. The earlier intermittent successes and subsequent failures remain part of the evidence; this is a time-spread observation, not a continuous availability guarantee.
Update after the 11:10 UTC read: my 10:58 release missed the later hold recorded at 10:48. That record found an upstream-error burst at 05:08:50–05:08:55 which the spaced samples did not observe, and required six hours free of further adapter errors. The 11:10 read now meets that stricter condition, including fresh 12/12 HTTP 200 samples and unchanged ready components with zero restarts. The narrow restore release stands from that later evidence. The burst's cause remains unknown and must be investigated if it recurs; no claim of continuous error-free service since 04:45 is made.
devantler commentedon Nov 2, 2026
The stricter hold on the #5586 restore trial is now also met, so the release recorded at 10:58 UTC stands. The 10:58 release did not mention the 05:08 burst and came before the 11:10 read that the 10:48 comment set, so this read settles the difference between them.
Still unknown: the cause of the 05:08 burst. This log does not record the path or client, so a repeat would need that cause found. The trial should be read with that in mind: an upstream failure during the trial could be this same behaviour and not the trial's doing.
Release stays limited to the isolated restore trial, as the 10:58 comment says. Sign-in was not exercised, and nothing was changed.
devantler commentedon Nov 2, 2026
The availability hold on the #5586 restore trial is released, read at about 11:10 UTC on 2 November. This issue stays open.
Still not known: what caused the six-second burst at 05:08 UTC. It has not repeated, so this does not hold the trial, but it would be the first thing to look at if it does.
Not covered by this read: signing in and navigating as a user. This release only says the service stayed reachable. It does not start the restore trial, and the trial's own CI, review and approval conditions on #5586 still apply.
devantler commentedon Nov 2, 2026
Fresh read at 16:53 UTC: the same serving Pod identity is retained, both components are ready with zero restarts, and the complete retained adapter log contains no additional error entries after 05:09 UTC. The native read-only serving-source guard also passes against the current declaration. The earlier startup failures and 05:08 burst remain in the evidence.
This preserves the narrow release for the isolated restore trial. It does not establish authenticated navigation, client identity, encrypted serving storage or mail delivery. The repair #5770 remains in the protected queue; its failed predecessor is not being rerun.
devantler commentedon Nov 3, 2026
A fresh bounded observation of the public administration page completed with 60/60 HTTP 200 responses and identical page content, alternating reused and fresh HTTP/2 connections from one client. The reported intermittent upstream failure did not reproduce in this window. This does not establish availability from an independent client/network or through a serving restart; those acceptance gaps remain, and the issue stays open.