Verify Email mailbox-data recovery before encrypted-storage cutover

🤖 Generated by the Agentic Engineer

Problem

The operational encryption key has passed independent recovery (#5540), and the native encrypted-volume canary passed (#5543). Before moving company Email data, an actual mailbox-data backup must also be restored and checked. A completed backup or storage-health check alone does not establish data recovery.

Expected outcome

Provide an explicitly confirmed, protected recovery test that restores the existing Email backup into separately owned test storage. Reuse the deployed backup service and its established custody. Preserve the serving claim, original backup and all recovery material. Keep configuration and runtime identities bound to the reviewed revision.

Acceptance criteria

Validate the deployed backup version, completed backup, complete Email volume record and source identity. Refuse incomplete, changed, ambiguous or mismatched data.
Restore into invocation-owned isolated storage without replaying arbitrary backed-up workloads, executing backup-provided hooks, changing production accounts, publishing routes or sending real messages. Never restore over or mount the serving claim.
Open the restored data with the pinned Email server and verify meaningful account and mailbox evidence privately. Preserve messages and legacy encryption dependencies; do not report backup or restoration success from empty data or storage readiness alone.
Test source/target confusion, missing and corrupt data, uncertain controller outcomes and ownership/cleanup failures. Print only closed outcomes; keep credentials, message data and internal recovery identifiers private.
Complete current-head tests and review, protected merge, the actual confirmed recovery exercise and verified cleanup. Production encrypted-storage selection and mailbox cutover remain separate reviewed changes.

Part of #5387. The service stays on its retained serving claim until data recovery and cutover acceptance pass. The test is disabled by default and does not change DNS, mail clients or delivery routes.

🔗 Related PRs

#5533 - feat(email): move active credentials and recovery to infrastructure [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

chore: agent-claim #5578 nonce=570007cf0e07b34533d9a461d59038cd

chore: agent-claim #5578 nonce=563c7093a83e8a371f4162e73d34ba09

chore: agent-claim #5578 nonce=0d49ea49f3f8ddf4d0e58ee239e7019c

🤖 Generated by the Agentic Engineer

The restore trial has not run yet, read at 14:50 UTC on 2 November. #5586 merged at 14:07 UTC, and Verify Email Mailbox Data Recovery is active on main with no runs.

The Claude lane did not start it, for two reasons:

The trial may run only once, and a Codex thread on this work may be about to start it. A second run would break that limit.
The approval is recorded only in disclosed agent comments, first on feat(email): verify live mailbox backup recovery #5586 at 20:07 UTC on 29 October. I could not read the maintainer's original words, so I cannot confirm its exact scope.

Next: the lane that holds the original approval runs the workflow once from main, or links that approval here so the other lane can. Check this workflow's run list immediately before dispatching.

chore: agent-claim #5578 nonce=d293335383a57c91a46c785344357d67

🤖 Generated by the Agentic Engineer

The original interactive approval was available in this session. The authorized first trial 41466671018, first attempt at validated main 12ee9ff5251604920949b3eccdd0b42b3d60940d, ran and stopped with EMAIL_DATA_RECOVERY=UNVERIFIED stage=backup cleanup=RETAINED. Do not duplicate or rerun that failed invocation.

Read-only diagnosis established two concrete driver defects. A bare Kubernetes Backup lookup selects a different installed API group and hides the Velero object that the driver created. The trial's backup also excluded its mounted claim; native controller evidence contains only a scratch-volume receipt, so the datastore backup is unproven despite the controller's Completed phase. This does not change the ordinary backup policy.

The repair explicitly selects the Velero API group and includes the related claim in this narrowly selected backup. The new regressions failed before the change and pass with it; the corrected resource lookup is also being exercised read-only against the retained native receipts. Keep storage activation held until a reviewed repair is merged and a new confirmed trial proves the actual data restore and cleanup. The current interactive direction authorizes continuing those repairs through the normal delivery gates.

chore: agent-claim #5578 nonce=c9f8bb723e8aff057c6fbc5ce2864b7f

chore: agent-claim #5578 nonce=c969590c5202c5d255d44d110f784d8a

chore: agent-claim #5578 nonce=269e3d86e7aba2298f56ffe26b7d5168

chore: agent-claim #5578 nonce=8a306e663c95e2000cf6eb4d947c6992

🤖 Generated by the Agentic Engineer

The confirmed fresh restore run 41561215415, from merged revision 7e4486cad02c326004ac9a1a637bb5205dbfe76a, is unverified: it stopped at the mailbox-proof stage and retained its diagnostic resources. The fresh backup and byte transfer completed; the retained reader is running without restarts and matches its reviewed workload declaration. This does not prove the restored message is readable.

No second trial or serving-storage switch was attempted. The current failure output does not distinguish forwarding, readiness, account-session and message-read failures. Preparing a fixed, read-only protected diagnostic for this retained invocation before deciding a repair or another trial.

chore: agent-claim #5578 nonce=9215289af5584fb558d3c13d2b327707

🤖 Generated by the Agentic Engineer

Further diagnosis of the retained failed trial has ruled out one candidate: an isolated copy of a synthetic datastore, using pinned Stalwart 0.16.27, still returned HTTP 200 for readiness and an authenticated mailbox session after its referenced certificate files were removed. The original synthetic source remained byte-for-byte unchanged. That warning alone does not explain the failed mailbox proof.

A startup race remains plausible: the current recovery driver waits for Pod Running before opening its temporary connection, while server startup can finish later. The pinned Kubernetes forwarding implementation closes its stream on a remote connection error. The original forwarding error was discarded, so this is a hypothesis rather than a confirmed cause of the failed invocation. The read-only diagnostic in #5802 remains queued; preserve the retained restore and do not repeat the restore trial until its result is assessed.

chore: agent-claim #5578 nonce=7066e8ecab1abbde84f48b973c132455

chore: agent-claim #5578 nonce=407bc790296686fa09b8f7cb6139144a

🤖 Generated by the Agentic Engineer

The protected read-only observer from merged #5802 passed at ee240d7f7b0920f88513627d08ed45d18e51dc5b: run 41677453033, job126064952491. Its actual receipts report HTTP readiness200 and EMAIL_RECOVERY_OBSERVATION=PASS readiness=VERIFIED session=VERIFIED writes=0.

This proves current readiness and account-session access only. Recovered-message content, owned disposal, a fresh recovery exercise and production storage cutover remain unverified. Draft #5839 prepares the separate read-only parity and UID-fenced disposal path; no cleanup has run. Its independently witnessed object inputs are held in protected production settings, with no new service credential.

🤖 Generated by the Agentic Engineer

The maintainer has now connected the existing account to Apple Calendar. Native account settings and calendar collection discovery were observed in the interactive session.

Future recovery and production-storage cutover checks must preserve calendar and task collections, their metadata and account settings as well as mailboxes and messages. Require a fresh backup of the complete serving store and private readback of the restored collections before switching storage; a message-only proof does not cover the new client use.

The retained diagnostic-copy disposal in #5839 remains an independent prerequisite and must preserve the serving store. Company Email transport, encrypted-storage cutover and the broader recovery acceptance remain open.

Navigation Menu

Repository navigation

Verify Email mailbox-data recovery before encrypted-storage cutover #5578

Problem

Expected outcome

Acceptance criteria

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Verify Email mailbox-data recovery before encrypted-storage cutover #5578

Description

Problem

Expected outcome

Acceptance criteria

Activity

coderabbitai commented on Oct 29, 2026

devantler commented on Nov 2, 2026

devantler commented on Nov 2, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions