Synchronize website-sandbox/database-url with the weekly RDS master rotation

[alexeygrigorev]

Evidence

The 2026-08-14 development outage tracked by #160 was caused by the RDS-managed master credentials of website-sandbox rotating on their 7-day schedule ("Reset master credentials" at 2026-08-14T13:12:27Z) while the application secret website-sandbox/database-url remained a static out-of-band copy. Web and worker tasks then failed password authentication failed for user "dtc_admin", /health/ready went 503, and every CI run failed at the deploy/capture gates with web captured terminal counts differ. Recovery on 2026-08-16 re-synced the secret manually (#160 recovery comment). The exact sandbox account, region, and RDS ARN are pinned in _docs/runbooks/development-release.md.

The managed rotation repeats every 7 days — next expected around 2026-08-21 — and will desync the static secret again unless a durable mechanism lands. The #160 recovery explicitly deferred this to its own issue and decision.

Owner decision required

Choose the durable mechanism; any of these is acceptable if it meets the acceptance criteria below:

1. scheduled automatic re-sync of website-sandbox/database-url from the managed rds!db-* secret (bounded, redacted, fail-closed) ahead of task refresh;
2. web/worker/migration tasks read the managed secret directly at task start;
3. change or disable the managed rotation cadence with documented compensating controls.

Scope

Implement the chosen mechanism so development tasks keep authenticating across rotations without a manual operator step, fail loudly when synchronization cannot be proven, and add a detection signal for authentication-failure storms (log-based alert or CI gate) so a future desync is attributed in minutes rather than days. Update the development runbook with the chosen mechanism, its verification, and its failure attribution path.

Non-goals

• No weakening of readiness, capture, or deploy fail-closed gates; no fallback credentials or fabricated data.
• No RDS resizing or engine change, no production-account work, and no Terraform/IAM change beyond what the chosen mechanism strictly requires (such a change gets its own review).
• No credentials, connection strings, or provider payloads in issues, comments, logs, or CI output.

Acceptance criteria

• The owner records the chosen approach and its tradeoff in this issue before implementation starts.
• A rotation-shaped verification proves tasks authenticate after a credential rotation without a manual secret update (safely staged rotation, or an equivalent contract test of the sync mechanism).
• If synchronization cannot be proven within a bounded window after a rotation, the failure is loud (alert or red gate) and never silently continues serving stale credentials.
• The Restore development database readiness for web.dtcdev.click #160 desync failure mode is reproduced or simulated and shown to be prevented or alerted by the mechanism.
• Repeated password authentication failed events on /ecs/website-sandbox/web and /ecs/website-sandbox/worker have a detection path with a bounded attribution procedure recorded in the development runbook.
• Development deploy/readiness gates keep their exact fail-closed success criteria; none is relaxed.

Dependencies

• Restore development database readiness for web.dtcdev.click #160 (recovery evidence and runbook identifiers). No application-code dependency.

Scenarios

• Operations: simulated rotation during a maintenance window with DEVELOPMENT_AUTO_DEPLOY=false, then proof that the next deploy/readiness cycle is green without manual secret edits.
• Repository: contract tests for the sync/alert mechanism run in CI without secrets present.

[alexeygrigorev]

PM readiness disposition — BLOCKED

Read-only audit on 2026-08-20: #191 is sufficiently specified to preserve as a groomed issue, but it cannot enter engineering. It has no needs-grooming label and no #191 worktree or branch.

The blocker is the unresolved owner choice in acceptance criterion 1 between three incompatible rotation mechanisms. The website currently reads DATABASE_URL from the ECS-injected runtime environment at process start; durable synchronization, task-definition or rotation changes, and rotation-shaped verification cross the DataTalksClub/aws-infra and authorized AWS Secrets Manager/ECS boundary. #160 is closed and supplies recovery evidence only.

Next owner/action: the service/AWS infrastructure owner must choose and record the mechanism and tradeoffs in this issue, name the implementation and operations owner, and authorize the redacted sandbox rotation verification. After that decision and scope freeze, hand the issue to engineering. No AWS, secret, or protected-environment action was taken in this audit.

[alexeygrigorev]

Read-only follow-up — predicted desync now observable

Audit on 2026-08-22 found materially new external evidence since the PM disposition:

• The public liveness endpoint is 200 on release 20260821-062421-11b2bd1; readiness is 503 with configuration=ok, database=error (database unavailable), and migrations=error (database unavailable). /courses and /sitemaps/courses.xml both return 500. This matches the previously recorded static-secret desynchronization symptom after the expected weekly rotation, but the RDS event itself was not re-read because the AWS gate is closed.
• CI run 32454238498 completed its deployment job successfully at 2026-08-21T07:28Z for the 11b2bd1 release, including read-only smoke and a successful-release artifact. This establishes that the release path was green before the current readiness failure.
• The later CI run 32517486416 failed at resolve-release; capture, publish, and deploy were skipped. It is not recovery evidence.
• The repository-level DEVELOPMENT_AUTO_DEPLOY read-back is true. The local AWS credential gate still returns HTTP 403 (AWS Gate is closed or denied), so no RDS, ECS, Secrets Manager, or CloudWatch state was read and no AWS, secret, or deployment mutation was attempted.

Conclusion: the owner decision remains unresolved. No approach, tradeoffs, implementation owner, or operations owner has been recorded in #191, so the engineering blocker remains. The new runtime failure increases urgency but does not choose among the three mechanisms in the issue.

Safe next action:

1. The service/AWS infrastructure owner records one mechanism and its tradeoffs in Synchronize website-sandbox/database-url with the weekly RDS master rotation #191, names the implementation and operations owners, and authorizes the redacted sandbox rotation-shaped verification.
2. Before any separately authorized outage recovery or AWS inventory, the operator must set the repository variable to exactly DEVELOPMENT_AUTO_DEPLOY=false and read it back, obtain the approved development-only AWS gate, and run the bounded read-only Gate B inventory from _docs/runbooks/development-release.md. No secret value or provider payload belongs in evidence.
3. After the decision and scope freeze, hand the issue to engineering for the selected mechanism and its rotation-shaped test; do not relax readiness or deployment fail-closed gates.

No AWS, secrets, deployments, or repository files were changed by this audit.

[alexeygrigorev]

Product/operations decision status — BLOCKED (2026-08-22)

Decision

The safe implementation phase for #191 is not unblocked. Acceptance criterion 1 is still open.

The exact missing owner decision is for the service/AWS infrastructure owner to select and record one durable mechanism, with its tradeoffs:

1. a bounded, redacted scheduled sync from the RDS-managed secret to website-sandbox/database-url before task refresh;
2. web/worker/migration tasks reading the managed secret directly at task start; or
3. changing/disabling the managed rotation cadence with documented compensating controls.

The decision record must also state the source-of-truth and refresh boundary, fail-closed/alert behavior, the named implementation owner and named operations owner, and whether the redacted development-only rotation-shaped verification is authorized. The issue has no individual assignee; only this owner role is named in the existing disposition.

This choice cannot be inferred from #160: #160 proved and repaired the incident by manually adopting the rotated credential into the static application secret, then forcing web/worker refresh, and explicitly deferred the durable mechanism to #191 (recovery evidence, closure/follow-up).

Evidence and boundary

• The latest Synchronize website-sandbox/database-url with the weekly RDS master rotation #191 audit records /health/live 200, /health/ready 503 with database/migrations unavailable, and /courses plus the course sitemap failing. It also records DEVELOPMENT_AUTO_DEPLOY=true; the AWS gate was closed, so the RDS event was not re-read and no current rotation attribution was claimed (latest #191 audit).
• CI run 32454238498 completed a successful deployment before that readiness failure. Run 32517486416 failed at source resolution and skipped capture/publish/deploy; it is not recovery evidence.
• P0: Restore the exact service pair within bounded compensation budgets #102's automated recovery scope is merged, but its remaining HUMAN choice is separate and does not select Synchronize website-sandbox/database-url with the weekly RDS master rotation #191's credential-rotation mechanism (#102 operations audit).
• Specifications 07/08/10 require independent secret rotation/recovery with named alerts/runbooks and redacted evidence; keep infrastructure/credential ownership explicit; validate RDS only at the exact-image migration/readiness/deployed-smoke boundary; and preserve fail-closed deployment/readiness gates. They do not authorize credential mutation.

Bounded next action

The service/AWS infrastructure owner should post the single mechanism decision, tradeoffs, implementation/operations owner names, and verification authorization in #191. Until that is recorded, leave #191 open and do not start code, Terraform/IAM, task-definition, secret, AWS, or deployment work for this issue.

If a separately authorized outage recovery or AWS inventory is needed before then, the operator must first read back DEVELOPMENT_AUTO_DEPLOY=false, obtain the development-only AWS gate, and run only the bounded redacted Gate B procedure. That is recovery/inventory work, not approval of #191 implementation. This update authorizes no AWS, secret, deployment, or production mutation.