Skip to content

fix: retain application volumes and protect database rollouts - #21

Merged
toto04 merged 2 commits into
mainfrom
fix/incident-20260917-workload-safety
Sep 30, 2026
Merged

toto04 merged 2 commits into
mainfrom
fix/incident-20260917-workload-safety

Conversation

@lorenzocorallo

@lorenzocorallo lorenzocorallo commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

Protect all seven managed Longhorn PVCs from Argo pruning and application deletion with Prune=false,Delete=false. Use Recreate for the single PostgreSQL instance so a rollout cannot start competing writers on its volume, and add pg_isready readiness without a liveness restart loop. Give the bot 90 seconds for graceful shutdown.

Validation: both manifest regression scripts pass, including retention and PostgreSQL rollout checks. Every changed manifest passed kubectl apply --dry-run=server against the current cluster. No manifests were applied by these checks.

Rollout: the PostgreSQL pod-template change causes a graceful database recreation and brief downtime; schedule the sync accordingly. The bot grace-period change also recreates its pod: confirm its Redis connection and fallback-buffer flush before sync. PVCs and data remain in place. Existing PVs were separately changed to Retain during the incident; PVC annotations complement that protection. There are no image or node-count changes in this PR.

Related incident PRs: backend, telegram, terraform.

@coderabbitai

coderabbitai Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 449e0940-d1ba-41c2-b7c2-15fdd7ca15a1

📥 Commits

Reviewing files that changed from the base of the PR and between c4b3f9b and 565f790.

📒 Files selected for processing (8)
  • bot-ts/src/deployment.yaml
  • influxdb/src/deployment.yaml
  • monitoring/src/deployment-grafana.yaml
  • monitoring/src/deployment-prometheus.yaml
  • postgres/src/deployment.yaml
  • redis/src/deployment.yaml
  • tests/workload-manifests.test.rb
  • uptime-kuma/src/deployment.yaml
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@toto04
toto04 merged commit 20c8bfc into main Sep 30, 2026
1 check was pending
@toto04
toto04 deleted the fix/incident-20260917-workload-safety branch September 30, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants