Skip to content

feat(reconcile): manage per-pool PodDisruptionBudgets - #290

Open
shurkus wants to merge 2 commits into
rustfs:mainfrom
shurkus:feat/issue-289-pod-disruption-budget
Open

shurkus wants to merge 2 commits into
rustfs:mainfrom
shurkus:feat/issue-289-pod-disruption-budget

Conversation

@shurkus

@shurkus shurkus commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Type of Change

  • New Feature
  • Bug Fix
  • Documentation
  • Performance Improvement
  • Test/CI
  • Refactor
  • Other:

Related Issues

Closes #289

Summary of Changes

The operator now manages one PodDisruptionBudget per Tenant pool, so node drains and autoscaler scale-downs cannot evict more pool Pods than the pool's erasure sets can lose while keeping write quorum.

  • New pod_disruption_budget reconcile step after the pool StatefulSets. The budget is named <tenant>-<pool>, owned by the Tenant, carries the pool labels and selects the pool Pods.
  • minAvailable = servers minus the servers that can be down while every erasure set keeps write quorum. It is derived from the layout RustFS reports in /rustfs/admin/v3/info (drive pool_index, set_index, endpoint host) and the standard parity, capped at the default parity for the set size, the same way RustFS computes write quorum.
  • Pools that cannot lose a server (single-server, two-server, or dense layouts such as 3 servers x 4 volumes with EC:2) get no budget, so they never block a drain.
  • Budgets are created or updated only while all pools are settled (rolled out, ready, no decommission) and all pool drives report ok. If the layout cannot be read, existing budgets stay as they are and the step retries in 60s.
  • New optional field spec.disablePodDisruptionBudget (default: budgets on). Setting it deletes only the budgets owned by the Tenant; budgets of removed pools are deleted as well.
  • rustfs-admin: RustfsServerInfo now deserializes servers[].drives[].
  • RBAC (chart and deploy/k8s-dev): policy/poddisruptionbudgets get, list, watch, create, patch, delete. The controller watches owned budgets.
  • CRD regenerated, user guide (EN and zh-CN) and CHANGELOG updated. The guide asks users to remove their own budgets that select pool Pods before upgrading, since Kubernetes rejects evictions of Pods selected by more than one budget.

Checklist

  • I have read and followed the CONTRIBUTING.md guidelines
  • Passed make pre-commit (fmt-check + clippy + test + console-lint + console-fmt-check)
  • Added/updated necessary tests
  • Documentation updated (if needed)
  • CHANGELOG.md updated under [Unreleased] (if user-visible change)
  • CI/CD passed (if applicable)

Impact

  • Breaking change (CRD/API compatibility)
  • Requires doc/config/deployment update
  • Other impact:

The CRD gains an optional field and the operator ClusterRole gains PodDisruptionBudget permissions, so the chart (CRD and RBAC) must be upgraded together with the operator image. Existing Tenants get budgets on the next reconcile unless they set spec.disablePodDisruptionBudget: true.

Verification

make pre-commit
cargo test --lib pod_disruption_budget

Additional Notes

  • Reconcile tests run the step against a mocked API server: a settled pool gets its budget, a pool without spare servers loses it, disabling deletes only Tenant-owned budgets (with a uid precondition), a removed pool's budget is deleted while pools are not settled, and an unreadable layout or an unhealthy drive keeps existing budgets and requeues.
  • Unit tests cover 3x1, 4x1 EC:2, 2x1, 8x2 in one 16-drive set with EC:4, 3x4 with EC:4 and EC:2, single-server pools, parity capping, missing or unhealthy drives, and other pools' drives.
  • Unlike AIStor, there is no "majority of servers" fallback when the layout is unknown: majority is not safe when a server holds several drives of one set or parity is below the default (8x2 with EC:4 tolerates 2 servers, majority would allow 3).
  • The operator deletes Pods directly, not through the eviction API, so the budgets do not block its own Pod cleanup or StatefulSet rolling updates.

Read the erasure layout through an injected future so the reconcile can run against a mocked API server, and document removing user-created budgets before upgrading.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: PodDisruptionBudgets for Tenant pools

1 participant