Skip to content

fix(kubernetes): defer fence binding checks during preparation - #4096

Closed
matthewgrossman wants to merge 1 commit into
mainfrom
fix/4095-preparing-fence-reconciliation/matthewgrossman
Closed

matthewgrossman wants to merge 1 commit into
mainfrom
fix/4095-preparing-fence-reconciliation/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The Kubernetes driver's periodic reconciler can suspend a sandbox while its creator is still preparing it: fence identity annotations are published only after the workload Pod is observed, but reconciliation currently treats their absence as a changed fence. Defer the persisted identity comparison during preparing, after validating the actual fence policy, so reconciliation leaves active provisioning to the creator and retains the existing expiration path.

Related Issue

Closes #4095.

Related investigation: #4072. This PR targets main independently and changes only production reconciliation, its tests, and crate documentation. The exact Kubernetes object behind the original CI sandbox not found failure remains unconfirmed; this PR fixes the independently demonstrated race.

Changes

  • Exclude preparing generations from the persisted fence UID/generation comparison.
  • Preserve live fence-policy validation, released-generation identity checks, and abandoned-bootstrap expiration.
  • Exercise the real reconciler with four API fixtures: fresh preparation, missing binding after release, altered policy during preparation, and expired preparation.
  • Document when reconciliation starts comparing the published binding.

Testing

  • All 274 Kubernetes driver library tests pass, including four new regression cases (cargo test --locked -p openshell-driver-kubernetes --lib). The original logic fails the fresh-preparation and expiration regression cases.
  • Scoped Clippy over all driver targets, Rust formatting, and diff checks pass.
  • Configured pre-commit hook passes, including repository formatting/lint, license checks, Helm checks, protobuf lint, and lockfile checks.
  • Fresh Kubernetes/K3s E2E with artifacts from this PR's own commit; results pending in Branch E2E Checks. Branch Checks is also queued. Both runs test 067776fb95b289be2082b1c9d03f610d87ba35ea.

Investigation evidence used the prior candidate 508035eefd57a1d13a6e126a34597db3ba735b2e: a dedicated K3s deployment completed 8 conformance scenarios and 20 additional creates, while logs and audit metadata captured two erroneous suspension attempts rejected by resource-version conflicts. That confirms the race occurs in a live deployment; it does not validate this fix or reproduce the original CI failure.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Crate lifecycle documentation updated; related skills reviewed, with no command, configuration, or deployment changes requiring skill updates.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added area:gateway Gateway server and control-plane work test:e2e Requires end-to-end coverage area:cluster Related to running OpenShell on k3s/docker labels Oct 2, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 2, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/4096 does not exist yet. A maintainer needs to comment /ok to test 067776fb95b289be2082b1c9d03f610d87ba35ea to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

1 similar comment
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/4096 does not exist yet. A maintainer needs to comment /ok to test 067776fb95b289be2082b1c9d03f610d87ba35ea to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cluster Related to running OpenShell on k3s/docker area:gateway Gateway server and control-plane work test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(kubernetes): reconciliation suspends creation before fence identity publication

1 participant