fix(kubernetes): defer fence binding checks during preparation - #4096
Closed
matthewgrossman wants to merge 1 commit into
Closed
matthewgrossman wants to merge 1 commit into
matthewgrossman wants to merge 1 commit into
Conversation
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Label |
1 similar comment
|
Label |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Kubernetes driver's periodic reconciler can suspend a sandbox while its creator is still preparing it: fence identity annotations are published only after the workload Pod is observed, but reconciliation currently treats their absence as a changed fence. Defer the persisted identity comparison during
preparing, after validating the actual fence policy, so reconciliation leaves active provisioning to the creator and retains the existing expiration path.Related Issue
Closes #4095.
Related investigation: #4072. This PR targets main independently and changes only production reconciliation, its tests, and crate documentation. The exact Kubernetes object behind the original CI
sandbox not foundfailure remains unconfirmed; this PR fixes the independently demonstrated race.Changes
preparinggenerations from the persisted fence UID/generation comparison.Testing
cargo test --locked -p openshell-driver-kubernetes --lib). The original logic fails the fresh-preparation and expiration regression cases.067776fb95b289be2082b1c9d03f610d87ba35ea.Investigation evidence used the prior candidate
508035eefd57a1d13a6e126a34597db3ba735b2e: a dedicated K3s deployment completed 8 conformance scenarios and 20 additional creates, while logs and audit metadata captured two erroneous suspension attempts rejected by resource-version conflicts. That confirms the race occurs in a live deployment; it does not validate this fix or reproduce the original CI failure.Checklist