Repository navigation
fix(compute): retry watch-driven sandbox removal after a concurrent write - #5666
Merged
matthewgrossman merged 2 commits intoNov 17, 2026
Merged
Conversation
…rite A driver deletion event is the only signal that removes a sandbox record after a driver-confirmed delete. Its removal compared the record version once and silently gave up when another replica wrote the record in between, such as the endpoint-status reset when the supervisor session ends. The record then stayed until the orphan sweep, about five minutes after creation. Re-read and retry the removal, and log driver deletion events, retries, retained records, and Kubernetes sandbox watcher relists. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Member
Author
|
/ok to test 4dba2a1 |
|
Label |
…anup The removal helper returns false for more than a version conflict, so log the retry neutrally and only before attempts remain. Assert that the successful retry reclaims driver-owned resources. The orphan-sweep removal log also gains the sandbox phase, to tell a stalled delete apart from a sandbox that never reached the driver. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Member
Author
|
/ok to test 0918090 |
matthewgrossman
marked this pull request as ready for review
November 17, 2026 21:58
TaylorMutch
approved these changes
Nov 17, 2026
matthewgrossman
enabled auto-merge
November 17, 2026 22:13
matthewgrossman
deleted the
fix/5520-retry-watch-delete-removal/matthewgrossman
branch
November 17, 2026 23:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On Kubernetes, a deleted sandbox's record could stay for about five minutes after its resource was gone, which stalls
delete+expect_absenton the k3s-ha conformance row. The lease holder's watch-driven removal gave up silently when another replica wrote the record mid-removal. It now retries against a fresh snapshot, and the delete path logs enough to confirm the cause in CI.In plain terms
The test deletes a sandbox and waits for the gateway to confirm it is gone. Usually that takes about a second, but sometimes it took about 5 minutes.
When a sandbox is deleted, Kubernetes removes the container. The gateway keeps its own record of the sandbox in a shared database and erases it once Kubernetes reports the sandbox gone. The k3s-ha setup runs three gateway copies that share that database, and one of them does the erasing.
The erase works in two steps:
At almost the same moment, another gateway copy often updates the same record because the sandbox's connection dropped. When that update landed between the two steps, the check skipped the erase, and nothing tried again. A cleanup job eventually removes leftover records, but only for sandboxes older than 5 minutes, so the test waited for that job.
The fix retries the erase up to 3 times when the record changed, the same way another delete path already does. New log messages show in CI when a retry happens. The failing run's logs strongly suggest this collision but don't fully cover the moment the stalled sandbox hit it, so those logs will confirm the cause.
Related Issue
Closes #5520
Related: #5333
Changes
apply_deleted_lockedre-reads the record and retries the version-checked removal up toDELETE_PHASE_CAS_RETRY_LIMITtimes, matchingremove_deleting_sandbox_record. Before, a concurrent write left the record until the orphan sweep, which waits until the sandbox is 5 minutes old.Root cause
The Kubernetes driver's
DeleteSandboxreturns once the resource is gone. The gateway record is removed only when the lease holder applies the watch deletion event. That removal read the record, then removed it only if the version still matched. On a mismatch or CAS conflict it returnedOk(false)with a debug log and never retried.Other replicas write the record during deletion. In particular,
reset_endpoint_status_after_supervisor_disconnectwrites it in any phase when the supervisor session ends. In the failing run (44020913674), the holder's removal and the session end on another replica landed within 8–40 ms of each other for several sandboxes.For the stalled sandbox
a7a42cf0, the holder logged nothing from the delete until the orphan sweep removed the record atage_secs=347. The logs of the replica that handled its delete do not cover that window, so the conflicting write for this sandbox is inferred. The new test reproduces the mechanism: one injected concurrent write leaves the record behind silently without the retry.Testing
cargo test -p openshell-server --lib compute::(407 passed)cargo test -p openshell-driver-kubernetes --lib(335 passed)cargo fmt --checkandcargo clippy -p openshell-server -p openshell-driver-kubernetes --all-targets -D warningswatch_deletion_retries_after_concurrent_replica_write(fails with a single attempt; also asserts driver cleanup runs)Checklist