Skip to content

fix(compute): retry watch-driven sandbox removal after a concurrent write - #5666

Merged
matthewgrossman merged 2 commits into
mainfrom
fix/5520-retry-watch-delete-removal/matthewgrossman
Nov 17, 2026
Merged

matthewgrossman merged 2 commits into
mainfrom
fix/5520-retry-watch-delete-removal/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

On Kubernetes, a deleted sandbox's record could stay for about five minutes after its resource was gone, which stalls delete + expect_absent on the k3s-ha conformance row. The lease holder's watch-driven removal gave up silently when another replica wrote the record mid-removal. It now retries against a fresh snapshot, and the delete path logs enough to confirm the cause in CI.

In plain terms

The test deletes a sandbox and waits for the gateway to confirm it is gone. Usually that takes about a second, but sometimes it took about 5 minutes.

When a sandbox is deleted, Kubernetes removes the container. The gateway keeps its own record of the sandbox in a shared database and erases it once Kubernetes reports the sandbox gone. The k3s-ha setup runs three gateway copies that share that database, and one of them does the erasing.

The erase works in two steps:

  1. Read the record and note its version.
  2. Erase it, but only if nobody changed it since step 1.

At almost the same moment, another gateway copy often updates the same record because the sandbox's connection dropped. When that update landed between the two steps, the check skipped the erase, and nothing tried again. A cleanup job eventually removes leftover records, but only for sandboxes older than 5 minutes, so the test waited for that job.

The fix retries the erase up to 3 times when the record changed, the same way another delete path already does. New log messages show in CI when a retry happens. The failing run's logs strongly suggest this collision but don't fully cover the moment the stalled sandbox hit it, so those logs will confirm the cause.

Related Issue

Closes #5520

Related: #5333

Changes

  • apply_deleted_locked re-reads the record and retries the version-checked removal up to DELETE_PHASE_CAS_RETRY_LIMIT times, matching remove_deleting_sandbox_record. Before, a concurrent write left the record until the orphan sweep, which waits until the sandbox is 5 minutes old.
  • New logs:
    • INFO when a driver deletion event arrives.
    • INFO when the removal does not complete and retries, such as after a concurrent write.
    • INFO when the record is kept for pending recovery (previously silent).
    • WARN when the retries run out.
  • The orphan-sweep removal log includes the sandbox phase.
  • The Kubernetes driver logs when its sandbox watcher finishes a full list, since a relist does not report sandboxes deleted during the gap.

Root cause

The Kubernetes driver's DeleteSandbox returns once the resource is gone. The gateway record is removed only when the lease holder applies the watch deletion event. That removal read the record, then removed it only if the version still matched. On a mismatch or CAS conflict it returned Ok(false) with a debug log and never retried.

Other replicas write the record during deletion. In particular, reset_endpoint_status_after_supervisor_disconnect writes it in any phase when the supervisor session ends. In the failing run (44020913674), the holder's removal and the session end on another replica landed within 8–40 ms of each other for several sandboxes.

For the stalled sandbox a7a42cf0, the holder logged nothing from the delete until the orphan sweep removed the record at age_secs=347. The logs of the replica that handled its delete do not cover that window, so the conflicting write for this sandbox is inferred. The new test reproduces the mechanism: one injected concurrent write leaves the record behind silently without the retry.

Testing

  • cargo test -p openshell-server --lib compute:: (407 passed)
  • cargo test -p openshell-driver-kubernetes --lib (335 passed)
  • cargo fmt --check and cargo clippy -p openshell-server -p openshell-driver-kubernetes --all-targets -D warnings
  • Unit test added: watch_deletion_retries_after_concurrent_replica_write (fails with a single attempt; also asserts driver cleanup runs)
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

…rite

A driver deletion event is the only signal that removes a sandbox record
after a driver-confirmed delete. Its removal compared the record version
once and silently gave up when another replica wrote the record in
between, such as the endpoint-status reset when the supervisor session
ends. The record then stayed until the orphan sweep, about five minutes
after creation.

Re-read and retry the removal, and log driver deletion events, retries,
retained records, and Kubernetes sandbox watcher relists.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the state:accepted A maintainer decided OpenShell should pursue this issue label Nov 17, 2026
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 17, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 4dba2a1

@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 4dba2a1. Open Branch E2E Checks, find the run for commit 4dba2a1, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

…anup

The removal helper returns false for more than a version conflict, so
log the retry neutrally and only before attempts remain. Assert that the
successful retry reclaims driver-owned resources.

The orphan-sweep removal log also gains the sandbox phase, to tell a
stalled delete apart from a sandbox that never reached the driver.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 0918090

@matthewgrossman
matthewgrossman marked this pull request as ready for review November 17, 2026 21:58
@matthewgrossman
matthewgrossman added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit d08935e Nov 17, 2026
266 of 269 checks passed
@matthewgrossman
matthewgrossman deleted the fix/5520-retry-watch-delete-removal/matthewgrossman branch November 17, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:accepted A maintainer decided OpenShell should pursue this issue test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(e2e): file_transfer::upload_download_round_trip stalls on the k3s-ha conformance row

2 participants