Conversation
A memory snapshot restores envd mid-execution rather than restarting it, so an envd that is already wedged when the snapshot is taken is recorded in that state and replayed on every later resume. The resume succeeds through resume-fc, reaches sandbox-wait-for-start, and then retries envd's /init for the whole request budget. Nothing marks the snapshot bad, there is no cold boot fallback, and the origin node is only a preference, so the sandbox retries on any node forever. Observed on Foxtrot: one sandbox accumulated 183 consecutive failed resumes over three hours. The pause path already had two signals that envd was gone. bestEffortFreeze and bestEffortCollapse each burn their full timeout (2s and 10s) and set the span to codes.Error, but both are best-effort by construction, neither returns an error, and bestEffortReclaim has no error return, so the pause proceeds to drain-balloon, pause-fc and create-snapshot-fc and reports success. Add an envd /health probe as its own snapshot-admission pre-flight, before any destructive step, gated on pause-envd-health-timeout-milliseconds (negative default = off, same sign convention as pause-admission-grace-milliseconds). It runs independently of the durable-header wait so the two roll out separately, and only for memory snapshots, since a filesystem-only pause boots a fresh envd on resume. The refusal is retryable and deliberately kept out of the latched/kill path. An unanswered probe predicts an unresumable snapshot, it does not prove one: of eight sandboxes sampled whose freeze and collapse both timed out, three went on to resume cleanly. Deferring a pause costs a retry, condemning a sandbox cannot be undone. Pair with pause-refusal-restore so a refusal keeps the sandbox running rather than leaving it to the orphan reconciler. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PR SummaryMedium Risk Overview Reviewed by Cursor Bugbot for commit 32fa5d4. Bugbot is set up for automated code reviews on this repo. Configure here. |
…Sandbox The probe read pause-envd-health-timeout-milliseconds off the Sandbox's own feature-flag client. That client is nil on the pkg/server test fixtures, so every Pause and Checkpoint test panicked on a nil dereference and took the whole package down with it: 77 failures, most of them in tests unrelated to this change. Follow the convention the neighbouring gate already uses. PauseAdmissionGraceMs is read from the SERVER's flag client at the call site and the sandbox method is pure, so AwaitEnvdAdmission now takes the timeout as a parameter and returns immediately when it is negative. The handlers do the lookup, which also puts the enablement next to the existing one where a reader expects it, and leaves the method callable from a Sandbox built without flags. Tests drop the LaunchDarkly test datasource with the flag read and exercise AwaitEnvdAdmission directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 32fa5d4. Configure here.
| // mid-execution, so one taken while envd is unresponsive is replayed wedged | ||
| // on every later resume and the sandbox never comes back. | ||
| if healthMs := s.featureFlags.IntFlag(ctx, featureflags.PauseEnvdHealthTimeoutMs); healthMs >= 0 && !in.GetFilesystemOnly() { | ||
| outcome, waited, admitErr := sbx.AwaitEnvdAdmission(ctx, time.Duration(healthMs)*time.Millisecond) |
There was a problem hiding this comment.
Team-targeted health flag never enables
High Severity
Moving the PauseEnvdHealthTimeoutMs lookup out of AwaitEnvdAdmission dropped the TeamContext and TemplateContext that used to be added before the flag was read. Pause and Checkpoint only attach a sandbox-kind context, so a team-targeted value never matches and the probe stays disabled.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 32fa5d4. Configure here.
1 participant
No description provided.