Repository navigation
Description
User Story
I use OpenShell on Amazon EKS with the Kubernetes driver and one supervisor Pod per sandbox to run an internal agent platform. Agents are long-running: each sandbox is expected to stay up for days and keeps its state in the workspace PVC (ReadWriteMany storage, mountable on any node). On 14 November one worker node lost network connectivity for about three minutes. 22 sandboxes had their supervisor Pod or workload Pod on that node. For each of them the gateway logged
sandbox-runtime supervisor is unavailable; suspending workload,
and the sandbox moved to Stopped. They stayed down for about 55 hours, until we ran openshell sandbox start for each one. The sandboxes were created with --restart-policy always. At the time we ran OpenShell 0.1.7; we now run 0.1.8 and have confirmed the same behavior in the 0.1.9 source.
Problem Statement
When the Kubernetes driver suspends a sandbox because its supervisor or runtime is lost, OpenShell treats the sandbox like one an operator stopped. It stays Stopped, and the restart policy is never consulted, because a replacement is only triggered by a reported exit of the main process. No setting recovers a sandbox from this state automatically, and its status does not distinguish it from a deliberate stop.
Impact / Why This Matters
Events that Kubernetes operators treat as routine (node failure, network partition, eviction, preemption) leave sandboxes down indefinitely, until a person notices. With supervisor and workload on separate nodes, a failure of either node, or of the network between them, stops the sandbox (see #5547). That widens the effect of a single node failure.
Our workaround idea is an external reconciler that calls openshell sandbox start on every Stopped sandbox that should be running. It is insufficient: it cannot tell a deliberate stop from a lost runtime, because both show the same phase, so it needs its own record of desired state; every operator has to build and run it; and it can race with lifecycle operations in the gateway.
--restart-policy always reads as if it covers this case, which makes the gap easy to miss until production. The fail-closed suspension itself is correct and should stay. The gap is what happens afterwards.
Proposed Design
-
When a sandbox's runtime is lost (supervisor unavailable, workload Pod or node lost, recovery window expired) and its restart policy is
on-failureoralways, the gateway starts a new runtime generation for the same sandbox, with the same ID, policy, providers and workspace PVC, under the existing restart backoff. Process memory is lost, as with any restart. This matches the RFC 0018 entry "Workload node or runtime lost: Restart if policy allows". -
The new generation starts only after the previous one can no longer run, so two instances of a sandbox never run at the same time. Fail-closed behavior is unchanged.
-
An explicit
openshell sandbox stopis never undone, and--restart-policy neverbehaves as today.
Acceptance Criteria
- A Kubernetes sandbox created with
--restart-policy alwaysreturns toReadywithout operator action after its supervisor Pod is deleted, keeping its sandbox ID and workspace PVC, with an incremented restart count.
The same happens when the node hosting its workload Pod or supervisor Pod is unreachable for longer than the recovery window, once a replacement can be scheduled.--restart-policy on-failurealso restarts in these cases. A sandbox stopped withopenshell sandbox stopis never restarted automatically, and--restart-policy neverbehaves as today. Two workload generations of one sandbox never run at the same time.openshell sandbox getshows the suspension cause and the restart count. If a replacement cannot start, for example because the previous Pod is still terminating on an unreachable node, status reports the blocking condition. Repeated failures end inErrorwith a reason instead of retrying forever.
Alternatives Considered
External reconciler (our current workaround): see Impact. - Multi-tenant supervisor failover (experimental): keeps the agent process alive when a shared supervisor fails, for up to supervisor.recoveryTimeoutSeconds (at most 300 s). It does not cover loss of the workload's own Pod or node, or outages longer than the window. It complements this proposal rather than replacing it.
Kubernetes-native restarts (Pod restart policy, Agent Sandbox controller): would bypass the gateway, which owns runtime generations, policy and the fail-closed decision. - Extension points: interceptors govern API operations and middleware governs sandbox traffic. We did not find an extension point that observes or reacts to a suspension the driver initiates.
Related Investigation
#2616 (closed as not planned for lack of a user story; this issue provides one), #3415, #5547, #5104, #4910.
Problem Statement
When the Kubernetes driver suspends a sandbox because its supervisor or runtime is lost, OpenShell treats the sandbox like one an operator stopped. It stays Stopped, and the restart policy is never consulted, because a replacement is only triggered by a reported exit of the main process. No setting recovers a sandbox from this state automatically, and its status does not distinguish it from a deliberate stop.
Impact / Why This Matters
- Events that Kubernetes operators treat as routine (node failure, network partition, eviction, preemption) leave sandboxes down indefinitely, until a person notices.
- With supervisor and workload on separate nodes, a failure of either node, or of the network between them, stops the sandbox (see feat(kubernetes): operator-set placement for dedicated supervisor pods, and same-node placement with the workload #5547). That widens the effect of a single node failure.
- Our workaround is an external reconciler that calls
openshell sandbox starton everyStoppedsandbox that should be running. It is insufficient: it cannot tell a deliberate stop from a lost runtime, because both show the same phase, so it needs its own record of desired state; every operator has to build and run it; and it can race with lifecycle operations in the gateway. --restart-policy alwaysreads as if it covers this case, which makes the gap easy to miss until production.
Proposed Design
- When a sandbox's runtime is lost (supervisor unavailable, workload Pod or node lost, recovery window expired) and its restart policy is
on-failureoralways, the gateway starts a new runtime generation for the same sandbox, with the same ID, policy, providers and workspace PVC, under the existing restart backoff. Process memory is lost, as with any restart. This matches the RFC 0018 entry "Workload node or runtime lost: Restart if policy allows". - The new generation starts only after the previous one can no longer run, so two instances of a sandbox never run at the same time. Fail-closed behavior is unchanged.
- An explicit
openshell sandbox stopis never undone, and--restart-policy neverbehaves as today. - Status shows why the sandbox was suspended, whether a restart is pending, and what blocks it.
Acceptance Criteria
- A Kubernetes sandbox created with
--restart-policy alwaysreturns toReadywithout operator action after its supervisor Pod is deleted, keeping its sandbox ID and workspace PVC, with an incremented restart count. - The same happens when the node hosting its workload Pod or supervisor Pod is unreachable for longer than the recovery window, once a replacement can be scheduled.
-
--restart-policy on-failurealso restarts in these cases. - A sandbox stopped with
openshell sandbox stopis never restarted automatically, and--restart-policy neverbehaves as today. - Two workload generations of one sandbox never run at the same time.
-
openshell sandbox getshows the suspension cause and the restart count. If a replacement cannot start, for example because the previous Pod is still terminating on an unreachable node, status reports the blocking condition. Repeated failures end inErrorwith a reason instead of retrying forever.
Alternatives Considered
- External reconciler (our current workaround): see Impact.
- Multi-tenant supervisor failover (experimental): keeps the agent process alive when a shared supervisor fails, for up to
supervisor.recoveryTimeoutSeconds(at most 300 s). It does not cover loss of the workload's own Pod or node, or outages longer than the window. It complements this proposal rather than replacing it. - Kubernetes-native restarts (Pod restart policy, Agent Sandbox controller): would bypass the gateway, which owns runtime generations, policy and the fail-closed decision.
- Extension points: interceptors govern API operations and middleware governs sandbox traffic. We did not find an extension point that observes or reacts to a suspension the driver initiates.
Agent Investigation
Checked against v0.1.9:
crates/openshell-driver-kubernetes/src/driver.rs(around L5254): the reconciler suspends a sandbox when it should be running but its supervisor availability isUnavailable. With a dedicated supervisor, availability comes from the state of the supervisor Pod, which runs withrestartPolicy: Never.crates/openshell-server/src/compute/mod.rs,should_restart_main_process(L7014): replacement is decided only from the main process's exit code, so a suspension initiated by the driver never reaches it.docs/kubernetes/high-availability.mdx, "Recover from a Supervisor Failure": recovery does not restore a sandbox whose own Pod or node is lost.rfc/0018-multi-sandbox-ha-supervisors(draft): its failure table lists "Workload node or runtime lost: Restart if policy allows."
Checklist
- I've reviewed existing issues and the published docs
- This is a design proposal, not a "please build this" request
Activity