fix(v1): typed sandbox fault codes

[faresobeid]

What

• SandboxError(message, *, code: str | None = None). code names the fault from typed evidence only — an exception type, an errno, or an HTTP/RPC status, never the message text: not_found, timeout, disk_full, unavailable, provisioning, denied; None when nothing typed says.
• trace.Error.code: str | None, recorded by Trace.record_error next to status_code.
• errors.sandbox_fault_code(e) maps the Python-level sources (FileNotFoundError, TimeoutError, OSError with ENOSPC, httpx status/transport errors).
• runtimes/prime.py maps the SDK's exception types and Connect RPC codes in one function, _fault_code. The SDK's base APIError carries the HTTP status only in its text ("HTTP 503: ...") and is raised inside the httpx handler without from; that function reads the typed status off the chained httpx.HTTPStatusError instead of parsing the message. start() falls back to provisioning.
• runtimes/subprocess.py: a bounded read of a missing path carries not_found.

Cancellation is not a fault

A cancelled task's RPC comes back from the SDK as an ordinary error: connectrpc turns the CancelledError raised inside a VM RPC into ConnectError(CANCELED, "Request was cancelled"), prime-sandboxes re-wraps it as APIError, and the runtime used to wrap that as SandboxError("prime exec failed: Connect RPC failed (canceled): ...") — so any caller that retries on sandbox faults (a harness turn, a rollout re-attempt) swallowed the cancel and kept working; a run failed to exit on SIGINT for 11 minutes and leaked 21 sandboxes. The same seam now decides: _failure(message, e) re-raises asyncio.CancelledError when the current task is being cancelled (Task.cancelling()) or the failure carries the cancel (the CANCELED code, or the CancelledError chained under it), else the SandboxError with its typed code. No path returns a SandboxError for a cancel, so there is no canceled code. The SDK's live-process stream pump (AsyncSandboxProcess._pump, its own task) meets the same swallow and re-attaches on it (live process stream dropped (Request was cancelled); re-attaching 1/5), so a cancelled pump streamed on until the remote process exited and asyncio.run never returned; _can_reconnect is wrapped (guarded by getattr, a stopgap until the SDK owns the cancel — connectrpc re-raising CancelledError, or prime-sandboxes listing CANCELED as fatal) so the pump ends instead.

Why

A pipeline running ~90 concurrent rollouts on prime sandboxes classifies sandbox faults (box gone, API unavailable, disk full, quota) by regex over SandboxError text, because the error carries no typed code. The runtime already knows the type of the failure at the point it wraps it; this records it once so consumers branch on trace.last_error.code instead of on prose.

Validation

• tests/v1/test_errors.py: the code survives record_error and a to_record round trip; typed sources map, text-only ones stay None (including the SDK's APIError("HTTP 404: ...") shape); a runtime read of a missing path raises SandboxError(code="not_found").
• Cancellation: a fake SDK that reports the cancel as the RPC does (APIError ← ConnectError(CANCELED) ← CancelledError), or with no typed trace at all, inside runtime.run() while the awaiting task is cancelled → CancelledError propagates and the task ends cancelled, not SandboxError; a fault without a cancel stays a SandboxError with its code. The SDK pump, cancelled while its stream reports CANCELED, ends without re-attaching; a dropped link (UNAVAILABLE) still re-attaches.
• uv run pytest tests/v1 -m "not e2e", ruff, ty check, pre-commit pass.

Not wired: the modal runtime (its SDK types are not audited here), harness._check_result's "runtime died" SandboxError (the alive probe is structural, not an exception type — left None for now), and PrimeProcess.write/terminate/kill, which never mapped to SandboxError and still surface a swallowed cancel as the SDK's raw APIError.

Note

Add typed fault codes to SandboxError and Prime runtime error handling

• Adds an optional code field to SandboxError and trace.Error so sandbox failures carry a named fault code (e.g. not_found, timeout, disk_full, denied, unavailable) alongside their message
• Introduces sandbox_fault_code and _http_fault_code classifiers in errors.py that map typed local OSError and HTTP status evidence to codes; unknown exception types and message-only strings produce no code
• Adds the _faults context manager in prime.py and wraps all Prime runtime operations (start, run, read, write, expose, run_background, open_process, prepare_execution) so provider failures become coded SandboxError and swallowed task cancellation is re-raised as asyncio.CancelledError
• Replaces the SDK AsyncSandboxProcess._can_reconnect predicate with a cancellation-aware wrapper so a cancelled stream pump does not attempt reconnection
• Applies sandbox_fault_code to subprocess reads in subprocess.py and records code on trace errors in trace.py
• Behavioral Change: Prime egress polling now uses EGRESS_APPLY_TIMEOUT; deadline expiry raises a timeout-coded SandboxError instead of propagating as cancellation. The import-time replacement of AsyncSandboxProcess._can_reconnect is guarded but modifies a SDK private method at module load.

^{Macroscope summarized 9671970.}

---

Note

High Risk
Changes Prime/async cancellation and error classification across all sandbox RPC paths; misclassification could break retries or leave sandboxes leaking, though behavior is heavily tested.

Overview
Adds typed sandbox fault codes so pipelines can branch on trace.last_error.code instead of parsing error messages.

SandboxError now accepts an optional code (not_found, timeout, disk_full, unavailable, provisioning, denied) derived only from typed evidence. sandbox_fault_code and Prime’s _fault_code (walking SDK/Connect/httpx exception chains, including HTTP status on chained APIError) classify failures; Trace.record_error persists code on trace errors.

Prime runtime routes SDK calls through _faults, which re-raises asyncio.CancelledError when Task.cancelling() is set (so retry-on-SandboxError cannot swallow SIGINT/cancel) and otherwise raises SandboxError with the mapped code. Egress apply timeouts explicitly use code="timeout". AsyncSandboxProcess._can_reconnect is wrapped so a cancelled stream pump does not reconnect forever. Subprocess bounded reads attach codes from OSError.

New tests/v1/test_errors.py covers trace round-trips, classification, cancel vs deadline behavior, and stream pump reconnect rules.

^{Reviewed by Cursor Bugbot for commit 9671970. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit a2882cb. Configure here.}

[macroscopeapp]

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — Beyond adding typed fault codes, this PR changes cancellation, timeout, error-wrapping, and live-process reconnection behavior across existing Prime runtime paths, including an import-time patch to a private SDK method. The resulting blast radius is materially wider than a small additive trace-schema change.

Notes:

• Diff unchanged. Approvability was decided on eligibility alone.

^{You can add or adjust custom eligibility rules. Learn more.}

[faresobeid]

Superseded by #2580, which reworks this in the shape the error module already uses: typed SandboxError subclasses (SandboxNotFoundError, SandboxTimeoutError, SandboxUnavailableError, SandboxDeniedError, SandboxDiskFullError, SandboxProvisioningError) with the code derived from the class rather than an instance string, so except SandboxUnavailableError and RetryConfig.include/exclude work; sandbox_error() mirrors model_error and the nine prime.py sites keep their except … raise … from e shape (no _faults). The cancellation handling and the _can_reconnect wrap are dropped here — that contract belongs in prime-sandboxes and goes upstream separately. Closing this one.