[ac-pplx]

Summary

The sandbox state a caller sees was decided in several places. The controller response (sandbox_to_cached_response in crates/libs/api/src/controllers/sandbox/responses.rs) overwrote the row's lifecycle and status with the node's observed status whenever an observed record existed. ECW's translator (crates/libs/api/src/routes/e2b/translate.rs) then re-read that merged string. No render could tell the stored lifecycle from the node's observation, so changing ECW's precedence meant editing the controller response, which also changes the admin API.

This PR changes no output. It adds a characterization table and routes the admin and ECW renders through one typed input:

• SandboxStateInput (crates/libs/api/src/models/sandbox_state.rs) carries the row's stored lifecycle and the node's observed status separately. It rides on SandboxResponse as state_input, which is never serialized.

• SandboxResponse::set_state (crates/libs/api/src/models.rs) is the one writer. It stores the input and renders lifecycle and status from it. sandbox_to_cached_response and observed_only_sandbox_response call it. sandbox_to_response keeps its struct literal, and both fields come from state_input.reported_lifecycle().

• One render per surface reads the input:

  • Admin lifecycle and status: SandboxStateInput::reported_lifecycle. Observed status first, else stored lifecycle, as before.
  • ECW state: ecw_sandbox_state.
  • ECW terminal classification: ecw_terminal_state.
  • Both ECW renders read ecw_lifecycle. It keeps today's precedence (observed status, then stored lifecycle, then started), trims and lowercases, and keeps the rule that every lifecycle ECW does not name reads as running. A later precedence in which the stored lifecycle wins is a change to ecw_lifecycle alone. It also moves the native timeout gate (routes/public/timeout.rs), which reads translate_sandbox_state. The ECW list never carries an observation (cached_sandbox_response), so a precedence in which the observation wins also needs the list to read it.

• The existing entry points keep their signatures and delegate:

  • translate_sandbox_state
  • terminal_sandbox_error
  • translate_lifecycle_state (the stored-only read in should_wake_on_connect)

  Their callers are unchanged: get, list, connect, touch recovery, timeout and refresh in routes/e2b/lifecycle.rs, and the native timeout gate in routes/public/timeout.rs.

• ECW translation stays at the HTTP boundary, and ECW pause still maps to SPACE suspend.

Native PublicSandbox.state and conditions are not routed. They do not render from SandboxResponse: routes/public/sandboxes.rs reads its own row query and renders from the stored lifecycle alone, through public_sandbox_projection::public_lifecycle_state. That query reads the node's observed status only for failure_code. The table pins native outputs next to the others.

The new table (routes/e2b/state_render_tests.rs and its snapshot) covers 21 stored values × 28 observations, 588 rows.

• Stored values: StoredSandboxLifecycle::all_as_str() (the 18 live and tombstone values), then the placement crate's test_support::NON_CANONICAL_ROWS (bogus, Suspended), then a padded terminal Stopped.
• Observations: no record, a NULL status, every sandbox-manager label, and padded, mixed-case, unknown, empty and whitespace-only statuses.

For each row it records what every surface returns through Postgres:

• the admin lifecycle/status;
• ECW list, get and terminal gate;
• the ECW connect wake, usable-pod and touch-recovery decisions;
• native state and Ready.

It shows the divergence the later fix addresses. For example, stored stopped with observed started is hidden from the ECW list, running on ECW get, and stopped on native. An empty observed status overrides the stored lifecycle in the admin response and ECW get. The table was generated by running it against main's code (main plus the test file). The same snapshot passes unchanged at this head.

Validation

At head 5baa765a (tree 005dfea3), on e9d36fd7.

Unit and Postgres tests

• cargo nextest run --locked -p pplx-space-api: 1563/1564 passed, 1 skipped (the #[ignore]d occupancy benchmark). With --features kubernetes: 2143/2144 passed, 1 skipped. Both Postgres and Redis setup scripts ran, and no fixture-backed test was skipped.
• The one failure in both runs, routes::e2b::tests::detail_reports_authoritative_federated_ownership_not_metadata, fails identically on the base commit. Its inline JSON snapshot assumes serde_json preserve_order, which these -p builds do not enable.
• Table generation: with main plus the test file, INSTA_UPDATE=always cargo nextest run --locked -p pplx-space-api -E 'test(each_surface_renders_stored_lifecycle_and_observed_status)' wrote the snapshot. At this head, the same test with INSTA_UPDATE=no passes against that snapshot.
• Mutation check: making ecw_lifecycle prefer the stored lifecycle fails the table test. Reverted, not committed.
• Test edits in routes/e2b/tests.rs: every fake that an ECW state read consumes goes through set_state.
  • native_sandbox_response builds with set_state(SandboxStateInput::stored("started")).
  • The two connect tests set SandboxStateInput::new(Some(<row lifecycle>), Some("started")) for each row, which is what tracked_response builds.
  • The remaining such fakes call set_state instead of writing lifecycle, status or state_input directly. The literal in metadata_uses_sandbox_metadata reads no state and is unchanged. No expectation changed.

Static checks

• cargo clippy --locked -p pplx-space-api -p pplx-space-api-svc --all-targets --features kubernetes -- -D warnings: clean. Without kubernetes: two warnings that are on main too, both in lines this PR does not touch.
• cargo arch: no new finding for files this PR touches. Its advisory that controllers/sandbox/mod.rs shrank (1086 → 1084) comes from main; this PR changes one import line there without changing its line count.

Kind at this tree (fresh cluster via infra/space/kind/scripts/up.sh, 5 min 10 s; images built from this head's tree, 005dfea3). Scenarios ran with scripts/tests/smoke-kind-typed.sh's environment and SPACE_SMOKE_KIND_COLD_RECEIPTS set as Kind CI does.

• ECW: all 13 passed, 6 min 17 s. These are ecw.cold_create_environment, ecw.command_round_trip, ecw.sandbox_info, ecw.sandbox_list_pagination, ecw.sandbox_delete, ecw.filesystem_round_trip, ecw.pause_connect, ecw.pause_connect_race, ecw.pause_timeout_requires_connect, ecw.access_idle_reconnect_preserves_workspace, ecw.refresh_keepalive, ecw.timeout_update and ecw.timeout_rearm.
• Native: all 15 passed, 7 min 4 s. These are the 14 non-ignored native_contract.* scenarios (--category native_contract, 6 min 49 s) and native.m1_generated_sdk_conformance (15 s).

Severity and risk

Severity: does not apply. It is a behaviour-preserving refactor.

Risk: API only. It changes no SQL, no wire shape and no stored data. state_input is #[serde(skip)], so admin JSON is byte-identical. The response constructors in responses.rs set it through set_state or from the same render. A struct literal or a direct status write still compiles, and ECW ignores it in favour of state_input; a default response reads as started in ECW, as it did before. The one hand-built response in production, the fork result in e2b.rs, uses only id. Rollback: revert the commit.

Rollout

No migration and no flag. Old and new API pods render identical outputs, so a mixed-version rollout is indistinguishable to callers. No sql! locator moves: no SQL changed.

Not covered

• The full scripts/tests/smoke-kind-typed.sh all suite was not run at this head; only the 13 ECW and 15 native scenarios listed under Validation.
• Native state is not routed through SandboxStateInput. It already renders from the stored lifecycle alone, and its row query reads the observed status only for failure_code (see Summary).
• Open PRs on the same files edit none of the functions or statements changed here, so nothing was left in place:
  • responses.rs: [space] fence every placement writer against the provider-terminate path #559, [cluster-controller] re-place an evacuated stopped sandbox as stopped on another node, materialized by the shim without a VM #826, [api] record the activating client and label sandbox pods with it #1068
  • models.rs: [api] record the activating client and label sandbox pods with it #1068 (it edits Sandbox, not SandboxResponse)
  • routes/e2b/tests.rs: [api] rate-limit activations per client key, atomically with the cluster bucket #1131, [api] record the activating client and label sandbox pods with it #1068, [space-api] preserve CommandIdle template default across ECW lifecycle endpoints #538, Accept client TLS interception exemptions on ECW creates #194
  • translate.rs: [space-api] preserve CommandIdle template default across ECW lifecycle endpoints #538, Accept client TLS interception exemptions on ECW creates #194
  • routes/e2b/lifecycle.rs (not modified): [api] rate-limit activations per client key, atomically with the cluster bucket #1131, [api] record the activating client and label sandbox pods with it #1068, [space] fence every placement writer against the provider-terminate path #559, [space-api] preserve CommandIdle template default across ECW lifecycle endpoints #538
  • routes/public/timeout.rs (not modified): [space-api] preserve CommandIdle template default across ECW lifecycle endpoints #538
  • controllers/sandbox/mod.rs: api: permalaunch space-api-timeout-lock-bypass #1178, api: cap each client's live sandboxes, counted from the pod reflector and a Redis pending set #1172, [api] rate-limit activations per client key, atomically with the cluster bucket #1131, [api] record the activating client and label sandbox pods with it #1068, api: refuse a cold create with a 503 when every eligible node's network slots are held #1061, [cluster-controller] give up on pod-less drain residents only when the node vouches #705, feat(api): list guest files matching a glob in one call (files:glob) #675, [space] fence every placement writer against the provider-terminate path #559, [space-api] preserve CommandIdle template default across ECW lifecycle endpoints #538, Bound reconnect recovery on unpublished local suspends #350

AI assistance: An AI agent (Claude Code) wrote the code, the tests and this description, and ran the validation.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[zibi-pplx]

Approving. The production constructors keep today's outputs, and the characterization table is a useful record of where the surfaces disagree. Two suggestions:

[medium] lifecycle and status are still writable alongside state_input. Both stay pub on SandboxResponse (crates/libs/api/src/models.rs). ECW now reads state_input. A future direct assignment to status would change the admin JSON while ECW keeps reading the unchanged state_input, even under today's precedence. The description acknowledges this ("a direct status write still compiles, and ECW ignores it"). No production caller makes such a write today, so this isn't blocking. But it's cheap to close now, before the follow-up: make the two fields private behind set_state, or render them from state_input at serialization with serialize_with or a custom Serialize.

[low] For the precedence follow-up. The 588-row snapshot will change in many rows at once. It would help reviewers if that PR listed which (stored, observed) classes it intends to change, so the snapshot diff can be checked against the list.