[ac-pplx]

Summary

On the Direct substrate, a worker that suspends a sandbox on its own (an idle suspend) publishes the row unplaced and clears sandboxes.active_node_id. It leaves the sandbox's Direct route identity (sandbox_direct_route_identities.current_node_id) naming the worker.

The data-plane route reads in crates/libs/internal/api/server/src/data_plane_sessions/route_rows.rs placed a sandbox on COALESCE(direct_route.current_node_id, s.active_node_id), so they still saw that row as placed. A session reattach was routed to the worker, where the sandbox no longer runs, and failed with 503 stream_unavailable. The resolver never asked for the wake it asks for on an unplaced row. The same row on Kubernetes has no route identity, so it reads as unplaced and the reattach wakes it.

After this change, the three route-row reads (fetch_route_row, update_route_row, fetch_sandbox_service_route_row) join the Direct route identity only while s.active_node_id is set. A row with no active_node_id is unplaced on both substrates, and a reattach to it queues the wake as it already does on Kubernetes. The API side of that wake for a Direct row with no active node is #1169. No join is added; the existing join gets one more condition.

Why this is safe: every Direct path that writes a route identity does one of three things.

• It requires active_node_id to name the worker already: start and wake find the node through it.
• It writes active_node_id in the same transaction: the owned restore.
• It writes the identity inside the claim's transaction: pool claims.

#1169's adoption writes active_node_id only when the route identity already names that node at the same execution generation, so it can't create a mismatch either. Direct has no drain or evacuation. The one live identity without active_node_id is a create's receipt before the create records its node. That row is creating, it is unplaced on Kubernetes too, and an attach to it gets the same retryable answer as on Kubernetes. Rows without a route identity, which includes every Kubernetes row, read exactly as before.

published_state.rs is unchanged; the route identity is not cleared at publication time.

Validation

• New Postgres-backed test data_plane_sessions::tests::a_reattach_wakes_an_unplaced_sandbox_whatever_its_direct_route_names. It resolves a terminal session's route for three rows sharing one Direct worker A:

  • A Kubernetes row, suspended, active_node_id NULL: fails with FailedPrecondition ("not assigned to a node") and queues a Command wake.
  • A placed started Direct row: routes to worker A's registered endpoint and queues no wake.
  • A Direct row, suspended, active_node_id NULL, route identity naming A: before this change it resolved to worker A and minted a route token, and the test fails on that case. With the change it behaves like the Kubernetes row.

  It also checks that the service-route row reads the same placement. Without the fix, only the suspended Direct case fails.

• At this head (94cdce8, on api: wake and delete a Direct sandbox the node suspended for idleness #1169's 6efc7a9 on current main):

  • cargo nextest run -p pplx-space-internal-api-server --all-features: 430 passed, 0 skipped, against the run's shared Postgres.
  • cargo clippy -p pplx-space-internal-api-server --all-targets --all-features -- -D warnings: clean.
  • cargo fmt --all -- --check: clean.
  • cargo arch: 0 required findings; nothing reported for route_rows.rs.

• Direct-on-Kind lane at this head: brought up with SPACE_KIND_SUBSTRATE=direct SPACE_KIND_STORAGE_CREDENTIALS=static infra/space/kind/scripts/up.sh, every SPACE container on this head. Each scenario ran five times with space_smoke run <id>, all passing 5/5:

  • native_contract.session_reattach_wakes_idle_suspended_sandbox
  • native_contract.session_reconnect_after_idle_suspend
  • native_contract.command_wakes_idle_suspended_sandbox
  • native_contract.process_session_wakes_idle_suspended_sandbox
  • ecw.access_idle_reconnect_preserves_workspace
  • ecw.pause_connect
  • ecw.pause_timeout_requires_connect

• The default Kubernetes Kind lane was not rerun at this head. This head changes api: wake and delete a Direct sandbox the node suspended for idleness #1169's Direct code and only the position of this PR's test.

Severity and risk

Severity: moderate. On Direct, a session created while a sandbox ran could not reattach after an idle suspend: the client got 503 stream_unavailable and no wake was asked for. This change makes the resolver ask for the wake instead of routing to the worker. With #1169, that wake restores the sandbox, or adopts its live record, and the reattach succeeds. Kubernetes was not affected.

Risk: low and limited to Direct. Kubernetes rows have no route identity, so their reads don't change. On Direct, only rows with no active_node_id read differently: they now look unplaced instead of placed on the worker named by a leftover route identity. Rollback is reverting this commit.

Rollout

internal-api only; no wire, proto, or schema change. Stacked on #1169, which must land first. Merged without it, a Direct reattach to an idle-suspended sandbox still fails: now as a retryable refusal whose queued wake fails at the API, instead of an immediate 503. Old and new internal-api replicas can run together. An old replica keeps routing such reattaches to the stale worker until it is replaced.

Not covered

Other readers of the same COALESCE(direct_route.current_node_id, s.active_node_id) placement likely still report the stale worker for an idle-suspended Direct sandbox. These are API sandbox responses, public sandbox joins, and the API bridge route token in crates/libs/api/src/controllers/sandbox/spaced.rs. Direct's access-path wake (access_runtime_client) no longer branches on that placement: with #1169 every implicit wake of a suspended or restoring Direct row goes through wake_execution.

The root cause is that a worker-side idle suspend leaves the Direct route identity in place, while the API-driven suspend clears it. Fixing that at publication time is a separate change. Clearing the identity there would also change which node's observations publish_sandbox_states accepts for the row, since it scopes by the same COALESCE.

AI assistance: Claude Code wrote the code change, the test, and this description.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[the-gigi-pplx]

Review — 1 finding

Low

1. crates/libs/internal/api/server/src/data_plane_sessions/route_rows.rs:14 — "The only live identity without active_node_id is a create's receipt" leaves out the race PR api: wake and delete a Direct sandbox the node suspended for idleness #1169 lists under "Not covered".
   There, a wake reaches the worker while the row is still placed, and finishes after the node's idle-suspend publication clears active_node_id. The VM is running on the worker the identity names, and before this change a reattach routed there and worked. With this change the reattach reads unplaced and queues a wake. That wake takes PR api: wake and delete a Direct sandbox the node suspended for idleness #1169's restore fallback, which is where my finding on PR api: wake and delete a Direct sandbox the node suspended for idleness #1169 sits. The code change is right for the common case. Please qualify the comment and the matching "Why this is safe" paragraph so the next reader doesn't treat the invariant as complete.

Verified against 5c736bcf5208b03318eee336adcfbb2c567689aa, diffed against PR #1169's head. Reproduced locally: cargo nextest run -p pplx-space-internal-api-server -p pplx-space-db-test-fixtures --all-features -E 'test(a_reattach_wakes_an_unplaced_sandbox_whatever_its_direct_route_names)' passed. Removing the new AND s.active_node_id IS NOT NULL from fetch_route_row alone makes it fail at tests.rs:562. I checked the other route-identity writers: the owned restore (direct.rs:971-985) and the pool claim (queries.rs:2953-2968) write the identity in the same transaction that sets placement. Start and wake check it against the node active_node_id names (direct.rs:819). I didn't mutate update_route_row's copy of the condition. It only runs after the first read saw a placed row (session_route.rs:115-126), and the test doesn't cover it.