[NancyKwongPerplexity]

Why

When a client creates a command with Accept: text/event-stream (#653), the API creates the session and then opens the data plane's events stream in-cluster with the session's fresh attach token. The data plane then resolves the route through ResolveDataPlaneSessionRoute. That is a cluster-API round trip that reads the route the API has just looked at and then writes it back to the session row. It sits on the path to the command's first event.

With this change, the API passes the route with its in-cluster request, and the data plane opens the stream on it without that round trip.

What changed

The handover (common-svc, space_api_route_ed25519::data_plane_handover)

• A route the API signs with the route signing key it already holds.
• It has its own token prefix and audience, and the prefix is part of the signed bytes. So nodes never accept a handover as a route token, and the data plane never accepts a route token as a handover.
• It lives for 10 seconds; verifiers allow at most 30 seconds with 5 seconds of clock skew. Its size is capped at 16 KiB.
• It names:
  • the session and its kind;
  • the SHA-256 of the attach token;
  • the org and sandbox;
  • the node and node-API endpoint;
  • the execution generation;
  • the revocation epochs;
  • the command's adapter state;
  • a node route token for that execution.

The API (command_stream::handover)

• Before opening the stream, it reads the session's route with the same joins route resolution uses.
• It mints a handover only for an attach that resolution would authorize:
  • the token matches the stored digest, is unexpired, and grants the kind's connect scope;
  • the session is active and has never bound a process;
  • the kind is terminal, process or file watch (port, protocol and ECW service routes are untouched);
  • the sandbox is placed on a node with a node-API endpoint.
• It hands over the route generation that resolution would store. The data plane compares generations when it later refreshes a route, so a long stream doesn't see a spurious route change.
• The node-API endpoint is derived by the same function resolution now uses (moved to common-grpc).
• Off unless SPACE_DATA_PLANE_ROUTE_HANDOVER is set. Any failure, or a read slower than 500 ms, sends no handover.

The data plane (native::handover)

• A new, optional SPACE_DATA_PLANE_INTERNAL_LISTEN listener serves the same routes and drains with the public one.
• Handovers are honored only on that listener, and only when the route verification keys are configured. A handover on the public listener is ignored.
• An accepted handover must also:
  • verify (signature, issuer, cluster, audience, lifetime);
  • name the request's session and kind;
  • match the presented token's SHA-256 (compared in constant time);
  • carry a node route token, verified with the same keys, for the handover's sandbox, node and execution generation.
• Every other request resolves the route as before. data_plane.route_handover{result} counts accepted and refused handovers by reason.

Freshness and safety

• Authentication. The data plane still authenticates every request. A handover only replaces resolution when the API vouched, within the last few seconds, for this exact session and token. The vouching is bound by a signature the data plane checks.
• Stale routes. A route that went stale between the handover and the open (a wake, restore or migration) carries an outdated node route token. The node refuses it, and the stream refreshes the route through resolution as it does for any refused route. If the old node is unreachable, the stream fails to open and the API answers with the JSON handle, so the client attaches through ordinary resolution.
• Revocation. Semantics are unchanged. The stream remembers the epochs the handover carries, and the periodic revocation check runs as before. A session revoked during the handover's few seconds is cut at the next check, as it would be if it had been revoked just after resolution.
• Process binding (data-plane: forward process events while the session binding commits #645) is unchanged.
• The skipped route write. The session row keeps no route until something resolves it. Only resolution reads those columns, and it writes them on its first call, at the generation the handover already claimed.

Rollout

Both sides are off by default. To enable:

1. Give the data plane SPACE_DATA_PLANE_INTERNAL_LISTEN, the route verification bundle and SPACE_CLUSTER_ID.
2. Expose that port only to in-cluster callers, and point the API's SPACE_DATA_PLANE_INTERNAL_URL at it.
3. Set SPACE_DATA_PLANE_ROUTE_HANDOVER=true on the API.

The handover carries the command's adapter state over the API-to-data-plane hop, which is plain HTTP inside the cluster, as the attach token already is.

Tests

• data_plane_handover:
  • round trip;
  • handovers and route tokens don't verify as each other;
  • a different key and a tampered payload are rejected;
  • expired, overlong, future-dated, foreign-issuer, wrong-audience, wrong-cluster and unknown-version handovers are rejected;
  • oversized handovers are neither minted nor parsed.
• Data-plane native::handover, against a fake cluster API and worker:
  • a valid handover opens the stream with no ResolveDataPlaneSessionRoute call, on the handed-over route token;
  • each of the following resolves the route once and starts on the resolved token: the public listener, a forged handover, an expired one, another session, another attach token, a route token for another execution, and garbage;
  • a handover whose execution the worker refuses (as after a restore) is resolved and retried on the resolved route;
  • no handover is accepted where no keys are configured.
• API command_stream::handover:
  • for IPv4 and IPv6 nodes, the handover for a streamed create matches what the real resolver returns for the same session and token: node, endpoint, route generation, revocation epochs, adapter state and node route identity;
  • a wrong, missing or expired token, a missing scope, a revoked or bound session, an unplaced sandbox, a node without an endpoint, and a port session get no handover.

Validation

At head ef04de78, rebased on main at 9f0fb596:

• cargo fmt --all --check, cargo check --workspace --all-targets --locked, and cargo clippy --all-targets --all-features --locked -- -D warnings on the touched crates (pplx-space-api, pplx-space-api-route, pplx-space-common-grpc, pplx-space-data-plane-server, pplx-space-data-plane-svc, pplx-space-internal-api-server, pplx-space-metrics): pass.
• cargo nextest run --all-features on those crates other than pplx-space-api, with the shared Postgres and Redis: 981 passed, none skipped. This includes data_plane_handover and the data plane's native::handover tests.
• cargo nextest run -p pplx-space-api --all-features --config 'profile.test.package.pplx-space-api.opt-level=0', with the shared Postgres and Redis: 2181 passed. One #[ignore] benchmark was skipped. The frozen M1/OpenAPI contract tests pass with no snapshot changes; this PR adds no public API.
• cargo arch: no required findings. The advisory direct_tables finding for the handover's read of sandbox_direct_route_identities and node_direct is unchanged by the rebase.
• The rebase adds one line to the handover's route read: main (internal-api: wake an unplaced Direct sandbox on reattach instead of routing it to a stale route identity #1170) now joins the Direct route identity only while sandboxes.active_node_id is set, and the handover keeps the same joins as resolution. A row it changes (no active_node_id, Direct identity present) is never started, so the handover already refused it as not_started; no test can tell the two apart, and none was added.

Not validated: a Kind run. The Kind charts set neither SPACE_DATA_PLANE_ROUTE_HANDOVER nor SPACE_DATA_PLANE_INTERNAL_LISTEN, so a Kind run never mints or accepts a handover.

[NancyKwongPerplexity]

Closing for now: the API-relayed command stream path this optimizes isn't enabled in production, so the handover token isn't worth carrying yet. Can revisit if the relay path is turned on.