[ac-pplx]

Summary

Cluster operators are managing-organization admins who support every
organization, but several of their actions only existed as org-scoped public
routes or were refused for other organizations. This lets the admin API do
them for any organization, under the owning organization, and records each one
in the persisted admin audit trail with the organization and resource it acted
on. Every route here already requires a managing-organization admin through the
admin middleware; tenant admins get 403 and unauthenticated callers 401.

• API keys for any organization. POST /orgs/{org_id}/api-keys no longer
  refuses organizations other than the caller's (it answers 404 for an unknown
  one). The key belongs to the target organization and keeps its
  allowed_surfaces permission model. It acts as a user chosen at creation:
  user_id, which must be a user of that organization (400 otherwise).
  Without it, a key for the caller's own organization acts as the caller, as on
  main. A key for another organization without it answers 400 "user_id is
  required for another organization's key": the route doesn't pick a tenant
  user for the operator. That user is stored as the key's created_by, the
  column the existing key lookup already reads, so every release, before or
  after this change, resolves the key to the same user. A chosen user deleted
  before the key is stored answers 409, asking to retry or choose another. For
  another organization's key, the calling admin is recorded in the audit event,
  not on the key.
  List and revoke already accepted any organization. This lets the console
  manage tenant keys without scripts/seed-api-key.sh, and choose the user a
  key acts as; ui: manage tenant keys, terminals, snapshots and rollback across orgs, and show node and template state #1387 adds that to the console.
• Admin terminal sessions. New POST /sandboxes/{sandbox_id}/terminal-sessions,
  the admin equivalent of the public route, beside the existing admin
  /commands. It resolves the sandbox without an org filter and creates the
  session under its owning organization through the same session store as the
  public route. So its credential is no wider: an attach token valid for this
  one session and the terminal:connect scope, expiring 5 minutes after
  creation, with a 60 s attach deadline and a 24 h session. Its idempotency keys
  are recorded under the admin route's own scope, and so are the existing admin
  /commands route's, so each route keeps its own records. The tenant
  sees the session in its own session list. This lets the console open terminals
  in other organizations' sandboxes.
• Admin snapshot list and rollback. New GET /sandboxes/{sandbox_id}/snapshots
  returns the public snapshot page (same filters, cursor and limit) under the
  owning organization, whether or not that organization can see the hidden
  public route. :force-rollback now takes the public :rollback body
  ({"snapshot_name", "auto_restart"?}), parsed by the public route's parser,
  and passes it to the operation. Before, it enqueued a rollback with no
  snapshot, which the executor always failed with "rollback action is missing
  rollback request". It now answers 400 without a body. Rollback stays behind
  the public :rollback's hidden-operation gate: :force-rollback answers 404
  "rollback is not enabled for the sandbox's organization" unless
  SPACE_NATIVE_HIDDEN_OPERATIONS_ORGS or the space-native-hidden-operations
  flag enables rollback for the sandbox's organization, the same check the
  public action makes. Closing the gate closes both routes. It is the only
  hidden action the admin surface gates: :force-checkpoint and
  :force-enable_snapshots, also hidden on the public API, already ran ungated
  on main and only add snapshots, while rollback, which the admin route could
  not run before this change, rolls the sandbox back to a snapshot. The body is
  part of the idempotency fingerprint; every other action's fingerprint is
  byte-identical to before, so replays across releases still match. This lets the
  console drop its read-only debug-route fallback for other organizations'
  snapshots, and roll back where rollback is enabled.
• API key metadata. Key metadata gains last_used_at and revoked_at.
  The auth path never wrote last_used_at. It now does, coarsely: a lookup
  writes only when the value it just read is more than 5 minutes old, so an
  active key costs at most one UPDATE per window. The UPDATE rechecks
  against the database clock and uses SKIP LOCKED, so concurrent requests
  don't queue on the row. The write is best effort and never fails the request.
  It records the last authentication through the API. Data-plane uses are not
  recorded: the ECW data-plane route resolver checks a key without writing, so a
  key used only for guest traffic keeps its last API use. Revocation now keeps
  the row, sets revoked_at, and replaces key_hash with revoked:<id>,
  which no SHA-256 digest equals. So the key stops authenticating on every
  reader of key_hash at once, including older API releases and the data-plane
  resolver. Revoked keys are listed only with
  GET .../api-keys?include_revoked=true, so the current console, which doesn't
  read revoked_at, never shows a revoked key as live. Revoking twice answers
  204 and keeps the first time.
• Org branding validation. PATCH /orgs/{org_id} and POST /orgs reject
  theme_primary/theme_secondary other than #rgb or #rrggbb, and a
  logo_url that isn't an absolute https URL without credentials (http
  only on localhost, 127.0.0.1 or [::1]). They answer 400 with
  {"error", "field"}. PATCH checks only a value that differs from the
  stored one, so the console's save, which re-sends all three fields, still
  works for an organization whose stored value predates validation. null
  still clears a value.
• Audit targets. A handler can return an AdminAuditTarget response
  extension; the admin middleware persists it into the new target_org_id,
  target_type (api_key or sandbox), target_id and target_user_id
  columns, which GET /audit-events lists. Handlers attach it once the
  resource resolves, on refusals as well as successes, and name only what
  resolved. That covers key create (with the key's user) and revoke, the
  terminal route, /commands, the snapshot list, admin POST /sandboxes and
  PATCH /sandboxes/{sandbox_id}, and every :force-* action on an existing
  sandbox. A refused revoke of another organization's key through this
  organization's path names no organization. Every admin write is persisted as
  before. Of the reads, only those that name a target are persisted, which today
  means only the snapshot list, including its refused queries. The secret never
  enters an audit event.

The new terminal and snapshot routes resolve the sandbox through
parse_admin_sandbox_id, as /commands, :force-rollback and every other
admin sandbox route do since #1321. All four accept sbx_<uuid>, a dashed or
undashed UUID in either case, or the sandbox pod name, and record the canonical
sbx_ ID in the session or operation, its idempotency fingerprint and the audit
target. /commands now lives with the terminal and snapshot routes in
sandboxes/guest_access.rs, which keeps admin/sandboxes.rs within archcheck's
file-size limit.

The admin OpenAPI document describes the new routes, fields and refusals.

A follow-up commit makes a key revoked after its lookup keep its last_used_at, drives the deleted-user 409 test through the admin router so it fails with the handler reverted, and documents that the admin /commands route applies neither the public route's lifecycle admission nor its implicit wake.

Validation

At this head, rebased onto main 0271415a after #1321:

• cargo nextest run -p pplx-space-api --all-features: 2212 passed, against
  Postgres and Redis started by the nextest setup scripts. The only test not
  run is the #[ignore] occupancy benchmark.

• Route tests through the real admin router and middleware: operator allowed;
  tenant admin 403; unauthenticated 401; audit event fields, including on
  refusals; validation errors. They also cover:

  • the key's chosen user, resolved by the unchanged lookup;
  • without user_id, the caller for an own-organization key, and a 400 that
    creates no key for another organization's, audited against that
    organization;
  • 400 for a foreign user_id, and 409 when the chosen user is deleted before
    the insert;
  • the rollback gate, closed and open;
  • admin and public terminal sessions, and command sessions, with the same key
    and body staying apart;
  • the terminal, /commands, :force-rollback and snapshot routes under
    sbx_, bare, upper-case undashed, pod-name and whitespace-padded IDs: one
    idempotency key under every form replays the first request, the rollback
    operation targets the canonical sbx_ ID, and so does each audit event,
    while SBX_, braced, sandbox- and dashed pod-name forms answer 400;
  • branding with a stored value that predates validation.

  The gate, terminal-scope, command-scope, own-organization default,
  required-user_id, deleted-user and ID-form tests fail with their fix undone.
  New Postgres-backed tests cover revocation, including inside a rolled-back
  transaction, and the last-use throttle.

• cargo clippy -p pplx-space-api --all-targets --all-features -- -D warnings
  and cargo fmt --all -- --check: clean; every commit builds on its own.
  cargo arch with main's baseline and ARCHITECTURE.toml: the same 112
  advisory findings as main, none required.

• On a local Kind stack at the previous head b077a034, with synthetic fixtures
  (since deleted), as a managing-organization admin:

  • issued tenant keys with user_id, and created a sandbox with each, owned
    by the key's user;
  • issued a key for the caller's own organization without user_id, and saw
    the caller as its created_by;
  • got 400 for a foreign user_id;
  • opened an admin terminal and ran a command in it; the tenant sending the same
    key and body to the public route got a separate session, and the operator's
    token was unchanged;
  • saw :force-rollback refused with the gate closed, with nothing enqueued;
  • got one audit event naming the sandbox or organization for each refusal
    case;
  • saved console-shaped branding over a stored pre-validation value;
  • revoked a key, and ran the migrations down and up.

  Kind was not rerun at this head. The 400 for another organization's key
  without user_id and the non-sbx_ IDs on the terminal route are covered by
  the router tests only. An earlier run of this branch covered listing a
  tenant's snapshots while the public route stayed hidden, last_used_at and
  include_revoked.

Not validated: a successful rollback. With the gate open, the operation reaches
the node, which fails it with "sandbox host mutation or reset is in progress;
retry" and then a missing rootfs.ext4. space-sbxmgr rollback run on the node
itself, with no API involved, fails the same way, so this is not specific to the
route.

Severity and risk

Severity: does not apply; this adds operator capabilities and fixes the inert
:force-rollback.

Risk:

• Wider operator powers. Managing-organization admins can mint keys for,
  open terminals in, and read the snapshots of any organization. Each is
  audited with the organization and resource.
• Rollback behind its gate. Rollback remains closed wherever the hidden
  :rollback is closed.
• Changed answers. Creating a key for another organization now answers 201
  with user_id, and 400 without it, instead of 403. A body-less
  :force-rollback answers 400 instead of enqueueing an operation that always
  failed, and a gated one 404. Revocation no longer deletes the row.
• Key users. A key's user is fixed at creation. An own-organization key
  without user_id acts as its caller, as on main. If a key's user is deleted,
  the key falls back to the organization's oldest user, as every release
  already does for a NULL created_by.
• Branding. A stored branding value that predates validation stays until
  someone changes it. PATCH it with a valid value, or null to clear it.
• Load. Each active key adds one small UPDATE per 5 minutes.
• Audit volume. Admin snapshot list reads now add audit rows.

Rollout

Requires the stacked migration. While old and new API pods run together:

• Old pods still reject revoked keys, because the hash is gone.
• Old pods list revoked keys without revoked_at until they are replaced.
• Old pods' DELETE still hard-deletes.
• Old pods' audit inserts leave the target columns NULL.
• Old pods' :force-rollback ignores its body and enqueues a rollback that
  fails, as it did before.
• Admin /commands idempotency keys are recorded under a new scope. A retry
  that moves between an old and a new pod misses the earlier record, so it can
  start a second command session and run the command twice. New pods
  don't fall back to looking up records under the old scope, so each route
  keeps only its own records from the start. The admin terminal route doesn't
  exist on old pods, so its new scope has no such effect.
• Keys created on new pods act as the same user on every pod, since the user is
  stored in the created_by column that every pod's lookup reads.

Old pods answer 422 to any body containing user_id, and 403 to another
organization's key without it; new pods require user_id for another
organization's key. Finish the API rollout before the console sends that field.
Today's console creates keys only for the caller's own organization, so the new
400 does not reach it. The
new include_revoked query parameter defaults to false.

To roll back, revert this change; the migration's columns can stay. After a
revert, keys revoked in the meantime still can't authenticate, because their
key_hash was replaced. But they list as live, since the reverted list doesn't
filter on revoked_at, until they are deleted, for example with
DELETE FROM api_keys WHERE revoked_at IS NOT NULL or the migration's down
step. :force-rollback again ignores its body.

Stack

Builds on #1301, now merged, which added the columns this uses
(api_keys.revoked_at, and the admin audit event targets including
target_user_id). Rebased onto #1321, which added the admin sandbox list's
paging and filters and the shared sandbox ID parser; both rewrote
admin/sandboxes.rs, and the rebase keeps both changes.

AI assistance: Claude Code drafted the code, the tests and this description.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[the-gigi-pplx]

Review — 4 findings

I'm holding my approval for now. The findings below don't block.

Low

1. crates/libs/api/src/routes/admin/api_keys.rs:259 — a key for another organization, created without user_id, acts as that organization's oldest user.
   The PR body documents this. The cost is that an operator-minted credential takes that user's role, and its traffic is attributed to a person who never created it; only the admin audit event names the real creator. Requiring user_id when the target organization isn't the caller's would make that choice explicit. Since the console now offers a picker, that costs one field.
2. crates/libs/api/src/routes/admin/sandboxes.rs:63 — moving admin /commands onto ADMIN_COMMAND_PATH_TEMPLATE orphans records the previous release wrote under the public template.
   A client retry that crosses the deploy misses its record and runs the command again. That's a short window, but a one-shot command running twice is the case idempotency exists for. A fallback lookup under the old template until its records expire would close it, or the rollout note could say so.
3. crates/libs/api/src/routes/admin/sandboxes.rs:548-556 — only :force-rollback checks the hidden-operation gate, while CHECKPOINT_SANDBOX and ENABLE_SANDBOX_SNAPSHOTS are IncubatingHidden too (crates/libs/public/api/src/routes.rs:231, :233).
   That's how main already behaves for those two, so nothing regresses. It's worth one sentence saying which hidden actions the admin surface gates and why rollback is the only one.
4. Sandbox ID parsing and the overlap with PR api: page and filter the admin sandbox list, count spaced versions, and share one sandbox ID parser #1321. The admin routes here, including the new terminal route (guest_access.rs:104), parse only sbx_ IDs (PublicSandboxId::from_str, sandboxes.rs:156, :622, :684, :1043).
   PR api: page and filter the admin sandbox list, count spaced versions, and share one sandbox ID parser #1321 moves the admin sandbox routes onto routes::admin::sandbox_id::parse_admin_sandbox_id, and both PRs rewrite large parts of routes/admin/sandboxes.rs. Whichever lands second has to rebase through that and route the new terminal and snapshot routes through the shared parser. Otherwise /sandboxes/{id}/terminal-sessions refuses the ID forms every other admin sandbox route accepts.

Verified against b077a03445. What I traced:

• The hidden-operation catalog.
• The admin command template change.

I also checked that the api_keys.revoked_at and admin_audit_events target columns this PR uses are already in main's migrations (20261102062705, 20261102062706). CI is green at this head. I didn't run the suite or a Kind stack.

[ac-pplx]

Thanks. All four are addressed; the head is now d40c0488, rebased onto main after #1321.

1. user_id for another organization's key. It is now required. Without it the route answers 400 "user_id is required for another organization's key". The oldest-user lookup and its 409 are gone. An own-org key without user_id still acts as the caller. The doc, the OpenAPI and the tests say so. Today's console creates keys only for the caller's own organization; ui: manage tenant keys, terminals, snapshots and rollback across orgs, and show node and template state #1387 will send user_id.
2. /commands idempotency records across the deploy. I kept this as a documented rollout cost. A fallback lookup under the old scope would keep the two routes reading each other's records for as long as those records live. The Rollout note says so.
3. The rollback gate. A comment and the body now say why only :force-rollback checks the hidden-operation gate. On main, checkpoint and enable-snapshots already ran ungated on the admin surface, and both only add snapshots. Rollback is new there, and it rolls the sandbox back. Gating the other two would remove actions that work today.
4. Shared ID parsing. The terminal and snapshot routes now resolve IDs with parse_admin_sandbox_id, like the other admin sandbox routes, and record the canonical sbx_ ID. A test sends every accepted form to all four routes, compares the answers, and checks the refused forms.

The rebase also moved /commands into sandboxes/guest_access.rs, with no behaviour change, to keep admin/sandboxes.rs under the line limit.

On a Kind cluster at 01bbc0ad:

• another organization's key without user_id got the 400; with user_id it acted as that user;
• a bare UUID and the sbx_ ID answered the same on the terminal route, and the audit rows named the sbx_ ID;
• /commands replayed the same session, and answered a different body under the same key with 409.

Could you take another look?

[the-gigi-pplx]

Re-review — 3 new findings

Of the four points in my earlier comment:

• Fixed: a key for another organization now needs user_id; without one, key_user refuses it (routes/admin/api_keys.rs:251-261).
• Fixed: the terminal and snapshot routes resolve sandbox IDs through the shared parser (resolve_sandbox → AdminSandboxTarget::parse, routes/admin/sandboxes/guest_access.rs:149).
• Documented: why only :force-rollback checks the hidden-operation gate (routes/admin/sandboxes.rs:416-424).
• Documented: the admin command idempotency scope change, with its cross-deploy retry cost, in the Rollout section.

Low

1. crates/libs/api/src/auth.rs:209-218 — record_api_key_use can stamp last_used_at on a key revoked after its lookup.
   The UPDATE's subquery checks only the staleness, not revoked_at. So a request that authenticated just before a revoke commits can still advance last_used_at afterwards, and include_revoked then lists the key as used after its revocation. Adding AND revoked_at IS NULL to the subquery closes it.
2. crates/libs/api/src/routes/admin/api_keys/tests.rs:345-366 — the deleted-user 409 test calls create_error on a store error directly, so it doesn't check that create_api_key routes insert failures through create_error. Reverting the handler to support::db_error would answer 500 for that race again and still pass. Driving it through the router, or asserting on the handler's response, would pin it.
3. routes/admin/sandboxes/guest_access.rs:67 — the admin command route passes || Ok(()) as its admit closure, so it skips the lifecycle admission and implicit wake the public route applies (routes/public/sessions.rs:145-177).
   On a stopped or suspended sandbox, it returns a session for a guest that isn't running and won't be woken. This predates the PR, and main has the same closure. I'm raising it because the moved doc now calls the route a mirror of the public one. Either apply the same admission, or say in the doc that it doesn't.

Verified against d40c04889522bf93969d5eeca123cfcc24256206, which is current with main. Reproduced locally: cargo nextest run -p pplx-space-api --features kubernetes over routes::admin::, auth::, public_sessions::, routes::public::sessions:: and routes::public::snapshots::: 379 run, 379 passed. Everything else I read at the lines cited.

[ac-pplx]

Thanks. All three are in 1130a69a, the PR's last commit. The branch is now restacked on #1384, which is on main after #1303 merged.

1. last_used_at after a revoke. record_api_key_use's subquery now has AND revoked_at IS NULL. A new test revokes a key between its lookup and its use, and checks that last_used_at stays put. It fails without the clause.
2. The deleted-user 409 test now goes through the router, so reverting create_api_key to support::db_error makes it fail.
3. Admin /commands admission. I documented it rather than changing it. The route's doc comment now says it applies neither the public route's lifecycle admission nor its implicit wake, so a stopped or suspended sandbox gets a session for a guest that isn't running. Mirroring the public admission would expose several private items of the public session route. It would also make an operator command implicitly wake a tenant's sandbox, which is a policy change of its own. It's tracked separately.

The rebase onto #1303 kept both PRs' behaviour in the two files they share:

• the identity error body carries api: close six admin API gaps the admin console works around #1303's reason and retained_until, plus this PR's field;
• the admin create runs this PR's precheck and passes api: close six admin API gaps the admin console works around #1303's requested_by.