[ac-pplx]

Summary

This change closes six admin API gaps that the admin console currently works around in ui/. It changes Rust code, docs and one OpenAPI stub snapshot, and adds no schema migration. It is one PR of eleven commits: one for each of the six changes below, three more for change 4's build key (when it stays reserved, how replays report it, and the bearer-token check), and two for change 2's rollback check.

1. User delete answers 409, not 500, while the user owns sandboxes. DELETE /users/{user_id} maps the foreign-key violations from sandboxes.user_id and sandbox_archives.user_id to 409 with a reason:

   • user_owns_sandboxes while the user owns a sandbox that is not deleted;
   • user_owns_retained_deleted_sandboxes when every such row is a deleted sandbox. Those rows are kept for 7 days after their last update and still hold the foreign key, mostly on Direct. The message says they are kept for retention until the time in the retained_until field and to retry after that, rather than asking the operator to delete sandboxes that are already deleted;
   • user_owns_archived_sandboxes while the user owns archived sandboxes.

   The self-target 409 carries reason: self_mutation, so callers can tell the cases apart. The field is reason, not code, because code is the status category elsewhere in /admin/api/v1. On main, both foreign keys surface as Internal. No other foreign key to users restricts a delete; the rest are SET NULL or CASCADE. The retention period lives in the sandbox store, which the cluster controller's tombstone retention reads.

2. Fork operations can record their source as sbx_<uuid>. Every other sandbox operation records that form. Every reader of a fork's target_id matches both forms:

   • the fork-source guard, active_fork_source_operation_exists, which the sandbox delete, reconcile finalize and snapshot retention use;
   • capture-leader election;
   • the ForkFailed event, which takes the canonical id from ClaimedPublicOperation::sandbox_id();
   • the admin /operations?target_id= filter. It always matches the string as given, and also both sandbox forms when that string parses as a sandbox id, so a template named by 32 hex digits still finds its own delete_template operation;
   • GET /v1/sandboxes/{id}/lifecycle-events, which carries the events of forks of the sandbox in either form, so what it streams does not depend on the switch or on a row's age.

   A wake no longer counts a later fork of the sandbox as other work in the sandbox's queue. New forks write sbx_<uuid> only while SPACE_FORK_PUBLIC_TARGET_ID_ENABLED is on; see Rollout. Matching the bare form can be removed one release after the switch is on everywhere, plus the 7 days a finished operation is kept; the code records where.

3. An admin create records who requested it. actor_user_id keeps its meaning: it is the owner, and the create executor runs as that user. The create payload gains an optional requested_by_user_id, and GET /operations and GET /operations/{operation_id} return it as a nullable field, read only from create_sandbox operations. The API writes it only while SPACE_ADMIN_CREATE_REQUESTED_BY_ENABLED is on; see Rollout.

4. System template builds accept an optional Idempotency-Key. POST /system-templates/{template}/builds records the key in public_idempotency_records. The builder starts a new job per submission and takes no caller identity to deduplicate on, so the key is reserved in its own committed transaction before dispatch, and the job is recorded in its place after. No transaction is held across the builder call. The reservation and the dispatch run together in a spawned task that keeps the request's tracing span, so a client disconnect interrupts neither. Within the record's 24-hour lifetime:

   • the same key and body return the original job_id without reaching the builder;
   • the same key with a different body answers 409;
   • a repeat answers a retryable 409 while the first request may still be submitting. It answers a non-retryable 409 once the API has recorded that the first request failed without proof that the builder did not take the build, and in any case 15 minutes after the key was reserved. That 409 carries reserved_at in details, and job_id when the builder named one; its message says the build may have started and may still be running, and points to Template Releases and the build's status. The 15 minutes outlast the builder RPC attempts, but not necessarily the database work and builder discovery around them.

   The template controller reports whether the builder may have taken a failed build. The key is freed for a retry only when it certainly did not: the submission failed before its request was sent (validation, ownership, grants, no builder to send to, or a node-api bearer token that cannot be read, which the controller reads before the call), the dial sent nothing, or the builder refused it while draining, which build-manager decides before it enqueues. On any other failure, such as a timeout, a transport error after sending, or a failed job-record write after acceptance, the reservation stays for the rest of the record's life, noted as unresolved with the job ID when the builder named one. So no repeat under a key starts a second job. The native :timeout route's response admission is unchanged.

5. The release tag move gains a compare-and-set precondition. PUT /template-releases/{release_id}/tag accepts an optional body field, expected_current_release_id:

   • omitted: the move is unconditional, as before;
   • null: the tag must not exist yet;
   • a release id: the tag must point at that release.

   A mismatch answers 412 failed_precondition with current_release_id in details. The check and the move run in one transaction under the tag row's lock. A move that loses to other writers creating and removing the tag answers 409 with retryable: true. Each move, and each refused precondition, emits a space.admin.audit entry, action: template_release_tag, with previous_release_id: the release the tag pointed at when the request ran, or empty when it did not exist.

6. The debug route returns environment variable names, not values. GET /debug/sandbox/{sandbox_id} keeps resources.env as an object with each value null, { "NAME": null }. This is data minimisation: diagnostics still show which variables are set. The route is in the admin OpenAPI stub, with that shape documented. Nothing in this repository reads the values: neither ui/, space-admin-cli nor the docs.

The admin console PRs #1245, #1253, #1254 and #1278 work around these gaps in ui/. #1245 and #1278 have merged with their workarounds, which a follow-up can drop; #1253 and #1254 are still open. After this merges, each can drop its workarounds. For the two changes behind a switch, that applies once the switch is on; for the tag precondition and the build key, once every API replica runs this release (see Rollout).

Validation

Rebased onto main at c5fdf0c7, after #1307 merged. Three textual conflicts were resolved by keeping both sides: two adjacent new debug-route tests, and the controllers imports in routes/admin/templates.rs. No behaviour changed. At this head, cargo nextest run -p pplx-space-api passes 2,191 of 2,191, and fmt and clippy with -D warnings are clean.

The results below are from 049cdf5b, before the rebase, on main at 17721a84:

• cargo nextest run --all-features for pplx-space-api, pplx-space-sandbox-store, pplx-space-snapshot-retention and pplx-space-cluster-controller-server, with the Postgres setup from pplx-space-db-test-fixtures: 3451 passed, 0 failed, with no dependency skips.

• cargo clippy --all-features --all-targets -- -D warnings on the API, sandbox-store, snapshot-retention and cluster-controller crates is clean, and cargo arch passes.

• Database-backed and route tests:

  • delete_user_answers_conflict_while_the_user_owns_sandboxes: live, deleted-only and archived owners, then 204, and self_mutation
  • user_sandbox_rows_reports_when_deleted_rows_leave_retention
  • an_active_fork_source_is_found_by_either_target_id_form
  • a_wake_joins_an_earlier_wake_past_a_later_fork
  • capture_leader_election_spans_both_target_id_forms
  • fork_operation_kind_persists_in_native_scope: the admin filter by either form, a 32-hex template name found as typed, and the lifecycle-events read finding a bare-UUID fork
  • an_admin_create_reports_its_requester_apart_from_the_owner
  • create_payload_without_a_requester_is_wire_compatible_with_old_shape
  • a_response_reservation_answers_replays_without_repeating_work: reserve, replay, release, unknown outcome, complete, and a noted reservation that reads as unknown at once and can be neither completed nor released
  • a_system_build_replay_with_its_idempotency_key_starts_one_job: replay, a different body, a failed record write, and a client that disconnects mid-dispatch
  • a_failed_system_build_frees_its_key_only_if_the_builder_did_not_take_it: a builder that takes the build and answers an error, and an accepted build whose job record fails, both keep the key, and their replays answer the non-retryable 409 at once (the second names the job); a node-api bearer token that cannot be read, and a refused connection, free it, and the same key then builds
  • a_failed_attempt_may_have_been_accepted_unless_it_proves_otherwise: the controller's classification table
  • a_release_tag_move_honours_its_expected_current_release: preconditions, and the lost race answering a retryable 409
  • the_debug_response_lists_env_names_without_values

• Red checks at this head, each fix reverted on its own, made its test fail: the fork reader fixes; dropping the typed filter string; awaiting the dispatch in the handler instead of a task; freeing the key when the record write fails; the old Raced envelope; user_owns_sandboxes for deleted rows; treating every RPC failure as not accepted, and every failure as possibly accepted; skipping the bearer-token read before the call; and leaving a failure without a job unnoted.

• On a local Kind stack built from this head, through the admin API with curl, with both builder nodes' build-manager stopped (SIGSTOP) behind their node-api: a keyed system build failed with node-api's own 503 after 15 s, an answer that does not prove nothing was taken. The reservation was noted unresolved, and a replay right after answered the non-retryable 409, as did one after the builder was resumed. template_build_jobs stayed empty.

  The cluster was removed afterwards.

Not validated: a mixed-version rollout with an older controller or API replica. That behaviour rests on the wire-compatibility test, the switches being off by default, and the Rollout steps.

Severity and risk

Severity: moderate. These are wrong statuses, missing provenance and missing concurrency controls on operator-only admin routes. No data is lost today.

Risk:

• With both switches off, most wire changes are additive: new response fields (reason, retained_until, requested_by_user_id), an optional header and an optional body field. These are not:
  • The debug route returns null for every env value. A script that read the values from resources.env gets null instead.
  • DELETE /users/{user_id} answers 409 with a reason where it answered 500 for a user who owns sandboxes or archived sandboxes. A client that retried the 500 gets a 409 to act on instead.
  • The admin operations target_id filter matches a sandbox's operations under both id forms, so filtering by either form returns operations main did not.
  • GET /v1/sandboxes/{id}/lifecycle-events carries the events of forks of the sandbox, in either id form, which it never did for forks recorded by the bare UUID. A client of the stream sees fork_sandbox operations' events it did not see before. The route is hidden by default.
  • An unconditional tag move that races another writer creating and then removing the same tag can answer a retryable 409 asking the caller to read the tag and retry; on main it overwrites silently. A race with a writer that only creates the tag is resolved inside the transaction.
• A system build key whose outcome is unknown stays unusable for the rest of its record's 24 hours. The operator checks Template Releases and submits again with a new key.

Rollback: revert the commits. The switches can be turned off on their own; for the fork switch, follow the steps below first.

Rollout

The API and the cluster controller can roll out in either order. Both switches default to off, and with them off the payloads and target_id values written are byte-identical to today's.

• SPACE_ADMIN_CREATE_REQUESTED_BY_ENABLED (API): older controllers reject unknown create-payload keys (deny_unknown_fields). Turn it on only once every space-cluster-controller replica runs this release. Turn it off, and let queued creates finish, before rolling a controller back. A new API reading an older operation reports requested_by_user_id: null. An older API never reads the field.
• SPACE_FORK_PUBLIC_TARGET_ID_ENABLED (API): older releases match a fork's source by the bare UUID alone, in the delete guard, snapshot retention and capture election, so they cannot see a fork recorded as sbx_<uuid>. Turn it on only once every space-api and space-cluster-controller replica runs this release. Before rolling either service back:
  1. turn the switch off on every space-api replica;
  2. wait for the fork requests those replicas were already admitting to finish;
  3. wait until SELECT operation_id, state, target_id FROM public_operations WHERE kind = 'fork_sandbox' AND state IN ('queued', 'running') AND target_id LIKE 'sbx\_%' returns no rows. The admin operations list cannot stand in for it: it returns at most 250 operations, and two separate reads can miss a fork that moves between queued and running;
  4. only then roll back.
• expected_current_release_id on the tag route: older API replicas reject unknown fields on that request with 422. Callers, the admin console included, should send it only once every API replica runs this release.
• Idempotency-Key on system template builds: older API replicas ignore the header, so during a rolling deploy a retry that reaches one starts a second build, and nothing returns an error. Callers, the admin console included, should rely on the key only once every API replica runs this release.

What an operator sees in the window after the fork switch is turned on: fork operations created before it keep the bare UUID in target_id. They stay until finished operations are garbage-collected, 7 days after creation, so for up to 7 days the operation lists show both forms side by side.

• The admin target_id filter finds both forms whichever form it is given.
• GET /v1/operations/{id} returns each operation's target_id as it was recorded.
• The source sandbox's lifecycle-events stream carries forks in both forms.
• Telemetry's canonical sandbox_id is the bare UUID either way.

Both switches, and the fork switch's rollback steps, are documented in docs/configuration.md.

Not covered

• The persisted audit row. The previous release of a tag move is recorded in the structured space.admin.audit log entry, not in the persisted admin_audit_events row that GET /audit-events serves. That table has no column for per-request details. Storing it there needs a separate migration first, adding a nullable details jsonb column to admin_audit_events; then the audit middleware can take details from a response extension the handler sets. Per db/README.md, that migration lands on its own, before the code that writes it.
• Other admin operations. requested_by_user_id is recorded only for admin creates. Admin lifecycle actions and updates already record the admin as actor_user_id.
• Builder-side deduplication. StartBuildRequest carries no caller identity, and build-manager creates a job per submission. Deduplicating in the builder would need a proto field and a build-manager change with its own rollout, so the key is enforced in the API.
• Reservations nothing settles. A panic in the submission task, an API process exit mid-submission, or a failed write of the reservation's note leaves the key's reservation pending without a note. Nothing recovers it: replays answer "in progress", then "outcome unknown" 15 minutes after the reservation, until the record expires 24 hours after it.
• Keys kept though nothing was taken. A builder refusal other than draining, such as an invalid argument, is not counted as proof that nothing was taken. A node-api that cannot reach build-manager, and a failed release, also keep the key reserved, though nothing reached build-manager; use a new key. Telling node-api's failure apart needs a node-api marker with its own rollout. A node-api bearer token that goes missing between the controller's read and the call's also keeps the key.

AI assistance: Claude Code drafted the code, tests and this description, and ran the validation above.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.