Skip to content

api: answer 409 for org deletes with users and duplicate emails, and name the actor on admin operations - #1384

Merged
ac-pplx merged 4 commits into
mainfrom
ac/admin-api-refusals
Nov 3, 2026
Merged

ac-pplx merged 4 commits into
mainfrom
ac/admin-api-refusals

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three admin API fixes on /admin/api/v1, extending the identity error's
reason field that #1303 introduces.

  • Org delete refusals. DELETE /orgs/{org_id} used to answer 500 for
    every foreign-key refusal. It now answers 409 with a reason naming what
    still references the org, and a message saying what to do:

    • org_has_users: delete or move the users first.
    • org_has_sandboxes: deleted sandboxes kept for retention included.
    • org_has_archived_sandboxes.
    • org_has_templates: at least one template is not deleted.
    • org_has_retained_deleted_templates: every template is deleted, but
      tombstone retention still keeps the rows, which the template routes no
      longer show. The body's retained_until says about when the last one
      goes. It is omitted, and the message says to retry later, while a node
      still caches one, because no fixed time bounds that.
    • org_has_orphaned_snapshot_storage: the org has no sandboxes left, but
      its master key still encrypts snapshot storage locations that its deleted
      snapshots left behind. The snapshot storage sweeper skips encrypted
      locations, so neither deleting sandboxes nor waiting clears them. While
      the org has sandbox rows, the same constraint answers org_has_sandboxes.
    • org_has_node_groups: release each group with :assign-org and a null
      org_id, or delete it once its nodes are relabeled or retired.
    • org_manages_cluster: the org became the managing org during the request.

    Every key that can refuse the delete was checked against the migrated
    schema. That includes the five encrypted-row constraints that restrict the
    delete of the org master key, which the org's delete cascades to. API keys,
    credentials, operations and the other ON DELETE CASCADE rows are deleted
    with the org and never block it. Child organizations are detached: their
    parent_id becomes null. That behavior predates this PR, and the OpenAPI
    text now says it.

  • Email in use. POST /orgs/{org_id}/users and PATCH /users/{user_id}
    used to answer 500 for a unique violation on users_email_key. They now
    answer 409 {"error": "email already in use", "reason": "email_in_use"}.
    Other unique violations still answer 500.

  • Actor email on admin operations. GET /operations and
    GET /operations/{operation_id} return a nullable actor_email on each
    operation. It is the email actor_user_id's user has when the operation is
    read, so it follows a change of email. It is null when the operation has no
    actor, which includes once the user is deleted, because the delete also
    clears actor_user_id. It is not the actor_email of an admin audit event,
    which is a never-null copy taken when the request ran. It is read with a
    primary-key LEFT JOIN users in the store's existing admin statements, so
    no read runs per row, and the console can drop its per-row
    GET /users/{id} lookups.

The refusal is refined only after a delete has failed:

  • A new admin template-store read, org_template_rows, mirrors the tombstone
    reaper's conditions to tell live templates from retained deleted ones.
  • A new sandbox-store verb, org_has_sandbox_rows, is one EXISTS on
    idx_sandboxes_org_id.

The OpenAPI documents the new 409s and actor_email. The body-shape comment
in routes/admin/mod.rs now says every identity 409 carries a reason, and
names both retained_until cases.

Validation

  • cargo nextest run -p pplx-space-api --all-features: 2195 passed and 1
    skipped (an existing #[ignore] benchmark), with Postgres and Redis from
    the nextest setup scripts and no dependency skips.

  • New Postgres-backed tests:

    • delete_org_answers_conflict_naming_what_still_references_it inserts
      each blocker alone. A soft-deleted template row checks retained_until
      against its delete request plus the retention period. A node inventory
      row for it checks that retained_until is omitted. An orphaned encrypted
      snapshot storage location with no sandboxes checks its reason, and
      adding a sandbox refines it to org_has_sandboxes. The final step checks
      success with a cascading API key and a detached child org.
    • org_delete_blockers_classify_the_database_constraint covers all five
      master-key constraints and the managing-org constraint, classified from
      real database errors.
    • an_email_in_use_answers_conflict_on_create_and_update.
    • admin_operations_name_the_actor_by_email covers a renamed actor, and a
      deleted actor whose operation reads as one with no actor, on both the
      list and the get.
  • Without the fixes, these tests fail:

    • With the 500 mapping, the org-delete test gets a 500, and the create
      and update halves of the email test each get a 500.
    • With deleted templates and the snapshot-storage constraint classified
      as the plain template and sandbox cases, the org-delete test gets
      org_has_templates and the classification test gets Sandboxes.
    • Without the join, the operations test gets a null email.
  • cargo clippy --all-targets --all-features -- -D warnings passes for
    pplx-space-api, pplx-space-sandbox-store and
    pplx-space-cluster-controller-server, and cargo fmt --all -- --check
    passes.

  • cargo arch against main's baseline and architecture reports the same
    findings at this head as at the previous head of this branch.

  • Kind (Kubernetes substrate), run at the previous head and not rerun for the
    refusal refinements:

    • Through /admin/api/v1 with a synthetic org and users, a duplicate email
      answered 409 email_in_use on create and on update.
    • Deleting the org with users answered 409 org_has_users, and with them
      gone, 204.
    • An admin sandbox create's operation reported its owner's actor_email
      on list and get.

    The other blockers can't be created through the API, so the Postgres-backed
    tests cover them.

Severity and risk

Severity: low to moderate. Operators saw 500s for ordinary refusals. They
could not tell what blocked an org delete, and for deleted templates and
orphaned snapshot storage, nothing they could do would clear it.

Risk: low. The changes are additive. Statuses move from 500 to 409 only for
the named constraints. retained_until and actor_email are new response
fields. The refinement reads run only after a refused delete. The response
structs are serialize-only, and nothing in the workspace decodes these
responses with deny_unknown_fields. Roll back by reverting.

Stack

Stacked on #1303 (ac/admin-api-console-gaps), which introduces the identity
reason field and the body-shape comment this extends. Merge after it.

Not covered

  • When several kinds of row block an org delete, the 409 names the first one
    the database checks. The next attempt names the next.
  • There is no admin API to move a user to another org. The org_has_users
    message says to delete or move them, as asked.
  • No production code writes encrypted snapshot storage locations today, so
    org_has_orphaned_snapshot_storage is reachable only through the schema.
    No endpoint purges them yet: the sweeper's encrypted-row skip is the
    documented follow-up in snapshot_storage_sweeper/mod.rs.
  • While a node still caches a deleted template, the refusal names no end to
    retention. It clears when that node's inventory no longer lists the
    template.

AI assistance: Claude Code drafted the code, tests and this description.

@github-actions github-actions Bot added the triage:pending Waiting for a maintainer to triage label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

@ac-pplx
ac-pplx added this pull request to stack #1403 November 3, 2026 21:06
@ac-pplx
ac-pplx force-pushed the ac/admin-api-refusals branch from 7f1c8b1 to 6e4a4ab Compare November 3, 2026 22:21
Base automatically changed from ac/admin-api-console-gaps to main November 3, 2026 22:21
… use

DELETE /orgs/{org_id} surfaced every foreign-key refusal as a 500. Name
the blocking rows instead: org_has_users, org_has_sandboxes,
org_has_archived_sandboxes, org_has_templates, org_has_node_groups, or
org_manages_cluster when the org became the managing org mid-request.

Creating or updating a user with another user's email now answers 409
email_in_use, matched on users_email_key rather than any 23505.
The admin operations list and get now return actor_email, the current
email of actor_user_id's user, joined in the same statement, or null when
there is no actor or the user is gone. The console no longer needs a user
read per row.
…ete refusals

A deleted template's row keeps its org_id until tombstone retention
removes it, a retention period after the delete request and only once no
node caches it. The template routes no longer show it, and deleting it
again answers 404, yet the org delete answered org_has_templates, asking
for a delete the operator cannot make. When every template the org owns
is deleted, the refusal is now org_has_retained_deleted_templates, with
retained_until when the end is knowable and "retry later" when a node
still caches one.

An org with no sandboxes left can still be held by a snapshot storage
location its deleted snapshots left behind, encrypted under its master
key. The snapshot storage sweeper skips encrypted locations, so neither
deleting sandboxes nor waiting clears it. The master-key constraint now
answers org_has_orphaned_snapshot_storage once the org has no sandbox
rows, and org_has_sandboxes while it has any.

The node-group refusal now names both ways out: release the group with
:assign-org, or delete it once its nodes are relabeled or retired. The
OpenAPI text says the delete detaches child organizations, and the
classification test covers all five constraints that restrict a master
key's delete.
…s audit events

The docs said actor_email is the field admin audit events carry. An audit
event's actor_email is a never-null copy taken when the request ran; an
operation's is read with it, so it follows a change of email, and it is
null once the user is deleted, because the delete also clears
actor_user_id. The OpenAPI text and the record's doc now say so.

That same clearing meant the test's deleted-actor case could not fail
apart from its no-actor case, so they are one case, which also asserts
the cleared actor_user_id. Two comments on the admin reads now describe
both extra columns they select, not one payload-derived expression.
@ac-pplx
ac-pplx force-pushed the ac/admin-api-refusals branch from 6e4a4ab to 0405325 Compare November 3, 2026 22:21
@ac-pplx
ac-pplx added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 5bce6db Nov 3, 2026
41 checks passed
@ac-pplx
ac-pplx deleted the ac/admin-api-refusals branch November 3, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage:pending Waiting for a maintainer to triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants