You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[cr-test] api: refuse a template delete while warm pools still use it - #1447
A template delete no longer hangs while warm pools still use the template. The admin route refuses it up front with a 409 that names each use and says how it clears. Once those uses clear, the delete removes the leftover disabled pool rows and completes.
Before this change, a template delete removed the template's releases in one transaction. Pool configs, pool slots and sandbox archives reference releases with ON DELETE RESTRICT, so the delete raised a foreign-key violation whenever any of them pointed at one of the template's releases. The operation treated that as a transient database error and requeued every 5 s until its retry budget ran out. The template stayed deleting, and every replacement delete failed the same way. Nothing removed the rows:
disabling a manual pool config keeps its row;
the reconciler deletes only empty, disabled autoscaler configs.
So a template that ever had a manual pool could not be deleted. The delete also left the name's warm_pool_policies rows for a recreated template to inherit.
Merging this:
Refuses up front. The admin delete route checks what still uses the template before it creates an operation or marks the template deleting. It answers 409 with a message naming each use, and structured details:
reason=template_in_use;
enabled_pool_config_ids;
enabled_pool_policies (shapes);
open_pool_slots, counting warming, ready, claimed and cleanup_pending slots;
sandbox_held_pool_slots, counting bound slots;
pool_slot_sandbox_ids, up to 5.
The message gives only the remedies that apply:
Disable the configs and policies and wait for open slots to drain. A slot claimed by an in-flight create settles within minutes.
A bound slot clears when its sandbox's pod is retired, for example when the sandbox is deleted or suspended.
Deleting a single release already answers 409 when anything still references it.
Rechecks in the operation. The operation runs the same check, and fails with a conflict and the same details instead of requeueing 16 times:
before it marks a template that is still present;
again in the transaction that deletes the releases.
An operation can find the template still present in two cases:
the route's mark failed, and the template stays present;
the dispatcher ran the operation before the route's mark. A use that commits in that gap fails the operation, then the mark sets deleting and the route answers 202.
As before, a template already deleting stays deleting, and a later delete with a new Idempotency-Key replaces the failed operation.
Cleans up what no longer matters. In that transaction the operation deletes the template's disabled pool configs that have no slots, manual or autoscaler, and its disabled policies. Enabled rows stay. The configs are locked before their slots are counted, so a slot that commits meanwhile is not deleted with its config.
The reconciler is unchanged, because space-admin-cli warm-pool ... --disabled stages a disabled manual config on purpose.
If a foreign-key violation still occurs and the re-read finds no use, for example a disabled config committed meanwhile, the operation requeues, and the retry cleans up.
Store methods.WarmPoolStore gains template_usage and delete_disabled_for_template.
Docs. The admin OpenAPI lists the new 409. docs/warm-pools.md and the admin CLI README say how each use clears, that a retry after a failed delete needs a new Idempotency-Key, and that an accepted delete can still fail. The warm-pool design doc states the cleanup rule.
Validation
At fbe8cab36, after review: the route runs the use check on the idempotency admission's transaction, so a request holds one connection, and the design note states that a policy enabled while the delete transaction runs survives it. New test an_in_use_refusal_needs_one_connection gets the 409 on a one-connection pool, 10 of 10 runs, and fails with a 500 after the acquire timeout on the old two-connection form. nextest passes 2307 tests in pplx-space-api and pplx-space-cluster-pool, and workspace clippy passes in CI.
Rebased again onto main after the warm-pool autoscale change (#1154) merged, as 0cd65cb94. The only conflicts were the cluster-pool crate's models re-export and import lists, which now name both PRs' types; every other line this PR changes is the same as before. At this head: cargo fmt, workspace clippy with all features and -D warnings, cargo check --workspace --all-targets, and cargo arch against main's baseline pass, and nextest passes 3554 tests against Postgres (pplx-space-api, pplx-space-cluster-pool, cluster-controller server).
Rebased onto the crate layer move (#1368) as a2f363b79; every changed line is the same as before, under the new crate paths. At this head: cargo fmt, cargo check --workspace --all-targets with default and all features, and cargo arch pass, and nextest passes 2302 tests against Postgres (pplx-space-api, pplx-space-cluster-pool).
At 6eaed99af:
cargo fmt, workspace clippy with -D warnings and cargo arch pass.
nextest passes, all features, against Postgres and Redis: pplx-space-api and pplx-space-cluster-pool, 2228 tests. One test is skipped: the #[ignore] benchmark.
New tests, each of which fails on main:
delete_is_refused_while_the_template_is_in_use: an enabled config, an enabled policy, an open slot, a claimed slot, one bound slot and six bound slots each get a 409 with only their own remedy and matching details. No operation is created, and the template stays present.
delete_removes_disabled_empty_pool_configs_and_policies: a disabled empty manual config, a disabled empty autoscaler config and a disabled policy are removed, and the delete reaches deleted.
an_unmarked_delete_of_a_template_in_use_leaves_it_present: an operation that finds a use before marking fails with the conflict, and leaves the template present.
a_use_added_after_the_delete_was_accepted_fails_the_operation: an enabled policy added after acceptance fails the operation with a conflict. After it is disabled, a new delete completes.
a_slot_committed_while_the_delete_runs_fails_it_terminally: a slot committed while the delete waits on its lock becomes a terminal conflict, and the slot is not deleted.
a_disabled_config_committed_while_the_delete_runs_is_retried_away: the foreign-key violation requeues the operation, and the retry removes the config and completes.
Each behaviour fails a named test when it is removed:
the route check;
the cleanup;
the lock-then-count order;
the pre-mark check;
the in-transaction re-check;
terminal versus requeue on the foreign-key path;
the open versus bound slot split, including claimed counted as held;
the details in the route and in the operation's error.
Not validated: a Kind or deployed run.
Severity and risk
Severity: medium. A template that ever had a manual warm pool could not be deleted. Each attempt retried for minutes, then left the name stuck in deleting.
Risk:
Deletes that used to hang now return 409 until the uses clear.
Policy-only templates: a template with an enabled policy but no pool config yet used to delete, and now gets 409 until the policy is disabled.
Right after a disable: a delete right after disabling a pool gets 409 until its open slots drain.
bound slots clear only when their sandboxes are deleted or suspended.
Idempotency replay: replaying the same Idempotency-Key after the original operation went terminal now gets 409 while something uses the template, rather than the old operation id. A refusal never creates an operation.
Rollback: revert. There is no schema change.
Rollout
The route runs in the API service, and the delete operation runs in the cluster controller, which links the same crate.
An old API replica accepts a delete without the check, as today. A new controller then completes it, or fails it with the conflict message once nodes have dropped the template.
While a pool's pods keep the template on a node, the node refuses to delete its copy, so the operation waits and ends internal at its retry budget, as before. Warm-pool pods count as dependent sandboxes on their node. This includes a pool config enabled after the delete was accepted, because the reconciler fills it regardless of the template's state.
An old controller runs any delete the old way.
There is no schema change, so either rollout order is safe.
Not covered
Policy writes during or after the delete: a policy enabled after the operation's in-transaction check, or written for the name after the delete, survives it, and a recreated template inherits it. Closing this needs both: the delete transaction locks the template row before its check, and the policy upsert refuses unless the template is present.
Single-release delete:DELETE /admin/api/v1/template-releases/{id} still answers 409 for a disabled, empty manual config. A whole-template delete now removes such a config; a per-release form of this cleanup would cover the single-release path.
Archives: a sandbox_archives row still blocks through its foreign key and the bounded retry, as on main. Nothing writes that table today.
AI assistance: Claude Code wrote the code, the tests and this description, and ran the validation and the mutation checks.
A template delete removes the template's releases in one transaction.
Pool configs, pool slots and sandbox archives reference releases with
ON DELETE RESTRICT, so when any of them pointed at one of the
template's releases the delete raised a foreign-key violation. The
operation treated that as a transient database error and requeued
every 5 seconds until its retry budget ran out. The template stayed
`deleting`, and each replacement delete failed the same way. Nothing
removed the rows: disabling a manual pool config keeps the row, and
the reconciler deletes only empty disabled autoscaler configs. The
delete also left the name's warm_pool_policies rows for a recreated
template to inherit.
- The admin delete route checks what warm pools still hold of the
template before it creates an operation or marks the template
`deleting`. It answers 409 naming each use. The message names enabled
pool configs by id and managed_by, enabled policy shapes, the number
of open slots (warming, ready, claimed or cleanup_pending), and the
number of slots held by sandboxes created from the pool (bound), with
up to five of those sandboxes' ids. `details` carries `reason:
template_in_use` and, for each use present, `enabled_pool_config_ids`
(ids only), `enabled_pool_policies`, `open_pool_slots`,
`sandbox_held_pool_slots` and `pool_slot_sandbox_ids`. A claimed slot
counts as open, not as held: it has no sandbox row until its bind
commits, and it settles without the operator. The advice matches
each kind: disable the configs and policies and wait for the open
slots to drain, a slot claimed by a create still in flight settling
within minutes when the create finishes or the stale claim is swept;
a slot held by a sandbox clears when that sandbox's pod is retired,
for example when the sandbox is deleted or suspended.
- The delete operation runs the same check before it marks a template
that is still `present` as `deleting`. The route commits the
operation before its mark transaction, so the operation sees a
`present` template in two cases. If the route's mark failed, a
same-key replay gets the route's 409, and the operation now fails
with the conflict and leaves the template `present`, instead of
marking it `deleting` and removing its node copies. If the
dispatcher runs the operation before the route's mark and a use
committed after the route's check, the operation fails with the
conflict, and the route's mark then sets `deleting` and answers 202;
as with any failed delete, a later delete with a new Idempotency-Key
replaces it.
- The operation checks again in the transaction that deletes the
releases. It then deletes the template's disabled pool configs that
have no slots, manual or autoscaler, and its disabled policies.
Enabled rows stay. The reconciler is unchanged, because
`warm-pool ... --disabled` stages a disabled manual config on
purpose. The configs are locked before slots are counted, so a slot
that commits meanwhile is not cascade-deleted with its config.
- A use found by either operation check fails the operation
terminally with a conflict and the same message and details. So does
a foreign-key violation from the release delete when a re-read finds
a use. When the re-read finds none, for example for a disabled
config written after the cleanup took its lock, the error goes back
to the dispatcher, which requeues it as before, and the retry's
cleanup deletes the config. A failed delete still leaves the
template `deleting`; a later delete with a new Idempotency-Key
replaces it.
- Sandbox archives are not checked, because nothing writes that table
yet. Their release key still blocks the delete, which then retries
until its budget runs out, as before.
- WarmPoolStore gains template_usage and delete_disabled_for_template.
The admin OpenAPI lists the new 409, and the operator docs say how
each use clears, that a pool config enabled after the delete was
accepted can instead keep the template on nodes until the operation
fails with `internal` at its retry budget, and that a retry after a
failed delete needs a new Idempotency-Key.
Mixed versions: there is no schema change. The route runs in the API
service; the delete operation runs in the cluster controller, which
links the same crate. An old API replica accepts a delete without the
check, as today. A new controller then completes it, or fails it with
the conflict message once nodes have dropped the template. While a
pool's pods keep the template on a node, the node refuses to delete
it, so the operation waits and ends at its retry budget, as before. An
old controller runs any delete the old way, retrying until its budget
runs out while something still references a release. Either rollout
order is safe.
Known gaps, not changed here. A policy enabled after the operation's
in-transaction check, or written for the name after the delete,
survives it, and a recreated template inherits it: the policy upsert
takes no template lock and does not check the template's state, and
the delete transaction would also have to lock the template row before
its check. Separately, DELETE /admin/api/v1/template-releases/{id}
still answers 409 for a disabled, empty manual config, which nothing
removes; a per-release form of this cleanup would cover it.
The route checked warm-pool use before looking up the Idempotency-Key,
so once a delete had failed because a use appeared, replaying its key
while the use remained got a fresh 409 instead of the original
operation, against the documented replay behaviour. The route now opens
the key's admission first and checks use only for a new key, as the
sandbox create path does for its quota; a replayed or conflicting key
answers as before.
A test replays the failed delete's key while the policy is still
enabled and gets the original operation, while a new key is still
refused with no operation recorded. The executor and test comments that
described a same-key replay getting the route's 409 are updated.
The delete route held its idempotency admission transaction while the
use check took a second pooled connection, so enough concurrent new
deletes could each hold one connection and wait for another. The check
now reads on the admission's transaction, as the sandbox create path
does for its quota check. A test refuses an in-use delete on a
one-connection pool.
The design note's delete paragraph also says what the final in-
transaction check catches, and that a policy enabled while the delete
runs survives it and applies to a template later created with the same
name.
Why: The deletion safeguards have relevant tests, but their changed database queries and cleanup behavior warrant validation before merge.
Commit: 392a32e96a59.
No actionable comments were generated in the recent review.
Walkthrough
Template deletion now checks whether warm-pool configs, policies, or slots use the template. The API returns structured conflicts for detected usage and removes eligible disabled configs and policies during deletion. Checks also run during deletion to detect uses added after admission. Tests and documentation cover these conditions, cleanup, and retry behavior.
Possibly related PRs
perplexityai/SPACE#1224: Adds related warm-pool usage checks, cleanup, conflict handling, tests, and documentation for template deletion.
Merge Risk: Moderate · up to 392a3
The deletion safeguards have relevant tests, but their changed database queries and cleanup behavior warrant validation before merge.
Security architecture risk: Moderate · up to 392a3
The admission checks and transactional cleanup improve deletion safety. However, detailed administrative diagnostics now enter operation records with an organization-scoped read path. Administrator-only access to those details has not been established end to end. The potential exposure is operational metadata, not demonstrated access to sandbox contents or administrative actions.
Retained concerns
Medium · security · inferred: New administrative warm-pool diagnostics may cross into a less-privileged operation-read surface. A delete encountering late usage stores template-wide pool metadata and sandbox identifiers in its terminal error. The shared lookup accepts organization-matching Admin operations and returns that error without redaction. The complete native owner-scoping enforcement remains unverified, so this is a conditional disclosure concern, not a verified authorization bypass. The lookup predates the PR; the additional diagnostic payload is new.
Security review details
Security Blast Radius
inferred — The potential diagnostic disclosure is bounded to a known deletion operation whose organization matches the authenticated effective organization, which is the managing organization for admin deletes. Its payload can describe cluster-wide use of that template, including sandbox identifiers outside the reader's own assets. Arbitrary-tenant access, operation enumeration, sandbox-content access, and additional write authority have not been established.
Security Findings and Attack Paths
inferred — A conditional read path exists from newly persisted administrative conflict details to native operation retrieval: a privileged delete encounters late usage, records detailed failure data, and an organization-matching reader requests its operation ID. The store does not exclude Admin-product rows and the response conversion preserves the terminal JSON. Whether the full native owner-scoping gate permits a non-admin reader remains unresolved; this is not a verified exploit.
Trust Boundaries and Controls
observed — The delete handler checks managing-organization-admin authorization before resource access, and admin operation retrieval independently requires the same privilege. Native operation retrieval instead receives AuthOrg, behind native authentication and owner-scoping middleware. Native authentication explicitly permits GET_OPERATION for service credentials without an actor. These are distinct enforcement paths, not evidence of a user-to-admin identity escalation.
Resilience and Maintainability Implications
observed — The new cleanup contains partial failure by keeping eligible-row removal and release deletion transactional. Name-based policy writes remain unsynchronized: a policy enabled after the final check can survive deletion and apply to a recreated template. Policy retention also existed at the merge base, so this remains a lifecycle limitation rather than a newly introduced security concern.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title "[cr-test] api: refuse a template delete while warm pools still use it" names the area (api) and describes the main change: template deletion is refused while warm pools still reference the …
Description Follows The Pr Template
The PR description contains all required sections and content: ## Summary explains the merged behavior; ## Validation names commands, test suites, and passing results; ## Severity and risk gives…
Bug Fixes Name A Regression Test
The pull request fixes application behavior under crates/. The Validation section names regression tests, including delete_is_refused_while_the_template_is_in_use, `delete_removes_disabled_empty_p…
Weakened Safeguards Are Justified
No weakened safeguard is introduced. The PR changes only Rust source, tests, and documentation; it does not change Cargo lints, CI workflows, nextest configuration, allowlists, baselines, or repositor…
Public-Safe Content
PASS. The reviewed diff contains no credentials, real tokens, private-key material, internal hostnames or URLs, private tracker/chat/document links, cloud account IDs or ARNs, workstation paths, produ…
Full details: Title check
Explanation
The title "[cr-test] api: refuse a template delete while warm pools still use it" names the area (api) and describes the main change: template deletion is refused while warm pools still reference the template. The changeset matches this, including the 409 admission check, the in-transaction recheck, and the new TemplateWarmPoolUsage model. The title uses the area-prefix form and starts the summary with a lowercase word. The bracketed "[cr-test]" tag is a test marker, not a ticket ID, so it does not violate the requirements. No trailing period is present.
Full details: Description Follows The Pr Template
Explanation
The PR description contains all required sections and content: ## Summary explains the merged behavior; ## Validation names commands, test suites, and passing results; ## Severity and risk gives medium severity and specific risks; and the line starting AI assistance: identifies Claude Code and the drafted or changed parts.
Full details: Bug Fixes Name A Regression Test
Explanation
The pull request fixes application behavior under crates/. The Validation section names regression tests, including delete_is_refused_while_the_template_is_in_use, delete_removes_disabled_empty_pool_configs_and_policies, and a_slot_committed_while_the_delete_runs_fails_it_terminally. It states that each test fails without the related fix and reports passing test runs. The diff confirms these tests were added with the template-delete behavior changes.
Full details: Weakened Safeguards Are Justified
Explanation
No weakened safeguard is introduced. The PR changes only Rust source, tests, and documentation; it does not change Cargo lints, CI workflows, nextest configuration, allowlists, baselines, or repository guards. The existing tests remain, and the diff adds deletion and warm-pool coverage with assertions. Removed lines are imports, refactored operation code, and documentation text, not tests or assertions.
Full details: Public-Safe Content
Explanation
PASS. The reviewed diff contains no credentials, real tokens, private-key material, internal hostnames or URLs, private tracker/chat/document links, cloud account IDs or ARNs, workstation paths, production logs or measurements, incident details, or undisclosed security vulnerability. The only credential-shaped test field is an empty password_hash in a fixture. The description contains a public GitHub link and test/validation counts only.
Recent review info
Automatic approval policy
Commit: 392a32e96a59.
Setting
Effective value
Source
Automatic approvals
Enabled
Global configuration
Merge risk
Low
Override 2
Security risk
Not set
Global configuration
Matching overrides (numbered as in the resolved configuration):
Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.
Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 3 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A template delete no longer hangs while warm pools still use the template. The admin route refuses it up front with a 409 that names each use and says how it clears. Once those uses clear, the delete removes the leftover disabled pool rows and completes.
Before this change, a template delete removed the template's releases in one transaction. Pool configs, pool slots and sandbox archives reference releases with
ON DELETE RESTRICT, so the delete raised a foreign-key violation whenever any of them pointed at one of the template's releases. The operation treated that as a transient database error and requeued every 5 s until its retry budget ran out. The template stayeddeleting, and every replacement delete failed the same way. Nothing removed the rows:So a template that ever had a manual pool could not be deleted. The delete also left the name's
warm_pool_policiesrows for a recreated template to inherit.Merging this:
Refuses up front. The admin delete route checks what still uses the template before it creates an operation or marks the template
deleting. It answers 409 with a message naming each use, and structureddetails:reason=template_in_use;enabled_pool_config_ids;enabled_pool_policies(shapes);open_pool_slots, countingwarming,ready,claimedandcleanup_pendingslots;sandbox_held_pool_slots, countingboundslots;pool_slot_sandbox_ids, up to 5.The message gives only the remedies that apply:
boundslot clears when its sandbox's pod is retired, for example when the sandbox is deleted or suspended.Deleting a single release already answers 409 when anything still references it.
Rechecks in the operation. The operation runs the same check, and fails with a conflict and the same details instead of requeueing 16 times:
present;An operation can find the template still
presentin two cases:present;deletingand the route answers 202.As before, a template already
deletingstaysdeleting, and a later delete with a newIdempotency-Keyreplaces the failed operation.Cleans up what no longer matters. In that transaction the operation deletes the template's disabled pool configs that have no slots, manual or autoscaler, and its disabled policies. Enabled rows stay. The configs are locked before their slots are counted, so a slot that commits meanwhile is not deleted with its config.
space-admin-cli warm-pool ... --disabledstages a disabled manual config on purpose.Store methods.
WarmPoolStoregainstemplate_usageanddelete_disabled_for_template.Docs. The admin OpenAPI lists the new 409.
docs/warm-pools.mdand the admin CLI README say how each use clears, that a retry after a failed delete needs a newIdempotency-Key, and that an accepted delete can still fail. The warm-pool design doc states the cleanup rule.Validation
At
fbe8cab36, after review: the route runs the use check on the idempotency admission's transaction, so a request holds one connection, and the design note states that a policy enabled while the delete transaction runs survives it. New testan_in_use_refusal_needs_one_connectiongets the 409 on a one-connection pool, 10 of 10 runs, and fails with a 500 after the acquire timeout on the old two-connection form. nextest passes 2307 tests inpplx-space-apiandpplx-space-cluster-pool, and workspace clippy passes in CI.Rebased again onto main after the warm-pool autoscale change (#1154) merged, as
0cd65cb94. The only conflicts were the cluster-pool crate'smodelsre-export and import lists, which now name both PRs' types; every other line this PR changes is the same as before. At this head:cargo fmt, workspace clippy with all features and-D warnings,cargo check --workspace --all-targets, andcargo archagainst main's baseline pass, and nextest passes 3554 tests against Postgres (pplx-space-api,pplx-space-cluster-pool, cluster-controller server).Rebased onto the crate layer move (#1368) as
a2f363b79; every changed line is the same as before, under the new crate paths. At this head:cargo fmt,cargo check --workspace --all-targetswith default and all features, andcargo archpass, and nextest passes 2302 tests against Postgres (pplx-space-api,pplx-space-cluster-pool).At
6eaed99af:cargo fmt, workspace clippy with-D warningsandcargo archpass.pplx-space-apiandpplx-space-cluster-pool, 2228 tests. One test is skipped: the#[ignore]benchmark.New tests, each of which fails on
main:delete_is_refused_while_the_template_is_in_use: an enabled config, an enabled policy, an open slot, a claimed slot, one bound slot and six bound slots each get a 409 with only their own remedy and matching details. No operation is created, and the template stayspresent.delete_removes_disabled_empty_pool_configs_and_policies: a disabled empty manual config, a disabled empty autoscaler config and a disabled policy are removed, and the delete reachesdeleted.an_unmarked_delete_of_a_template_in_use_leaves_it_present: an operation that finds a use before marking fails with the conflict, and leaves the templatepresent.a_use_added_after_the_delete_was_accepted_fails_the_operation: an enabled policy added after acceptance fails the operation with a conflict. After it is disabled, a new delete completes.a_slot_committed_while_the_delete_runs_fails_it_terminally: a slot committed while the delete waits on its lock becomes a terminal conflict, and the slot is not deleted.a_disabled_config_committed_while_the_delete_runs_is_retried_away: the foreign-key violation requeues the operation, and the retry removes the config and completes.Each behaviour fails a named test when it is removed:
boundslot split, includingclaimedcounted as held;Not validated: a Kind or deployed run.
Severity and risk
Severity: medium. A template that ever had a manual warm pool could not be deleted. Each attempt retried for minutes, then left the name stuck in
deleting.Risk:
boundslots clear only when their sandboxes are deleted or suspended.Idempotency-Keyafter the original operation went terminal now gets 409 while something uses the template, rather than the old operation id. A refusal never creates an operation.Rollout
The route runs in the API service, and the delete operation runs in the cluster controller, which links the same crate.
internalat its retry budget, as before. Warm-pool pods count as dependent sandboxes on their node. This includes a pool config enabled after the delete was accepted, because the reconciler fills it regardless of the template's state.Not covered
present.DELETE /admin/api/v1/template-releases/{id}still answers 409 for a disabled, empty manual config. A whole-template delete now removes such a config; a per-release form of this cleanup would cover the single-release path.sandbox_archivesrow still blocks through its foreign key and the bounded retry, as onmain. Nothing writes that table today.AI assistance: Claude Code wrote the code, the tests and this description, and ran the validation and the mutation checks.