Skip to content

api: reject custom shapes on Direct one-shot create and state its retire contract - #499

Merged
ac-pplx merged 1 commit into
mainfrom
ac/direct-explicit-gaps
Oct 15, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/direct-explicit-gaps

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

One-shot create refuses a custom shape. Direct's create_execution now returns SubstrateError::Unsupported (HTTP 501, "custom vCPU/memory is not supported for one-shot create on the Direct substrate") when it would take the one-shot path (use_one_shot_create: wait_until_ready and a cold boot) and the request sets vcpu or memory_mib. Before, such a create would have succeeded and booted the VM with node-side defaults while public.sandboxes recorded the requested shape, because the one-shot CreateRequest has no vCPU/memory fields. The refusal comes before any node call (cached_worker_create_candidates only queries the database). The shared create path has already inserted the sandbox row by then. Its failure path deletes that row best effort, as for any substrate create failure; if the DELETE fails, the row stays in creating and the error is demoted. A refusal also spends one cluster-admission token and emits the usual create-requested/create-failed events, again like any substrate create failure.

No client sees a change today; this is a guard at the controller boundary for future callers. No API surface sends a wait-until-ready create with a custom shape:

  • Native /v1 and admin creates go through controller_create_request (public_sandbox_operations.rs), which sets boot_mode: None and wait_until_ready: false. They take the two-call create+start path, which forwards vCPU and memory.
  • ECW/E2B create (routes/e2b/translate.rs) sets wait_until_ready: true but always vcpu: None and memory_mib: None.
  • Fork restore (fork_operation_executor.rs) sets restore_from and stops at Direct's existing fork refusal.

So the "succeeds silently today" gap #357 recorded couldn't be reached from the API either. If a future native operation caller did send this combination, it wouldn't fail cleanly: controller_error_result (public_sandbox_operations.rs) retries any 5xx except a 503, this 501 included, until max_retry_attempts.

What counts as an override. The request's own vcpu and memory_mib. On Direct, resolves_cold_create_shape is false, so nothing comes from the template build shape or a pool config, and default creates are never refused. disk_mib alone is accepted, because one-shot create forwards it (rootfs_size_mb). Only a cold boot reaches the check: CreateBootMode::from_api already answers 400 for a snapshot or pool boot with a custom shape, and a custom shape with no boot_mode becomes cold.

Not implemented (alternative). Direct could send such creates through the two-call create+start path, which forwards the shape in StartRequest, instead of refusing them.

No shared code changes: the shared create path (controllers/sandbox/queries.rs), its test doubles and the Kubernetes adapter are identical to main. The resolves_cold_create_shape doc from #368 now says Direct refuses, instead of silently dropping, a one-shot vCPU/memory override.

Retire contract (no behaviour change)

I made the contract truthful rather than adding a backup enqueue to Direct's retire.

  • The trait doc (substrate/mod.rs) said retire "explicitly enqueues a stopped-disk backup". Only the Kubernetes teardown (kubernetes/adapter/teardown.rs) does that: stop, then enqueue_artifact_backup_with_client(Disk, CurrentInstance). It skips stop and enqueue when the pod has no reachable runtime. It lets the retire finish without a backup on the terminal refusals retire_can_skip_backup_enqueue_error names (no sbx-mgr row, a failed sandbox, the full-suspend-snapshot marker). On a retryable enqueue error it fails the retire, so the failed-cleanup retry keeps the execution tracked. The trait doc now says so.
  • Direct reaches retire_execution only from the transient-lifecycle sweeper's best-effort cleanup_after_sweep, and nothing retries a failed Direct retire. The Kubernetes fail-and-retry pattern doesn't carry over.
  • A best-effort enqueue wouldn't be a cheap side call either. sbx-mgr's EnqueueArtifactBackup returns FailedPrecondition on a node without cluster-api wiring (reupload_precondition_for_cluster_api_env, crates/libs/sbx/mgr/server/src/service/grpc_lifecycle/artifact_backup.rs). For a stopped instance it also takes a btrfs snapshot through node-manager. sbx-mgr's Stop (grpc_core.rs) keeps the filesystem and enqueues nothing itself.
  • There's no test seam: the sim worker's enqueue_sandbox_artifact_backup accepts every request and records nothing.

The trait doc now says only Kubernetes enqueues the stopped-disk backup. The direct.rs module doc and the retire_execution comment say Direct enqueues none, and why.

Docs

The direct.rs module doc drops "Known gaps". The one-shot item moves to the Unsupported list, worded as a guard with the three reachability facts above. The retire item becomes documented behaviour. In the substrate AGENTS.md, I only dropped the references to "known gaps"; it keeps the rule that a Direct behaviour change belongs in its own PR. It never described the guard, so nothing else there changes. I only touched the create/retire bullets and that section of the module doc, not the drain bullet or the SQL literals.

Tests

direct/tests.rs, through a SandboxController over Substrate::Direct and one sim worker with seed cached. Each test is listed with the mutation that makes it fail:

Test Request Expects Fails under
one_shot_direct_create_boots_a_default_shape_on_the_worker cold, waits, no override succeeds, worker has the sandbox guard refuses every one-shot create
one_shot_direct_create_accepts_a_custom_disk cold, waits, disk_mib succeeds, worker has the sandbox guard counts disk_mib as an override
one_shot_direct_create_refuses_a_custom_vcpu cold, waits, vcpu 501 one-shot create, no sandboxes row left, worker empty guard removed
one_shot_direct_create_refuses_a_custom_memory cold, waits, memory_mib same guard removed
one_shot_direct_create_refuses_a_custom_vcpu_with_an_inferred_cold_boot no boot_mode, waits, vcpu same guard removed
two_call_direct_create_boots_a_custom_shape_on_the_worker the native shape: no boot_mode, no wait, vcpu + memory_mib succeeds, worker has the sandbox guard ignores wait_until_ready

The sim worker doesn't record StartRequest.vcpu/memory_mib, so the two-call test shows the create isn't refused but not that the shape arrives. That's a coverage gap. No existing test pinned the silent drop.

Validation

Check Head Result
cargo fmt --all -- --check 934268a pass
pre-commit (Gitleaks) 934268a pass
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-api --all-features --retries 0 -E 'test(direct)' 934268a 95 passed
scripts/check-rust-error-policy.sh 7fa1426 pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings 7fa1426 pass
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-api --all-features --retries 0 ×3 7fa1426 1761 passed, 1 skipped (an #[ignore] benchmark), all three runs
One mutation per new or changed test (table above) local mutations each targeted test fails, the rest pass
Shared create path and Kubernetes adapter vs main (command and output below) 934268a empty
Kind, fresh cluster: smoke-kind-e2e.sh and smoke-kind-typed.sh all previous revision 49c1038 pass (ECW 78/78; native 23 + 1 inconclusive autoscaler-bounds harness step that passed when rerun alone)
$ git diff --stat origin/main -- crates/libs/api/src/controllers/sandbox/queries.rs crates/libs/api/src/controllers/sandbox/queries/create_postgres_tests.rs crates/libs/api/src/controllers/substrate/kubernetes/
$ echo $?
0

The final revision 934268a changes only doc comments since 7fa1426, so I didn't rerun clippy or the ×3 suite on it. Kind wasn't rerun for this revision either. It runs the Kubernetes substrate, and this revision leaves the shared create path and the Kubernetes adapter identical to main; it changes only Direct code, the trait's doc comments, and tests.

🤖 Generated with Claude Code

@ac-pplx
ac-pplx force-pushed the ac/direct-explicit-gaps branch from b330c7f to 7fa1426 Compare October 14, 2026 23:16
…ire contract

On the Direct substrate, a create with wait_until_ready and a cold boot
takes the one-shot path: a single sbx-mgr CreateRequest, which has no
vCPU or memory fields, so the VM would boot with node-side defaults while
public.sandboxes recorded the requested shape. Direct's create_execution
now returns SubstrateError::Unsupported (HTTP 501) for such a create
before any node call; the shared create path's failure handling deletes
the row it inserted, best effort, as for any substrate create failure. The check reads the request's own vcpu and
memory_mib. Direct fills nothing from the template, so default and
disk-only creates are unaffected, and the two-call create and start still
carries the full shape.

No API surface sends that combination today, so this guards future
callers rather than changing what a client sees: native and admin creates
never wait (controller_create_request), ECW creates carry no vCPU or
memory, and fork restores stop at Direct's fork refusal. A snapshot or
pool boot with a custom shape was already a 400 from
CreateBootMode::from_api.

Direct's retire_execution still enqueues no stopped-disk backup. The
trait doc now says only Kubernetes enqueues one. Its teardown fails the
retire when a retryable enqueue fails and relies on the failed-cleanup retry,
which never runs for Direct. A best-effort enqueue would not be a cheap
side call either: sbx-mgr refuses it on a node without cluster-api
wiring and otherwise takes a btrfs snapshot of the stopped sandbox. The
direct.rs module doc drops its "Known gaps" section, and both items are
now documented behavior.
@ac-pplx
ac-pplx force-pushed the ac/direct-explicit-gaps branch from 7fa1426 to 934268a Compare October 14, 2026 23:29
@ac-pplx
ac-pplx marked this pull request as ready for review October 14, 2026 23:53
@ac-pplx
ac-pplx added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit f4bfdf3 Oct 15, 2026
20 checks passed
@ac-pplx
ac-pplx deleted the ac/direct-explicit-gaps branch October 15, 2026 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants