Repository navigation
api: reject custom shapes on Direct one-shot create and state its retire contract - #499
Merged
Merged
Conversation
ac-pplx
force-pushed
the
ac/direct-explicit-gaps
branch
from
October 14, 2026 23:16
b330c7f to
7fa1426
Compare
…ire contract On the Direct substrate, a create with wait_until_ready and a cold boot takes the one-shot path: a single sbx-mgr CreateRequest, which has no vCPU or memory fields, so the VM would boot with node-side defaults while public.sandboxes recorded the requested shape. Direct's create_execution now returns SubstrateError::Unsupported (HTTP 501) for such a create before any node call; the shared create path's failure handling deletes the row it inserted, best effort, as for any substrate create failure. The check reads the request's own vcpu and memory_mib. Direct fills nothing from the template, so default and disk-only creates are unaffected, and the two-call create and start still carries the full shape. No API surface sends that combination today, so this guards future callers rather than changing what a client sees: native and admin creates never wait (controller_create_request), ECW creates carry no vCPU or memory, and fork restores stop at Direct's fork refusal. A snapshot or pool boot with a custom shape was already a 400 from CreateBootMode::from_api. Direct's retire_execution still enqueues no stopped-disk backup. The trait doc now says only Kubernetes enqueues one. Its teardown fails the retire when a retryable enqueue fails and relies on the failed-cleanup retry, which never runs for Direct. A best-effort enqueue would not be a cheap side call either: sbx-mgr refuses it on a node without cluster-api wiring and otherwise takes a btrfs snapshot of the stopped sandbox. The direct.rs module doc drops its "Known gaps" section, and both items are now documented behavior.
ac-pplx
force-pushed
the
ac/direct-explicit-gaps
branch
from
October 14, 2026 23:29
7fa1426 to
934268a
Compare
ac-pplx
marked this pull request as ready for review
October 14, 2026 23:53
johnlouie-pplx
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
One-shot create refuses a custom shape. Direct's
create_executionnow returnsSubstrateError::Unsupported(HTTP 501, "custom vCPU/memory is not supported for one-shot create on the Direct substrate") when it would take the one-shot path (use_one_shot_create:wait_until_readyand a cold boot) and the request setsvcpuormemory_mib. Before, such a create would have succeeded and booted the VM with node-side defaults whilepublic.sandboxesrecorded the requested shape, because the one-shotCreateRequesthas no vCPU/memory fields. The refusal comes before any node call (cached_worker_create_candidatesonly queries the database). The shared create path has already inserted the sandbox row by then. Its failure path deletes that row best effort, as for any substrate create failure; if the DELETE fails, the row stays increatingand the error is demoted. A refusal also spends one cluster-admission token and emits the usual create-requested/create-failed events, again like any substrate create failure.No client sees a change today; this is a guard at the controller boundary for future callers. No API surface sends a wait-until-ready create with a custom shape:
/v1and admin creates go throughcontroller_create_request(public_sandbox_operations.rs), which setsboot_mode: Noneandwait_until_ready: false. They take the two-call create+start path, which forwards vCPU and memory.routes/e2b/translate.rs) setswait_until_ready: truebut alwaysvcpu: Noneandmemory_mib: None.fork_operation_executor.rs) setsrestore_fromand stops at Direct's existing fork refusal.So the "succeeds silently today" gap #357 recorded couldn't be reached from the API either. If a future native operation caller did send this combination, it wouldn't fail cleanly:
controller_error_result(public_sandbox_operations.rs) retries any 5xx except a 503, this 501 included, untilmax_retry_attempts.What counts as an override. The request's own
vcpuandmemory_mib. On Direct,resolves_cold_create_shapeis false, so nothing comes from the template build shape or a pool config, and default creates are never refused.disk_mibalone is accepted, because one-shot create forwards it (rootfs_size_mb). Only a cold boot reaches the check:CreateBootMode::from_apialready answers 400 for a snapshot or pool boot with a custom shape, and a custom shape with noboot_modebecomes cold.Not implemented (alternative). Direct could send such creates through the two-call create+start path, which forwards the shape in
StartRequest, instead of refusing them.No shared code changes: the shared create path (
controllers/sandbox/queries.rs), its test doubles and the Kubernetes adapter are identical to main. Theresolves_cold_create_shapedoc from #368 now says Direct refuses, instead of silently dropping, a one-shot vCPU/memory override.Retire contract (no behaviour change)
I made the contract truthful rather than adding a backup enqueue to Direct's retire.
substrate/mod.rs) said retire "explicitly enqueues a stopped-disk backup". Only the Kubernetes teardown (kubernetes/adapter/teardown.rs) does that: stop, thenenqueue_artifact_backup_with_client(Disk, CurrentInstance). It skips stop and enqueue when the pod has no reachable runtime. It lets the retire finish without a backup on the terminal refusalsretire_can_skip_backup_enqueue_errornames (no sbx-mgr row, a failed sandbox, the full-suspend-snapshot marker). On a retryable enqueue error it fails the retire, so the failed-cleanup retry keeps the execution tracked. The trait doc now says so.retire_executiononly from the transient-lifecycle sweeper's best-effortcleanup_after_sweep, and nothing retries a failed Direct retire. The Kubernetes fail-and-retry pattern doesn't carry over.EnqueueArtifactBackupreturnsFailedPreconditionon a node without cluster-api wiring (reupload_precondition_for_cluster_api_env,crates/libs/sbx/mgr/server/src/service/grpc_lifecycle/artifact_backup.rs). For a stopped instance it also takes a btrfs snapshot through node-manager. sbx-mgr'sStop(grpc_core.rs) keeps the filesystem and enqueues nothing itself.enqueue_sandbox_artifact_backupaccepts every request and records nothing.The trait doc now says only Kubernetes enqueues the stopped-disk backup. The
direct.rsmodule doc and theretire_executioncomment say Direct enqueues none, and why.Docs
The
direct.rsmodule doc drops "Known gaps". The one-shot item moves to theUnsupportedlist, worded as a guard with the three reachability facts above. The retire item becomes documented behaviour. In the substrateAGENTS.md, I only dropped the references to "known gaps"; it keeps the rule that a Direct behaviour change belongs in its own PR. It never described the guard, so nothing else there changes. I only touched the create/retire bullets and that section of the module doc, not the drain bullet or the SQL literals.Tests
direct/tests.rs, through aSandboxControlleroverSubstrate::Directand one sim worker withseedcached. Each test is listed with the mutation that makes it fail:one_shot_direct_create_boots_a_default_shape_on_the_workerone_shot_direct_create_accepts_a_custom_diskdisk_mibdisk_mibas an overrideone_shot_direct_create_refuses_a_custom_vcpuvcpuone-shot create, nosandboxesrow left, worker emptyone_shot_direct_create_refuses_a_custom_memorymemory_mibone_shot_direct_create_refuses_a_custom_vcpu_with_an_inferred_cold_bootboot_mode, waits,vcputwo_call_direct_create_boots_a_custom_shape_on_the_workerboot_mode, no wait,vcpu+memory_mibwait_until_readyThe sim worker doesn't record
StartRequest.vcpu/memory_mib, so the two-call test shows the create isn't refused but not that the shape arrives. That's a coverage gap. No existing test pinned the silent drop.Validation
cargo fmt --all -- --check934268a934268aSPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-api --all-features --retries 0 -E 'test(direct)'934268ascripts/check-rust-error-policy.sh7fa1426cargo clippy --workspace --locked --all-targets --all-features -- -D warnings7fa1426SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-api --all-features --retries 0×37fa1426#[ignore]benchmark), all three runs934268asmoke-kind-e2e.shandsmoke-kind-typed.sh all49c1038The final revision
934268achanges only doc comments since7fa1426, so I didn't rerun clippy or the ×3 suite on it. Kind wasn't rerun for this revision either. It runs the Kubernetes substrate, and this revision leaves the shared create path and the Kubernetes adapter identical to main; it changes only Direct code, the trait's doc comments, and tests.🤖 Generated with Claude Code