Repository navigation
api, cluster-controller: sort the tests that use the Direct substrate as a fixture (tests only) - #402
Merged
Merged
Conversation
… as a fixture (tests only) Production and Kind run the Kubernetes substrate, but several api and cluster-controller tests build a Direct substrate because it needs no apiserver. Sort the tests that construct it by what they actually depend on: - `stale_heartbeat_direct_worker_is_not_empty_cluster` pins Direct's own inventory SQL (a registered worker with a stale heartbeat still counts as eligible), so it moves, byte for byte, from `cluster_admission::tests` into `substrate::direct::cluster_admission_tests`. The admission suite's `direct_substrate` and `enabled_config` become `pub(crate)` inside that `#[cfg(test)]` module. Its `admission` helper takes a bucket type private to `cluster_admission`, so the new module builds the same bucketless admission through `ClusterSandboxAdmission::with_config`. - Tests in which Direct only fills a constructor slot, or answers a question whose answer is the same on Kubernetes (zero or some eligible nodes, or `managed_by` bound into a query with no Kubernetes-specific term), get a short comment saying Direct is a placeholder. - Tests of shared logic whose outcome or path differs on Kubernetes are left unchanged, for a later change to run on a Kubernetes fixture. No production behavior, schema, wire format, metric or configuration changes. The only edits outside test files are a `#[cfg(test)]` module declaration and `pub(crate)` on an existing `#[cfg(test)]` module declaration. The test count is unchanged.
ac-pplx
force-pushed
the
ac/resort-direct-fixture-tests
branch
from
October 13, 2026 22:06
eec96d0 to
bcd77b4
Compare
ac-pplx
marked this pull request as ready for review
October 13, 2026 22:31
Collaborator
Review: 5 findingsLow
I checked these against |
the-gigi-pplx
approved these changes
Oct 13, 2026
the-gigi-pplx
left a comment
the-gigi-pplx
left a comment
Collaborator
There was a problem hiding this comment.
Approved. 5 non-blocking findings in the review comment above.
This was referenced Oct 14, 2026
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Tests only. #402 sorted the tests that build a DirectSubstrate and listed the ones whose shared-lifecycle outcome or path differs on Kubernetes (bucket 4). This adds a Kubernetes-backed twin for 21 of the 23 api tests in that bucket and keeps every Direct test. Each twin keeps its Direct twin's setup and assertions, and its doc says whether the Kubernetes outcome is the same and, where it differs, why. Where the Direct name's claim does not hold on Kubernetes (four twins), the twin is renamed and its doc names the Direct test. The twins live in a `kubernetes` submodule beside each Direct test file and use the `kubernetes::tests` substrate fixtures and the scripted apiserver double. Shared fixtures are in `test_support::kubernetes`: an in-memory pod-store apiserver, a live sandbox's active execution row and pod, and the node-api simulator registered as a Kubernetes worker node. That module is the only place these tests construct a `Substrate` variant. A seeded execution has published its route identity unless a test asks for the legacy, route-less shape; only the two tests that hook the route publication do. The fixture's reflectors are seeded once, so pods the apiserver double creates or deletes later reach the substrate only through its uncached reads. `kubernetes::tests` gains `substrate_with_store_client_proxy_port_and_pods`, because suspend and observed-only delete need the node's node-api pod and a sandbox pod in the reflectors at once. Finding: the claim of `completed_actions_do_not_wait_for_a_key_share_lock` holds on Direct only. On Kubernetes, start and wake block behind a transaction holding `FOR KEY SHARE` on the sandbox row, which any foreign-key insert into a child table takes. They block on `SELECT lifecycle FROM sandboxes WHERE id = $1 FOR UPDATE`, run by `lock_sandbox_execution_lifecycle_in`; reading the code, that is the execution claim `ensure_pod_exists` takes before materializing the pod. Suspend blocks on its commit's `SELECT id ... FOR UPDATE`, as Direct suspend does on its artifact lock. Resume, pause, stop and kill do not block. In the test's interleaving the holder ends promptly and each waiter then completes; a longer holder can instead end the waiter through lock or statement timeouts or the activation deadline. The twin `activations_and_suspend_wait_for_a_key_share_lock` pins this with waits tied to the holder's pid through `pg_blocking_pids` and to the waiting statement's `sql!` module, all bounded. Also pinned: an untracked suspend of a `started` or `paused` Kubernetes sandbox writes its row `suspended`/`unplaced`, because Kubernetes suspend claims and finalizes the row whether or not the caller tracked it. Production code is unchanged. Not covered here: the two template builder-dispatch tests, because the Kubernetes builder client dials a fixed in-cluster service name, so a deterministic failure needs host DNS state or a production seam; and the cluster controller's `converge_cas_moves_only_restoring_rows`, which has no Kubernetes substrate fixture in that crate yet.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Tests only. #402 sorted the tests that build a DirectSubstrate and listed the ones whose shared-lifecycle outcome or path differs on Kubernetes (bucket 4). This adds a Kubernetes-backed twin for 21 of the 23 api tests in that bucket and keeps every Direct test. Each twin keeps its Direct twin's setup and assertions, and its doc says whether the Kubernetes outcome is the same and, where it differs, why. Where the Direct name's claim does not hold on Kubernetes (four twins), the twin is renamed and its doc names the Direct test. The twins live in a `kubernetes` submodule beside each Direct test file and use the `kubernetes::tests` substrate fixtures and the scripted apiserver double. Shared fixtures are in `test_support::kubernetes`: an in-memory pod-store apiserver, a live sandbox's active execution row and pod, and the node-api simulator registered as a Kubernetes worker node. That module is the only place these tests construct a `Substrate` variant. A seeded execution has published its route identity unless a test asks for the legacy, route-less shape; only the two tests that hook the route publication do. The fixture's reflectors are seeded once, so pods the apiserver double creates or deletes later reach the substrate only through its uncached reads. `kubernetes::tests` gains `substrate_with_store_client_proxy_port_and_pods`, because suspend and observed-only delete need the node's node-api pod and a sandbox pod in the reflectors at once. Finding: the claim of `completed_actions_do_not_wait_for_a_key_share_lock` holds on Direct only. On Kubernetes, start and wake block behind a transaction holding `FOR KEY SHARE` on the sandbox row, which any foreign-key insert into a child table takes. They block on `SELECT lifecycle FROM sandboxes WHERE id = $1 FOR UPDATE`, run by `lock_sandbox_execution_lifecycle_in`; reading the code, that is the execution claim `ensure_pod_exists` takes before materializing the pod. Suspend blocks on its commit's `SELECT id ... FOR UPDATE`, as Direct suspend does on its artifact lock. Resume, pause, stop and kill do not block. In the test's interleaving the holder ends promptly and each waiter then completes; a longer holder can instead end the waiter through lock or statement timeouts or the activation deadline. The twin `activations_and_suspend_wait_for_a_key_share_lock` pins this with waits tied to the holder's pid through `pg_blocking_pids` and to the waiting statement's `sql!` module, all bounded. Also pinned: an untracked suspend of a `started` or `paused` Kubernetes sandbox writes its row `suspended`/`unplaced`, because Kubernetes suspend claims and finalizes the row whether or not the caller tracked it. Production code is unchanged. Not covered here: the two template builder-dispatch tests, because the Kubernetes builder client dials a fixed in-cluster service name, so a deterministic failure needs host DNS state or a production seam; and the cluster controller's `converge_cas_moves_only_restoring_rows`, which has no Kubernetes substrate fixture in that crate yet.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Tests only. #402 sorted the tests that build a DirectSubstrate and listed the ones whose shared-lifecycle outcome or path differs on Kubernetes (bucket 4). This adds a Kubernetes-backed twin for 21 of the 23 api tests in that bucket and keeps every Direct test. Each twin keeps its Direct twin's setup and assertions, and its doc says whether the Kubernetes outcome is the same and, where it differs, why. Where the Direct name's claim does not hold on Kubernetes (four twins), the twin is renamed and its doc names the Direct test. The twins live in a `kubernetes` submodule beside each Direct test file and use the `kubernetes::tests` substrate fixtures and the scripted apiserver double. Shared fixtures are in `test_support::kubernetes`: an in-memory pod-store apiserver, a live sandbox's active execution row and pod, and the node-api simulator registered as a Kubernetes worker node. That module is the only place these tests construct a `Substrate` variant. A seeded execution has published its route identity unless a test asks for the legacy, route-less shape; only the two tests that hook the route publication do. The fixture's reflectors are seeded once, so pods the apiserver double creates or deletes later reach the substrate only through its uncached reads. `kubernetes::tests` gains `substrate_with_store_client_proxy_port_and_pods`, because suspend and observed-only delete need the node's node-api pod and a sandbox pod in the reflectors at once. Finding: the claim of `completed_actions_do_not_wait_for_a_key_share_lock` holds on Direct only. On Kubernetes, start and wake block behind a transaction holding `FOR KEY SHARE` on the sandbox row, which any foreign-key insert into a child table takes. They block on `SELECT lifecycle FROM sandboxes WHERE id = $1 FOR UPDATE`, run by `lock_sandbox_execution_lifecycle_in`; reading the code, that is the execution claim `ensure_pod_exists` takes before materializing the pod. Suspend blocks on its commit's `SELECT id ... FOR UPDATE`, as Direct suspend does on its artifact lock. Resume, pause, stop and kill do not block. In the test's interleaving the holder ends promptly and each waiter then completes; a longer holder can instead end the waiter through lock or statement timeouts or the activation deadline. The twin `activations_and_suspend_wait_for_a_key_share_lock` pins this with waits tied to the holder's pid through `pg_blocking_pids` and to the waiting statement's `sql!` module, all bounded. Also pinned: an untracked suspend of a `started` or `paused` Kubernetes sandbox writes its row `suspended`/`unplaced`, because Kubernetes suspend claims and finalizes the row whether or not the caller tracked it. Production code is unchanged. The untracked-actions twin boxes its action future. Held inline in the test's future, that future overflows libtest's default 2 MiB stack when the api crate is built at opt-level 0, as Linux CI builds it. Not covered here: the two template builder-dispatch tests, because the Kubernetes builder client dials a fixed in-cluster service name, so a deterministic failure needs host DNS state or a production seam; and the cluster controller's `converge_cas_moves_only_restoring_rows`, which has no Kubernetes substrate fixture in that crate yet.
ac-pplx
added a commit
that referenced
this pull request
Oct 17, 2026
Tests only. #402 sorted the tests that build a DirectSubstrate and listed the ones whose shared-lifecycle outcome or path differs on Kubernetes (bucket 4). This adds a Kubernetes-backed twin for 21 of the 23 api tests in that bucket and keeps every Direct test. Each twin keeps its Direct twin's setup and assertions, and its doc says whether the Kubernetes outcome is the same and, where it differs, why. Where the Direct name's claim does not hold on Kubernetes (four twins), the twin is renamed and its doc names the Direct test. The twins live in a `kubernetes` submodule beside each Direct test file and use the `kubernetes::tests` substrate fixtures and the scripted apiserver double. Shared fixtures are in `test_support::kubernetes`: an in-memory pod-store apiserver, a live sandbox's active execution row and pod, and the node-api simulator registered as a Kubernetes worker node. That module is the only place these tests construct a `Substrate` (`Substrate::from_kubernetes`). A seeded execution has published its route identity unless a test asks for the legacy, route-less shape; only the tests that hook the route publication do. The fixture's reflectors are seeded once, so pods the apiserver double creates or deletes later reach the substrate only through its uncached reads. The double creates pods already Running and never binds one to a node, and deletes them without a Terminating phase; like a real apiserver, it refuses with a 409 a delete whose `preconditions.uid` names another pod. `kubernetes::tests` gains `substrate_with_store_client_proxy_port_and_pods`, because suspend and observed-only delete need the node's node-api pod and a sandbox pod in the reflectors at once. Finding: the claim of `completed_actions_do_not_wait_for_a_key_share_lock` holds on Direct only. On Kubernetes, start and wake block behind a transaction holding `FOR KEY SHARE` on the sandbox row, which any foreign-key insert into a child table takes. They block on `SELECT lifecycle FROM sandboxes WHERE id = $1 FOR UPDATE`, run by `lock_sandbox_execution_lifecycle_in`; reading the code, that is the execution claim `ensure_pod_exists` takes before materializing the pod. Stop and kill of a sandbox whose pod is gone block too, earlier, on `SELECT id FROM sandboxes WHERE id = $1 FOR UPDATE` (`lock_sandbox_for_execution_mutation_in`), which, reading the code, the pod-tracking cleanup of the missing pod runs before the pod is recreated. Suspend blocks on its commit's `SELECT id ... FOR UPDATE`, as Direct suspend does on its artifact lock. Resume, pause, stop and kill of a sandbox whose pod is cached do not block. In the test's interleaving the holder ends promptly and each waiter then completes. A longer holder is ended by the `space_user` role's `statement_timeout`, which `bootstrap-roles` sets; the api sets no session timeout of its own. In the api, only start and wake are also bounded, by the activation wait budget around `ensure_pod_exists`. Neither the missing-pod stop and kill wait (in pod resolution) nor suspend's commit lock is under an api deadline. The twin `activations_and_suspend_wait_for_a_key_share_lock` pins this with waits tied to the holder's pid through `pg_blocking_pids` and to the waiting statement's `sql!` module, all bounded. The Kubernetes twin of `completed_actions_keep_a_terminal_lifecycle_that_landed_mid_flight` also lands the terminal lifecycle after the node call for start and suspend, which re-read the row and refuse one that landed before them. Start's lands once the node acknowledged its Wake, from a trigger on the legacy route publication. Suspend's lands with the substrate's own `suspended` finalization, from a trigger on that write. Both actions then reach the controller's terminal-state guard, which keeps the lifecycle. Also pinned: an untracked suspend of a `started` or `paused` Kubernetes sandbox writes its row `suspended`/`unplaced`, because Kubernetes suspend claims and finalizes the row whether or not the caller tracked it. Production code is unchanged. Four twins box their action future. Held inline in the test's future, the untracked-actions twin's overflows libtest's default 2 MiB stack when the api crate is built at opt-level 0, as Linux CI builds it; the others are boxed for margin. Not covered here: the two template builder-dispatch tests, because the Kubernetes builder client dials a fixed in-cluster service name, so a deterministic failure needs host DNS state or a production seam; and the cluster controller's `converge_cas_moves_only_restoring_rows`, which has no Kubernetes substrate fixture in that crate yet.
meefs
pushed a commit
to meefs/SPACE
that referenced
this pull request
Oct 17, 2026
Tests only. perplexityai#402 sorted the tests that build a DirectSubstrate and listed the ones whose shared-lifecycle outcome or path differs on Kubernetes (bucket 4). This adds a Kubernetes-backed twin for 21 of the 23 api tests in that bucket and keeps every Direct test. Each twin keeps its Direct twin's setup and assertions, and its doc says whether the Kubernetes outcome is the same and, where it differs, why. Where the Direct name's claim does not hold on Kubernetes (four twins), the twin is renamed and its doc names the Direct test. The twins live in a `kubernetes` submodule beside each Direct test file and use the `kubernetes::tests` substrate fixtures and the scripted apiserver double. Shared fixtures are in `test_support::kubernetes`: an in-memory pod-store apiserver, a live sandbox's active execution row and pod, and the node-api simulator registered as a Kubernetes worker node. That module is the only place these tests construct a `Substrate` (`Substrate::from_kubernetes`). A seeded execution has published its route identity unless a test asks for the legacy, route-less shape; only the tests that hook the route publication do. The fixture's reflectors are seeded once, so pods the apiserver double creates or deletes later reach the substrate only through its uncached reads. The double creates pods already Running and never binds one to a node, and deletes them without a Terminating phase; like a real apiserver, it refuses with a 409 a delete whose `preconditions.uid` names another pod. `kubernetes::tests` gains `substrate_with_store_client_proxy_port_and_pods`, because suspend and observed-only delete need the node's node-api pod and a sandbox pod in the reflectors at once. Finding: the claim of `completed_actions_do_not_wait_for_a_key_share_lock` holds on Direct only. On Kubernetes, start and wake block behind a transaction holding `FOR KEY SHARE` on the sandbox row, which any foreign-key insert into a child table takes. They block on `SELECT lifecycle FROM sandboxes WHERE id = $1 FOR UPDATE`, run by `lock_sandbox_execution_lifecycle_in`; reading the code, that is the execution claim `ensure_pod_exists` takes before materializing the pod. Stop and kill of a sandbox whose pod is gone block too, earlier, on `SELECT id FROM sandboxes WHERE id = $1 FOR UPDATE` (`lock_sandbox_for_execution_mutation_in`), which, reading the code, the pod-tracking cleanup of the missing pod runs before the pod is recreated. Suspend blocks on its commit's `SELECT id ... FOR UPDATE`, as Direct suspend does on its artifact lock. Resume, pause, stop and kill of a sandbox whose pod is cached do not block. In the test's interleaving the holder ends promptly and each waiter then completes. A longer holder is ended by the `space_user` role's `statement_timeout`, which `bootstrap-roles` sets; the api sets no session timeout of its own. In the api, only start and wake are also bounded, by the activation wait budget around `ensure_pod_exists`. Neither the missing-pod stop and kill wait (in pod resolution) nor suspend's commit lock is under an api deadline. The twin `activations_and_suspend_wait_for_a_key_share_lock` pins this with waits tied to the holder's pid through `pg_blocking_pids` and to the waiting statement's `sql!` module, all bounded. The Kubernetes twin of `completed_actions_keep_a_terminal_lifecycle_that_landed_mid_flight` also lands the terminal lifecycle after the node call for start and suspend, which re-read the row and refuse one that landed before them. Start's lands once the node acknowledged its Wake, from a trigger on the legacy route publication. Suspend's lands with the substrate's own `suspended` finalization, from a trigger on that write. Both actions then reach the controller's terminal-state guard, which keeps the lifecycle. Also pinned: an untracked suspend of a `started` or `paused` Kubernetes sandbox writes its row `suspended`/`unplaced`, because Kubernetes suspend claims and finalizes the row whether or not the caller tracked it. Production code is unchanged. Four twins box their action future. Held inline in the test's future, the untracked-actions twin's overflows libtest's default 2 MiB stack when the api crate is built at opt-level 0, as Linux CI builds it; the others are boxed for margin. Not covered here: the two template builder-dispatch tests, because the Kubernetes builder client dials a fixed in-cluster service name, so a deterministic failure needs host DNS state or a production seam; and the cluster controller's `converge_cas_moves_only_restoring_rows`, which has no Kubernetes substrate fixture in that crate yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What merging this does
This PR is tests only. It sorts every test function in the 17 files that construct
DirectSubstrateby what the test actually depends on. The aim is that none of these tests quietly relies on Direct for behavior that differs on Kubernetes, the substrate that production and Kind run.stale_heartbeat_direct_worker_is_not_empty_clusterpins Direct's own inventory SQL: a registered worker with a stale heartbeat still counts as eligible. It moves byte-identical into a new#[cfg(test)]module,crates/libs/api/src/controllers/substrate/direct/cluster_admission_tests.rs.managed_bybound into a query with no Kubernetes-specific term).completed_actions_do_not_wait_for_a_key_share_lock, carries a possible finding: its invariant may not hold on Kubernetes (see its row).No production behavior, schema, wire format, metric or configuration changes. Outside test files, the diff adds only a
#[cfg(test)] moddeclaration indirect.rsandpub(crate)on the existing#[cfg(test)] mod testsincluster_admission/mod.rs. The test count is unchanged: api 1730 → 1730, cluster-controller 955 → 955.Out of scope. About 16 more test files get Direct implicitly through
AppState::from_pool/SandboxController::from_env, becauseSubstrate::from_envfalls back to Direct when Kubernetes isn't configured or thekubernetesfeature is off. They are not traced or changed here.Moved test
cluster_admission::tests::stale_heartbeat_direct_worker_is_not_empty_cluster→controllers::substrate::direct::cluster_admission_tests::stale_heartbeat_direct_worker_is_not_empty_cluster.git show 38b61d15:crates/libs/api/src/cluster_admission/tests.rsand from the new file at this head. Both copies are 42 lines with sha256bd325aee4c363f10…: identical. The body needed no path or visibility qualifier.cluster_admission/mod.rs:#[cfg(test)] mod tests→pub(crate) mod tests.cluster_admission/tests.rs:pub(crate)ondirect_substrateandenabled_config.direct.rs:#[cfg(test)] mod cluster_admission_tests;.cluster_admission::tests::admissiontakesOption<RedisTokenBucket>, a type private tocluster_admission. The new module therefore defines a three-lineadmissionwith the same call shape. It builds the same bucketless admission throughClusterSandboxAdmission::with_config:build_bucket(None)isNone, the config is sanitized the same way, and feature flags are disabled.direct.rsDirectSubstrate::eligible_start_node_countto also requireCOALESCE(nss.stale_after > NOW(), TRUE). The moved test fails at its new location withEmptyCluster. The file was then restored byte-for-byte (cmp) and touched.Decisions made unattended
node_direct) as setup leaves a test in bucket 4 when the behavior under test is shared. Only the stale-heartbeat test pins Direct's own SQL.direct/tests.rs. It keeps theadmissionstand-in scoped to the one test that needs it.KubernetesSubstrate::eligible_start_node_countgives the same answer for a synced node cache. An unsynced cache returnsNone, which fails open, so a later Kubernetes version needs a synced fixture;substrate_with_storeprovides one.capacity_probe_failure_fails_openis the exception, and is bucket 1: only Direct's count can fail.lifecycle/tests.rs::direct_start_restore_spends_deferred_admission_at_entryandfork_operation_executor/tests.rs::direct_fork_copy_fallback_requires_a_fresh_ready_node_statusare bucket 1, which means no change. Both are candidates for the Direct suite later.runtime_rpc/tests.rsstays bucket 1 (the substrate layer'sRuntimeClient::call), even though Direct only fills its&Substrateargument. No comment was added there.lifecycle_writer_tests.rs::direct_sink,cluster_admission/tests.rs::direct_substrateandpublic_sandbox_operations/tests.rs::direct_sandbox_controller.Overlaps with open PRs
queries/create_postgres_tests.rsandsubstrate/direct.rs, but no hunks overlap.For every PR listed, I ran
git apply --checkon that PR's own diff to the shared files (merge-base..head). It applies cleanly both onmain(38b61d15) and on this head.Validation
All checks ran at head
bcd77b446980on Linux. The nextest setup scripts started Postgres and Redis, withSPACE_CI_REQUIRE_TEST_DEPS=1.SPACE_TEST_DEP_SKIP_LOGwas empty for every run.cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features --locked -- -D warningsscripts/check_no_inline_rust_test_modules.pycrates/libs/observability, identical at basescripts/source_guards/*.pythin_service_mains.pyreports pre-existing findings incrates/apps/*/main.rs, identical at basescripts/check-rust-error-policy.shcargo nextest run --locked -p pplx-space-api --all-features --retries 0, 3 consecutive runs#[ignore]benchmark skipped, as at base)cargo nextest run --locked -p pplx-space-cluster-controller-server --all-features --retries 0, 3 consecutive runscargo nextest list, base38b61d15→ head)bd325aee4c363f10…before and after)Classification of every test in the 17 files
Buckets:
Kubernetes fixture names:
substrate_with_store,substrate_with_store_and_clientandsubstrate_with_store_client_proxy_port_and_podare thepub(crate)constructors incontrollers/substrate/kubernetes/tests.rs.scripted_kube_clientis inkubernetes/scripted_test_doubles.rs, which api, cluster-controller: reach Kubernetes-only state through one accessor #386 makespub(crate).api/controllers/substrate/direct/tests.rs(78 tests)direct.rsfunctions orDirectSubstratemethods (placement, restore leases, receipts, route identities)deleting_snapshot_blocks_artifact_without_non_orphaned_ready_copypersist_optional_current_local_artifact; the Direct fixture is unusedlive_cpu_update_preserves_allocation_across_cold_restartSandboxController::updatevCPU invariant through Direct's restart; on Kubernetes the restart is a pod recreate. Fixture:substrate_with_store_client_proxy_port_and_podplus a sim or scripted workerapi/controllers/substrate/runtime_rpc/tests.rs(2 tests)peer_errors_are_returned_without_replaying_the_requestRuntimeClient::call); Direct only fills the&Substrateargument and is never reacheddeadline_bounds_the_rpc_and_prevents_expired_requestsRuntimeClient::calldeadline); Direct is never reachedapi/fork_operation_executor/tests.rs(37 tests)direct_fork_copy_fallback_requires_a_fresh_ready_node_statusDirectSubstrate::select_ready_worker_node_api_target36 tests in bucket 0 (never construct or reach Direct)
executor_invariants_are_not_reported_as_invalid_client_requests,invalid_capture_leader_state_is_terminal,create_request_applies_requested_fork_timeout,create_request_applies_timeout_to_disabled_source_lifecycle_policy,persisted_sandbox_result_contains_only_minting_inputs,terminal_cleanup_uses_the_preallocated_target,fork_metrics_follow_the_persisted_workflow_phase,fork_retry_metrics_distinguish_transient_failures_from_workflow_progress,fork_response_preserves_untagged_source_template_id,frozen_source_template_uses_the_resolved_content_hash,target_config_without_source_template_id_uses_the_normalized_template_reference,sandbox_error_uses_typed_e2b_shape,restore_activation_lease_preserves_the_existing_floor,deadline_with_a_failed_target_is_not_reported_as_success,failed_targets_are_reported_as_a_failed_fork_outcome,executor_retry_uses_queued_state,executor_wait_preserves_retry_budget,suspended_source_selects_the_durable_artifact_path,snapshot_errors_have_explicit_workflow_dispositions,live_source_requires_a_positive_execution_generation,byok_organizations_are_rejected_before_capture,retryable_restore_controller_error_requeues_the_operation,unrelated_restore_conflicts_are_terminal,concurrent_target_creation_is_reobserved,suspended_target_replays_as_success_after_restore,command_pending_target_cannot_replay_as_success,external_work_has_an_overall_deadline,cleanup_failure_does_not_overturn_a_successful_fork,cold_copy_destination_uses_snapshot_identity_and_preserves_connection_query,cold_copy_database_identities_are_stable_per_operation,copied_snapshot_is_initially_owned_by_the_existing_source_sandbox,terminal_cold_copy_cleanup_recovers_ids_before_payload_publication,terminal_cleanup_retires_a_ready_copied_artifact_consistently,node_operation_identity_uses_the_unprefixed_uuid,kubernetes_copy_target_requires_a_substrate_node_id,direct_copy_target_accepts_a_direct_endpointapi/controllers/sandbox/lifecycle/characterization_tests.rs(16 tests)wake_refuses_teardown_and_terminal_lifecycles_before_admissionsuspended/restoringpass-through arms end in Direct'swake_execution; Kubernetes enters the restore-lease claim. Fixture:kubernetes::tests::substrate_with_storenon_lifecycle_actions_are_bad_requests_and_write_nothingdrain_gated_actions_conflict_and_leave_the_rowensure_resolved_node_not_draining)unreachable_runtime_fails_actions_without_writing_lifecyclesubstrate_with_store_and_clientwith ascripted_test_doubles::scripted_kube_clientthat fails pod createswake_completion_outcome_maps_to_the_wake_responsesubstrate_with_store_client_proxy_port_and_podplus a sim or scripted workercompleted_actions_keep_a_terminal_lifecycle_that_landed_mid_flightsubstrate_with_store_client_proxy_port_and_podplus a sim or scripted workernative_start_overwrites_a_stopped_written_mid_flight_without_a_retrymaybe_update_sandbox_state(.., "started", "on_node")(lifecycle.rs), which a Kubernetes non-restore start also reaches; the trigger on Direct'ssandbox_direct_route_identitiesis only the injection hook. Fixture:substrate_with_store_client_proxy_port_and_podplus a sim or scripted worker, with a hook on a Kubernetes-path writecompleted_actions_do_not_wait_for_a_key_share_lockpersist_optional_current_local_artifact, called only fromdirect.rs) is Direct-specific. Possible finding: a Kubernetes start's pod materialization takesSELECT lifecycle FROM sandboxes … FOR UPDATE(kubernetes/store/execution_rows.rs, viastore/pod_placement.rs), which conflicts withFOR KEY SHARE, so the invariant in the test's name may not hold on Kubernetes; a Kubernetes version should report that, not paper over it. Fixture:substrate_with_store_client_proxy_port_and_podplus a sim or scripted workerstop_and_kill_on_a_draining_node_still_reach_the_workerreachable_runtime_client == None). Fixture:substrate_with_store_client_proxy_port_and_podplus a sim or scripted worker on a draining nodeuntracked_start_and_restore_validate_then_charge_admissionSome(0)too, so admission rejects the same wayuntracked_runtime_actions_without_placement_are_unavailablesubstrate_with_store_and_clientwith ascripted_test_doubles::scripted_kube_clientuntracked_actions_report_status_and_leave_the_rownode_directendpoint; Kubernetes uses the pod IP. Fixture:substrate_with_store_client_proxy_port_and_podplus a sim or scripted workerobserved_only_delete_without_an_observation_is_not_founddelete_observed_executionobserved_only_delete_removes_the_resident_on_the_observing_nodedelete_observed_executionfirst claims a reserved warm-pool slot, else resolves the node from its node cache. Fixture:kubernetes::tests::substrate_with_storewith a node cache entryredrive_completes_only_a_restoring_rowsubstrate_with_store_and_clientwith ascripted_test_doubles::scripted_kube_client1 tests in bucket 0 (never construct or reach Direct)
failed_delete_compensation_refuses_a_row_that_left_deletingapi/cluster_admission/tests.rs(25 tests)admission_enabled_applies_the_env_master_first_policycapacity_probe_failure_fails_openErrfail-open branch of admission is reachable only through Direct: Kubernetes' count cannot fail (Ok(substrate.eligible_start_node_count()))disabled_admission_admits_without_touching_storeempty_cluster_rejects_activationSome(0)too, so admission rejects the same wayfails_open_when_no_redis_configuredstale_heartbeat_direct_worker_is_not_empty_clusterDirectSubstrate::eligible_start_node_count)exhausted_bucket_rate_limits_activationwith_config_builds_bucket_from_config_redis_url17 tests in bucket 0 (never construct or reach Direct)
sanitize_config_falls_back_to_absolute_cluster_defaults,sanitize_config_clamps_sub_one_burst_up_to_one,should_reject_empty_only_on_confident_zero,retry_after_hint_spreads_over_every_whole_second_from_one_to_five,widest_jitter_offset_lands_on_the_gateway_retry_after,earliest_refill_collapses_to_the_header_floor_at_or_above_one_token_per_second,earliest_refill_is_a_real_wait_below_one_token_per_second,the_cap_is_the_only_reason_a_caller_returns_before_its_token,a_subnormal_refill_rate_is_capped_rather_than_panicking,a_shed_caller_is_never_told_to_return_before_a_token_can_exist,a_rate_limited_hint_is_paced_by_the_configured_refill_rate,an_unusable_refill_rate_is_capped_without_flattening_the_jitter,every_hint_in_the_window_survives_header_rendering_unchanged,unreachable_redis_consume_errors,redis_consume_records_client_span_on_error,shared_bucket_allows_burst_then_rejects,concurrent_consumers_cannot_overspend_shared_bucketapi/controllers/sandbox/updates/transport_tests.rs(5 tests)untransmitted_update_is_replayed_once_on_a_fresh_clientsend_access_idle_deadlineandRuntimeClient::call's fresh-client replay;Fixture::move_workerrewritingnode_directis setup. Fixture: a Kubernetes substrate whose pod IP can be repointed between calls (substrate_with_store_client_proxy_port_and_podplus a sim worker)interrupted_update_is_observed_not_resentsend_access_idle_deadline/observe_access_idle_update; repointing the worker throughnode_directis setup. Fixture: same as above3 tests in bucket 0 (never construct or reach Direct)
deadline_send_failures_split_by_whether_the_worker_saw_the_request,unconfirmed_deadline_update_is_a_paced_retryable_unavailable,only_an_observed_extension_floor_proves_an_ambiguous_updateapi/controllers/sandbox/lifecycle/tests.rs(17 tests)rejected_activation_emits_no_orphaned_requested_eventSome(0)too, so admission rejects the same waydirect_start_restore_spends_deferred_admission_at_entrydirect.rsstart entry); Kubernetes charges in the restore-lease claim winnerinvalid_activation_options_reject_before_admission_and_requested_eventSome(0)too, so admission rejects the same wayexhausting_the_finalize_retry_budget_stays_retryabledelete_execution(Unavailable, swallowed); Kubernetesteardown_executionwritesforget_pod_tracking. The 503 should still hold. Fixture:kubernetes::tests::substrate_with_storea_contended_delete_precheck_stays_retryabledelete_execution; Kubernetes tears down with writes. Fixture:kubernetes::tests::substrate_with_storeresume_uses_resolved_runtime_without_async_observationnode_direct; Kubernetes resolves the pod IP from the active execution and pod cache. Fixture:substrate_with_store_client_proxy_port_and_podplus a sim or scripted worker11 tests in bucket 0 (never construct or reach Direct)
cluster_delete_miss_reason_classifies_by_substrate_error,protected_node_delete_defers_control_plane_delete,finalize_sandbox_db_delete_removes_live_sandbox,delete_fork_guard_persists_delete_intent_before_runtime_teardown,deferred_delete_restores_the_previous_lifecycle,finalize_failure_restores_the_previous_lifecycle,finalize_failure_accepts_an_already_deleted_row,delete_fork_guard_allows_observed_only_cleanup_without_a_sandbox_row,persisted_delete_intent_denies_later_fork_admission,finalize_delete_retries_bounded_lock_waits,delete_guard_contention_is_retryable_not_internalapi/controllers/sandbox/ingress/tests.rs(7 tests)stored_ingress_access_resumes_paused_execution_before_reporting_readywake_executionfences the resume with a seeded Direct route identity; Kubernetes resumes through the pod IP. Fixture:substrate_with_store_client_proxy_port_and_podplus a sim or scripted worker6 tests in bucket 0 (never construct or reach Direct)
accepted_activation_outlives_timeout_and_disconnect_and_drains,restore_followers_only_observe_completion_or_expiry,wake_disabled_refuses_dormant_but_allows_running_access,losing_restore_claim_observes_without_retrying_activation,healthy_ingress_bypasses_exhausted_activation_capacity,lifecycle_change_during_probe_enters_the_cancellation_independent_ownerapi/controllers/sandbox/spaced/tests.rs(10 tests)reconnect_node_api_is_blocked_by_access_drain_gateaccess_runtime_client9 tests in bucket 0 (never construct or reach Direct)
process_stream_open_timeout_error_is_request_timeout_without_retry_marker,runtime_operation_timeout_error_is_request_timeout_without_retry_marker,stream_open_retry_accepts_node_api_drain_status,stream_open_retry_accepts_shutdown_transport_status,stream_open_retry_accepts_unavailable_transport_close_status,stream_open_retry_accepts_internal_transport_close_status,stream_open_retry_rejects_plain_unavailable_status,stream_open_retry_rejects_plain_unknown_status,stream_open_retry_rejects_application_statusapi/controllers/sandbox/updates/postgres_tests.rs(4 tests)update_persists_metadata_and_reads_it_backupdate_with_no_fields_leaves_the_row_aloneupdate_metadata_bumps_updated_at_only_on_a_real_changeupdate_metadata_replaces_the_whole_documentapi/controllers/sandbox/queries/create_postgres_tests.rs(34 tests)postgres_cold_federated_create_is_atomic_with_grant_upsertmanaged_by; Cold mode is pool-ineligible on both and the grant trigger rolls back either waypostgres_federated_create_denies_api_key_org_specific_templatepostgres_federated_create_allows_mapped_org_specific_templatepostgres_failed_pooled_create_finalizes_after_claim_cleanupnode_directlookup and the row is deleted; Kubernetescleanup_failed_pooled_preparemarks the slot cleanup-pending and keeps the row. Fixture:kubernetes::tests::substrate_with_storewith a claimed slotpostgres_federated_restore_inserts_restore_ownershipmanaged_byonly; the trigger raises beforecreate_executionunconfigured_falcon_create_continues_to_database_accessmaterialize_ownerbefore the substrate28 tests in bucket 0 (never construct or reach Direct)
create_finalization_accepts_receipt_converged_started_state,create_finalization_accepts_operation_leased_restoring_state,restoring_convergence_clears_fork_only_artifact_retention,restore_finalization_requires_the_receipt_converged_node_binding,create_finalization_records_operation_leased_restore_failure,postgres_insert_pooled_sandbox_writes_api_row,postgres_insert_pooled_sandbox_can_write_federated_owner_row,postgres_pooled_federated_sandbox_persists_service_grant,postgres_pool_configured_resources_uses_enabled_release_shape,postgres_template_build_shape_prefers_the_builder_inventory,postgres_kubernetes_create_fills_unset_resources_from_the_template_shape,postgres_pool_miss_can_retain_rolling_policy_shape_before_materialization,postgres_pool_miss_retains_policy_shape_without_legacy_opt_in,postgres_pool_miss_retains_shape_for_release_outside_warming_window,postgres_pool_miss_uses_same_policy_priority_as_zero_demand_materialization,postgres_pooled_create_retries_prepare_in_place_before_reclaiming,postgres_terminal_pooled_prepare_failure_transfers_claim_to_cleanup,postgres_byok_org_bypasses_warm_pool_and_cold_creates,postgres_pinned_sandbox_id_bypasses_warm_pool_and_cold_creates,postgres_admitted_pool_claim_adopts_the_claimed_slot_under_its_id,postgres_admitted_pool_claim_released_slot_builds_cold_without_pool_calls,postgres_admitted_pool_claim_waits_while_a_slot_holds_the_id,postgres_admitted_pool_claim_releases_a_slot_it_may_not_adopt,postgres_admitted_pool_claim_prepare_failure_never_reclaims,postgres_pooled_create_reclaims_after_prepare_retry_budget_exhaustion,postgres_pooled_sandbox_network_policy_override_enqueues_reconcile,postgres_strict_pool_placement_refusal_is_a_conflict_not_unavailable,admission_claim_executes_and_runs_under_the_claimed_idapi/public_sandbox_operations/tests.rs(20 tests)unconfirmed_creating_row_retries_through_executorexisting_row_create_persists_the_renewed_grantdeleting_row_retries_through_executorexisting_rows_are_classified_exhaustively_through_executorcreate_executor_retries_unpinned_transient_substrate_failurecreate_executionfinds no worker (Unavailable); Kubernetes goes through the network-slot preflight and an apiserver pod create. Fixture:substrate_with_store_and_clientwith ascripted_test_doubles::scripted_kube_clientanswering pod creates with 503admitted_create_re_execution_after_success_does_not_create_againsandbox_k8s_executionshas 0 rows is vacuous on Direct today (only Kubernetes code writes that table), and Direct'sadmitted_pool_cleanup_pendingis a constantfalse. Fixture:kubernetes::tests::substrate_with_storewith a claimed warm-pool slot14 tests in bucket 0 (never construct or reach Direct)
native_create_executor_persists_exemptions_without_changing_egress,generic_429_is_not_marked_admission_backpressure,cluster_admission_429_is_marked_backpressure,controller_create_request_uses_public_contract_defaults,controller_create_request_threads_node_pool,create_operation_payload_is_wire_compatible_with_old_shape,controller_update_request_uses_public_contract_fields,lifecycle_projection_hides_internal_started_label,completed_lifecycle_target_state_accepts_idempotent_targets,firecracker_snapshot_restore_incompatibility_is_terminal,create_precondition_is_terminal_with_controller_evidence,completed_lifecycle_target_state_rejects_active_transitions,create_operation_payload_pool_claim_is_optional_on_the_wire,pool_slot_release_wait_is_workflow_progressapi/controllers/template/tests.rs(17 tests)build_submission_grants_the_base_before_dispatchsubstrate_with_storedials a fixed service name)accepted_build_pins_only_its_own_base_grantaccepted_build_without_builder_identity_records_unpinned_routingaccepted_build_with_builder_identity_records_routingaccepted_e2b_build_records_client_build_mappingaccepted_build_with_invalid_builder_identity_is_rejectedsystem_build_withdraws_its_publication_grant_when_dispatch_fails10 tests in bucket 0 (never construct or reach Direct)
build_admission_accepts_owner_digest,build_admission_reports_database_contention_as_retryable,build_admission_authorizes_and_pins_the_base_release,template_build_submission_retries_a_draining_builder,template_build_submission_does_not_retry_other_unavailable_errors,template_build_submission_maps_to_builder_request,template_build_status_timeout_is_transient_rpc,template_build_status_transport_error_is_transient_transport,template_build_status_contention_is_transient_contention,template_build_status_not_found_is_not_transientapi/controllers/substrate/kubernetes/tests.rs(209 tests)stale_start_snapshot_cannot_bypass_admission_after_lifecycle_refreshSome(0)too, so admission rejects the same waystart_restore_takeover_charges_admissionSome(0)too, so admission rejects the same wayjoining_follower_never_charges_admissionSome(0)too, so admission rejects the same waylease_takeover_charges_admission_and_releases_claim_on_rejectionSome(0)too, so admission rejects the same wayrejected_fresh_restore_claim_reverts_to_suspendedSome(0)too, so admission rejects the same wayrecovery_follower_does_not_charge_cluster_admissionSome(0)too, so admission rejects the same waysuspending_access_defers_before_charging_cluster_admissionSome(0)too, so admission rejects the same wayrejected_admission_blocks_pod_recreation_for_started_sandbox_with_lost_podSome(0)too, so admission rejects the same wayno_activity_spaced_path_charges_recovery_for_lost_podSome(0)too, so admission rejects the same waycontrol_runtime_client_propagates_admission_rejection_without_recreating_podSome(0)too, so admission rejects the same way199 tests in bucket 0 (never construct or reach Direct)
wake_recovery_lifecycle_excludes_stop_and_delete_states,suspending_wake_recovery_preserves_retryable_suspend_ownership,paused_restore_does_not_join_started_recovery,shortlist_from_candidates_puts_source_first_dedupes_and_is_deterministic,scheduling_outcome_classifies_scheduler_decision,required_source_pin_detects_only_the_required_matchfields_pin,network_slot_preflight_admits_matching_node_with_free_capacity,network_slot_preflight_allows_unknown_empty_cache,network_slot_preflight_rejects_when_matching_nodes_are_full,network_slot_preflight_ignores_terminal_pods,network_slot_preflight_reports_unknown_without_matching_schedulable_capacity,started_recovery_local_source_must_pass_direct_placement_gates,sandbox_start_candidate_selector_requires_extra_labels,sandbox_start_candidate_filter_rejects_unschedulable_and_not_ready_but_counts_tainted_nodes,split_operation_producer_does_not_call_uncapable_node_api,suspend_handoff_accepts_projection_before_pod_retirement,suspend_handoff_accepts_projection_between_lookup_and_commit,resident_restore_joins_shim_after_missing_context_wake,split_operation_waits_for_transient_receipt_capability,split_operation_capability_deadline_is_a_clean_no_send_failure,legacy_paused_restore_preserves_handoff_without_route_identity,paused_restore_requires_existing_route_acknowledgement,only_a_proven_no_send_wake_failure_may_release_the_operation,dormant_access_preserves_operation_after_dispatched_wake_failure,started_restore_does_not_use_command_readiness_as_identity_evidence,route_ack_join_uses_phase_deadline_without_clean_rollback,substrate_enum_forwards_observed_delete_to_kubernetes_override,process_start_requires_sandbox_connect_to_create_k8s_execution,concurrent_started_wakes_with_existing_execution_wait_for_one_pod_create,concurrent_started_wakes_without_active_execution_wait_for_one_pod_create,concurrent_started_access_returns_when_leader_lacks_restore_context,started_access_redrives_when_suspend_finalization_retires_execution,reconnect_waits_for_local_suspend_backup,started_recovery_replaces_authoritative_terminal_pod_with_local_restore,started_recovery_restores_locally_after_reflector_cache_miss,started_recovery_stale_pod_preserves_explicit_local_restore_plan,non_owner_ensure_does_not_duplicate_started_recovery_pod_create,stale_suspended_wake_reuses_owner_completion_without_pod_create,stale_restoring_wake_reuses_owner_completion_without_pod_create,stale_restoring_wake_retries_after_owner_rollback,restore_claim_reuses_owner_completion_without_pod_create,restore_claim_preserves_conflict_when_started_is_unroutable,wake_losing_restore_lease_joins_owner_binding_without_pod_create,user_restore_losing_lease_joins_owner_completion,published_activation_releases_only_its_completed_lease,start_failure_before_wake_releases_its_activation_lease,start_losing_restore_lease_reports_joined_completion,wake_follower_respects_readiness_budget_while_restore_remains_in_flight,wake_follower_defers_if_suspend_claims_during_restore_join,fence_blocked_conflict_redrive_respects_readiness_budget,follower_losing_expiry_reclaim_keeps_original_budget_and_new_lease_hint,wake_follower_reclaims_expired_restore_with_remaining_readiness_budget,stale_restore_owner_cannot_materialize_after_lease_reclaim,failed_restore_pod_create_without_pod_retires_execution_before_rollback,disconnected_restore_leader_finishes_node_work_and_cleanup,joined_restore_does_not_resurrect_concurrent_delete,concurrent_wakes_for_one_sandbox_hold_one_claim_connection,owned_restore_does_not_resurrect_concurrent_delete,expired_recovery_is_reclaimed_once_by_runtime_access,expired_recovery_reclaimed_by_runtime_access_publishes_completion,ambiguous_pod_create_keeps_started_recovery_pending,pod_create_conflict_with_failed_get_keeps_started_recovery_pending,concurrent_started_wake_returns_when_owner_publishes_running,concurrent_started_wake_stops_waiting_after_readiness_budget,concurrent_started_wake_stops_joining_when_owner_lease_expires,process_start_does_not_replace_stale_k8s_execution,dedupe_inventory_prefers_ready_over_not_ready,dedupe_inventory_breaks_tie_by_newer_created_at,inventory_skips_kubernetes_node_with_invalid_internal_ip,inventory_listing_does_not_downgrade_ready_node_on_unready_pod,every_pod_build_path_renders_the_node_group_boundary,stale_existing_pod_reason_treats_suspended_sandbox_as_stale,stopped_restart_replaces_only_execution_retired_by_current_stop,execution_pod_lookup_in_flight_includes_started_recovery,missing_recovering_execution_is_left_for_recovery_owner,recent_succeeded_pod_without_suspend_ownership_is_cleaned_up,recent_succeeded_pod_with_suspend_ownership_preserves_publication,started_recovery_without_prior_state_materializes_on_demand,unavailable_started_recovery_source_preserves_claim_across_retries,missing_pod_observation_preserves_failed_started_recovery_claim_across_retries,started_recovery_reports_pending_restore_source_without_naming_suspend_finalization,missing_started_recovery_remote_restore_replaces_generation,started_recovery_created_execution_keeps_generation_for_local_restore,stale_restore_execution_create_requires_old_restoring_claim,ensure_pod_exists_replaces_stale_first_create_claim,ensure_pod_exists_refreshes_stale_started_before_pod_create,started_recovery_materialization_does_not_recreate_after_teardown,suspending_materialization_does_not_consume_successor_generation,ordinary_restore_materialization_does_not_recreate_after_teardown,restore_create_defers_when_pod_reflector_cache_is_cold,materialization_inherits_policy_committed_before_execution_claim,pod_materialization_claim_rejects_teardown_lifecycle,teardown_revokes_slow_pod_materialization,pod_is_deleted_when_post_create_persistence_fails,unconfirmed_delete_after_metadata_persist_failure_keeps_recovery_pending,restore_shortlist_drops_nodes_with_not_ready_node_services,restore_shortlist_keeps_not_ready_nodes_when_the_bias_is_off,a_local_restore_from_an_out_of_set_source_is_refused_before_any_pod,restore_shortlist_keeps_in_set_nodes_when_the_group_reads_degraded,restore_shortlist_drops_fallbacks_outside_the_candidate_set,restore_shortlist_ignores_the_signal_when_most_of_the_fleet_reads_degraded,eligible_start_node_count_reads_live_node_cache,ensure_pod_exists_preserves_observed_invalid_stateful_generation_create_claim,observed_create_claim_replaced_before_return_is_retryable_not_bad_gateway,failed_create_rejects_stale_pod_loaded_by_recovery_get,ensure_pod_exists_recovers_started_after_final_replaced_conflict,mock_api_repro_stale_create_replacement_restores_from_source_snapshot,expired_restore_redrive_with_vanished_source_uses_remote_snapshot_without_hard_pin,on_demand_pod_create_allows_initial_generation,on_demand_pod_create_rejects_stopped_restart_after_initial_generation,stopped_pod_create_preserves_node_local_restore_intent,stopped_pod_create_prefers_the_source_node_once_the_disk_artifact_is_durable,on_demand_pod_create_rejects_live_recovery_after_initial_generation,on_demand_pod_create_rejects_transient_lifecycle_after_initial_generation,pod_create_context_preserves_restore_intent_after_initial_generation,pod_create_context_allows_storage_only_restore_after_initial_generation,stale_existing_pod_reason_keeps_in_progress_restore_pod_for_suspended_sandbox,stale_existing_pod_reason_replaces_pending_local_restore_when_s3_becomes_ready,stale_existing_pod_reason_treats_terminal_pod_as_stale,delete_skips_runtime_cleanup_for_unschedulable_pod,delete_propagates_genuine_infra_errors,retire_teardown_never_sends_sandbox_delete,retire_teardown_requires_stop_success,retryable_sbxmgr_delete_refusal_defers_kubernetes_teardown,retire_skips_backup_for_missing_source_or_terminal_refusal,unschedulable_message_extracts_scheduler_reason,cancel_guard_defused_is_noop,cancel_guard_unarmed_drop_is_noop,create_ending_early_gives_back_exactly_what_it_claimed,is_nodemgr_unreachable_matches_transport_signatures,is_nodemgr_unreachable_matches_timeout_after_unavailable_probe,phase_only_timeout_messages_do_not_name_nodemgr,ready_wait_bounds_phase_deadline_fires_before_the_outer_backstop,caller_ready_budget_is_not_shortened_by_phase_margin,warm_pool_setup_rpc_timeout_stays_below_lifecycle_wait_budget,command_ready_timeout_error_carries_typed_phase_and_phase_label,is_nodemgr_unreachable_does_not_match_other_variants,is_nodemgr_unreachable_does_not_match_unrelated_connect_substring,terminal_pod_phase_matches_only_succeeded_and_failed,claiming_a_warm_slot_consults_the_candidate_set,a_dedicated_only_sandbox_reports_a_refusal_not_a_bare_miss,an_unlabeled_node_releases_the_slot_instead_of_draining_the_pool,an_unprovisioned_cluster_refuses_a_slot_on_a_labeled_node,an_unprovisioned_cluster_still_adopts_warm_slots,an_unresolvable_candidate_set_never_errors_out_of_the_claim_step,an_org_specific_pool_config_drains_instead_of_recycling,an_unusable_warm_pod_is_judged_before_its_node,a_warm_pod_outside_the_candidate_set_is_not_adopted,pooled_claim_pod_is_usable_requires_present_matching_pod,pooled_adoption_request_persists_cluster_route_without_resuming,terminal_selected_pod_requires_current_suspend_evidence,selected_wake_uses_authoritative_pod_uid_and_ip,wake_action_preserves_fence_intent_and_separates_split_receipts,wake_ack_requires_the_complete_expected_route_identity,route_publication_classification_uses_materialization_evidence,parse_create_command_ready_timeout_applies_default_for_missing_or_invalid,parse_create_command_ready_timeout_honors_explicit_value,parse_create_data_plane_ready_timeout_defaults_to_120s,parse_create_data_plane_ready_timeout_honors_explicit_value,restore_cleanup_requires_clean_error_and_proof_wake_was_not_dispatched,missing_runtime_context_wake_requires_identity_ack_retry,command_ready_poll_backoff_preserves_fast_window_then_grows_and_caps,command_ready_poll_backoff_clamps_delays_to_deadline,fenced_retire_ignores_a_runtime_the_fenced_pod_does_not_have,fenced_retire_collects_prior_generation_pods,fenced_retire_of_an_already_absent_pod_reports_retired,a_name_only_fenced_retire_deletes_no_pod,a_fenced_stop_never_rides_a_stale_cache,a_name_only_refusal_records_the_uid_it_could_not_verify,a_no_execution_fence_refuses_an_execution_that_appeared,a_no_execution_fence_never_clears_a_row_it_did_not_see,the_uid_backfill_touches_nothing_it_should_not,a_no_execution_fence_retires_when_the_absence_holds,a_failed_orphan_delete_keeps_the_row_that_names_it,a_no_execution_fence_collects_orphans_on_inactive_rows,a_name_only_retire_defers_to_a_pod_that_materialized,fenced_retire_spares_a_same_generation_replacement,fenced_retire_refuses_a_replacement_activation_attempt,fenced_retire_refuses_an_attempt_that_appeared_where_none_was_observed,a_name_only_retire_releases_a_row_that_still_records_a_uid,reachable_runtime_probe_leaves_missing_and_terminal_pods_for_recovery,fenced_retire_deletes_a_terminal_pod_only_under_its_precondition,fenced_retire_refuses_when_a_successor_lands_mid_teardown,the_caller_side_probe_does_not_trust_the_reflector,fenced_retire_sends_the_identity_precondition_with_the_delete,fenced_retire_refuses_a_replaced_pod_and_keeps_the_execution,fenced_retire_refuses_when_a_newer_execution_took_over,every_field_of_the_identity_is_part_of_the_comparison,absence_is_a_value_except_where_it_is_unobservable,a_retire_with_nothing_left_to_retire_succeeded,a_terminal_pod_behind_a_name_only_fence_is_retired,a_terminal_pod_is_identified_by_the_row_that_records_it,a_rejected_delete_precondition_is_resolved_by_reading_the_pod,one_place_answers_whether_a_caller_may_undo,only_the_same_pod_moving_proves_the_stop_reached_what_is_live,a_completed_replacement_activation_is_a_different_owner,a_re_claimed_row_is_caught_even_with_no_pod_left_to_compare,a_fenced_delete_always_preconditions_on_the_uid,fenced_pod_delete_maps_404_benign_and_409_to_precondition_failurecluster-controller/public_operation_dispatcher/tests.rs(18 tests)native_executor_requeues_create_when_cluster_admission_rejectsSome(0)too, so admission rejects the same way;managed_by = 'k8s'adds pool-config reads that find nothingnative_executor_persists_exemptions_on_policy_update_and_template_reset16 tests in bucket 0 (never construct or reach Direct)
config_defaults_enabled,tick_defaults_to_25ms_and_the_test_override_takes_whole_seconds,m1_max_age_env_defaults_and_zero_disables,execution_concurrency_parses_and_clamps_from_the_environment,tick_report_merge_sums_outcomes_without_claim_double_counting,native_executor_claims_create_sandbox,tick_fails_operation_after_retry_budget,tick_does_not_exhaust_backpressure_retries,delete_execution_renews_claim_while_a_phase_is_blocked,run_slow_execution_does_not_block_newly_queued_operation,run_never_claims_beyond_execution_concurrency,run_shutdown_drains_in_flight_executions,tick_does_not_exhaust_workflow_progress_requeues,expired_native_m1_requeues_fail_with_evidence,backdated_fork_and_admin_operations_remain_outside_m1_deadline,success_after_native_m1_age_threshold_still_winscluster-controller/transient_lifecycle_sweeper/lifecycle_writer_tests.rs(10 tests)fail_cas_claims_only_the_lifecycle_it_was_listed_undermanaged_by; the query has no Kubernetes-specific termfail_cas_clears_the_moved_columns_and_spares_a_published_evacuationmanaged_by; the query has no Kubernetes-specific termfail_undo_reverts_only_its_own_failed_writemanaged_by; the query has no Kubernetes-specific termfail_cas_waiting_on_a_row_lock_decides_on_the_committed_rowmanaged_by; the query has no Kubernetes-specific termconverge_cas_moves_only_restoring_rowsmanaged_by <> 'k8s' OR routable execution) passes every Direct row; with'k8s'rows it needs a routable execution. Fixture: a Kubernetes substrate reachable from this crate (none is exported yet) plus seeded running executionsrevert_cas_moves_only_restoring_rowsmanaged_by; the query has no Kubernetes-specific termrevert_undo_reverts_only_its_own_suspended_writemanaged_by; the query has no Kubernetes-specific termunrestorable_fail_cas_moves_only_unplaced_restoring_rows_of_its_substratemanaged_by; the query's twoNOT EXISTS (… sandbox_k8s_executions …)guards hold trivially for these Direct rows, which have no executions, and are covered at the query level bypu4_execution_history_naming_a_node_shieldsandpu5_active_execution_shieldsintransient_lifecycle_sweeper/tests.rsreap_moves_only_marked_started_rows_and_heal_never_moves_a_lifecyclemanaged_by; the query has no Kubernetes-specific termreap_undo_reverts_only_its_own_failed_writemanaged_by; the query has no Kubernetes-specific termcluster-controller/warm_pool/tests.rs(55 tests)observe_warming_slots_retries_not_started_probe_and_promotes_slotwarm_pool.rsisdelete_executionfor cleanup-pending slotsobserve_warming_slots_keeps_young_slot_missing_from_cachereconcile_config_refills_after_observation_failurereconcile_confignever calls the substratereconcile_config_refused_by_placement_still_runs_teardownreconcile_config_does_not_refill_when_slot_counts_failreconcile_once_starts_preparing_republished_exact_releasedelete_executionis never reachedreconcile_config_stage_or_continue_marks_failure_and_returns_empty_workreceipt_backstops_are_noops_inside_the_cadence_interval47 tests in bucket 0 (never construct or reach Direct)
timed_io_times_out_slow_future,pod_create_timeout_keeps_warming_slot_row_for_later_observation,warm_slot_create_concurrency_follows_refill_per_tick,warm_slot_create_concurrency_clamps_to_minimum_one,warm_slot_create_concurrency_clamps_to_ceiling,sandbox_namespace_prefers_space_sandbox_namespace,sandbox_namespace_falls_back_to_pod_namespace,pod_config_uses_api_compatible_env_defaults,slots_to_create_respects_target_refill_and_warming_budget,slots_to_create_stops_when_open_slots_meet_target,slots_to_create_stops_when_pool_is_disabled,target_open_slots_is_zero_when_pool_is_disabled,slot_matches_config_requires_cached_policy_and_namespace,ready_slot_requires_matching_pod_identity,claimable_slot_metadata_rejects_running_pod_until_sandbox_started,claimable_slot_metadata_accepts_started_sandbox,warm_slot_probe_retry_classification_treats_not_started_as_transient,warm_slot_probe_retry_delay_desynchronizes_slots,summary_add_assign_accumulates_deleted_excess_slots,summary_delete_counts_maps_every_delete_field_to_its_reason,summary_add_assign_accumulates_required_operation_failures,cleanup_pending_slot_delete_result_keeps_sweep_best_effort,cleanup_pending_slot_batch_attempts_remaining_slots_after_delete_failure,cleanup_pending_slot_batch_attempts_remaining_slots_after_timeout,stale_claimed_slot_batch_attempts_remaining_slots_after_delete_failure,reap_stale_claimed_slot_runs_recheck_then_pod_delete_then_row_finalize,reap_stale_claimed_slot_skips_pod_delete_when_recheck_fails,reap_stale_claimed_slot_keeps_row_when_pod_delete_fails,reap_stale_claimed_slot_reports_skip_when_locked_verify_fails,template_write_request_targets_exact_pool_release_digest,template_write_node_list_params_filters_to_sandbox_nodes,cached_warm_pods_by_name_filters_to_pool_config,cached_cordoned_node_names_filters_reflector_state,matching_node_without_template_presence_needs_template_write_annotation,node_with_template_presence_does_not_need_template_write_annotation,node_with_existing_matching_template_write_annotation_is_current,unready_warming_pod_deletion_keeps_young_pending_pods,unready_warming_pod_deletion_removes_old_pending_pods,unready_warming_pod_deletion_removes_failed_pods,target_open_slots_is_zero_for_encryptable_configs,node_is_cordoned_true_only_when_unschedulable_set,node_is_cordoned_false_when_schedulable_unset_or_no_spec,create_warming_slots_bounds_in_flight_concurrency,create_permits_bound_in_flight_creates_across_config_batches,create_warming_slots_returns_every_result_on_partial_failure,create_warming_slots_does_nothing_when_count_is_zero,reconcile_configs_runs_configs_concurrently_and_isolates_failures🤖 Generated with Claude Code