Repository navigation
api: tie node_addressing to endpoint resolution and fold the wake-fence test into its table - #597
Conversation
1c026fd to
61ad054
Compare
…ce test into its table Follow-ups to the review of #368 (`SubstrateCapabilities`). Tests and comments only; no production behavior changes. - `node_addressing_names_the_field_endpoint_resolution_reads` (`substrate/tests.rs`) runs each substrate's `resolve_node_api_endpoint` on a target carrying only the field its `capabilities().node_addressing` names (it resolves) and on one carrying only the other field (NotFound). Before, flipping either constant's `node_addressing` failed no test. It follows the shape of `node_drain_capability_matches_set_node_schedulability` and covers Kubernetes when built with the `kubernetes` feature. - `wake_completion_cas_outcome_for_every_prior_lifecycle` now runs `SubstrateCapabilities::DIRECT` and `::KUBERNETES` through `PublishedSuspendFence::for_capabilities` instead of naming the fences, and asserts each running substrate's `capabilities()` returns its constant. That absorbs `wake_completion_fence_follows_the_running_substrate`, which is removed; its `suspended` row outcomes are table rows. - `wake_completion_outcome_maps_to_the_wake_response` (lifecycle characterization) also wakes a Direct sandbox from `suspended`, with the simulated resident suspended to the shared directory, across the same mid-flight lifecycle moves. With no move it ends `started` / `on_node`, which pins the Wake arm's fence choice: hard-coding `PublishedSuspendFence::Enforced` there leaves the row `suspended`. - Comments: the table test credits Direct's `suspended` baseline to the worker-local wake, not a direct restore, and says only Direct's side of the Wake arm's fence choice is pinned outside the race. The Restore arm states the known Direct gap, which the Wake arm's lease-restore fallback shares: Direct's suspend writes no `suspending` claim, only `suspended` after its node call, so a `suspended` landing between the lease restore's `started` write and the completion CAS is overwritten back to `started`; and because the lease write is not guarded on lifecycle, a `suspended` written after the restore placed the row `on_node` (a span that, for a remote restore, includes the source cleanup call) is overwritten by the lease write itself. `delete_observed_sandbox` states why defaulting a missing node `managed_by` to `Space` is harmless: neither substrate's delete path reads it, only `role` and the address fields.
61ad054 to
81bf669
Compare
johnlouie-pplx
left a comment
There was a problem hiding this comment.
Thanks for working through the #368 follow-ups. The new node_addressing test and folding the fence test into the table both look good, and the mutation checks in the description make the coverage easy to trust.
One ask: could you open a tracking issue for the Direct restore race the new lifecycle.rs comment describes? A suspended write that lands after the lease restore's started write, or after the restore puts the row on_node, gets overwritten back to started, because complete_direct_restore_lease matches only restore_lease_token. That leaves the row saying started while the sandbox is suspended. Fine to keep it out of this PR, but it'd be good to have an issue to link from the comment so it doesn't get lost.
Follow-ups to the non-blocking review suggestions on #368 (
SubstrateCapabilities). Tests and comments only; no production behavior, wire format, schema, metric or configuration changes.Suggestion → change
node_addressingrestates whatresolve_node_api_endpointrequires; flipping it fails no testnode_addressing_names_the_field_endpoint_resolution_reads(substrate/tests.rs). For each substrate, it resolves a target carrying only the fieldcapabilities().node_addressingnames (expects the endpoint) and a target carrying only the other field (expectsNotFound). Direct reads the stored endpoint. Kubernetes runs the realtry_resolve_node_api_endpointagainst a ready node-api pod in thekubernetes::testscaches, with no apiserver.wake_completion_fence_follows_the_running_substraterepeats the table testwake_completion_cas_outcome_for_every_prior_lifecyclenow runsSubstrateCapabilities::DIRECT/::KUBERNETESthroughPublishedSuspendFence::for_capabilitiesrather than naming the two fences, and the duplicate test is removed. Its assertions are all kept: the table test asserts that each running substrate'scapabilities()returns its constant, and thesuspendedrow outcomes (Direct completes, Kubernetes defers) are table rows.wake_completion_outcome_maps_to_the_wake_response(lifecycle characterization, against the simulated Direct worker) now also wakes fromsuspended. The resident is suspended to the simulator's shared directory, and the test covers the same mid-flight lifecycle moves. With no move the row endsstarted/on_node, so hard-codingPublishedSuspendFence::Enforcedin the Wake arm now fails it (row stayssuspended); before, that test seeded onlypaused. This pins only Direct's side of the Wake arm's choice: hard-codingDirectRestoreBaselinein the Wake arm still passes. A Kubernetes wake fromsuspendedgoes throughbegin_restore_lifecycle, so the row isrestoringat the CAS. Like the Restore arm, the Kubernetes side is observable only in the race. Restore arm: its fence isn't observable outside the race. On Direct the row is alreadystarted(the lease restore wrote it); on Kubernetes it isrestoring. Neither issuspended, the only lifecycle on which the two fences differ, so a test would have to inject the race. It stays covered only throughfor_capabilitiesin the table test.suspendedlifecycle.rs). Direct's suspend writes nosuspendingclaim, onlysuspendedafter its node call. So asuspendedthat lands after the lease restore'sstartedwrite and before this CAS is overwritten back tostarted.complete_direct_restore_leasematches onlyrestore_lease_token, not lifecycle. So asuspendedwritten after the restore placed the rowon_nodeis overwritten by the lease write itself. For a remote restore, that span includes the source cleanup call. The Wake arm's lease-restore fallback has the same window: on NotFound, a Direct wake falls back torestore_execution_with_candidates, then runs the sameDirectRestoreBaselineCAS. The comment says so. Behavior is unchanged and no issue is filed.delete_observed_sandboxdefaults a missing or unparseable nodemanaged_bytoSpaceNodeApiTarget::managed_by. They read onlyroleand the address fields. Direct's (the trait default) resolves bydirect_endpoint. Kubernetes' deletes a pool slot or resolves bysubstrate_node_id+role.Why a test and not a trait method (suggestion 1). A
node_api_target_is_completemethod next toresolve_node_api_endpointwould be a production change. It would also be a second statement of the requirement that nothing checks against the resolver: the same drift risk, moved. The test checks the declared field against the real resolver. It is one table over both substrates, the same shape asnode_drain_capability_matches_set_node_schedulability, rather than two near-identical functions. The Kubernetes case compiles only with--features kubernetes, which CI's--all-featuresrun includes.Validation
cargo fmt --all -- --checkcargo clippy --workspace --locked --all-targets --all-features -- -D warningspre-commit run --all-files(Gitleaks)SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-api --all-features --retries 0×3benchmark_single_node_occupancy.PublishedSuspendFence::Enforced(full suite)wake_completion_outcome_maps_to_the_wake_responsefails (woken from suspended: rowsuspended, expectedstarted)DIRECT.node_addressing→SubstrateNodeIdnode_addressing_names_the_field_endpoint_resolution_readsfailsKUBERNETES.node_addressing→DirectEndpointnode_addressing_names_the_field_endpoint_resolution_readsfailsDIRECT.caller_completes_claimed_restore→truesuspended, direct)KUBERNETES.caller_completes_claimed_restore→falsesuspended, kubernetes)DirectSubstrate::capabilities()returnsKUBERNETESfor_capabilitiesinvertedNo Kind run, since no production code changed.
🤖 Generated with Claude Code