Skip to content

api: tie node_addressing to endpoint resolution and fold the wake-fence test into its table - #597

Merged
ac-pplx merged 1 commit into
mainfrom
ac/substrate-capabilities-followup
Oct 16, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/substrate-capabilities-followup

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Follow-ups to the non-blocking review suggestions on #368 (SubstrateCapabilities). Tests and comments only; no production behavior, wire format, schema, metric or configuration changes.

Suggestion → change

Suggestion on #368 Change
1. node_addressing restates what resolve_node_api_endpoint requires; flipping it fails no test New node_addressing_names_the_field_endpoint_resolution_reads (substrate/tests.rs). For each substrate, it resolves a target carrying only the field capabilities().node_addressing names (expects the endpoint) and a target carrying only the other field (expects NotFound). Direct reads the stored endpoint. Kubernetes runs the real try_resolve_node_api_endpoint against a ready node-api pod in the kubernetes::tests caches, with no apiserver.
2a. wake_completion_fence_follows_the_running_substrate repeats the table test wake_completion_cas_outcome_for_every_prior_lifecycle now runs SubstrateCapabilities::DIRECT / ::KUBERNETES through PublishedSuspendFence::for_capabilities rather than naming the two fences, and the duplicate test is removed. Its assertions are all kept: the table test asserts that each running substrate's capabilities() returns its constant, and the suspended row outcomes (Direct completes, Kubernetes defers) are table rows.
2b. The Wake and Restore call sites stay uncovered Wake arm: the controller-level wake_completion_outcome_maps_to_the_wake_response (lifecycle characterization, against the simulated Direct worker) now also wakes from suspended. The resident is suspended to the simulator's shared directory, and the test covers the same mid-flight lifecycle moves. With no move the row ends started / on_node, so hard-coding PublishedSuspendFence::Enforced in the Wake arm now fails it (row stays suspended); before, that test seeded only paused. This pins only Direct's side of the Wake arm's choice: hard-coding DirectRestoreBaseline in the Wake arm still passes. A Kubernetes wake from suspended goes through begin_restore_lifecycle, so the row is restoring at the CAS. Like the Restore arm, the Kubernetes side is observable only in the race. Restore arm: its fence isn't observable outside the race. On Direct the row is already started (the lease restore wrote it); on Kubernetes it is restoring. Neither is suspended, the only lifecycle on which the two fences differ, so a test would have to inject the race. It stays covered only through for_capabilities in the table test.
3a. Nit: the table comment credits a "direct restore" with keeping suspended The table comment now credits Direct's worker-local wake.
3b. The Direct Restore-arm window Comment above the Restore arm's completion CAS (lifecycle.rs). Direct's suspend writes no suspending claim, only suspended after its node call. So a suspended that lands after the lease restore's started write and before this CAS is overwritten back to started. complete_direct_restore_lease matches only restore_lease_token, not lifecycle. So a suspended written after the restore placed the row on_node is overwritten by the lease write itself. For a remote restore, that span includes the source cleanup call. The Wake arm's lease-restore fallback has the same window: on NotFound, a Direct wake falls back to restore_execution_with_candidates, then runs the same DirectRestoreBaseline CAS. The comment says so. Behavior is unchanged and no issue is filed.
4. delete_observed_sandbox defaults a missing or unparseable node managed_by to Space Comment: the default is harmless because neither substrate's delete path reads NodeApiTarget::managed_by. They read only role and the address fields. Direct's (the trait default) resolves by direct_endpoint. Kubernetes' deletes a pool slot or resolves by substrate_node_id + role.

Why a test and not a trait method (suggestion 1). A node_api_target_is_complete method next to resolve_node_api_endpoint would be a production change. It would also be a second statement of the requirement that nothing checks against the resolver: the same drift risk, moved. The test checks the declared field against the real resolver. It is one table over both substrates, the same shape as node_drain_capability_matches_set_node_schedulability, rather than two near-identical functions. The Kubernetes case compiles only with --features kubernetes, which CI's --all-features run includes.

Validation

Check Result
cargo fmt --all -- --check pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings pass
pre-commit run --all-files (Gitleaks) pass
SPACE_CI_REQUIRE_TEST_DEPS=1 cargo nextest run --locked -p pplx-space-api --all-features --retries 0 ×3 1832/1832 passed each run. The 1 skip is the existing ignored benchmark benchmark_single_node_occupancy.
Mutation: Wake arm's fence hard-coded to PublishedSuspendFence::Enforced (full suite) only wake_completion_outcome_maps_to_the_wake_response fails (woken from suspended: row suspended, expected started)
Mutation: DIRECT.node_addressing → SubstrateNodeId node_addressing_names_the_field_endpoint_resolution_reads fails
Mutation: KUBERNETES.node_addressing → DirectEndpoint node_addressing_names_the_field_endpoint_resolution_reads fails
Mutation: DIRECT.caller_completes_claimed_restore → true table test fails (suspended, direct)
Mutation: KUBERNETES.caller_completes_claimed_restore → false table test fails (suspended, kubernetes)
Mutation: DirectSubstrate::capabilities() returns KUBERNETES table test fails (running-substrate assertion)
Mutation: for_capabilities inverted table test fails

No Kind run, since no production code changed.

🤖 Generated with Claude Code

@ac-pplx
ac-pplx force-pushed the ac/substrate-capabilities-followup branch from 1c026fd to 61ad054 Compare October 16, 2026 22:14
…ce test into its table

Follow-ups to the review of #368 (`SubstrateCapabilities`). Tests and
comments only; no production behavior changes.

- `node_addressing_names_the_field_endpoint_resolution_reads`
  (`substrate/tests.rs`) runs each substrate's `resolve_node_api_endpoint`
  on a target carrying only the field its `capabilities().node_addressing`
  names (it resolves) and on one carrying only the other field (NotFound).
  Before, flipping either constant's `node_addressing` failed no test.
  It follows the shape of `node_drain_capability_matches_set_node_schedulability`
  and covers Kubernetes when built with the `kubernetes` feature.
- `wake_completion_cas_outcome_for_every_prior_lifecycle` now runs
  `SubstrateCapabilities::DIRECT` and `::KUBERNETES` through
  `PublishedSuspendFence::for_capabilities` instead of naming the fences,
  and asserts each running substrate's `capabilities()` returns its
  constant. That absorbs `wake_completion_fence_follows_the_running_substrate`,
  which is removed; its `suspended` row outcomes are table rows.
- `wake_completion_outcome_maps_to_the_wake_response` (lifecycle
  characterization) also wakes a Direct sandbox from `suspended`, with the
  simulated resident suspended to the shared directory, across the same
  mid-flight lifecycle moves. With no move it ends `started` / `on_node`,
  which pins the Wake arm's fence choice: hard-coding
  `PublishedSuspendFence::Enforced` there leaves the row `suspended`.
- Comments: the table test credits Direct's `suspended` baseline to the
  worker-local wake, not a direct restore, and says only Direct's side of
  the Wake arm's fence choice is pinned outside the race. The Restore arm
  states the known Direct gap, which the Wake arm's lease-restore fallback
  shares: Direct's suspend writes no `suspending` claim, only
  `suspended` after its node call, so a `suspended` landing between the
  lease restore's `started` write and the completion CAS is overwritten
  back to `started`; and because the lease write is not guarded on
  lifecycle, a `suspended` written after the restore placed the row
  `on_node` (a span that, for a remote restore, includes the source
  cleanup call) is overwritten by the lease write itself.
  `delete_observed_sandbox` states why defaulting a missing node
  `managed_by` to `Space` is harmless: neither substrate's delete path
  reads it, only `role` and the address fields.
@ac-pplx
ac-pplx force-pushed the ac/substrate-capabilities-followup branch from 61ad054 to 81bf669 Compare October 16, 2026 22:23
@ac-pplx
ac-pplx marked this pull request as ready for review October 16, 2026 22:39

@johnlouie-pplx johnlouie-pplx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through the #368 follow-ups. The new node_addressing test and folding the fence test into the table both look good, and the mutation checks in the description make the coverage easy to trust.

One ask: could you open a tracking issue for the Direct restore race the new lifecycle.rs comment describes? A suspended write that lands after the lease restore's started write, or after the restore puts the row on_node, gets overwritten back to started, because complete_direct_restore_lease matches only restore_lease_token. That leaves the row saying started while the sandbox is suspended. Fine to keep it out of this PR, but it'd be good to have an issue to link from the comment so it doesn't get lost.

@ac-pplx
ac-pplx added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 34d97ac Oct 16, 2026
20 checks passed
@ac-pplx
ac-pplx deleted the ac/substrate-capabilities-followup branch October 16, 2026 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants