[ac-pplx]

Summary

Each node now has a page of its own, with its health, occupancy and sandboxes, and the node actions the nodes list allows, plus cordon, uncordon and uploading node-local snapshots.

Before this change:

• Nodes had no page. Their sandboxes were a drawer of bare UUIDs, and health was a status tag.
• The console offered only gentle drain, undrain, reconcile and delete.
• The admin API also supports cordon, uncordon, forced drain, uploading node-local snapshots to the object store, and terminate.

/dashboard/nodes/{id} shows:

• Node: ID, address, Kubernetes node, manager, state, scheduling, last heartbeat, when its status was observed, and the last drain error.
• Occupancy: active sandboxes, node-local snapshots, and network slots (leased, ready, degraded, quarantined).
• Health: reachability, the time of the status snapshot it comes from, node-ctl's own error when it is unreachable, each service, and each node-ctl reconciler's last success, next run and last error. It is read from the node response's own status snapshot (status.health, with status.observed_at), so health is never paired with metadata from a different read. Only worker and builder nodes have health; other roles show no Health card.
• Sandboxes on this node: paged as the API pages them (100 at a time), each linking to its page. A failed request shows an error with Retry instead of an empty node, an answer for an older request is dropped, and Refresh reloads the list.

The page offers actions by the nodes list's rules: only on worker and builder nodes, and drain, cordon and uncordon only on substrates that support node drain (the Direct substrate answers 501). Undrain clears an earlier drain on any substrate. Without a drain: Cordon, Uncordon and a gentle Drain. While draining or drained: Undrain. Upload local snapshots only where the API accepts it: drained, terminating, or a forced drain (it refuses during a gentle drain). Each action sits behind a confirmation that says what happens to resident sandboxes; dangerous ones focus Cancel. Both drain confirmations, here and on the nodes list, say that once the node is drained it is terminated automatically wherever the drained-node terminator is enabled, unless the node carries the space.pplx.net/drain-keep annotation.

Forced drain and terminate stay CLI-only (space-admin-cli drain-node --forced, space-admin-cli terminate-node), as the nodes list already decided: they are not dashboard buttons one misclick away from evacuating or ending a node.

Uploading node-local snapshots asks the node to queue each upload; an accepted request is not yet a durable snapshot, which it becomes once the upload completes and registers it in the object store. One call lists a limited number of pinned sandboxes, in sandbox ID order, and attempts uploads for only the first of them. The confirmation says so, and that the result says what was left out, without naming numbers: the limits are the server's (FORCE_UPLOAD_MAX_PER_INVOCATION and FORCE_UPLOAD_ENUMERATION_CAP in crates/libs/api/src/routes/admin/nodes.rs) and are not copied into the console. The result is titled "Upload requests accepted for N sandboxes" and lists the requests that failed with their errors. What the call left out comes only from the answer: deferred gives "N listed sandboxes not attempted in this call", and truncated gives "More sandboxes are pinned to this node than this call listed". The result also notes that a retry starts with the same sandboxes, failed ones included, because the order is stable.

After every action, successful or not, the page reloads the node, because some actions commit and then report an error. Node reads carry a request generation and an action invalidates earlier reads as it starts, so a Refresh that began before an action cannot land after it and restore the previous state; the actions stay disabled until the post-action read settles.

Node names on the nodes page link here. The active-sandbox count opens the existing drawer.

The admin node response does not report whether the substrate node is cordoned, so Cordon and Uncordon are both offered outside a drain.

Stack

This is stacked on "ui: show each node's sandbox and slot occupancy" (#1252); both change the nodes page, and #1252 reads occupancy as a separate best-effort read. Merge #1252 first. The node page itself has no polling interval: it reads on open, on Refresh and after an action.

Validation

• npm ci, npm test (56 pass, including src/lib/node-actions.test.mjs for the action rules across roles, substrates and drain phases, a check that the upload confirmation names no numbers, and the result notes built from deferred and truncated), npx tsc --noEmit -p . (no errors) and npm run build in ui/. No ui/ source mentions the server's upload limits.
• A Playwright loop against a local Kind stack with synthetic data, on the published head and on this branch (the Kind space-ui image):
  • the busiest worker's page shows occupancy, health and its sandboxes, and a sandbox link opens the sandbox;
  • on an idle builder node, Cordon and then Uncordon each ask for confirmation and succeed;
  • an idle worker offers Cordon, Uncordon and Drain only, and a node presented as cluster-role offers none;
  • both drain confirmations carry the termination warning, and the node page's focuses Cancel;
  • a node presented as drained offers Upload and Undrain only, and a synthetic upload result shows accepted requests, the failed request, and the sandbox not attempted in this call;
  • a Refresh read held back past a Drain does not restore the undrained view, and the actions stay disabled until the post-action read settles;
  • the node reloads after a successful and after a failed action;
  • a failed sandbox list shows an error with Retry, a stale answer is dropped, and Refresh reloads the list;
  • the health card shows its own snapshot's time and node-ctl's error from that snapshot;
  • afterwards every Kubernetes node is schedulable again.
• No new console errors, page errors or failed requests compared with main.
• A Playwright run on the same Kind stack compared production builds of an earlier head of this branch and of this head on a worker's page, with the node read shown as drained and the upload answered synthetically (2 uploaded, 1 failed, 3 deferred, truncated), so nothing on the node changed. The earlier head's confirmation and result named 32 and 256. This head's confirmation names no numbers, and its result says "3 listed sandboxes not attempted in this call." and "More sandboxes are pinned to this node than this call listed."

Visual confirmation

Drain on the node page, with the termination warning; Cancel has focus:

Video: after.

Node page drain confirmation	Nodes list drain confirmation

The upload confirmation, on an earlier head of this branch and on this head: it no longer names the server's limits:

Before	After

The upload result for a synthetic answer with deferred sandboxes and truncated, before and after: what was left out now comes from the answer alone, without 32 or 256:

Before	After

Severity and risk

Severity: medium. Node health and drain state are what an operator checks first in a capacity incident.

Risk: medium. The page exposes cordon, uncordon, gentle drain and snapshot upload, which change where sandboxes run. A gentle drain can end with the node terminated where the drained-node terminator is enabled; the confirmation says so. Forced drain and terminate are not offered. Each action needs a confirmation that describes the effect, and the API's own guards still apply. Roll back by reverting the commits.

AI assistance: Claude Code wrote the code and this description, and ran the validation; a maintainer reviews it.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[zibi-pplx]

The confirmations, and keeping forced drain and terminate off the dashboard, look right. One maintainability note.

• [low] The confirmation text mirrors server constants with nothing linking the two.
  ui/src/lib/node-actions.ts:18-19 copies FORCE_UPLOAD_MAX_PER_INVOCATION = 32 and FORCE_UPLOAD_ENUMERATION_CAP = 256 (crates/libs/api/src/routes/admin/nodes.rs:1805,1818), and node-actions.test.mjs asserts on the literal numbers. If either constant changes, the dialog describes the old limits and the test still passes. Suggestion: add a comment next to the Rust constants pointing at node-actions.ts. Alternatively, word the confirmation without the numbers and rely on the response's deferred and truncated fields, which the result view already reads.

Withdrawing this approval for now. Any feedback above still stands.

[johnlouie-pplx]

Approving with one nonblocking follow-up. I reviewed f8169b3; the UI build and all 56 tests passed in my earlier validation of this same head. I also checked the page against a mocked admin API, not a live cluster. The upload dialog now reports what the server returned instead of repeating its limits.

In ui/src/app/dashboard/nodes/[id]/page.tsx, loadNode still requests include_occupancy=true. If the count query fails, the entire page becomes "Could not load the node", hiding health, slots and the recovery buttons. A transient failure on Refresh does the same even after a successful load.

Could we fetch counts separately, as #1252 does, so a count failure doesn't hide the rest of the page? For transient refresh failures, keep the last loaded node with an explicit stale/error notice. Permission failures and not-found responses should still replace the page.