You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ui: show what holds a node and why on the node page - #1675
The node page shows what holds the node and why, and its sandbox list becomes a collapsed, filterable History.
When a drain stalls or :terminate is refused, the page listed every sandbox that ever named the node, by id, and counted that page by lifecycle. History dominated it, and nothing said which sandboxes held the node or what to do about them. The page now reads GET /v1/nodes/{id}/holds from the server PR and adds a What holds this node panel. It groups the rows by who acts on them next, under the node's drain stage: no drain, a gentle or a forced drain while the node is draining, or parked once it is drained or terminating. The drain moves, uploads and fails a node's sandboxes only while the node is draining; on a parked node it no longer acts on what still holds it, though restores of sandboxes it already evacuated continue. The groups, in this order:
Group
Reasons
Needs you
sandboxes the node's report omits, while a gate still counts them; transitions past the drain's 10-minute limit, unless a forced drain is running; on a parked node, running sandboxes and suspended sandboxes whose only snapshot is on the node
Handled automatically
evacuations, uploads in flight and young transitions; suspended sandboxes whose only snapshot is on the node, unless the node is parked; under a forced drain, also running sandboxes and stuck transitions
Waiting on users
running sandboxes under a gentle drain or none
Blocks terminate
stopped, created, deleting and quarantined sandboxes, and a stopped sandbox an evacuation left part-way, outside a forced drain
Listed, not counted
rows no gate counts, such as sandboxes the node's report omits under node-census residency
Rows: each row has the sandbox link, lifecycle and age, one sentence for its reason in the node's drain stage, and the gates that count it. A created or stopped row under a forced drain or on a drained node says Undrain lets its owner start it; on a terminating node, which the API refuses to undrain, only deleting it lets the node terminate.
Header: shows the residency evidence once ("rows only" or "node census"), with a tooltip on what it means for rows the node does not report.
Actions: Upload local snapshots appears beside the local-only rows, in whichever group holds them, whenever the API accepts it.
Refused terminate: the refusal links to the panel and sets its counts beside the panel's.
Summary row: the Drain section's sandbox summary now counts the groups, leaving out the rows no gate counts. When no gate term counts any row, it and the panel read "Nothing holds this node", and the panel still lists the rows it does not count.
History: the node's sandbox list becomes a collapsed History, newest lifecycle change first, with a lifecycle filter and Failed during this drain (failed rows changed since the drain started). Ending or restarting the drain clears that toggle and the paging, and the reload after an action keeps the filters on screen.
Polling: the holds read is polled like the page's other reads, for worker and builder nodes only, serialized and bounded, and stops after a 403 or 404. An API without the route answers 404, and the panel says it does not list holds.
npm test in ui/ (389 pass; the tests cover the group of every reason under each of the four drain stages, a stopped and a restoring evacuation, rows no gate counts, the sentences per stage, including the Undrain exit and the terminating case, and the refusal's counts) and npm run build. The console loop's typecheck and test pass too.
Kind (local cluster): current main, the server commit, ui: add forced drain, terminate and a drain section to the node pages #1662 and this commit, on a worker with a hold for every reason: 15 rows, 14 synthetic (including a stopped evacuation, an evacuation restoring onto the node, and created, deleting and quarantined rows) and the loop's seeded running sandbox. The cluster-controller was paused so its sweeps left the synthetic rows alone, and the node's drain phase and mode were set in turn to none, gentle, forced and drained, then terminating.
Under each stage the panel listed all 15 rows in the groups the table above gives, with the uncounted census-absent row last, and node-holds-panel passed all 53 of its checks, including each row's Why and its exact Counted by.
The rendered sentences for the created and stopped rows give the Undrain exit under the forced drain and on the drained node, and keep "only deleting it lets the node terminate" on the terminating node.
On the drained node, the refused terminate's counts (11 occupying, 3 held) matched the panel's, 14 sandboxes in all.
node-history-filters passed in every stage: after the Reload the History showed the answer's 20 rows in its order, newest change first; the failed filter keeps 3 of 3 rows; and with Failed during this drain on, it showed the 2 rows that failed after the drain started, in the answer's order.
A synthetic node holding only an uncounted census-absent row read "Nothing holds this node" in the panel and the Drain section, and the panel still listed the row.
Two UI mutations, each served from a copy of the console: with the History's rows reversed, the four new History checks failed; with the Drain section showing its group tags when no term counts a row, the "Nothing holds this node" check failed. Both checks pass on the unmutated build.
npm run loop -- --video --no-mutate against a production build of main and then this branch, on the same stack: 0 new error types, 7 resolved (main has no Drain section or holds panel).
Not validated: the flows steps were not run, since --no-mutate skips them, including ui: add forced drain, terminate and a drain section to the node pages #1662's two drain flows, which also refuse while a node is draining. The older-API notice was checked only against a substituted answer, not against an older API.
Severity and risk
Severity: does not apply. It adds an operator view.
Risk: it touches the node page only. The History now reads the node's sandboxes newest change first, through optional parameters the server PR adds. Against an API without that PR, the History's filters, change times and paging do not work, since the API ignores the parameters and answers its first page in id order. The History shows a notice saying so, and its change times show as "—"; merge and deploy the server PR first. Roll back by reverting the commit.
Stack
Stacked on #1662 (ac/console-node-drain-panel); merge #1662 first. It also needs #1674 (GET /v1/nodes/{id}/holds and the history parameters) merged and deployed first. Without it the panel reports that the API does not list holds.
AI assistance: Claude Code made the change, ran the validation and drafted this description. A human reviews before merge.
Operators could cordon, gently drain and undrain a node from the console,
but a forced drain and a terminate needed `space-admin-cli`, and the node
page showed only the drain phase, mode and last error. Following a drain
meant reloading the page, its history was only in the audit log page, and
the nodes list judged a heartbeat stale by a client-side guess (twice the
longest reconciler poll interval) rather than by the node's `stale_after`.
The node page and the nodes list now offer Forced drain (on an idle node,
and on a gentle drain, which the API escalates in place) and Terminate (on
a drained node). Both send the same requests as the CLI and open a
confirmation that says what happens and stays disabled until the node's
name is typed. A terminate refused with 409 is shown as a list of reasons:
the sandboxes still running or held for the node, or the phase it is in.
The substrate rules stay in `nodeActions`, which the nodes list now uses as
well, so the Direct substrate still gets only Undrain.
The node page gains a Drain section: phase, mode, start time and elapsed,
last error, heartbeat age and staleness, whether the API's node cache holds
the Kubernetes Node, the node's sandboxes by lifecycle, and the node's
actions from the admin audit log (action, operator, time, result), newest
first. The audit log endpoint has no node filter, so the newest 250 events
are read and filtered by path, and the section says when that page is full
and what the log does not record. The page re-reads the node every 5 s, and
its sandboxes and the audit log every 15 s, while the tab is visible.
The nodes list's stale heartbeat now follows the node's `stale_after`, the
rule behind the API's own `status.stale` flag, and gains a heartbeat filter.
The logic is in `lib/node-drain.ts` and `lib/node-actions.ts`, with unit
tests.
The console loop gains steps for the heartbeat filter, the Drain section,
the remaining sandboxes, a forced drain and undrain of an empty Kind
worker, and a refused terminate. The drain steps set the drain-keep
override first, because the Kind terminator otherwise deletes a drained
node, and remove it after undraining.
No API, database, controller or CLI code changes: the console calls
existing admin endpoints with the bodies the CLI already sends. What the
console itself shows and offers changes as described above.
…steps
The drain steps' cordon probe checked only the kube context's Node for
an existing cordon, so with a mismatched context its uncordon could lift
a cordon the step had not added. It now also refuses a worker the API
shows cordoned. A drain-keep annotation that an interrupted run of the
loop left counts as the loop's own. The README's recovery steps use the
run's kube context and cover a cordoned node, and the idle worker's
action names are asserted again beside its labels.
A drain step treated a leftover drain-keep value of console-loop as its
own, so a second run against the same stack could adopt a live run's
override and remove it before that run drained, and the Kind terminator
could then delete the drained worker. Each run now writes an override
carrying its own run id, claims it with an annotate that fails when the
key exists, and removes it only while it still carries that id. A
worker that already has an override, including one an interrupted run
left, is refused and left as it is.
A claim whose annotate reached the server but whose answer was lost read
back as someone else's override, so the step skipped its cleanup and
left the override on the node. Each claim attempt now writes its own
token and counts a read-back of that token as a won claim. Cleanup after
a failed step records each error and keeps going where it is safe: it
keeps the step's own error, undrains even when reading the node fails,
and removes the override only once the node shows no drain phase.
When a node's status poll fails, the status snapshot is observed at
the failure, so the Drain section showed a recent heartbeat for a node
that did not answer. An unreachable status no longer counts as a
heartbeat. The reload after an action now reads the page the operator is
on, not the one they were on when the action started, and the forced
drain tests check every action's mode, not just its label.
Draft PRs are not automatically reviewed by default.
Trigger a manual review
To automatically review draft PRs, update your CodeRabbit configuration:
reviews:
auto_review:
drafts: true
No actionable comments were generated in the recent review.
Walkthrough
The node detail page adds a holds panel for worker and builder nodes, with grouped hold details and occupancy counts. Sandbox history becomes a collapsible, filterable, cursor-paged view ordered by lifecycle-change time. Termination refusals can include active and drain-blocker counts. The console loop adds checks for the holds panel and history filters.
perplexityai/SPACE#1662 — This change extends the Drain section, ActionRefusal, and the node-drain.ts module introduced there.
perplexityai/SPACE#1674 — This change reads the holds endpoint and uses the sandbox-history query parameters added there.
Merge Risk: Low · up to 5db13
The node page relies on backend changes in #1662 and #1674. Merge those first. Without them the holds panel reports that the API does not list holds, and History filters and paging do not work. The page shows a warning in that case.
Security architecture risk: Low · up to 5db13
The change presents node-hold information through the existing administrative session and reuses existing confirmed node actions. No new permission or action-targeting capability was demonstrated. The corresponding server authorization and response contracts remain unverified.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
observed — The new consumer requests hold metadata for one route-selected worker or builder node, including sandbox identity and lifecycle information. Actual cross-node or tenant exposure depends on the unavailable server authorization implementation; the client role check is only a request-scheduling control.
Trust Boundaries and Controls
observed — The holds read uses the existing administrative API client, which supplies console authentication headers and same-origin credentials and rejects unsuccessful responses. This establishes transport continuity, not proof that the new server route enforces authorization or node scope.
Resilience and Maintainability Implications
observed — Failed holds reloads preserve previously loaded rows with a stale-data warning, and the contextual upload shortcut can remain visible. That shortcut still invokes the same confirmed action already offered in the page header; cached hold rows do not supply its authority or request targets.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title "ui: show what holds a node and why on the node page" starts with the lowercase area prefix "ui:" followed by a lowercase summary word. It has no trailing period, ticket ID, or internal link…
Description Follows The Pr Template
The description contains a Summary, Validation results, Severity and risk, and an AI assistance line naming the tool and its contributions.
Bug Fixes Name A Regression Test
This PR adds a holds panel and History filters. The diff and description identify an application feature, not a bug fix. The check does not apply.
Weakened Safeguards Are Justified
The removed lifecycle-count test covered a removed summary. The description explains the replacement. The new holds check also explains why it checks the link instead of clicking it: a census-absent s…
Public-Safe Content
The diff, title, and description contain no content that meets the listed failure conditions. The validation measurements describe a local test cluster, not production.
Full details: Title check
Explanation
The title "ui: show what holds a node and why on the node page" starts with the lowercase area prefix "ui:" followed by a lowercase summary word. It has no trailing period, ticket ID, or internal link. It matches the main change: a new What holds this node panel on the node detail page that loads GET /v1/nodes/{id}/holds.
Full details: Weakened Safeguards Are Justified
Explanation
The removed lifecycle-count test covered a removed summary. The description explains the replacement. The new holds check also explains why it checks the link instead of clicking it: a census-absent sandbox may not be readable through the public API.
Recent review info
Automatic approval policy
Commit: 5db13b53384b.
Setting
Effective value
Source
Automatic approvals
Enabled
Global configuration
Merge risk
Low
Override 2
Security risk
Not set
Global configuration
Matching overrides (numbered as in the resolved configuration):
Any matching disabled override withholds approval, regardless of risk. Otherwise, the strictest explicitly set limit wins for each score. Limits omitted by all matching overrides inherit the global configuration.
Not set means no limit was configured after inheritance. Use @coderabbitai configuration to inspect the resolved settings and their sources.
Reviewing files that changed from the base of the PR and between 85bf503 and 5db13b5.
Files selected for processing: 6
scripts/console-loop/README.md
scripts/console-loop/src/nodes.ts
scripts/console-loop/src/steps.ts
ui/src/app/dashboard/nodes/[id]/page.tsx
ui/src/lib/node-drain.test.mjs
ui/src/lib/node-drain.ts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
When a drain stalls or :terminate is refused, the node page listed every
sandbox that ever named the node, in any lifecycle, ordered by id, and
counted that page by lifecycle. History dominated the list, and the page
could not say which sandboxes held the node, why, or what to do about
them.
The page now reads GET /v1/nodes/{id}/holds and shows a "What holds this
node" panel. It groups the rows by who acts on them next, under the
node's drain stage: none, a gentle or forced drain while the node is
draining, or parked once it is drained or terminating. The groups are
Needs you (census-absent rows; stuck transitions unless a forced drain
is running; running residents and local-only snapshots on a parked
node), Handled automatically (evacuations, uploads in flight, young
transitions, suspended sandboxes whose only snapshot is on the node,
unless the node is parked, and stuck transitions and running residents
under a forced drain), Waiting on users (running residents under a
gentle drain or none), Blocks terminate (created, stopped, deleting and
quarantined rows, and a stopped evacuation outside a forced drain) and
Listed, not counted (rows no gate counts, such as census-absent rows
under node-census residency). Each row has its sandbox link, lifecycle
and age, one sentence for its reason in that stage, and the gates that
count it. A created or stopped row under a forced drain or on a drained
node says Undrain lets its owner start it; on a terminating node, which
the API refuses to undrain, only deleting it lets the node terminate.
The header shows the residency evidence once ("rows only" or "node
census"), with what it means for rows the node does not report. Upload
local snapshots sits beside the local-only rows when the API accepts it.
A refused terminate links to the panel and sets its counts beside the
panel's. The Drain section's sandbox summary counts the groups instead,
leaving out the rows no gate counts; a node no term counts reads
"Nothing holds this node".
The node's sandbox list becomes a collapsed History, newest lifecycle
change first, with a lifecycle filter and "Failed during this drain"
(failed rows changed since the drain started), through the list's new
parameters. Ending or restarting the drain clears that toggle and the
paging, and the reload after an action keeps the filters on screen. An
API that answers without next_cursor ignores these parameters; the
History then says it shows the first 100 rows by id, unfiltered, and an
unreadable change time shows as "—". The holds read is polled like the
page's other reads, for worker and builder nodes only, serialized and
bounded, and stops after a 403 or 404, which is also what an API without
the route answers. lifecycleCounts lost its only caller and is removed,
and terminateRefusal returns the refusal's counts.
Unit tests cover the group of every reason under each drain stage, the
stopped and restoring evacuations and the rows no gate counts, and the
sentences per stage. The grouping follows the drain, which moves,
uploads and fails a node's sandboxes only while the node is draining: a
drained node with a forced mode is parked, not forced, and its running
residents and local-only snapshots need an operator. The console loop
gains node-holds-panel and node-history-filters, which check the rows of
the holds answer the page received, each row's gates exactly, "Nothing
holds this node" in the panel and the Drain section when no term counts
a row, the filtered History's row count, and that after the Reload and
the toggle the History shows the answer's rows in its order, newest
change first.
The change is confined to the console and the console loop; no API,
controller or store path changes.
The pull request changed after this approval. CodeRabbit approves again after it reviews the current head.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Summary
The node page shows what holds the node and why, and its sandbox list becomes a collapsed, filterable History.
When a drain stalls or
:terminateis refused, the page listed every sandbox that ever named the node, by id, and counted that page by lifecycle. History dominated it, and nothing said which sandboxes held the node or what to do about them. The page now readsGET /v1/nodes/{id}/holdsfrom the server PR and adds a What holds this node panel. It groups the rows by who acts on them next, under the node's drain stage: no drain, a gentle or a forced drain while the node isdraining, or parked once it isdrainedorterminating. The drain moves, uploads and fails a node's sandboxes only while the node isdraining; on a parked node it no longer acts on what still holds it, though restores of sandboxes it already evacuated continue. The groups, in this order:Stills: panel under a forced drain, panel on the drained node, refusal, refusal link to the panel, failed filter, failed during this drain.
Validation
npm testinui/(389 pass; the tests cover the group of every reason under each of the four drain stages, a stopped and a restoring evacuation, rows no gate counts, the sentences per stage, including the Undrain exit and the terminating case, and the refusal's counts) andnpm run build. The console loop'stypecheckandtestpass too.main, the server commit, ui: add forced drain, terminate and a drain section to the node pages #1662 and this commit, on a worker with a hold for every reason: 15 rows, 14 synthetic (including a stopped evacuation, an evacuation restoring onto the node, and created, deleting and quarantined rows) and the loop's seeded running sandbox. The cluster-controller was paused so its sweeps left the synthetic rows alone, and the node's drain phase and mode were set in turn to none, gentle, forced and drained, then terminating.node-holds-panelpassed all 53 of its checks, including each row's Why and its exact Counted by.node-history-filterspassed in every stage: after the Reload the History showed the answer's 20 rows in its order, newest change first; the failed filter keeps 3 of 3 rows; and with Failed during this drain on, it showed the 2 rows that failed after the drain started, in the answer's order.npm run loop -- --video --no-mutateagainst a production build ofmainand then this branch, on the same stack: 0 new error types, 7 resolved (mainhas no Drain section or holds panel).flowssteps were not run, since--no-mutateskips them, including ui: add forced drain, terminate and a drain section to the node pages #1662's two drain flows, which also refuse while a node is draining. The older-API notice was checked only against a substituted answer, not against an older API.Severity and risk
Severity: does not apply. It adds an operator view.
Risk: it touches the node page only. The History now reads the node's sandboxes newest change first, through optional parameters the server PR adds. Against an API without that PR, the History's filters, change times and paging do not work, since the API ignores the parameters and answers its first page in id order. The History shows a notice saying so, and its change times show as "—"; merge and deploy the server PR first. Roll back by reverting the commit.
Stack
Stacked on #1662 (
ac/console-node-drain-panel); merge #1662 first. It also needs #1674 (GET /v1/nodes/{id}/holdsand the history parameters) merged and deployed first. Without it the panel reports that the API does not list holds.AI assistance: Claude Code made the change, ran the validation and drafted this description. A human reviews before merge.