[ac-pplx]

Summary

Operators can now rebuild the system templates, seed and code-runtime, from the Templates page, check what a new release carries before promoting it, and see which spaced guest versions the running sandboxes report, per template. Before, a rebuild needed the admin CLI, and nothing in the console showed spaced versions outside a single sandbox's page.

• Rebuild system templates (administrators of the managing organization only), above Template Releases, walks the seed README's release procedure one explicit step at a time. Nothing is built, created or promoted unless the operator clicks it in this tab:
  • Build seed (POST /admin/api/v1/system-templates/seed/builds), then the page polls it.
  • When seed succeeds, Build code-runtime on seed@<digest> (base: "seed@<digest>"), on that exact seed, then the page polls that. Neither build carries publish tags, so each registers a release named only by its digest.
  • For each new release, Create a test sandbox creates a sandbox from the pinned reference (seed@<digest>, code-runtime@<digest>) through the public create route the Sandboxes page's form uses. The panel links to it: its page shows the spaced version its guest reports, and whether it boots on this cluster. The panel reminds the operator to delete it.
  • Promote to latest… opens a confirmation that reads latest of both templates fresh, and shows what moves from which digest to which. If latest changed since the run started, it says so, and Promote waits for an explicit "Promote anyway". On Promote, the page reads the tags again; if what would move is no longer what the dialog showed, it shows the new moves and asks again. Before the first tag request it saves the promotion record, for both templates: the digests latest named (the rollback target the panel names) and the digests it moves to. That includes a template whose latest already names its new release; only the tag requests leave it out. The first confirmation writes the record and a retry keeps it. If this browser cannot save it, no tag request is sent and the dialog says why.
  • The record is the only promotion state the page stores. Whether the promotion is complete is read from the tags each time the page loads them, and only the newest read's answer is applied, so a slower, older one cannot bring back a state that has since changed. The panel shows success only when both templates' latest names the new releases, says which template still has to move, and says when someone moved latest to a third release. A retry plans only what still has to move, reads the tags again, and asks before overwriting a latest someone else moved. Nothing retries by itself. The tag route has no compare-and-set, so a change made between the page's final read and a tag request is not detected; the dialog says so.
  • When every latest that can move back is on the release the record names as before, as after a rollback with the tag dialog, the panel says latest is back on it and offers Promote again…, with the same confirmation. The tags do not tell a rollback from a tag request that never committed, and the panel says so.
  • Rolling back is assigning latest to the releases it named before, with the existing tag dialog. The success panel lists those releases under "Rollback target", each template's as a full digest with a copy button, and says to record them: the run that holds them expires a day after it started and lives only in this browser. The incomplete-promotion panel and the promote-again dialog show the same list. A template that had no latest before is listed as keeping the new release, because the API does not delete latest.
• The stored run only lets a reload, or another tab, show the run and keep polling it. It is keyed by cluster and user, and kept for at most a day, which is shorter than the builder's four-day job retention. The panel shows when the run started and when it is forgotten, and clears itself then. A status answer applies only to the run and job that asked for it. Polling backs off while the status route fails, and the panel shows that error. Dismiss is always available. A step's button disappears once the run has a job for that step, and a button is disabled while its request is in flight. A new run that this browser cannot save is not started, and the page says so.
• Several tabs. Before every request that changes something, and before every write to the stored run, the page checks that the run is still stored: not dismissed in another tab and not expired. Every write of the stored run, including Dismiss and dropping an expired or unreadable record, takes a lock the browser's tabs share and re-reads the run inside it, so a poll in one tab cannot write back a run that Dismiss removed in another. The lock is the Web Locks API where the browser offers it, which needs a secure context (https, or localhost). Elsewhere it is a short-lived lock in localStorage; its read-then-write is not atomic across tabs, so there two tabs' writes can still interleave in a narrow window.
• An answer is recorded only into a step that is still empty. If two tabs click Build seed, or Create a test sandbox, at the same time, the run keeps the first answer, and the tab whose click lost names the other above the run panel: "Another click also started seed job …; this run follows job …", or "Another click also created test sandbox …; this run keeps …. Delete …". These notices stay until closed, also after Dismiss, as do the notice for an answer that arrives after the run was dismissed and an unknown test-sandbox outcome. A promotion dialog closes if the run, or the digests it would promote, change in another tab, and a confirmation is refused for a run that is no longer the stored one.
• Unknown outcomes. A request that fails with a 5xx or a network error is reported as an unknown outcome, not as one that did nothing. For a build: the builder may have accepted it, an accepted build appears in Template Releases only when it finishes, and building again starts a second job, so wait, refresh Template Releases, and build again only if no new release appears. For a test sandbox: it may have been created, so look for its name (rebuild-check-<template>-<run>) in the Sandboxes list before creating another.
• spaced in running sandboxes, under Template Releases: the running sandboxes per template and the spaced version their guest reports, sampled from the admin sandbox list (its newest rows, at most 5,000, with deleted ones left out). A line says what that covers: "running sandboxes among the newest N listed, created since <the oldest listed row's creation time>", where N and that time come from the list's database rows only. Rows that nodes report without a database row, such as warm-pool slots, are named in a separate sentence. When the list reached its limit, the line says that long-lived sandboxes created earlier are not counted, even when none of the listed rows is a sandbox that is not deleted. A sandbox that reports no version counts as "unknown", and a template with no running sandbox among them does not appear. A failed refresh is shown even when the previous counts remain. A server-side count by template and version would need a backend change.
• Seed README. Its "Spaced Version Rollout" section said sandbox-manager refuses templates whose compatibility family does not match its own, and that host binaries and the seed template must come from the same checkout. Sandbox-manager admits guests whose version is in SUPPORTED_SPACED_GUEST_VERSIONS (crates/libs/spaced/common/src/compatibility.rs) and whose family agrees with that version; host and guest families do not have to match. The section now says so, without copying the range, and links to the guest-compatibility section of docs/versioning.md. It also no longer says the stamp comes from the workspace Cargo.toml: it is the version of the guest binaries the build packages. Its "Build and publish system templates" section now also says that between the two tag moves, or after one fails, seed's latest can name the new release while code-runtime's still names the old one, and that this pair is consistent because each code-runtime release was built on a pinned seed digest.

No service, API or schema changes.

Stack

Stacked on #1255 (templates first on the Templates page), which also rewrites the Template Releases header. Merge #1255 first.

Validation

• npm ci, npm test (68 pass), tsc --noEmit and npm run build in ui/. The new tests are in two files:
  • src/lib/system-template-rebuild.test.mjs: which step is offered; status answers applied only to their run and job; answers accepted only into an empty step, and the extra one reported; no notice for an answer the run recorded, and the notice for an extra or late one; polling backoff; unknown outcomes for a build and for a test sandbox; the promotion plan, its "changed since the run started" and "moved by someone" flags, and the comparison that makes a changed plan ask again; the rollback target captured once and kept by a retry; a record that names both templates, including one already at its target, whose later change reads as moved by someone; complete, incomplete, moved-by-someone and back-on-the-previous-release promotions read from the tags; stored runs ignored for another cluster or user, after expiry, or in any shape the page does not write, including older shapes of the promotion record.
  • src/lib/spaced-counts.test.mjs: the per-template counts, the coverage line, and the 5,000-row limit. That case fails if node-reported rows count toward the limit, or if 5,000 rows do not count as capped. Node-reported rows do not move the "created since" time, and the limit warning stays when no listed row counts.
• A Playwright loop against a local Kind stack with synthetic data, on this branch's Kind space-ui image, at an earlier head of this branch:
  • Explicit steps. "Rebuild system templates" shows a Build seed button and sends nothing; neither does a reload. There is no promote switch. Exactly one seed and one code-runtime build request were sent, both on clicks, and nothing was tagged until Promote.
  • A real seed build. Seed built on the Kind builder in about a minute. A reload while it built sent nothing, and polling resumed. The next step was the button "Build code-runtime on seed@".
  • code-runtime is stubbed. On this Kind stack the API's bundled recipes pin a guest kernel (vmlinux-6.1.155-bpf-modsig) that the Kind builder and nodes do not have installed. A real code-runtime build fails when its build sandbox starts, and a sandbox from a rebuilt seed does not boot there. So code-runtime's build, its release and that release's tag request were answered by the loop. The page asked for code-runtime with base: "seed@<the new seed digest>" and no publish tags.
  • Two tabs. A second tab showed the same run, offered no step the run already had a job for, and sent nothing.
  • No false notices. After normal Build seed, Build code-runtime and Create a test sandbox clicks, the page showed no "dismissed" or "another click" notice.
  • Two tabs clicking at once (stubbed answers). Two tabs that opened the rebuild together got one run. Both clicked Build seed at the same time: both requests were sent, the run kept the first job, and the other job was named above the run panel of the tab whose click lost. Both then clicked Create a test sandbox: the run kept one sandbox, and the other was named, with "Delete", the same way. Each extra appeared once. After Dismiss in that tab, the run was gone and the sandbox to delete was still shown.
  • Without the Web Locks API (stubbed answers). In a tab with navigator.locks removed, a build was recorded and polled to success with no notice, the localStorage lock was released after each write, and Dismiss forgot the run.
  • Dismiss in another tab. While one tab polled a build, Dismiss in the other tab cleared the run in the polling tab, which then sent no further status request. The run stayed dismissed after a reload.
  • Test sandboxes. From the real run's new seed, "Create a test sandbox" requested seed@<new digest>. That sandbox could not boot on Kind, as above, and was deleted. To show the page a test sandbox links to, a second run's seed "build" reported the current seed release. Its test sandbox, created for real from seed@sha256:…, started, and its page showed spaced 0.8.0. The button disappeared once the run had a test sandbox.
  • Promotion. "Promote to latest…" showed seed moving from its latest at the start of the run. With the dialog open, seed's latest was moved, through the tag route, to another seed release built by a first real run. Promote then sent nothing. The dialog showed the new moves, warned that seed's latest had changed since the run started, and kept Promote disabled until "Promote anyway" was checked.
  • Interrupted promotion. The seed tag request (real) committed. The code-runtime one (stubbed) was held, and the page reloaded. The panel said code-runtime still had to move, not that the promotion had succeeded. The stored record held the rollback target and nothing read back. A retry planned only code-runtime. With its tag request answered 500, the panel still read the promotion as incomplete. A second retry completed it. The panel then showed success as read from the tags, with the rollback target and without the failed attempt's note, and a reload showed the same.
  • Rollback. The tag dialog moved seed back to the rollback target. The panel said latest was back on the release it named before, and that code-runtime, which had no latest before, keeps the new release. It offered Promote again, not Retry. Promote again opened the same confirmation, with seed moving forward and nothing extra to acknowledge, and sent nothing (it was cancelled). Moving seed on to the release it had before the run made the panel say someone had moved latest.
  • seed already at its target (stubbed seed answer). A run whose seed "build" reported the release seed's latest already named planned only code-runtime's tag request. The record named both templates, seed with its latest as read. After seed's latest was moved to another release, the panel read the promotion as incomplete, with seed moved by someone, not as success. A retry planned seed and asked before overwriting it. seed's latest was then put back.
  • A wedged status route (stubbed). A build request answered 500 was reported as an unknown outcome ("the builder may have accepted it"), with "Build seed again". A status route answering 503 every time showed its error and when it would retry, and Dismiss cleared the run.
  • Another cluster. A run stored under another cluster's key, and one under this page's key naming another cluster, both showed nothing and sent nothing.
  • spaced in running sandboxes listed the running seed sandboxes by version, with "Running sandboxes among the newest 10 listed, created since …".
• The full loop (162 steps) on that head's image found four error types main's baseline lacks. "owner row listed" is the user-page fixture this stack no longer has. "callback is not a function", in three Templates steps, came from the loop itself: an earlier step wraps navigator.locks.request and handled only its three-argument form, while the panel uses the two-argument form and now also takes the lock on load. With the wrapper fixed, those steps and the one that installs it reran with no new error type. A refused connection in one user-page step did not recur when it reran.
• A Playwright run on the same Kind stack compared production builds of an earlier head of this branch and of this head on the Templates page, with no build and no tag move. A synthetic rebuild record, with both builds succeeded and a promotion confirmed two minutes earlier, was written under the page's own localStorage key and removed afterwards. The release list was answered synthetically with latest on the promoted releases, and other methods on that route were refused. The earlier head's success panel named the rollback target as short hashes in a sentence. This head lists each template's full digest with a copy button and says to record them, because this browser keeps them only until the run expires and no other browser has them.
• Not validated: a real code-runtime build, and booting a rebuilt seed, which this Kind stack cannot do (above). The release list's newest-answer rule was not exercised either: no loop step delays an older read past a newer one. The rollback list in the incomplete-promotion panel and the promote-again dialog was not captured.

Visual confirmation

On a local Kind stack with synthetic data. The steps, after a real seed build (the next step is an explicit button):

The promotion confirmation after seed's latest moved while it was open: Promote sent nothing and asks again:

After a reload that interrupted the promotion between its two tag requests, the panel reads the tags and says what still has to move:

In the two pairs below, before is an earlier head of this branch and after is this head, from a synthetic rebuild record and release list with placeholder digests.

The success panel after a promotion: the rollback target is now each template's full digest, with a copy button and a note to record it:

Before	After

The same panel on the Templates page:

Before	After

After the tag dialog moved seed back to the rollback target, the panel says so and offers Promote again:

A promotion whose seed was already at its target records seed too; after someone moved seed's latest, the panel does not read it as complete:

Two tabs that clicked Build seed, and then Create a test sandbox, at the same time (stubbed answers): the run keeps one of each, and the tab whose clicks lost names the extra ones above the run panel:

The same notices after Dismiss in that tab:

A test sandbox created from a pinned seed digest; its page shows the spaced version:

The per-template spaced table, with its coverage line:

Severity and risk

Severity: medium. Rebuilding and promoting the system templates is how a guest or spaced change reaches new sandboxes, and the console could neither do it nor show which spaced versions sandboxes run.

Risk: low. The page uses existing admin and public routes with their own permission checks (the system build and status routes, the tag route, the admin sandbox list, and the public sandbox create route). It changes nothing unless the operator clicks a step, and moves latest only through the same tag route the dialog uses, after a confirmation. Each build costs builder time, and each test sandbox runs until the operator deletes it. Roll back by reverting the commit.

Not covered

• The page does not cancel a build: the API has no cancel route.
• The page does not wait for or judge a test sandbox: the operator opens its page to see whether it booted and which spaced version it reports, and deletes it afterwards. A release can build and publish yet fail to boot on the cluster's nodes, as on the Kind stack used here (see Validation).
• The tag route has no compare-and-set. The page reads the tags again when the operator confirms, but a change made between that read and a tag request is not detected, and the request overwrites it. Refusing it would need an API change.

AI assistance: Claude Code wrote the code, the tests and this description, and ran the validation; a maintainer reviews it.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[zibi-pplx]

The flow is careful about interruption: completion is read back from fresh tags, an incomplete promotion is detected, and Retry asks first when a tag has diverged. That leaves two questions about what promotion has to guarantee.

• [medium] Does promoting seed and code-runtime need to be atomic?
  ui/src/app/dashboard/templates/system-template-rebuild.tsx:461 moves latest with one PUT /v1/template-releases/{id}/tag per template, in sequence. Between the two writes, or after a failure the operator never retries, seed's latest names the new release and code-runtime's names the old one. The re-read before writing narrows the race, but it can't stop a concurrent writer, because the tag route has no compare-and-set. If a mixed pair is acceptable (each template stands alone), a sentence in the seed README's release section saying so would settle it. If not, the fix belongs on the server: a route that moves both tags in one transaction, or an expected-current-release precondition on the tag route. Either way, a tracking issue would help.

• [low] The rollback target lives only in this browser.
  ui/src/lib/system-template-rebuild.ts:15,35: Promotion.previous is stored in localStorage for this cluster and user, and is dropped after RUN_MAX_AGE_MS (one day). Another operator, another browser, or the same operator a day later can't see what latest named before. The admin audit log records the release ID each tag request moved to, but not the one it replaced. Template Releases still allows recovery by hand. Suggestion: show the previous digests prominently and make them copyable in the success panel, so the operator can record them. Should promotions eventually be recorded server-side?

• [low] Would an idempotency key on the build route simplify this?
  system-template-rebuild.tsx:351: POST /v1/system-templates/{template}/builds takes no Idempotency-Key, so two tabs that both click Build seed start two builder jobs, and the page has to name the extra one. If that route accepted the run ID as an idempotency key, the cross-tab "extra job" handling could be removed. This is a server change and fits a tracking issue rather than this PR.

Withdrawing this approval for now. Any feedback above still stands.

[the-gigi-pplx]

Review — 2 findings

Low

1. ui/src/lib/template-format.ts:7: shortHash cuts a BLAKE3 digest down to 5 hex characters, and new releases are BLAKE3 by default.
   shortHash strips only a sha256: prefix and then keeps 12 characters, so blake3:<64 hex> comes out as blake3:01234. New rootfs registrations default to BLAKE3 (DEFAULT_PRODUCTION_ALGORITHM in crates/libs/template/mgr/src/content_hash.rs:37), and this PR's own tests use blake3: for the new builds. So the promotion dialog's "latest now → moves to" columns (digest() at system-template-rebuild.tsx:82), the "Build code-runtime on seed@…" button (:545), and the "latest is back on …" and "incomplete" texts show the new release as 5 hex characters next to 12 for an older SHA-256 one. That's the dialog whose job is to show which digest moves where. The rollback list prints full digests, so nothing is lost. It's still the hardest place in the flow to read a short prefix. The Hash and releases columns had the same truncation before this PR, but the helper is now shared, so one fix covers all of them: strip any <alg>: prefix (for example hash.replace(/^[a-z0-9]+:/, '')), or keep the algorithm and 12 hex characters after it.

2. ui/src/app/dashboard/templates/system-template-rebuild.tsx:71: without the Web Locks API, a browser whose storage refuses writes makes exclusive throw before the page can say why.
   The fallback calls tryAcquireViewRefreshLock, which writes the lock with an unguarded localStorage.setItem (ui/src/lib/view-refresh.ts:65). When storage is full or blocked, that write throws inside exclusive. So persist/store, which catch the same failure and show "could not save the rebuild" or "could not save the promotion record", never run. start() then rejects inside withBusy and the click does nothing visible. In confirm(), running: true is already set (:414), so the dialog keeps spinning with no message. It fails safe, because no tag request goes out. It does contradict the "the dialog says why" behavior the description promises, and it only shows up on http deployments, where the fallback is in use. A fix that keeps the safety: catch a throw from the fallback acquisition and run work() without the lock. Its own write then fails the same way and reports through the existing messages. Retrying the lock instead would just spin, because the storage that refused the lock refuses every write.

Verified against 9ec59a1c8130e433b3f92c7f06c8ba54e0bd5ce9, reviewed as the diff on top of 7f0164ae0233 (the base PR's head). Neither finding repeats an earlier review point. The atomicity and rollback-target questions raised earlier are answered by the README paragraph and the full-digest rollback list in this head.

What I ran: npm ci, npm test (68 pass), tsc --noEmit and npm run build in ui/ on this head. I removed the movedForward assignment in promotionState and two tests failed, so the complete-versus-back-on-previous split is covered. I also called shortHash on a blake3: and a sha256: digest to confirm the 5-versus-12 output above.

What I traced in the API, not just the UI:

• The system build route treats an empty publish_tags as no tags (start_system_template_build), so a build doesn't move latest.
• base: "seed@<digest>" parses as an artifact digest (TemplateReference), the same shape the seed README's CLI procedure uses.
• The tag route takes {tag} and has no compare-and-set.
• The admin sandbox list clamps limit to 5,000, newest first, deleted rows included, so truncated and the coverage line match what the server returns.
• The README's spaced paragraph matches the code: rootfs-builder.sh stamps the version build-mgr reads from the guest binaries, and sbx-mgr admits guests through validate_spaced_guest_compatibility against SUPPORTED_SPACED_GUEST_VERSIONS. The docs/versioning.md#guest-compatibility-in-08 anchor exists.

What I didn't check: anything in a browser. I didn't run the Playwright loops, the two-tab races, or a real code-runtime build.

[the-gigi-pplx]

Approved. 2 non-blocking findings in the review comment above.