[ac-pplx]

Summary

This shortens a cold Kind bring-up by moving two Tilt steps off the critical path. Each step was held back for reasons unrelated to its own work. The charts and the seed-publish Job's gates do not change. space-internal-api stops waiting for the console step, and the bring-up gains one wait for it (below). build-seed.sh now requires flock.

• The seed starts early. Tilt holds a custom_build until its resource's resource_deps are ready, and space-seed-publish waits for space-internal-api. So the privileged rootfs build in build-seed.sh could start only once internal-api was Ready, even though it needs only the Rust artifact image's guest binaries. A new seed-artifacts local_resource (allow_parallel = True, after rust-artifacts) now runs it beside the space-api deploy chain. This mirrors how rust-artifacts was already pulled forward.
• The seed is never stale. The space-seed-publish custom_build still runs build-seed.sh before its docker build. Both steps use one SEED_BUILD command, which resolves SPACE_SEED_OUTPUT_DIR the same way. The image build finds the seed up to date or rebuilds it. The publish Job keeps its gate on space-internal-api and space-object-store-bucket-init.
• The seed directory is locked. build-seed.sh takes an exclusive file lock in its output directory before it reads any input, the recipe included, and holds it until the outputs and stamp are written. A build that waited therefore reads its inputs as they are once it runs, and the later caller finds the seed up to date. The image build reads the directory through build-seed.sh --read-lock -- <command>, which holds the lock shared, so no seed build rewrites the directory while the docker build reads it. A seed build that takes the lock after the check and before the read finishes first, and the image is built from its output. The script now refuses to run without flock, rather than running two privileged builds unguarded. The cache-only build writes nothing there and takes no lock.
• The console step runs in parallel. provision-local-console only waits for the API rollout and execs space-db-migrator set-user-password in the API pod, so it is now allow_parallel. As a serial resource it waited for Tilt to go idle and blocked every other update while it ran. That is why admin-api, internal-api and the credential authority were all applied the instant it finished.
• A failed console step fails fast. The bring-up now waits for provision-local-console to succeed before the node-publication wait, which signs in as root@local. A failed console step ends the bring-up with its log, like any failed Tilt resource; before, it surfaced as a "nodes not published" timeout. wait-node-publication.sh also says when it timed out on rejected logins.
• Console dependencies. space-internal-api no longer lists the console step in resource_deps. Nothing it runs signs in as root@local, and its build does not link the session-login code. Its callers present ServiceAccount tokens or the internal token; the seed publish Job uses a projected token. The only password logins during a bring-up are the node-publication wait, which now runs after the console step succeeds, and the smokes, which run after the bring-up returns. space-ui keeps the dependency; see "Not covered".

Validation

Measured at 0afac29d (the scheduling change; later commits change only the locking and the failure path) on one 48-CPU box with scripts/benchmarks/kind-bringup-timeline.py, using SPACE_KIND_TELEMETRY_MODE=otel as the Kind CI shards do. Each run used a fresh cluster and an empty SPACE_SEED_OUTPUT_DIR, with Rust and image caches warm. Times are seconds from the start of smoke-kind-e2e.sh. "Applied" is the time Tilt applied admin-api, internal-api and the authority. "Ready" lists the time each was Ready, in that order.

run	smoke PASS	seed rootfs	console	applied	Ready	seed Job
main 1	397.4	155–347	128–146	146.4	153/154/148	367–386
branch 1	203.1	103–135	137–148	137.2	143/144/138	166–186
main 2	239.6	161–185	135–152	151.6	157/158/152	207–227
branch 2	189.7	103–130	128–148	128.1	134/135/129	157–178
main 3	232.5	156–179	127–147	147.1	153/154/148	198–218
branch 3	191.7	102–128	128–146	127.5	133/135/129	156–178

Pair 1 is not a fair comparison. Main 1's rootfs spent 155 s fetching apt indexes. Branch 1 also rebuilt the UI image for its new commit. In pairs 2 and 3 the branch is 50 s and 41 s faster. On the branch the rootfs finishes before internal-api is Ready, so the seed-publish image build starts as soon as internal-api is Ready.

Seed freshness: on the live branch stack, I changed a string compiled into spaced-svc and ran up.sh again. seed-artifacts and the seed-publish image build started together. The image build logged that the lock was held, waited, and then found the seed up to date. These all match sha256:e0460394…:

• build-seed.sh's manifest.json content_hash;
• the sha256 of the seed directory's rootfs;
• the manifest.json and the sha256 of the rootfs inside the Job's image;
• the release the Job registered.

The image's rootfs contains the new string, and the staged spaced-svc matches the artifact image's binary. Reverting the change and running up.sh again also passed smoke-kind-e2e.sh.

At this head, a fresh-cluster branch run with an empty seed directory passed smoke-kind-e2e.sh at 198.3 s. The seed rootfs ran at 108–135 s, and the image build ran under --read-lock. The freshness check above passed again through the new path: on each guest change, the image build logged that it was waiting on the lock.

Regression tests, each failing on the commit before its fix:

• scripts/tests/test-kind-build-seed-lock.sh runs the real script against a scratch checkout with docker stubbed. It shows that a build that waited on the lock builds with the recipe saved during the wait, that a build waits while a --read-lock command reads the directory, and that the script exits nonzero without flock.
• Case 14c in test-kind-stale-tilt-recovery.sh: a failed console step ends the bring-up with its log before the node-publication wait.

Live check of the console failure, using a scratch edit that is not committed: provision-local-console sleeps 90 s, then sets the password of a user that does not exist.

• At 0afac29d, with SPACE_BOOTSTRAP_TIMEOUT=300, the bring-up failed after 466 s with "nodes not published as ready within 300s".
• At this head, with the default 1800 s timeout, it failed after 260 s with the console's log ("user does not exist").

Against the live stack, wait-node-publication.sh with a wrong password now fails with "root@local login still rejected (HTTP 401)".

Earlier, at 0afac29d, on the branch stack:

• a warm up.sh rerun reused the running Tilt (same pid, 11 s);
• smoke-kind-e2e.sh passed;
• SKIP_BOOTSTRAP=1 bash scripts/tests/smoke-kind-typed.sh all 1/4 passed 17 of 17 (ECW 10, native 7), with the shard step's admin token and cold-receipt settings. The GHCR image mirror was not used locally.

At this head, a warm up.sh rerun reused Tilt (11 s) and smoke-kind-e2e.sh passed. test_kind_prune_superseded_images.py, test-kind-localhost-ports.sh, test-kind-rust-input-key.sh, test-kind-up-restart.sh and test-kind-stale-tilt-recovery.sh (all cases) pass. The 4 Kind smoke shards run in CI on this head.

Severity and risk

Severity: does not apply. This is a local and CI bring-up speedup.

Risk: the main new behavior is two concurrent callers of build-seed.sh, plus the image build reading what they write. The lock serializes every writer and keeps them out while the image build reads. The custom_build still rebuilds the seed if it is stale. If seed-artifacts fails, the bring-up fails the same way a failing seed-publish build does today. Kind hosts now need flock, which util-linux provides on the apt-based Linux hosts Kind already requires. Revert this PR to roll back.

Merge order: #1017 rewrites the same seed-publish block. Whichever lands second keeps SEED_BUILD, deps = SEED_DEPS + [...] and #1017's image call, with --read-lock wrapping only the image build, not the node load.

Not covered

• space-ui still depends on provision-local-console. The UI pod does not sign in at start, so the dependency looks removable. It also gates the UI image build, but that build is off the critical path here.
• The Tiltfile itself takes about 70 s to load, before any resource starts. This PR does not address it.
• In scripts/tests/lib/bootstrap-kind-stack.sh, assignments such as status="$(… | awk …)" run under set -euo pipefail. They end the bring-up silently when tilt get uiresources fails. This affects the console wait added here and two older lines on main, in __die_on_failed_builds and the quiet wait.

AI assistance: Claude Code drafted the code changes, the measurements and this description.

[github-actions]

Thanks for opening this pull request! A SPACE maintainer will take a look.

CONTRIBUTING.md explains how we review outside contributions.

[zibi-pplx]

The scheduling change looks right. It needs a rebase, and some of its evidence predates two merges.

• [low] The rebase onto #1017 and #1168 should refresh the claims and timings
  Tiltfile (seed-publish custom_build, space-internal-api deps); scripts/tests/lib/bootstrap-kind-stack.sh
  The PR conflicts with main. The merge-order note covers #1017, but #1168 also changed the Tiltfile, bootstrap-kind-stack.sh, wait-node-publication.sh and test-kind-stale-tilt-recovery.sh. After #1168, the description's statement that "the only password logins during a bring-up are the node-publication wait … and the smokes" no longer holds. On the Direct lane, register-direct-nodes signs in as root@local. It stays safe because that resource lists provision-local-console in its own resource_deps (Tiltfile:929 on main). Dropping the console step from space-internal-api's deps therefore doesn't affect it. The bring-up timings and "the 4 Kind smoke shards" were measured before #1017 changed image loading and before CI moved to six shards.
  Suggestion: after the rebase, keep register-direct-nodes' explicit dependency and update that sentence. Then re-run kind-bringup-timeline.py on both substrates so the before/after table reflects current main.

Withdrawing this approval for now. Any feedback above still stands.