Skip to content

kind: authenticate node-api and build-manager callers by TokenReview - #484

Merged
ac-pplx merged 1 commit into
ac/kind-fidelity-node-runtimefrom
ac/kind-fidelity-internal-auth
Oct 15, 2026
Merged

ac-pplx merged 1 commit into
ac/kind-fidelity-node-runtimefrom
ac/kind-fidelity-internal-auth

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Stack: Part of the Kind fidelity stack, bottom to top: #485 → #480 → #472 → #484 → #507. Based on #472 (ac/kind-fidelity-node-runtime).
This PR is one commit, and that commit is the only thing to review.

What Kind now does like deployments

Kind's node-api (worker and builder) and build-manager accepted only the shared static bearer tokens, so the Kubernetes TokenReview path those listeners run in deployments never ran locally. They now authenticate every caller by TokenReview of its projected ServiceAccount token, with the static fallback off, as deployed services do.

  • node-api, both listeners (SPACE_K8S_TOKEN_AUTH_ENABLED=true, SPACE_K8S_TOKEN_AUTH_STATIC_FALLBACK_ENABLED=false). It uses the code's audiences: space-node-api for WorkerApi, BuilderApi and NodeInfo, and space-node-api-internal for WorkerApiInternal (DEFAULT_NODE_API_AUDIENCE, DEFAULT_NODE_API_INTERNAL_AUDIENCE, auth/k8s.rs). The per-surface allowlists come from the callers the code shows:

    Surface Admits
    worker WorkerApi, NodeInfo space-api (shared and admin api), space-cluster-controller, space-data-plane (guest bridge), space-internal-api, both node-api accounts (template transfers, connect_remote in transfer.rs)
    worker WorkerApiInternal space-api, space-cluster-controller
    worker trusted wait-ready and trusted policy lists space-api, space-cluster-controller
    worker guest bridge space-data-plane
    builder BuilderApi, NodeInfo space-api, space-cluster-controller, both node-api accounts
    builder trusted policy (the api's TemplateController template and pool calls); BuildTemplate (template-build, system and tenant lists) space-api
    peer template transfers on both roles (SPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS) space-api, space-node-api, space-builder-node-api
    build-manager (allowed, system and tenant lists) space-builder-node-api
  • Peer template transfers. Authorize peer-to-peer template transfers #312 gates PullTemplate and NodeExportTemplate behind a separate transfer list. With Kubernetes auth on, only listed accounts get the grant, and an empty list refuses every transfer (AuthPolicy::authenticated_caller; require_template_transfer_authority and prepare_template_export in auth/template_transfer.rs). Both roles list:

    • space-api, the account that starts a pull with PullTemplate on the target;
    • both node-api accounts, because the target's node-api, of either role, calls the source's NodeExportTemplate with its own projected space-node-api token (SPACE_NODE_API_TOKEN_PATH, 600 s).

    Both node-api accounts were already in the surface allowlists, and node-api already mounted that token, so neither changed. One list covers both RPCs, so space-api may also request an export. The guest bridge is always excluded.

  • Who starts a transfer in Kind. No product path does. The only in-tree caller of PullTemplate is admin-api's pull_node_template. It serves only the direct substrate: require_admin_node_registry answers 501 on the Kubernetes substrate, in Kind as in any Kubernetes deployment. So in Kind a pull is an operator's direct PullTemplate call to the target node-api with a space-api token. The README says so next to the allowlist table, and docs/configuration.md says the route is direct-substrate only.

  • build-manager reviews the builder node-api's token (audience space-build-manager). It has no static fallback (auth/build.rs, auth/build/k8s.rs).

  • API access for the reviewers. node-api and build-manager mount their own ServiceAccount's API credentials (automountServiceAccountToken: true). A per-release ClusterRole limits each account to tokenreviews/create. They read the namespace their allowlists qualify from that mount (auth_namespace), as a release binary does.

  • No auth test hook. The SPACE_K8S_TOKEN_AUTH_NAMESPACE test hook is removed from the api and internal-api charts, so no Kind auth path depends on it.

  • Caller tokens. Every caller mounts projected tokens under /var/run/space-auth and names them with SPACE_NODE_API_TOKEN_PATH, SPACE_NODE_API_INTERNAL_TOKEN_PATH and SPACE_BUILD_MANAGER_TOKEN_PATH:

    • api and admin-api: normal and internal tokens.
    • cluster-controller: normal and internal tokens.
    • data-plane and internal-api: normal token.
    • worker and builder node-api: normal token, for transfers.
    • builder node-api: build-manager token.

    An explicit path makes a missing token an error instead of a silent fall back to the static token.

  • Lifetimes, as deployed.

    • 600 s for every token presented to node-api, and for the node services' ClusterApi and credential-manager tokens.
    • 7200 s for the builder's build-manager token.

    The kubelet rotates each one at 80% of its lifetime.

  • Static tokens stay mounted. SPACE_INTERNAL_TOKEN and SPACE_INTERNAL_CONTROL_TOKEN are still mounted because the binaries require them at startup, but they admit nobody. node-ctl's own auth is unchanged.

  • Feature list. pplx-space-build-mgr-svc/k8s is added to Kind's cook and build feature lists (docker/Dockerfile.rust-build-kind), matching Dockerfile.rust-build-common.

  • Manual wiring removed. scripts/k8s/projected-tokens goes: the charts now do what it wired by hand, and its 01-apply-auth.sh would turn the static fallback back on and set the namespace hook. Its guidance moves:

    • policy authorization goes to docs/configuration.md ("Policy authorization");
    • Authorize peer-to-peer template transfers #312's template-transfer paragraph goes to docs/configuration.md ("Template transfer authority"): which two callers need the grant, that both also need surface admission, and that an empty list disables transfers with a startup warning;
    • the note on Kind apiserver timeouts during parallel builds (SPACE_TILT_MAX_PARALLEL_UPDATES) goes to the Kind README ("Kind apiserver timeouts during parallel builds").
  • Comments cite symbols. The chart comments this PR adds cite the code by symbol, not by line number. The rebase onto Authorize peer-to-peer template transfers #312 had already moved several of the cited lines.

  • Namespace hook comment. The comment on SPACE_K8S_TOKEN_AUTH_NAMESPACE in auth/k8s.rs no longer says Kind uses the override. This is a comment-only change.

Code paths this exercises in every Kind run:

  • NodeApiAuthLayer in Kubernetes mode, including the per-surface policy, wait-ready, policy-mutation and guest-bridge authority derivation (auth/k8s.rs).
  • BuildTokenReviewer on the builder gateway and on build-manager.
  • Projected-token reads and rotation in every node-api and build-manager client (bearer_token.rs, build/mgr/client).
  • The in-cluster kube client for TokenReview on a host-network pod.

The template-transfer grant and node-api's outbound peer authentication run whenever a pull is started, which in Kind means a direct PullTemplate call. This round's validation ran them that way (below).

The smoke harness and the space-node-debug CLIs call neither surface: the harness drives the public API, and the shell's CLIs use node-local Unix sockets. For an operator calling build-manager directly with space-buildmgr, the README documents the deployed way: mint a short-lived token for an admitted account with the build-manager audience, and give the CLI that token and the internal RPC CA. A job lives only in the node-local SQLite store of the build-manager that ran it, and the CLI's Service is internalTrafficPolicy: Local, so the recipe asks each builder in turn until one knows the job.

What stays different, and why

README "Differences from the EKS deployment modules" gains one row. Kind binaries are built with test-hooks, so smokes can shrink timers. No chart now sets a hook for authentication, and the TokenReview listeners read their namespace from the ServiceAccount mount, as release binaries do. No existing row described this area, so none is resolved. The transfer wiring adds no row: the route that starts a pull is unavailable on any Kubernetes substrate, not only in Kind. Two things are deliberately unchanged:

  • Control-plane ClusterApi tokens keep 3600 s. The deployed lifetimes given for this work cover the node-side tokens and the build-manager token.
  • No operator ServiceAccount for direct build-manager calls. Kind admits only the builder node-api, and operator builds go through the public template build API.

#472's "Release binaries read no test hooks" row is in this tree. This commit removes the TokenReview namespace from its list of hooks the charts set, so it names no hook a chart no longer sets; this PR's own "Release binaries, built without test-hooks" row sits beside it.

Product findings

1. A rejected build TokenReview is reported as retryable.

  • What happens. With build-manager's TokenReview on, a token for another audience is refused, but with UNAVAILABLE: build token review failed rather than UNAUTHENTICATED. The same applies to the builder node-api's BuildTemplate gateway. The probe below shows it again this round.
  • Root cause. BuildTokenReviewer treats any TokenReview error string as an outage. The apiserver puts the reason in error on every ordinary rejection (forged, expired, wrong audience, with authenticated: false). So bad tokens are reported as retryable, and counted as token_review_unavailable, while node-api's reviewer reports the same answer as UNAUTHENTICATED. The existing unit fixture modelled the rejection without an error, which is why it passed.
  • Fix branch. ac/kind-fidelity-internal-auth-fix-build-review-rejection has a single commit that checks authenticated first and adds a test with the apiserver's response shape; the test fails without the fix. It is pushed as a branch only, with no separate PR.
  • This PR does not depend on it. The request is still refused, and no scenario fails.

2. A peer export of a received template never becomes ready. This is new this round, and not about auth.

  • What happens. NodeExportTemplate of seed from any node, worker or builder, answers UNAVAILABLE: compressed rootfs not ready …; retry after background generation completes, and keeps answering that. Retried every 15 s for 7.5 minutes, it never succeeded.
  • Cause. The export needs a compressed rootfs. When it is missing, template-manager starts background generation in the template's digest directory. A template that arrived by btrfs receive (seed here) lives in a read-only received subvolume (ro=true, Flags: readonly) with no rootfs.ext4.zst. So generation fails every time with Read-only file system (os error 30), which template-manager logs as a warning, and the caller sees only the retryable UNAVAILABLE.
  • Scope. Templates built or imported on the node have a compressed rootfs and transfer fine (below). The failure comes after both auth checks and does not depend on this PR's settings. By the code path it is PRE-EXISTING, but I did not reproduce it on main.
  • Not fixed here. It is a template-manager change, outside this Kind-only PR.

Validation

This round

What was validated. Commit 4f8a766, based on main at 8dda85a. The pushed head d274012 is that commit rebased onto main at 6b427da. git range-diff shows the patch unchanged. The three upstream commits it gained (#477 operator egress profiles, and two docs-site commits) touch none of this PR's files except docs/configuration.md, where #477 renamed a different row of the same table. None touches node-api auth. CI's Kind Pull CI also passed all four smoke shards on 672feac, an earlier push of this round with identical rendered manifests.

Settings. One fresh cluster, brought up from 4f8a766 with CI's settings: SPACE_KIND_TELEMETRY_MODE=otel, DATA_PLANE_URL=http://127.0.0.1:8090, and the admin token file and cold-receipts directory created the way kind-ci.yml creates them. Before the suite, the health gate required: Tilt quiet, no ReplicaSet younger than 180 s, every node Ready and published (wait-node-publication.sh), then a 120 s settle.

Check Result
Existing Kind chart checks (test_kind_node_service_env.py and five other Kind chart tests, plus test_env_test_hook_boundary.py), on 672feac 202 passed
helm template of node-api, both roles: 672feac against the final tree byte-identical (this round's later edits are comments and docs)
Bring-up and smoke-kind-e2e.sh (cluster create → e2e round trip) pass, 523 s
smoke-kind-typed.sh all, unsharded, SKIP_BOOTSTRAP=1, after the gate ECW 78/78 pass, native 24/24 pass, exit 0, 3060 s. native_contract.command_replay_credentials is ignored by its own declaration, as on main
admin-api POST /admin/api/v1/nodes/{id}/template-pulls 501: "direct node registry is only available for the direct/self-managed substrate"
Direct PullTemplate with a space-api token, template transfer-probe (built for this check through the admin template build API, cached only on one builder) builder → worker: 120,632,144 bytes; builder → builder: 120,632,144 bytes; worker → worker: 120,632,144 bytes. The admin node-template inventory then listed it on all four nodes
Direct PullTemplate of seed (worker → worker, builder → worker, builder → builder) both auth legs passed and the source's template-manager answered; the export then failed UNAVAILABLE (product finding 2)
Refusal and transfer-authority probe as expected, every row (table below)

Each successful pull exercised, on the real path:

  • the target's transfer grant for space-api;
  • the target node-api's ClusterApi registry lookup of the source;
  • the source's transfer grant for the target's own account: space-node-api for a worker target, space-builder-node-api for a builder target;
  • template-manager's check of the forwarded grant.

Provenance. Every pod's image ID was recorded after the gate and again after the last check: 88 pods both times, and the same set of image IDs.

  • node-api (both roles), build-manager and the builder gateway report version 4f8a76654bdd, with mode=k8s_token_review or mode=k8s and static_fallback_enabled=false.
  • All four node-api pods carry SPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS=space-api,space-node-api,space-builder-node-api.
  • No node-api logged the empty-transfer-list warning.
  • No pod sets SPACE_K8S_TOKEN_AUTH_NAMESPACE.

Probe. Each call went from the host to a node's host_ip:9800, or through a port-forward to build-manager. It used grpcurl, the cluster's internal RPC CA, the in-repo protos, and tokens from kubectl create token. space-auth-probe is a throwaway ServiceAccount on no list, deleted afterwards. "Admitted, not transfer-listed" means the account passes the surface allowlist but is not on the transfer list.

Surface / credential Result
worker NodeInfo, static SPACE_INTERNAL_TOKEN Unauthenticated: invalid token
worker NodeInfo, space-api aud space-cluster-api Unauthenticated: token validation failed
worker NodeInfo, unlisted space-auth-probe PermissionDenied: service account system:serviceaccount:space:space-auth-probe is not allowed for this node-api surface
worker NodeInfo, space-api (control) OK: { "nodeId": "space-local-worker", "schedulingStatus": "NODE_SCHEDULING_STATUS_READY", "schedulingReason": "ready s
worker ListSandboxes, space-data-plane (guest bridge) PermissionDenied: guest-bridge service account is not allowed for this node-api method
worker Internal WakeSandbox, static control token Unauthenticated: invalid token
worker Internal WakeSandbox, space-api aud space-node-api Unauthenticated: token validation failed
worker Internal WakeSandbox, space-data-plane internal aud PermissionDenied: service account system:serviceaccount:space:space-data-plane is not allowed for this node-api surface
worker Internal WakeSandbox, space-api internal aud (control) InvalidArgument: InternalWakeSandbox.sandbox_id must be a full UUID
builder NodeInfo, static token Unauthenticated: invalid token
builder NodeInfo, space-data-plane PermissionDenied: service account system:serviceaccount:space:space-data-plane is not allowed for this node-api surface
builder BuildTemplate, static token Unauthenticated: invalid token
builder BuildTemplate, space-cluster-controller PermissionDenied: caller may not submit builds
worker NodeExportTemplate, static token Unauthenticated: invalid token
worker NodeExportTemplate, space-node-api aud space-cluster-api Unauthenticated: token validation failed
worker NodeExportTemplate, unlisted space-auth-probe PermissionDenied: service account system:serviceaccount:space:space-auth-probe is not allowed for this node-api surface
worker NodeExportTemplate, space-cluster-controller (admitted, not transfer-listed) PermissionDenied: template transfer requires artifact-transfer authority
worker NodeExportTemplate, space-internal-api (admitted, not transfer-listed) PermissionDenied: template transfer requires artifact-transfer authority
worker NodeExportTemplate, space-data-plane (guest bridge) PermissionDenied: guest-bridge service account is not allowed for this node-api method
worker PullTemplate, space-cluster-controller (admitted, not transfer-listed) PermissionDenied: template transfer requires artifact-transfer authority
worker NodeExportTemplate, space-node-api (listed, control) NotFound: template release not found
worker NodeExportTemplate, space-builder-node-api (listed, control) NotFound: template release not found
worker NodeExportTemplate, space-api (listed, control) NotFound: template release not found
worker PullTemplate, space-api (listed, control) Unimplemented: auto-resolve not yet supported, provide sources
builder NodeExportTemplate, static token Unauthenticated: invalid token
builder NodeExportTemplate, space-node-api aud space-cluster-api Unauthenticated: token validation failed
builder NodeExportTemplate, unlisted space-auth-probe PermissionDenied: service account system:serviceaccount:space:space-auth-probe is not allowed for this node-api surface
builder NodeExportTemplate, space-cluster-controller (admitted, not transfer-listed) PermissionDenied: template transfer requires artifact-transfer authority
builder PullTemplate, space-cluster-controller (admitted, not transfer-listed) PermissionDenied: template transfer requires artifact-transfer authority
builder NodeExportTemplate, space-node-api (listed, control) NotFound: template release not found
builder NodeExportTemplate, space-builder-node-api (listed, control) NotFound: template release not found
builder NodeExportTemplate, space-api (listed, control) NotFound: template release not found
builder PullTemplate, space-api (listed, control) Unimplemented: auto-resolve not yet supported, provide sources
worker PullTemplate from a real source, space-cluster-controller (admitted, not transfer-listed) PermissionDenied: template transfer requires artifact-transfer authority
worker PullTemplate from a real source, space-internal-api (admitted, not transfer-listed) PermissionDenied: template transfer requires artifact-transfer authority
worker PullTemplate from a real source, unlisted space-auth-probe PermissionDenied: service account system:serviceaccount:space:space-auth-probe is not allowed for this node-api surface
worker PullTemplate from a real source, static token Unauthenticated: invalid token
build-manager GetBuildStatus, static token Unauthenticated: build RPC requires a projected ServiceAccount token
build-manager GetBuildStatus, space-builder-node-api aud space-node-api Unavailable: build token review failed
build-manager GetBuildStatus, space-api aud space-build-manager PermissionDenied: caller may not submit builds
build-manager GetBuildStatus, space-builder-node-api aud space-build-manager (control) InvalidArgument: job_id: invalid auth UUID: invalid length: found 0

Cleanup. transfer-probe was deleted through the admin API (operation succeeded, template 404 afterwards). The probe ServiceAccount was deleted, and the cluster was torn down.

ENVIRONMENTAL, first attempt. The first fresh cluster, from 672feac, passed bring-up, smoke-kind-e2e.sh and the first 50 typed scenarios. Then the box's 49 GB Docker data volume filled: the local Rust build's cache and stale images left about 5 GB for the nodes. Postgres failed with No space left on device, so every later scenario failed at bootstrap. The cluster was torn down, and the authorized cleanup steps freed 11.5 GiB (build cache), 0.2 GiB (dangling images) and 26.6 GiB (unused tagged images). The run above was then made from scratch on a fresh cluster, with 20 GB free after its bring-up. No scenario failed for any other reason.

Every scenario (this round)
Scenario Result
ecw.access_idle_reconnect_preserves_workspace (ecw pass) pass
ecw.authenticated_access_with_public_traffic_disabled (ecw pass) pass
ecw.background_process_control (ecw pass) pass
ecw.binary_file_round_trip (ecw pass) pass
ecw.cold_create_environment (ecw pass) pass
ecw.command_round_trip (ecw pass) pass
ecw.domain_access_survives_default_deny (ecw pass) pass
ecw.filesystem_mutations (ecw pass) pass
ecw.filesystem_ownership (ecw pass) pass
ecw.filesystem_private_boundary (ecw pass) pass
ecw.filesystem_read_identity (ecw pass) pass
ecw.filesystem_round_trip (ecw pass) pass
ecw.filesystem_watcher (ecw pass) pass
ecw.filesystem_write_identity (ecw pass) pass
ecw.guest_vlan_frames_dropped (ecw pass) pass
ecw.immediate_egress (ecw pass) pass
ecw.internet_access_allowed (ecw pass) pass
ecw.internet_access_denied (ecw pass) pass
ecw.large_stdout_integrity (ecw pass) pass
ecw.loopback_route_stays_open (ecw pass) pass
ecw.pause_connect (ecw pass) pass
ecw.pause_connect_race (ecw pass) pass
ecw.pause_timeout_requires_connect (ecw pass) pass
ecw.process_identity (ecw pass) pass
ecw.process_reattach (ecw pass) pass
ecw.public_port_to_all_interfaces (ecw pass) pass
ecw.public_port_to_loopback (ecw pass) pass
ecw.refresh_keepalive (ecw pass) pass
ecw.sandbox_delete (ecw pass) pass
ecw.sandbox_info (ecw pass) pass
ecw.sandbox_list_pagination (ecw pass) pass
ecw.timeout_rearm (ecw pass) pass
ecw.timeout_update (ecw pass) pass
ecw.warm_pool_environment (ecw pass) pass
ecw.watcher_identity (ecw pass) pass
local.btrfs_send_restore_fails_closed_without_parent_source (ecw pass) pass
local.btrfs_send_restore_pulls_parent_template (ecw pass) pass
local.btrfs_send_restore_rejects_corrupted_stream (ecw pass) pass
local.cold_sandbox_eviction_wake_round_trip (ecw pass) pass
local.concurrent_restore_converges_to_one_execution (ecw pass) pass
local.deferred_btrfs_send_capture_restores_cross_node (ecw pass) pass
local.ecw_template_build_from_oci_boots (ecw pass) pass
local.encrypted_resuspend_preserves_cow (ecw pass) pass
local.forced_drain_evacuates_sandbox (ecw pass) pass
local.gentle_drain_leaves_sandbox_resident (ecw pass) pass
local.node_ctl_fence_refuses_and_reconciles (ecw pass) pass
local.node_group_boundary_schedules (ecw pass) pass
local.node_selector_extras_render_and_enforce (ecw pass) pass
local.orphaned_restore_pod_gc_reaps_only_superseded (ecw pass) pass
local.pending_backup_census_reports_blocked_upload (ecw pass) pass
local.private_template_build_isolation (ecw pass) pass
local.quarantine_recovery_backs_off (ecw pass) pass
local.rate_limiters_enforce_and_fail_open (ecw pass) pass
local.retention_claim_finish_forward (ecw pass) pass
local.retention_restore_source_guard (ecw pass) pass
local.sandbox_network_isolation_holds (ecw pass) pass
local.snapshot_reaper_reclaims_backed_fork_snapshot (ecw pass) pass
local.suspend_restore_cross_node_from_object_storage (ecw pass) pass
local.suspend_snapshot_rows_are_per_attempt (ecw pass) pass
local.telemetry_reaches_local_backends (ecw pass) pass
local.template_cache_lru_eviction_reclaims (ecw pass) pass
local.template_startup_recovery_decision_table (ecw pass) pass
network_policy.v1_cidr_allow (ecw pass) pass
network_policy.v1_cidr_deny (ecw pass) pass
network_policy.v1_deny_precedes_allow (ecw pass) pass
network_policy.v1_domain_binding (ecw pass) pass
network_policy.v1_exact_ip_allow (ecw pass) pass
network_policy.v1_exact_ip_deny (ecw pass) pass
network_policy.v1_fastly_shared_edge_bypass (ecw pass) pass
network_policy.v1_platform_guardrail_precedes_allow (ecw pass) pass
network_policy.v1_rules_survive_restore (ecw pass) pass
network_policy.v1_tcp_dns_transport_respects_ip_allow (ecw pass) pass
network_policy.v1_udp_deny_precedes_allow (ecw pass) pass
network_policy.v1_udp_respects_cidr_allow (ecw pass) pass
network_policy.v1_udp_respects_cidr_deny (ecw pass) pass
network_policy.v1_udp_respects_ip_allow (ecw pass) pass
network_policy.v1_udp_respects_ip_deny (ecw pass) pass
network_policy.v1_wildcard_domain_allow (ecw pass) pass
ecw.background_process_control (native pass) pass
ecw.command_round_trip (native pass) pass
ecw.pause_connect (native pass) pass
ecw.sandbox_delete (native pass) pass
ecw.sandbox_info (native pass) pass
ecw.sandbox_list_pagination (native pass) pass
native.encrypted_double_restore_retires_prior_pods (native pass) pass
native.resident_pause (native pass) pass
native.stopped_restart_preserves_rootfs (native pass) pass
native_concurrency.command_idempotency (native pass) pass
native_concurrency.create_idempotency (native pass) pass
native_concurrency.independent_commands (native pass) pass
native_concurrency.lifecycle_contention (native pass) pass
native_concurrency.slow_reader (native pass) pass
native_contract.command_validation (native pass) pass
native_contract.completed_delete_replay (native pass) pass
native_contract.idempotency_conflict (native pass) pass
native_contract.permanent_command_error (native pass) pass
native_contract.tenant_isolation (native pass) pass
native_contract.token_scope (native pass) pass
native_contract.zero_vcpu_admission (native pass) pass
native_recovery.command_survives_data_plane_restart (native pass) pass
native_recovery.queued_create_survives_api_restart (native pass) pass
native_recovery.slow_operation_does_not_block_later_work (native pass) pass

TokenReview costs, from the code

  • node-api caches an accepted verdict for at most 30 s, or until the token expires if that is sooner (accepted_token_cache_expiry). The cache is per listener and keyed by token and audience. So each caller token is reviewed again about every 30 s per listener and audience. Rejections are cached for 2 s.
  • BuildTokenReviewer (build-manager and the builder gateway's BuildTemplate) caches nothing. It reviews every build RPC within its in-flight permit and per-second budget.

An earlier round compared full suites of this change and of main on fresh clusters of the same box: the branch's suite was not slower (2997 s against 3113 s, different bases). That comparison was not repeated this round.

Rust (fix branch only). cargo fmt --check, cargo clippy -p pplx-space-common-svc --features k8s,test-support --all-targets -D warnings, and cargo test -p pplx-space-common-svc --features k8s,test-support --lib auth::build all pass, 9 tests. The new test fails without the fix. This branch changes one Rust comment and no Rust code.

Review follow-up

Restack onto #472 (content changed):

  • kind: run node and runtime configuration like a deployed node #472's "Release binaries read no test hooks" row no longer lists api's and internal-api's TokenReview namespace among the hooks the charts set, since this PR removes that hook. This PR's own "Release binaries, built without test-hooks" row is kept.
  • The space-cluster umbrella from Add cluster deployment overlays for existing SPACE charts #391 (infra/space/cluster/chart/values-builder.yaml) now carries the builder node-api's auth allowlists from values-builder.yaml, as it carries that file's other builder settings. Without them its space-builder release rendered the worker lists, which grant no BuildTemplate. Checked by diffing that release's rendered allowlists against the Kind builder render.

Previous round (rebase onto #312):

  • Rebased as it was, the PR would have refused every peer template transfer under Kubernetes auth. Fixed: both node-api roles set the transfer list, derived from the code (space-api starts a pull; the target node-api requests the export with its own token).
  • The deleted setup scripts added node-api's own account to the surface allowlist and gave it an outbound token. No change was needed: both node-api accounts were already admitted on both roles, and node-api already presents a projected space-node-api token (600 s) to peers. Verified live.
  • Authorize peer-to-peer template transfers #312's transfer guidance in the deleted README. Fixed: it is now in docs/configuration.md, merged with Authorize peer-to-peer template transfers #312's own section there.
  • The premise that the API starts pulls in Kind. Rejected after checking the code: the admin template-pull route is direct-substrate only and answers 501 on Kubernetes, so the transfer was exercised by a direct PullTemplate. The comments, README and docs say so.
  • kind: run node and runtime configuration like a deployed node #472's "Release binaries read no test hooks" row. Not in that tree; resolved in the restack above.
  • Comments that cited lines the rebase had moved, and a builder trusted-policy comment that named a direct-substrate-only route as its caller. Fixed: symbols are cited, and the real caller (TemplateController) is named.

Earlier rounds:

  • TokenReview caching was described wrongly. Fixed: the cache behaviour is described from the code.
  • Evidence bases were incomplete. Fixed: each run states its base.
  • The operator space-buildmgr recipe always picked the first builder. Fixed: it asks each builder in turn.
  • A README cross-reference pointed the wrong way, and the namespace-hook comment in auth/k8s.rs said Kind uses the override. Both fixed.
  • Removing scripts/k8s/projected-tokens/README.md lost the Kind apiserver timeout note. Fixed: moved to the Kind README.

Rollback

Apply or revert the caller token projections and the listeners' auth settings together. A partial revert, where callers present static tokens to listeners that require TokenReview or the reverse, rejects RPCs. The charts change in one release, so reverting the whole commit is safe; reverting one chart on its own is not.


Prepared with AI assistance (Claude Code), reviewed against the repository's public-review checklist before pushing.

🤖 Generated with Claude Code

@ac-pplx
ac-pplx force-pushed the ac/kind-fidelity-internal-auth branch from 72c7fe0 to 1fcebfd Compare October 14, 2026 22:00
@ac-pplx
ac-pplx force-pushed the ac/kind-fidelity-internal-auth branch 4 times, most recently from d274012 to 41b9d23 Compare October 15, 2026 14:26
@ac-pplx
ac-pplx changed the base branch from main to ac/kind-fidelity-node-runtime October 15, 2026 14:27
@ac-pplx
ac-pplx marked this pull request as ready for review October 15, 2026 16:54
@ac-pplx
ac-pplx added this pull request to stack #533 October 15, 2026 17:00

@nickdujay-pplx nickdujay-pplx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 41b9d23 against #472. No additional blocker in this layer.

Traced the required caller → listener edges, normal/internal/build audiences, worker and builder allowlists, independent policy/build/transfer authority, token file rotation and TokenReview-only RBAC. Both node-api accounts have the peer-transfer grant; the umbrella builder release also renders its build and transfer permissions correctly. Static fallback is disabled, while startup-required static secrets remain available.

113 existing common-svc auth/bearer/peer-auth tests passed locally, as did the relevant chart checks and worker/builder renders. All four Kind CI shards passed. The Rust source edit here is comment-only. I reviewed the reported live refusal/transfer probes but did not independently repeat them on Linux/KVM; the two pre-existing product findings described in the PR remain separate follow-ups.

Apply or revert the caller token projections and listener auth configuration together: a partially reverted static/projected-token setup will reject RPCs. Approval covers this delta; #480's separate installer finding remains outstanding.

Kind's node-api (both roles) and build-manager accepted only the shared
static bearer tokens, so the Kubernetes TokenReview path those listeners
run in deployments never ran locally. Run it the way deployed services do:

- node-api turns on Kubernetes token auth with the static fallback off on
  both listeners (audiences space-node-api and space-node-api-internal),
  with the per-surface allowlists derived from the callers the code shows:
  the api substrate (shared and admin api, cluster-controller), the data
  plane as the guest bridge, internal-api's service-port routes, and
  node-api itself for peer template transfers. Workers carry the internal,
  lifecycle-probe and trusted-policy lists; builders admit BuildTemplate
  from the api's account with system and tenant authority.
- Both node-api roles grant peer template-transfer authority
  (SPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS) to space-api,
  the account that starts a pull with PullTemplate on the target, and to
  both node-api accounts, since the target's node-api calls the source's
  NodeExportTemplate with its own projected node-api token. Without the
  list, Kubernetes auth refuses every peer transfer. The admin template-pull
  route serves only the direct substrate, so in Kind, as on any Kubernetes
  deployment, a pull is a direct PullTemplate call.
- The chart comments cite the code by symbol rather than line number, and
  the trusted-policy lists name their real callers.
- build-manager reviews the builder node-api's token (audience
  space-build-manager) with no static fallback, admitting only that account
  with system and tenant build authority.
- Both mount their own ServiceAccount's API credentials, limited by RBAC to
  creating TokenReviews, and read their namespace from that mount as a
  release binary does. The SPACE_K8S_TOKEN_AUTH_NAMESPACE test hook is
  removed from the api and internal-api charts, so no Kind auth path
  depends on a test hook, and the README's test-hooks row no longer lists
  it among the hooks the charts set.
- The space-cluster umbrella's builder release (infra/space/cluster) takes
  the builder node-api's allowlists from its values-builder.yaml, as it
  takes that file's other builder settings; otherwise it would render the
  worker lists, which grant no BuildTemplate.
- Every caller mounts projected tokens and names them with
  SPACE_NODE_API_TOKEN_PATH, SPACE_NODE_API_INTERNAL_TOKEN_PATH and
  SPACE_BUILD_MANAGER_TOKEN_PATH. Tokens presented to node-api and the node
  services' ClusterApi and credential-manager tokens live 600 s, the
  builder's build-manager token 7200 s, as deployed.
- Kind's Rust build adds pplx-space-build-mgr-svc/k8s, matching release.
- The manual scripts/k8s/projected-tokens wiring is removed: the charts now
  do it, and the script would turn the static fallback back on and set the
  namespace hook. Its policy-authorization and template-transfer guidance
  moves to docs/configuration.md, and its note on Kind apiserver timeouts
  during parallel builds (SPACE_TILT_MAX_PARALLEL_UPDATES) moves to the Kind
  README.
- The namespace override's comment no longer says Kind uses it.

The smoke harness and the space-node-debug CLIs call neither listener. The
README documents how an operator gives space-buildmgr a token, and asks each
builder in turn for a job, because a job lives only in the node-local store of
the build-manager that ran it.
@ac-pplx
ac-pplx force-pushed the ac/kind-fidelity-internal-auth branch from 41b9d23 to 0dc0ea2 Compare October 15, 2026 18:07
@ac-pplx
ac-pplx added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit dd6e88a Oct 15, 2026
25 checks passed
@ac-pplx
ac-pplx deleted the ac/kind-fidelity-internal-auth branch October 15, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants