Repository navigation
kind: authenticate node-api and build-manager callers by TokenReview - #484
Conversation
72c7fe0 to
1fcebfd
Compare
d274012 to
41b9d23
Compare
nickdujay-pplx
left a comment
There was a problem hiding this comment.
Reviewed 41b9d23 against #472. No additional blocker in this layer.
Traced the required caller → listener edges, normal/internal/build audiences, worker and builder allowlists, independent policy/build/transfer authority, token file rotation and TokenReview-only RBAC. Both node-api accounts have the peer-transfer grant; the umbrella builder release also renders its build and transfer permissions correctly. Static fallback is disabled, while startup-required static secrets remain available.
113 existing common-svc auth/bearer/peer-auth tests passed locally, as did the relevant chart checks and worker/builder renders. All four Kind CI shards passed. The Rust source edit here is comment-only. I reviewed the reported live refusal/transfer probes but did not independently repeat them on Linux/KVM; the two pre-existing product findings described in the PR remain separate follow-ups.
Apply or revert the caller token projections and listener auth configuration together: a partially reverted static/projected-token setup will reject RPCs. Approval covers this delta; #480's separate installer finding remains outstanding.
Kind's node-api (both roles) and build-manager accepted only the shared static bearer tokens, so the Kubernetes TokenReview path those listeners run in deployments never ran locally. Run it the way deployed services do: - node-api turns on Kubernetes token auth with the static fallback off on both listeners (audiences space-node-api and space-node-api-internal), with the per-surface allowlists derived from the callers the code shows: the api substrate (shared and admin api, cluster-controller), the data plane as the guest bridge, internal-api's service-port routes, and node-api itself for peer template transfers. Workers carry the internal, lifecycle-probe and trusted-policy lists; builders admit BuildTemplate from the api's account with system and tenant authority. - Both node-api roles grant peer template-transfer authority (SPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS) to space-api, the account that starts a pull with PullTemplate on the target, and to both node-api accounts, since the target's node-api calls the source's NodeExportTemplate with its own projected node-api token. Without the list, Kubernetes auth refuses every peer transfer. The admin template-pull route serves only the direct substrate, so in Kind, as on any Kubernetes deployment, a pull is a direct PullTemplate call. - The chart comments cite the code by symbol rather than line number, and the trusted-policy lists name their real callers. - build-manager reviews the builder node-api's token (audience space-build-manager) with no static fallback, admitting only that account with system and tenant build authority. - Both mount their own ServiceAccount's API credentials, limited by RBAC to creating TokenReviews, and read their namespace from that mount as a release binary does. The SPACE_K8S_TOKEN_AUTH_NAMESPACE test hook is removed from the api and internal-api charts, so no Kind auth path depends on a test hook, and the README's test-hooks row no longer lists it among the hooks the charts set. - The space-cluster umbrella's builder release (infra/space/cluster) takes the builder node-api's allowlists from its values-builder.yaml, as it takes that file's other builder settings; otherwise it would render the worker lists, which grant no BuildTemplate. - Every caller mounts projected tokens and names them with SPACE_NODE_API_TOKEN_PATH, SPACE_NODE_API_INTERNAL_TOKEN_PATH and SPACE_BUILD_MANAGER_TOKEN_PATH. Tokens presented to node-api and the node services' ClusterApi and credential-manager tokens live 600 s, the builder's build-manager token 7200 s, as deployed. - Kind's Rust build adds pplx-space-build-mgr-svc/k8s, matching release. - The manual scripts/k8s/projected-tokens wiring is removed: the charts now do it, and the script would turn the static fallback back on and set the namespace hook. Its policy-authorization and template-transfer guidance moves to docs/configuration.md, and its note on Kind apiserver timeouts during parallel builds (SPACE_TILT_MAX_PARALLEL_UPDATES) moves to the Kind README. - The namespace override's comment no longer says Kind uses it. The smoke harness and the space-node-debug CLIs call neither listener. The README documents how an operator gives space-buildmgr a token, and asks each builder in turn for a job, because a job lives only in the node-local store of the build-manager that ran it.
41b9d23 to
0dc0ea2
Compare
What Kind now does like deployments
Kind's node-api (worker and builder) and build-manager accepted only the shared static bearer tokens, so the Kubernetes TokenReview path those listeners run in deployments never ran locally. They now authenticate every caller by TokenReview of its projected ServiceAccount token, with the static fallback off, as deployed services do.
node-api, both listeners (
SPACE_K8S_TOKEN_AUTH_ENABLED=true,SPACE_K8S_TOKEN_AUTH_STATIC_FALLBACK_ENABLED=false). It uses the code's audiences:space-node-apifor WorkerApi, BuilderApi and NodeInfo, andspace-node-api-internalfor WorkerApiInternal (DEFAULT_NODE_API_AUDIENCE,DEFAULT_NODE_API_INTERNAL_AUDIENCE,auth/k8s.rs). The per-surface allowlists come from the callers the code shows:space-api(shared and admin api),space-cluster-controller,space-data-plane(guest bridge),space-internal-api, both node-api accounts (template transfers,connect_remoteintransfer.rs)space-api,space-cluster-controllerspace-api,space-cluster-controllerspace-data-planespace-api,space-cluster-controller, both node-api accountsTemplateControllertemplate and pool calls);BuildTemplate(template-build, system and tenant lists)space-apiSPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS)space-api,space-node-api,space-builder-node-apispace-builder-node-apiPeer template transfers. Authorize peer-to-peer template transfers #312 gates
PullTemplateandNodeExportTemplatebehind a separate transfer list. With Kubernetes auth on, only listed accounts get the grant, and an empty list refuses every transfer (AuthPolicy::authenticated_caller;require_template_transfer_authorityandprepare_template_exportinauth/template_transfer.rs). Both roles list:space-api, the account that starts a pull withPullTemplateon the target;NodeExportTemplatewith its own projectedspace-node-apitoken (SPACE_NODE_API_TOKEN_PATH, 600 s).Both node-api accounts were already in the surface allowlists, and node-api already mounted that token, so neither changed. One list covers both RPCs, so
space-apimay also request an export. The guest bridge is always excluded.Who starts a transfer in Kind. No product path does. The only in-tree caller of
PullTemplateis admin-api'spull_node_template. It serves only the direct substrate:require_admin_node_registryanswers 501 on the Kubernetes substrate, in Kind as in any Kubernetes deployment. So in Kind a pull is an operator's directPullTemplatecall to the target node-api with aspace-apitoken. The README says so next to the allowlist table, anddocs/configuration.mdsays the route is direct-substrate only.build-manager reviews the builder node-api's token (audience
space-build-manager). It has no static fallback (auth/build.rs,auth/build/k8s.rs).API access for the reviewers. node-api and build-manager mount their own ServiceAccount's API credentials (
automountServiceAccountToken: true). A per-release ClusterRole limits each account totokenreviews/create. They read the namespace their allowlists qualify from that mount (auth_namespace), as a release binary does.No auth test hook. The
SPACE_K8S_TOKEN_AUTH_NAMESPACEtest hook is removed from the api and internal-api charts, so no Kind auth path depends on it.Caller tokens. Every caller mounts projected tokens under
/var/run/space-authand names them withSPACE_NODE_API_TOKEN_PATH,SPACE_NODE_API_INTERNAL_TOKEN_PATHandSPACE_BUILD_MANAGER_TOKEN_PATH:An explicit path makes a missing token an error instead of a silent fall back to the static token.
Lifetimes, as deployed.
The kubelet rotates each one at 80% of its lifetime.
Static tokens stay mounted.
SPACE_INTERNAL_TOKENandSPACE_INTERNAL_CONTROL_TOKENare still mounted because the binaries require them at startup, but they admit nobody. node-ctl's own auth is unchanged.Feature list.
pplx-space-build-mgr-svc/k8sis added to Kind's cook and build feature lists (docker/Dockerfile.rust-build-kind), matchingDockerfile.rust-build-common.Manual wiring removed.
scripts/k8s/projected-tokensgoes: the charts now do what it wired by hand, and its01-apply-auth.shwould turn the static fallback back on and set the namespace hook. Its guidance moves:docs/configuration.md("Policy authorization");docs/configuration.md("Template transfer authority"): which two callers need the grant, that both also need surface admission, and that an empty list disables transfers with a startup warning;SPACE_TILT_MAX_PARALLEL_UPDATES) goes to the Kind README ("Kind apiserver timeouts during parallel builds").Comments cite symbols. The chart comments this PR adds cite the code by symbol, not by line number. The rebase onto Authorize peer-to-peer template transfers #312 had already moved several of the cited lines.
Namespace hook comment. The comment on
SPACE_K8S_TOKEN_AUTH_NAMESPACEinauth/k8s.rsno longer says Kind uses the override. This is a comment-only change.Code paths this exercises in every Kind run:
NodeApiAuthLayerin Kubernetes mode, including the per-surface policy, wait-ready, policy-mutation and guest-bridge authority derivation (auth/k8s.rs).BuildTokenRevieweron the builder gateway and on build-manager.bearer_token.rs,build/mgr/client).The template-transfer grant and node-api's outbound peer authentication run whenever a pull is started, which in Kind means a direct
PullTemplatecall. This round's validation ran them that way (below).The smoke harness and the
space-node-debugCLIs call neither surface: the harness drives the public API, and the shell's CLIs use node-local Unix sockets. For an operator calling build-manager directly withspace-buildmgr, the README documents the deployed way: mint a short-lived token for an admitted account with the build-manager audience, and give the CLI that token and the internal RPC CA. A job lives only in the node-local SQLite store of the build-manager that ran it, and the CLI's Service isinternalTrafficPolicy: Local, so the recipe asks each builder in turn until one knows the job.What stays different, and why
README "Differences from the EKS deployment modules" gains one row. Kind binaries are built with
test-hooks, so smokes can shrink timers. No chart now sets a hook for authentication, and the TokenReview listeners read their namespace from the ServiceAccount mount, as release binaries do. No existing row described this area, so none is resolved. The transfer wiring adds no row: the route that starts a pull is unavailable on any Kubernetes substrate, not only in Kind. Two things are deliberately unchanged:#472's "Release binaries read no test hooks" row is in this tree. This commit removes the TokenReview namespace from its list of hooks the charts set, so it names no hook a chart no longer sets; this PR's own "Release binaries, built without
test-hooks" row sits beside it.Product findings
1. A rejected build TokenReview is reported as retryable.
UNAVAILABLE: build token review failedrather thanUNAUTHENTICATED. The same applies to the builder node-api'sBuildTemplategateway. The probe below shows it again this round.BuildTokenReviewertreats any TokenReviewerrorstring as an outage. The apiserver puts the reason inerroron every ordinary rejection (forged, expired, wrong audience, withauthenticated: false). So bad tokens are reported as retryable, and counted astoken_review_unavailable, while node-api's reviewer reports the same answer asUNAUTHENTICATED. The existing unit fixture modelled the rejection without anerror, which is why it passed.ac/kind-fidelity-internal-auth-fix-build-review-rejectionhas a single commit that checksauthenticatedfirst and adds a test with the apiserver's response shape; the test fails without the fix. It is pushed as a branch only, with no separate PR.2. A peer export of a received template never becomes ready. This is new this round, and not about auth.
NodeExportTemplateofseedfrom any node, worker or builder, answersUNAVAILABLE: compressed rootfs not ready …; retry after background generation completes, and keeps answering that. Retried every 15 s for 7.5 minutes, it never succeeded.seedhere) lives in a read-only received subvolume (ro=true,Flags: readonly) with norootfs.ext4.zst. So generation fails every time withRead-only file system (os error 30), which template-manager logs as a warning, and the caller sees only the retryableUNAVAILABLE.main.Validation
This round
What was validated. Commit
4f8a766, based onmainat8dda85a. The pushed headd274012is that commit rebased ontomainat6b427da.git range-diffshows the patch unchanged. The three upstream commits it gained (#477 operator egress profiles, and two docs-site commits) touch none of this PR's files exceptdocs/configuration.md, where #477 renamed a different row of the same table. None touches node-api auth. CI's Kind Pull CI also passed all four smoke shards on672feac, an earlier push of this round with identical rendered manifests.Settings. One fresh cluster, brought up from
4f8a766with CI's settings:SPACE_KIND_TELEMETRY_MODE=otel,DATA_PLANE_URL=http://127.0.0.1:8090, and the admin token file and cold-receipts directory created the waykind-ci.ymlcreates them. Before the suite, the health gate required: Tilt quiet, no ReplicaSet younger than 180 s, every node Ready and published (wait-node-publication.sh), then a 120 s settle.test_kind_node_service_env.pyand five other Kind chart tests, plustest_env_test_hook_boundary.py), on672feachelm templateof node-api, both roles:672feacagainst the final treesmoke-kind-e2e.sh(cluster create → e2e round trip)smoke-kind-typed.sh all, unsharded,SKIP_BOOTSTRAP=1, after the gatenative_contract.command_replay_credentialsis ignored by its own declaration, as onmainPOST /admin/api/v1/nodes/{id}/template-pulls501: "direct node registry is only available for the direct/self-managed substrate"PullTemplatewith aspace-apitoken, templatetransfer-probe(built for this check through the admin template build API, cached only on one builder)PullTemplateofseed(worker → worker, builder → worker, builder → builder)UNAVAILABLE(product finding 2)Each successful pull exercised, on the real path:
space-api;space-node-apifor a worker target,space-builder-node-apifor a builder target;Provenance. Every pod's image ID was recorded after the gate and again after the last check: 88 pods both times, and the same set of image IDs.
version 4f8a76654bdd, withmode=k8s_token_reviewormode=k8sandstatic_fallback_enabled=false.SPACE_NODE_API_AUTH_TEMPLATE_TRANSFER_SERVICE_ACCOUNTS=space-api,space-node-api,space-builder-node-api.SPACE_K8S_TOKEN_AUTH_NAMESPACE.Probe. Each call went from the host to a node's
host_ip:9800, or through a port-forward to build-manager. It used grpcurl, the cluster's internal RPC CA, the in-repo protos, and tokens fromkubectl create token.space-auth-probeis a throwaway ServiceAccount on no list, deleted afterwards. "Admitted, not transfer-listed" means the account passes the surface allowlist but is not on the transfer list.Cleanup.
transfer-probewas deleted through the admin API (operationsucceeded, template404afterwards). The probe ServiceAccount was deleted, and the cluster was torn down.ENVIRONMENTAL, first attempt. The first fresh cluster, from
672feac, passed bring-up,smoke-kind-e2e.shand the first 50 typed scenarios. Then the box's 49 GB Docker data volume filled: the local Rust build's cache and stale images left about 5 GB for the nodes. Postgres failed withNo space left on device, so every later scenario failed at bootstrap. The cluster was torn down, and the authorized cleanup steps freed 11.5 GiB (build cache), 0.2 GiB (dangling images) and 26.6 GiB (unused tagged images). The run above was then made from scratch on a fresh cluster, with 20 GB free after its bring-up. No scenario failed for any other reason.Every scenario (this round)
ecw.access_idle_reconnect_preserves_workspace(ecw pass)ecw.authenticated_access_with_public_traffic_disabled(ecw pass)ecw.background_process_control(ecw pass)ecw.binary_file_round_trip(ecw pass)ecw.cold_create_environment(ecw pass)ecw.command_round_trip(ecw pass)ecw.domain_access_survives_default_deny(ecw pass)ecw.filesystem_mutations(ecw pass)ecw.filesystem_ownership(ecw pass)ecw.filesystem_private_boundary(ecw pass)ecw.filesystem_read_identity(ecw pass)ecw.filesystem_round_trip(ecw pass)ecw.filesystem_watcher(ecw pass)ecw.filesystem_write_identity(ecw pass)ecw.guest_vlan_frames_dropped(ecw pass)ecw.immediate_egress(ecw pass)ecw.internet_access_allowed(ecw pass)ecw.internet_access_denied(ecw pass)ecw.large_stdout_integrity(ecw pass)ecw.loopback_route_stays_open(ecw pass)ecw.pause_connect(ecw pass)ecw.pause_connect_race(ecw pass)ecw.pause_timeout_requires_connect(ecw pass)ecw.process_identity(ecw pass)ecw.process_reattach(ecw pass)ecw.public_port_to_all_interfaces(ecw pass)ecw.public_port_to_loopback(ecw pass)ecw.refresh_keepalive(ecw pass)ecw.sandbox_delete(ecw pass)ecw.sandbox_info(ecw pass)ecw.sandbox_list_pagination(ecw pass)ecw.timeout_rearm(ecw pass)ecw.timeout_update(ecw pass)ecw.warm_pool_environment(ecw pass)ecw.watcher_identity(ecw pass)local.btrfs_send_restore_fails_closed_without_parent_source(ecw pass)local.btrfs_send_restore_pulls_parent_template(ecw pass)local.btrfs_send_restore_rejects_corrupted_stream(ecw pass)local.cold_sandbox_eviction_wake_round_trip(ecw pass)local.concurrent_restore_converges_to_one_execution(ecw pass)local.deferred_btrfs_send_capture_restores_cross_node(ecw pass)local.ecw_template_build_from_oci_boots(ecw pass)local.encrypted_resuspend_preserves_cow(ecw pass)local.forced_drain_evacuates_sandbox(ecw pass)local.gentle_drain_leaves_sandbox_resident(ecw pass)local.node_ctl_fence_refuses_and_reconciles(ecw pass)local.node_group_boundary_schedules(ecw pass)local.node_selector_extras_render_and_enforce(ecw pass)local.orphaned_restore_pod_gc_reaps_only_superseded(ecw pass)local.pending_backup_census_reports_blocked_upload(ecw pass)local.private_template_build_isolation(ecw pass)local.quarantine_recovery_backs_off(ecw pass)local.rate_limiters_enforce_and_fail_open(ecw pass)local.retention_claim_finish_forward(ecw pass)local.retention_restore_source_guard(ecw pass)local.sandbox_network_isolation_holds(ecw pass)local.snapshot_reaper_reclaims_backed_fork_snapshot(ecw pass)local.suspend_restore_cross_node_from_object_storage(ecw pass)local.suspend_snapshot_rows_are_per_attempt(ecw pass)local.telemetry_reaches_local_backends(ecw pass)local.template_cache_lru_eviction_reclaims(ecw pass)local.template_startup_recovery_decision_table(ecw pass)network_policy.v1_cidr_allow(ecw pass)network_policy.v1_cidr_deny(ecw pass)network_policy.v1_deny_precedes_allow(ecw pass)network_policy.v1_domain_binding(ecw pass)network_policy.v1_exact_ip_allow(ecw pass)network_policy.v1_exact_ip_deny(ecw pass)network_policy.v1_fastly_shared_edge_bypass(ecw pass)network_policy.v1_platform_guardrail_precedes_allow(ecw pass)network_policy.v1_rules_survive_restore(ecw pass)network_policy.v1_tcp_dns_transport_respects_ip_allow(ecw pass)network_policy.v1_udp_deny_precedes_allow(ecw pass)network_policy.v1_udp_respects_cidr_allow(ecw pass)network_policy.v1_udp_respects_cidr_deny(ecw pass)network_policy.v1_udp_respects_ip_allow(ecw pass)network_policy.v1_udp_respects_ip_deny(ecw pass)network_policy.v1_wildcard_domain_allow(ecw pass)ecw.background_process_control(native pass)ecw.command_round_trip(native pass)ecw.pause_connect(native pass)ecw.sandbox_delete(native pass)ecw.sandbox_info(native pass)ecw.sandbox_list_pagination(native pass)native.encrypted_double_restore_retires_prior_pods(native pass)native.resident_pause(native pass)native.stopped_restart_preserves_rootfs(native pass)native_concurrency.command_idempotency(native pass)native_concurrency.create_idempotency(native pass)native_concurrency.independent_commands(native pass)native_concurrency.lifecycle_contention(native pass)native_concurrency.slow_reader(native pass)native_contract.command_validation(native pass)native_contract.completed_delete_replay(native pass)native_contract.idempotency_conflict(native pass)native_contract.permanent_command_error(native pass)native_contract.tenant_isolation(native pass)native_contract.token_scope(native pass)native_contract.zero_vcpu_admission(native pass)native_recovery.command_survives_data_plane_restart(native pass)native_recovery.queued_create_survives_api_restart(native pass)native_recovery.slow_operation_does_not_block_later_work(native pass)TokenReview costs, from the code
accepted_token_cache_expiry). The cache is per listener and keyed by token and audience. So each caller token is reviewed again about every 30 s per listener and audience. Rejections are cached for 2 s.BuildTokenReviewer(build-manager and the builder gateway'sBuildTemplate) caches nothing. It reviews every build RPC within its in-flight permit and per-second budget.An earlier round compared full suites of this change and of
mainon fresh clusters of the same box: the branch's suite was not slower (2997 s against 3113 s, different bases). That comparison was not repeated this round.Rust (fix branch only).
cargo fmt --check,cargo clippy -p pplx-space-common-svc --features k8s,test-support --all-targets -D warnings, andcargo test -p pplx-space-common-svc --features k8s,test-support --lib auth::buildall pass, 9 tests. The new test fails without the fix. This branch changes one Rust comment and no Rust code.Review follow-up
Restack onto #472 (content changed):
test-hooks" row is kept.space-clusterumbrella from Add cluster deployment overlays for existing SPACE charts #391 (infra/space/cluster/chart/values-builder.yaml) now carries the builder node-api'sauthallowlists fromvalues-builder.yaml, as it carries that file's other builder settings. Without them itsspace-builderrelease rendered the worker lists, which grant no BuildTemplate. Checked by diffing that release's rendered allowlists against the Kind builder render.Previous round (rebase onto #312):
space-apistarts a pull; the target node-api requests the export with its own token).space-node-apitoken (600 s) to peers. Verified live.docs/configuration.md, merged with Authorize peer-to-peer template transfers #312's own section there.PullTemplate. The comments, README and docs say so.TemplateController) is named.Earlier rounds:
space-buildmgrrecipe always picked the first builder. Fixed: it asks each builder in turn.auth/k8s.rssaid Kind uses the override. Both fixed.scripts/k8s/projected-tokens/README.mdlost the Kind apiserver timeout note. Fixed: moved to the Kind README.Rollback
Apply or revert the caller token projections and the listeners' auth settings together. A partial revert, where callers present static tokens to listeners that require TokenReview or the reverse, rejects RPCs. The charts change in one release, so reverting the whole commit is safe; reverting one chart on its own is not.
Prepared with AI assistance (Claude Code), reviewed against the repository's public-review checklist before pushing.
🤖 Generated with Claude Code