[ac-pplx]

Stack: Part of the Kind fidelity stack, bottom to top: #485 → #480 → #472 → #484 → #507. Based on main.
This PR is one commit, and that commit is the only thing to review.

Summary

Kind's Postgres now behaves like the database SPACE deployments run, so a green Kind run covers the production connection and provisioning paths that Kind used to skip.

What Kind now does like deployments, and the code paths it exercises

Deployed behaviour	Kind now	Code path exercised
PostgreSQL 17	postgres:17 (17.11); CI takes it from the GHCR mirror at a pinned linux/amd64 digest (ci/mirrored-images.txt)	every query and migration on the major version SPACE is built and tested against
sslmode=require on every client connection	The space-postgres Secret's connection string carries sslmode=require, so api, admin-api, internal-api, cluster-controller, credmgr-authority and the migrate Job all connect over TLS. So do the harness's psql helpers (kind_stack_psql, space_smoke/adapters.py, smoke-kind-sandbox-fork.py, run-local-smokes.sh, bootstrap-prod-seed.sh), over loopback TLS inside the pod. The server's certificate is self-signed per pod start, and pg_hba.conf refuses unencrypted TCP	sqlx's rustls client path in every service (database_connect_options keeps the URL's TLS options when it swaps in the managed identity), the migrator's TLS connect, PgListener over TLS
Bootstrap and migrations run as the managed service's master user, which is not a superuser but a member of rds_superuser	The connection string names space_admin, which has CREATEDB CREATEROLE, holds rds_superuser WITH ADMIN OPTION and owns the database. The pod's init script creates that setup	bootstrap_roles.rs: converge_superadmin_attributes grants rds_superuser instead of SUPERUSER; converge_role_attributes verifies the privileged attributes instead of resetting them. The migrator's assume_migrator_role does its SET ROLE from a non-superuser session
api tier pools far above the default, internal-api somewhat above it, the rest on the default; no pooler	POSTGRES_POOL_MAX_CONNECTIONS is 100 for space-api and space-admin-api, 30 for internal-api, and unset elsewhere. max_connections is 1400, which holds the sum of every replica's pool maxima (HPA maxima plus a rollout surge; the arithmetic is in space-postgres/values.yaml) plus the Job and headroom. The memory limit goes from 512 MiB to 8 GiB, measured to hold that many backends (see Validation); the request stays 256 MiB	postgres_pool_options (crates/libs/common/svc/src/sql/mod.rs) with non-default sizes. Exhausting a pool, not a server refusal, is now what limits a service. On main, Kind's max_connections of 100 was below the sum of the default pools
log_lock_waits = on; the roles' lock_timeout off	log_lock_waits=on; SPACE_DB_APP_LOCK_TIMEOUT stays unset, which leaves lock_timeout off the same way	—
The migration Job fails fast	backoffLimit: 0, restartPolicy: Never, activeDeadlineSeconds: 1800 passed to the migrator as SPACE_DB_MIGRATE_JOB_DEADLINE_SECONDS; bootstrap → migrate → bootstrap unchanged. bootstrap.sh stops on a Failed Job and prints its logs, where it used to wait out its timeout	migration_retry::LockRetryPolicy::for_job_deadline sized against the Job's real deadline

The managed-service role setup, and what only a managed service provides

A stock postgres:17 has none of this, so space-postgres/templates/configmap.yaml creates it once, when the image initializes an empty data directory. I checked every line by dropping it and rerunning bootstrap-roles as the master:

• rds_superuser (the role itself). Without it, bootstrap-roles takes the plain non-superuser path, which fails below.
• CREATEROLE on the master: role creation and memberships. CREATEDB on the master: without it, ALTER ROLE space_superadmin CREATEDB fails with "permission denied to alter role".
• rds_superuser WITH ADMIN OPTION: without it, GRANT rds_superuser TO space_superadmin fails with "permission denied to grant role".
• GRANT SET ON PARAMETER log_statement TO rds_superuser. This is the one thing a managed service provides that stock PostgreSQL needs emulated: prepare_session mutes statement logging before it writes role passwords and refuses to continue otherwise ("permission denied to set parameter").
• Database ownership by the master, so it owns schema public (pg_database_owner). Without it, bootstrap-roles still reports converged, but the GRANT … ON SCHEMA public statements only raise "no privileges were granted" warnings and space_migrator never receives CREATE (see Observations).

The image's own superuser remains, used only by the image's init and the harness's direct-state psql. That is a README row.

Clusters from before this change

PostgreSQL 17 cannot start on the PostgreSQL 16 data directory of a cluster brought up from main, and the init script runs only on an empty one, so such a cluster resets its database. A new infra/space/kind/scripts/check-postgres-data.sh reads the data directory's PG_VERSION on the node that holds the volume (the local-path PV's host path), so it answers the same whether the old server is still running, the new one is crash-looping on the old data, or no pod runs. On a mismatch it prints the reset commands and exits 1, changing nothing. bootstrap.sh (beside the other existing-cluster checks), the smoke bring-up (kind_stack_assert_postgres_data, beside the mount check, so SKIP_BOOTSTRAP=1 runs it too) and the Tiltfile (a load-time local(), like the other cluster checks) all run it before they apply the chart or query the database.

What stays different (README rows)

• One in-cluster pod on a 1 GiB volume with static Secrets, instead of RDS with External Secrets.
• A per-pod self-signed certificate. require does not verify it, as deployed clients do not. Kind also refuses unencrypted TCP, so no Kind client runs in plaintext and one that sets sslmode=disable fails. A client that omits sslmode still connects over TLS (libpq and sqlx default to prefer), so Kind cannot catch a connection string that loses sslmode=require.
• The managed-service role setup above comes from an init script. The image superuser exists for the harness only.
• Other server parameters are the image's defaults, not a managed parameter group.
• Clusters created before this change must reset their database (README "Recovering from an unbundled migration version"); the check above stops every bring-up path with the commands.

Validation

All runs were on one Linux/KVM devbox with CI's settings (SPACE_KIND_TELEMETRY_MODE=otel, DATA_PLANE_URL=http://127.0.0.1:8090, the admin token file and cold-receipts directory created as in kind-ci.yml). Before every suite a health gate waited for Tilt to be quiet, no ReplicaSet younger than 180 s, every node Ready and published (wait-node-publication.sh), then a 120 s settle.

The whole sequence below ran twice. The first time was on 7f65b45 and then fd230d3, based on 0f1d3be. The second time was on the pushed head 20c0205, based on 9254f09, from a torn-down cluster. The results were the same both times; the numbers are from the second run. The pushed head's own diff is identical to fd230d3's (only hunk offsets moved in the rebase).

Stale bench. Brought up with bootstrap.sh from main 9254f09: postgres:16-alpine, server_version_num 160015, ssl off. main's smoke-kind-e2e.sh passed on it first, so it had real state.

Step, from the PR head	Result
Before the fix (d99c57b, on the first run's 0f1d3be bench): smoke bring-up with SKIP_BOOTSTRAP=1	rc 1: server does not support SSL, but SSL was required, then could not read node_k8s_tombstones. This is the reported failure
(a) smoke bring-up with SKIP_BOOTSTRAP=1	rc 1 after 1 s, with the reset commands; nothing ran
(b) bootstrap.sh	rc 1 after 1 s, with the reset commands; the StatefulSet is still postgres:16-alpine
(c) tilt up	Tiltfile load error at check-postgres-data.sh after about 2 s, with the reset commands; nothing applied
Re-apply state: the new chart applied by hand over the 16 data directory	The pod crash-loops (database files are incompatible with server), and the check still refuses (rc 1)
The printed reset (three kubectl delete), then tilt trigger "(Tiltfile)"	The check passes. Tilt brings everything up with 0 errors; migrate and seed-publish complete

Fresh (reset) bench, from the head. Pod image IDs of every SPACE service were the same before and after the suites; the only difference is the sandbox pods' pause image. The deployed migrator and API report git_sha 20c0205.

Suite	Result
smoke-kind-e2e.sh (full bring-up path, Tilt live)	PASS, 11 s
ecw-surface:local.concurrent_restore_converges_to_one_execution	PASS, 74 s
native-surface:native_concurrency.create_idempotency	PASS, 5 s
native-surface:native_recovery.queued_create_survives_api_restart	PASS, 22 s
ecw-surface:local.suspend_restore_cross_node_from_object_storage	PASS, 18 s
ecw-surface:ecw.sandbox_list_pagination	PASS, 15 s

Postgres on the live head cluster:

• SHOW ssl is on, max_connections 1400, log_lock_waits on, server_version 17.11.
• pg_stat_ssl × pg_stat_activity: every client backend is TLS 1.3 (space_user, space_creds and the harness psql). A sampler every 30 s through the scenarios saw 186 backend observations, all encrypted, peak 69.
• The server log's only no pg_hba.conf entry … no encryption lines (431) came in the 52 s after the reset. They came from two pod IPs that no pod holds now: the stale bench's main-era control-plane pods, still on the old Secret's sslmode=disable, until Tilt replaced them. None came after that.
• POSTGRES_POOL_MAX_CONNECTIONS is 100 on space-api and space-admin-api, 30 on internal-api, and unset on cluster-controller and credmgr-authority. The migrate Job has backoffLimit 0, restartPolicy Never and activeDeadlineSeconds 1800, and it completed. The only superuser is the image's own (spaceuser), and all 72 public tables are owned by space_migrator.
• The TLS key is mode 0600 and owned by the image's postgres account, chowned by name. The server accepted it.

Memory limit (method and result). Each test session is a psql over TLS that reads every public table once (SELECT count(*) on all 72) and then sits in pg_sleep. Memory is the Postgres container's cgroup memory.current and memory.peak.

• In Kind, at the earlier 4 GiB limit: 1,262 sessions (the budget's sum) put 1,283 backends on the server. The container passed 3.9 GiB and was OOM-killed (OOMKilled, exit 137). Postgres restarted on its volume and the stack recovered.
• A standalone postgres:17 with the same flags, TLS and the Kind database's schema, with no limit: 1,397 sessions (max_connections minus the superuser reserve) settle at 5.3 GiB (4.6 GiB anon). That is linear at about 3.3 MiB per backend (1.66 GiB anon at 500, 3.3 GiB at 1,000).
• In Kind at the new 8 GiB limit: the same 1,262 sessions (1,287 backends) peak at 4.9 GiB with no restart, and smoke-kind-e2e.sh passes afterwards.

Not rerun this round: the full typed suite (an earlier revision passed all 102 scenarios on both main and this branch), and the earlier round's fail-fast drill for the migrate Job (its chart and bootstrap.sh code did not change this round).

Review follow-up

Restack onto main (after #391 and #241):

• Conflicts in the api and internal-api deployments: Add cluster deployment overlays for existing SPACE charts #391's space-cluster.dbPassword include is kept, with POSTGRES_POOL_MAX_CONNECTIONS after it. In bootstrap-kind-stack.sh, Unify Kind telemetry and log routing under observability profiles #241's detached bring-up text is kept beside kind_stack_assert_postgres_data, which still returns early for a missing cluster. The commit's content is otherwise unchanged.

Earlier rounds:

• The smoke bring-up's Postgres data check now returns early when the cluster does not exist, as the mount check beside it does. It used to resolve REPO_ROOT first, which failed scripts/tests/test-kind-tombstone-sweep.sh (it sources the library without one).
• sslmode claim: fixed. An omitted sslmode defaults to prefer (libpq, sqlx 0.9) and connects over TLS, so the README, the configmap.yaml comment and this text now say the server refuses plaintext and sslmode=disable fails. I added no live check: a pg_stat_ssl check sees prefer as ssl = t, so it could not catch a dropped sslmode.
• Stale Postgres 16 benches: fixed. check-postgres-data.sh runs in bootstrap.sh, the smoke bring-up (SKIP_BOOTSTRAP=1 too) and the Tiltfile. It reads PG_VERSION from the volume, not the server's socket, which also covers the crash-looping re-apply state.
• Unevidenced memory figure: fixed. Reproducing it showed 4 GiB was OOM-killed at about 1,280 backends, so the limit is now 8 GiB, measured as above; the comment now gives the reasoning and the measured figure, and drops an unevidenced claim about 512 MiB.
• runAsUser: 999: fixed. The init container chowns the key to postgres by name instead of pinning a uid; its comment also notes that the alpine images ship no openssl CLI.

Observations (not changed here)

• cargo test fixtures still use postgres:16-alpine (crates/libs/db-test-fixtures/src/lib.rs, crates/libs/db/migrator/src/pg_test_support.rs, ci/scripts/lib/test-deps.sh, and the test-container line in ci/mirrored-images.txt). Moving them to 17 changes every Rust test run, so it belongs in its own follow-up.
• If the connecting master does not own schema public, bootstrap-roles reports converged even though its schema GRANTs only raised "no privileges were granted" warnings, and the failure surfaces later in migrations.
• On the managed-service branch, bootstrap-roles logs nothing at the default log level. Only sqlx notices at info show which branch ran.
• The reference deployment's in-cluster PostgreSQL (infra/space/aws/reference/addons/charts/postgresql/templates/secret.yaml) still connects with sslmode=disable.

Rollback

This is not an in-place database upgrade or downgrade. A Kind cluster whose Postgres data an earlier chart initialized (PostgreSQL 16) needs the documented reset, and the new preflight says so. Going back is the same: a PostgreSQL 16 chart, for example a branch from before this PR on a cluster created after it, cannot use PostgreSQL 17 data, and the old chart has no check to name the problem. Reset the database, or recreate the cluster, whenever you cross that boundary in either direction.

This change was prepared with AI assistance (Claude Code) and validated as described above.

🤖 Generated with Claude Code

[nickdujay-pplx]

Reviewed ff45e93 against its base, including the stacked consumers. No blocking issue found.

Independently ran the rendered Postgres configuration on real PostgreSQL 17.11: TLS accepted, plaintext TCP rejected, space_admin was non-superuser with the required role-management privileges, and bootstrap-roles → migrations → bootstrap-roles succeeded. A further bootstrap was convergent; all 72 public tables belonged to space_migrator, and a database restart preserved the migrated state and TLS access. Reviewed the migration deadline/failure handling and the PG_VERSION guard in all three bring-up paths. Exact-head chart renders passed, and all four Kind CI shards passed.

Upgrade/rollback constraint: an existing PG16 Kind database requires the documented deliberate reset. Reverting the image to PG16 cannot reuse PG17 data; recreate/reset the local database when reverting across that boundary. This is not an in-place database upgrade or downgrade. I did not independently rerun the Linux/KVM suite or the maximum-connection memory stress test.