kind: run Postgres like the deployed database

Tiltfile

@@ -55,6 +55,12 @@ local(["bash", "infra/space/kind/scripts/check-object-store-transition.sh", KIND
 # "Upgrading from the node-stack layout" in infra/space/kind/README.md.
 local(["bash", "infra/space/kind/scripts/check-node-layout.sh", KIND_CLUSTER], quiet = True)
 
+# A cluster whose Postgres data directory an earlier space-postgres chart
+# initialized (PostgreSQL 16) cannot run this chart's PostgreSQL 17; refuse it
+# with the database reset rather than crash-loop the pod. See "Recovering from
+# an unbundled migration version" in infra/space/kind/README.md.
+local(["bash", "infra/space/kind/scripts/check-postgres-data.sh", KIND_CLUSTER], quiet = True)
+
 # Select one metrics sink per process. The datadog and otel modes also select
 # the matching trace exporter; off disables both.
 # These local selectors do not activate any deployed environment.

ci/mirrored-images.txt

@@ -59,7 +59,7 @@ kindest/node:v1.34.0@sha256:7416a61b42b1662ca6ca89f02028ac133a309a2a30ba309614e8
 
 # Images the Kind charts run (infra/space/kind/charts/*/values.yaml), which
 # scripts/tests/lib/bootstrap-kind-stack.sh loads into the cluster.
-postgres:16-alpine       sha256:1a66d744c1b459e13b05a8fca341da84cb63383e99ce262210efee5a319d4551 ghcr.io/perplexityai/space-ci-mirror:postgres-16-alpine
+postgres:17              sha256:3cec7eb015ba8adb28139fa5c83b8489cdf0e666e53dfdf20f598ae0cc8739e3 ghcr.io/perplexityai/space-ci-mirror:postgres-17
 redis:7-alpine           sha256:ca0acbb137c1dc3339c8b147a58fd6f42775d4599327b50e7b116c23de501af2 ghcr.io/perplexityai/space-ci-mirror:redis-7-alpine
 # Bump versitygw to the first release that contains versity/versitygw#2415
 # (see the space-object-store chart's values); Content-MD5 stays.

infra/space/kind/README.md

@@ -868,6 +868,22 @@ tilt trigger space-seed-publish
 Deleting the publish Job is required because the database reset removes the
 template-release tags that resolve `seed:latest`.
 
+The same reset applies to a cluster whose database was created by an earlier
+revision of the `space-postgres` chart (PostgreSQL 16, superuser connection
+string): PostgreSQL 17 cannot start on that data directory, and the role
+setup the pod's init script provides runs only when it initializes an empty
+one. `bootstrap.sh`, the smoke suites' `scripts/tests/lib/bootstrap-kind-stack.sh`
+(under `SKIP_BOOTSTRAP=1` too) and the `Tiltfile` run
+`infra/space/kind/scripts/check-postgres-data.sh` before they apply anything
+or query the database. It reads the data directory's `PG_VERSION` on the node
+that holds the volume, so it answers the same whether the old server is still
+running or the new one is crash-looping, and on a mismatch it prints the reset
+commands and stops without changing anything (the scripts exit 1; Tilt
+reports a Tiltfile load error). Without Tilt, rerun `bootstrap.sh` (or the
+smoke suite) after the three `kubectl delete` commands above; a running Tilt
+reloads with `tilt trigger "(Tiltfile)"`. Recreating the cluster has the same
+effect.
+
 ### Diagnosing iteration disk pressure
 
 Every build loads its images into each Kind node's containerd store, which
@@ -977,7 +993,7 @@ kind cluster: space-local (kind-cluster.yaml.tmpl — 6 nodes)
 ├── worker2 [pool=sandbox, space-enabled=true]
 │   └── identical to worker
 ├── worker3 [pool=infra,    space.pplx.net/infra=true]
-│   ├── space-postgres        (StatefulSet)
+│   ├── space-postgres        (StatefulSet; PostgreSQL 17 serving TLS only, non-superuser master user)
 │   ├── space-object-store    (versitygw S3 gateway + bucket-init Job)
 │   ├── space-api             (Deployment, 2–3 replicas by HPA; SPACE_K8S_CONTROL_PLANE=true)
 │   ├── space-admin-api       (Deployment; the same binary on the dedicated admin surface)
@@ -1378,7 +1394,13 @@ asks of the hosting deployment.
 | One DaemonSet per node service, systemd watchdogs via k8s probes | One DaemonSet per node service and role from the Kind charts (`charts/space-<service>-local`), probed on each service's health socket; runtime-supervisor is the same host systemd unit node-manager's init container installs |
 | Node image (packer) provides `/space/cache`, the cgroup subtree, loop devices, device permissions, SPACE accounts and the `/space` skeleton before any pod starts | Sandbox workers and builders boot the SPACE Kind node image, which bakes in the same as systemd units that run before kubelet on every node start (see [The node layer](#the-node-layer)) |
 | `/space/cache` is btrfs on instance-store NVMe            | btrfs on a sparse loopback image at `/var/lib/space-cache.img` on the kind node (40 GiB, same code paths, less I/O headroom), mounted on the node's `/space/cache` by `space-cache.service`, so it outlives pod restarts like the NVMe does |
-| Postgres on RDS, secrets via ExternalSecrets / AWS Secrets Manager | In-cluster `space-postgres` StatefulSet, static Kubernetes Secrets             |
+| Postgres on RDS, secrets via ExternalSecrets / AWS Secrets Manager | In-cluster `space-postgres` StatefulSet, one pod on a 1 GiB volume, static Kubernetes Secrets |
+| PostgreSQL 17 | The same major version (`postgres:17`; CI takes it from the image mirror at a pinned digest) |
+| Every client connects with `sslmode=require`              | The same `sslmode=require` on every connection: the services and the migrate Job through the `space-postgres` Secret's connection string, and the harness's `psql` helpers over loopback TLS inside the pod. The certificate is self-signed per pod start, which `require` accepts because it does not verify the server. Kind's server also refuses unencrypted TCP, so no Kind client runs in plaintext and one that sets `sslmode=disable` fails. A client that omits `sslmode` still connects over TLS, because libpq and sqlx default to `prefer`, so Kind cannot catch a connection string that loses `sslmode=require` |
+| Bootstrap and migrations run as the managed service's master user, which is not a superuser but a member of `rds_superuser` | The same branch of `bootstrap-roles`: every connection string names `space_admin`, which has `CREATEDB CREATEROLE` and `rds_superuser` WITH ADMIN OPTION and owns the database. A stock server has none of that, so the pod's init script creates it, including the `rds_superuser` role itself and its `SET` privilege on `log_statement`, which a managed service grants that role and `bootstrap-roles` needs before writing passwords. The image's superuser remains, used only by the image's init and the harness's direct-state `psql` |
+| Per-service pools: the api tier far above the binary's default of 25, internal-api somewhat above it, every other service on the default; no connection pooler | `POSTGRES_POOL_MAX_CONNECTIONS` 100 for space-api and space-admin-api, 30 for internal-api, unset elsewhere; no pooler. `max_connections` (1400) holds every replica's pools at their HPA maxima plus a rollout surge, the migrate Job and headroom, so a service runs out of its own pool before the server refuses it. The pod's memory limit (8 GiB) is sized for that many backends rather than for the suites' usual load |
+| Server logs lock waits; the application roles' `lock_timeout` is off (`SPACE_DB_APP_LOCK_TIMEOUT` `0`) | `log_lock_waits = on`; `SPACE_DB_APP_LOCK_TIMEOUT` is unset, which leaves `lock_timeout` off the same way. Other server parameters are the image's defaults rather than a managed parameter group |
+| The migration Job fails fast: one attempt, no restarts, its deadline passed to the migrator | The same `space-db-migrate` Job shape (`backoffLimit: 0`, `restartPolicy: Never`, `activeDeadlineSeconds` = `SPACE_DB_MIGRATE_JOB_DEADLINE_SECONDS`), bootstrap-roles → migrate → bootstrap-roles. `bootstrap.sh` stops on a failed Job with its logs; rerunning it replaces the Job |
 | S3 for snapshots + template artifacts                     | In-cluster versitygw (real S3 protocol, sigv4, path-style)                    |
 | `space-credmgr-authority` Deployment whose platform KEK comes from an external intermediate-key broker, with TLS from its own certificate (`credmgr_authority_tls.tf`; the direct KMS backend is gone, `credmgr_authority_kms.tf`) | two replicas on the same broker path against the in-repo broker fixture (`space-kind-key-broker`), which derives keys instead of protecting them; TLS from a self-signed certificate `internal-rpc-tls.sh` issues per cluster |
 | Internal RPC CA and listener leaves issued by Terraform and synced by External Secrets (`internal_rpc_tls.tf`) | the same Secrets issued per cluster by `scripts/internal-rpc-tls.sh`; the ingress activation leaf names `space-api` rather than `space-ingress-activation` |

infra/space/kind/charts/space-api-local/templates/deployment.yaml

@@ -148,6 +148,8 @@ spec:
               {{- include "space-cluster.dbPassword" (dict "ctx" . "role" "USER" "value" .Values.postgres.managedUserPassword) | nindent 14 }}
             - name: SPACE_DB_CREDS_PASSWORD
               {{- include "space-cluster.dbPassword" (dict "ctx" . "role" "CREDS" "value" .Values.postgres.managedCredentialsPassword) | nindent 14 }}
+            - name: POSTGRES_POOL_MAX_CONNECTIONS
+              value: {{ .Values.postgres.poolMaxConnections | quote }}
             - name: SPACE_INTERNAL_TOKEN
               valueFrom:
                 secretKeyRef:

infra/space/kind/charts/space-api-local/templates/migrate-job.yaml

@@ -6,6 +6,12 @@
 # runner can turn a retried lock race into a failed run
 # (crates/libs/db/migrator/src/migration_retry.rs). Tilt makes space-api wait
 # for this Job, and bootstrap.sh waits for it before applying the API.
+#
+# It fails fast, as a deployment's migration Job does: one attempt, no pod
+# restarts, inside a deadline the migrator sizes its lock-retry window against
+# (--job-deadline-seconds, crates/libs/db/migrator/src/lib.rs). A failed run
+# stays Failed for an operator to read rather than being retried into a
+# different state; bootstrap.sh and Tilt replace the Job to run it again.
 apiVersion: batch/v1
 kind: Job
 metadata:
@@ -15,14 +21,15 @@ metadata:
     app.kubernetes.io/name: space-db-migrate
     app.kubernetes.io/instance: {{ .Release.Name }}
 spec:
-  backoffLimit: 6
+  backoffLimit: 0
+  activeDeadlineSeconds: {{ .Values.migrate.jobDeadlineSeconds }}
   template:
     metadata:
       labels:
         app.kubernetes.io/name: space-db-migrate
         app.kubernetes.io/instance: {{ .Release.Name }}
     spec:
-      restartPolicy: OnFailure
+      restartPolicy: Never
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
@@ -33,8 +40,9 @@ spec:
         # pre-run creates the managed logins (space_user, space_creds, …)
         # the services connect as, and the post-run grants them on any
         # tables the migrations just created. The kind Postgres connection
-        # string authenticates as the image-created superuser, which can
-        # create roles.
+        # string authenticates as the master user, which, as on a managed
+        # database, is not a superuser but can create roles (space-postgres
+        # chart, configmap.yaml).
         - name: bootstrap-roles
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
@@ -60,6 +68,10 @@ spec:
                 secretKeyRef:
                   name: {{ .Values.postgres.secretName | default "space-postgres" }}
                   key: POSTGRES_CONNECTION_STRING
+            # Kubernetes does not expose activeDeadlineSeconds to the
+            # container, so the Job passes its own.
+            - name: SPACE_DB_MIGRATE_JOB_DEADLINE_SECONDS
+              value: {{ .Values.migrate.jobDeadlineSeconds | quote }}
       containers:
         - name: bootstrap-roles-grants
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

infra/space/kind/charts/space-api-local/values.yaml

@@ -84,6 +84,10 @@ clusterId: "kind-local"
 # space-db-migrator + space-api-svc).
 migrate:
   enabled: true
+  # activeDeadlineSeconds, also passed to the migrator. The migrator's own
+  # default for an unstated deadline (MIGRATION_JOB_ACTIVE_DEADLINE,
+  # crates/libs/db/migrator/src/migration_retry.rs).
+  jobDeadlineSeconds: 1800
 
 postgres:
   # Secret holding POSTGRES_CONNECTION_STRING.
@@ -99,6 +103,12 @@ postgres:
   bootstrapSuperadminPassword: space-dev-superadmin
   bootstrapReadOnlyPassword: space-dev-ro
   bootstrapMigratorPassword: space-dev-migrator
+  # POSTGRES_POOL_MAX_CONNECTIONS for space-api and space-admin-api, each of
+  # whose two pools (space_user and space_creds) it sizes
+  # (crates/libs/common/svc/src/sql/mod.rs). The api tier runs a pool far
+  # above the binary's default of 25, as deployed. The space-postgres chart's
+  # max_connections is sized to hold every replica's pools.
+  poolMaxConnections: 100
 
 # K8s control-plane substrate. SPACE_K8S_CONTROL_PLANE=true runs the
 # api-server on the KubernetesSubstrate: it discovers nodes through the Pod

infra/space/kind/charts/space-internal-api-local/templates/deployment.yaml

@@ -133,6 +133,8 @@ spec:
                   key: {{ .Values.postgres.secretKey }}
             - name: SPACE_DB_USER_PASSWORD
               {{- include "space-cluster.dbPassword" (dict "ctx" . "role" "USER" "value" .Values.postgres.managedUserPassword) | nindent 14 }}
+            - name: POSTGRES_POOL_MAX_CONNECTIONS
+              value: {{ .Values.postgres.poolMaxConnections | quote }}
             - name: SPACE_INTERNAL_TOKEN
               valueFrom:
                 secretKeyRef:

infra/space/kind/charts/space-internal-api-local/values.yaml

@@ -59,8 +59,13 @@ postgres:
   secretName: space-postgres
   secretKey: POSTGRES_CONNECTION_STRING
   # Dev-default space_user password; the role is provisioned by the api
-  # chart's bootstrap-roles initContainer with the same value.
+  # chart's space-db-migrate Job (bootstrap-roles) with the same value.
   managedUserPassword: space-dev-user
+  # POSTGRES_POOL_MAX_CONNECTIONS for the general space_user pool
+  # (crates/libs/common/svc/src/sql/mod.rs); the template-visibility sideline
+  # pool keeps its own fixed cap. Somewhat above the binary's default of 25,
+  # as deployed. The space-postgres chart's max_connections holds it.
+  poolMaxConnections: 30
 
 nodeSelector:
   space.pplx.net/infra: "true"

infra/space/kind/charts/space-postgres/Chart.yaml

@@ -3,4 +3,4 @@ name: space-postgres
 description: Single-pod Postgres for the local KinD SPACE stack
 type: application
 version: 0.1.0
-appVersion: "16"
+appVersion: "17"

infra/space/kind/charts/space-postgres/templates/configmap.yaml

@@ -0,0 +1,51 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: {{ .Release.Name }}-postgres
+  namespace: {{ .Release.Namespace }}
+  labels:
+    app.kubernetes.io/name: space-postgres
+    app.kubernetes.io/instance: {{ .Release.Name }}
+data:
+  # Every TCP client must encrypt: an unencrypted connection matches no line
+  # and is refused, so no Kind client runs in plaintext and one that sets
+  # sslmode=disable fails. A client that omits sslmode still connects, over
+  # TLS: libpq and sqlx both default to prefer. The Unix socket serves only
+  # the image's init scripts; the in-pod harness and probes connect over
+  # loopback TLS.
+  pg_hba.conf: |
+    # TYPE   DATABASE  USER  ADDRESS       METHOD
+    local    all       all                 trust
+    hostssl  all       all   127.0.0.1/32  trust
+    hostssl  all       all   ::1/128       trust
+    hostssl  all       all   all           scram-sha-256
+  # Run once by the image's entrypoint when it initializes an empty data
+  # directory, as the image superuser. It provides what a managed service gives
+  # its master user, which bootstrap-roles' managed-service branch relies on
+  # (crates/libs/db/migrator/src/bootstrap_roles.rs):
+  # - rds_superuser exists, so converge_superadmin_attributes grants it to
+  #   space_superadmin instead of making that role a real superuser. The master
+  #   holds it WITH ADMIN OPTION, which that GRANT needs.
+  # - CREATEROLE, for ensure_roles and converge_memberships, and CREATEDB, which
+  #   a non-superuser needs to give space_superadmin CREATEDB.
+  # - SET on log_statement for rds_superuser: prepare_session mutes statement
+  #   logging before it writes role passwords and fails without it. A managed
+  #   service lets its rds_superuser set that parameter; stock PostgreSQL needs
+  #   the grant.
+  # - Ownership of the database, so the master owns schema public
+  #   (pg_database_owner) and can converge its privileges
+  #   (converge_schema_privileges), as a managed service's master owns the
+  #   database it creates.
+  10-managed-service-master.sh: |
+    #!/usr/bin/env bash
+    set -Eeuo pipefail
+    psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \
+      --set=master_user="$SPACE_DB_MASTER_USER" \
+      --set=master_password="$SPACE_DB_MASTER_PASSWORD" \
+      --set=database="$POSTGRES_DB" <<'SQL'
+    CREATE ROLE rds_superuser NOLOGIN;
+    GRANT SET ON PARAMETER log_statement TO rds_superuser;
+    CREATE ROLE :"master_user" WITH LOGIN CREATEDB CREATEROLE PASSWORD :'master_password';
+    GRANT rds_superuser TO :"master_user" WITH ADMIN OPTION;
+    ALTER DATABASE :"database" OWNER TO :"master_user";
+    SQL

infra/space/kind/charts/space-postgres/templates/secret.yaml

@@ -11,4 +11,8 @@ stringData:
   POSTGRES_USER: {{ .Values.credentials.user | quote }}
   POSTGRES_PASSWORD: {{ .Values.credentials.password | quote }}
   POSTGRES_DB: {{ .Values.credentials.database | quote }}
-  POSTGRES_CONNECTION_STRING: "postgres://{{ .Values.credentials.user }}:{{ .Values.credentials.password }}@{{ .Values.service.name }}:{{ .Values.service.port }}/{{ .Values.credentials.database }}?sslmode=disable"
+  SPACE_DB_MASTER_USER: {{ .Values.credentials.masterUser | quote }}
+  SPACE_DB_MASTER_PASSWORD: {{ .Values.credentials.masterPassword | quote }}
+  # The master user over TLS, as deployed: sslmode=require encrypts without
+  # verifying the server, so the pod's self-signed certificate is enough.
+  POSTGRES_CONNECTION_STRING: "postgres://{{ .Values.credentials.masterUser }}:{{ .Values.credentials.masterPassword }}@{{ .Values.service.name }}:{{ .Values.service.port }}/{{ .Values.credentials.database }}?sslmode=require"