feat: bootstrap new instances in-process instead of with Jobs#11231
Draft
armru wants to merge 11 commits into
Draft
feat: bootstrap new instances in-process instead of with Jobs#11231armru wants to merge 11 commits into
armru wants to merge 11 commits into
Conversation
armru
requested review from
a team,
NiccoloFei,
jsilvela and
litaocdl
as code owners
July 20, 2026 16:30
dosubot
Bot
added
size:XXL
cnpg-bot
added
backport-requested ◀️
Contributor
|
❗ By default, the pull request is configured to backport to all release branches.
|
armru
marked this pull request as draft
July 20, 2026 16:39
Member
Author
|
/test barman_plugin=pr-1025 |
Contributor
|
@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/29762494135 |
Member
Author
|
/test barman_plugin=pr-1025 |
Contributor
|
@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/29812087177 |
The five bootstrap subcommands (initdb, join, pgbasebackup, restore, restoresnapshot) each carried their PGDATA initialization logic inline in their cobra command, which made it impossible to run a bootstrap without spawning the matching subcommand in a dedicated Job. Move that logic into a new pkg/management/postgres/bootstrap package exposing an Instruction type and a single Execute dispatch, and reduce the subcommands to flag parsing that delegates to it. The per-mode behavior is preserved exactly, including that only restoresnapshot skips the target-directory cleanup. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Add a per-instance bootstrapProgress flag carrying the bootstrap mode and start time, so the instance manager can advertise that PostgreSQL is not up yet while it initializes the data directory in-process. While the flag is set the startup probe is skipped (the copy phase has no socket and can outlast the probe budget), /pg/status replies with a 503 and a recognizable body instead of a valid payload (a valid one would make the operator short-circuit its reconciliation guards on an unready pod), and an instance-manager update is refused with a 409. The flag is intentionally separate from mightBeUnavailable so a bootstrapping pod is never counted as reporting its status, which would break primary-before-replica ordering. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Teach instance run to initialize an uninitialized PGDATA in-process before the controller-runtime manager starts, so a data directory can be bootstrapped without a dedicated Job. When run is given a --bootstrap-mode (plus the same per-mode flags the Job builders used to emit) and the data directory has no completion marker yet, run flips the instance bootstrap flag, serves a minimal status server and the local web server, dispatches the bootstrap through the shared library, and on success writes a durable completion marker inside PGDATA before falling through to the normal run. The manager cannot start earlier because its in-pod reconciler would write into an empty PGDATA, and the marker makes a container restart after a successful bootstrap skip straight to the normal run. The marker is fsynced together with its parent directory because WriteFileAtomic only flushes the file contents, not the rename. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Prepare the operator to bootstrap an instance by turning its own pod into a bootstrapping pod instead of creating a dedicated Job. The per-mode flag lists that the six bootstrap Job builders assembled inline are extracted into shared arg builders, and a new ApplyBootstrapOverlay stamps those flags (prefixed with --bootstrap-mode) plus the volumes, mounts, environment and annotation each mode needs onto a steady-state instance pod produced by NewInstance. The overlay leaves the podSpec and podEnvHash annotations alone so a just-bootstrapped pod is not mistaken for a drifted one. Nothing calls the overlay yet, so the Job builders keep working exactly as before. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The five bootstrap paths (initdb, join, pgbasebackup, recovery and volume-snapshot restore) no longer run in a dedicated batch Job before the instance Pod is created. The operator now builds the steady-state instance Pod and applies the bootstrap overlay to it, so the Pod initializes its own data directory in-process and then starts PostgreSQL. Primary creation, replica join and reattachment go through a single shared builder, and the lost-Job recovery that guarded #11036 becomes a plain Pod recreation for any serial whose data PVC never finished bootstrapping, covering both the interrupted first primary and a bootstrapping Pod evicted before it completed. A data PVC is now flipped to ready only once its owning Pod reports ready, instead of as soon as any Pod attaches, so a Pod still bootstrapping in-process no longer marks its not-yet-initialized volume as ready and cannot be mistaken for a fully bootstrapped one on the reattach path. The pg_upgrade Job and every Job read, guard and cleanup path are left untouched. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
…of mounting it The bootstrap overlay used to mount the barman endpoint CA Secret on the instance's certificates directory and point AWS_CA_BUNDLE at it. That was harmless for the short-lived bootstrap Job, but the job-free bootstrap keeps the same pod running as the instance, and a read-only Secret mount over that directory shadows the place where the instance manager writes its server certificate: the certificate reconciler then fails with a read-only filesystem error and the pod never becomes ready. Drop the CA mount from the overlay and instead write the recovery-source endpoint CA to disk during the in-process bootstrap, before barman-cloud runs, so the CA files are owned by the instance manager exactly as on every other pod. The write reuses the certificate reconciler's existing secret-to-file helper and resolves the source CA with the same precedence the Job builders used: the recovery backup reference, then the origin backup status, then the recovery source external cluster. The container-level AWS_CA_BUNDLE the Job injected is not restored, as the in-process restore and wal-restore build their own environment and only need the CA file present at the unchanged location. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
…identity The in-process phase-0 bootstrap writes a completion marker inside PGDATA so a container restart can skip work that already finished. Because the marker lives inside PGDATA, it travels inside volume snapshots: a PVC cloned from a snapshot already carries the source instance's marker, so IsCompleted reported the data directory as already bootstrapped and runBootstrap skipped the whole restoresnapshot phase-0. That lost the PITR recovery configuration (the cluster promoted from the snapshot data without fetching the archived WAL up to the requested target) and left snapshot cloned scale-up replicas without their demotion and replica configuration. Scope the marker to the instance that wrote it. The marker payload now records the namespace, cluster name, cluster UID and pod name, and IsCompleted reports completed only when the marker exists, parses, and matches the current instance. A marker inherited from another instance (or an unparseable or old-format one) is treated as not completed, so phase-0 runs and rewrites the marker with the new identity. The cluster is now loaded before the marker check to provide the cluster UID. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Issue #11228 replaces every initdb/join/pgbasebackup/restore/restoresnapshot bootstrap Job with a self-bootstrapping instance Pod, so e2e assertions that checked batch/v1 Job objects for these paths no longer describe what the operator does. The microservice import-failure test now watches the bootstrap pod's restart count instead of a Job's failed count (the exact tier of the failure-surfacing policy is still open on #11228, so the assertion only relies on the pod restarting rather than the threshold); the #10985 regression test now checks that no new instance Pod appears for the recreated serial while the previous WAL PVC is terminating, instead of checking for the absence of a join Job. The initdb, join, pgbasebackup, and barman/volume-snapshot recovery suites (in-tree and via plugin-barman-cloud) gained a shared assertion that no bootstrap Job is ever created while a cluster is provisioned, plus a check that the resulting instance pods reach Ready without restarting their postgres container. A small tests/utils/jobs package and two new assertions in tests/internal/asserts/cluster (AssertNoBootstrapJobCreatedDuring, AssertClusterInstancesHaveNoRestart) back these checks so they can be reused across suites instead of inlining the same polling logic everywhere. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The e2e list for #11228 had no walStorage variant: the bootstrap Job used to lay out pg_wal before run's ReconcileWalDirectory executed, and the in-process replacement needs the same ordering verified with a dedicated WAL volume configured. The existing dedicated-WAL-volume test already creates a 3-instance cluster with walStorage, exercising both initdb and join, so it is extended with the same no-bootstrap-Job and no-restart assertions used by the other bootstrap suites rather than adding a near-duplicate spec. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
…t CA secret Phase-0 bootstrap now reads the recovery endpoint CA through the Kubernetes API instead of the bootstrap Job's Secret volume mount, so the instance Role must list that secret among the ones it can get. The Role already enumerated the origin Backup credentials and the external cluster endpoint CA, but not the endpoint CA coming from the origin Backup status or from a recovery Backup reference in the cluster spec. Without those entries every restore from a Backup reference against a TLS-protected object store crashloops on a forbidden Secret get. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
With in-process bootstrap the operator reuses the freed instance serial and recreates the pod with the same name within seconds, so the old primary pod name never stays absent long enough for the test to observe it. Capture the old primary pod UID before deletion and assert the pod is either gone or has been replaced by a new one carrying a different UID, keeping the existing timeout. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bootstrapping an instance today goes through a dedicated
batch/v1Job per mode (<instance>-initdb,-join,-pgbasebackup,-full-recovery,-snapshot-recovery): the operator creates the PVCs, waits for the Job to initialize them, marks them ready, and only then creates the instance Pod. Every bootstrap therefore pays two pod schedules and image pulls, reconciliation stays blocked while the Job runs, and the operator has to coordinate two consumers of the same PVCs.With this change the operator creates the instance Pod directly. When the pgdata PVC is not marked ready yet, the Pod carries a bootstrap overlay:
instance runreceives--bootstrap-mode=<mode>plus exactly the flags the Job builders used to emit, and the Pod is annotated withcnpg.io/bootstrapInstancefor observability. Before starting the instance manager,instance runinitializes PGDATA in-process with the same library code theinstance <mode>subcommands are built on, while a minimal webserver keeps the kubelet probes andwal-restoreworking: the startup probe is answered during the whole data copy instead of timing out, readiness stays negative, and/pg/statusanswers 503 so the operator sees the instance as not reporting yet. The overlay is deliberately left out of the pod-template annotation, so a freshly bootstrapped Pod is not considered drifted and no rollout is triggered.Completion is tracked at two levels. A marker file inside PGDATA records the mode and the instance identity (namespace, cluster name and UID, pod name): a container restarting mid-bootstrap finds no marker and retries the mode from scratch (except snapshot recovery, which resumes in place, since a snapshot-seeded PGDATA cannot be re-obtained), while a marker that traveled inside a volume snapshot belongs to a different identity and is ignored, so a snapshot-cloned PVC is bootstrapped properly instead of being mistaken for an already-initialized one. The
cnpg.io/pvcStatusannotation remains the operator-side memory, and now flips toreadyonly when the owning Pod becomes Ready.User-visible effects: no bootstrap Jobs are created anymore (the
pg_upgradeJob is unchanged); bootstrap progress and failures surface on the instance Pod itself, in its logs and restart count; and since the Pod's hostname is the instance name from the start (the Jobs used the Job name),application_nameand everything else derived fromPOD_NAMEis correct during the initial data copy too. Recovery sources protected by a private CA no longer rely on a Secret mounted over/controller/certificates: the bootstrap writes the CA file itself, which also removes a mount that would have shadowed the directory where the instance manager maintains the server certificates.Operator upgrades need no migration: the Job read/guard/cleanup paths are all kept, so an in-flight legacy bootstrap Job still blocks reconciliation, still marks its PVC ready on completion, and still surfaces its failure; from then on those paths only ever see the
pg_upgradeJob.Recovery through a CNPG-I plugin now runs inside the instance Pod, so plugins must serve the restore protocol from their instance sidecar (there is no restore Job carrying a dedicated sidecar anymore): cloudnative-pg/plugin-barman-cloud#1025 is the companion change. Removing the now-unused Job builders and the five bootstrap subcommands, and reworking how a failed bootstrap reaches
PhaseUnrecoverablewithout a JobbackoffLimit, are left to follow-ups; the failure policy is the open question tracked on the issue.The e2e bootstrap assertions are retargeted from Jobs to Pods and now also verify that no bootstrap Job is ever created; a
walStoragevariant covers the separate WAL volume layout.Closes #11228