perf(cli): reuse shadow baseline cache in migration squash and pgadmin diff - #6220
perf(cli): reuse shadow baseline cache in migration squash and pgadmin diff#6220avallete wants to merge 113 commits into
Conversation
…lta-next # Conflicts: # apps/cli-go/cmd/db.go # apps/cli/docs/go-cli-porting-status.md # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/diff/diff.layers.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/pull/pull.handler.ts # apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts # apps/cli/src/legacy/commands/db/pull/pull.layers.ts # apps/cli/src/legacy/commands/db/push/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/reset/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.seam.service.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts # apps/cli/src/legacy/shared/legacy-db-connection.sql-pg.layer.ts # apps/cli/src/legacy/shared/legacy-http-errors.ts # apps/cli/src/legacy/shared/legacy-migration-apply.ts # apps/cli/src/legacy/shared/legacy-migration-apply.unit.test.ts # apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts # packages/api/src/effect.ts # packages/api/src/internal/client.ts # packages/api/src/internal/client.unit.test.ts
…lta-next # Conflicts: # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts
…n diff The two remaining shadow-database consumers still provisioned bare, uncached shadows. Both now acquire through `legacyWithShadowDatabase` (`shadow-cache.ts`), the same seam `db diff`'s native branch and `db pull`'s migration path already use, so a key-matching PGDATA snapshot is restored in a few seconds instead of cold-provisioning the platform baseline every run. `migration squash` keeps its dump/apply/dump sequence intact by resuming at the baseline seam: `legacyOpenShadowBaselineSession` — Go's `SetupShadowDatabase`/`MigrateShadowDatabase` shared prologue, extracted from `shadow-database.ts` so all three compositions share one implementation — hands back the still-open session, and squash's before-dump / apply-migrations / after-dump / full-dump steps are unchanged. Unlike diff/pull it passes no `webhooks` override, matching its long-standing config-following `SetupDatabase` call, so it keys to its own snapshots. `db diff --use-pgadmin` becomes structurally identical to the native branch: same cached acquire, same forced-on `webhooks: "enabled"` policy (so it shares the native branch's tars), with `legacyMigrateShadowDatabase` now told whether the cluster already carries the baseline. A warm shadow skips `SetupDatabase`, so `Initialising schema...` / `Seeding globals from roles.sql...` no longer print and the PG15+ one-shot migrate jobs do not run — progress text reflects the work actually performed. Everything else, including the dumped artifacts and the differ output, is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Supabase CLI previewnpx --yes https://pkg.pr.new/supabase/cli/supabase@d6289e62c2e0685e53de5dd402153eabec34242cPreview package for commit |
💡 Codex ReviewWhen AGENTS.md reference: apps/cli/AGENTS.md:L524-L526 cli/apps/cli/src/legacy/shared/db-bootstrap/health-check.ts Lines 585 to 588 in d6289e6 When the published port blackholes TCP connections rather than refusing them, each probe can consume its full 2-second connect timeout, while this schedule still permits AGENTS.md reference: apps/cli/AGENTS.md:L53-L56 When When another live test runs later in the same Vitest worker, this direct ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…cli-pr-conflicts-529506 # Conflicts: # apps/cli/docs/go-cli-divergences.md # apps/cli/package.json # apps/cli/src/legacy/commands/db/diff/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/pull/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/pull/pull.handler.ts # apps/cli/src/legacy/commands/db/pull/pull.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.flow.unit.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/declarative.orchestrate.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.command.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/generate/generate.integration.test.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/SIDE_EFFECTS.md # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.handler.ts # apps/cli/src/legacy/commands/db/schema/declarative/sync/sync.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.layer.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.integration.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-engine.next.layer.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-adapter.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-artifacts.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.service.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta.write.unit.test.ts # apps/cli/src/legacy/commands/db/shared/legacy-shadow-source.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.ts # apps/cli/src/legacy/shared/db-bootstrap/db-setup.unit.test.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts # apps/cli/src/legacy/shared/legacy-db-push-core.ts # apps/cli/src/legacy/shared/legacy-pgdelta-next-flag.ts # apps/cli/src/legacy/shared/legacy-pgdelta-next-flag.unit.test.ts # apps/cli/tests/fixtures/compiled-libpg-query.ts # pnpm-lock.yaml # pnpm-workspace.yaml
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bypass pg-delta's same-identity guard by snapshot lineage (same cache key), not by requiring both plan shadows to be warm restores — the first cold plan exports the tar its declarative sibling restores, and the guard rejected that clone (Codex P1). - Fold LEGACY_START_ENABLE_DATABASE_WEBHOOKS_SQL into the baseline SQL digest so editing it invalidates stale pg_net baselines. - Bound legacyWaitForShadowReady by elapsed time (+ one connect allowance) so hung dials cannot stretch the wait ~3x its budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eview findings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l run db diff/db pull hardcoded webhooks: "enabled" into the cache key even when pgdelta-next's config-following migrate would run, and db diff's explicit migrations-catalog path omitted the policy entirely while its provisioner forces pg_net on — either mismatch lets the two engines restore each other's tars. Derive the key from migrationMode at the diff/pull seams, declare "enabled" at the explicit catalog call, and drop exportViaShadowCatalog's opts default so callers must state their policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rapid-fire publishes can land within the filesystem's timestamp granularity, making the "oldest" tar ambiguous and the eviction pick arbitrary (CI-only flake). Age the first tar explicitly — the test asserts the keep-cap, not tie-breaking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
db diff and db pull each gain a cache-enabled scenario proving a legacy-engine baseline tar is never restored into a pg-delta-next run (and vice versa each publishes its own key). Mutation-verified: reverting either call site to a hardcoded webhooks: "enabled" fails the new test at the warm-restore assertion. The shadow-cache suite's stateful Docker model is hoisted into tests/helpers for reuse; the stateless shadow spawner mock stays byte-identical for existing callers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…key token - A warm hit now scans the tar's entry headers for the cluster marker (data/PG_VERSION) before restoring: an empty or foreign-but-valid tar no longer initdbs a bare cluster that reports baselinePresent and skips setup — it is discarded as suspect and the run cold-provisions. - The cache key hashes the EFFECTIVE api.auto_expose_new_tables behavior (grants kept vs revoked): unset and explicit false execute identical revoke SQL, so they no longer force separate ~90MB tars. - Docs: generate joins the cache divergence entry; roles.sql cache-key reads listed across diff/pull/sync/generate SIDE_EFFECTS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The export now writes data/SUPABASE_BASELINE into the stopped shadow's PGDATA right before the outbound docker cp, and warm-hit validation requires it alongside data/PG_VERSION — a valid bare-cluster tar is discarded as suspect instead of restoring into baselinePresent: true and skipping setup. Only snapshotBaseline stamps, strictly after the platform baseline, so a snapshot taken too early can never carry the marker (regression guard). Uppercase name keeps the entry at the front of docker cp's sorted tar so warm validations stay a first-blocks read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tants The baseline marker's content is now the snapshot's own cache key and warm validation verifies it, so a valid snapshot copied over another key's filename is discarded (wrong-key verdict, distinct from a missing marker) instead of restoring a mismatched baseline. The baseline digest (renamed baseline_embedded_digest) now also hashes the Realtime seed constants persisted by the one-shot job (tenant id, encryption key, db user/name/port), so editing them without an image bump invalidates stale tars. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cli-pr-conflicts-529506 # Conflicts: # apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts
Docker-aware gate (describeDockerLive), ephemeral shadow port instead of a fixed one, best-effort network removal in the scope finalizer, and scoped SUPABASE_HOME/SUPABASE_SHADOW_CACHE overrides restored on every outcome. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One runSupabaseLive golden path: init + minimal start, then db diff --local --use-pg-delta twice against the same SUPABASE_HOME — the cold run must export the tar (shadow-debug: baseline-export), the warm run must restore it (baseline-restore, refreshed mtime, identical stdout). Replaces the in-process legacyAcquireShadowDatabase calls, which could stay green while the command wiring or env propagation broke; mechanics coverage lives in the integration suite. Shadow port comes from SUPABASE_DB_SHADOW_PORT with a bounded retry on bind conflicts — true reservation is impossible since Docker must bind the port itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n's pid Eight candidates from a pid-seeded base in the IANA dynamic range replace the fixed 54987/54988 pair, so independently concurrent runs start from different bases and an occupied port costs one retry step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-exec-wrappers # Conflicts: # packages/stack/src/ServiceCatalog.ts
…-parallel-provision-fn9uue # Conflicts: # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.ts # apps/cli/src/legacy/commands/db/shared/legacy-pgdelta-next-shadow.layer.unit.test.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-cache.ts
…te/shadow-cache-squash-pgadmin # Conflicts: # apps/cli/docs/go-cli-divergences.md # apps/cli/src/legacy/commands/db/diff/diff.handler.ts # apps/cli/src/legacy/commands/db/diff/diff.integration.test.ts # apps/cli/src/legacy/commands/db/pull/pull.handler.ts # apps/cli/src/legacy/commands/migration/squash/squash.integration.test.ts # apps/cli/src/legacy/shared/db-bootstrap/shadow-database.ts # apps/cli/src/legacy/shared/legacy-pgdelta.cache.ts
Summary
Stacked on #6215 (← #6203 ← #6184 ← #6102).
migration squashanddb diff --use-pgadminwere the last two shadow-database consumers still provisioning bare cold shadows — every run re-paid the full platform baseline (init schema + the PG15+ realtime/storage/auth one-shot migrate jobs, ~15s) that the shadow-baseline cache already eliminates fordb diff/db pull. Both now provision throughlegacyWithShadowDatabaseand warm-restore the cached pg_data tar when eligible.The shared prologue (connect → prelude →
legacySetupDatabase, with the warm/cold/cold-snapshotting branches) is extracted intolegacyOpenShadowBaselineSessioninshadow-database.ts;legacySetupShadowDatabaseand the migrate path delegate to it, and squash calls it directly so its pre-migration auth/storage dumps stay exactly at the baseline seam. Squash keys config-following (its baseline always followedconfig.toml); pgadmin keyswebhooks: "enabled"to share the legacy migra branch's forced-pg_nettars.Two defects surfaced while wiring this in, both fixed here:
db pullanddb diff's native branch hardcoded{ webhooks: "enabled" }as the cache-key opts even in pg-delta next mode, where the actual provisioning (legacyMigrateNextShadowDatabase) followsconfig.toml. On a project with webhooks disabled, a cold next-mode run published apg_net-less tar under the "enabled" key — which the newly cached pgadmin branch (and the reverse direction) could then silently warm-restore. The key opts now follow the migration mode at all call sites;legacyResolveMigrationsCatalogRefinlegacy-pgdelta.cache.tshad the same drift (keyed config-following, provisioned forced-on) and is fixed too. A regression test asserts a next-mode diff and a pgadmin diff on a webhooks-disabled config never share tars (verified failing pre-fix).legacyWaitForHealthyServices(Docker HEALTHCHECK, first probe at t+10s, ~6.5s pure wait) instead of thelegacyWaitForShadowReadyconnect probe every other cached consumer uses. Swapped, so the warm restore actually lands its win. The now-deadneverHealthymock knob is deleted; the interrupt/timeout tests model a connect-refusing shadow instead.Test-helper cleanups ride along: a shared
withLegacyShadowCacheEnabledintests/helpers/legacy-mocks.tsreplaces per-file env stash/restore copies, and the squash test useslegacyShadowBaselineCacheDirinstead of a hand-built path.Transcript note: on a warm hit both commands skip the
Initialising schema.../Seeding globals from roles.sql...lines, matching the existing warm behavior ofdb diff/db pull. All dumps, migration-replay lines, and result output are unchanged.Linked issue
Closes #
open-for-contributionlabel (or I'm a Supabase maintainer).Checklist
fix(cli): …).pnpm check:allandpnpm testpass for the workspace(s) I touched.🤖 Generated with Claude Code