diff --git a/docs/specs/007-data-copy-schemas.md b/docs/specs/007-data-copy-schemas.md new file mode 100644 index 0000000..fdd5a15 --- /dev/null +++ b/docs/specs/007-data-copy-schemas.md @@ -0,0 +1,129 @@ +# S-007: Data copy schemas + +status: approved + +## Problem + +`cloudfall data copy` dumps the whole source database: `cloudfall_data_migration` runs +`pg_dump --no-owner --no-privileges --format custom` with no schema or extension filter +(`engine/ansible/roles/cloudfall_data_migration/tasks/main.yml`). Its checks read only +`public`: the empty-target check and `cloudfall_data_migration_count_query` in +`defaults/main.yml` both filter `table_schema = 'public'`. So the dump copies every schema, +but the receipt's "per-table row counts matched" vouches for one of them. `pg_restore` runs +without `--exit-on-error` or `--single-transaction`, so a restore that hits a missing +extension or role keeps going, exits non-zero, and leaves a half-restored database that the +empty-target check (on `public` only) may not even see. A Supabase source shows all of it at +once: platform schemas, Supabase-only extensions, and RLS policies on `anon` and +`authenticated` (cloudfall-dev/cloudfall#82). + +## Behaviour + +**Choosing the schemas.** `cloudfall data copy` gains a repeatable `--schema NAME` flag, +and `cloudfall migrate` gains a repeatable `--data-schema DATABASE=NAME` flag beside its +`--data DATABASE=PATH`. The `data_copy` MCP tool, generated from the command, gains the +same `schema` list. A flag, not a declared field, because a copy is configured by flags +today (`--database`, `--source-url-file`; `DataMigrateArgs` in `sdk/src/cloudfall/app.py`) +and the `Service` declares only the target: the source and what to take from it belong to +the one-time action, not to the desired state. + +A schema name must match `^[a-z_][a-z0-9_]*$` and be at most 63 characters. A name that +doesn't, a name given twice, or a `--data-schema` for a database with no `--data` stops the +command with `PRECONDITION` before any playbook runs. The plan shown without `--yes` names +the schemas it will copy, or says it copies every non-system schema of the source. + +**What gets copied.** With `--schema`, `pg_dump` gets one `--schema` per name and one +`--extension` per extension the preflight found the selected schemas need, so only those +schemas and their extensions reach the target. Without `--schema`, the dump command is +today's, unchanged, and the selected schemas are every schema of the source except +`pg_catalog`, `information_schema`, `pg_toast`, and the `pg_temp_*` and `pg_toast_temp_*` +families: what `pg_dump` dumps today. + +**Checks cover exactly the selected schemas.** The empty-target check counts tables in the +selected schemas of the target, so a table in another schema doesn't block the copy. The +row-count query reads every base table of the selected schemas on both sides and returns +one `schema,table,rows` line per table, ordered by schema and table. The source must hold +at least one table across them. + +**Preflight before dumping.** On the target host, after the empty-target check and before +`pg_dump`, the role reads the source and the target catalogs and collects every finding: + +- `missingSchemas`: a named schema the source doesn't have +- `extensions`: an extension the selected schemas need that is not in the target's + `pg_available_extensions`. "Need" means an object in a selected schema depends on it in + `pg_depend`, or it is installed in a selected schema; without `--schema`, every extension + installed in the source except `plpgsql` counts, since the whole dump creates them all +- `roles`: a role named in a row-level security policy (`pg_policy.polroles`) on a table in + a selected schema that the target's `pg_roles` lacks, such as `anon` or `authenticated` +- `authReferences`: a function in a selected schema whose definition references `auth.`, or + a policy whose `USING` or `WITH CHECK` expression does, when `auth` is not a selected + schema. Each is named by its qualified name, as a blocker the team must rewrite + +Any finding refuses the copy. Nothing is dumped, the target is untouched, no receipt is +written, and `data copy` (and the `data:` step of `cloudfall migrate`) exits +`PRECONDITION` with code `data_copy_preflight_refused` and the four lists, each item named, +in its error context. The role writes the findings to a file under the `--data-migrations` +directory for the SDK to read, so the list isn't cut to the 2000-character output tail +that `_execute_step_with_output` in `sdk/src/cloudfall/lifecycle.py` keeps. + +**No half-failed restore.** `pg_restore` runs with `--single-transaction`, which also stops +at the first error. A restore that fails rolls back whole: the target holds no table in the +selected schemas, no receipt is written, the command exits `ENGINE_STEP_FAILED`, and a +re-run needs no manual cleanup. + +**The receipt.** `DataMigrationReceipt` keeps its id, path, and fields. `spec.schemas` lists +the copied schemas in sorted order, and each `spec.tables[]` entry gains `schema` beside +`table` and `rows`. Both are additions; a reader of `table` and `rows` keeps working. + +**Compatibility.** Without `--schema`, what is dumped and restored stays the same as 0.7.0: +the whole database. The default stays "every non-system schema", not `public`, because +`public` was only what the checks read, never what the copy covered. Narrowing the default +would silently drop schemas existing migrations copy today (an ORM's migration table, such +as Drizzle's `drizzle.__drizzle_migrations`, or an app's own schemas) while the receipt still +said the counts matched. Two things do change for a copy without `--schema`. The row-count +check now covers every copied schema, so a mismatch outside `public` that passed before now +fails. And the preflight can refuse, but only for extensions and policy roles the target +lacks, which make today's restore fail too; `auth` is a selected schema in this mode +whenever the source has one, so `authReferences` stays empty. A Supabase project is copied +with `--schema public` (or the app's own schemas), the way the refusal points to. + +Touches `sdk/src/cloudfall/app.py` (`data copy`, `DataMigrateArgs`, `cloudfall migrate`), +`sdk/src/cloudfall/lifecycle.py` (`plan_data_migration`, `migrate_data`, +`preview_data_migration`), `sdk/src/cloudfall/migrate.py`, the `data_copy` MCP tool, +`engine/ansible/playbooks/data.yml`, `engine/ansible/roles/cloudfall_data_migration` +(`tasks/main.yml`, `defaults/main.yml`, `meta/argument_specs.yml`), and +`docs/neon-migration-guide.md` and `docs/render-migration-guide.md` where they describe the +copy. + +## Acceptance criteria + +- S-007-1: Against a source with tables in schemas `public`, `app`, and `audit`, `cloudfall data copy --database --source-url-file --schema public --schema app --yes` leaves the target with the tables of `public` and `app` only, and the receipt's `spec.schemas` is `["app", "public"]` +- S-007-2: Against the same source, the copy without `--schema` leaves the target with the tables of all three schemas, as 0.7.0 does, and the receipt's `spec.schemas` is `["app", "audit", "public"]` +- S-007-3: Each `spec.tables[]` entry of the receipt has `schema`, `table`, and `rows`, and the entries are exactly the base tables of the selected schemas +- S-007-4: The rendered row-count query for schemas `app` and `public` reads only those two schemas and returns one `schema,table,rows` line per base table, ordered by schema and table +- S-007-5: A target with a table in a selected schema refuses the copy with `ENGINE_STEP_FAILED` naming the table count, as today, and a target whose only table is in a schema not selected goes ahead +- S-007-6: `--schema missing` for a schema the source lacks exits `PRECONDITION` with code `data_copy_preflight_refused` and `missingSchemas` of `["missing"]`, and no dump file is written +- S-007-7: A selected table using a type from an extension that is not in the target's `pg_available_extensions` exits `PRECONDITION` with code `data_copy_preflight_refused` naming that extension under `extensions`, the target holds no new table, and no receipt is written +- S-007-8: A policy `TO anon, authenticated` on a selected table, with neither role on the target, exits `PRECONDITION` with code `data_copy_preflight_refused` and `roles` of `["anon", "authenticated"]` +- S-007-9: With `--schema public`, a function `public.current_tenant()` whose body calls `auth.uid()` and a policy on `public.notes` whose `USING` calls `auth.uid()` are both named under `authReferences`, and the copy is refused +- S-007-10: A source with a missing extension, a missing role, and an `auth.` reference at once is refused with all three named in one answer +- S-007-11: A restore that fails after the preflight passes exits `ENGINE_STEP_FAILED`, leaves no table in the selected schemas of the target, writes no receipt, and a second run against the same target passes the empty-target check +- S-007-12: `--schema 'Bad;Name'`, `--schema app --schema app`, or `cloudfall migrate --data-schema crm=public` without `--data crm=` exits `PRECONDITION` and runs no playbook +- S-007-13: `cloudfall migrate --data crm= --data-schema crm=public --yes` copies only `public` into `crm`, as `data copy --schema public` does +- S-007-14: `cloudfall data copy` without `--yes` names the selected schemas in its plan, or says it copies every non-system schema when no `--schema` is given + +## Out of scope + +- A Supabase migration guide: it needs a proving run on a real Supabase project before it goes on the site, and nothing here depends on it +- Recreating Supabase Auth, Storage, or Realtime: those stay with Supabase or get replaced by the team. The preflight names the `auth.` references to rewrite and stops there +- Rewriting `auth.` references or creating missing extensions and roles on the target: Cloudfall refuses and lists them; the operator decides how to provide them +- Narrowing the default to `public`: it would change what existing migrations copy, as the Compatibility paragraph says +- Copying owners, grants, or roles: the copy keeps `--no-owner --no-privileges` and restores as the declared owner +- Selecting single tables: schemas are the unit the dump, the checks, and the preflight share + +## Decisions relied on + +- none + +## Issues + +## Verification