improvement: add rebuild_table, a migration helper that rebuilds a table

documentation/topics/development/migrations-and-tasks.md

@@ -37,6 +37,63 @@ For single-step changes or when you know the final feature name:
 
 For more information on generating migrations, run `mix help ash_sqlite.generate_migrations` (the underlying task that is called by `mix ash.codegen`)
 
+### Changes SQLite cannot make in place
+
+SQLite can mostly add, rename and drop a column in place. Changing a column's type or default, adding or dropping a foreign key, changing a primary key, making a table `STRICT` and, as the adapters write it, making a column required or optional all need the table to be created again in its new shape, with the rows copied over.
+
+By default the generator writes a statement that fails when the migration runs, or leaves the change out (a renamed attribute that also changes, a table that becomes `STRICT`). With `--rebuild-tables`, or `rebuild_tables: true` in the repo's config, it writes a `rebuild_table` instead, which shows the table's new shape and what triggered it:
+
+```elixir
+def up do
+  # SQLite cannot change `comments` in place, so it is rebuilt from the resource's
+  # snapshot. Anything on the table that the resource does not describe (a column,
+  # index or trigger added by hand) is not kept. Changes:
+  # - dropping the foreign key comments_post_id_fkey
+  rebuild_table :comments do
+    add :id, :uuid, null: false, primary_key: true
+    add :post_id, :uuid
+  end
+end
+```
+
+A rebuild needs an `ecto_sql` that runs the `before_transaction/1` and `after_transaction/1` migration callbacks, which is how foreign keys are switched off outside the transaction. Without them, asking for rebuilds raises, and the generator only prints its hint, without suggesting one.
+
+The rebuild runs in one transaction with foreign keys off, and changes nothing if it fails. Columns both tables have are copied as they are. When the right value for the rows depends on your data, as when a column becomes required and has no default, a `REVIEW` comment says so and shows how to give them one.
+
+The custom statements of the resource are dropped (their `down`) before the rebuild and recreated (their `up`) after it, so what they hold is lost. That suits what can be dropped and recreated without losing anything, such as a trigger, a view or an index, and a full-text index, which is rebuilt from its table. Data that exists nowhere else does not belong in a custom statement. The full-text index needs a command that you add to the generated migration, after the statement's `up`:
+
+```elixir
+statement :items_fts do
+  up "CREATE VIRTUAL TABLE items_fts USING fts5(name, content='items', content_rowid='rowid')"
+  down "DROP TABLE items_fts"
+end
+```
+
+```elixir
+def up do
+  # SQLite cannot change `items` in place, so it is rebuilt ... Changes:
+  # ...
+  execute("""
+  DROP TABLE items_fts
+  """)
+
+  rebuild_table :items do
+    add :id, :uuid, null: false, primary_key: true
+    add :name, :text
+  end
+
+  execute("""
+  CREATE VIRTUAL TABLE items_fts USING fts5(name, content='items', content_rowid='rowid')
+  """)
+
+  execute("INSERT INTO items_fts(items_fts) VALUES('rebuild')")  # added
+end
+```
+
+The `default:` of an attribute (a number, string, boolean, atom, decimal, date or time) is then also the default of its column. SQLite cannot change a default in place either, so the first run rebuilds the tables whose attributes have one: `--dry-run` shows which.
+
+Snapshots keep the defaults, so a run without the option after one with it sees them as removed. If you use rebuilds, set `rebuild_tables: true` in the repo's config instead of passing the flag, so that every run agrees, `mix ash.codegen --check` in CI included.
+
 ### Regenerating Migrations
 
 Often, you will run into a situation where you want to make a slight change to a resource after you've already generated and run migrations. If you are using git and would like to undo those changes, then regenerate the migrations, this script may prove useful:

lib/migration.ex

@@ -0,0 +1,242 @@
+# SPDX-FileCopyrightText: 2023 ash_sqlite contributors <https://github.com/ash-project/ash_sqlite/graphs/contributors>
+#
+# SPDX-License-Identifier: MIT
+
+defmodule AshSqlite.Migration do
+  @moduledoc """
+  Helpers for migrations that rebuild a table.
+
+  SQLite can mostly add, rename and drop columns in place. For most other changes it is
+  [documented](https://www.sqlite.org/lang_altertable.html#otheralter) to create the table in
+  its new shape, copy the rows, drop the old table and rename the new one:
+
+      use AshSqlite.Migration
+
+      def up do
+        rebuild_table :posts do
+          add :id, :uuid, null: false, primary_key: true
+          add :title, :text
+        end
+      end
+
+  `use AshSqlite.Migration` does what `use Ecto.Migration` does, and adds what makes a rebuild
+  safe: foreign keys are off while it runs (dropping a table would otherwise run the `ON DELETE`
+  actions of the tables that point to it) and they are checked before it commits. A rebuild
+  either completes or changes nothing.
+
+  Foreign keys can only be switched off by an `ecto_sql` that runs the `before_transaction/1`
+  and `after_transaction/1` migration callbacks. Without them, a rebuild refuses to run
+  unless the connection has foreign keys off already.
+  """
+
+  @pragmas_key :ash_sqlite_rebuild_pragmas
+
+  @doc false
+  defmacro __using__(_opts) do
+    quote do
+      use Ecto.Migration
+      import AshSqlite.Migration, only: [rebuild_table: 2, rebuild_table: 3]
+
+      defdelegate before_transaction(repo), to: AshSqlite.Migration, as: :__before_transaction__
+      defdelegate after_transaction(repo), to: AshSqlite.Migration, as: :__after_transaction__
+
+      def before_commit, do: AshSqlite.Migration.__check_foreign_keys__(repo())
+    end
+  end
+
+  # An `ecto_sql` without the callbacks never calls the ones `use AshSqlite.Migration` adds.
+  @doc false
+  def rebuild_supported? do
+    {:before_transaction, 1} in Ecto.Migration.behaviour_info(:optional_callbacks)
+  end
+
+  # `PRAGMA foreign_keys` does nothing inside a transaction, so it is set before Ecto opens one,
+  # on the connection it will use. The previous values are kept to put them back afterwards.
+  @doc false
+  def __before_transaction__(repo) do
+    Process.put(@pragmas_key, {pragma(repo, "foreign_keys"), pragma(repo, "legacy_alter_table")})
+    quiet!(repo, "PRAGMA foreign_keys = OFF")
+  end
+
+  # `legacy_alter_table` is put back too: a rebuild that fails does not turn it off itself.
+  @doc false
+  def __after_transaction__(repo) do
+    case Process.delete(@pragmas_key) do
+      nil ->
+        :ok
+
+      {foreign_keys, legacy_alter_table} ->
+        quiet!(repo, "PRAGMA legacy_alter_table = #{legacy_alter_table}")
+        quiet!(repo, "PRAGMA foreign_keys = #{foreign_keys}")
+    end
+  end
+
+  @doc false
+  def __check_can_rebuild__(repo, table) do
+    check_foreign_keys_off!(repo, table)
+    check_in_transaction!(repo, table)
+  end
+
+  # Ecto does not run the callbacks for a migration without a transaction
+  # (`@disable_ddl_transaction true`), and dropping a table with foreign keys on deletes
+  # from the tables that reference it.
+  defp check_foreign_keys_off!(repo, table) do
+    if pragma(repo, "foreign_keys") != 0 do
+      raise "refusing to rebuild #{table}: foreign keys are on, so dropping it would run " <>
+              "the ON DELETE actions of the tables that reference it. They are switched " <>
+              "off by the callbacks `use AshSqlite.Migration` adds. Ecto runs them only when " <>
+              "the migration runs in its own transaction (remove `@disable_ddl_transaction " <>
+              "true`), and only in an `ecto_sql` that has them."
+    end
+  end
+
+  # Without one nothing keeps other connections from writing during the rebuild, and a failure
+  # leaves the table half rebuilt.
+  defp check_in_transaction!(repo, table) do
+    unless repo.in_transaction?() do
+      raise "refusing to rebuild #{table}: the migration is not in a transaction, so a rebuild " <>
+              "could lose concurrent writes or fail halfway. Remove `@disable_ddl_transaction true`."
+    end
+  end
+
+  @doc false
+  def __legacy_alter_table__(repo, value),
+    do: quiet!(repo, "PRAGMA legacy_alter_table = #{value}")
+
+  @doc """
+  Rebuilds `table` with the shape `block` describes, as the block of a `create table`.
+
+  Every column the old and the new table both have is copied as it is. A column the
+  old table did not have gets the `DEFAULT` of the new table.
+
+  Options:
+
+    * `:copy` - what to copy differently, as a keyword list from the new column to
+      where its value comes from: `new: :old` copies the column `old` into `new` (a
+      rename), and `new: "sql"` copies an SQL expression, which can look at the old
+      table's columns (for example `COALESCE(name, 'unknown')` for a column that is
+      becoming required, or `'unknown'` for a new required one).
+    * any other option is an option of the temporary `table/2`, for example
+      `options: "STRICT"`.
+  """
+  defmacro rebuild_table(table, opts \\ [], do: block) do
+    quote do
+      table = unquote(table)
+      {copy, opts} = Keyword.pop(unquote(opts), :copy, [])
+      temporary = "#{table}_rebuild"
+
+      execute(fn -> AshSqlite.Migration.__check_can_rebuild__(repo(), table) end)
+
+      create table(temporary, [primary_key: false] ++ opts) do
+        unquote(block)
+      end
+
+      # the columns of both tables are only known once the new one exists
+      execute(fn -> AshSqlite.Migration.__copy_rows__(repo(), table, temporary, copy) end)
+
+      # While the old table is dropped, a view or trigger that mentions it points at nothing, and
+      # the rename that follows refuses. The legacy behaviour leaves them alone. It is not on for
+      # the rest of the migration, so that a `rename table` in it still updates what mentions it.
+      execute(fn -> AshSqlite.Migration.__legacy_alter_table__(repo(), 1) end)
+      drop(table(table))
+      rename(table(temporary), to: table(table))
+      execute(fn -> AshSqlite.Migration.__legacy_alter_table__(repo(), 0) end)
+    end
+  end
+
+  @doc false
+  def __copy_rows__(repo, table, temporary, copy) do
+    copy_sequence(repo, table, temporary)
+
+    old_columns = column_names(repo, table)
+    new_columns = column_names(repo, temporary)
+
+    unknown =
+      copy |> Keyword.keys() |> Enum.map(&to_string/1) |> Enum.reject(&(&1 in new_columns))
+
+    if unknown != [] do
+      raise ArgumentError,
+            "`copy:` for #{table} names #{Enum.join(unknown, ", ")}, which the new table " <>
+              "does not have (it has #{Enum.join(new_columns, ", ")})"
+    end
+
+    expressions =
+      for name <- new_columns,
+          expression = copy_expression(copy, name, old_columns),
+          do: {name, expression}
+
+    if expressions == [] do
+      raise ArgumentError,
+            "`rebuild_table` for #{table} would copy nothing: the new table has no column in " <>
+              "common with the old one and no `copy:`, so its rows would be lost"
+    end
+
+    names = Enum.map_join(expressions, ", ", fn {name, _} -> quote_name(name) end)
+    selected = Enum.map_join(expressions, ", ", fn {_, expression} -> expression end)
+
+    query!(
+      repo,
+      "INSERT INTO #{quote_name(temporary)} (#{names}) SELECT #{selected} FROM #{quote_name(table)}"
+    )
+  end
+
+  # An AUTOINCREMENT table keeps the highest id it ever handed out in `sqlite_sequence`, which
+  # dropping the table deletes: ids that were used and deleted would be handed out again.
+  defp copy_sequence(repo, table, temporary) do
+    %{rows: [[autoincrement]]} =
+      quiet!(
+        repo,
+        "SELECT count(*) FROM sqlite_master WHERE name = ? AND sql GLOB '*PRIMARY KEY AUTOINCREMENT*'",
+        [temporary]
+      )
+
+    if autoincrement > 0 do
+      query!(
+        repo,
+        "INSERT INTO sqlite_sequence (name, seq) SELECT ?, seq FROM sqlite_sequence WHERE name = ?",
+        [temporary, to_string(table)]
+      )
+    end
+  end
+
+  defp copy_expression(copy, name, old_columns) do
+    case Enum.find(copy, fn {new, _} -> to_string(new) == name end) do
+      {_, from} when is_atom(from) -> quote_name(from)
+      {_, sql} when is_binary(sql) -> sql
+      nil -> if name in old_columns, do: quote_name(name)
+    end
+  end
+
+  defp column_names(repo, table) do
+    %{rows: rows} = quiet!(repo, "PRAGMA table_info(#{quote_name(table)})")
+    Enum.map(rows, fn [_cid, name | _] -> name end)
+  end
+
+  defp quote_name(name), do: ~s("#{name}")
+
+  # not `repo.query!/1`: ecto_libsql's repo has none
+  # sobelow_skip ["SQL.Query"]
+  defp query!(repo, statement, params \\ []),
+    do: Ecto.Adapters.SQL.query!(repo, statement, params)
+
+  # sobelow_skip ["SQL.Query"]
+  defp quiet!(repo, statement, params \\ []),
+    do: Ecto.Adapters.SQL.query!(repo, statement, params, log: false)
+
+  defp pragma(repo, name) do
+    %{rows: [[value]]} = quiet!(repo, "PRAGMA #{name}")
+    value
+  end
+
+  @doc false
+  def __check_foreign_keys__(repo) do
+    case quiet!(repo, "PRAGMA foreign_key_check") do
+      %{rows: []} ->
+        :ok
+
+      %{rows: rows} ->
+        raise "foreign key check failed after rebuilding tables (table, rowid, parent, constraint): " <>
+                inspect(rows)
+    end
+  end
+end