Skip to content

Deploy has no release-phase hook, so database migrations run by hand after the health gate #88

Open

Description

@shipmill-romamo

Problem

Found while deploying Abecra's staging on 2026-10-19. The API's schema is managed by Alembic. cloudfall_deploy unpacks the release, runs uv sync, switches current, restarts the unit and gates on the health check; there is no step where a declared command runs against the new release before (or after) traffic moves to it. component.schema.json has no field for one: service.command is the only command a component declares.

The workaround, after cloudfall deploy returns, is an SSH command that sources shared/env/<component>.env by hand and runs .venv/bin/alembic upgrade head from current/ as the app user. That means:

  • the health gate checks a release against the old schema, so a release whose code needs the new schema either fails the gate (and rolls back) or passes it and then serves errors until the manual step runs
  • a failed migration leaves no receipt and does not stop or roll back the deploy
  • the environment the migration sees is reassembled by hand instead of the unit's own EnvironmentFile

Proposal

A declared release command, run by the deploy role on one server after uv sync and before the current switch:

spec:
  release:
    command: [.venv/bin/alembic, upgrade, head]
    workingDirectory: services/api
    timeoutSeconds: 300
  • Runs as the application's linuxUser, in the new release directory, with the same environment file the systemd unit gets
  • Runs once per deploy even when deployment.servers lists several servers (on the first server only, recorded in the receipt)
  • A non-zero exit or timeout fails the deploy before current moves; the previous release keeps serving. The command's exit code and the tail of its output go into the release receipt
  • cloudfall deploy without --yes shows the command in the plan
  • Rollback does not run it (downgrades stay a human decision); the plan says so

Alternatives

  • Run migrations at application start: works for one server but races on several, and turns a failed migration into a failed health check with less evidence
  • A separate cloudfall run <component> -- <command> one-off command: useful on its own (shells, backfills), but it leaves ordering with the deploy to the operator

Acceptance

  • A component with release.command that adds a table: after cloudfall deploy --yes the table exists, and the receipt records the command, exit code 0 and output tail
  • A release command that exits 1: the deploy fails with ENGINE_STEP_FAILED, current still points at the previous release, and the service was not restarted
  • With two servers declared, the command runs once
  • The command sees every variable from the component's environment and secretRefs

Activity

  1. shipmill-romamo commented on Oct 20, 2026

    @shipmill-romamo
    ContributorAuthor

    Triage: feature.

    Confirmed on main (348ed73): cloudfall_deploy unpacks, runs uv sync, switches current, restarts and gates on health, and component.schema.json declares no command besides service.command. A declared release command run before the current switch is a new component field, a new deploy step, new receipt fields and a once-per-deploy rule across servers, so it gets a spec with acceptance criteria first.

    Writing it alongside #87's spec, since both extend the deploy plan and receipts.

    The spec PR is being written now (docs/specs/005-release-command.md); I'll put its link and the hold line here. Merging it is the approval.

  2. shipmill-romamo commented on Oct 20, 2026

    @shipmill-romamo
    ContributorAuthor

    The spec is in #102 (docs/specs/005-release-command.md). Where it departs from the issue, following the code: a failed release command writes no release receipt, because ReleaseReceipt only records a passed health check, so the evidence is the ENGINE_STEP_FAILED message. "First server" means the first entry of deployment.servers, and the deploy runs that server first. The output tail (2000 characters) is stored without redaction. Each of these is a choice to settle in review.

    On hold: the build waits on #102, the spec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions