Skip to content

CI: boot the release candidate against data fixtures from every published stable #990

Description

@frahlg

Why

Our own validation walks beta-by-beta on our sites, but users jump: the 2026-08-29 incident was a v1.15.0 box meeting v2.3.0 (#988). No automated check ever boots the candidate against the on-disk state a real older box brings — config.yaml written by an old version, state.db with old schema, driver overlays.

Proposal

An e2e job in the release pipeline (beta.yml, and as a make target locally):

  1. Keep a small library of data-directory fixtures, one per published stable (config.yaml + state.db + drivers dir), captured from the e2e stack right after that version's suite ran — not hand-written.
  2. For each fixture: start the candidate Core against a copy of it and require /api/health 200 and a fully started /api/status within a deadline.
  3. Assert zero ERROR-level config/load lines except known, listed repairs (e.g. Duplicate is_site_meter bricks boot: config rejection exits before the HTTP server binds #988's demotion warning).

This turns "an old config bricks the new version" from a field incident into a red check before a beta tag exists. Start with fixtures for the still-deployed stables (1.15.0, 2.0.0, 2.3.0) rather than every historical version.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions