Skip to content

Updater rollback does not survive a Restart: the failed target version comes back #989

Description

@frahlg

Field observation (2026-08-29, same incident as #988)

A v1.15.0 box updated toward v2.3.0. The new Core failed its health check (config rejection, #988) and the updater rolled back — the UI showed live data on v1.15.0 again and reported the previous image restored. The operator then pressed Restart, and the stack came up on v2.3.0 again (FTW starting version=v2.3.0 crash-looping in the logs), taking the box dark until SSH.

Suspected mechanism

Rollback restores the running container from the previous image, but the deploy pin (compose FTW_IMAGE_TAG / whatever records the target) appears to keep the failed target. Any later restart or recreate then resolves to the failed version instead of the restored one — the rollback is cosmetic across a container lifecycle boundary.

Expected

A rollback is not complete until the box would boot the restored version from every path: updater restart, operator Restart, docker compose up, host reboot. The failed target must not be re-applied unless the operator (or a new update attempt) chooses it again.

To verify first

Reproduce in a compose sandbox: pin version A, trigger an update to a B that fails health, confirm rollback, then docker compose up -d --force-recreate and check which version boots. The fix likely lives where the updater writes/undoes the pin, plus a test asserting the pin after rollback equals the restored version.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions