Skip to content

build(build): fail CI when a dependency update doesn't resolve or regenerate #873

@algattik

Description

@algattik

Summary

Dependabot can open a PR that edits pyproject.toml into a state that does not
resolve
, or that resolves to an import-incompatible set — and today that PR
passes CI green, because no workflow re-resolves the manifest or regenerates the
generated dependency files. The breakage surfaces later, at container build or
training runtime, instead of on the PR.

PR #779 mitigates the churn from such updates; this follow-up makes the
breaking ones fail CI. It covers both generated-file mechanisms in the repo.

The core problem

Dependabot edits pyproject.toml directly. Two failure modes, neither caught
today:

  1. Unresolvable / incompatible manifest. A bump can leave the constraint set
    with no valid solution, or one that breaks at import time.
    Real example: security(deps): bump the training-dependencies group in /training/rl with 36 updates #743 bumped marshmallow to 4.x against
    azure-ai-ml==1.33.0, which imports symbols 4.x removed. Caught only by
    manual review, hand-pinned back to marshmallow==3.26.2. CI was green
    throughout.
  2. Stale generated file. Even a valid bump leaves the generated dependency
    file out of step, because no Dependabot ecosystem regenerates a
    uv pip compile requirements.txt, and uv sync in CI re-resolves silently
    rather than asserting the existing uv.lock.

Two mechanisms, two gates

A. uv pip compilerequirements.txt (committed, installed at runtime)

  • training/rl/requirements.txttraining/rl/pyproject.toml
  • training/il/lerobot/requirements.txttraining/il/lerobot/pyproject.toml

Installed at runtime via uv pip install --no-deps --requirement requirements.txt
in training/rl/scripts/train.sh and
training/il/scripts/lerobot/azureml-train-entry.sh, so they must stay
committed. CI never touches them today (pytest-training.yml installs from
pyproject.toml), so they are neither validated nor exercised.

Gate — recompile in place, then diff:

uv pip compile pyproject.toml -o requirements.txt \
  --python-version <3.11 rl | 3.12 il> --python-platform manylinux_2_28_x86_64  # fails if manifest unresolvable
git diff --exit-code requirements.txt                                           # fails if generated file stale
  • Compile into the committed file so uv reads existing pins as
    preferences; without --upgrade, routine transitive minor releases produce no
    diff (verified empirically on rl/il). The diff fires only when
    pyproject.toml changed without a matching regen.
  • --python-version is load-bearing: requires-python is a range/floor
    (il is >=3.12), not a resolution target. uv resolves against the ambient
    interpreter unless pinned, so the output would vary by runner. Pin it to each
    component's container Python (rl→3.11, il→3.12).

B. uv.lock projects

./uv.lock, data-management/viewer/uv.lock,
data-management/viewer/backend/uv.lock, data-pipeline/uv.lock,
evaluation/uv.lock.

uv sync in the pytest workflows does not assert these — and three
workflows sync the root project, so data-pipeline/uv.lock and
data-management/viewer/uv.lock are currently asserted by nothing.

Gate — assert each lock is up to date:

uv lock --check   # per project; fails if uv.lock is out of date vs pyproject.toml

Adding --locked to the existing uv sync calls is cheaper but incomplete
(misses the two unsynced locks), so an explicit per-project uv lock --check
is preferred.

Implementation notes

  • Both gates follow the existing drift-gate convention (table-format.yml,
    terraform-docs-check.yml: regenerate, fail on diff). Support soft-fail.
  • Pin the uv version in the workflow so the resolver cannot introduce diffs.
  • Document the exact per-file requirements.txt regeneration command (e.g. in
    docs/contributing/component-updates.md). The two committed headers currently
    disagree (rl: --python-version 3.11 ... --refresh; il:
    --python-version 3.12) — pin them.

Acceptance criteria

  • CI recompiles both requirements.txt files in place and fails on an
    unresolvable manifest or a git diff, with uv pinned.
  • CI runs uv lock --check (or equivalent) for all five uv.lock projects.
  • The per-file requirements.txt regeneration command is documented.
  • build(deps): align dependabot config with uv ecosystem and pin caps #779 remains the Dependabot-churn mitigation; this issue closes the
    resolve/regenerate gap.

Notes

  • Known residual: a yanked/removed pinned version forces uv off the pin and
    trips the requirements.txt gate. Rare, and surfacing it is desirable.
  • Scope is catching broken/stale dependency updates in CI — not a redesign of
    the runtime dependency model.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-triageNeeds initial review and categorization

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions