Skip to content

ci: let the land gate stop just ci at the first failed module - #7466

Merged
kixelated merged 1 commit into
mainfrom
ci/gate-fail-fast
Oct 31, 2026
Merged

kixelated merged 1 commit into
mainfrom
ci/gate-fail-fast

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Problem

The host land gate ejects a whole batch when any module fails, yet just ci keeps every lane running to the end. On 2026-10-31, moq's failed batches ran 20-50 more minutes after their first failure: check quest in 3 s, check rs in about 40 s, test rs in 226 s. No moq PR landed between 11:27 and 16:30 while 60+ waited.

Approach

  • CHECK_FAIL_FAST=1 is an opt-in that the land gate sets.
    • A lane returns at its first failed module.
    • The dispatcher cancels the running lanes, using the existing cancel, which records each one as (cancelled).
    • It starts no more lanes.
    • The failed module is still named, and the seconds-per-module report still prints.
  • Without the variable, just ci and just check still run every lane to the end, as the dispatcher intends: one unavailable check shouldn't hide unrelated failures.
  • Host side: kixelated/nixos's moq check command exports CHECK_FAIL_FAST=1. That lands in kixelated/nixos#16 once this merges.

Impact

  • No public API changes.

Alternatives

  • The host watches just ci's output and kills the run at the first ci: … failed line. Rejected as a workaround for something the dispatcher should own.
  • Dropping the batching gate for per-PR CI and auto-merge. Rejected by the maintainer, because PRs that pass alone but break together would break main.

Follow-ups

  • This is the planned quest/m0/ci/ci-fail-fast.md from #7456, which is still queued. That quest is deleted once both land.

Checks

  • python3 sh/dispatch-query.test.py: the two new tests pass, and both fail without the change. The serial one fails outright; the parked one hangs until its timeout. The existing failure and continuation tests still pass.
  • just check passes.
  • just test passes.

Review

quest review --base origin/main (Codex), round 1 on head c367838: no findings.

The maintainer approved landing this past the queue with the admin bypass, since the queue it unblocks is the one it would wait in.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

The host land gate ejects a failing batch whatever else fails, yet each
failed batch on 2026-10-31 ran 20-50 more minutes after its first failure
(check quest in 3s, check rs in 40s). CHECK_FAIL_FAST=1 stops a lane at its
first failed module, cancels the running lanes, and starts no more. Without
it, just ci still runs every lane to the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated merged commit 6a260cd into main Oct 31, 2026
3 checks passed
@kixelated
kixelated deleted the ci/gate-fail-fast branch October 31, 2026 23:16

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Opt-in fail-fast in the dispatcher — CHECK_FAIL_FAST=1 makes run_lane return at its first failed phase and the main loop set stopping, cancel the running lanes via the existing cancel, and start no further lanes; unset, every lane still runs to the end.
  • Cancelled lanes stay distinct from failures — the [[ -z "$stopping" ]] guard keeps a cancelled lane's nonzero status out of failed, so only the first failure is named while cancelled phases show as (cancelled).
  • Tests — a parallel test proves parked rs/cpp lanes are cancelled and later lanes never start, and a serial test proves a first failure stops the run with exactly one failure line.
  • Comment updates — justfile and the sh/dispatch.sh header document the new host-gate variable.

Verified locally: the full sh/dispatch-query.test.py (31 tests) passes, shellcheck sh/dispatch.sh is clean, and both new tests fail without the change (the parallel one hangs to its 60s timeout; the serial one reports two failures instead of one).

One environment note, not a code change: the two new tests need zizmor on PATH because gh is a repository-wide module selected on every diff and CI_STRICT=1 turns its absence into an error before any lane runs. That tool is present in CI's dev shell, so it only affects bare checkouts.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash (free via Pullfrog for OSS) | 𝕏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant