Skip to content

fix(scheduler): idle-fill must not out-race a repeated infra_error (AGT-4305) - #623

Merged
unohee merged 1 commit into
mainfrom
fix/agt-4305-infra-idle-fill-streak
Sep 10, 2026
Merged

fix(scheduler): idle-fill must not out-race a repeated infra_error (AGT-4305)#623
unohee merged 1 commit into
mainfrom
fix/agt-4305-infra-idle-fill-streak

Conversation

@unohee

@unohee unohee commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • AGT-4257's idle-fill unconditionally cleared a task's infra_error backoff whenever a free slot was available, with no cap on how many times the same issue could be bypassed for the same recurring infra_error. On vela, AX-1272's reviewer stage died to the same OpenRouter 360s timeout 9 of ~13 attempts over 4h+, retried every ~90s instead of waiting the intended 1h backoff, because it was the only runnable candidate.
  • Adds consecutiveInfraErrorCounts (issueId → streak), incremented on every infra_error result and reset at every existing terminal/recovery site. Both idle-fill bypass points (legacy failedTaskRetryTimes gate, and the durable-ledger RETRY_AT idleLiftable check) now refuse to bypass once an issue's current infra_error streak reaches 3, honoring the real 1h backoff instead. The durable-ledger check is additionally scoped to durableRun.lastErrorCode === 'infra_error' so a later, unrelated RETRY_AT (e.g. an ordinary rejection) on the same issue keeps normal idle-fill treatment.
  • Two adjacent gaps found during review are tracked separately rather than expanding this fix's scope: AGT-4306 (idle-fill also un-parks an INFRA_CIRCUIT_PARK_REASON NEEDS_HUMAN with no operator action) and AGT-4307 (a thrown/rejected executor error surfaces via a separate scheduler 'error' event that this streak never sees).

Review history

  • Layer-2 independent subagent review, 3 rounds.
  • Round 1: CRITICAL — the fix was dead code on the durable-ledger (primary) path, the exact branch the production incident (AX-1272) actually looped through. Fixed by adding the same gate to idleLiftable, plus a new test that seeds a real RunLedger RETRY_AT row.
  • Round 2: MEDIUM — the streak gate wasn't scoped to lastErrorCode === 'infra_error', so it over-applied to later unrelated RETRY_AT rows on the same issue. Fixed. Also fixed a LOW test-hygiene nit (unclosed ledger handle).
  • Round 3: no new findings in the diff itself.
  • Manually reverted the durable-ledger fix and re-ran the new test to confirm it actually catches the regression (not trivially green).

Test plan

  • npx tsc --noEmit -p tsconfig.check.json clean
  • npx oxlint src/automation/autonomousRunner.ts src/automation/autonomousRunner.infraError.test.ts → 0 warnings/errors
  • npx vitest run src/automation/autonomousRunner → 139/139 passed (after rebase onto origin/main)
  • Manual revert-and-rerun confirms the new durable-ledger test fails without the fix

🤖 Generated with Claude Code

…GT-4305)

AGT-4257's idle_fill unconditionally cleared a task's infra_error backoff
whenever a free slot was available, with no cap on how many times the SAME
issue could be bypassed for the SAME recurring infra_error. On vela, AX-1272's
reviewer stage died to the same OpenRouter 360s timeout 9 of ~13 attempts over
4h+, retried every ~90s instead of waiting the intended 1h backoff, because it
was the only runnable candidate — burning a full worker+tester+reviewer cycle
each time with no forward progress.

Adds consecutiveInfraErrorCounts (issueId -> streak), incremented on every
infra_error result and reset at every existing terminal/recovery site. Both
idle-fill bypass points (legacy failedTaskRetryTimes gate, and the durable-
ledger RETRY_AT idleLiftable check) now refuse to bypass once an issue's
CURRENT infra_error streak reaches 3, honoring the real 1h backoff instead.
The durable-ledger check is additionally scoped to durableRun.lastErrorCode
=== 'infra_error' so a later, unrelated RETRY_AT (e.g. an ordinary rejection)
on the same issue keeps normal idle-fill treatment.

Two related gaps found during review are tracked separately rather than
expanding this fix's scope: AGT-4306 (idle-fill also un-parks an
INFRA_CIRCUIT_PARK_REASON NEEDS_HUMAN with no operator action) and AGT-4307
(a thrown/rejected executor error surfaces via a separate scheduler 'error'
event that this streak never sees, so it bypasses this protection entirely).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@unohee
unohee merged commit 586f6d8 into main Sep 10, 2026
7 checks passed
@unohee
unohee deleted the fix/agt-4305-infra-idle-fill-streak branch September 10, 2026 23:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant