Skip to content

fix: don't fail the check group on transient GitHub API errors - #44

Merged
ethanwharris merged 1 commit into
Lightning-AI:masterfrom
dhedey:david/retry-transient-github-api-errors
Aug 18, 2026
Merged

fix: don't fail the check group on transient GitHub API errors#44
ethanwharris merged 1 commit into
Lightning-AI:masterfrom
dhedey:david/retry-transient-github-api-errors

Conversation

@dhedey

@dhedey dhedey commented Aug 17, 2026

Copy link
Copy Markdown

Why are we doing this work?

During the GitHub incident on 17 Aug, check-group failed PRs whose required checks were all green — this run logged All required checks were successful! and then crashed, because the PATCH updating the PR comment came back 503 after octokit had exhausted its own retries. The action has a 40-minute budget to poll through exactly this kind of blip, and wasn't using it.

What does this PR change?

Makes the check-group action ride out GitHub API errors that say nothing about the PR:

  • notifyProgress is now awaited — the unawaited call turned a rethrown error into an unhandled rejection that killed the process.
  • Failing to write the PR comment now only warns; it's informational, and the check statuses it summarises are unaffected.
  • Transient errors in the poll loop retry on the next interval instead of calling core.setFailed immediately.
  • The timeout message names the last API error, so an outage doesn't masquerade as failing checks.
  • The one-shot calls before the loop starts (listing PR files, reading checkgroup.yml) retry with backoff.
  • Transient/403 classification duck-types on status rather than instanceof RequestError.

Why instanceof RequestError had to go

The existing 403 branch could never match. Several copies of @octokit/request-error are installed side by side, and the one that throws is not the one src/ imports:

$ node -e "..."
top-level version 2.1.0
nested version 3.0.1
same class? false
instanceof top-level? false name HttpError status 503

So every comment failure — including the expected 403 on fork PRs — took the throw e path.

Verification

There's no test harness in this repo, so I drove CheckGroup from a script with a stubbed octokit that reproduces the failure and its neighbours. Against the pre-fix dist, the 503-on-comment case crashes with an unhandled rejection and exits 1; after the change:

Scenario Result
503 on the PR comment, checks green warns, exits 0
403 on the PR comment logs once, exits 0
503 on the check-run listing, then recovers retries per interval, exits 0
503 on the PR file listing, then recovers retries with backoff, exits 0
503 for the whole timeout window fails, naming the API error
404 on the check-run listing still fails fast, unchanged

yarn lint, yarn build and pre-commit run all pass. yarn format is deliberately not run — prettier is not clean on master and would rewrite every file.

Follow-up

Once this lands, gridai/grid's .github/workflows/probot-check-group.yml should be repinned from 08fa537 to the merge commit.

A GitHub incident could fail `check-group` on a PR whose required checks
were all green. In gridai/grid run 32048244470 the loop logged "All
required checks were successful!" and then crashed, because the PATCH
updating the PR comment came back 503 after octokit had exhausted its
own retries.

Three things went wrong, all fixed here:

- `notifyProgress` was called without `await`, so the rethrow of a
  non-403 error became an unhandled rejection that took the process
  down. It is now awaited, and a failure to write the comment is only
  warned about — the comment is informational, and the check statuses it
  summarises are unaffected by GitHub refusing to store it.

- Any error inside the poll loop went straight to `core.setFailed`,
  so a single blip ended a run that had 40 minutes of budget left.
  Transient errors now warn and poll again on the next interval; the
  timeout timer still bounds the run, and names the API error if one was
  the last thing seen.

- The 403 branch tested `e instanceof RequestError`, which never matched:
  several copies of `@octokit/request-error` are installed side by side
  and the throwing one is a different class. Transient and 403
  classification now duck-type on `status`.

The one-shot calls made before the loop starts — listing the PR's files
and reading checkgroup.yml — have no later poll to fall back on, so they
retry with backoff.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ethanwharris
ethanwharris merged commit dcf2158 into Lightning-AI:master Aug 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants