Skip to content

perf(pr-pipeline): wait for the rate-limit reset in ci-status instead of failing - #9430

Merged
kyle-sexton merged 12 commits into
mainfrom
perf/ci-status-wait-rate-limit-reset
Oct 23, 2026
Merged

kyle-sexton merged 12 commits into
mainfrom
perf/ci-status-wait-rate-limit-reset

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

No related issue: efficiency register E5 (owner directive 2026-10-23)

Summary

ci-status failed closed as soon as a GitHub API read hit the repository's GITHUB_TOKEN rate limit, then told people to re-run. The re-run landed in the same exhausted window. In the 05:26Z and 09:24Z runs (39905801496, 39931677333, 39931558895) the reset was 399 to 523 s away.

With this change, ci-status's inline reads wait for the reset when it falls inside the job's existing timeout. Otherwise they fail red at once, naming the reset time and saying a re-run after it is needed. The owner's decision (Q10) was the most time-efficient option that adds no spam: no timeout raise, and no new re-run loop.

Fix

gh_read. A new first step writes gh_read to $RUNNER_TEMP, and later steps source it. On a rate limit it waits once and reads again.

  • Detection. gh's error names a rate limit, or a 403/429 comes back while the probe below reads remaining 0 or retry-after.
  • Length of the wait. It reads the x-ratelimit-* and retry-after headers of one gh api -i probe to the same resource (core, or graphql for a graphql call).
    • retry-after seconds when that header is present.
    • Otherwise, 5 s past x-ratelimit-reset while remaining is 0.
    • 1 s when the probe shows the limit has already cleared.
    • Otherwise, 60 s.
  • Why a probe. This PR's first live run showed that GET /rate_limit does not report the installation limit. It read core remaining > 0 while every read failed, and the failing responses' own headers said remaining=0, reset 15:35:06.
  • Wait budget. A wait is taken only when it ends inside the job's existing 11-minute timeout, less a 300 s margin for the job's setup and the steps after the wait. The deadline is measured from when the helper step starts: GH_READ_JOB_SECONDS: 660 minus GH_READ_MARGIN_SECONDS: 300, so 360 s. The margin covers aggregate-results' 202 s of waits (60 s rate limit, 45 s own-gate, 7 s status writes, 90 s rerun wait) plus retry sleeps.
  • Past the budget. A reset after the deadline fails the read at once with ::error::, which turns the check red, never neutral. The error names the reset and says a re-run after it is needed. Later reads of that resource before the reset fail without a call. Any other error returns at once, so each step's existing retries and fail-closed messages are unchanged.
  • What the budget covers. A wait is taken only for a reset about 6 minutes or less away. The cited resets were 399 to 523 s away, past the 360 s budget, so those cases still fail red as before. The change is that the error now names the reset time and says a re-run is needed after it, so the re-run does not land in the same window. The composites' 60 s rate-limit-wait-seconds matches main; this PR does not change it.
  • No new re-run mechanism. The next review lane's rerun-ci-status, or the rate-limit wait from E10 in perf(pr-pipeline): re-run ci-status once per head after the review lanes finish #9429, picks the PR up after the reset.

Which reads use it. The API reads that remain after #9433:

  • the PR file list in the broker step, and in the review-lane fallback
  • the label reads in the human-gated and merge-group steps
  • the comment and event reads in hold-lifted-by
  • the read_api helpers in the review-lane and lanes-hold steps

Two reads keep plain gh api. The cost-hold step keeps it because a wait there would act on reads taken before a newer hold. The best-effort hold-reason read keeps it too. Both are marked gh-read-exempt, and a test fails on any new unmarked read.

What does not change. ci-status keeps its timeout-minutes: 11. The composites keep their 60 s rate-limit-wait-seconds default. rerun-ci-status and the pr-pipeline README are unchanged from main.

Verification

  • scripts/ci-gh-read.test.sh (new, 33 checks).
    • The budget env equals timeout-minutes in seconds, and the margin leaves room for the steps after a wait.
    • The step sets the deadline at its start plus the timeout, less the margin.
    • No composite gets a rate-limit-wait-seconds override.
    • Waits are taken for a primary limit, a graphql limit, retry-after, a secondary limit, and a cleared limit.
    • A second limit after the wait fails the read.
    • A reset past the deadline fails at once, and later reads of that resource make no call while graphql reads are still sent.
    • A later wait that would end past the deadline is not taken.
    • A 403 or 404 that is not a rate limit returns at once.
    • The helper is the first step, every caller sources it, and every other gh api read is a write or marked exempt.
  • scripts/ci-hold-human-gated.test.sh, end to end.
    • A rate-limited label read sleeps 125 s and passes.
    • A reset past the deadline fails closed, naming 2026-10-24T00:50:00Z.
    • A 403 that is not a rate limit fails closed with only the step's own 5 s and 10 s retries.
  • Mutation. Dropping the margin from the deadline fails the suite.
  • Full suite and checks. Every scripts/ci-*.test.sh passes, as do actionlint, shellcheck, shfmt, and check-affected.sh --tests on the merged head.
  • Left out. A live run against a real exhausted rate limit, because one cannot be produced on demand.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 7 commits October 23, 2026 11:06
… of failing

ci-status failed closed whenever a GitHub API read hit the repository
GITHUB_TOKEN rate limit and asked for a re-run that landed in the same
empty window. The aggregate-results composite needed 399 to 523 s waits
against a 60 s ceiling in the 05:26Z and 09:24Z runs.

- Set RATE_LIMIT_WAIT_SECONDS: 600 on the job and pass it to the
  check-contract and aggregate-results composites, whose gh-api.sh already
  waits for the reset.
- Add a first step that writes gh_read, the same wait for the inline
  steps: on a rate limit it reads the reset from GET /rate_limit, waits
  until five seconds past it (60 s for a secondary limit) and reads again,
  under a job-wide ceiling; past it the read fails closed naming the reset.
  Every fail-closed read in the six hold and review steps goes through it.
- Raise timeout-minutes from 9 to 20 to hold one reset wait.
- Test the helper in scripts/ci-gh-read.test.sh and the recover, past the
  ceiling and non-rate-limit cases in ci-hold-human-gated.test.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the first commit found two gaps and a fail-open:

- gh_read's job-wide budget did not include the composites' waits, and
  core and graphql reset on separate windows, so two reset waits plus the
  other waits could pass the 20-minute timeout. gh_read now waits only
  until a deadline RATE_LIMIT_WAIT_SECONDS after the job starts, which
  check-contract's wait also spends; the timeout is 25 minutes for that
  window, aggregate-results' own and five minutes of other waits.
- A block past the deadline was stored for every resource, so a graphql
  block stopped later core reads. Blocks are now kept per resource.
- The cost-hold step reads plain gh api again: its failure keeps the
  hold, and a wait there would remove the label on reads made before a
  newer hold landed.
- gh_read waits once per call, like the composites' gh-api.sh, and reads
  again after one second when the window has already reset.
- pr-feedback-dispatch polls an in-progress ci-status for 26 minutes, past
  its new timeout, so the re-run after a review is not skipped.
- A test fails any new plain gh api read in ci-status that is neither a
  write nor marked gh-read-exempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 23, 2026 15:28
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T15:30:37.757030Z 64b6bb4 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

The first live run of this branch hit the GITHUB_TOKEN installation
limit: every read failed "API rate limit exceeded for installation" with
x-ratelimit-remaining 0 and a reset five minutes out, while GET
/rate_limit reported core remaining above 0, so gh_read waited one
second and the review-lane step failed closed.

gh_read now reads the wait from the x-ratelimit-* and retry-after
headers of one `gh api -i` probe to the same resource, the headers the
composites' gh-api.sh already uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 23, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane stopped on this pull request and needs a person: a conflicted path is a workflow, submodule config or a settings file, which the lane does not resolve.

  • Reason: protected-path-conflict
  • Resume key: 64b6bb4744bd0f888752b28e59a1d59f24f9326b (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=9430 and activity=resolve-conflicts:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=9430 -f activity=resolve-conflicts

Resolves the ci-status timeout comment against the new lanes-hold step:
its eight retried reads go through gh_read, and its retry sleeps raise
the other-waits allowance, so timeout-minutes is 28 and
pr-feedback-dispatch polls an in-progress ci-status for 29 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 89f0c702e71c50bb3396234fd4bf2871a62095b4: 0 finding(s), 0 as inline comments.

I found no security issues in the incremental change (.github/workflows/pr-require-checks.yml, scripts/ci-gh-read.test.sh, scripts/ci-hold-human-gated.test.sh, scripts/lib/ci-gh-read.sh). I read only those four files' change and used the PR context to check what it claims to do.

  • Rate-limit deadline: pr-require-checks.yml now sets the deadline to GH_READ_JOB_SECONDS - GH_READ_MARGIN_SECONDS (660 − 300 s) after the helper step starts. A wait is taken only if it ends before that deadline.
    • A reset past the deadline still fails the read at once with ::error::, and later reads of that resource before the reset make no call.
    • Callers' own retries and fail-closed exits are unchanged, so no new fail-open path appears.
  • Timeout cut from 28 to 11 minutes: this shortens the job's runtime and widens nothing. A job that hits the timeout ends red, never neutral, so it can't count as a passing required check.
  • Removed RATE_LIMIT_WAIT_SECONDS and the two rate-limit-wait-seconds: inputs: both composites fall back to their own 60 s default, a smaller wait than the 600 s it replaces. The composites are still pinned by SHA (@5e18f3d…, v0.58.0).
  • Error message text: the new ::error:: text interpolates the same values as before (resource, $*, and timestamps computed with jq). The $* interpolation was already there and the diff adds no new injection surface.
  • scripts/lib/ci-gh-read.sh: the test helper reads the two env values from the workflow with a strict digit-only sed regex. It passes them as environment to bash -e on a step body extracted from the repository's own workflow file. No untrusted input reaches it.
  • Test files: scripts/ci-gh-read.test.sh and scripts/ci-hold-human-gated.test.sh only update expected strings and assertions. They also add checks that the budget equals the timeout in seconds and that the margin is at least 240 s. This tightens the guard against drifting from the timeout.
  • Embedded instructions: the PR context contains no directives aimed at the review lane.
  • Manifests: no plugin.json or marketplace.json is touched, so there are no manifest <field> lines.
  • Instruction surfaces: no standing instruction (CLAUDE.md, skill, hook allowlist) is removed or softened.
  • Trust-gate paths: none are touched, so I set no hold.

critical-findings: 0

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 89f0c702e71c50bb3396234fd4bf2871a62095b4: 1 finding(s), 1 as inline comments.

Incremental review of 9a1c034..89f0c70 (ci-status timeout kept at 11 min, RATE_LIMIT_WAIT_SECONDS replaced by GH_READ_JOB_SECONDS and GH_READ_MARGIN_SECONDS, composites back on their 60 s default, test and helper updates). I read the diffs only and ran nothing. The pass and fail counts in the PR description are author-claimed and I did not check them.

I found no correctness bugs in the changed lines:

  • The error text matches in the workflow, ci-gh-read.test.sh and ci-hold-human-gated.test.sh.
  • ci_gh_read::install now passes the same two env values the workflow job defines, and it returns non-zero when either is missing.
  • The new budget checks are consistent with each other: the job seconds equal 60 × the timeout, and the deadline equals the step start plus the job seconds less the margin.

One advisory finding, anchored inline: the 360 s wait budget is smaller than the 399 to 523 s resets the PR cites as its motivation, so those cases will still fail red rather than wait (see the comment on the env lines).

There is no cost or trust-gate hold. The change lowers the composites' rate-limit waits and adds no model call, schedule or runner. The PR has no closing issue, so I checked no acceptance criteria. No plugin manifests changed.

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane escalated this pull request as protected-path-conflict, and that no longer holds at b56afc3d44580e536a7f14e38c1ad0bc468b47e0: GitHub reports no conflict with main. pr-merge removed needs-human, so the lanes run on it again. To keep a person on it, add needs-human back: a label a person adds is never removed.

@melodic-automation-lanes melodic-automation-lanes Bot removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 23, 2026
Takes main's base-config checkout in place of the contents-API reads
and their retry loops in the human-gated, hold-lifted, merge-group and
broker steps, so gh_read leaves those call sites. It stays on the
surviving reads: the PR file list (the broker step's pages read and the
review-lane step's fallback), the label, comment and event reads, and
the review-lane and lanes-hold read_api helpers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@melodic-automation-lanes melodic-automation-lanes Bot added the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 23, 2026
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane stopped on this pull request and needs a person: a conflicted path is a workflow, submodule config or a settings file, which the lane does not resolve.

  • Reason: protected-path-conflict
  • Resume key: b56afc3d44580e536a7f14e38c1ad0bc468b47e0 (the head the lane stopped on)

To resume, deal with the reason above, remove the needs-human label (and do-not-merge, if set), then dispatch the pr-unblock workflow with pr-number=9430 and activity=resolve-conflicts:

gh workflow run pr-unblock.yml -R melodic-software/claude-code-plugins -f pr-number=9430 -f activity=resolve-conflicts

@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

The pr-unblock lane escalated this pull request as protected-path-conflict, and that no longer holds at 9a1c0343b68047bdfccc4a5456e54577749ad2c8: GitHub reports no conflict with main. pr-merge removed needs-human, so the lanes run on it again. To keep a person on it, add needs-human back: a label a person adds is never removed.

@melodic-automation-lanes melodic-automation-lanes Bot removed the needs-human Human-in-the-loop required; autonomous sessions must not resolve items carrying this. label Oct 23, 2026
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 23, 2026
@kyle-sexton
kyle-sexton removed this pull request from the merge queue due to a manual request Oct 23, 2026
@kyle-sexton kyle-sexton added the do-not-merge Hard merge gate: do not merge while applied. label Oct 23, 2026
kyle-sexton and others added 2 commits October 23, 2026 12:47
The owner chose the most time-efficient wait that adds no re-run loop
and no timeout raise. ci-status's timeout-minutes, and rerun-ci-status's
poll and job timeout, go back to main's values, and check-contract and
aggregate-results keep their 60 s rate-limit-wait-seconds default.

gh_read still waits for the reset, but only when the wait ends by the
job's timeout less a 300 s margin for the setup and the steps after it,
counted from the helper step's start (GH_READ_JOB_SECONDS 660, pinned
to timeout-minutes by the test). A reset past that fails the read red at
once, naming the reset and saying a re-run after it is needed. The next
review lane's rerun-ci-status, or a person, re-runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton kyle-sexton removed the do-not-merge Hard merge gate: do not merge while applied. label Oct 23, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 89f0c702e71c50bb3396234fd4bf2871a62095b4; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread .github/workflows/pr-require-checks.yml
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 23, 2026
Merged via the queue into main with commit 6537ae8 Oct 23, 2026
138 of 142 checks passed
@kyle-sexton
kyle-sexton deleted the perf/ci-status-wait-rate-limit-reset branch October 23, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant