Skip to content

perf(ci): cut GITHUB_TOKEN calls in pr-verify and pace timer lanes on burn rate - #11046

Merged
kyle-sexton merged 8 commits into
mainfrom
perf/github-token-cuts
Oct 30, 2026
Merged

kyle-sexton merged 8 commits into
mainfrom
perf/github-token-cuts

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

No related issue: these cuts come from the 2026-10-29 17:58-18:58Z GITHUB_TOKEN spend ranking (gh-monitor trend report). That report names no issue; the one number it cites, #10488, is the earlier pr-merge change it compares against.

Summary

The window spent about 4,289 GITHUB_TOKEN calls, and every budget trip stops all automation. This PR covers three of the four planned items.

What check-intent spent, measured from the window's run data. The data is per-job api-calls notices, annotation counts, and post-verify-notes' verdict-or-none spend:

  • 20 judgments: 334 calls, 16.7 each.
  • About 24 runs that judged nothing: 138 calls, 5.75 each, in clusters of 4-6 and 11-12 calls.
  • That totals the report's 472. post-verify-notes agrees: 20 of its runs found a verdict, and 25 spent 1 call finding none.
Item Window spend Change Estimated calls cut per hour
1. pr-verify check-intent 472 over 45 runs The read-only resolver runs before the trigger gate, so a run it skips no longer pays the gate's reads About 70-95. That is about 24 skips times the gate's 3-4 calls, a figure derived from the clusters: the resolver's 2 or its pinned 8, plus the gate. The ceiling is the 138 that skips spend; the 334 judgment calls are one per head.
2. ci-status hold-read carry 541 over 48 runs Not in this PR; see Related 0
3. api-calls notices 32% of spend had no notice intake-triage apply-verdict and pr-review-main record now log their GITHUB_TOKEN calls 0. This is coverage: the next ranking can count these jobs.
4. Burn-rate pacer reconcile 137, pr-unblock-dispatch schedule about 41 While the remaining quota projected at reset is under pace (0.2), even-numbered schedule ticks skip their work About 84 at this window's burn: half of 178, less 4 pacer reads. That burn projects about 8% left at reset from the first minute, so pacing would have held all hour.

Fix

  • Item 1: resolver first.
    • Order. In all three pr-verify jobs, Resolve the pull request now runs on the kill switch. Check the trigger runs only when the resolver proceeds, and the agree step only when the gate proceeds. A gate refusal therefore stays a refusal; it doesn't become a mismatch error. A judged run makes the same reads as before.
    • Verify-on-red stays. It is relied on, so this keeps one run per head rather than skipping red runs:
      • pr-verify.yml:3-6 and resolve-pr.mjs:5-9,588 (heldOnly) judge a red run that only an approval-liftable hold or do-not-merge holds, so a person approves a judged head.
      • pr-feedback-dispatch.yml:18-35 turns a FAIL into threads that address-feedback works, or into the pr-verify-failed label that ci-status holds.
    • No carried state. A per-head record in the Actions cache was rejected: a restore extracts anywhere in the workspace, including .base (Harden pr-merge's run-cache restore in the token-holding jobs #9455, pr-require-checks.yml:5210-5213). A lookup-first precheck was rejected too, because it would add reads to all 20 judgments an hour.
  • Item 3: notices.
    • apply-verdict's resolve-config step records under github-token, and the job logs lanes-app github-token.
    • pr-review-main record counts its check-run POSTs and ends with the notice in api_budget_emit_calls' shape, pinned against the emitter. This adds no checkout and no permission.
  • Item 4: pacer (log-api-budget/pace.mjs).
    • Config. limits.api github-token pace is new, default 0.2, in rate-limits.json. It is validated like breaker and allowed in pr-pipeline.schema.json.
    • Wiring. The lost-run reconciler and pr-unblock-dispatch read it on schedule ticks only. A paced projection skips even-numbered ticks and ends them green. Odd ticks still run, so the lanes slow to half cadence rather than stopping, and the reconciler keeps re-running the runs a trip stops.
    • Backstop. Any other event, or a failed read, runs as before. The breaker is unchanged.

No token scope or workflow permission is added. No hold is changed, and ci-status is untouched.

For review:

  • Held paths. rate-limits.json and docs/conventions/pr-pipeline/ sit on the merge hold lists, so this PR waits for a person.
  • Pacing at today's burn. About 4,300 calls an hour keeps the projection under 20%, so both lanes would run at half cadence most of the time. pace: 0 in limits.api turns the pacer off.

Verification

  • pr-verify:

    • The new order test failed before the workflow change and passes after.
    • node --test .github/scripts/pr-verify/*.test.mjs: 147 pass.
    • test_verify_rules_sync: OK.
  • node --test .github/actions/log-api-budget/*.test.mjs: all pass. This includes:

    • pace.test.mjs, with hand-worked projections, the evidence window's reading and the odd-tick floor.
    • The record-job notice test against api_budget_emit_calls.

    Both new tests failed before their workflow changes.

  • Node suites for pr-verify, pr-unblock, pr-merge, resolve-config, check-trusted-trigger, report-check-run, check-wip-cap, reconcile, intake-triage, intake-reenter, pr-review-main and triage-sweep: 3,958 pass.

  • Shell suites: ci-lanes-pr-hold 243, check-convention-yaml 45, check-loop-lane-floor-drift 28, read-runner-dispatcher-gate 9, plan-test-lanes 55.

  • babysit-prs Python (unittest): test_babysit_gh_rate_guard (45), test_merge_holds_sync, test_babysit_risk, test_babysit_postmerge and test_verify_rules_sync all pass.

  • actionlint and zizmor --offline are clean on every changed workflow. markdownlint is clean. scripts/check-changelog-fragments.sh --check and --check-required pass.

Related

  • Item 2 is not done. ci-status's hold reads are live state (labels, comments, reviews, statuses). A carry may only keep a hold, never lift one, but those reads cost calls exactly when they lift. So a keep-only carry saves calls only on held heads. The safe lever is per-head immutable reads: completed review-lane run records, the changed files at a head and base, and compares. They would go in pr-merge's main-uploaded run cache, re-validated against a live listing as pr-merge already does for review-lane runs. That design needs its own review.
  • Next lever for item 1. On a judged run, the gate and the resolver both read commit pulls and the upstream run. Sharing those reads would cut about 2 calls from each of the 20 judgments an hour.
  • Out of local reach. pr-review-hosted, pr-review-security-hosted and pr-require-checks-trust-paths run their jobs in ci-workflows' reusable workflows, so their notices need a ci-workflows change. pr-review-main's review-handoffs and subscription-handoffs hold permissions: {} and make no API calls.

🤖 Generated with Claude Code

kyle-sexton and others added 4 commits October 29, 2026 16:05
check-intent spent 472 GITHUB_TOKEN calls over the 2026-10-29
17:58-18:58Z window. The job annotations and post-verify-notes'
verdict-or-none counts split them into 20 judgments (334 calls, 16.7 each)
and about 24 runs that judged nothing (138 calls, 5.75 each). The skips sat
in two clusters, 4-6 and 11-12 calls: the trigger gate's own reads, about 3
or 4, plus the resolver's 2 (no-single-pr, draft, author) or its pinned 8
(already-judged).

The resolver only reads and already skips every run the brief names: a red
run that only an approval does not lift, a PR whose head moved on, and a
head already judged. So it now runs first, the gate runs only when the
resolver proceeds, and the agree step only after the gate proceeds. A
judged run makes the same reads as before.

This keeps one run per head, not one per attempt, rather than skipping
red runs, because PR authors and lanes rely on verify-on-red:
- pr-verify.yml:3-6 and resolve-pr.mjs:5-9,588 (heldOnly) judge a red run
  held only by an approval-liftable hold or do-not-merge, so a person
  approves a judged head.
- pr-feedback-dispatch.yml:18-35 turns a FAIL into threads that
  address-feedback works, or the pr-verify-failed label that ci-status holds.

No state is carried. A per-head record in the Actions cache was rejected,
because a restore extracts anywhere in the workspace (#9455,
pr-require-checks.yml:5210-5213). A lookup-first precheck was rejected too,
because it would add reads to all 20 judgments an hour.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r-review-main record

About 32% of the 2026-10-29 17:58-18:58Z window's GITHUB_TOKEN spend had no
per-job api-calls notice. Two of the listed jobs are local and spend the
job token without counting it:

- intake-triage apply-verdict reads through resolve-config with the job
  token; that step now records under github-token and the job logs both
  lanes-app and github-token, as work-unblock's label-unblocked does.
- pr-review-main record POSTs one check run per verdict file with no checkout
  of log-api-budget; it counts its POSTs and ends with the notice in
  api_budget_emit_calls' shape, pinned against the emitter by a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A budget trip (the breaker share of the header's limit left) turns the kill
switch on and stops every lane. In the 2026-10-29 17:58-18:58Z window the
pool reached used=4289 remaining=711 about 4 minutes before reset. The
scheduled lanes were still spending at full rate then: reconcile at 68.5
calls per run, pr-unblock-dispatch at 20.6.

On a schedule tick, pace.mjs reads the pool once and projects the remaining
at the reset from the window's burn rate so far. When that falls under
limits.api github-token `pace`, an even-numbered tick writes paced=true.
`pace` is a new config value, default 0.2 in rate-limits.json, validated
like the breaker and allowed in the schema.

The lost-run reconciler and pr-unblock-dispatch skip their work on a paced
tick and end green. Odd ticks still run, so a paced lane runs at half its
cadence rather than stopping, and the reconciler keeps re-running the runs a
trip stops. Any other event, or a failed read, runs as before. The breaker
stays the backstop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 29, 2026 20:17
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T20:19:28.081913Z 0adc993 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 0adc99331845085c9b32ae093a2d8d8c84af1ed4; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread .github/actions/log-api-budget/pace.mjs Resolved
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

claude-review reviewed 10475b1370cbe8cb09a471a4a85a113200b393c7: 1 finding(s), 0 as inline comments.

Review of the pr-verify reorder, the pacer and the api-calls notices. I found one problem, in the pacer's tick selection.

Run-number parity does not give half cadence for pr-unblock-dispatch (inline comment on pace.mjs). github.run_number counts every run of the workflow, and pr-unblock-dispatch also triggers on push to main. The parity of a scheduled tick therefore depends on how many pushes landed since the previous tick. If that count is odd every time, which is plausible with one merge per half hour, every scheduled tick has the same parity. If that parity is even, paced=true holds on every tick for as long as the projection stays under pace. The PR's claim that "odd ticks still run" and that lanes slow to half cadence rather than stopping is then false for that lane. The reconciler is schedule-plus-dispatch only, so there the parity does alternate.

Checked and found fine

  • pr-verify, all three jobs:
    • The resolver now runs on the kill switch, the gate on steps.pr.outputs.proceed, and the agree step on steps.gate.outputs.proceed.
    • Every later step that uses the resolver's outputs (trusted text, diff, model, credential, judgment) hangs off steps.agree.outcome == 'success', so nothing the resolver writes reaches the model before the trust gate passes.
  • pace is added to FIELDS in rate-guard.mjs, to rate-limits.json, to the schema (same range as breaker), and to the docs. Only github-token carries the default.
  • Pacer wiring:
    • In the reconciler, the Find lost runs and Re-run lost runs steps are gated on the pacer. The record-upload steps depend on plan outputs, which a skipped plan leaves empty.
    • In pr-unblock-dispatch, Install resolve-config's dependencies, the restore and the dispatch steps are gated. Save the dispatch record and the dispatched-runs record steps depend on steps.dispatch, which is skipped on a paced tick, so they skip too.
    • A failed read or any non-schedule event runs the tick as before.
  • The pr-review-main record step's trap writes its count on every exit, and the notice step validates the count (^[0-9]{1,6}$).
  • The intake-triage API_BUDGET_TOKEN override and the tokens: lanes-app github-token change are consistent with the updated test.

I read the diff and the surrounding workflows and ran no tests. The PR's verification claims are the author's and I did not confirm them.

Cost: the pacer's one GET /repos/{repo} per scheduled tick spends GITHUB_TOKEN rate-limit calls, not a billed resource. This is not a hold.

critical-findings: 1

Findings not posted inline (their lines are not in the diff):

  • .github/actions/log-api-budget/pace.mjs:134 (RIGHT): github.run_number is not a per-schedule-tick counter: it counts every run of the workflow. pr-unblock-dispatch also runs on push to main, and each push run takes a number. When the number of pushes between two ticks is odd every time (for example one merge per half hour), all scheduled ticks land on the same parity. If that parity is even, every tick is paced for as long as the projection stays under pace, so the lane stops instead of running at half cadence. The PR body, the changelog fragment and the header comment all promise that odd ticks still run. Cancelled push runs also take a number, so the pattern is not rare.

The reconciler is schedule-plus-dispatch only, so its parity does alternate. The failure is specific to pr-unblock-dispatch.

One fix: derive the tick parity from the clock, e.g. Math.floor(now / 1800) % 2 for a half-hourly cron, or from the minute, rather than from run_number. The tests only pass RUN_NUMBER directly, so none of them would catch a lane whose scheduled ticks all share one parity.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0adc993318

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/actions/log-api-budget/pace.mjs Resolved
Comment thread .github/workflows/pr-review-main.yml Resolved
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 10475b1370cbe8cb09a471a4a85a113200b393c7: 0 finding(s), 0 as inline comments.

No security issues found. critical-findings: 0

I read the full diff and the PR context. I also read .github/workflows/pr-verify.yml and .github/scripts/pr-verify/resolve-pr.mjs in full around the changed steps.

pr-verify: the resolver now runs before the trust gate (all three jobs). This is the one change that touches a trust control, so I checked what the resolver does before the gate and what still depends on the gate.

  • Every step that feeds the model or writes a verdict is still gated on steps.agree.outcome == 'success'. That includes Select the trusted PR text, Write the diff, the credential and managed-settings steps, Isolate the runner, Write the prompt and schema, and the judge. Write the verdict and the upload depend on the judge. agree now requires steps.gate.outputs.proceed == 'true' (pr-verify.yml:165, :532, :901). A gate refusal therefore still ends the job without a judgment. Nothing the gate admitted before is admitted differently.
  • The gate still checks the triggering actor, the upstream run's actors and the PR author against the trusted-actor list. Its inputs and the trusted-actor path are unchanged. The agree step still requires the gate and the resolver to name the same PR and head.
  • What the resolver does before the gate is limited. It reads the PR, the upstream run and the job log, and writes SPEC_FILE and SPEC_TEXT_FILE into the workspace. It appends fixed-shape lines to GITHUB_OUTPUT: proceed, reason, pr-number, head-sha, base-sha and head-ref. The head ref is checked against ^[A-Za-z0-9._/-]+$ before it is written (resolve-pr.mjs:725). It makes no POST, PUT, PATCH or DELETE call. The spec files are only read by steps behind agree.
  • The resolver now runs for runs triggered by an untrusted actor, so it parses upstream job-log text it did not parse before. That text can only change whether the resolver skips and which PR and head it names. Both agree and the gate still have to confirm that PR and head before a judgment.
  • The kill switch step and its report step are unchanged, and the resolver is still gated on steps.kill.outputs.proceed.

Burn-rate pacer (pace.mjs, maintenance-reconcile-lost-runs.yml, pr-unblock-dispatch.yml).

  • The pacer can only skip a schedule tick's work. It cannot widen anything.
  • Every failure path sets paced=false and runs the tick. That covers a non-schedule event, a failed or unparseable read, and a run number that is not a number. The breaker and the kill switch are untouched.
  • The pace override is read from .base/docs/conventions/pr-pipeline.yaml (the default-branch checkout), not from PR-controlled content.
  • The pacer uses only github.token for one GET /repos/{repo}. No token scope or permissions: block changes.
  • The notice interpolates only a validated number (SHARE, 0 to 1) and a rounded projection.
  • In pr-unblock-dispatch.yml the added && steps.pace.outputs.paced != 'true' clauses only narrow when the dispatch steps run. The kill-switch and gate conditions are unchanged.

Rate-limit config (rate-guard.mjs, rate-limits.json, pr-pipeline.schema.json, pr-pipeline.yaml). The new pace field is validated as a share in 0..1, the same as breaker. The breaker, writer-pause and read-floor values are unchanged. The pr-pipeline.yaml change is comment-only. No merge hold is weakened.

pr-review-main.yml record job. The step counts its check-run POSTs and writes the count in an EXIT trap. A new always() step reads the count back, only accepts ^[0-9]{1,6}$, and otherwise emits 0. This adds no permission, no checkout and no untrusted interpolation.

intake-triage.yml. The added API_BUDGET_TOKEN: github-token env and the extra lanes-app github-token entry only affect which tokens the notice counts.

Manifests and instruction surfaces. No plugin.json, marketplace.json, CLAUDE.md, AGENTS.md or other standing instruction changed, so no manifest <field> lines apply.

Spend. The pacer and the reordered pr-verify steps make fewer GITHUB_TOKEN calls. The change adds no billed resource and no paid-plan setting.

kyle-sexton and others added 2 commits October 29, 2026 17:05
… verdicts download

Neither paced workflow set RESOLVED_CONFIG, so pace.mjs read only
rate-limits.json and limits.api github-token pace in pr-pipeline.yaml,
0 included, never reached it. Each now writes the base config's
limits.api with yq, in resolve-config's output shape, and hands that
file to the pacer; a failed read leaves the defaults.

pr-review-main's record job: when the verdict download failed, the
record step was skipped, no counter file was left, and the notice
step's cat ended it red under bash -e. It now reads the file only when
it exists, else 0. The test runs that step with -eo pipefail, as
Actions does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude-review: 1 inline finding(s) on 0db806ce85aa82f5e148d9b688063dcc88c04dc4; the summary is in this lane's pull request comment.

claude-lane: claude-review

Comment thread docs/conventions/pr-pipeline/README.md Resolved
@melodic-automation-lanes

Copy link
Copy Markdown
Contributor

Unattended merge: risk MED at head 0db806ce85aa82f5e148d9b688063dcc88c04dc4. Only LOW, or MED on class, size and reported review-finding reasons alone, merges without a human. The lanes clear its holding reasons (a size reason holds nothing); pr-merge judges it again each run.

MED evidence-unbound:claude-review: not bound: evidence-missing
MED evidence-unbound:claude-security-review: not bound: evidence-missing

kyle-sexton and others added 2 commits October 29, 2026 20:49
Reorder the limits.api pace sentence so "the timer lanes slow down before
the breaker stops every lane" describes pace itself and "`0` turns pacing
off" is its own clause, per the review thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit 5c3529f Oct 30, 2026
50 checks passed
@kyle-sexton
kyle-sexton deleted the perf/github-token-cuts branch October 30, 2026 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant