Skip to content

perf(ci): log api-budget from the demotion alert and dependabot-fragments jobs - #11172

Merged
kyle-sexton merged 2 commits into
mainfrom
perf/api-budget-lanes
Oct 30, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
perf/api-budget-lanes

Conversation

@kyle-sexton

Copy link
Copy Markdown
Member

Closes #9497

Summary

Issue #9497 asks every token-spending job in its workflow list to log an api-budget line, so
GITHUB_TOKEN spend can be attributed per job. Most of the list already logs, through #10141,
#10710 and the jobs added since. This PR covers the one real gap left: the demotion alert job,
which sends about 20 gh api calls (reads, --paginate, GraphQL, issue writes and dispatches)
and logged nothing. It also adds the job-end step to dependabot-fragments, matching
release-publish-plugins. The PR only measures: each job sends the same requests and writes the
same outputs.

Fix

  • .github/workflows/post-merge-demotion-alert.yml, job alert:
    • Job env API_BUDGET_TOKEN: github-token. With it set, babysit_gh.py records its readings
      under the judge (babysit_postmerge.py --judge-run). Without it set, babysit_gh.py already
      plans to github-token's limits in Actions, so the limits it uses do not change.
    • "Read the failed run" and "File or comment on the demotion issue" now open with the
      api_budget_wrap_gh github-token line, verbatim from pr-feedback-dispatch.yml.
    • The job ends with the standard if-always Log the API budget at the end step. The step runs
      from the job's existing .base checkout of main.
  • .github/workflows/dependabot-fragments.yml, job write: ends with the same step, run from
    the root checkout, as release-publish-plugins does (ci: log GITHUB_TOKEN api-calls from the jobs that logged none #10710). Its gh calls use the release
    App token, not GITHUB_TOKEN. They are left unwrapped so App spend is never booked under
    github-token, and the line reads github-token=0.
  • One behavior change comes with the wrapper: the alert job's gh calls now honor the job's
    rate-guard stop file ($RUNNER_TEMP/api-budget/stop). A call the guard refuses fails through
    the step's existing error path; it is not sent. Before this PR it would have reached a pool
    GitHub had already reported as exhausted.
  • No new permissions, secrets or triggers. The mechanism is the existing log-api-budget action
    and api-budget.bash. Node-step attribution through API_BUDGET_SCRIPT (perf(pr-merge): attribute enable-auto-merge's API calls per step #11151) does not apply,
    because neither job has a Node step that calls GitHub through rate-guard.mjs.

Coverage of the issue's list

Workflow / job State
post-merge-demotion-alert alert Instrumented here.
dependabot-fragments write Instrumented here (GITHUB_TOKEN line only; see below).
pr-verify: check-intent, check-intent-subscription, check-intent-fallback Already logged (#10141).
pr-test-windows: all 9 jobs Already logged (#10710).
intake-triage: born-triaged, apply-verdict, refine-select, apply-refinement Already logged.
intake-reenter reenter Already logged.
work-unblock: wip-state, label-unblocked Already logged.
post-merge-demotion-alert report-red-main Already logged (#10141).
post-merge-detect-silent-revert detect-silent-revert Already logged (#10710).
release-publish-plugins: merge, build Already logged (#10710).
pr-merge enable-auto-merge Already logged. Per-step attribution is #11151.

Left out, and why

Job Reason
intake-triage kill-switch No token API call. The kill-switch values arrive as action inputs.
intake-triage refine-issue, refine-issue-subscription No GITHUB_TOKEN API call (contents: read: checkout, download-artifact, model run). Model tokens are out of scope (issue decision 5).
intake-triage intake-label-needs-triage, intake-triage Pinned reusable workflows in melodic-software/ci-workflows, where a caller cannot add a step.
intake-triage redispatch-refined One dispatch POST per run, and no checkout. Logging it would add a checkout and contents: read to count one known call. This follows #10710's mark-lanes row.
intake-triage triage-split-children One dispatch POST per sub-issue (the job's children output), with no checkout, for the same reason.

Known uncounted calls

  • failing-checks.mjs (demotion alert), .github/scripts/pr-merge/rotation.mjs:65-78
    (gh run download) and last-report.mjs run gh through execFileSync. That runs the gh
    binary, not the shell wrapper function, and these calls never reach countRequests, so the
    readings file never sees them. Counting them would need new Node code in rate-guard.mjs. That
    file is perf(pr-merge): attribute enable-auto-merge's API calls per step #11151's and on the hold list, so these stay listed and are not patched here.
  • Release App calls in release-publish-plugins and dependabot-fragments stay unattributed.
    The token names were settled as github-token and lanes-app (issue decision 4).

Metrics

  • New per-job lines in the demotion alert job log:
    ::notice title=api-calls::github-token=N not-modified=M graphql=K, the api-budget cost
    notice, and the content-creation line for its issue and comment writes. N covers the two shell
    steps and the judge's babysit_gh.py calls; failing-checks.mjs is not included.
  • dependabot-fragments write: api-calls::github-token=0 .... The line confirms the job
    spends no GITHUB_TOKEN REST budget.

Verification

  • node --test .github/actions/log-api-budget/*.test.mjs: 143 of 143 pass. The three new or
    extended tests failed before the workflow edits:
    • alert and report-red-main added to the API_BUDGET_TOKEN table (alert names github-token
      failed);
    • dependabot-fragments write added to the no-guarded-client table (failed);
    • a new test checks that every alert step calling gh api wraps gh first. To prove it can
      fail, the File step's wrap line was removed: the test failed with "File or comment on the
      demotion issue wraps gh before its first call", then passed once the line was restored.
  • bash scripts/post-merge-demotion-alert.test.sh: 173 of 173 pass. Its shape check allowed only
    the checkout action in alert, and it now also allows the .base log-api-budget action from
    that checkout of main.
  • bash scripts/dependabot-fragments.test.sh: 8 of 8 pass.
  • .github/scripts/post-merge-demotion-alert suites (58), label-unblocked.test.mjs (226) and
    check-trusted-trigger (370): all pass.
  • actionlint on both workflows: clean. zizmor --offline: no findings.
  • scripts/check-changelog-fragments.sh --check-required origin/main: no plugin changed, so no
    fragment is required.

Related

🤖 Generated with Claude Code

kyle-sexton and others added 2 commits October 30, 2026 01:30
…ents jobs

The demotion alert job's gh calls now run through api-budget.bash's
wrapper under API_BUDGET_TOKEN=github-token, and the job ends with the
log-api-budget step. dependabot-fragments' write job ends with the same
step, as release-publish-plugins does.

Refs: #9497

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kyle-sexton
kyle-sexton marked this pull request as ready for review October 30, 2026 06:39
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T06:43:16.987131Z 31d5a49 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

claude-security-review reviewed 31d5a49aa6d5e98679886b8f7509207f522caebf: 0 finding(s), 0 as inline comments.

No security issues found in this pull request. critical-findings: 0

What I read: all four changed files, the pull request context, the demotion alert workflow's checkout steps, and the log-api-budget action.

  • post-merge-demotion-alert.yml, alert job:
    • The new api_budget_wrap_gh lines source api-budget.bash from .base. .base is a credential-less checkout of refs/heads/main (lines 157–159), so the code comes from main's tip and not from pull request code.
    • The job-end ./.base/.github/actions/log-api-budget step runs from that same checkout. The action only reads rate-limit readings in $RUNNER_TEMP/api-budget and writes notices and the step summary. It validates token names against ^[a-z][a-z0-9-]{0,39}$ and has no network or secret access.
    • The added API_BUDGET_TOKEN: github-token job env is a plain label, and no secret is involved.
    • The job's permissions: block, triggers and secrets are unchanged. The wrapper only adds the rate-guard stop-file check to gh calls, which can refuse a call but cannot send one.
  • dependabot-fragments.yml, write job:
    • The new step runs the root checkout's action. That checkout is ref: main with persist-credentials: false.
    • The release App token exists only as steps.app-token.outputs.token and in the earlier step's GH_TOKEN env. The log step does not receive it, so the added step cannot reach the App credentials. Because the App calls stay unwrapped, the step reports github-token=0.
    • The job still has permissions: contents: read, and the environment: plugin-release gate on main is unchanged.
  • Tests: The test edits only add assertions. The updated uses allowlist in scripts/post-merge-demotion-alert.test.sh admits exactly one extra action, the .base log-api-budget, so the alert job's "no other action" check still holds.
  • Trust controls: None are widened. No trusted-actor list, token scope, kill switch, merge hold or pr-pipeline.yaml setting changes. post-merge-demotion-alert.yml and dependabot-fragments.yml are on the trusted-path list, so I read their whole change.
  • Manifests: No plugin manifest is touched.
  • Embedded text: The pull request context contains no directives aimed at the review lane.
  • Spend: The change only adds logging. It makes no new paid API calls and no billing change, so there is no hold.

@github-actions

Copy link
Copy Markdown
Contributor

claude-review reviewed 31d5a49aa6d5e98679886b8f7509207f522caebf: 0 finding(s), 0 as inline comments.

No findings. I read the full diff, the PR context and closing issue #9497, and the files the change depends on. I did not run any tests, so the PR's pass counts are author-claimed and unverified.

  • post-merge-demotion-alert.yml, job alert:
    • Both steps that call gh api ("Read the failed run" and "File or comment on the demotion issue") source api-budget.bash from the .base checkout of main and wrap gh before their first call.
    • .base is main's tip, not PR-controlled code.
    • The wrapper keeps stdout, stderr and the exit status, so the existing --jq, --paginate, -X POST and IDENTITY=… gh api graphql call shapes keep working.
    • The one behavior change is that a call the rate guard refuses now fails through the step's existing error path. The PR states this.
    • The job-end step is if: always(), continue-on-error: true and timeout-minutes: 1, and uses the local action from .base.
    • API_BUDGET_TOKEN: github-token at job level matches the pattern the other lane jobs use.
  • dependabot-fragments.yml, job write:
    • The job-end step runs from the existing checkout of main, as release-publish-plugins does.
    • The gh calls stay on the release App token and are left unwrapped, so no App spend is booked under github-token.
    • The comment says the line reads 0, which is accurate. The job has contents: read, which the step needs.
  • Tests:
    • The alert and write rows are added to the existing tables in log-api-budget.test.mjs.
    • The new wrap-before-first-call test finds gh api steps by regex and skips comment lines, so it fails if a step's wrap line is removed.
    • The edit to post-merge-demotion-alert.test.sh allows exactly the checkout plus the .base log-api-budget action. It strips the trailing zizmor comment before comparing.
  • Closing issue perf(ci): log api-budget lines from pr-verify, pr-test-windows, intake, work-unblock, post-merge and enable-auto-merge #9497: the PR lists the jobs left out of the change with reasons, as the acceptance criteria require. It says the other jobs in the issue's list were already wired by perf(ci): measure unattributed GITHUB_TOKEN calls and cut three without changing behavior #10141 and ci: log GITHUB_TOKEN api-calls from the jobs that logged none #10710. I did not re-check that for each of them.

No cost hold applies. The change adds a job-end step and a gh wrapper, and it spends no extra API credits or paid resources.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit 5192875 Oct 30, 2026
79 checks passed
@kyle-sexton
kyle-sexton deleted the perf/api-budget-lanes branch October 30, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(ci): log api-budget lines from pr-verify, pr-test-windows, intake, work-unblock, post-merge and enable-auto-merge

1 participant