Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
202 changes: 202 additions & 0 deletions .agents/skills/github-runner-image-maintenance/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
---
name: github-runner-image-maintenance
description: Use when auditing or updating GitHub runner images.
version: 0.1.0
author: Dave Bunten (d33bs), Hermes Agent
license: MIT
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [github-actions, runners, maintenance, packages, ci]
related_skills: [python-minimum-version-maintenance, python-latest-version-maintenance]
---

# GitHub Runner Image Maintenance

Proactively audit GitHub-hosted runner images across maintained, published
packages, then propose small upgrades on separate branches and draft PRs.
Prefer the newest viable, explicit, generally available OS label that is not
being deprecated. A registry entry, a runner catalog row, and a green local
lint check are three different pieces of evidence; none proves hosted CI passed.

## When to Use

- Asked to audit runner freshness or retirement exposure across maintained repos.
- Asked to replace deprecated runners or floating `*-latest` OS labels.
- Asked to act on brownout notices without waiting for an email or CI outage.
- Don't use for blanket action-version, language-version, dependency, or hardware
upgrades. Don't migrate self-hosted runners or Azure pipelines with this skill.

## Prerequisites

- Authenticated GitHub CLI: check with `terminal(command="gh auth status")`.
Workflow pushes require a credential authorized to change workflow files.
Never print tokens or change permissions to get around a rejected push.
- Python 3.11+ and `uv` for the read-only helper's declared PyYAML dependency.
The helper reads registry/GitHub APIs; it never clones, edits, commits, pushes,
creates PRs, or dispatches workflows. An explicit `--output` writes a local
report only. Use `read_file`, `search_files`, `patch`, and `write_file` for
project inspection and edits; use `terminal` for CLI commands and checks.
- Establish approved repos and publishing scope. Building this skill or asking
for an audit is not permission to update every discovered package. Preserve
local-only/no-commit/no-push instructions. Do not merge PRs or publish releases.

## How to Run

From the repository root, use `terminal` to run a read-only PyPI-maintainer audit:

```sh
uv run --script .agents/skills/github-runner-image-maintenance/scripts/audit.py \
--pypi-user HANDLE --output runner-audit.json
```

Put the report in the task's scratch directory rather than committing it.
Add repeated `--repo OWNER/REPO` arguments to limit the published-package
inventory to selected projects. Run `--help` for the exact supported options.

The helper covers PyPI discovery. For other formal registries, use their
current official APIs and perform the same publication/maintainership checks
manually; do not claim automated npm, crates.io, CRAN, or Bioconductor discovery.
A source manifest or a GitHub Release alone is not proof of a registry package.

## Procedure

1. Establish maintainer and package scope. For PyPI, use the helper's live
owner/maintainer discovery and registry project URLs, then cross-check the
canonical GitHub repo and current maintenance status. When PyPI omits source
URL fields, the helper requires a published-description GitHub link plus a
matching, SHA-pinned package manifest; name similarity alone is not proof.
Unlinked or legacy setup-only sources remain explicit manual-review outcomes.
For other registries,
record the package name, published version, source URL, and maintainer proof.
Exclude archived repos and unrelated mirrors; report unreachable, ambiguous,
or missing sources instead of guessing. The helper conservatively excludes
forks: review a genuinely published canonical fork separately if requested.
Account for every discovered entry, including skipped/error outcomes; persist
batches and compute totals programmatically before reporting a shortlist.
Completion: selected repos are maintained, published, non-archived, and scoped.
2. Refresh official runner evidence on every run. Read the
[runner image catalog](https://github.com/actions/runner-images), its current
[announcements](https://github.com/actions/runner-images/issues?q=label%3AAnnouncement),
the relevant image's software README and completed image releases, and
[GitHub's runner reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners).
Record UTC access time, catalog commit SHA, source URLs, image status,
architecture, size, and any deprecation/brownout/removal dates. Follow an
announcement before trusting an unbadged catalog row. Check closed notices
too: closed does not necessarily mean a retirement was cancelled. Exclude
previews, betas, retired/deprecating images, and unconfirmed future releases.
Resolve contradictory official sources or flag uncertainty; do not use an
email, search snippet, remembered label mapping, or OS vendor lifecycle alone.
Completion: each proposed label has fresh, source-backed GA/support evidence.
3. Inventory runner selection on each repo's pinned default-branch SHA. Inspect
all workflow files, reusable workflows and their inputs/defaults, job-level
`runs-on`, groups/labels, matrix axes/include/exclude, variables, and invoked
local actions/scripts. Trace dynamic selectors to definitions and callers;
report unresolved expressions and remote reusable workflows for manual review.
Arrays in `runs-on` combine labels; they are not an OS matrix. Preserve
self-hosted/custom-group selectors. A label in an unrelated example/comment
is not an active job, and changing a matrix axis must also update matching
conditional branches, includes/excludes, artifact names, and cache keys where
they actually depend on that value. Inspect test and Dependabot jobs together
so essential setup steps do not drift.
Completion: every affected job has a resolved selector or explicit review flag.
4. Classify each selector: deprecated/retired, floating alias, older supported
GA, current explicit GA, preview, or unknown/custom. Prioritize imminent
removals and brownouts, but also consider supported pins behind the newest GA.
Resolve aliases from current evidence, not string substitution. Select the
newest viable explicit version within the same OS family, architecture,
runner size, and required toolchain flavor. Compare version numbers
numerically; keep Ubuntu Slim/container and Windows client/server variants
distinct. Never invent a label from an OS version or add a patch number that
GitHub does not expose. Catalog-generated suggestions are review candidates,
not proof that the replacement is supported for this particular project.
Completion: old/new label, reason, evidence, and compatibility risks are stated.
5. Check the project's actual compatibility constraints before editing. Inspect
minimum and newest tested language versions, action architecture support,
dependency wheels, native libraries, browser/GPU/headless setup, Xcode/SDKs,
signing, containers, shell behavior, caches, and hardcoded install paths.
For data-heavy jobs, measure free space, inodes, mount types, and quotas for
`/tmp`, the job temp directory, and workspace on the actual target image.
A memory-backed `/tmp` can exhaust its quota while the regular disk is free.
Use a small hosted probe when documentation is insufficient; only then scope
an appropriate temporary-directory override to the affected step. Do not skip
the large-data test, disable quotas, or assume a paid runner is necessary.
Compare old/new included software; explicitly provision required tools rather
than relying on an image's incidental defaults. Inspect pre-commit's
`default_language_version` and per-hook `language_version` separately from
the package test matrix. Ensure the lint job explicitly installs those
interpreters before hooks run; a local installation or warm cache can hide
a missing runtime on the new image. Align tool setup with the existing hook
requirement rather than downgrading hooks or raising the package minimum.
For Python, read both
`python-minimum-version-maintenance` and `python-latest-version-maintenance`
once to review the whole supported range. A runner change does not authorize
raising the Python floor or dropping an existing OS/architecture test job.
Keep paid runner size/cost and Intel/ARM transitions separate unless explicitly
authorized. If the newest GA cannot run the support matrix, record the blocker
and propose an explicit, non-deprecated fallback with evidence; never silently
use `latest`, preview, fewer tests, or a different architecture to get green.
Completion: the candidate preserves support promises or has an explicit blocker.
6. For approved repos, inspect status/remotes/guidance and fetch the current
upstream default branch. Create a fresh, non-tracking branch from that base,
not from another package upgrade PR. Check existing PRs for duplicates and
avoid overwriting uncommitted work. Make the narrow runner/related selector
edits with `patch`; preserve triggers, permissions, secrets, release guards,
dependencies, supported/test Python versions, and test assertions. Do not
globally replace matching strings or retrofit archival docs. Run a dry-run/diff
inspection before staging and account for every changed file.
Completion: a small diff changes only intended runner selection and necessary
compatibility setup, with unrelated work untouched.
7. Run configured workflow/YAML/action lint and relevant local package checks.
Exercise the supported minimum and newest tested runtime locally where
available, plus build/installed-artifact smoke paths when image tools affect
packaging. If current lint tooling lacks an officially confirmed GA label,
update the tool first, then document only the exact source-backed exception
if still necessary; never disable runner validation wholesale.
A local Mac or Linux container is not the GitHub-hosted image.
Record commands, actual interpreters/platforms, skips, and pre-existing errors
separately; keep unexercised remote combinations provisional.
Completion: local checks pass or the draft accurately names remaining evidence.
8. When publishing is authorized, commit only the scoped files, push to the
intended fork, and verify the remote SHA. Open one draft PR per project with
explicit base/head and a short title plus two-sentence body: the runner change
and actual verification. Include the official notice link when deprecation
motivated the change. Keep detailed logs and the full audit in scratch, not
in the PR description. Read back exact URL, draft/open state, base/head repos,
branches, head SHA, title, and body. Do not mark ready or merge automatically.
Completion: the correct draft PR exists for each approved changed project.
9. Inspect hosted CI on the exact published head. Use check-run app ownership
and workflow jobs, not just a review bot's green status. Read the job's actual
runner labels/architecture and image/setup logs; compare expected matrix jobs
with those executed. A required job skipped by a stale condition is missing
coverage, not a pass. Distinguish image brownouts, missing tools, architecture
failures, and code/test regressions. Fix small in-scope issues, rerun checks,
and verify new head state. Report queued, approval-required, draft-skipped,
absent, timed-out, or failing CI honestly. Do not bypass security guards,
dispatch publishing workflows, or mark a draft ready just to force execution.
Completion: PR URLs and observed CI outcomes are reported, with unverified
combinations/blockers explicit and all PRs left as drafts.

## Pitfalls

- Read [source notes](references/runner-sources.md) for a dated worked example;
its version labels and retirement dates must be refreshed, not reused as policy.
- `macos-14`, `macos-14-large`, and `macos-14-xlarge` are not interchangeable
hardware classes. The same warning applies to newer labels and `*-latest`.
- An explicit OS label still receives rolling image/tool updates; it is not an
immutable image digest. Record the actual image version from hosted CI.
- An image README's announcement table can mention a different image's retirement.
An issue's unchecked platform list and mitigation labels are not affected labels.
- Preserve UTC date boundaries and the source's exact tokens. Flag impossible or
contradictory brownout intervals rather than silently correcting their dates.
- Draft PRs can be skipped by project policy or need fork approval. Never call
zero check runs, external-only statuses, or a skipped matrix green validation.

## Verification

Every scoped repo has a published-package eligibility decision, a complete runner
inventory or explicit gaps, current official evidence, and a no-change/proposed/
blocked result. Every published change has a read-back-verified draft PR and an
honest hosted-CI outcome. Skill tests and a live read-only audit must run before
calling the helper verified; they do not establish that package runner upgrades
passed hosted CI.
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Runner Evidence and Migration Notes

Use live official sources on every run. This worked example was checked on
October 16, 2026; it is evidence for the skill's first trial, not a permanent
replacement map or retirement calendar.

## Source precedence

1. [Current runner catalog](https://github.com/actions/runner-images/blob/main/README.md):
supported label spelling, architecture, image status, and image README links.
Record its commit SHA; do not use a cached search result as the catalog.
2. [Announcement issues](https://github.com/actions/runner-images/issues?q=label%3AAnnouncement):
GA promotion, deprecation, image removals, brownouts, alias migrations, and
breaking software changes. Read the body and relevant updates, not just tags.
3. The chosen image's software README and completed release: actual tools,
image version, OS version, and deployment evidence. A software release's
pre-release flag can mean deployment is underway; do not confuse it with an
OS being generally available everywhere.
4. [Runner reference](https://docs.github.com/en/actions/reference/runners/github-hosted-runners)
and [larger-runner labels](https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/running-jobs-on-larger-runners):
hardware class, access/billing, architecture restrictions, and label syntax.
5. Exact-head hosted job logs: the image and architecture actually exercised.
This final observation, not a local test or an available-image row, establishes
that the proposed project's jobs ran on the intended image.

If sources disagree, compare dates and the specific image/architecture in the
newer announcement. Stop or flag manual review if the conflict remains. A current
catalog row can lack a deprecation badge even when an active notice exists.

## First-trial snapshot

Catalog commit:
`ee1ff4569e381a9649b400338bdc8ceb7e18a80e`.

The catalog's macOS 26 rows distinguish standard ARM64 `macos-26`, ARM64 larger
`macos-26-xlarge`, and Intel `macos-26-intel`/`macos-26-large`. Preserve size and
architecture, then verify availability and project compatibility. Do not map
all retired macOS labels to one new label or assume a suffix guarantees the same
CPU count, memory, price, or toolchain forever.

The [macOS 14 notice](https://github.com/actions/runner-images/issues/13518)
sets retirement for November 2, 2026 and affects `macos-14`, `macos-14-large`,
and `macos-14-xlarge`. Its October 16 brownout spans 14:00 UTC on October 16
through 00:00 UTC on October 17. Keep the cross-midnight boundary explicit.
The catalog identifies the standard/xlarge variants as ARM64 and large as Intel.
The [macOS 26 ARM64 software inventory](https://github.com/actions/runner-images/blob/main/images/macos/macos-26-arm64-Readme.md)
provides a separate check on tools and preinstalled runtimes; provision missing
supported runtimes with the project's setup action rather than removing tests.

The [Ubuntu 26.04 GA announcement](https://github.com/actions/runner-images/issues/14747)
confirms GA on September 17, 2026 for `ubuntu-26.04` and `ubuntu-26.04-arm`.
The catalog still associates `ubuntu-latest` with 24.04 in this snapshot, so
resolving `latest` alone does not find the newest available explicit GA label.
A public-preview entry in an older documentation/search copy is not enough to
contradict a newer, explicit GA announcement.

The [Ubuntu 22.04 notice](https://github.com/actions/runner-images/issues/14254)
states retirement on April 17, 2027, although the catalog's 22.04 rows lack a
retirement badge. Several brownout lines in that notice give a same-day end time
earlier than the start time. Preserve that discrepancy, flag it for confirmation,
and do not invent corrected intervals. Its unchecked lists of other images do
not mean that those images are also deprecated.

## Validator catalog lag

The current actionlint release, 1.7.12, recognizes macOS 26 but does not yet
recognize the confirmed GA `ubuntu-26.04` label. Update an old validator first;
if its current release still lags GitHub's official catalog, a narrowly documented
additional-label entry can validate the real hosted label without disabling
workflow lint. The skill's draft CI uses that exact exception in
`.github/actionlint.yaml`. Despite actionlint's configuration key being named
`self-hosted-runner`, it does not change how GitHub schedules the workflow.
Remove the exception when the validator's built-in catalog catches up. Never
allow arbitrary unknown labels or suppress all runner-label diagnostics.

## Required compatibility review

- Compare image tools against the commands actually used by the workflow:
compilers, SDKs, Xcode, browsers, native dependencies, display/headless setup,
shell features, package manager paths, signing, containers, and caches.
- Check action/runtime and wheel availability for both the package's minimum
and newest tested runtime on the target architecture. Do not infer ARM support
from an x64 wheel, or successful installation from a resolver-only result.
- Compare temporary-storage mounts and quotas, not just total VM disk size.
In the October 16 pilot, Ubuntu 26.04 used a roughly 8 GB quota-limited tmpfs
for `/tmp`, while the job temp directory was on an ext4 disk with about 88 GB
free. Ubuntu 24.04 used the disk for both. The unchanged large-data test wrote
beneath `/tmp` and hit `Errno 122`; a larger paid runner was not the first
remedy. Diagnose with `df`/`findmnt`, then scope `TMPDIR` to a measured,
disk-backed directory for the affected test step and rerun the complete test.
Refresh these measurements rather than hardcoding capacities as guarantees.
- Check hook-specific runtime declarations as well as the package test matrix.
A lint job that sets up Python 3.11 can still require Python 3.12 through
pre-commit's `default_language_version`. A newer OS image may no longer supply
the latter on PATH. Explicitly install the configured hook runtime; preserve
the package's minimum and matrix. Verify on the hosted image, not just a local
machine that already happens to have both interpreters installed.
- Preserve Intel/ARM and standard/larger classes by default. A larger-runner
label may require organization configuration and paid access even when the
image itself is GA; ask before changing cost or resource class.
- Keep test selection conditions and matrix include/exclude entries consistent.
A new runner label can cause `if: matrix.os == 'old-label'` to skip the only
coverage job while the rollup still appears green.
- Read duplicated test/Dependabot/release setup paths together. Do not drop a
display service, system package, tool setup, or release guard in one path.
- Retain the draft PR until a maintainer reviews the actual hosted CI result.
Migrating the runner does not authorize merging, publishing, or rewriting
another open Python-version PR.
Loading
Loading