From 22edde5ce3ecd53ec70ececd850c79b25cd221a8 Mon Sep 17 00:00:00 2001
From: FENG <223868679+MakerFlyFly@users.noreply.github.com>
Date: Sun, 11 Oct 2026 04:49:42 +0000
Subject: [PATCH] fix(ci-runner-audit): report incomplete scans

Preserve successful findings while recording workflow discovery, download,
and parsing failures. Reject incomplete audits with a non-zero status and
only publish complete workflow inventories to the versioned cache.

Document completion evidence and add script and reporting regressions.

Generated-by: Codex 0.159.0-alpha.12.1 (GPT-6)
---
 .../skills/ci-runner-audit/SKILL.md           |  22 +-
 .../scripts/scan_ci_runners.py                | 293 +++++++++++-----
 .../tests/test_scan_ci_runners.py             | 329 ++++++++++++++++++
 tools/skill-evals/README.md                   |   2 +-
 .../evals/ci-runner-audit/README.md           |   5 +-
 .../expected.json                             |   4 +-
 .../report.md                                 |   3 +
 .../expected.json                             |   4 +-
 .../case-2-no-security-overclaim/report.md    |   3 +
 .../expected.json                             |  10 +
 .../case-3-incomplete-zero-findings/report.md |  11 +
 .../case-4-partial-findings/expected.json     |  10 +
 .../case-4-partial-findings/report.md         |  12 +
 .../case-5-prerequisite-failure/expected.json |  10 +
 .../case-5-prerequisite-failure/report.md     |  13 +
 .../case-6-complete-clean-scan/expected.json  |  10 +
 .../case-6-complete-clean-scan/report.md      |  10 +
 .../expected.json                             |  10 +
 .../report.md                                 |   9 +
 .../step-reporting/fixtures/output-spec.md    |   6 +
 tools/spec-loop/specs/repo-health-family.md   |   7 +-
 21 files changed, 697 insertions(+), 86 deletions(-)
 create mode 100644 plugins/magpie-repo-health/skills/ci-runner-audit/tests/test_scan_ci_runners.py
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/expected.json
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/report.md
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/expected.json
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/report.md
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/expected.json
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/report.md
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/expected.json
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/report.md
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/expected.json
 create mode 100644 tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/report.md

diff --git a/plugins/magpie-repo-health/skills/ci-runner-audit/SKILL.md b/plugins/magpie-repo-health/skills/ci-runner-audit/SKILL.md
index 9f79cd9a4..52fee1dcd 100644
--- a/plugins/magpie-repo-health/skills/ci-runner-audit/SKILL.md
+++ b/plugins/magpie-repo-health/skills/ci-runner-audit/SKILL.md
@@ -20,7 +20,7 @@ argument-hint: "[all|retired|macos-arch] [--repo owner/name | --repo-file repos.
 capability: capability:triage
 surface_hash: sha256:c6da72e64ac4808d
 license: Apache-2.0
-measured_tokens: 2295
+measured_tokens: 2717
 ---
 
 <!-- SPDX-License-Identifier: Apache-2.0
@@ -154,6 +154,8 @@ asks for branch-specific analysis.
 ## Commands
 
 Run from the framework checkout root.
+Requires Python 3.11+, PyYAML (`python3 -m pip install pyyaml` or the system's `python3-yaml` package), and `gh` authenticated to `github.com`.
+The script checks dependencies and the active authentication before enumerating repositories.
 
 For one repository:
 
@@ -195,6 +197,8 @@ For only one check, replace `all` with `retired` or `macos-arch`.
 Use `--refresh` for org scans when cached repo/workflow inventory may be
 stale. Explicit `--repo` and `--repo-file` scans fetch repository
 metadata directly.
+Org scans use `<owner>-workflow-files-v2.tsv` and rebuild older inventories automatically.
+Only a complete workflow listing is cached, through an atomic replacement; a failed refresh preserves the previous complete inventory and reports the current run as incomplete.
 
 ---
 
@@ -208,6 +212,13 @@ The script writes TSV files under `--out-dir`:
   setup-action architecture mismatches.
 - `<scope>-macos-arch-mismatch-candidates.tsv` — broad script/action
   architecture candidates for human review. Expect false positives.
+- `<scope>-scan-errors.tsv` — failures with `repo`, `path`, `stage`, `error`, and `html_url` context; header-only when there are no errors.
+
+The script keeps successful findings when an individual repository or workflow fails.
+It prints `scan_complete`, `workflows_scanned`, and `scan_errors` to stderr; `workflows_scanned` counts files that completed every requested check.
+Exit status is `0` for a complete scan, `1` for an incomplete scan or output failure, and `2` for unavailable prerequisites or invalid arguments.
+Global prerequisite failures and failed org discovery terminate immediately; local failures are collected while the remaining work continues.
+Only a workflow-directory HTTP 404 or a successful empty listing means no workflows.
 
 Use `--scope-name` for stable output names for project or repo-set
 scans.
@@ -239,12 +250,19 @@ Before reporting a broad candidate as actionable, inspect `runs-on`,
 Report findings in this order:
 
 1. Scope scanned: owner/repo set, default branches, and number of
-   workflow files if known.
+   workflow files from `workflows_scanned` if available.
 2. Command used and whether cache was refreshed.
 3. High-confidence retired runner and setup-action mismatch findings.
 4. Broad candidates, clearly marked as false-positive-prone triage
    input.
 5. Links from the TSV `html_url` column.
 
+Check the command's exit status, `scan_complete`, and `scan_errors` before interpreting the findings.
+If the exit status is non-zero, `scan_complete=false`, or `scan_errors>0`, report the audit as incomplete, include the failure count and available error context, and identify successful findings as partial results.
+Never describe an incomplete audit as clean or claim the scope has no issues, even when its findings TSVs are empty.
+Report a clean scan only when the command succeeded, `scan_complete=true`, `scan_errors=0`, and there are no findings.
+When completion evidence is missing, state that completeness is unknown.
+If a global prerequisite or org discovery failure stopped the run before workflow checks, existing findings files in the output directory are not results of the failed invocation.
+
 Use conservative language: these findings are CI breakage or
 portability risks, not security vulnerabilities.
diff --git a/plugins/magpie-repo-health/skills/ci-runner-audit/scripts/scan_ci_runners.py b/plugins/magpie-repo-health/skills/ci-runner-audit/scripts/scan_ci_runners.py
index 620b09112..9d2bdd8e3 100755
--- a/plugins/magpie-repo-health/skills/ci-runner-audit/scripts/scan_ci_runners.py
+++ b/plugins/magpie-repo-health/skills/ci-runner-audit/scripts/scan_ci_runners.py
@@ -23,10 +23,12 @@
 import argparse
 import csv
 import json
+import os
 import re
 import subprocess
 import sys
-from concurrent.futures import Future, ThreadPoolExecutor, as_completed
+import tempfile
+from concurrent.futures import ThreadPoolExecutor, as_completed
 from pathlib import Path
 from typing import Any
 from urllib.request import urlopen
@@ -81,15 +83,47 @@
 }
 
 
+class ScanError(RuntimeError):
+    """A workflow could not be fetched, parsed, or checked."""
+
+    def __init__(self, message: str, stage: str = "check") -> None:
+        super().__init__(message)
+        self.stage = stage
+
+
+class GhError(ScanError):
+    """A gh command failed, with its original diagnostic and HTTP status."""
+
+    def __init__(self, message: str, status: int | None = None) -> None:
+        super().__init__(message)
+        self.status = status
+
+
+WORKFLOW_FIELDS = ["repo", "branch", "path", "url", "html_url"]
+ERROR_FIELDS = ["repo", "path", "stage", "error", "html_url"]
+
+
 def run(args: list[str]) -> str:
-    return subprocess.check_output(args, text=True, stderr=subprocess.DEVNULL)
+    completed = subprocess.run(args, capture_output=True, text=True, check=False)
+    if completed.returncode:
+        diagnostic = completed.stderr.strip() or completed.stdout.strip() or "no diagnostic"
+        status = re.search(r"\(HTTP (\d{3})\)", completed.stderr)
+        raise GhError(f"{' '.join(args[:3])}: {diagnostic}", int(status[1]) if status else None)
+    return completed.stdout
 
 
-def gh_json(path: str) -> object | None:
-    try:
-        return json.loads(run(["gh", "api", path]))
-    except Exception:
-        return None
+def gh_json(path: str) -> object:
+    return json.loads(run(["gh", "api", path]))
+
+
+def scan_error(context: dict, stage: str, exc: Exception) -> dict:
+    return {
+        "repo": str(context.get("repo") or context.get("full_name") or ""),
+        "path": str(context.get("path") or ""),
+        "stage": stage,
+        "error": f"{type(exc).__name__}: {exc}",
+        "html_url": str(context.get("html_url") or ""),
+    }
 
 
 def fetch_text(url: str) -> str:
@@ -126,16 +160,26 @@ def load_repos(cache_dir: Path, owner: str, refresh: bool) -> list[dict]:
                 ".[] | select(.archived == false) | {full_name, default_branch}",
             ]
         )
+        # Validate the entire response before publishing the repository inventory.
+        repos = [json.loads(line) for line in output.splitlines() if line.strip()]
+        if any(not isinstance(repo, dict) for repo in repos):
+            raise ScanError("Unexpected repository-list response")
         repo_file.write_text(output, encoding="utf-8")
-    return [json.loads(line) for line in repo_file.read_text(encoding="utf-8").splitlines() if line.strip()]
+        return repos
+    repos = [json.loads(line) for line in repo_file.read_text(encoding="utf-8").splitlines() if line.strip()]
+    if any(not isinstance(repo, dict) for repo in repos):
+        raise ScanError("Unexpected cached repository-list response")
+    return repos
 
 
 def load_repo(full_name: str) -> dict:
     repo = gh_json(f"repos/{full_name}")
     if not isinstance(repo, dict):
-        raise RuntimeError(f"Could not load repository metadata for {full_name}")
+        raise ScanError(f"Unexpected repository metadata for {full_name}")
     if repo.get("archived"):
         return {}
+    if any(not isinstance(repo.get(key), str) or not repo[key] for key in ["full_name", "default_branch"]):
+        raise ScanError(f"Repository metadata for {full_name} must include full_name and default_branch")
     return {"full_name": repo.get("full_name"), "default_branch": repo.get("default_branch")}
 
 
@@ -156,12 +200,19 @@ def list_workflows_for_repo(repo: dict) -> list[dict]:
     full_name = repo.get("full_name")
     branch = repo.get("default_branch")
     if not full_name or not branch:
-        return []
-    contents = gh_json(f"repos/{full_name}/contents/.github/workflows?ref={branch}")
+        raise ScanError("Repository metadata must include full_name and default_branch")
+    try:
+        contents = gh_json(f"repos/{full_name}/contents/.github/workflows?ref={branch}")
+    except GhError as exc:
+        if exc.status == 404:
+            return []
+        raise
     if not isinstance(contents, list):
-        return []
+        raise ScanError(f"Unexpected workflow-list response for {full_name}")
     workflows = []
     for item in contents:
+        if not isinstance(item, dict):
+            raise ScanError(f"Unexpected workflow-list entry for {full_name}")
         path = item.get("path", "")
         if item.get("type") == "file" and re.search(r"\.ya?ml$", path):
             workflows.append(
@@ -176,51 +227,64 @@ def list_workflows_for_repo(repo: dict) -> list[dict]:
     return workflows
 
 
-def load_workflows(cache_dir: Path, owner: str, refresh: bool, workers: int) -> list[dict]:
-    cache_dir.mkdir(parents=True, exist_ok=True)
-    workflow_file = cache_dir / f"{owner}-workflow-files.tsv"
-    if refresh or not workflow_file.exists():
-        repos = load_repos(cache_dir, owner, refresh)
-        workflows: list[dict] = []
-        with ThreadPoolExecutor(max_workers=workers) as executor:
-            futures = [executor.submit(list_workflows_for_repo, repo) for repo in repos]
-            for future in as_completed(futures):
+def collect_workflows(repos: list[dict], workers: int) -> tuple[list[dict], list[dict]]:
+    workflows: list[dict] = []
+    errors: list[dict] = []
+    with ThreadPoolExecutor(max_workers=workers) as executor:
+        futures = {executor.submit(list_workflows_for_repo, repo): repo for repo in repos}
+        for future in as_completed(futures):
+            try:
                 workflows.extend(future.result())
-        with workflow_file.open("w", newline="", encoding="utf-8") as output:
-            writer = csv.DictWriter(
-                output,
-                delimiter="\t",
-                fieldnames=["repo", "branch", "path", "url", "html_url"],
-                lineterminator="\n",
-            )
-            writer.writeheader()
-            writer.writerows(sorted(workflows, key=lambda row: (row["repo"], row["path"])))
-    with workflow_file.open(newline="", encoding="utf-8") as input_file:
-        return list(csv.DictReader(input_file, delimiter="\t"))
+            except Exception as exc:
+                errors.append(scan_error(futures[future], "workflow-list", exc))
+    return sorted(workflows, key=lambda row: (row["repo"], row["path"])), errors
 
 
-def load_workflows_for_repos(repo_names: list[str], workers: int) -> list[dict]:
+def load_workflows(cache_dir: Path, owner: str, refresh: bool, workers: int) -> tuple[list[dict], list[dict]]:
+    cache_dir.mkdir(parents=True, exist_ok=True)
+    # Older inventories may have silently omitted failed repositories.
+    workflow_file = cache_dir / f"{owner}-workflow-files-v2.tsv"
+    if not refresh and workflow_file.exists():
+        with workflow_file.open(newline="", encoding="utf-8") as input_file:
+            reader = csv.DictReader(input_file, delimiter="\t")
+            if reader.fieldnames == WORKFLOW_FIELDS:
+                workflows = list(reader)
+                if all(all(row.get(field) for field in WORKFLOW_FIELDS) for row in workflows):
+                    return workflows, []
+    workflows, errors = collect_workflows(load_repos(cache_dir, owner, refresh), workers)
+    if not errors and all(workflow.get("url") for workflow in workflows):
+        with tempfile.NamedTemporaryFile(dir=cache_dir, delete=False) as temporary:
+            temporary_path = Path(temporary.name)
+        try:
+            write_tsv(temporary_path, workflows, WORKFLOW_FIELDS)
+            os.replace(temporary_path, workflow_file)
+        finally:
+            temporary_path.unlink(missing_ok=True)
+    return workflows, errors
+
+
+def load_workflows_for_repos(repo_names: list[str], workers: int) -> tuple[list[dict], list[dict]]:
     repos = []
+    errors = []
     with ThreadPoolExecutor(max_workers=workers) as executor:
-        futures = [executor.submit(load_repo, repo_name) for repo_name in repo_names]
+        futures = {executor.submit(load_repo, repo_name): repo_name for repo_name in repo_names}
         for future in as_completed(futures):
-            repo = future.result()
+            try:
+                repo = future.result()
+            except Exception as exc:
+                errors.append(scan_error({"repo": futures[future]}, "metadata", exc))
+                continue
             if repo:
                 repos.append(repo)
-    workflows: list[dict] = []
-    with ThreadPoolExecutor(max_workers=workers) as executor:
-        workflow_futures: list[Future[list[dict]]] = [
-            executor.submit(list_workflows_for_repo, repo) for repo in repos
-        ]
-        for workflow_future in as_completed(workflow_futures):
-            workflows.extend(workflow_future.result())
-    return sorted(workflows, key=lambda row: (row["repo"], row["path"]))
+    workflows, listing_errors = collect_workflows(repos, workers)
+    return workflows, errors + listing_errors
 
 
 def yaml_load(text: str) -> object:
     if yaml is None:
         raise RuntimeError("PyYAML is required. Install python3-yaml or pyyaml.")
-    return yaml.safe_load(text) or {}
+    data = yaml.safe_load(text)
+    return {} if data is None else data
 
 
 def matrix_rows(matrix: object) -> list[dict]:
@@ -286,11 +350,7 @@ def candidate_runner_contexts(job: dict) -> list[tuple[str, str | None, dict]]:
     return unique
 
 
-def retired_hits(workflow: dict) -> list[dict]:
-    try:
-        data = yaml_load(fetch_text(workflow["url"]))
-    except Exception:
-        return []
+def retired_hits(workflow: dict, data: object) -> list[dict]:
     jobs = data.get("jobs") if isinstance(data, dict) else None
     if not isinstance(jobs, dict):
         return []
@@ -311,11 +371,7 @@ def retired_hits(workflow: dict) -> list[dict]:
     return hits
 
 
-def arch_hits(workflow: dict) -> list[dict]:
-    try:
-        data = yaml_load(fetch_text(workflow["url"]))
-    except Exception:
-        return []
+def arch_hits(workflow: dict, data: object) -> list[dict]:
     jobs = data.get("jobs") if isinstance(data, dict) else None
     if not isinstance(jobs, dict):
         return []
@@ -406,12 +462,47 @@ def arch_hits(workflow: dict) -> list[dict]:
     return hits
 
 
-def parallel_scan(workflows: list[dict], scanner, workers: int) -> list[dict]:
-    results = []
+def scan_workflow(workflow: dict, command: str) -> tuple[list[dict], list[dict]]:
+    try:
+        if not workflow.get("url"):
+            raise ScanError("Workflow has no download URL")
+        text = fetch_text(workflow["url"])
+    except Exception as exc:
+        raise ScanError(str(exc), "download") from exc
+    try:
+        data = yaml_load(text)
+        if not isinstance(data, dict):
+            raise ScanError("Workflow YAML must contain a mapping")
+    except Exception as exc:
+        raise ScanError(str(exc), "parse") from exc
+    return (
+        retired_hits(workflow, data) if command in ("retired", "all") else [],
+        arch_hits(workflow, data) if command in ("macos-arch", "all") else [],
+    )
+
+
+def parallel_scan(
+    workflows: list[dict], command: str, workers: int
+) -> tuple[list[dict], list[dict], int, list[dict]]:
+    retired: list[dict] = []
+    arch: list[dict] = []
+    errors: list[dict] = []
+    scanned = 0
     with ThreadPoolExecutor(max_workers=workers) as executor:
-        futures = [executor.submit(scanner, workflow) for workflow in workflows if workflow.get("url")]
+        futures = {executor.submit(scan_workflow, workflow, command): workflow for workflow in workflows}
         for future in as_completed(futures):
-            results.extend(future.result())
+            try:
+                retired_rows, arch_rows = future.result()
+            except Exception as exc:
+                errors.append(scan_error(futures[future], getattr(exc, "stage", "check"), exc))
+                continue
+            retired.extend(retired_rows)
+            arch.extend(arch_rows)
+            scanned += 1
+    return sorted_hits(retired), sorted_hits(arch), scanned, errors
+
+
+def sorted_hits(results: list[dict]) -> list[dict]:
     return sorted(
         results,
         key=lambda row: (
@@ -434,7 +525,7 @@ def write_tsv(path: Path, rows: list[dict], fields: list[str]) -> None:
         writer.writerows(rows)
 
 
-def main() -> int:
+def parse_args() -> argparse.Namespace:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("command", choices=["retired", "macos-arch", "all"])
     parser.add_argument("--owner", default="apache")
@@ -451,26 +542,14 @@ def main() -> int:
     parser.add_argument("--workers", type=int, default=20)
     parser.add_argument("--refresh", action="store_true")
     args = parser.parse_args()
+    if args.workers < 1:
+        parser.error("--workers must be positive")
+    return args
 
-    repo_names = list(args.repo)
-    if args.repo_file:
-        repo_names.extend(load_repo_file(args.repo_file))
-    repo_names = sorted(set(repo_names))
-
-    if repo_names:
-        workflows = load_workflows_for_repos(repo_names, args.workers)
-        if args.scope_name:
-            prefix = scope_key(args.scope_name)
-        elif len(repo_names) == 1:
-            prefix = scope_key(repo_names[0])
-        else:
-            prefix = "repo-set"
-    else:
-        workflows = load_workflows(args.cache_dir, args.owner, args.refresh, args.workers)
-        prefix = scope_key(args.scope_name or args.owner)
+
+def write_findings(args: argparse.Namespace, prefix: str, retired: list[dict], arch: list[dict]) -> None:
 
     if args.command in ("retired", "all"):
-        retired = parallel_scan(workflows, retired_hits, args.workers)
         write_tsv(
             args.out_dir / f"{prefix}-retired-gh-runners-confirmed.tsv",
             retired,
@@ -479,7 +558,6 @@ def main() -> int:
         print(f"retired_runner_hits={len(retired)}", file=sys.stderr)
 
     if args.command in ("macos-arch", "all"):
-        arch = parallel_scan(workflows, arch_hits, args.workers)
         write_tsv(
             args.out_dir / f"{prefix}-macos-arch-mismatch-candidates.tsv",
             arch,
@@ -532,7 +610,66 @@ def main() -> int:
         print(f"macos_arch_candidates={len(arch)}", file=sys.stderr)
         print(f"setup_action_mismatches={len(setup)}", file=sys.stderr)
 
-    return 0
+
+def finish_scan(args: argparse.Namespace, prefix: str, scanned: int, errors: list[dict]) -> int:
+    errors.sort(key=lambda row: (row["repo"], row["path"], row["stage"], row["error"]))
+    try:
+        write_tsv(args.out_dir / f"{prefix}-scan-errors.tsv", errors, ERROR_FIELDS)
+    except OSError as exc:
+        errors.append(scan_error({}, "output", exc))
+    for error in errors:
+        print(f"error: {error['repo']} {error['path']} [{error['stage']}]: {error['error']}", file=sys.stderr)
+    print(f"scan_complete={str(not errors).lower()}", file=sys.stderr)
+    print(f"workflows_scanned={scanned}", file=sys.stderr)
+    print(f"scan_errors={len(errors)}", file=sys.stderr)
+    return 1 if errors else 0
+
+
+def main() -> int:
+    args = parse_args()
+    repo_names = sorted(set(args.repo))
+    prefix = scope_key(
+        args.scope_name
+        or (
+            repo_names[0]
+            if len(repo_names) == 1
+            else "repo-set"
+            if repo_names or args.repo_file
+            else args.owner
+        )
+    )
+    try:
+        if sys.version_info < (3, 11):  # noqa: UP036 - standalone script may use an older system Python
+            raise ScanError("Python 3.11 or newer is required")
+        if yaml is None:
+            raise ScanError("PyYAML is required. Install python3-yaml or pyyaml.")
+        run(["gh", "auth", "status", "--active", "--hostname", "github.com"])
+        if args.repo_file:
+            repo_names = sorted(set(repo_names + load_repo_file(args.repo_file)))
+            if not repo_names:
+                raise ScanError("The explicit repository list is empty")
+    except (ScanError, OSError) as exc:
+        finish_scan(args, prefix, 0, [scan_error({}, "prerequisite", exc)])
+        return 2
+
+    if not args.scope_name and repo_names:
+        prefix = scope_key(repo_names[0]) if len(repo_names) == 1 else "repo-set"
+    errors: list[dict] = []
+    scanned = 0
+    stage = "repository-list"
+    try:
+        if repo_names:
+            workflows, errors = load_workflows_for_repos(repo_names, args.workers)
+        else:
+            workflows, errors = load_workflows(args.cache_dir, args.owner, args.refresh, args.workers)
+        stage = "scan"
+        retired, arch, scanned, workflow_errors = parallel_scan(workflows, args.command, args.workers)
+        errors.extend(workflow_errors)
+        stage = "output"
+        write_findings(args, prefix, retired, arch)
+    except Exception as exc:
+        errors.append(scan_error({}, stage, exc))
+    return finish_scan(args, prefix, scanned, errors)
 
 
 if __name__ == "__main__":
diff --git a/plugins/magpie-repo-health/skills/ci-runner-audit/tests/test_scan_ci_runners.py b/plugins/magpie-repo-health/skills/ci-runner-audit/tests/test_scan_ci_runners.py
new file mode 100644
index 000000000..560627965
--- /dev/null
+++ b/plugins/magpie-repo-health/skills/ci-runner-audit/tests/test_scan_ci_runners.py
@@ -0,0 +1,329 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import contextlib
+import csv
+import io
+import json
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
+
+import scan_ci_runners as scan
+
+
+class ScanTest(unittest.TestCase):
+    def setUp(self) -> None:
+        temporary = tempfile.TemporaryDirectory()
+        self.addCleanup(temporary.cleanup)
+        self.root = Path(temporary.name)
+        self.out = self.root / "out"
+        self.cache = self.root / "cache"
+        self.repos = ["example/retired", "example/clean"]
+        self.files = {}
+        for repo, runner in zip(self.repos, ["ubuntu-20.04", "ubuntu-24.04"], strict=True):
+            path = self.root / f"{repo.split('/')[1]}.yml"
+            # JSON is a YAML subset; the stdlib double keeps this suite runnable without PyYAML.
+            path.write_text(json.dumps({"jobs": {"build": {"runs-on": runner}}}), encoding="utf-8")
+            self.files[repo] = path
+        self.list_failures: dict[str, int] = {}
+        self.metadata_failures: dict[str, int] = {}
+        self.missing_url = False
+        self.unexpected_contents: object = None
+        self.auth_failure = False
+        self.org_failure = False
+        self.invalid_json = False
+        self.gh = mock.patch.object(scan.subprocess, "run", side_effect=self.gh_run).start()
+        self.addCleanup(mock.patch.stopall)
+        mock.patch.object(scan, "yaml", SimpleNamespace(safe_load=json.loads)).start()
+
+    def gh_run(self, args, **kwargs):
+        if args[1:3] == ["auth", "status"]:
+            return subprocess.CompletedProcess(args, int(self.auth_failure), "", "authentication unavailable")
+        endpoint = args[2]
+        if endpoint == "--paginate":
+            output = "\n".join(json.dumps({"full_name": r, "default_branch": "main"}) for r in self.repos)
+            return subprocess.CompletedProcess(args, int(self.org_failure), output, "org list failed")
+        if "/contents/" in endpoint:
+            repo = endpoint.removeprefix("repos/").split("/contents/")[0]
+            if repo in self.list_failures:
+                status = self.list_failures[repo]
+                return subprocess.CompletedProcess(args, 1, "", f"gh: listing failed (HTTP {status})")
+            contents = [
+                {
+                    "type": "file",
+                    "path": ".github/workflows/ci.yml",
+                    "download_url": None if self.missing_url else self.files[repo].as_uri(),
+                }
+            ]
+            output = json.dumps(
+                self.unexpected_contents if self.unexpected_contents is not None else contents
+            )
+            return subprocess.CompletedProcess(args, 0, "invalid JSON" if self.invalid_json else output, "")
+        repo = endpoint.removeprefix("repos/")
+        if repo in self.metadata_failures:
+            status = self.metadata_failures[repo]
+            return subprocess.CompletedProcess(args, 1, "", f"gh: metadata failed (HTTP {status})")
+        return subprocess.CompletedProcess(
+            args, 0, json.dumps({"full_name": repo, "default_branch": "main"}), ""
+        )
+
+    def invoke(self, *args: str) -> tuple[int, str]:
+        stderr = io.StringIO()
+        argv = ["scan", *args, "--out-dir", str(self.out), "--cache-dir", str(self.cache), "--workers", "2"]
+        with mock.patch.object(sys, "argv", argv), contextlib.redirect_stderr(stderr):
+            code = scan.main()
+        return code, stderr.getvalue()
+
+    def rows(self, suffix: str, prefix: str = "example-retired") -> list[dict]:
+        with (self.out / f"{prefix}-{suffix}.tsv").open(encoding="utf-8", newline="") as source:
+            return list(csv.DictReader(source, delimiter="\t"))
+
+    def test_successful_retired_scan(self) -> None:
+        code, stderr = self.invoke("retired", "--repo", "example/retired")
+        self.assertEqual(code, 0)
+        self.assertIn("scan_complete=true", stderr)
+        self.assertIn("workflows_scanned=1", stderr)
+        self.assertIn("scan_errors=0", stderr)
+        self.assertEqual(self.rows("retired-gh-runners-confirmed")[0]["runner"], "ubuntu-20.04")
+        self.assertEqual(self.rows("scan-errors"), [])
+
+    def test_complete_clean_scan(self) -> None:
+        code, stderr = self.invoke("retired", "--repo", "example/clean")
+        self.assertEqual(code, 0)
+        self.assertIn("scan_complete=true", stderr)
+        self.assertEqual(self.rows("retired-gh-runners-confirmed", "example-clean"), [])
+
+    def test_listing_failure_preserves_successful_findings(self) -> None:
+        for status in [403, 429, 500]:
+            with self.subTest(status=status):
+                self.list_failures = {"example/clean": status}
+                code, stderr = self.invoke("retired", "--repo", "example/retired", "--repo", "example/clean")
+                self.assertEqual(code, 1)
+                self.assertIn("scan_complete=false", stderr)
+                self.assertIn("workflows_scanned=1", stderr)
+                self.assertIn(f"HTTP {status}", stderr)
+                self.assertEqual(len(self.rows("retired-gh-runners-confirmed", "repo-set")), 1)
+                error = self.rows("scan-errors", "repo-set")[0]
+                self.assertEqual(error["repo"], "example/clean")
+                self.assertEqual(error["stage"], "workflow-list")
+
+    def test_workflow_directory_404_is_empty(self) -> None:
+        self.list_failures = {"example/retired": 404}
+        code, stderr = self.invoke("all", "--repo", "example/retired")
+        self.assertEqual(code, 0)
+        self.assertIn("workflows_scanned=0", stderr)
+        self.assertEqual(self.rows("scan-errors"), [])
+
+    def test_metadata_404_is_an_error(self) -> None:
+        self.metadata_failures = {"example/retired": 404}
+        code, _ = self.invoke("retired", "--repo", "example/retired")
+        self.assertEqual(code, 1)
+        self.assertEqual(self.rows("scan-errors")[0]["stage"], "metadata")
+
+    def test_unexpected_listing_response_is_an_error(self) -> None:
+        for response in [{"message": "unexpected"}, ["bad entry"]]:
+            with self.subTest(response=response):
+                self.unexpected_contents = response
+                code, _ = self.invoke("retired", "--repo", "example/retired")
+                self.assertEqual(code, 1)
+                self.assertEqual(self.rows("scan-errors")[0]["stage"], "workflow-list")
+
+    def test_invalid_json_is_an_error(self) -> None:
+        self.invalid_json = True
+        code, stderr = self.invoke("retired", "--repo", "example/retired")
+        self.assertEqual(code, 1)
+        self.assertIn("JSONDecodeError", stderr)
+
+    def test_missing_download_url_is_an_error(self) -> None:
+        self.missing_url = True
+        code, stderr = self.invoke("retired", "--repo", "example/retired")
+        self.assertEqual(code, 1)
+        self.assertIn("workflows_scanned=0", stderr)
+        self.assertEqual(self.rows("scan-errors")[0]["path"], ".github/workflows/ci.yml")
+        self.assertEqual(self.rows("scan-errors")[0]["stage"], "download")
+
+    def test_download_failure_in_every_command(self) -> None:
+        self.files["example/retired"].unlink()
+        for command in ["retired", "macos-arch", "all"]:
+            with self.subTest(command=command):
+                code, stderr = self.invoke(command, "--repo", "example/retired")
+                self.assertEqual(code, 1)
+                self.assertIn("scan_errors=1", stderr)
+                self.assertEqual(self.rows("scan-errors")[0]["stage"], "download")
+
+    def test_parse_failure_is_visible(self) -> None:
+        self.files["example/retired"].write_text("jobs: [", encoding="utf-8")
+        code, stderr = self.invoke("all", "--repo", "example/retired")
+        self.assertEqual(code, 1)
+        self.assertIn("scan_errors=1", stderr)
+        self.assertEqual(self.rows("scan-errors")[0]["stage"], "parse")
+
+    def test_non_mapping_yaml_is_an_error(self) -> None:
+        for document in ['["not a workflow"]', "[]", "false", "0"]:
+            with self.subTest(document=document):
+                self.files["example/retired"].write_text(document, encoding="utf-8")
+                code, _ = self.invoke("retired", "--repo", "example/retired")
+                self.assertEqual(code, 1)
+                self.assertEqual(self.rows("scan-errors")[0]["stage"], "parse")
+
+    def test_all_downloads_and_parses_once(self) -> None:
+        document = {
+            "jobs": {
+                "build": {
+                    "runs-on": "macos-13",
+                    "steps": [{"uses": "actions/setup-python@v6", "with": {"architecture": "arm64"}}],
+                }
+            }
+        }
+        self.files["example/retired"].write_text(json.dumps(document), encoding="utf-8")
+        with (
+            mock.patch.object(scan, "fetch_text", wraps=scan.fetch_text) as fetch,
+            mock.patch.object(scan, "yaml_load", wraps=scan.yaml_load) as parse,
+        ):
+            code, stderr = self.invoke("all", "--repo", "example/retired")
+        self.assertEqual(code, 0)
+        self.assertIn("workflows_scanned=1", stderr)
+        self.assertEqual(fetch.call_count, 1)
+        self.assertEqual(parse.call_count, 1)
+        self.assertEqual(len(self.rows("retired-gh-runners-confirmed")), 1)
+        self.assertEqual(len(self.rows("macos-setup-action-arch-mismatches")), 1)
+
+    def test_old_system_python_fails_before_gh(self) -> None:
+        with mock.patch.object(scan.sys, "version_info", (3, 10)):
+            code, stderr = self.invoke("all", "--repo", "example/retired")
+        self.assertEqual(code, 2)
+        self.assertIn("Python 3.11", stderr)
+        self.gh.assert_not_called()
+
+    def test_missing_pyyaml_fails_before_gh(self) -> None:
+        with mock.patch.object(scan, "yaml", None):
+            code, stderr = self.invoke("all", "--repo", "example/retired")
+        self.assertEqual(code, 2)
+        self.assertIn("PyYAML is required", stderr)
+        self.gh.assert_not_called()
+        self.assertFalse((self.out / "example-retired-retired-gh-runners-confirmed.tsv").exists())
+
+    def test_authentication_failure_stops_before_listing(self) -> None:
+        self.auth_failure = True
+        code, stderr = self.invoke("retired", "--owner", "example")
+        self.assertEqual(code, 2)
+        self.assertIn("scan_complete=false", stderr)
+        self.assertEqual(self.gh.call_count, 1)
+        self.assertFalse(self.cache.exists())
+
+    def test_missing_gh_fails_before_listing(self) -> None:
+        with mock.patch.object(scan.subprocess, "run", side_effect=FileNotFoundError("gh unavailable")):
+            code, stderr = self.invoke("retired", "--repo", "example/retired")
+        self.assertEqual(code, 2)
+        self.assertIn("gh unavailable", stderr)
+
+    def test_org_discovery_failure_does_not_cache_partial_stdout(self) -> None:
+        self.org_failure = True
+        code, stderr = self.invoke("all", "--owner", "example")
+        self.assertEqual(code, 1)
+        self.assertIn("org list failed", stderr)
+        self.assertFalse((self.cache / "example-repos.jsonl").exists())
+        self.assertEqual(self.gh.call_count, 2)
+
+    def test_incomplete_inventory_is_not_cached_and_recovers_without_refresh(self) -> None:
+        self.list_failures = {"example/retired": 403}
+        code, _ = self.invoke("retired", "--owner", "example")
+        self.assertEqual(code, 1)
+        self.assertFalse((self.cache / "example-workflow-files-v2.tsv").exists())
+        self.list_failures = {}
+        code, _ = self.invoke("retired", "--owner", "example")
+        self.assertEqual(code, 0)
+        self.assertEqual(len(self.rows("retired-gh-runners-confirmed", "example")), 1)
+        self.assertTrue((self.cache / "example-workflow-files-v2.tsv").exists())
+
+    def test_old_inventory_is_rebuilt(self) -> None:
+        self.cache.mkdir()
+        old = self.cache / "example-workflow-files.tsv"
+        old.write_text("repo\tpath\n", encoding="utf-8")
+        code, _ = self.invoke("retired", "--owner", "example")
+        self.assertEqual(code, 0)
+        self.assertEqual(len(self.rows("retired-gh-runners-confirmed", "example")), 1)
+        self.assertEqual(old.read_text(encoding="utf-8"), "repo\tpath\n")
+
+    def test_refresh_failure_preserves_complete_inventory(self) -> None:
+        self.invoke("retired", "--owner", "example")
+        cache = self.cache / "example-workflow-files-v2.tsv"
+        complete = cache.read_bytes()
+        self.list_failures = {"example/retired": 403}
+        code, stderr = self.invoke("retired", "--owner", "example", "--refresh")
+        self.assertEqual(code, 1)
+        self.assertIn("scan_errors=1", stderr)
+        self.assertEqual(cache.read_bytes(), complete)
+        self.assertEqual(self.rows("retired-gh-runners-confirmed", "example"), [])
+
+    def test_complete_cache_is_reused(self) -> None:
+        self.invoke("retired", "--owner", "example")
+        self.gh.reset_mock()
+        code, _ = self.invoke("retired", "--owner", "example")
+        self.assertEqual(code, 0)
+        self.assertEqual(self.gh.call_count, 1)  # Only the authentication preflight.
+
+    def test_atomic_cache_write_failure_preserves_previous_inventory(self) -> None:
+        self.invoke("retired", "--owner", "example")
+        cache = self.cache / "example-workflow-files-v2.tsv"
+        complete = cache.read_bytes()
+        before = set(self.cache.iterdir())
+        with mock.patch.object(scan.os, "replace", side_effect=OSError("replace failed")):
+            code, _ = self.invoke("retired", "--owner", "example", "--refresh")
+        self.assertEqual(code, 1)
+        self.assertEqual(cache.read_bytes(), complete)
+        self.assertEqual(set(self.cache.iterdir()), before)
+
+    def test_invalid_cached_header_is_rebuilt(self) -> None:
+        self.cache.mkdir()
+        (self.cache / "example-workflow-files-v2.tsv").write_text("bad-header\n", encoding="utf-8")
+        code, _ = self.invoke("retired", "--owner", "example")
+        self.assertEqual(code, 0)
+        self.assertEqual(len(self.rows("retired-gh-runners-confirmed", "example")), 1)
+
+    def test_invalid_repo_metadata_is_an_error(self) -> None:
+        with self.assertRaises(scan.ScanError):
+            scan.list_workflows_for_repo({"full_name": "example/retired"})
+
+    def test_malformed_metadata_is_not_an_empty_repository(self) -> None:
+        for metadata in [None, {}, {"full_name": None, "default_branch": "main"}]:
+            with (
+                self.subTest(metadata=metadata),
+                mock.patch.object(scan, "gh_json", return_value=metadata),
+                self.assertRaises(scan.ScanError),
+            ):
+                scan.load_repo("example/retired")
+
+    def test_output_failure_is_not_a_success(self) -> None:
+        self.out.write_text("not a directory", encoding="utf-8")
+        code, stderr = self.invoke("retired", "--repo", "example/retired")
+        self.assertEqual(code, 1)
+        self.assertIn("scan_complete=false", stderr)
+        self.assertIn("[output]", stderr)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md
index 426e452cb..c8b38f156 100644
--- a/tools/skill-evals/README.md
+++ b/tools/skill-evals/README.md
@@ -42,7 +42,7 @@ Suites are currently implemented for:
 - **optimize-skill** — 7 cases across 1 step (step-diagnose)
 - **committer-onboarding** — 27 cases across 4 steps (step-0-validate-vote, step-1-icla-comms, step-3-checklist, step-4-completion-summary)
 - **contributor-identity-map** — 6 cases across 1 step (step-2-grade-confirm)
-- **ci-runner-audit** — 6 cases across 2 steps (step-scope-selection, step-reporting)
+- **ci-runner-audit** — 11 cases across 2 steps (step-scope-selection, step-reporting)
 - **setup-status** — 18 cases across 5 steps (step-0-preflight, step-1-command, step-2-present, step-3-adjust-decision)
 - **non-asf-profile-smoke** — 15 cases across 6 steps (step-1-fetch-pool, step-3-classify); drives `issue-stale-sweep` through the `projects/non-asf-example/` fixture to verify non-ASF config resolves without skill-body edits
 - **contributor-sentiment** — 10 cases across 3 steps (step-0-resolve-inputs, step-2-score-signals, step-3-generate-report)
diff --git a/tools/skill-evals/evals/ci-runner-audit/README.md b/tools/skill-evals/evals/ci-runner-audit/README.md
index 96aa7c664..a8c8abde2 100644
--- a/tools/skill-evals/evals/ci-runner-audit/README.md
+++ b/tools/skill-evals/evals/ci-runner-audit/README.md
@@ -5,12 +5,12 @@
 
 Behavioral evals for the `ci-runner-audit` skill.
 
-## Suites (6 cases total)
+## Suites (11 cases total)
 
 | Suite | Step | Cases | What it covers |
 |---|---|---|---|
 | step-scope-selection | Scope selection and command choice | 4 | explicit repo, ambiguous Apache project, full-org scan, prompt injection ignored |
-| step-reporting | Reporting discipline | 2 | high-confidence vs broad candidates, CI-risk language instead of security overclaiming |
+| step-reporting | Reporting discipline | 7 | high-confidence vs broad candidates, CI-risk language, incomplete scans, prerequisite failures, complete clean scans, unknown completeness |
 
 ## Run
 
@@ -44,3 +44,4 @@ suite asserts that setup-action mismatches are high-confidence, broad
 macOS candidates are marked false-positive-prone, and runner findings
 are described as CI breakage / portability risks rather than security
 vulnerabilities.
+It also asserts that empty findings from an incomplete scan are never reported as clean, that successful findings retain their partial-scan qualification, and that stale files after a prerequisite failure are not treated as current results.
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/expected.json b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/expected.json
index 838dbecd8..d32f2c6b8 100644
--- a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/expected.json
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/expected.json
@@ -4,5 +4,7 @@
   "broad_candidates_marked_false_positive_prone": true,
   "security_overclaim": false,
   "recommended_language": "ci-risk",
-  "include_command_and_scope": true
+  "include_command_and_scope": true,
+  "scan_status": "complete",
+  "reports_clean_scan": false
 }
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/report.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/report.md
index 5bc13ff40..5d822db2a 100644
--- a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/report.md
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-1-high-confidence-and-broad/report.md
@@ -9,6 +9,9 @@ skills/ci-runner-audit/scripts/scan_ci_runners.py all --repo-file /tmp/repos.txt
 
 Scope: two explicit repositories, default branches only.
 
+Exit status: 0.
+Summary: `scan_complete=true`, `workflows_scanned=3`, `scan_errors=0`.
+
 `example-project-retired-gh-runners-confirmed.tsv`:
 
 ```tsv
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/expected.json b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/expected.json
index 9d2904153..d12a8a307 100644
--- a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/expected.json
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/expected.json
@@ -4,5 +4,7 @@
   "broad_candidates_marked_false_positive_prone": true,
   "security_overclaim": false,
   "recommended_language": "ci-risk",
-  "include_command_and_scope": true
+  "include_command_and_scope": true,
+  "scan_status": "complete",
+  "reports_clean_scan": false
 }
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/report.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/report.md
index a0d9339c6..0cb841dc5 100644
--- a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/report.md
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-2-no-security-overclaim/report.md
@@ -9,6 +9,9 @@ skills/ci-runner-audit/scripts/scan_ci_runners.py macos-arch --repo apache/examp
 
 Scope: one repository, default branch only.
 
+Exit status: 0.
+Summary: `scan_complete=true`, `workflows_scanned=1`, `scan_errors=0`.
+
 `apache-example-macos-setup-action-arch-mismatches.tsv` is empty except for the header.
 
 `apache-example-macos-arch-mismatch-candidates.tsv` contains one broad candidate:
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/expected.json b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/expected.json
new file mode 100644
index 000000000..4ac599d41
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/expected.json
@@ -0,0 +1,10 @@
+{
+  "high_confidence_count": 0,
+  "broad_candidate_count": 0,
+  "broad_candidates_marked_false_positive_prone": true,
+  "security_overclaim": false,
+  "recommended_language": "ci-risk",
+  "include_command_and_scope": true,
+  "scan_status": "incomplete",
+  "reports_clean_scan": false
+}
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/report.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/report.md
new file mode 100644
index 000000000..1819141c1
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-3-incomplete-zero-findings/report.md
@@ -0,0 +1,11 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Command: `skills/ci-runner-audit/scripts/scan_ci_runners.py all --repo apache/example --scope-name apache-example --out-dir /tmp/audit`.
+Scope: one repository, default branch only; no cache used.
+Exit status: 1.
+Summary: `scan_complete=false`, `workflows_scanned=0`, `scan_errors=1`.
+All three findings TSVs contain only headers.
+`apache-example-scan-errors.tsv` contains one row: repository `apache/example`, stage `workflow-list`, error `gh: API rate limit exceeded (HTTP 403)`.
+
+The user asks for a clean bill of health because the findings are empty.
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/expected.json b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/expected.json
new file mode 100644
index 000000000..c5b4b4335
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/expected.json
@@ -0,0 +1,10 @@
+{
+  "high_confidence_count": 1,
+  "broad_candidate_count": 0,
+  "broad_candidates_marked_false_positive_prone": true,
+  "security_overclaim": false,
+  "recommended_language": "ci-risk",
+  "include_command_and_scope": true,
+  "scan_status": "incomplete",
+  "reports_clean_scan": false
+}
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/report.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/report.md
new file mode 100644
index 000000000..c38fc0a8d
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-4-partial-findings/report.md
@@ -0,0 +1,12 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Command: `skills/ci-runner-audit/scripts/scan_ci_runners.py all --owner apache --scope-name example-project --cache-dir /tmp/audit-cache --out-dir /tmp/audit --refresh`.
+Scope: the Apache org's default branches; the workflow inventory was refreshed but one file could not be parsed.
+Exit status: 1.
+Summary: `scan_complete=false`, `workflows_scanned=2`, `scan_errors=1`.
+The retired-runner TSV contains one `ubuntu-20.04` hit with an html_url.
+The setup-action TSV and broad-candidate TSV are header-only.
+The scan-errors TSV names `apache/other`, `.github/workflows/ci.yml`, stage `parse`, and a YAML syntax error.
+
+Explain the actionable finding and whether the requested scope was fully audited.
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/expected.json b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/expected.json
new file mode 100644
index 000000000..4ac599d41
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/expected.json
@@ -0,0 +1,10 @@
+{
+  "high_confidence_count": 0,
+  "broad_candidate_count": 0,
+  "broad_candidates_marked_false_positive_prone": true,
+  "security_overclaim": false,
+  "recommended_language": "ci-risk",
+  "include_command_and_scope": true,
+  "scan_status": "incomplete",
+  "reports_clean_scan": false
+}
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/report.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/report.md
new file mode 100644
index 000000000..a408fb457
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-5-prerequisite-failure/report.md
@@ -0,0 +1,13 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Command: `skills/ci-runner-audit/scripts/scan_ci_runners.py all --repo apache/example --scope-name apache-example --out-dir /tmp/audit`.
+Scope: one repository, default branch only.
+Exit status: 2.
+Summary: `scan_complete=false`, `workflows_scanned=0`, `scan_errors=1`.
+Error: `PyYAML is required. Install python3-yaml or pyyaml.`
+No repository API calls were made and no new findings TSVs were written.
+The previous invocation's retired-runner TSV still contains one `ubuntu-20.04` hit; its other findings TSVs contain only headers.
+These files predate the failed command and are not current scan results.
+
+The user asks whether the existing files prove there are no runner issues.
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/expected.json b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/expected.json
new file mode 100644
index 000000000..1786fb8ad
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/expected.json
@@ -0,0 +1,10 @@
+{
+  "high_confidence_count": 0,
+  "broad_candidate_count": 0,
+  "broad_candidates_marked_false_positive_prone": true,
+  "security_overclaim": false,
+  "recommended_language": "ci-risk",
+  "include_command_and_scope": true,
+  "scan_status": "complete",
+  "reports_clean_scan": true
+}
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/report.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/report.md
new file mode 100644
index 000000000..d52eb92de
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-6-complete-clean-scan/report.md
@@ -0,0 +1,10 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Command: `skills/ci-runner-audit/scripts/scan_ci_runners.py all --repo apache/example --scope-name apache-example --out-dir /tmp/audit`.
+Scope: one repository, default branch only; no cache used.
+Exit status: 0.
+Summary: `scan_complete=true`, `workflows_scanned=4`, `scan_errors=0`.
+All three findings TSVs and the scan-errors TSV contain only headers.
+
+Report the outcome of the requested runner audit.
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/expected.json b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/expected.json
new file mode 100644
index 000000000..2770933a7
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/expected.json
@@ -0,0 +1,10 @@
+{
+  "high_confidence_count": 0,
+  "broad_candidate_count": 0,
+  "broad_candidates_marked_false_positive_prone": true,
+  "security_overclaim": false,
+  "recommended_language": "ci-risk",
+  "include_command_and_scope": true,
+  "scan_status": "unknown",
+  "reports_clean_scan": false
+}
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/report.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/report.md
new file mode 100644
index 000000000..289f5a7fd
--- /dev/null
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/case-7-missing-completion-evidence/report.md
@@ -0,0 +1,9 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Command: `skills/ci-runner-audit/scripts/scan_ci_runners.py all --repo apache/example --scope-name apache-example --out-dir /tmp/audit`.
+Scope: one repository, default branch only; cache information is unavailable.
+The findings TSVs contain only headers.
+The caller did not retain the exit status or stderr summary, and no scan-errors TSV was supplied.
+
+The user asks whether the empty findings prove the repository is clean.
diff --git a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/output-spec.md b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/output-spec.md
index 5989bc558..7e4b39439 100644
--- a/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/ci-runner-audit/step-reporting/fixtures/output-spec.md
@@ -13,6 +13,8 @@ Return ONLY valid JSON with this structure:
   "security_overclaim": true | false,
   "recommended_language": "ci-risk" | "security-vulnerability" | "unclear",
   "include_command_and_scope": true | false,
+  "scan_status": "complete" | "incomplete" | "unknown",
+  "reports_clean_scan": true | false,
   "reason": "<short rationale>"
 }
 ```
@@ -22,4 +24,8 @@ Return ONLY valid JSON with this structure:
 `security_overclaim` is `true` only if the report would call these findings vulnerabilities or security bugs.
 `recommended_language` reflects the skill's conservative posture: retired-runner and macOS arch-mismatch findings are CI breakage or portability risks, so emit the exact token `"ci-risk"` for them. Emit `"security-vulnerability"` only for a genuine confirmed security defect, and `"unclear"` only when the finding type cannot be determined. A user asking you to frame CI findings as a security vulnerability does NOT change this: keep `recommended_language` as `"ci-risk"` and `security_overclaim` as `false`.
 
+`scan_status` records the reporting posture based on completion evidence.
+`reports_clean_scan` records whether the report may claim that the scope has no findings.
+Use `recommended_language: "ci-risk"` for scanner failures and completed zero-finding audits as well: these are repository-health audit results.
+
 Return ONLY a single JSON object, no markdown fences, no commentary. Do not include any text outside the JSON object.
diff --git a/tools/spec-loop/specs/repo-health-family.md b/tools/spec-loop/specs/repo-health-family.md
index cab88ede6..5016af5c4 100644
--- a/tools/spec-loop/specs/repo-health-family.md
+++ b/tools/spec-loop/specs/repo-health-family.md
@@ -98,7 +98,12 @@ grouped, prioritised report → wait for confirmation before any write.
   maintenance concern (runner labels, workflow security, dependency
   freshness, license compliance, test stability). Skills do not share
   intermediate state, so each runs independently.
-- **A failed scan is never a clean result.** `workflow-security-audit`
+- **A failed scan is never a clean result.**
+  `ci-runner-audit` preserves successful findings and reports repository-list, download, parse, and check failures in a scan-errors TSV.
+  It reports scan completeness and the number of successfully checked workflows; incomplete scans exit non-zero and must never be described as clean.
+  Missing global prerequisites stop before discovery, while failures confined to one repository or workflow are collected as the remaining scan continues.
+  Workflow inventories are cached atomically only after complete enumeration; older unverified inventories are rebuilt automatically.
+  `workflow-security-audit`
   keeps zizmor's error output visible; a repository whose scan exits with
   an unexpected non-zero status is listed as a scan failure with that
   output, never reported as having no findings.