Skip to content

fix(backlog-stats): stop listing STALE-CANDIDATE as a triage class - #1753

Open
sharpca wants to merge 2 commits into
apache:mainfrom
sharpca:fix/backlog-stats-stale-not-a-class
Open

sharpca wants to merge 2 commits into
apache:mainfrom
sharpca:fix/backlog-stats-stale-not-a-class

Conversation

@sharpca

@sharpca sharpca commented Oct 21, 2026

Copy link
Copy Markdown

Summary

  • Step 2's class table listed STALE-CANDIDATE as one of five triage classes
    while the same row called staleness "orthogonal", so the skill contradicted
    itself. Sonnet followed the table and returned triage_class: STALE-CANDIDATE for the stale fixture; Opus and Fable followed the
    "orthogonal" half and returned UNTRIAGED. The fixture expects UNTRIAGED,
    so Sonnet failed step-2-classify/case-4-stale-candidate on 10 of 10 runs.
  • Staleness is a flag, not a class. The table now has four rows, and a
    paragraph under it states that an issue keeps its real class while
    is_stale_candidate carries staleness. Step 3's stale_candidate bucket
    reads the boolean instead of naming a class.
  • The step-2 eval output spec drops STALE-CANDIDATE from the allowed
    triage_class values and says the rule explicitly, so the spec no longer
    re-introduces the contradiction the skill just dropped.

Type of change

  • Skill change — prose only, no new fixture needed (see the test plan)
  • Documentation (docs/, README.md, CONTRIBUTING.md)

Test plan

  • markdownlint-cli2 on both changed files: 0 issues
  • measured_tokens re-stamped with the repo's own counter
    (4984 → 5055, verified stamp == measured by re-running the tool's
    measure() against the stamped file)
  • surface_hash unchanged, and checked rather than assumed: the
    fingerprint covers requires_config plus step/golden-rule headings, and
    git diff contains no heading line (grep -E '^[+-]#{1,6} ' is empty)
  • No fixture changes, because none are needed:
    case-4-stale-candidate/expected.json already asserts
    UNTRIAGED + is_stale_candidate: true. This PR makes the skill agree
    with the fixture it was already graded against; it does not change what
    the correct answer is.
  • Full eval suite not run locally — the runner needs a model CLI and
    credentials. The reproduction in the issue is the harness for this.

RFC-AI-0004 compliance

  • Vendor neutrality — the new prose says "an untriaged issue last
    updated 216 days ago", taken from the fixture, and names no model as the
    expected behaviour. The issue's model comparison stays in the issue.

Linked issues

Closes #1678

Notes for reviewers

  • The 216-day example in the new paragraph is the fixture's own numbers, so the
    prose and the graded case now describe the same issue.
  • I left Step 3's pressure score (stale-candidate → 2 pts each (regardless of triage status)) untouched: it already reads staleness as orthogonal to the
    class, which is what this PR makes the rest of the skill say.
  • golden-rule-details.md and fetch-queries.md carry no class wording, so the
    contradiction existed in exactly the two places changed here.

Generative AI disclosure

This change was authored with an AI coding agent (WorkBuddy AI, DeepSeek-V4.1-Flash)
under human review. The commit carries the Generated-by: trailer; the
contributor has reviewed the diff and owns the change.

sharpca added 2 commits October 21, 2026 17:57
Why: `fetch_issues.py` ran `gh` once and turned any non-zero exit into a
`SystemExit`, so a single transient `HTTP 502` aborted the whole dashboard
run. Because this fetch is the first of six steps, the operator had to
restart the pipeline by hand. `run_gh` now retries the failures that can
clear on their own (5xx, gateway timeouts, connection resets) with
exponential backoff, and still fails fast on the ones that cannot: an auth
error, a bad repo slug, or an unknown flag never fixes itself.

The default cache also moves off a hardcoded `/tmp`. A sandboxed run is
granted its own scratch directory and cannot write `/tmp` at all, so the
first fetch step failed before it ever reached the API. The default is now
`<tempfile.gettempdir()>/tracker-stats-cache`, which honours `$TMPDIR` on
POSIX and `%TEMP%` on Windows; an explicit `TRACKER_STATS_CACHE` still wins.
The default output path moves with it, for the same reason.

Closes apache#1744.

Generated-by: WorkBuddy AI (DeepSeek-V4.1-Flash)
Why: Step 2 told the model to pick exactly one triage class from a table
that listed `STALE-CANDIDATE` as one of the five rows, while the same row
called staleness "orthogonal". Both instructions cannot be followed, so the
model picked one: Sonnet took the table literally and returned
`triage_class: STALE-CANDIDATE` for the stale fixture, while Opus and Fable
followed the "orthogonal" half and returned `UNTRIAGED`. The fixture expects
`UNTRIAGED`, so Sonnet failed `step-2-classify/case-4-stale-candidate` on 10
of 10 runs.

Staleness is a flag, not a class. The class table now has four rows, and the
paragraph under it says plainly that an issue keeps its real class while
`is_stale_candidate` carries staleness: an issue untriaged for 216 days is
`UNTRIAGED` with `is_stale_candidate: true`, never `STALE-CANDIDATE`. Step 3's
`stale_candidate` bucket and the untriaged funnel already read the boolean, so
only the two places that still named a class had to move. The eval output spec
drops `STALE-CANDIDATE` from the allowed `triage_class` values and states the
rule explicitly.

No fixture changes: `case-4-stale-candidate/expected.json` already expected
`UNTRIAGED` plus `is_stale_candidate: true`. This makes the skill agree with
the fixture it was already being graded against.

Closes apache#1678.

Generated-by: WorkBuddy AI (DeepSeek-V4.1-Flash)

@Kaap10 Kaap10 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling #1678, @sharpca! The prompt fix itself is very well thought out — removing STALE-CANDIDATE from the 5-row mutually exclusive triage table and clarifying that staleness is an orthogonal boolean (is_stale_candidate: true) completely resolves the model divergence between Sonnet and Opus/Fable, while keeping the existing eval fixture case-4-stale-candidate/expected.json intact.

However, there is 1 major git branch issue that needs to be addressed before this PR can be reviewed and merged:


Major Blocker: Stacked PR / Branch Pollution

This PR was accidentally branched off your previous PR branch (#1752) rather than a clean branch off main:

  • Commit 9f1da7d9 ("fix(security-tracker-stats): retry 5xx and default cache to temp root") from PR #1752 is included here.
  • As a result, this PR currently touches 14 files across two unrelated skills/tools (security-tracker-stats-dashboard and issue-backlog-stats).

To fix this:
Please rebase your branch on origin/main to drop the PR #1752 commit:

git fetch origin main
git rebase --onto origin/main 9f1da7d93418ba7f0bc9150c171efad592eb7812 fix/stale-candidate-orthogonal
git push --force-with-lease origin fix/stale-candidate-orthogonal

This will isolate the PR diff to only the 2 files relevant to #1678 (plugins/magpie-issue/skills/backlog-stats/SKILL.md and tools/skill-evals/evals/issue-backlog-stats/step-2-classify/fixtures/output-spec.md).


Assessment of the Issue #1678 Changes (Commit f432930e)

Once the branch is rebased, the actual fix looks solid:

  1. SKILL.md Step 2 & 3: Clear and unambiguous. The four-class table plus the explanatory paragraph under it prevents the LLM from trying to assign STALE-CANDIDATE as a primary class.
  2. output-spec.md: The JSON schema enum and description explicitly drop STALE-CANDIDATE from allowed triage_class values.
  3. Token Count: Restamped correctly (4984 -> 5055) and surface_hash is preserved since headings did not move.


Staleness is **not** a class. It is the separate `is_stale_candidate`
boolean (`days_since_updated >= warn_days`), so an issue keeps its real
class and can be stale on top of it. An untriaged issue last updated 216

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph is the exact missing contract piece. Grounding it with the 216-day / 90 warn_days example (which directly matches case-4-stale-candidate) gives Sonnet unambiguous context to emit triage_class: "UNTRIAGED" and is_stale_candidate: true.

```json
{
"triage_class": "UNTRIAGED | TRIAGED | IN-PROGRESS | STALE-CANDIDATE | SKIP-SECURITY",
"triage_class": "UNTRIAGED | TRIAGED | IN-PROGRESS | SKIP-SECURITY",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping STALE-CANDIDATE from the JSON schema enum ensures the LLM output validator and schema strictly agree with the updated 4-class definition.

@github-actions github-actions Bot added family:security security-* skills family:setup setup-* skills family:tools tools/* family:docs Docs, MISSION.md, READMEs family:issue issue-* skills capability:stats Read-only dashboards, metrics, governance evidence substrate:analytics Tool substrate: read-only metrics / dashboards / renderers labels Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:stats Read-only dashboards, metrics, governance evidence family:docs Docs, MISSION.md, READMEs family:issue issue-* skills family:security security-* skills family:setup setup-* skills family:tools tools/* substrate:analytics Tool substrate: read-only metrics / dashboards / renderers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

issue-backlog-stats: STALE-CANDIDATE is listed as a triage class but defined as orthogonal

2 participants