Repository navigation
Conversation
Why: `fetch_issues.py` ran `gh` once and turned any non-zero exit into a `SystemExit`, so a single transient `HTTP 502` aborted the whole dashboard run. Because this fetch is the first of six steps, the operator had to restart the pipeline by hand. `run_gh` now retries the failures that can clear on their own (5xx, gateway timeouts, connection resets) with exponential backoff, and still fails fast on the ones that cannot: an auth error, a bad repo slug, or an unknown flag never fixes itself. The default cache also moves off a hardcoded `/tmp`. A sandboxed run is granted its own scratch directory and cannot write `/tmp` at all, so the first fetch step failed before it ever reached the API. The default is now `<tempfile.gettempdir()>/tracker-stats-cache`, which honours `$TMPDIR` on POSIX and `%TEMP%` on Windows; an explicit `TRACKER_STATS_CACHE` still wins. The default output path moves with it, for the same reason. Closes apache#1744. Generated-by: WorkBuddy AI (DeepSeek-V4.1-Flash)
Why: Step 2 told the model to pick exactly one triage class from a table that listed `STALE-CANDIDATE` as one of the five rows, while the same row called staleness "orthogonal". Both instructions cannot be followed, so the model picked one: Sonnet took the table literally and returned `triage_class: STALE-CANDIDATE` for the stale fixture, while Opus and Fable followed the "orthogonal" half and returned `UNTRIAGED`. The fixture expects `UNTRIAGED`, so Sonnet failed `step-2-classify/case-4-stale-candidate` on 10 of 10 runs. Staleness is a flag, not a class. The class table now has four rows, and the paragraph under it says plainly that an issue keeps its real class while `is_stale_candidate` carries staleness: an issue untriaged for 216 days is `UNTRIAGED` with `is_stale_candidate: true`, never `STALE-CANDIDATE`. Step 3's `stale_candidate` bucket and the untriaged funnel already read the boolean, so only the two places that still named a class had to move. The eval output spec drops `STALE-CANDIDATE` from the allowed `triage_class` values and states the rule explicitly. No fixture changes: `case-4-stale-candidate/expected.json` already expected `UNTRIAGED` plus `is_stale_candidate: true`. This makes the skill agree with the fixture it was already being graded against. Closes apache#1678. Generated-by: WorkBuddy AI (DeepSeek-V4.1-Flash)
There was a problem hiding this comment.
Thanks for tackling #1678, @sharpca! The prompt fix itself is very well thought out — removing STALE-CANDIDATE from the 5-row mutually exclusive triage table and clarifying that staleness is an orthogonal boolean (is_stale_candidate: true) completely resolves the model divergence between Sonnet and Opus/Fable, while keeping the existing eval fixture case-4-stale-candidate/expected.json intact.
However, there is 1 major git branch issue that needs to be addressed before this PR can be reviewed and merged:
Major Blocker: Stacked PR / Branch Pollution
This PR was accidentally branched off your previous PR branch (#1752) rather than a clean branch off main:
- Commit
9f1da7d9("fix(security-tracker-stats): retry 5xx and default cache to temp root") from PR #1752 is included here. - As a result, this PR currently touches 14 files across two unrelated skills/tools (
security-tracker-stats-dashboardandissue-backlog-stats).
To fix this:
Please rebase your branch on origin/main to drop the PR #1752 commit:
git fetch origin main
git rebase --onto origin/main 9f1da7d93418ba7f0bc9150c171efad592eb7812 fix/stale-candidate-orthogonal
git push --force-with-lease origin fix/stale-candidate-orthogonalThis will isolate the PR diff to only the 2 files relevant to #1678 (plugins/magpie-issue/skills/backlog-stats/SKILL.md and tools/skill-evals/evals/issue-backlog-stats/step-2-classify/fixtures/output-spec.md).
Assessment of the Issue #1678 Changes (Commit f432930e)
Once the branch is rebased, the actual fix looks solid:
SKILL.mdStep 2 & 3: Clear and unambiguous. The four-class table plus the explanatory paragraph under it prevents the LLM from trying to assignSTALE-CANDIDATEas a primary class.output-spec.md: The JSON schema enum and description explicitly dropSTALE-CANDIDATEfrom allowedtriage_classvalues.- Token Count: Restamped correctly (
4984 -> 5055) andsurface_hashis preserved since headings did not move.
|
|
||
| Staleness is **not** a class. It is the separate `is_stale_candidate` | ||
| boolean (`days_since_updated >= warn_days`), so an issue keeps its real | ||
| class and can be stale on top of it. An untriaged issue last updated 216 |
There was a problem hiding this comment.
This paragraph is the exact missing contract piece. Grounding it with the 216-day / 90 warn_days example (which directly matches case-4-stale-candidate) gives Sonnet unambiguous context to emit triage_class: "UNTRIAGED" and is_stale_candidate: true.
| ```json | ||
| { | ||
| "triage_class": "UNTRIAGED | TRIAGED | IN-PROGRESS | STALE-CANDIDATE | SKIP-SECURITY", | ||
| "triage_class": "UNTRIAGED | TRIAGED | IN-PROGRESS | SKIP-SECURITY", |
There was a problem hiding this comment.
Dropping STALE-CANDIDATE from the JSON schema enum ensures the LLM output validator and schema strictly agree with the updated 4-class definition.
Summary
STALE-CANDIDATEas one of five triage classeswhile the same row called staleness "orthogonal", so the skill contradicted
itself. Sonnet followed the table and returned
triage_class: STALE-CANDIDATEfor the stale fixture; Opus and Fable followed the"orthogonal" half and returned
UNTRIAGED. The fixture expectsUNTRIAGED,so Sonnet failed
step-2-classify/case-4-stale-candidateon 10 of 10 runs.paragraph under it states that an issue keeps its real class while
is_stale_candidatecarries staleness. Step 3'sstale_candidatebucketreads the boolean instead of naming a class.
STALE-CANDIDATEfrom the allowedtriage_classvalues and says the rule explicitly, so the spec no longerre-introduces the contradiction the skill just dropped.
Type of change
docs/,README.md,CONTRIBUTING.md)Test plan
markdownlint-cli2on both changed files: 0 issuesmeasured_tokensre-stamped with the repo's own counter(
4984→5055, verifiedstamp == measuredby re-running the tool'smeasure()against the stamped file)surface_hashunchanged, and checked rather than assumed: thefingerprint covers
requires_configplus step/golden-rule headings, andgit diffcontains no heading line (grep -E '^[+-]#{1,6} 'is empty)case-4-stale-candidate/expected.jsonalready assertsUNTRIAGED+is_stale_candidate: true. This PR makes the skill agreewith the fixture it was already graded against; it does not change what
the correct answer is.
credentials. The reproduction in the issue is the harness for this.
RFC-AI-0004 compliance
updated 216 days ago", taken from the fixture, and names no model as the
expected behaviour. The issue's model comparison stays in the issue.
Linked issues
Closes #1678
Notes for reviewers
prose and the graded case now describe the same issue.
stale-candidate → 2 pts each (regardless of triage status)) untouched: it already reads staleness as orthogonal to theclass, which is what this PR makes the rest of the skill say.
golden-rule-details.mdandfetch-queries.mdcarry no class wording, so thecontradiction existed in exactly the two places changed here.
Generative AI disclosure
This change was authored with an AI coding agent (WorkBuddy AI, DeepSeek-V4.1-Flash)
under human review. The commit carries the
Generated-by:trailer; thecontributor has reviewed the diff and owns the change.