feat(observability): distinguish a rejected credential from an exhausted quota

[JSONbored]

Summary

A provider 401 and a provider 429 need opposite first responses, but every surface we expose rendered them identically — loopover_ai_provider_failures_total has no reason dimension, and selfhost_ai_provider_failed carries only the raw error string. Decoding which one you're looking at meant grepping logs for a status code.

That cost us real time: a burst of claude_code_error_429 on the hosted ORB box read as a dead token and prompted a credential rotation. The credential was valid the whole time — a 429 is a quota signal, and a new token on the same account behaves identically.

classifyProviderFailure maps a failure to credential_invalid / quota_exhausted / timeout / not_configured / other. It matches on the structured error shapes this module itself throws (claude_code_error_<status>, the auth/timeout sentinels) rather than free-text provider prose, which is not a stable contract. The HTTP providers, which report auth failures as prose rather than an error_NNN code, get a separate narrower arm.

Anything unrecognised stays other — a confidently wrong label sends an operator to the wrong runbook, which is worse than an honest unknown.

Two deliberate choices

A new counter, not a new label. loopover_ai_provider_failures_total is already referenced by shipped alert rules and dashboards. Adding a reason label would silently multiply it into one series per reason and change what those existing queries return. loopover_ai_provider_failure_reason_total{provider,reason} is purely additive and breaks nothing.

Severity encodes the difference. LoopoverAiProviderCredentialInvalid is critical — it does not self-heal, and its runbook points straight at the rotation path from #9543 (noting no restart is needed). LoopoverAiProviderQuotaExhausted is warning with a longer for:, since a short burst against a rate limit is normal and self-correcting, and its runbook opens by saying explicitly not to rotate.

Validation

• npm run typecheck — clean (0 errors)
• npx vitest run test/unit/selfhost-ai.test.ts — 216 tests pass
• npm run selfhost:validate-observability — dashboards and alert rules valid

Tests cover every classification arm and both sides of the branches, including: each status code, the prose-based HTTP-provider arm, non-Error and nullish rejections, and the other fallback. Three pin behaviour that matters specifically:

• Regression: a 429 classifies as quota_exhausted, not as a credential problem — the exact misreading this closes.
• Invariant: an unrecognised failure stays other rather than being mislabelled.
• No false positive: "failed after 401 bytes of output" classifies as other. The status pattern requires the error_ prefix, so a PR number or byte count can never masquerade as an HTTP status.

Plus an assertion that the credential never leaks into the new metric's labels.

Closes #9549. Follow-up to #9543.

[loopover-orb]

Warning

⏸️ LoopOver review result - manual review recommended

_{Review updated: 2026-07-28 08:52:42 UTC}

4 files · 1 AI reviewer · no blockers · CI green · clean

⏸️ Suggested Action - Manual Review

Review summary
Adds classifyProviderFailure() to src/selfhost/ai.ts, mapping a provider failure's structured error string (or, for the two HTTP providers, its prose) into credential_invalid/quota_exhausted/timeout/not_configured/other, wires it into a new additive loopover_ai_provider_failure_reason_total counter plus a `reason` log field, and adds two severity-differentiated Prometheus alerts (critical for credential_invalid, warning for quota_exhausted) with runbooks that explicitly say not to rotate on a 429. I traced the regex logic against every test case (structured error_NNN extraction requires the literal `error_` prefix so a bare '401 bytes' string can't false-positive, the HTTP-prose branch requires both a 401/403 token AND an auth keyword, and the 'other' fallback is genuinely unreachable-safe) and it holds up correctly. The design choice to add a new counter rather than a label on the existing loopover_ai_provider_failures_total is well-reasoned and avoids breaking existing dashboards/alert queries, and `incr` is already imported in this file so the new call site is valid.

Nits — 7 non-blocking

• src/selfhost/ai.ts:1001/1018 — the 401/403/429 status codes are repeated as bare numeric literals across the structured and prose branches; a small `const AUTH_STATUSES = new Set([401, 403])` / `QUOTA_STATUS = 429` would make the two branches obviously in sync.
• test/unit/selfhost-ai.test.ts:2603 — the external secret scan flagged `sk-ant-oat01-should-never-appear` as a potential leaked credential; it's an obviously-fake placeholder token used to assert the reason metric never leaks it, not a real secret — no action needed.
• classifyProviderFailure treats both 401 and 403 as credential_invalid uniformly; a 403 can sometimes mean 'valid credential, insufficient scope' rather than 'rejected credential' — worth a one-line comment on why the PR collapses these (or confirm the CLI/HTTP providers here never emit a scope-403 distinct from an auth-403).
• Extract the repeated 401/403/429 literals in src/selfhost/ai.ts:1001-1023 into named constants for readability.
• Consider a short comment noting why 403 is bucketed with 401 rather than getting its own reason, given 403 can also mean insufficient scope rather than invalid credential.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
• Possible secret-shaped assignment in the diff (generic_secret_assignment) — Verify the value is not a real credential.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#9549
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	❌ 8/20	High review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 14 registered-repo PR(s), 13 merged, 326 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 14 PR(s), 326 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Addressed
The PR adds classifyProviderFailure matching structured error shapes, a new loopover_ai_provider_failure_reason_total counter and reason log field (not a new label on the existing counter), and two alert rules with a critical/warning severity split whose runbooks explicitly state whether rotation helps, plus tests covering the classification and metric/log behavior.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
• Official Gittensor activity: 14 PR(s), 326 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Then work through the remaining 2 steps in the Signals table above.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record

• action: hold · clause: success
• config: 070e846bd47423c42d59a1f2d2aafa335c5b1a1fabd09cc61cf3d5945b35a020 · pack: oss-anti-slop · ci: failed
• record: 546fadb9e585304127de251e7e94cefa73327eaf892a0d1b77111558626cc3f7 (schema v5, head 6e892da)

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[superagent-security]

Superagent did not find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.70%. Comparing base (c2d6102) to head (9244d4d).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9550      +/-   ##
==========================================
- Coverage   89.60%   88.70%   -0.90%     
==========================================
  Files         860      860              
  Lines      110573   110584      +11     
  Branches    26330    26337       +7     
==========================================
- Hits        99078    98098     -980     
- Misses      10231    11513    +1282     
+ Partials     1264      973     -291     

Flag	Coverage Δ
backend	93.68% <100.00%> (-1.63%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/selfhost/ai.ts	98.38% <100.00%> (+0.03%)	⬆️
src/selfhost/metrics.ts	100.00% <ø> (ø)

... and 3 files with indirect coverage changes

[JSONbored]

Conflicts resolved and both CI failures fixed; rebased onto current main.

1. Unregistered metric. loopover_ai_provider_failure_reason_total was emitted and referenced by two alert rules but never added to DEFAULT_METRIC_META, so the metrics-drift guard failed. Registered, with help text stating why it exists as a separate counter rather than a new label on loopover_ai_provider_failures_total (that series is already in shipped alerts and dashboards — adding a label would silently change what those queries return).

2. The runbook pointed at tooling that does not exist. The LoopoverAiProviderCredentialInvalid runbook told an on-call operator to run ./scripts/rotate-secret.sh and mentioned a loopover_admin_rotate_secret MCP tool. I checked: neither exists — not in the repo, not on edge-nl-01, and not in MCP_TOOL_CATEGORIES. The metric-name checker only caught it because the tool name happens to be loopover_-prefixed, but the real defect is worse than a lint failure: a runbook that sends someone to a nonexistent script during a credential outage is worse than no runbook.

A correction I had to make to my own fix. My first rewrite said a docker compose up -d --force-recreate was required, because a bind-mounted secret file rewritten in place leaves the process holding the stale inode. That was true — but #9543 landed on main in the meantime, and resolveClaudeOauthToken now re-reads CLAUDE_CODE_OAUTH_TOKEN_FILE on every AI call, with an empty/unreadable file falling back to the last known good value rather than dropping a working credential mid-rewrite. So the original annotation's "no restart is needed" claim was right, and my correction would have shipped an unnecessary-restart instruction. The runbook now states the real procedure and cites #9543 for why no restart is needed. Caught this because the rebase conflict was in #9543's new tests — the conflict was the signal.

Conflicts. test/unit/selfhost-ai.test.ts — both sides append independent describe blocks; kept both. The conflict region had swallowed the closing braces of #9543's block, so tsc failed until those were restored; all 227 tests in that file now run (previously the file would not parse).

284 tests green across selfhost-ai, alerts-metric-name-references, and selfhost-metrics; typecheck clean.