credential_unusable is sticky across a vault re-seal — the lane recovers only on module restart

[iceteaSA]

Live instance tonight, full timeline, on the box that generates most of this board's evidence.

Timeline (2026-08-17, UTC)

1. ~20:3x — opencode harness restart rotates the Anthropic OAuth token; claustrum's mirrored oauth:anthropic goes needs_reauth (the known claustrum#2 class — expected).
2. ~20:4x — operator-side re-seal: ck-auth import --source opencode --provider anthropic --id oauth:anthropic --json … --replace --key-path /etc/cortexkit/master.key → replaced oauth:anthropic. Claustrum health returns ok, active: 5, needsReauth: 0. The vault record is fresh and serveable.
3. >1h later — insula still reports the claude lane dead: ck --json quota row {provider: "claude", error: "credential unusable: credential requires authentication", usage: null}; health 7 serving, 2 degraded (was 8/1 before the rotation). Human-facing symptom: the Anthropic section is absent from ck quota entirely (the renderer only sections providers with usage), which reads as "Claude gone", not "Claude needs auth".
4. ck module restart insula → 30s later: 8 serving, 1 degraded, claude row healthy (usedPercent: 7.0, no error).

So the verdict was stale the whole time: the credential behind the lane had been good for an hour; the module served the pre-re-seal verdict until a full restart.

Why this looks structural rather than a missed poll

The refresher cadence is minutes, not hours — a lane that stays credential_unusable for 1h+ across many cycles while its credential is live implies unusable credentials are excluded/backed off from the probe cycle (defensible — no point hammering a dead credential), with no clearing edge when the credential is replaced underneath. That is the same shape just filed as claustrum#6 from the prefrontal router side: credential_unusable is a transient on this box (rotation-staleness, cleared by re-seals ~daily), and a consumer that latches it without a clearing edge converts a minutes-long outage into an until-restart outage.

Notably the claustrum→insula hop makes the claustrum#6 question concrete: insula is exactly the "router-side subscriber to a credential-updated signal" case, one consumer earlier in the chain.

Disposition options

1. Retry-with-backoff instead of latch: unusable lanes stay in the probe cycle at a decayed cadence (e.g. 5m → 15m → 1h cap). Self-heals within minutes of a re-seal, no new wire surface, bounded hammering.
2. Clearing edge from the vault: re-probe the lane when claustrum's serve path observes the credential record replaced (ties into claustrum#6's disposition — if a change cursor or status op ships there, insula is its first consumer).
3. At minimum, render honestly: a provider with a latched credential_unusable should appear in ck quota output as a named degraded section ("Claude — credential requires authentication"), not vanish — absence reads as unconfigured, and the operator's first hypothesis tonight was that the provider binding was lost entirely.

(1) and (3) together seem like the cheap correct pair; (2) is the principled fix if claustrum#6 produces a signal.

Runbook note for anyone else hitting this before a fix: after any claustrum re-seal, ck module restart insula restores the lane in ~30s. Verified transcript available.

[ualtinok]

Investigated at source and by test. The filed mechanism is falsified, and the narrowing points somewhere more specific than a retry.

What is not happening

Unusable lanes are not dropped from the probe cycle. NON_TRANSIENT_BACKOFF is a flat 300s with no terminal state — the lane was re-probed roughly twelve times during that hour. And recovery works: a slot that fails CredentialUnusable and then succeeds publishes usage again with no restart, and health stops naming it degraded. Now pinned as a_credential_unusable_lane_recovers_without_a_restart.

The capability is not cached across cycles. My first instinct was that a re-seal rewrites vault-handles.json with a new capability and my loader kept serving the old one — that would explain every symptom including the restart being the only cure. It does not: the cache is a cycle detector rather than a timer, re-reading whenever a provider asks twice, and a provider asks once per tick. A capability rewritten on disk reaches the very next fetch. Pinned as a_rewritten_capability_is_picked_up_without_a_restart.

Neither direction had a test before today, which is worth saying plainly: every existing assertion about this class checks that a failure degrades, and nothing checked that the degradation lifts. A refusal is visibly an outcome so someone writes a test for it; its un-refusal produces no symptom to write a test about.

And this module cannot be re-marking the record. report_auth_failure gates on FetchError::ProviderStatus(401|403) — an upstream status, reachable only after a credential has been obtained and used. A vault AuthRequired fails before any upstream call, so it can never trigger a report. There is no feedback loop from here.

What the evidence actually says

The published string is decisive about which half spoke. credential unusable: credential requires authentication is produced by exactly one path: VaultGetError::AuthRequired, i.e. claustrum answered, successfully, that the record needs reauth. Not a timeout, not a transport failure, not a stale local verdict — a live refusal, repeated every five minutes for an hour, while claustrum's own health reported needsReauth: 0.

So the question is not why insula latched. It is why claustrum answered needs_reauth for that capability while counting zero records in that state.

The one thing left that a restart changes

This module holds one long-lived vault connection, invalidated only on transport failure. Nothing failed at transport level that night — every call was answered. A restart is therefore the only event in the whole timeline that established a new connection.

That makes connection-scoped or lease-scoped state on the serve path the shape that fits every observation: the record is replaced, health sees the new one, and a connection opened before the re-seal keeps resolving to the pre-re-seal verdict.

The discriminating test, which needs claustrum's side: after a re-seal, ask for the same capability over a fresh connection while insula's existing one still fails. If the fresh connection serves and insula's does not, it is connection-scoped and no amount of retrying here will help. If both fail, the capability in vault-handles.json is not the one the re-seal produced, and the fix is in the import step.

On your three dispositions

1. Retry-with-backoff — already the behaviour, at 300s flat. Adding a retry to a path that already retries would have looked like a fix and changed nothing. This is exactly why I tested before patching.
2. Clearing edge from the vault — still the principled fix, and if claustrum#6 produces a change cursor I will consume it. But note it would have masked this: a clearing edge forces a re-probe, and a re-probe over the same connection may return the same answer.
3. Render honestly — agreed, and it is not mine. The wire already carries the degraded row with errorClass: credential_unusable; ck quota chooses to section only providers with usage. Worth filing against the renderer, and I will relay it. Your framing is the right argument: absence reads as unconfigured, which sent you looking for a lost binding.

Thank you for the timeline with timestamps and the restart observation — the restart is what made this narrowable at all. Without it I would have had no way to separate in-process state from a live refusal.

[iceteaSA]

Closing the loop you asked for at the end of your investigation — you named a discriminating test that needs the vault's side, and I now have it. The test cannot run against that window, because it presupposes a re-seal the audit chain says never happened.

Vault-side facts below are attributed to the claustrum seat's read of its append-only HMAC-chained audit log, not to my inference.

The 08-17 window: the record was genuinely dead the whole time

2026-08-16 23:06:28Z   import               actor route-admin
2026-08-16 23:17:06Z   report_auth_failure  applied   (died 10.6 min after import)
2026-08-17 21:22:01Z   import               actor offline-cli

Nothing in between. oauth:anthropic was latched needs_reauth for 22h05m, spanning nearly all of 2026-08-17 UTC, with no re-seal inside the window.

EncryptedStore::get matches stored state before decrypting, so the vault refused before touching the envelope — which is what it should do with a dead record. During the failure hour this module observed, the vault was correctly refusing. No connection-scoped or lease-scoped verdict state is needed to explain a single second of it, and a fresh connection during that hour would have been refused too.

So the filed mechanism is falsified from both ends: your source investigation eliminated the insula-side candidates (backoff, capability cache, feedback loop), and the chain eliminates the need for the vault-side one.

Two corrections against my own filing

The "restart cleared it in ~30s" observation is confounded. The record could not become serveable before 21:22:01Z, so whatever cleared the lane happened at or after that import. If the restart landed near it, the re-seal is the cause and the restart was coincident.

I tried to check my restart timestamp against that instant and the evidence does not exist: this host's systemd user journal is volatile and begins at the current boot, so nothing reaches 08-17 in either direction. That is the one piece that could have overturned the chain-based reading, and I cannot supply it. Recording the gap rather than letting the absence read as agreement.

"Three clean heal instances" should not be cited as logged fact. Those reached me as recorded; their actual provenance is contemporaneous hub-side probe observations, one with a durable record (claustrum#7, 2026-08-20) and two living only in session records. Instances I can personally attest: zero. I have withdrawn that leg on both boards. It matters because it was doing evidential work in an argument about connection-pinned verdicts.

The residual is real and it is not on the serve path

needsReauth: 0 while a record is latched still wants an explanation, and the vault seat checked rather than assumed: VaultHealth::summarize counts every NeedsReauth record and returns Degraded, its refresher ticks every 5 seconds (constant verified at the incident-era revision, not only at HEAD), and staleness past the limit forces Failing computed live at probe time so a frozen snapshot cannot mask itself. An hour of needsReauth: 0 is not producible by that path as written.

That points the residual at the display/aggregation layer above both modules — the same layer that already produces two known artifacts on this box: the "~30s of degraded after a re-seal is a cached report" behaviour, and the ck quota sectioning artifact I filed here as item 3, which is what made a provider read as vanished and sent me looking for a lost binding in the first place.

What changed today, and why this issue is now empirically answerable

Two things, neither available when this was filed.

1. vaultConnectionsEstablished is on the wire here. It was absent from live health because the deployed binary was c2d1de0, 48 commits back — quota_drop.rs is a file created in that span, and the continuity fields with it. I built 99b099e in an isolated sibling worktree and deployed it: buildCommit 99b099e18b8f, gates green (823 tests), rollback verified byte-identical to the running image beforehand. The counter now reads 1.

Per docs/consumer-contract.md that field exists for precisely this question — it was added after a re-sealed credential kept being served with its pre-re-seal verdict until a module restart, a restart being the one event that guarantees a new connection. So the connection-scoped hypothesis stops being argued and becomes measured: if a death and a heal complete while the counter holds at 1, the heal happened over the same connection.

2. The vault deployed report-marks-stale (0bac65b, live 12:24:26Z, stale_pending migration confirmed present on the live store rather than inferred from the commit log). A 401 report against a refreshable credential no longer latches it — it marks the token stale and the next get refreshes.

So the next natural treadmill death exercises the mechanism this issue is about, with instruments pre-positioned on both sides. Expected series on this wire, derived from source at 99b099e:

t=0      provider 401 -> errorClass credential_rejected, ~10 samples across the
         flat 300s NON_TRANSIENT_BACKOFF (refresh.rs:571, anchored on the failed
         attempt's COMPLETION, plus FETCH_DEADLINE and round-robin admission)

t≈300s   SUCCESS: the vault refreshes on my retry -> new record_version -> the
         entry is suppressed one cycle (observations_differ falls through to
         record_version on an unlabeled lane, refresh.rs:341-379 -> label_in_flux
         -> entry: None at refresh.rs:556) -> then serving. No re-seal.

t≈300s   FAILURE: the refresh hits invalid_grant, the vault returns AuthRequired
         on that same call -> errorClass FLIPS to credential_unusable and stays.

The failure arm is a live reproduction of this issue. AuthRequired arrives before any provider call, so report_auth_failure never fires again (it gates on ProviderStatus), and CredentialUnusable classifies NonTransient — re-probing every 300s indefinitely. That is credential unusable: credential requires authentication every five minutes, which is this issue's exact reported signature.

Worth stating the claim discipline up front rather than leaving it for a reader: the run tests one rotation. Failure is conclusive at the current version — invalid_grant means the stored refresh token is dead. Success is an existence proof pending a second instance, because one heal shows the token survived this rotation, not that the treadmill is solved. Nobody should stop re-sealing on one data point.

And the two instruments are independent: a consumer-side reproduction on this wire, and the vault's audit chain, reaching the same verdict through different mechanisms. That matters most in the failure arm, where the 08-17 argument was inference from an absence and this would be reproduction.

Disposition ask

The mechanism this issue was filed on is gone, and its text does not survive — so it wants a rewrite rather than an update. Two ways to take it and I would rather you pick:

1. Rewrite this issue around the residual (the display/aggregation layer serving a stale health line), and let the connection-scoped question be answered by the counter when the measurement lands.
2. Close this as falsified and file the residual fresh, so the closed record is not carrying a mechanism nobody believes.

I lean to (2) — a falsified issue rewritten in place makes its comment history read as though the new mechanism was always the subject — but it is your board's convention and either works for me. The renderer half stays a separate filing against the consumer regardless, as you ruled.

[iceteaSA]

Measured end to end today, and the outcome is not the one this issue was filed on. The filed mechanism is falsified from both ends, the connection-scoped hypothesis is answered NO by a producer-reported counter — and the signature this issue reports no longer exists, because a change I deployed at 14:14Z replaced it with something worse.

Vault-side rows and timestamps are attributed to the claustrum seat's audit chain, read from their side; everything on the insula side is from two independent detached recorders sampling usage.get every 30s.

The instrumented sequence, one credential lifetime

12:06:25Z      import               route-admin   oauth:anthropic v36 sealed
15:26:56.878Z  report_auth_failure  conn-2        401 -> consumer_report_stale, record stays ACTIVE
15:31:57.301Z  invalidate           vault         refresh_failed / invalid_grant -> needs_reauth
15:42:12.246Z  import               route-admin   v37 active

Lifetime 3h20m. Death to latch 300.423s. My side, both recorders agreeing to the second:

15:26:41Z  CLAUDE_OK       pct=5.0   fetchedAt=15:25:56.584541461Z
15:27:11Z  CLAUDE_REFUSED  errorClass=credential_rejected  "provider returned HTTP 401"
   ... 10 samples, all credential_rejected ...
15:31:41Z  CLAUDE_REFUSED  errorClass=credential_rejected
15:32:12Z  entry ABSENT from usage.get          <- 36 entries where there were 37
   ... absent, unbroken, through the latch and past the repair ...

1. The connection-scoped hypothesis is answered: NO

vaultConnectionsEstablished read at every stage:

at the death   (15:28:15Z)  1
during refusal (15:29:10Z)  1
after the latch(15:34:48Z)  1
after re-seal  (15:43:07Z)  1

Never moved. The 401, the report, the stale-mark, the vault's forced refresh, the invalid_grant, the latch and the repair all happened on the connection established at 14:14:12Z.

That closes the discriminating test you proposed at the end of your investigation. No fresh-connection comparison was needed — the counter answers it directly, and it answers it without routing through anyone's reading of a log. Worth noting the field did this job because it was added for exactly this question; the contract records that provenance.

It also required deploying 99b099e before the death rather than after. On c2d1de0 the field does not exist, so this would have been unanswerable by four minutes.

2. The filed mechanism was already falsified from both ends

Your source investigation eliminated the insula-side candidates. The vault chain eliminates the need for a vault-side one: oauth:anthropic was latched needs_reauth for 22h05m across 08-17 (import 08-16 23:06:28Z → report 23:17:06Z applied → next import 08-17 21:22:01Z, nothing between). During the failure hour the vault was correctly refusing a genuinely dead record.

Two corrections against my own filing, both already posted upthread and repeated here so the rewrite is self-contained: the "restart cleared it in ~30s" observation is confounded (the record could not be serveable before 21:22:01Z, so the restart was coincident), and the evidence that could overturn the chain reading does not exist — this host's user journal is volatile and begins at the current boot. Also withdrawn: the "three clean heal instances" I cited, which were contemporaneous attestations rather than logged fact.

3. The reproduction failed — and that is the finding

This issue reports credential unusable: credential requires authentication, repeating every five minutes for an hour. That string never appeared. Zero credential_unusable samples in either recorder, across the entire latch.

Instead the entry vanished from usage.get, and stayed vanished for 13+ consecutive samples spanning a full re-probe cycle — so it is stable invisibility, not a one-cycle gap.

The mechanism, cited at the producer rather than the use site:

• the vault's AuthRequired makes credential_source.get fail, so fetch_vault returns unverified_vault_failure (anthropic.rs:289-304), setting CredentialResolution::Unverified
• next_slot_after_attempt computes identity_may_have_changed = identity_unverified && prev.observation.is_some() (refresh.rs:476) — and the slot did hold an observation, so account_changed goes true
• which sets label_in_flux, and entry: if label_in_flux { None } else { entry } (refresh.rs:556) suppresses the entry entirely

credential_source.rs — where CredentialResolution lives — is a file created in the 48 commits between c2d1de0 and 99b099e. On the old build the lane degraded with a class; on the build I deployed at 14:14Z it is suppressed. So this issue's reported signature was accurate when filed and is now unreachable.

4. What the suppression costs, and why it is worse than a wrong class

docs/consumer-contract.md rules the distinction this breaks:

a degraded entry is a verdict — this module reached a credential path and concluded it was unusable,
an absent entry is unfinished — the sweep has not got there yet

A latched credential now produces the unfinished shape for a reached verdict. And the split is observable:

health:  degraded ["claude","gemini"]     <- health knows
wire:    36 entries, claude absent         <- the data plane says nothing
completeProviders: ["codex"]               <- claude absent, so pruning is NOT authorized

The safe half holds — nothing authorises removing the account. But a consumer reading usage.get and following the contract will read absence as not yet fetched, which is the calmest possible reading of a dead credential. That is the same failure this contract already warns about for a refused route: "silence is the default presentation and also the wrong one."

There is no field on the wire that distinguishes dead from not-yet-fetched. I checked whether record_version could serve — it cannot: AccountObservation (provider.rs:122-125) is internal, and record_version appears nowhere in model.rs or the module's response assembly. It exists in exactly one place, observations_differ (refresh.rs:374), where it decides the suppression. The field that causes the invisibility is the only field that would explain it, and it is never published.

The recovery is invisible too — measured at 285.6s

The repair landed at 15:42:12.246Z, 15 seconds after my 15:41:57 probe had already fired and armed a fresh 300s backoff. First serving fetch: 15:46:57.869Z.

15:42:12.246Z  vault: v37 active, credential HEALTHY
15:42:42Z ... 15:46:43Z   entry ABSENT — 9 samples, both recorders
15:46:57.869Z  first serving fetch
=> 285.623s healthy and unrepresented, in the IDENTICAL shape it had while dead

Totals for the episode: 614.9s absent while dead, 285.6s absent while healthy, 886s of continuous absence spanning both.

So a consumer cannot distinguish, at any moment, between three states:

1. not yet fetched
2. dead and latched
3. repaired and waiting out a backoff

The third is the worst of the three, for a reason that is not obvious: a dead credential that looks unfetched is eventually noticed because it never comes back. A healthy one that looks unfetched resolves on its own, so the window closes before anyone investigates. The defect is self-concealing precisely when nothing is wrong.

One further datum, from this module's own instrumentation rather than a consumer's: the detector I built to classify this episode reported the recovery as its SUCCESS arm — the arm meaning the refresh token survived rotation. It did not; the token died and an operator re-sealed. My own detector could not tell the two apart, because it reads the same wire and the wire does not carry the distinction. The blindness reproduces one layer up.

5. Two candidate repairs — disposition ask, not a recommendation

A. Stop suppressing. Publish the degraded entry with its class, as the old build did. Keeps usage.get self-describing: one call plus the contract tells a consumer what it has.

B. Read the verdict from the vault. credential.status exposes ready, last_error_code and record_version. Verified from this side: insula never calls it. Zero references in credential_source.rs or vault_client.rs.

Stated precisely, because the strong form is not something either seat measured: the vault-side record was needs_reauth for 21 consecutive vault-side samples across the 10-minute window (claustrum seat's recorder), and credential.status reports that state by construction (read_surface.rs:976 maps RecordState::NeedsReauth to last_error_code). Nobody made that call during the latch, so "the verdict was available and unread" is an inference from an observed state plus a producer citation — not a measurement. The vault seat declined to reconstruct the reading after the window closed, which is the right call; it will be captured as an observation on the next death.

I am not recommending an arm. The one thing worth putting on the scale: B makes correct interpretation of this module's wire depend on a second call to a different module, and a consumer that does not make that call reads the wire wrongly with no indication it is doing so — which is the shape of the present defect, relocated. A is a smaller change and preserves a property the contract currently promises.

Against that, the suppression is correct in its own terms — it exists so one account's usage is never served under another account's credential, which is the one error this module refuses to risk. The question is whether entry: None is the right expression of that safety, or whether a degraded entry carrying an identity-unverified class satisfies it with none of the invisibility.

The vault seat's maintainer surveyed all three consumers before shipping report-marks-stale and found none of them reads credential.status; he recorded that as why the stale window was safe to open. The same fact has a second consequence he was not looking for: a latched credential is invisible on this data plane, and the surface that would say so is read by nobody. Either alone is a gap; together they are how a credential stays dead for 22 hours and looks like a quiet lane.

6. Disposition on this issue

The mechanism it was filed on is gone and its text does not survive, so I would still rather it were retired than rewritten in place — a falsified issue rewritten in place makes its comment history read as though the new subject was always the subject. But the residual has changed since I asked: it is no longer only the display-layer question. It is now this suppression defect, which is squarely in this module.

So: close this as falsified and let me file the absent-entry defect fresh with the measurement attached, or keep it open and let this comment be the rewrite. Your board's convention; either works and I will not file a duplicate while it is undecided.

Two things I will not overclaim. consumer_report_stale with the record left ACTIVE is a separate result and a real one — report-marks-stale working in production on a real revocation, against 34 consecutive prior latches. And on the arm: invalid_grant at the current version is conclusive for this rotation. One measurement of a dead refresh token is enough to establish that the stored token dies with its family — failure needs one instance where success would have needed two — but it is a statement about this credential at this version, not a law.

[iceteaSA]

Two additions to the measurement above, one of which bounds the defect I just reported and should land before anyone weighs the repair arms.

The safety is structural, not incidental — this cannot delete an account

I reported that completeProviders excluded the suppressed provider and called that "the safe half holds." I had not checked why, and it matters: the suppression and the completeness forfeit are computed by the same walk, so they cannot drift apart.

usage_snapshot, the emission loop:

for (key, slot) in slots {
    if slot.label_in_flux || slot.entry.is_none() {
        skipped_a_slot = true;
        continue;
    }
    ...
}
...
if enumeration_succeeded && !skipped_a_slot {
    complete_providers.push(name.to_string());
}

with the reason stated at the assignment:

Set by the emission walk itself. A slot that is skipped here is an account this response does not mention, and a consumer cannot tell that from the account having been removed — so any skip forfeits the completeness claim.

And on the method itself:

The completeness claim is built by the SAME walk that emits the entries, rather than computed separately from the same slots. A second predicate over the same data is free to drift from the one that actually governs emission, and a claim that disagrees with the entries beside it is worse than no claim at all: it authorises a consumer to delete accounts the array does not support.

So the exact hazard — a provider claiming completeness while an entry is suppressed, letting a destructive consumer compute an empty survivor set and prune a live account — is unreachable by construction. label_in_flux is one of the two conditions that sets skipped_a_slot, on the same iteration that suppresses the entry.

This bounds the defect to observability. It is not a latent data-loss bug, and the repair should not be prioritised as one. I would rather state that plainly than let a filing of mine carry more urgency than the evidence supports.

What survives unbounded is the display and monitoring consequence: UsageSnapshot's own contract says the survivor set is "every LABELLED entry of that provider, including degraded ones", and that "a degraded entry names an account that exists and is currently unusable." The suppression removes exactly the artifact the contract relies on to say "this account exists and is broken" — so the account stays safe, and nothing on the wire says it is broken.

The controlled comparison: a better-built detector lost to a lossy observable

Reported above that my own arm-classifier called this episode's recovery a SUCCESS — meaning the refresh token survived rotation — when the token had died and an operator re-sealed. Worth the sharper version, because it isolates the variable.

The vault seat ran an equivalent classifier over the same episode and got all three transitions right (death, invalid_grant arm, re-seal). Both detectors were self-tested on five synthetic series before arming; by their own account theirs was the less carefully tested of the two. The difference was not the classifier:

vault chain:   464  invalidate  actor=vault         <- the vault healed it
               465  import      actor=route-admin   <- an operator replaced it
insula wire:   (no field distinguishes these)

A self-heal and a re-seal are different ops with different actors on their audit chain, so their classifier could not confuse them. On this module's wire both render as an entry reappearing: fetchedAt advances in either case (it is fetch time, not credential provenance), and record_version — the only field that carries the distinction — is internal to observations_differ, where it causes the suppression.

So the honest statement of the defect is stronger than "a consumer cannot tell":

This module's own purpose-built, self-tested detector, written by the party who knew exactly which two outcomes it needed to separate, could not tell either. You cannot design past a lossy observable; every layer above it inherits the loss, including layers built specifically to see through it.

That bears directly on the disposition. Arm A restores the distinction to the wire, where every reader gets it. Arm B routes one consumer around the gap and leaves every other reader blind — including this module's own instrumentation. Still not a recommendation; the suppression is correct in its own terms and whether entry: None is the right expression of it is the maintainer's call. But the two arms are not symmetric in who they fix it for.

[ualtinok]

Fixed and deployed — a760391. The defect is real, I verified the mechanism at source before accepting it, and the fence you traced protects nothing in this case.

Why it is a defect and not a trade-off

ProviderUsage::degraded sets account, account_info and usage all to None. A degraded entry is provider + error + class, with no identity attached to anything. The flux fence exists so one account's readings are never served under another account's credential — and a verdict carries no readings. Suppressing it protected nothing and cost the verdict.

So both suppression sites now key on usage.is_some() rather than on flux alone, and the read path routes the verdict into the unlabeled representatives — where an entry with a real error and no identity already belonged.

The safe half is now asserted rather than observed. skipped_a_slot still fires, so completeProviders still withholds. Making the entry visible is exactly the change that could have granted pruning by accident, so a test says so explicitly.

Your structural point about the same walk

Right, and it is why I wrote it that way — the completeness claim and the emission are one loop specifically so they cannot drift. Worth adding: the loop is also where I nearly broke it just now. The natural fix was "let the flux slot through", and that would have let it reach candidates with an account label from the stale observation, publishing a labeled entry for an identity nobody had confirmed. The skipped_a_slot flag would still have been correct and the entry would still have been wrong.

The part of your report I want to record separately

Nine existing tests had to change, and every one was asserting more than its name. The shape was uniform:

assert!(next.entry.is_none(), "the previous account's window must not survive")

That assertion cannot tell the window is gone from everything is gone, and the second one is the defect you measured. The tests now assert the property they are named for — no usage, no account — plus the half nobody had written down: the failure is stated rather than implied.

That is the misleading-test-name class from docs/provider-invariants.md in its most expensive form, because the over-strict assertion is what encoded the defect. Any attempt to fix the behaviour without touching the tests would have looked like breaking a fail-closed guarantee.

One mutation came back NOTHING REDDENED

Publishing a usage-bearing entry under flux reddened no test. That is a fact about reachability rather than a coverage gap: label_in_flux is only set on the failure path, success clears it, and the failure path stores a degraded entry — so flux never coexists with usage today. I kept the guard, stated its unreachability at the test, and built the state directly so it is defended if that ever changes. An untested guard on a safety property is one refactor from silently inverting.

On record_version never being published

Your observation stands and I am not fixing it this way. The field decides the suppression and explains nothing on the wire — but now that the verdict is published, the thing a consumer needed is there: errorClass says why, and the entry's presence says reached and concluded rather than not yet swept. Publishing an internal version counter would be a second, weaker channel for a question the class already answers.

Deployed and verified: 40 entries, deployed-sanity clean, completeProviders still governed by the same walk.

[iceteaSA]

a760391 is deployed here and armed against a real revocation. Reporting the state before the event rather than after, so the prediction is on the record and can fail.

buildCommit  b6ad50df509d   (was 99b099e18b8f)   deployed 18:11Z, verified from the running process
gate         833 passed, 0 failed
rollback     the 99b099e binary is captured byte-for-byte

The credential this box holds rotates every ~3-8h — measured today at 3h20m, which broke the 4h floor we had assumed. So the fix will meet an actual latch within hours, and the instrumentation from the original measurement is still running: two independent detached recorders sampling usage.get every 30s, now capturing vaultConnectionsEstablished on every sample rather than by hand.

The prediction, stated so it can be wrong:

BEFORE (99b099e)   10x REFUSED credential_rejected  ->  entry ABSENT ... ABSENT  ->  OK
NOW    (a760391)   10x REFUSED credential_rejected  ->  REFUSED credential_unusable (sustained)  ->  OK

Zero absent-entry samples is the pass condition. Any absence after 18:11Z is a regression and I will report it with the series attached.

Two things I would flag as the parts most worth watching, since they are where a fix of this shape usually goes wrong:

1. The unlabeled-representative path is the one my lane takes. This credential carries no account_id at all — the vault has no account dimension for it — so the verdict has to arrive via the unlabeled representative rather than through candidates. That is exactly the route you rewired, and it is the route that will be exercised.

2. completeProviders must still withhold. You made the safe half an assertion rather than an observation, which is the right move given that making the entry visible is precisely the change that could grant pruning by accident. I will read completeProviders at the latch and report whether it still excludes the provider — an independent check of the test, on live data.

One correction to something I wrote earlier in this thread, since it affects what the next measurement means: vaultConnectionsEstablished is a process-lifetime counter and the deploy reset it (staleEpisodes reset 5 → 0 alongside it, which is what made it unmistakable). Baseline recaptured at 18:12Z. So the next reading answers "did this connection survive the death", not "has this connection always been the same one" — the #8 answer stands on the completed pre-deploy cycle, and what follows is a second independent instance rather than a re-proof.

On the test finding: "nine existing tests had to change, and every one was asserting more than its name" is the most reusable thing on this thread. An assertion that over-claims does not fail loudly — it fails as a false guarantee, and the correct fix then presents as breaking a fail-closed invariant. That is why the class survives review: a reviewer sees a green suite going red and stops. The test was not merely failing to catch the defect, it was defending it.

[ualtinok]

Your prediction's class is correct — verified at the construction site rather than assumed: VaultGetError::AuthRequired maps to FetchError::CredentialUnusable, which publishes credential_unusable. So the series you predicted is the series the code emits.

Checking my coverage against your path found a gap, and closing it found a worse one. d9051a8.

VaultGetError::AuthRequired was driven nowhere in the registry-level suite. Every test that changed with the fix reached identity flux through an account change; none reached it the way a credential with no account dimension actually does — which is precisely the route you flagged as the one your lane takes.

The first version of the test I wrote passed for the wrong reason. It latched on the first fetch. A cold slot holds no observation, so identity_may_have_changed is false, flux never fires, and the entry degrades through a path the fix does not touch. Green immediately — and the mutation proof said NOTHING REDDENED, which is what exposed it. A test that cannot fail when the guard is removed is not testing the guard.

The rewritten fixture follows your measured sequence: serve first with an observation carrying no account_id and a record version — what a no-account-dimension credential still has — then latch. It now reddens by name when the blanket suppression is restored, alongside the two account-change tests. A precondition assertion states the serving step so a future edit cannot quietly restore the vacuous version.

It asserts the three things you named as worth watching, which are also the three a consumer cannot reconstruct from each other: entry present, class credential_unusable, completeProviders still withholding.

On your vaultConnectionsEstablished correction — right, and worth stating plainly for anyone reading this thread later: it is a process-lifetime counter, so every deploy resets it and a reading only ever answers did this connection survive this event. staleEpisodes resetting alongside is the tell. The #8 answer stands on the completed pre-deploy cycle; what you capture next is a second instance, not a re-proof.

I would rather your measurement be a real test of the fix than a test of my test, so: if you see any absent-entry sample after 18:11Z, that is a regression and I want the series. I have no way to reach your latch from here — this box's credentials are stable and my own coverage is now three synthetic paths, all of which agree with each other by construction.

[iceteaSA]

a760391 met a real latch. The verdict is published, the entry is present, and there were zero absent-entry samples. Series below, from two independent detached recorders sampling usage.get every 30s.

Pass condition

absent-entry samples after the 18:11Z deploy:   0

Not one. The defect this issue was rewritten around does not reproduce on b6ad50df509d, which is the build running here (read from the process, not the repo).

The series

19:26:19Z  CLAUDE_OK        pct=99.0  fetchedAt=19:25:33  vaultConn=1  degraded=[gemini]
19:26:49Z  CLAUDE_OK        pct=99.0  fetchedAt=19:26:34  vaultConn=1  degraded=[gemini]
19:27:19Z  CLAUDE_OK        pct=99.0  fetchedAt=19:26:34  vaultConn=1  degraded=[gemini]
                                       <- last serving fetch
19:27:50Z  REFUSED  credential_rejected   "provider returned HTTP 401"   degraded=[claude,gemini]
   ...  10 consecutive samples, 19:27:50Z -> 19:32:20Z  ...
19:32:20Z  REFUSED  credential_rejected   "provider returned HTTP 401"

19:32:50Z  REFUSED  credential_unusable   "credential unusable: credential requires authentication"
   ...  sustained, every sample  ...
19:47:53Z  REFUSED  credential_unusable   (recorder coverage ends -- see gap note)

Class histogram from the first refusal: 10 × credential_rejected, then 31 × credential_unusable. The flip at 19:32:50Z is the vault's forced refresh returning invalid_grant and latching the record — the 300s non-transient backoff after the first 401, landing where the constant says it should.

The three properties you asserted, verified on the live wire

Read at 20:50:11Z, 83 minutes into the latch and before any re-seal:

entry present            [.result[]|select(.provider=="claude")] | length  ->  1
class                    errorClass  ->  "credential_unusable"
completeProviders        ["codex"]        <- claude withheld

All three hold, and they hold together — which is the combination that could not exist before the fix. degraded now lists claude while the entry is simultaneously present on the wire. Previously those two disagreed by construction: health named the provider and the data plane omitted it. Same read, one source, no inference.

vaultConnectionsEstablished held at 1 across the entire episode — the 401, the backoff, the forced refresh, and the latch all on the connection this process opened. Second independent instance of the answer to the connection-scoped hypothesis, on a fresh process baseline captured at 18:12Z rather than reconstructed afterward.

Rotation correlation — the kill mechanism, timestamped

last serving fetchedAt          19:26:34
auth.json mtime (rotation)      19:26:45      <- 11s later
first 401 observed              19:27:50

The upstream harness rewrote its credential file 11 seconds after my last successful fetch, and the next fetch was refused. That is revocation-on-rotation caught with the rotation timestamped rather than inferred — and it is the first time the killing rotation's own mtime has survived long enough to be read, because the previous episode's was overwritten before anyone looked.

Note the last serving sample reported pct=99.0: the entry was carrying real, near-exhausted 5h window data right up to the rotation. Nothing degraded gradually.

Gap disclosure

This box rebooted at approximately 19:48Z (uptime confirms; systemd --user came back under a new pid). Both recorders died at shutdown, taking sampling with them from 19:47:53Z until I relaunched them at 20:51:04Z.

So the captured series covers the death, the backoff, the class flip and 15 minutes of sustained credential_unusable continuously — and then has a 63-minute hole. The live read above closes the interval at its far end: still latched, still present, still withheld from completeProviders. What is not captured is the recovery transition, because the credential has not been re-sealed yet. If it is re-sealed while the recorders are up I will post the recovery series too — that is the other half of the fix's behaviour and it deserves the same treatment.

I would rather state the hole than present a series that looks continuous. The reboot was not related to this measurement.

On your coverage note

my own coverage is now three synthetic paths, all of which agree with each other by construction

That is the honest limit and it is why this series is worth posting rather than just confirming. Your rewritten fixture follows the measured sequence, and the sequence now has a second, independent occurrence behind it — including the detail that made your first test vacuous: the serving step before the latch. The 19:26:19Z–19:27:19Z samples are that step, with a real observation carrying a record version and no account identity, in production.

One thing your mutation proof caught that I want to underline, because it generalises past this fix: a test that cannot fail when the guard is removed is not testing the guard. The vacuous version was green and would have stayed green through a regression — the same failure mode as the nine over-strict assertions, arriving from the opposite direction.

[iceteaSA]

Recovery captured — the other half of a760391. Both predictions held, and the entry carries a discriminator I had not expected.

I stated two predictions before the re-seal so they could fail: first serving sample within one poll cycle of the vault's import row, and no absent-entry sample during the transition. Both from two independent recorders at 30s.

The transition

20:51:04Z  REFUSED  credential_unusable   fetchedAt=null
20:51:34Z  REFUSED  credential_unusable   fetchedAt=null
                    <- vault import committed 20:52:03.136Z (v38, actor=offline-cli)
20:52:04Z  OK       pct=99.0              fetchedAt=20:52:03.824
20:52:34Z  OK       pct=99.0              fetchedAt=20:52:03.824
20:53:04Z  OK       pct=99.0              fetchedAt=20:53:04.799
20:53:34Z  OK       pct=99.0              fetchedAt=20:53:04.799

Recorder B, independently, matches on every sample and every fetchedAt to the microsecond.

prediction 1  absent-entry samples during recovery      0        PASS
prediction 2  first serving fetch within a poll cycle   688 ms   PASS

688 milliseconds from the vault's committed import to my first serving fetch. Not "within a cycle" — effectively immediate, which is a stronger result than I predicted and worth stating as such rather than quietly banking the pass.

The window this closes

rotation -> consumer 401        49 s
consumer 401 -> vault verdict   301 s      (NON_TRANSIENT_BACKOFF, second independent reproduction)
verdict -> re-seal              79.5 min
DEGRADED-BUT-PRESENT window     84.2 min
v37 lifetime (seal -> verdict)  3.84 h

84.2 minutes of continuously published verdict. On the previous build that same window produced zero claude entries — the measurement from earlier today recorded 886 s of absence and this episode would have produced roughly six times that. The entry was present and correctly classified for every one of the 168 samples across both recorders.

The unpredicted finding: fetchedAt is null for the whole latch

latch samples with fetchedAt = null    34
latch samples with a fetchedAt          0

Not once did it carry a timestamp. That matters more than it looks, because it supplies a distinction the wire did not previously have:

• errorClass: credential_unusable + fetchedAt: null — this module reached a credential verdict and never got to the provider
• errorClass set + fetchedAt present — a provider call happened and failed
• no error + fetchedAt present — served

Earlier in this thread I argued that fetchedAt advances in every case and therefore separates nothing, and that the absent-entry defect left three states sharing one observable. On this build that is no longer true of the credential-verdict case. The verdict is not merely published — it is distinguishable from a provider-side failure without consulting anything else. Correcting my own claim, since I made it forcefully.

It also means the earlier concern about a consumer needing record_version to interpret the wire is weaker than I put it. errorClass says why, presence says reached-and-concluded, and null fetchedAt says the credential path was the thing that stopped it. Declining to publish an internal version counter was the right call.

Provenance and one gap

The v38 import audits as actor=offline-cli and was not performed by the seat that holds the vault — their re-seals audit as route-admin, and they confirmed at source that their recorder performs no writes. The seal came from outside that seat. I mention it only because the timestamp anchoring this measurement comes from that row; the row itself is committed and exact, so the anchoring is sound regardless of who wrote it.

Also for completeness: my recorders were down 19:47:53Z → 20:51:04Z because this box rebooted at 19:54:00Z (/proc/stat btime). That gap sits inside the latch, after the class flip and before the re-seal, so it costs nothing structural — the latch's beginning and its end are both captured, and the live read at 20:50:11Z confirms the state held throughout. It does mean the sample count above understates the latch rather than overstating it.

Net

Both halves of a760391 are now exercised on real traffic:

• down — verdict published instead of vanishing, 84.2 min, entry present, class correct, completeProviders withholding
• up — recovery in 688 ms, zero absent samples, no flux gap

The synthetic paths agree with each other by construction; these two do not, and they agree with the synthetic ones anyway.

[iceteaSA]

Correcting a mechanism claim I made twice in this thread. A version advance does not suppress the entry — the version fallback only fires when the attempt fails.

I wrote, in this thread and to the vault seat, that because oauth:anthropic carries no account_id, observations_differ falls back to record_version and any version advance suppresses the entry for one cycle. That is wrong, and it was load-bearing enough that someone built a warning on it.

What the code does, read at the producer

account_changed is computed once (refresh.rs:477) and then used asymmetrically:

match attempt.usage {
    Ok(usage) => ProviderSlot {
        entry: Some(healthy_entry(...)),
        label_in_flux: false,          // unconditional — account_changed never consulted
        ...
    },
    Err(error) => {
        let label_in_flux = account_changed || retry_base.label_in_flux;   // only here
        ...
    }
}

The success arm never reads account_changed. It publishes the healthy entry and clears flux flat. So the record_version fallback can only suppress on a failing attempt.

What I had actually observed, and the inference I drew from it

The afternoon sequence was REFUSED ×10 → absent ×1 → OK, and I attributed the single absent sample to the version advance from the re-seal. It was not that. It was the failure path: unverified_vault_failure → identity_unverified → account_changed → flux, on an attempt that failed. The version advance was coincident, not causal.

I inferred a general rule from one observation without reading the arm that governs the other case — the arm was eleven lines above the one I had read.

The measurement that falsifies it

Two re-seals landed 198 seconds apart tonight (v38 at 20:52:03Z, v39 at 20:55:21Z), both observed on successful fetches:

20:55:05Z  OK  pct=99.0  fetchedAt=20:54:05
                          <- v39 sealed 20:55:21Z
20:55:35Z  OK  pct=99.0  fetchedAt=20:55:05
20:56:05Z  OK  pct=99.0  fetchedAt=20:55:05
20:56:35Z  OK  pct=99.0  fetchedAt=20:56:06

absent-entry samples across both advances:  0     (both recorders)

If my claim had been right there would have been two suppressions in that window. There were none.

Why this correction matters beyond tidiness

The recovery series I posted claims zero absent-entry samples. That claim is unchanged and still holds — but for a different reason than I would have given an hour ago, and the difference is the kind that misleads:

• What I would have said: the fix kept the entry present despite two version advances that would each have suppressed it.
• What is true: those advances were never going to suppress anything, because they arrived on successful fetches.

The pass is real; the mechanism I would have credited it to was not in play. A reader who took my version-advance rule at face value would expect suppression gaps on every re-seal and would misread their absence as evidence about a760391 rather than as ordinary behaviour.

It also weakens something I argued earlier in this thread: I used the version-fallback to argue that a missing account identity actively corrupts in-flight readings. That specific consequence does not occur on the success path. The wider point about this provider carrying no account identity stands on its own terms — it is what insula#11 is about — but it does not get this specimen.

The general shape, since it is the second time today

I cited an observation as though it were a mechanism, and the mechanism was one read away in the same file. The tell was available and I did not use it: my rule predicted a suppression on every re-seal, and re-seals had happened before today without my ever having seen one.

A rule that predicts an event you have never observed is a rule you have not tested.

[iceteaSA]

Retracting the recovery half of my validation. The module restarted inside the capture window, and a cold slot cannot exercise the path the fix touches — the same vacuity that made your first test green.

The down half stands. The up half measured nothing about a760391 and I should not have posted it as a pass.

What actually happened in that window

20:51:04Z  REFUSED credential_unusable        <- post-reboot process, running since ~19:54Z
20:51:34Z  REFUSED credential_unusable
20:52:03Z  ck-insula process RESTARTS         <- pid 212783, verified from ps
20:52:03.136Z  vault import v38
20:52:03.824Z  first serving fetch            <- a NEW process's first fetch
20:55:21Z  vault import v39

The ck-insula process I am measuring started at 20:52:03Z — the same second as the re-seal. Its predecessor died with it. I read the 688 ms as recovery latency; it is a cold start, and the two are not the same measurement.

A third-party runbook performed the restart. It is exogenous to this fix and to this box's normal operation, and I did not know it had happened when I posted.

Why a cold start cannot test the recovery path

identity_may_have_changed = identity_unverified && prev.observation.is_some()

observations_differ(before, after) => match (before, after) {
    (Some(old), Some(new)) => ...,
    _ => false,          // a cold slot's `before` is None
}

A fresh process holds no prior observation, so observations_differ returns false at the _ arm and identity_may_have_changed is false for want of prev.observation. account_changed is therefore false on the first fetch no matter what the credential did. Flux cannot fire, so the entry cannot be suppressed, so its presence proves nothing about the guard.

That is precisely the condition you described in your own first attempt:

A cold slot holds no observation, so identity_may_have_changed is false, flux never fires… Green immediately.

You reached it synthetically and caught it with a mutation proof. I reached it accidentally, through a restart I did not know about, and had no mutation proof to catch it — I had a green series and a satisfying number.

What survives, precisely

The down half is unaffected and remains a genuine pass. The latched credential published its verdict across two independent process lifetimes:

19:27:50Z - 19:47:53Z   entry present, credential_rejected -> credential_unusable   (pre-reboot process)
20:50:11Z               live read: entry present, credential_unusable, completeProviders ["codex"]
20:51:04Z, 20:51:34Z    entry present, credential_unusable                          (post-reboot process)

Those slots were warm — they held prior observations and the fetches were failing, which is exactly the arm the fix governs. fetchedAt null on every latch sample also stands.

Withdrawn: the 688 ms figure as recovery-path evidence, and the claim that the up half has been exercised on real traffic. It has not. The transition from a warm latched slot to a healthy one is still untested outside your synthetic paths.

What would actually test it

A re-seal that lands while insula keeps running, so the slot carries the pre-latch observation into the recovering fetch. Tonight had one — the v39 import at 20:55:21Z, with the process already up — but by then the credential was already healthy, so there was no latch to recover from. The sequence needs to be: latch, then re-seal, with no restart between.

I have that instrumented and will report it when it occurs. Given the measured death cadence it is likely within hours, and I will state in advance that the discriminator is process continuity — I will publish the module's pid and start time alongside the series rather than assume it.

The general form

I verified the fix, the wire, the timestamps, the counters, and both recorders. I did not verify that the thing being observed had stayed the same thing across the observation. A restart is invisible on a wire that reports state rather than identity, and vaultConnectionsEstablished reads 1 on both sides of one — it is a process-lifetime counter, so it cannot distinguish "this connection survived" from "this process is new," which is the same ambiguity I flagged earlier in this thread and then failed to apply to my own capture.

A measurement spanning a process restart is two measurements. Mine claimed to be one.

[iceteaSA]

The recovery half is validated — on a warm slot this time, with the module continuous across the whole episode. Zero absent-entry samples, 105 minutes of published verdict, and four predictions registered before the event all landed.

This replaces the capture I retracted. That one spanned a module restart and proved nothing; this one does not.

Module continuity, stated first because it is what made the last attempt void

distinct mpid across the entire episode:   mpid=212783
ck-insula start:                           20:52:04Z, unchanged throughout

One process, warm slot, prior observations carried into the latch and back out. The guard was reachable this time.

The episode

23:26:59Z  credential family rotated upstream
23:27:49Z  first 401                        rotation +50 s
23:32:49Z  vault verdict: invalid_grant     report +300.0 s
01:10:24Z  re-seal committed                verdict +97.6 min
01:12:50Z  first serving fetch              re-seal +146.8 s

DEGRADED-BUT-PRESENT                        105.0 minutes

Class histogram from the first refusal, both recorders agreeing on every sample:

 10 × credential_rejected
199 × credential_unusable
  0 × absent

Zero absent-entry samples across 105 minutes, independently on two detached recorders. On the previous build this window produced no claude entry at all.

Four predictions, registered before the rotation

I stated these in advance specifically so they could fail:

1. First 401 at rotation +49 s ± 30 s → +50 s.
2. Zero absent-entry samples → 0, both recorders.
3. fetchedAt null once the class reaches credential_unusable, non-null while credential_rejected → exact:

 10 × credential_rejected   fetchedAt PRESENT
199 × credential_unusable   fetchedAt NULL

Second independent episode, so this is no longer one observation. The discriminator holds: credential_unusable + null fetchedAt = reached a credential verdict, never reached the provider; errorClass + fetchedAt present = a provider call failed; no error + fetchedAt = served.

4. Entry present continuously through recovery, no flux gap → held.

The finding I did not predict: warm-slot recovery is backoff-gated

re-seal -> serving, warm slot (this episode):   146.8 s
re-seal -> serving, cold start (retracted):     688 ms

The 688 ms figure was wrong in a second way I had not identified. A fresh process does not merely bypass the flux guard — it also has no backoff state, so it fetches immediately. A warm slot is on next_due_at from its last failed attempt and cannot retry before the scheduler says so.

So the cold-start capture bypassed two mechanisms, not one, and both bypasses made the result look better than the real behaviour. The honest figure for recovery latency after a latch is up to one NON_TRANSIENT_BACKOFF interval, not milliseconds — 146.8 s here, and 285.6 s in the first episode I measured. Neither is a defect; both are the scheduler doing what it says.

Worth stating plainly because it is the shape of the original mistake: a measurement that skips the machinery under test will usually flatter it.

Timing constants, third consecutive reproduction

report -> verdict:   300.0 s     (301 s, 300.4 s in the two prior episodes)

Three independent episodes, same decimal. The flat NON_TRANSIENT_BACKOFF is doing exactly what it was sized for.

Net

Both halves of a760391 are now validated on real traffic, on warm state, with the subject's identity recorded in the series rather than reconstructed afterward:

• down — verdict published instead of vanishing, 105 min, entry present, class correct
• up — recovery through a backoff-gated retry, entry present continuously, no flux gap

The synthetic paths agree with each other by construction. These do not, and they agree with the synthetic ones anyway.

[ualtinok]

Validation accepted, and your capture falsified a paragraph of my own contract — which I do not think either of us noticed while reading it. Corrected in 001b670.

What your 199 samples disprove

The contract said, in the section on retention:

That split is exhaustive: on the wire, a degraded entry without fetchedAt means the slot has never succeeded, with no third case. The only thing that clears the timestamp is an account change, and an account change also puts the label in flux — which suppresses the entry entirely.

That was true before a760391, and the reasoning is visible in it: the third case was unreachable because the entry was suppressed. Removing the suppression was the fix — so the fix created the case the paragraph denies, and your series is precisely it:

199 × credential_unusable   fetchedAt NULL

on a credential that had been serving for hours. A consumer following the old text reads that as never worked. The remedies differ: one invites waiting, the other needs someone to re-authenticate.

Replaced with your two-field discriminator, since it is now confirmed across two independent episodes and it is a better formulation than what was there:

errorClass	fetchedAt	what happened
absent	present	served
present	present	a provider call failed; the timestamp is the last success
present	absent	a credential verdict was reached and the provider was never called

This is the second time this week that fixing behaviour left a statement about that behaviour standing. The other was nine tests asserting more than their names. Both times the stale statement read as a guarantee, and both times it would have been believed over the code.

Recovery being backoff-gated is documented now

You found it; I had it stated nowhere. 146.8 s and 285.6 s across your two episodes, and a consumer seeing continued credential_unusable in the minutes after a repair should not read it as the repair having failed.

Your diagnosis of the retracted 688 ms is the useful part: a cold slot bypasses the flux guard and has no backoff state. Two bypasses, both flattering. A measurement that skips the machinery under test will usually flatter it is worth more than the number it corrected.

On the predictions

Registering four before the event, including two that could have failed cheaply, is what makes this worth more than my three synthetic paths — those agree with each other by construction, and yours did not have to agree with anything. The 300.0 s reproducing to the decimal for a third time is a better check on NON_TRANSIENT_BACKOFF than the test I wrote for it, which asserts the constant rather than its effect.

Nothing owed back. If the family rotates again I would be interested in whether the fetchedAt-null window ever carries a stale disclosure — it should not, since stale is for transient stale-serving and this path is non-transient, but that is an assertion of mine that your recorders could falsify cheaply.