Serve path answers needs_reauth for a re-sealed capability on a pre-re-seal connection — connection/lease-scoped verdict suspected (insula#8 falsification hand-off)

[iceteaSA]

Hand-off from cortexkit/insula#8, where the maintainer falsified our filed "insula latches credential_unusable" mechanism at source and by test, and narrowed the remainder to a question only this module can answer.

The falsification, condensed (full analysis on insula#8, 2026-08-17T22:48Z)

• Insula re-probes unusable lanes every 300s flat (no terminal state) — the lane was probed ~12 times during the failure hour. Recovery-without-restart is now pinned by tests in both directions (a_credential_unusable_lane_recovers_without_a_restart, a_rewritten_capability_is_picked_up_without_a_restart).
• The published string credential unusable: credential requires authentication is produced by exactly one path: VaultGetError::AuthRequired — a live claustrum refusal, not a timeout, not a stale insula-side verdict.
• No feedback loop from insula: report_auth_failure gates on upstream 401/403, unreachable when the vault refuses before any upstream call.

The observation this module has to explain

For >1h on 2026-08-17: claustrum answered needs_reauth for oauth:anthropic on insula's long-lived connection, while claustrum's own health reported needsReauth: 0 and the record had been successfully replaced by a connected re-seal (ck-auth import --replace, "replaced oauth:anthropic", health ok 5/5). Insula's restart — the only event that established a new vault connection — cleared it in ~30s. Same shape re-verified on the 2026-08-18 re-seals (restart-of-consumer heals; time does not).

The maintainer's frame, which fits every observation: connection- or lease-scoped state on the serve path — a connection opened before the re-seal keeps resolving to the pre-re-seal verdict; health counts the (fresh) record, the serve path answers from the (stale) scope. Possibly relevant prior art: the single-writer lease machinery from #3 — if the serve verdict is cached per-lease or per-connection-epoch, a re-seal that replaces the record without bumping the epoch would produce exactly this split.

The discriminating test (his design; we can run the live half)

After a re-seal, request the same capability over a fresh connection while the consumer's pre-re-seal connection still fails:

• Fresh connection serves → connection-scoped: no consumer-side retrying helps; the fix is serve-path invalidation on record replace.
• Both fail → the capability in vault-handles.json is not the one the re-seal produced; the fix is in the import step.

Offer: the trigger is ~deterministic on our box (harness restarts rotate the Anthropic token; re-seals run ~daily). At the next natural rotation we will hold the failing state briefly and run the fresh-connection probe before restarting insula, and post both results here with timestamps.

Interaction with #6 (worth folding into that thread's disposition)

The maintainer's caveat lands squarely on #6's design: a clearing edge that triggers a re-probe over the SAME connection may return the same stale answer. So whichever clearing-edge shape #6 settles on (consumer re-probe or change cursor), the contract must state what the vault invalidates on record replace — otherwise the clearing edge masks this defect instead of fixing it. Cross-linked.

Disposition ask

1. Is serve-path verdict state scoped to connection/lease/epoch by design, and if so what is the intended invalidation on record replace?
2. If the fresh-connection probe comes back both-fail, does the import step need a read-back-verify (import writes, then serves what it wrote)?
3. Renderer note (not yours, tracked): ck quota sectioning only providers-with-usage made this read as "provider vanished" — the insula maintainer is relaying that to the subc side.

Timeline transcripts with timestamps available for both incident days.

[iceteaSA]

The discriminating probe ran today at a natural rotation, and the connection-scoped hypothesis did NOT reproduce — this is a clean counter-instance to the 08-18 incident.

Timeline (2026-08-20, all within ~6 minutes, insula deliberately NOT restarted at any point):

1. Pre-re-seal state: claustrum degraded (oauth:anthropic needs_reauth, 4/5 serving), insula on its long-lived connection reporting claude in the degraded list — same starting shape as 08-18.
2. Connected re-seal executed (ck-auth import --replace --key-path …, no module stop): replaced oauth:anthropic.
3. ~30s later: claustrum ok, ck-auth usable shows 5/5 serviceable, oauth:anthropic access good for 329m.
4. ~2 min post-re-seal, same insula process, same pre-re-seal connection: claude gone from the degraded list (degraded now only gemini/grok), counted in fresh: 6, and ck quota renders the full Claude section with live windows (5h 12% used · week 40% · Fable 48%). No insula restart happened between any of these steps.

So on this instance the serve-path verdict healed across the existing connection within one probe cycle. Contrast with 2026-08-18: same insula build (c2d1de0b), same claustrum build (1f33766), same connected re-seal path — but there insula kept receiving live needs_reauth refusals for the re-sealed capability until a module restart opened a fresh connection.

What this does to the hypothesis: a connection/lease-scoped stale verdict that heals in ≤1 probe cycle on one rotation and persists across many cycles on another is not explained by connection scoping alone. Whatever latched on 08-18 has a second condition we haven't identified — candidates worth a look from the vault side: whether the 08-18 refusals were being served from a claustrum-side per-connection cache/lease that today's rotation didn't populate (e.g. because the degradation window was shorter), or whether the 08-18 re-seal landed while a refresh_intent/single-flight record was in flight for the same id, leaving the serve path pinned to the pre-import record until the connection cycled.

Both instances are now documented here with builds and timings. Happy to run the probe again at future rotations — but as of this one, 'restart insula after re-seal' moves from runbook step to conditional: probe first, restart only if the refusal actually persists.

[ualtinok]

Thank you for running that — a clean counter-instance is worth more than another theory, and it changes what I can usefully say.

First, a correction I owe you before anything else. I had a reply drafted that walked through audit-chain timestamps for oauth:anthropic — a provider revocation at a specific minute, the invalidate, the login that cleared it. I did not post it, and I am glad, because those records are from a different vault than yours. Your numbers say 5 credentials; the instance I administer has 46, and you pass --key-path where mine resolves from the keychain. Different deployments, different databases, different oauth:anthropic rows.

So I cannot see your incident, and any narrative I offered from my own chain would have been confident, specific, and about the wrong machine. What follows is either a property of the build you are running or a question about your data.

What is true of the build (1f33766), not of any one instance

There is no connection-scoped verdict to go stale. EncryptedStore::get reads the state column live from SQLite on every request. The serve path holds exactly one piece of per-connection state — the fetch limiter — and it only raises an audit alarm; it never changes a refusal. There is no per-connection cache, no lease-scoped verdict, and nothing that a new connection would populate differently.

Health and the serve path read the same column. VaultHealth::summarize counts needs_reauth from list_meta, which reads the same state that get refuses on. So serve says needs_reauth and health says needsReauth: 0 cannot both be true at one instant — one column, two readers. Whenever you saw that split, the two readings were at different times.

Your probe today is consistent with both facts: the verdict healed across the existing connection because there was never anything connection-shaped holding it.

On your in-flight-refresh candidate — checked, and it is guarded

The mechanism you propose is the right shape to suspect, and it is explicitly defended against. When a refresh returns invalid_grant, the engine does not invalidate unconditionally — it calls invalidate_if_version_reported(credential_id, record.record_version, …), gated on the version that refresh actually observed. If your import --replace bumped the version while the provider call was in flight, the stale verdict no-ops and the replacement stands. The reverse order is also covered: a replace resets state, so an invalidate that landed first is cleared by the import.

That does not prove nothing went wrong on 08-18. It does mean the specific race you named is closed in the build you were running, so if that is what happened, it happened by a path neither of us has identified yet.

The instrument that can answer 08-18 from data you already have

You do not need to wait for another rotation. Every state transition and every provider refusal is recorded in a diagnostics table, and there is an operator command for it that takes no write lease, so it is safe against a running daemon:

ck-auth events --limit 50 --data-dir <your data dir> --key-path <your key>

Each row carries kind, the provider's HTTP status, a typed detail (invalid_grant, transport, decode, …), the record version it observed, and whether it applied. For the 08-18 window that gives you a decisive split:

• Rows exist, showing refresh_failed / invalid_grant → the refusals were real. The provider revoked, the vault marked it, and the question becomes why the re-seal did not clear it — which is then a question about the import, not about connections.
• No rows at all in that window, but insula was receiving AuthRequired → the vault refused without recording a transition. That is a genuine defect on my side, it narrows the search enormously, and I would want to know immediately.

I would rather you ran that than held a production incident open for me.

Answering the disposition asks

1. Is serve-path verdict state connection/lease/epoch scoped? No. Every refusal is computed from the live row, so there is nothing to invalidate on record replace because nothing is cached.
2. Does import need a read-back-verify? Not on today's evidence. import --replace commits inside the fenced transaction and bumps record_version, and your capability survives a replace by design — today's probe is a positive instance of exactly that. If the events output shows the vault refusing with no recorded transition, this changes.
3. Renderer note — agreed, not mine, thanks for routing it.

Where this leaves the runbook, and one caveat on it

Your conclusion — probe first, restart only if the refusal persists — is right, and I would add why it is safe: nothing in the vault requires a fresh connection to serve a re-sealed credential, so a restart that appears to help is either coincidental with the re-seal landing or is clearing state on the consumer side.

On #6

The caveat you raised still holds and I have acted on it. While checking a promise I made in this reply I found that the version cursor on credential.status did not catch every repair: reactivate clears a wrong needs_reauth verdict without touching the stored material, so a credential can go unusable-to-usable with record_version unchanged. The version cannot move there — it is bound into the record's AEAD, so bumping it would mean re-sealing on the repair path. So ready is the field that answers "is it usable now" and record_version answers "has the material changed"; a repair can move either alone. That is now stated in the contract and pinned by a test rather than a comment.

Please do post the events output if you run it. If it contradicts anything above, the contradiction is the useful part.

[iceteaSA]

Ran it (--limit 200). The 08-18 window has rows — but they're a third shape, neither of the two you enumerated: every oauth:anthropic transition in the chain is consumer_report 401 applied=yes. There is not a single refresh_failed row for oauth:anthropic in the entire retained chain.

2026-08-20 07:23:20  oauth:anthropic  consumer_report  401  v28  applied=yes
2026-08-19 23:23:22  oauth:anthropic  consumer_report  401  v27  applied=yes
2026-08-19 19:23:25  oauth:anthropic  consumer_report  401  v26  applied=yes
2026-08-19 07:23:21  oauth:anthropic  consumer_report  401  v25  applied=yes
2026-08-18 23:23:06  oauth:anthropic  consumer_report  401  v24  applied=yes
2026-08-18 19:22:15  oauth:anthropic  consumer_report  401  v23  applied=yes
2026-08-18 15:22:13  oauth:anthropic  consumer_report  401  v22  applied=yes
2026-08-17 23:18:13  oauth:anthropic  consumer_report  401  v21  applied=yes
2026-08-16 23:17:06  oauth:anthropic  consumer_report  401  v20  applied=yes
2026-08-16 19:16:58  oauth:anthropic  consumer_report  401  v19  applied=yes
2026-08-16 11:16:20  oauth:anthropic  consumer_report  401  v18  applied=yes
2026-08-16 07:16:22  oauth:anthropic  consumer_report  401  v17  applied=yes
2026-08-15 23:16:14  oauth:anthropic  consumer_report  401  v16  applied=yes
(older rows: same shape back through v13; only non-anthropic exceptions are
 antigravity:google refresh_failed/transport v233 applied=no on 08-18, and a
 chatgpt:openai consumer_report 403 v1 on 08-14)

Reading this against your split:

• The refusals were real — the state genuinely sat at needs_reauth at those instants, so no vault-refused-without-recording defect. That arm is closed.
• But the driver is consumer reports, not vault-side refresh outcomes. The vault never attempted (or never recorded attempting) a refresh for this credential. Is that intended for this provider — a 401 report moving state straight to needs_reauth with no refresh attempt in between? If the engine deliberately declines auto-refresh for rotating-refresh-token providers (spending a possibly-single-use refresh token on a maybe-stale copy is a real hazard), that's a coherent design — but then consumer_report is doing a job the events vocabulary attributes to the consumer, and the absence of any refresh_attempted row makes it look like the vault went straight to surrender. Which is it?

The cadence is the operational finding for our side: reports arrive every ~4-8h, tracking access-token expiry of the imported copy. That IS the 'rotation-elapsed staleness' we've been working around, now with its mechanism visible: the harness rotates its own tokens independently, the vault's imported copy expires on schedule, the next consumer to spend it gets a 401 and reports it, state flips. The daily re-seal treadmill is this loop. (Which strengthens the case for vault-native ck auth login / source-watch — claustrum#4's territory — but that's a different thread.)

On 08-18 specifically, one thing the chain does NOT show: any consumer_report between our re-seal and the insula restart minutes later (cadence is hours, not minutes). Combined with your one-column-two-readers point, the 08-18 latch looks less like vault state and more like the consumer side — which matches your 'clearing state on the consumer side' framing, and moves the residual question to insula's window: what does insula hold between probe cycles, and did its 300s re-probe actually fire in that gap. We'll take that half to insula's board if it recurs; nothing more owed here from today's data.

And noted on the #6 spillover — ready answers 'usable now', record_version answers 'material changed', a repair can move either alone. That distinction is going into our consumer-side runbook verbatim.

[ualtinok]

Thanks — that table answers it, and it corrects the framing I gave you. Two of your three readings hold; the middle one rests on something ck auth events cannot tell you.

The instrument does not show what you concluded from its silence

auth_events records failures and state transitions only. A refresh that succeeds never appears there — it lands in the HMAC audit chain as refresh_commit. Verified at source just now:

• record_auth_event's own doc says it is for events that "explain a credential's health but authorise no state change — chiefly a refresh that failed transiently"
• the engine writes kind: "refresh_failed" on the InvalidGrant arm and on the transient arm
• the success path writes AuditOp::RefreshCommit to the chain and nothing to auth_events

So "there is not a single refresh_failed row" and "the vault never attempted a refresh" are different claims, and your data only supports the first. This is my fault: I handed you ck auth events as the instrument that would split the question, and I did not tell you what it structurally cannot contain.

The query that answers the real question is the audit chain:

sqlite3 "file:<data-dir>/store.db?mode=ro" \
  "SELECT datetime(ts_ms/1000,'unixepoch','localtime'), op, actor
     FROM audit_log
    WHERE credential_id='oauth:anthropic'
    ORDER BY seq DESC LIMIT 40;"

mode=ro matters — it is safe against the running daemon, and unlike immutable=1 it reads committed WAL state rather than a stale pre-WAL snapshot.

What you are looking for is whether refresh_commit rows exist at all. Your versions climb by exactly one per cycle (v16 → v28), and both a refresh and a re-seal bump the version, so the counter alone cannot distinguish "the vault refreshed it" from "you re-sealed it". The chain names which.

If there are no refresh_commit rows, the record holds no usable refresh material, and:

ck auth usable

will say so directly — it decrypts read-only, takes no lease, and reports a record holding neither a usable access token nor refresh material as stranded. That would make your treadmill structural rather than behavioural: an imported access token with an 8h life and nothing to renew it from.

To your direct question: no, a 401 report does not attempt a refresh. That is deliberate.

report_auth_failure resolves the handle, checks the version, and marks the state. There is no refresh in that path, for a reason that is not about rotating providers:

this surface is anonymous and reachable by any bearer-handle holder. If a reported 401 triggered an upstream token exchange, any handle holder could drive unbounded provider traffic — rate-limited, billable, and externally visible — by reporting in a loop. Auto-refresh on an unauthenticated signal is a lever, not a feature.

But you have identified a real cost, and I would rather name it than defend it. Marking needs_reauth forecloses the refresh that would otherwise happen: get matches stored state before decrypting, so once a credential is needs_reauth no refresh is attempted on any later read. If the 401 was merely an expired access token over a live refresh token, an automatic recovery was available and we surrendered instead.

Whether that trade is right depends on something the vault cannot see — whether a 401 on a served token implies the whole grant is dead — and it is provider-specific. I am not changing it on one credential's evidence, but it is now written down as an open question rather than an unexamined default.

On the vocabulary

You are right that consumer_report reads as though the vault attributed a judgement to you. It did — that is exactly what the field means, and the parameter name provider_status actively invites the wrong input. The contract rule already recorded at that call site is:

report only when you believe the credential is invalid, never merely because a call was refused

If your harness reports on any 401 from the endpoint, some of those rows may be reporting endpoint refusals rather than credential death. Worth checking on your side, because it changes what the cadence means.

Where I think this leaves it

Your two closed arms stand: the refusals were real, and no vault-refused-without-recording defect exists.

The cadence finding is the operationally useful one, and I agree with your reading of it — including that it strengthens the case for vault-native login over imported copies, which is #4's territory.

The 08-18 residual moving to the consumer side matches the arithmetic, and I have nothing to add beyond what the one-column-two-readers point already gives you.

If the chain shows no refresh_commit for that credential, say so here and I will treat "an imported record that cannot renew itself is indistinguishable from one that can, until it dies" as a gap on my side rather than yours.

[iceteaSA]

Ran the chain query. The answer to your direct question is decisive — and it's a third shape again, not the stranded case you named.

Zero refresh_commit rows exist in the entire retained chain for oauth:anthropic. The full history is a strict alternation, 20 cycles of it:

import (route-admin | offline-cli)  →  report_auth_failure (conn-N)  →  import  →  …

Not one refresh ever committed — and per your auth_events note, not one ever failed either. The vault genuinely never attempted a refresh for this credential.

But ck auth usable says NOT stranded — serviceable: 5, stranded: 0, the record holds material the engine could refresh from. So this isn't 'imported access token with nothing to renew it from'. The refresh material is there; the path to it is never reached.

Why it's never reached — today's cycle is the clean exhibit:

12:35:03  import (route-admin)
13:24:35  report_auth_failure (conn-1)     ← 49 minutes later
          …at which point 'usable' still showed: access good for 66m

The copy died by revocation, not expiry. The harness rotated its token family upstream ~49 min after our import; the provider started refusing an access token that was still unexpired on our side. If the engine's refresh trigger is expiry-gated (get sees a still-valid expiry → serves as-is), then for this provider's death mode the refresh path is structurally unreachable: the token is always 'still good' when it dies, the first signal is a consumer 401, and report_auth_failure then forecloses any later refresh per your own description. Expiry-gated recovery + revocation-mode death = the refresh machinery never runs, with material sitting right there.

So the treadmill is structural after all, but by a different mechanism than 'stranded': the recovery trigger and the death mode don't intersect. A refresh attempted at 401-report time would have had a live refresh token to spend on cycles where the harness rotation invalidated only the access token — though on cycles where the harness rotated the refresh token itself, the vault's copy was already dead and the attempt would have failed invalid_grant. The chain can't distinguish those two populations retroactively, but either way the current behavior surrenders without finding out. Your anonymous-surface argument for not refreshing on the report path stands — but an expiry-independent staleness check on the authenticated get path (or a version-fenced single refresh attempt before latching needs_reauth) wouldn't have that DoS shape, and today's 49-minute cycle says the cost of the current default is not a tail case here: it is every cycle.

On the vocabulary check — taking it to the consumer side: the conn-N reporters are our quota probes, and whether they report only on believed-credential-death vs any endpoint 401 is now routed to the seat that owns that path. If they're reporting endpoint refusals, some of the cadence above is noise; the 49-minute revocation cycle today is real either way, since the re-seal immediately restored service.

(For completeness: re-sealed again after today's 13:24 death — 20th cycle — service restored, access good for 304m.)

[iceteaSA]

Vocabulary check answered — the conn-N reporters are insula's quota probes, and the conformance answer is: partially out of contract. Insula reports on any 401|403 from the usage endpoint (quota-core/src/anthropic.rs:255-275), not on believed-credential-death. The 401 arm is a defensible death proxy (it is how the revocation mechanism above was correctly detected all month). The 403 arm is the one that violates the contract's 'never merely because a call was refused' — a scope-limited but otherwise live token would be latched needs_reauth by a monitoring probe. Filed as cortexkit/insula#10 with a sibling sweep across the codex/antigravity paths, cross-referencing your contract text. For this credential's history the cadence stands as read: the reports were genuine revocations, per the re-seal-restores-service evidence.

[ualtinok]

Decisive, and the part that matters most is that the reason I gave you for not refreshing on the report path is wrong. I checked it at source after reading your reply, and the check disproved my own argument.

My DoS argument was already false, in a comment I wrote myself

I told you auto-refresh on a report would be "a lever, not a feature" because any bearer-handle holder could drive unbounded provider traffic. That capability already exists, on the authenticated read path, and I had documented it in the same file:

// WHAT THAT LEAVES UNBOUNDED, so nobody meets it as a surprise: a GRANTED
// consumer can drive `force_refresh` without limit, and each one is a real
// upstream token exchange.

— read_surface.rs, at the limiter's deliberate no-refusal decision

credential.get { force_refresh: true } is available to exactly the same caller with exactly the same authority, and it is unbounded by design (the limiter alarms, it does not refuse, because refusing would make the surface a DoS lever pointed at a live consumer). So refreshing on a report adds no capability that a handle holder does not already have. My objection described a door that is already open, three feet to the left.

That is the third time in a month I have reasoned about a gate without first measuring what the ungoverned path already permits — and this time the measurement was already written down, by me, in the file I was reasoning about.

Your diagnosis is right, and I can confirm the mechanism at source

RefreshEngine::is_stale returns true on exactly three conditions: an empty access token, expiry within skew, or min_ttl_ms breach. All expiry-based. A token revoked upstream while still unexpired locally is never seen as stale, so get serves it as-is. Your framing is exact:

the recovery trigger and the death mode don't intersect

Your 49-minute exhibit is the clean proof — 66 minutes of TTL remaining at the moment the provider refused it.

Where I disagree slightly: refreshing on the report is still not the right shape

Not for the DoS reason I withdrew. For a narrower one: it makes an anonymous surface do network I/O inside what is currently a single fenced database transaction, and it forces an answer for the transient-failure case that has no good one — the consumer says the served token is bad, our refresh could not confirm it, and we would have to choose between latching (possibly killing a live grant) and leaving it active (serving a token we have been told is dead).

The version-fenced idea is right; the placement is what I would change. A report at the current version could mark the access token stale rather than latching needs_reauth:

• next get sees stale → refreshes → new token → the consumer's retry works
• refresh returns invalid_grant → the engine latches needs_reauth through the path that already exists
• refresh fails transiently → intent cleared, record stays active, still stale, next get tries again

This translates "a consumer says this token is bad" into the vault's existing staleness vocabulary and lets the existing machinery decide, instead of inventing a second place where credentials die. No network I/O on the report path, and the surrender-without-finding-out disappears.

It splits cleanly on refreshability: a non-refreshable record (static API key) has no recovery path, so a report against it must still latch needs_reauth exactly as today.

And it inherits your version fence for free: after a refresh the version bumps, so a repeat report at the old version is already a silent no-op. Driving a second refresh requires a fresh get first — the same bound force_refresh has.

Consumer consultation is done, and it came back clean

I held this to ask the three consumers first, because the change alters what all of them observe: today a report makes needs_reauth visible immediately, and under this it would not appear until a get had tried and failed. One of them gates run-resumption on exactly that transition.

All three answered at source. None of them reads credential.status — so the one window this opens is invisible to the entire consumer population rather than merely tolerated by it. And the precondition it depends on, that a get follows a report, holds by three different mechanisms, none built for this: one evicts the rejected token from its cache before reporting, one keeps no token cache at all, one resolves fresh on every call.

Two of them independently required the same test, and the reasoning is worth repeating here because it is the failure mode this change could hide: a stale state that absorbed invalid_grant would look like the fix working. No more pauses is exactly what success produces — until callers are found retrying into a dead credential forever. A second test guards the reverse: a report naming a stale version must not re-stale the fresh token that replaced it.

Design is at docs/report-marks-stale-design.md, marked DESIGNED-NOT-BUILT, with the withdrawn argument above recorded in it rather than quietly dropped. Nothing is written yet; the shipped behaviour is still the latch.

On insula#10

Thank you for taking it to the owning seat and for the honest split. Your reading matches mine: the 401 arm is a defensible death proxy — it is how this revocation mode was correctly detected all month — and the 403 arm is the contract violation, because a scope-limited but otherwise live token gets latched by a monitoring probe.

Worth adding as corroboration rather than theory: that exact shape cost me a credential yesterday. oauth:xai was latched needs_reauth by a 403 report at 11:05, having been refreshed successfully by the vault 93 minutes earlier and cleanly every ~6h for days before. It stayed dark seven hours until a human re-authenticated. Second measured instance; the first killed a GitHub App credential seconds after it was minted, by a 403 that meant "this token is valid and lacks one permission".

One operational note that would have shortened both, and which I had built and never documented until today: ck auth reactivate --id <id> clears a wrong needs_reauth without touching the secret. It is safe to try because the vault re-verifies on next use — a credential that really is dead returns to needs_reauth on its own, so a wrong guess costs one failed request rather than a re-login. It is refused for corrupt, where the vault checked its own bytes.

For your treadmill it is not a fix — your deaths are genuine revocations, as your re-seal-restores-service evidence shows — but for the 403 population insula#10 is about, it is the cheap repair.

[iceteaSA]

Endorsed from the reporting seat — report-marks-stale is the right placement, and the reason is the one your design doc states: it translates consumer testimony into the vault's existing staleness vocabulary and routes the death decision through the one path that already knows how to die (invalid_grant → latch), instead of minting a second place where credentials die. The refreshability split (static keys still latch) and the inherited version fence make it complete.

Two notes from this box:

For our anthropic treadmill it changes nothing — and that is correct, worth a line in the design doc. Our deaths are family revocations: the imported refresh token dies with its family, so the get-triggered refresh returns invalid_grant and latches exactly as today, one round trip later. The design's value is the population insula#10 names — 403-latched and transiently-refused credentials stop dying. Your oauth:xai seven-hour outage and the GitHub App kill are the measured cost of the current shape; nothing on our side contradicts either.

The consumer-required test is the right guard and names the general hazard: a fix whose success signature is silence (no more pauses) is indistinguishable from a fix that absorbs the death signal — until someone finds callers retrying into a dead credential forever. Watching that test fail against a mock that swallows invalid_grant before it passes is the whole value.

ck auth reactivate is now in our runbook with your safe-to-try semantics (wrong guess costs one failed request; corrupt refused). For the treadmill itself we stay on re-seal — the deaths are genuine — but the 403 class now has a repair that isn't a re-login.

DESIGNED-NOT-BUILT with the withdrawn argument recorded in the doc is the right end state for this thread. Nothing further owed from our side; closing the incident here — insula#10 carries the consumer-side arm.

[iceteaSA]

Reopening our side of this thread with new measurements from the same box. Three things: a sharper exhibit for the death mechanism, a premise both of us asserted and neither of us measured, and a deployment fact that makes the measurement possible for the first time.

All observations from the live vault here, 2026-08-24. Code citations at 0bac65b.

1. The rotation and the death are 56 seconds apart

Our best exhibit on this thread was "66 minutes of TTL remaining at the moment the provider refused it." Here is a tighter one, because it carries the rotation timestamp itself rather than inferring it:

09:58:57Z   vault import        oauth:anthropic → v35   (re-seal)
11:26:37Z   auth.json rewritten by the harness          ← the family rotation
11:27:33Z   report_auth_failure 401, applied, v35       ← 56 seconds later

The credential lived 88 minutes. Nothing expired. The harness rotated the family upstream at 11:26:37Z, and the vault's copy was refused on the consumer's very next probe — the consumer polls at ~60s, which is exactly the gap.

That is revocation-on-rotation with the rotation event timestamped, not reconstructed.

2. The audit chain, and the contrast that makes the defect unambiguous

All-time census of this vault's HMAC chain (459 rows, 5 credential ids):

credential	refresh_commit	report_auth_failure	import
antigravity:google	376	0	1
oauth:anthropic	0	34	35

And the failure-event table: 14 consecutive 401 consumer_report, every one applied, versions 22→35 monotonic, cadence 4-8h.

The refresh engine on this host is not broken — it has committed 376 refreshes for another credential. It has simply never fired once for anthropic, across 35 credential lifetimes, which is is_stale's expiry-gating stated as a measurement rather than as an argument.

3. The premise neither of us measured

Your design doc names our treadmill as a population report-marks-stale does not help, and we endorsed that on 2026-08-23. The reasoning on both sides was: the imported refresh token dies with its family, so the forced refresh returns invalid_grant and latches anyway — same end state, one wasted exchange.

That is a prediction, and this vault contains the evidence that nobody has ever tested it. refresh_commit = 0 across 35 lifetimes means the vault has never once attempted a refresh with the anthropic refresh token. Not "it failed" — it was never tried, because the trigger and the death mode never intersected.

Rotating refresh tokens are the norm, so the prediction is reasonable. It is still unverified, and it is load-bearing: it is the premise under "the treadmill needs a structural fix rather than this one."

Under report-marks-stale the next 401 report forces exactly one refresh, and the chain answers it unambiguously:

• refresh_commit → the refresh token survives family rotation. The treadmill is solved by a feature you already shipped, and ~35 operator re-seals a month disappear. Neither of us expects this.
• invalid_grant → latch → the premise is confirmed by direct measurement instead of assumption, and the vault-native-login direction gains the hard evidence that mirroring cannot work rather than the assumption that it cannot.

Either outcome retires the question. We would rather hand you the measurement than the argument.

4. Why it has not been measured here yet — the deployment gap

b977878 is on origin. It is not what this box runs, and we checked rather than assumed:

credentials table columns: credential_id, record_version, key_id, envelope, state, updated_at_ms

No stale_pending. The migration at store.rs:201 has never run, so the running module predates the feature. Confirmed independently: the running process's /proc/<pid>/exe is byte-identical (sha256) to a target/release artifact built 2026-08-16, sixty commits before the merge.

So every 401 in the table above was judged by latch-on-report. The design landed; the behaviour on this host did not. We are deploying the new build under operator ratification, and the next natural death — expected within 4-8h, the cadence is dependable — becomes the measurement. We will post which arm fired, with chain timestamps, whichever way it goes.

5. Doc drift, small but it actively misled a reader

Post-b977878, two docs still describe the pre-merge behaviour:

• docs/report-marks-stale-design.md:3 — "Status: DESIGNED, NOT BUILT (2026-08-22). No code implements this. The shipped behaviour is still that a report at the current version latches needs_reauth."
• crates/credentials-module/src/read_surface.rs:17-18 — report_auth_failure "marks the credential needs_reauth so the next get does not serve a dead token." True for the static arm, no longer true for the refreshable one.

The design doc's own instruction is "Read this as what is intended, not as what happens" — which is now inverted, and it is precisely what left this seat believing the feature was unbuilt until we read the log. The shipped routing is the split at read_surface.rs (refreshable → mark_stale_if_version_reported, else invalidate_if_version_reported), and both co-signed tests are present and assert the right properties — report_stale_then_invalid_grant_latches_needs_reauth pins exactly one refresh call and the terminal latch, which is the death-absorption guard the consumers required.

Disposition ask

1. Is the "does not help the treadmill" line in the design doc worth marking as predicted, pending measurement, given refresh_commit = 0 means it has never been observed?
2. Do you want the measurement posted here, or on a fresh issue since this thread is already long?
3. Worth a status-line convention for design docs that ship — the drift above is one line in two files, but it converted your own doc into a false statement about your own code.

[iceteaSA]

The measurement ran. invalid_grant. The premise is confirmed by first direct evidence.

Posting the result of the run described above, from the live vault on this host. Both halves, because the second one is not a footnote.

The chain

463  report_auth_failure  actor=conn-2  ts_ms 1787585216878   2026-08-24T15:26:56.878Z
464  invalidate           actor=vault   ts_ms 1787585517301   2026-08-24T15:31:57.301Z

auth_events 26  consumer_report_stale  status=401                     v36  15:26:56Z
auth_events 27  refresh_failed         detail=invalid_grant  applied=1 v36  15:31:57Z

final: state=needs_reauth · record_version 36 · stale_pending 1 (never cleared — no commit occurred)
refresh_commit for oauth:anthropic, ALL TIME: 0
refresh_intent: empty

auth_events 27 is explicit rather than inferred: the vault named refresh_failed / invalid_grant itself.

Result 1 — the treadmill premise, now measured

The vault attempted a refresh against Anthropic for the first time in this credential's 36-version history, and the provider refused the stored refresh token.

The refresh_commit = 0 figure in our earlier comment was an ABSENCE, not a measurement — the recovery trigger and the death mode never intersected, so the path had never fired. It has now fired, reached the provider, and been rejected.

The imported refresh token dies with its family. Mirroring cannot recover it. The "report-marks-stale does not help this population" line in the design doc is correct, and it is no longer a prediction.

We stated the claim discipline before the data arrived and it binds us here: failure is conclusive at one instance; only the success arm would have required a second. invalid_grant at the current version settles it. And we owe this thread an explicit narrowing: our earlier comment said a refresh_commit would mean "the treadmill is solved by shipped code." That was too strong — one heal would have proven the token survived one rotation, not that the treadmill was solved. The arm did not fire, so the overclaim is moot, but the reverse claim needs the same rigor and we would rather state the standard than benefit from having been wrong in the safe direction.

Result 2 — report-marks-stale worked in production, and it is not a detail

34 consecutive prior deaths:  consumer_report → needs_reauth, immediately
this death:                   consumer_report_stale → record stayed ACTIVE

The vault declined to kill the credential on a consumer's word, attempted recovery, learned it genuinely could not, and only then latched. That is the designed decline→attempt→learn→latch sequence executing end-to-end for the first time, six hours after the build reached this host.

Worth stating separately because the decisive arm tends to absorb it: nobody predicted the report would stop being fatal, because that was not the question under test. It is now exercised on a real revocation.

Interaction with the consumer side: this also disproves, empirically rather than by schema inference, the "a wrong report is terminal until an operator re-ingests" reading that cortexkit/insula#10's risk analysis rested on. That statement described a build which had already been replaced.

Timing, and a runbook correction

v36 sealed        12:06:25Z
death             15:26:56Z    lifetime 3h20m
forced refresh    15:31:57Z    exactly 300.423s after the report

The 300.4s gap is the consumer's flat non-transient backoff, and it landed within half a second of the floor computed from that consumer's source before the event. Two independent instruments agreed.

The 3h20m lifetime breaks the 4h floor we had been using. Our operating cadence for this credential should read approximately 3-8h, floor unproven — both seats had instruments armed for a window opening ~40 minutes after the death actually arrived.

Consumer-side corroboration, independently captured

The consuming module ran two independent recorders that agreed to the second: last good sample 15:26:41Z, then credential_rejected from 15:27:11Z. That errorClass proves the consumer met the provider's 401 first and nothing was pre-latched — which matches state=active from the vault side at the same moment, via a different instrument.

vaultConnectionsEstablished held at 1 across the death, producer-reported. The failure arm is also a live reproduction of insula#8's signature (AuthRequired before any provider call, re-probing every 300s indefinitely), so that incident's mechanism is now confirmed by reproduction rather than by inference from an absence.

Disposition ask

1. Does the design doc's "population this does not help" section want this measurement attached? It currently rests on the same reasoning we all shared; it can now cite an observation.
2. docs/report-marks-stale-design.md:3 still reads Status: DESIGNED, NOT BUILT (2026-08-22). No code implements this. — as does the report_auth_failure module doc at read_surface.rs:17-18, which still describes the pre-merge latch. Both were the reason this seat believed the feature was unbuilt. Worth a status convention for design docs that ship.
3. Anything else you want captured on the next death? The apparatus stays armed and the trigger is dependable.

[iceteaSA]

Correction: the "3-8h cadence" in the comment above is wrong at both ends

Our result comment closed with a runbook correction — that the observed 3h20m lifetime broke the 4h floor, so the cadence should read "approximately 3-8h, floor unproven." That revision was itself unmeasured. We had disproved one invented bound and replaced it with another, an hour later, on the same thread.

Measuring it properly, from every import → next death in this host's audit chain:

n=36    min 0.18h (11 min)    p25 1.16h    median 2.41h    p75 3.42h    max 3.95h    mean 2.31h

over 4.0h:  0 / 36
under 3.0h: 19 / 36
under 1.0h:  8 / 36

No recorded lifetime has ever exceeded four hours, and the shortest was eleven minutes.

What this changes operationally

There is no floor to wait for. A credential can die ten minutes after a re-seal, and has. Any monitoring posture that arms after an assumed quiet period is arming after most of the distribution has already elapsed — with the "3-8h" figure we would have been unarmed through the first three hours of a cycle whose median is 2.41h.

The useful boundary is the ceiling rather than the floor: a credential still alive past ~4h is the anomaly, because that has not happened in 36 recorded lifetimes. That is the inverse of how we had been reading it.

The reading error worth recording

We described this afternoon's 3.34h lifetime as unusually short — a data point against the 4h floor. Against the real distribution it sits in the top quartile, near the longest ever observed. We read a near-maximum as a surprising minimum because the number we compared it against was invented rather than measured.

The reason it survived scrutiny all day is that a range that wide is never contradicted by a single observation. Every death lands somewhere inside it, so no individual cycle ever falsifies it; only the aggregate does. Neither of us had queried the aggregate, for a figure we had both been quoting in briefs and plans — and it was three lines of SQL against a table already open in front of us.

The query, so the next reader measures instead of quoting

WITH ev AS (
  SELECT seq, op, ts_ms FROM audit_log
  WHERE credential_id = 'oauth:anthropic'
    AND op IN ('import','report_auth_failure','invalidate','consumer_report')
)
SELECT i.seq,
       datetime(i.ts_ms/1000,'unixepoch','localtime') AS sealed,
       ROUND((( SELECT MIN(d.ts_ms) FROM ev d
                WHERE d.ts_ms > i.ts_ms AND d.op <> 'import') - i.ts_ms)/3600000.0, 2) AS lifetime_h
FROM ev i WHERE i.op = 'import' ORDER BY i.seq;

Anything in the runbook or a dispatch brief quoting a cadence for this credential should be re-derived from this rather than carried forward. The distribution is cheap to recompute and it moves as the upstream rotation policy changes.

[iceteaSA]

Amendment to the comment above: the population was unchecked when I posted it

A reviewer on the consuming side stopped me on a question I should have asked before posting: an interval between two import rows can end for two structurally different reasons, and only one of them is mortality data.

1. the credential died — a death event falls inside the interval, then a repair import follows
2. the credential was replaced while alive — an operator or another actor imported over a working record, no death involved

My original query took, for each import, the next death event in the chain. When a live credential is replaced, that query walks past the replacement and attributes the next credential's death to the earlier seal — inflating that interval.

The check, per interval

37 imports  ->  35 DEATH   ·   1 CENSORED   ·   1 OPEN (the live record)

One censored interval: seq 35, sealed 2026-08-09 09:12:22, replaced 1.46h later while still alive. My original figure reported it as a 3.69h lifetime. Excluding it:

DEATHS ONLY
n=35   min 0.18h   p25 1.11h   median 2.37h   p75 3.38h   max 3.95h   mean 2.27h
under 1h:  8/35
over  4h:  0/35

Against the unfiltered numbers the median moves by about two minutes. Every conclusion in the comment above stands — no floor, practical ceiling ~4h, median ~2.4h — but they now rest on a population that has been classified rather than assumed.

The amended query, with the filter included

WITH imp AS (
  SELECT seq, ts_ms FROM audit_log
  WHERE credential_id = 'oauth:anthropic' AND op = 'import'
),
dth AS (
  SELECT ts_ms FROM audit_log
  WHERE credential_id = 'oauth:anthropic'
    AND op IN ('report_auth_failure','consumer_report','invalidate')
)
SELECT ROUND(((SELECT MIN(ts_ms) FROM dth d WHERE d.ts_ms > i.ts_ms) - i.ts_ms)/3600000.0, 2)
FROM imp i
WHERE (SELECT MIN(ts_ms) FROM dth d WHERE d.ts_ms > i.ts_ms) IS NOT NULL
  AND NOT ( (SELECT MIN(ts_ms) FROM imp n WHERE n.ts_ms > i.ts_ms) IS NOT NULL
        AND (SELECT MIN(ts_ms) FROM imp n WHERE n.ts_ms > i.ts_ms)
          < (SELECT MIN(ts_ms) FROM dth d WHERE d.ts_ms > i.ts_ms) )
ORDER BY 1;

One counter-intuitive result, since it affects how anyone else reads this data

The prediction going in was that the sub-one-hour intervals would be the contaminated ones — a credential dying eleven minutes after a seal sounds extraordinary, while an operator importing twice in eleven minutes sounds routine. All eight sub-hour intervals are genuine deaths.

The reason is structural rather than behavioural: censoring requires a replacement to land before the death, so contamination probability rises with interval length. Short intervals have the least room for an intervening import and are the least contaminable. The one censored row is among the longer ones.

Worth stating because the intuition points the wrong way, and because it means the ceiling claim is unaffected — removing an over-stated interval cannot raise a maximum.

Why post the amendment rather than quietly edit

The comment above replaced an invented range with a measured one. That is an upgrade in accuracy and also an upgrade in authority: a vague figure reads as a guess and gets treated as one, while n=36, median 2.41h reads as evidence and gets planned against. If the population had been wrong, the error would have been harder to dislodge than the hand-wave it replaced — the next reader would have had to redo the analysis rather than notice an obvious gap.

Precision raises the cost of the next correction, so the population deserved to be stated before the numbers were, not after.

[iceteaSA]

Second amendment: the causal reading of that distribution is wrong, and that matters more than the numbers

The distribution above is correct as measured. What it means is not what the comment implies, and the implication is the part a maintainer would plan against.

What the intervals actually measure

This credential dies by revocation-on-rotation: the upstream harness rewrites its token file with a fresh family, and that rewrite revokes the family the vault holds. The vault's copy does not expire on its own — it is killed from outside, on someone else's schedule.

So an import → death interval is not a token's lifespan. It is the gap until the next upstream rotation, with vault mortality acting as a proxy for it.

median 2.37h   ->  "the harness rotates roughly every 2.4h"
               NOT "tokens live roughly 2.4h"

The numbers are unchanged. Every operational conclusion in the previous comment still holds — no floor, arm from the moment of any re-seal, a credential past ~4h is unusual. But the reason they hold has moved, and that changes what should be built on them:

• A change in the harness's rotation cadence moves this entire distribution, with no change in the vault, the credential, or anything in this repository.
• The distribution is therefore not a property of this system and should not be treated as a stable characteristic of it. It is a measurement of an external actor's current behaviour.
• Anything sized against it — a monitoring interval, a retry budget, a runbook expectation — inherits that dependency and should say so.

How it surfaced

An observation window opened tonight where the credential passed the median with the record intact, and both this seat and the consuming module had instruments armed to report "survival past the recorded maximum" as a finding.

Checking the kill mechanism before trusting that framing:

last upstream rotation   2026-08-24T15:33:58Z
current seal             2026-08-24T15:42:12Z   (8 min after that rotation)
elapsed with no rotation 3.17h

Nothing has rotated since the seal. The credential is not surviving adversity; the mechanism that kills it has not fired once this cycle. A crossing would have meant "the harness has gone unusually long without rotating" — an ordinary schedule fact — and both instruments would have reported it in language suggesting the token had withstood something.

Both probes now carry the rotation state alongside the crossing, so the alert arrives with its interpretation attached:

rotated since seal + still alive  ->  the token survived a rotation   (would contradict the invalid_grant result; major)
no rotation + still alive         ->  a quiet rotation period          (ordinary)

The general point, since it is the reusable part

An alert that cannot distinguish its own two interpretations is half an alert. Both instruments were correct, tested, and would have fired accurately — and the reading would still have been wrong, because the discriminating fact lived outside everything either of them was watching.

Worth stating plainly for anyone reading this distribution later: untested is not the same as proven. A credential that has not been attacked has demonstrated nothing about its durability, and an instrument measuring elapsed time cannot tell those apart.

[iceteaSA]

Correction: that distribution measured OUR re-seal latency, not the harness's cadence. Do not use those numbers.

The interval series I posted earlier (n=35, median 2.37h, max 3.95h, "no floor — be armed from any seal") is wrong about what it measures, and the arithmetic is fine — which is what makes it dangerous. The numbers are correct; the quantity is not.

The decomposition

Those intervals were seal → verdict. Against the rotation that actually causes the death:

seal(N)      = R_N       + L_N          rotation N, plus our re-seal lag
verdict(N)   = R_(N+1)   + 301s         the NEXT rotation kills it; verdict 301s later
measured(N)  = I_N + 301s - L_N

L ≥ 301s for every instance without exception — a death cannot be diagnosed before the 301s NonTransient backoff verdict, so no re-seal can precede it. Therefore measured ≤ I always: the series systematically understates, by our own response time.

What the real intervals are

Recovered from consumer-report timestamps in the audit chain, which cancel L entirely (report(N) = R(N+1) + d, where d ≤ one 60s poll cycle). 25 death episodes, 2026-08-14 → 2026-08-24:

  4.01h  16.01h   8.33h   8.00h   4.00h   8.01h   4.00h  24.02h
 16.07h   4.00h   4.01h   8.00h  12.00h   4.00h   8.00h  12.00h
  4.01h   8.01h  28.00h   4.01h   3.99h  40.05h   3.99h   4.01h

Every interval is an integer multiple of 4.00h. 23 of 24 land within ±4 minutes of exact; worst residual 19.8 min. Rotation phase is stable at 24–27 minutes past the hour, drifting ~50 s/day.

The larger values are missed rotations, not long intervals: once a credential is latched and not yet re-sealed, the next rotation kills nothing and produces no report. Every "16h interval" is four rotations, three of which landed on an already-dead credential.

So the harness rotates on a 4.00-hour cron

There is no distribution, no tail, and no "no floor" property. The apparent spread in the old series was entirely our own latency, inverted: a fast re-seal produced a long apparent lifetime, a slow one a short one. The headline "0.18h shortest lifetime ever" was a credential sealed 3.8 hours after its rotation — the slowest response in the series, recorded as the most fragile token.

Method note, for anyone re-deriving this

Three timestamp lanes witness the same event and they are not interchangeable:

lane	contamination	usable?
import (seal)	carries L, our response time, unbounded	no
consumer_report	carries d ≤ one poll cycle	yes, ±60s
refresh_failed / invalidate verdict	carries a constant 301s that cancels in deltas	yes, exact

auth.json mtime holds only the most recent rotation, so per-instance lag cannot be recovered from the filesystem — but it does not need to be. The consumer's own reports were in the chain the whole time.

Operational consequence

The death clock starts at the last rotation, not at the vault import. A seal that lags its rotation by 88 minutes has 88 minutes less runway than seal-age suggests — and the next death is predictable rather than probabilistic: last-rotation mtime + 4.00h, consumer 401 within one poll cycle, vault verdict ~301s later.