Admin CLI vs running module: single-writer lease makes reads and mint-handle a consumer outage

[iceteaSA]

Operator-flow question from today's provisioning (same box as #1's runbook notes).

Observed

With the claustrum module running under the daemon, every admin verb fails with:

error: the credentials daemon is running (holds the single-writer lease); stop it before running an admin command

Correct and fail-closed for writes — but it also covers pure reads (list, status, audit) and mint-handle, making routine operations a consumer outage:

• Checking what's in the vault requires stopping the module that insula is actively consuming credentials through.
• Minting a handle for a newly sealed credential (a metadata write, not a payload write) requires the same downtime.

The runbook's bootstrap flow assumes a stopped daemon, which is fine for day-0; day-N operations are where this bites.

Asks

1. Is the intended day-N flow "stop module → admin → start module"? If so, a runbook section saying that (with the consumer-impact caveat) would settle it — we'll add it to our ops notes.
2. Enhancement, if welcome: route admin verbs through the running module (it holds the lease and already speaks the ops on its management surface), falling back to direct-store access when the daemon is down. Even read-only routing (list/status/audit) would remove most downtime cases; mint-handle through the module would remove the rest.

No urgency — we're through provisioning; this is for the next box (a macOS work machine) where we'd rather not schedule vault downtime around insula.

[ualtinok]

Fixed, and the root cause was not where either of us was looking. Thanks for the report — the detail about which box and which flow is what made it findable.

Your premise was right and your diagnosis was half right. The enhancement you asked for already exists: --subc <connection-file> routes admin verbs through the running module, and list / status / events / usable are lease-free reads that work against a live daemon with no flag at all. So the intended day-N flow is not "stop module → admin → start module". It is "just run the verb".

Why you hit a wall anyway: auto-discovery had a hole. subc publishes its connection file to $XDG_RUNTIME_DIR/subc-connection.json, and falls back to <temp>/subc-<user-token>.connection.json when that variable is unset. The ck dispatcher checks three locations; ck-auth was checking two, missing the temp-dir fallback.

Stock macOS sets no XDG_RUNTIME_DIR, so on the macOS box you mentioned that fallback is the default arrangement rather than an edge case. Standard Linux sets /run/user/<uid>, which is why your Linux box got as far as it did.

The failure mode is the one worth naming: discovery returns nothing, the CLI concludes no daemon is running, takes the offline path, hits the single-writer lease, and prescribes stopping the daemon — the one remedy an operator should not use, while never mentioning the route that would have worked.

Fixed in 3c7b83e, with a cross-platform correction in 73ede12.

Two details in the fix that may matter if you read it:

• It globs for subc-*.connection.json rather than recomputing the filename. The token comes from user_connection_token(), whose unix arm derives a uid by writing a probe file — side-effecting, several fallbacks deep, and silently wrong if reimplemented slightly differently.
• Ambiguity refuses rather than guesses. Two connection files in a shared temp dir belong to different OS users; picking one could point an admin op at another user's daemon. It names the candidates and tells you to pass --subc.

One thing that has not changed: rotate-master-key and bootstrap remain offline-only and will still refuse while the daemon runs. That refusal names itself as having no route path, so it should not send you looking for a --subc that cannot help.

[ualtinok]

Follow-up, because I found a second cause of the same symptom and would rather you hear it from me than hit it.

The fix I reported covered discovery missing the temp-directory location. It did not cover discovery failing there. The last arm read the directory with .ok()?, so a permission error or an odd TMPDIR was discarded and returned as the same "nothing found" an empty directory produces — and the caller cannot tell those apart, because both arrive as None.

The result is the exact behaviour you reported: the CLI concludes no daemon is running, takes the offline path, hits the single-writer lease, and tells you to stop the daemon. The one action that makes it worse, recommended by the tool, for a second distinct reason.

Now fixed and deployed. The distinction lives in the return type rather than in a log line — Found / NotPresent / Unreadable / Ambiguous — with each mapped to its own advice. Verified against the placed binary rather than the source:

note: could not read /var/folders/…/T/tmp.vPXgWvHOnJ while looking for a running
daemon (Permission denied (os error 13)). Proceeding as if none is running; if
one IS running, pass --subc <connection-file> rather than stopping it.

Two things worth recording from the fix, since the first one nearly shipped:

My first attempt emitted that note and added a test asserting the outcome, with a comment claiming the mutation would catch a regression. It would not have — both arms returned None, the test asserted None, and restoring the original .ok()? left every test green. A negative test plus a false claim that it is mutation-proof is worse than no test, because the claim stops the next reader looking. The difference existed only on stderr, and a difference only visible on stderr is one no test can hold. That is why the outcome became a type.

And three end-to-end probes read clean before I noticed why: auto-discovery is skipped whenever --data-dir is passed explicitly, and every probe pinned a data directory to avoid touching a real vault. The gate was mine and I had forgotten it. If you ever try to reproduce discovery behaviour, leave --data-dir defaulted or you are testing a path that never runs.

Leaving this open until you have had a chance to hit it or not on your box — you found the first cause from a real deployment, and that is a better signal than my probes.

[iceteaSA]

Hit the path from a real deployment last night — before reading this follow-up, which makes it a fair datum: the first-cause fix verifies live on Linux; the second cause we cannot hit on this box, and here's why that's expected rather than absence-of-evidence.

Real-deployment verification of discovery (Found arm): a rotation-staled oauth:anthropic was re-sealed at ~23:00Z against the running module — ck-auth import --source opencode ... --replace --key-path /etc/cortexkit/master.key, no --data-dir, so per your gate note this exercised the real auto-discovery path, not the pinned-directory bypass. Discovery found the daemon via the standard $XDG_RUNTIME_DIR/subc-connection.json (stock Linux, /run/user/1000), the import committed connected, zero downtime, module health ok 5/5 after. That's the intended day-N flow working end-to-end on the box that filed this issue.

The Unreadable arm is structurally unreachable here in normal operation: this box always has XDG_RUNTIME_DIR set, so the temp-dir fallback arm — where your .ok()? collapse lived — never runs on the happy path. We can't give you a natural reproduction of the second cause; a contrived one (sabotaged TMPDIR perms with XDG_RUNTIME_DIR unset) would only re-prove what your placed-binary probe already showed. Not claiming the arm correct from here — claiming it unreachable on this host's configuration, which is the honest scope of what this box can attest.

One wrinkle from the same live run, adjacent rather than contradicting: the connected path without --key-path failed with cannot resolve the master key to authorize the op: keychain command failed: No such file or directory — the key-resolution fallback chain reaches for a macOS keychain binary on Linux and dies with an error naming a keychain problem when the actual remedy is "pass --key-path". Same shape as the discovery defect you just fixed: a platform-absent mechanism rendering as a confusing failure rather than a typed "no keychain on this platform, pass --key-path" hint. Filing separately if you'd rather track it — flagging here first since it's the same error-classification family and the same CLI.

From our side this issue can close on your call: both discovery causes fixed and deployed, the first verified from a real deployment, the second covered by your placed-binary probe on the platform where it's reachable.