Local CLI refused with a cache-directory conflict while the internal daemon is still draining after its last client

[DeusData]

Symptom

A local CLI command (config get …, or any CBM_DAEMON_PROCESS_LOCAL_CLI role) that starts within ~0.3–0.6 s after the last stdio MCP client of an internal daemon (--cbm-daemon-internal) has exited, and runs under a different CBM_CACHE_DIR, is refused:

CBM could not start because the active account daemon uses a different cache directory
(active cache …; requested cache …). Close all CBM sessions and commands, then retry with
one consistent CBM_CACHE_DIR.

There is no conflicting session at that point — the daemon is already draining and exits a few hundred milliseconds later. The refusal is a race, not a policy outcome.

Repro (macOS, main)

scripts/smoke-test.sh Phase 3z2 → 3z3 is exactly this sequence: 3z2 drives an MCP stdio session (initialize + 24 list_projects), which auto-starts an internal daemon; 3z3 immediately runs CBM_CACHE_DIR=<fresh tmp> $BINARY config get auto_watch. With a shared CBM_RUNTIME_DIR so both phases meet at the same endpoint:

• main: passes — but only because the local CLI's self-fingerprint (cbm_index_supervisor_capture_build_fingerprint, scalar SHA-256 of the whole binary) takes ~1 s locally and 3–4 s on a macos-14 runner, which is longer than the daemon's linger. The pass is an accident of startup cost.
• With a fast self-hash (PR fix(hook-augment): cache the build fingerprint so the startup deadline is met #1767 switches Apple to CommonCrypto, ~0.1 s), the CLI arrives inside the window and is refused 2/2 locally and on the runner (pr-smoke (macos-14), run 33799501343). Inserting a pause between the phases makes it pass again, which confirms the mechanism.
• Polling the daemon after client exit: --cbm-daemon-internal lingers 0.57 s (main) / 0.31 s (fix(hook-augment): cache the build fingerprint so the startup deadline is met #1767 build) after its last client disconnects.

Root cause

Every participant — daemon, bootstrap, local CLI — holds the version-cohort lifetime file lock (SH) until it releases its lease at process exit. cbm_version_cohort_acquire() (src/daemon/version_cohort.c) takes maintenance SH → admission EX → probes lifetime: if a holder is present it reads the holder's identity record; an exact build with a different cache_fingerprint is CBM_DAEMON_HELLO_CACHE_CONFLICT, after which the lifetime-EX try_acquire fails and the call returns CBM_VERSION_COHORT_CONFLICT immediately — even though the caller supplied a finite deadline_ms (10 s in every production path) that it never uses for this case.

A draining daemon (coordinator STOPPING, zero clients, no longer admitting) still holds that SH lock during its teardown. host.c already recognises the same class of transient for the daemon claim marker (HOST_DAEMON_CLAIM_TIMEOUT_MS: "a just-superseded daemon generation can still hold the cohort claim … wait out that handoff rather than failing startup on a transient BUSY"). Admission conflicts never got the same treatment.

Fix

cbm_version_cohort_acquire() retries a mismatched live holder (all conflict kinds: version / build / ABI / cache) until the caller's deadline_ms, releasing every guard between attempts so compatible participants and activations are never blocked behind the wait. On deadline it returns the same CONFLICT with conflict_out populated as today; an indefinite deadline (UINT64_MAX) keeps the current fail-fast behaviour, since a mismatched holder may never leave. The existing conflict tests already pass an expired deadline (cbm_now_ms()), so the contract change is additive.

Consequence for a genuine conflict (a long-lived daemon with another cache directory or build): the refusal arrives after the startup budget instead of instantly, and succeeds if the user closes the other session in the meantime — the message already tells them to do exactly that.

Follow-up worth considering separately: the daemon could release its cohort lease at the start of STOPPING rather than at the end of teardown, shrinking the window it holds the lock while admitting nobody.

[DeusData]

Fixed on main by #2047 — merge commit 13b7d067018c (head 9104feb67f78).

Root cause was a pure lifecycle race: an internal daemon that has just lost its last stdio client keeps the cohort lifetime lock (SH) through its teardown, so a local CLI (config get, index_status, …) arriving in that few-hundred-ms window met a holder whose cache root differed and was refused immediately with "active account daemon uses a different cache directory". It had been masked by the slow scalar self-hash and surfaced as soon as the hash got fast (#1767's pr-smoke on macOS).

cbm_version_cohort_acquire now retries a CONFLICT until the caller's finite deadline_ms, holding no guard between attempts — the same wait host.c already does for the daemon claim marker. Every production caller passes a finite deadline; UINT64_MAX keeps failing fast so a genuine conflict against a long-lived peer is never waited on indefinitely. Two deterministic tests (retry-until-deadline + waiter admitted the moment the holder releases) both fail with the retry removed.