Skip to content

fix(daemon): retry a mismatched cohort holder until the caller's deadline - #2047

Merged
DeusData merged 1 commit into
mainfrom
fix/2046-cohort-conflict-retry-until-deadline
Sep 4, 2026
Merged

fix(daemon): retry a mismatched cohort holder until the caller's deadline#2047
DeusData merged 1 commit into
mainfrom
fix/2046-cohort-conflict-retry-until-deadline

Conversation

@DeusData

@DeusData DeusData commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Fixes #2046.

Root cause

Every cohort participant — daemon, bootstrap, local CLI — holds the cohort lifetime file lock SH from admission until its lease is released at exit. The coordinator moves an internal daemon to STOPPING at client_count == 0 && !permanent, but the daemon keeps that SH lock through its teardown (~0.3–0.6 s on this machine). A local CLI arriving inside that window takes the SH lock, reads the holder record, and cbm_daemon_hello_compare reports CBM_DAEMON_HELLO_CACHE_CONFLICT when its CBM_CACHE_DIR differs — and cbm_version_cohort_acquire returned CBM_VERSION_COHORT_CONFLICT at once, ignoring deadline_ms entirely.

Main's slow scalar self-hash happened to push the CLI's admission past the window; #1767's CommonCrypto hash on macOS removes that accidental margin and the pr-smoke leg trips on config get under a fresh cache directory.

Fix

cbm_version_cohort_acquire retries a CONFLICT until the caller's finite deadline_ms, releasing every guard between attempts (the single-attempt body is now version_cohort_acquire_once). This is the same wait host.c already performs for the daemon claim marker (HOST_DAEMON_CLAIM_TIMEOUT_MS). All production callers pass a finite deadline (MAIN_STARTUP_TIMEOUT_MS, HOST_INITIAL_CLIENT_TIMEOUT_MS, bootstrap startup_timeout_ms); UINT64_MAX keeps failing fast, so an indefinite waiter is never parked behind a genuine long-lived peer. One version_cohort.conflict_retry info line is logged per acquire (reason + active build fingerprint).

Behaviour change worth stating plainly: a genuine conflict against a peer that stays now surfaces after the caller's deadline (≤10 s) instead of immediately. The existing conflict tests keep passing because they hand in an already-expired deadline.

Tests

tests/test_version_cohort.c, both deterministic — they assert stable states, never a transient window:

  • version_cohort_conflict_is_retried_until_the_deadline — mismatched live holder; the requester returns CONFLICT no earlier than its deadline with the retry counter raised and the original conflict payload intact; a UINT64_MAX requester still returns immediately with the counter unchanged.
  • version_cohort_conflict_waiter_is_admitted_when_the_holder_leaves — a waiter thread meets the holder (proven via the retry seam), is still waiting, the holder releases, the waiter is admitted with CBM_VERSION_COHORT_OK.

The retry counter is a CBM_ENABLE_TEST_SEAMS-only atomic; it compiles to nothing in production.

Verification on the branch:

  • version_cohort suite 16/16; revert-check with the retry disabled → exactly the two new tests fail (:325 returned_at >= deadline, :391 met_holder).
  • Widened daemon version_cohort daemon_version daemon_runtime daemon_application daemon_frontend daemon_bootstrap daemon_ipc → 224 passed.
  • End-to-end on fix(hook-augment): cache the build fingerprint so the startup deadline is met #1767's head + this fix: the failing config get sequence passes 3/3 with version_cohort.conflict_retry reason=cache_root followed by version_cohort.claimed_fresh (same sequence failed 2/2 without the fix).
  • make -f Makefile.cbm lint-ci clean.

Follow-up (not in this PR)

The daemon could release its cohort lease at the start of STOPPING rather than at the end of teardown, which would shrink the window this PR waits out. Noted in #2046.

…line

A participant admitted to the version cohort holds the cohort lifetime lock
SH until its lease is released at exit. An internal daemon that has just
lost its last stdio client keeps that lock through its teardown, so a local
CLI (`config get`, `index_status`, ...) arriving in that few-hundred-ms
window met a holder whose cache root differed and was refused immediately
with "active account daemon uses a different cache directory" — a pure
lifecycle race, previously masked by the slow scalar self-hash and exposed
as soon as the hash got fast (#1767 pr-smoke on macOS).

cbm_version_cohort_acquire now retries a CONFLICT until the caller's finite
deadline_ms, holding no guard between attempts, exactly as host.c already
waits out the same handoff for the daemon claim marker. Every production
caller passes a finite deadline; UINT64_MAX keeps failing fast so a genuine
conflict against a long-lived peer is never waited on indefinitely.

Tests (deterministic, no timing assertions on transient windows): one proves
the retry runs until the deadline and a UINT64_MAX caller still fails
immediately; one proves a waiter that met the mismatched holder is admitted
the moment the holder releases. Both fail with the retry removed.

Fixes #2046

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local CLI refused with a cache-directory conflict while the internal daemon is still draining after its last client

1 participant