fix(daemon): retry a mismatched cohort holder until the caller's deadline - #2047
Merged
Merged
Conversation
…line A participant admitted to the version cohort holds the cohort lifetime lock SH until its lease is released at exit. An internal daemon that has just lost its last stdio client keeps that lock through its teardown, so a local CLI (`config get`, `index_status`, ...) arriving in that few-hundred-ms window met a holder whose cache root differed and was refused immediately with "active account daemon uses a different cache directory" — a pure lifecycle race, previously masked by the slow scalar self-hash and exposed as soon as the hash got fast (#1767 pr-smoke on macOS). cbm_version_cohort_acquire now retries a CONFLICT until the caller's finite deadline_ms, holding no guard between attempts, exactly as host.c already waits out the same handoff for the daemon claim marker. Every production caller passes a finite deadline; UINT64_MAX keeps failing fast so a genuine conflict against a long-lived peer is never waited on indefinitely. Tests (deterministic, no timing assertions on transient windows): one proves the retry runs until the deadline and a UINT64_MAX caller still fails immediately; one proves a waiter that met the mismatched holder is admitted the moment the holder releases. Both fail with the retry removed. Fixes #2046 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2046.
Root cause
Every cohort participant — daemon, bootstrap, local CLI — holds the cohort lifetime file lock SH from admission until its lease is released at exit. The coordinator moves an internal daemon to STOPPING at
client_count == 0 && !permanent, but the daemon keeps that SH lock through its teardown (~0.3–0.6 s on this machine). A local CLI arriving inside that window takes the SH lock, reads the holder record, andcbm_daemon_hello_comparereportsCBM_DAEMON_HELLO_CACHE_CONFLICTwhen itsCBM_CACHE_DIRdiffers — andcbm_version_cohort_acquirereturnedCBM_VERSION_COHORT_CONFLICTat once, ignoringdeadline_msentirely.Main's slow scalar self-hash happened to push the CLI's admission past the window; #1767's CommonCrypto hash on macOS removes that accidental margin and the pr-smoke leg trips on
config getunder a fresh cache directory.Fix
cbm_version_cohort_acquireretries a CONFLICT until the caller's finitedeadline_ms, releasing every guard between attempts (the single-attempt body is nowversion_cohort_acquire_once). This is the same waithost.calready performs for the daemon claim marker (HOST_DAEMON_CLAIM_TIMEOUT_MS). All production callers pass a finite deadline (MAIN_STARTUP_TIMEOUT_MS,HOST_INITIAL_CLIENT_TIMEOUT_MS, bootstrapstartup_timeout_ms);UINT64_MAXkeeps failing fast, so an indefinite waiter is never parked behind a genuine long-lived peer. Oneversion_cohort.conflict_retryinfo line is logged per acquire (reason + active build fingerprint).Behaviour change worth stating plainly: a genuine conflict against a peer that stays now surfaces after the caller's deadline (≤10 s) instead of immediately. The existing conflict tests keep passing because they hand in an already-expired deadline.
Tests
tests/test_version_cohort.c, both deterministic — they assert stable states, never a transient window:version_cohort_conflict_is_retried_until_the_deadline— mismatched live holder; the requester returns CONFLICT no earlier than its deadline with the retry counter raised and the original conflict payload intact; aUINT64_MAXrequester still returns immediately with the counter unchanged.version_cohort_conflict_waiter_is_admitted_when_the_holder_leaves— a waiter thread meets the holder (proven via the retry seam), is still waiting, the holder releases, the waiter is admitted withCBM_VERSION_COHORT_OK.The retry counter is a
CBM_ENABLE_TEST_SEAMS-only atomic; it compiles to nothing in production.Verification on the branch:
version_cohortsuite 16/16; revert-check with the retry disabled → exactly the two new tests fail (:325 returned_at >= deadline,:391 met_holder).daemon version_cohort daemon_version daemon_runtime daemon_application daemon_frontend daemon_bootstrap daemon_ipc→ 224 passed.config getsequence passes 3/3 withversion_cohort.conflict_retry reason=cache_rootfollowed byversion_cohort.claimed_fresh(same sequence failed 2/2 without the fix).make -f Makefile.cbm lint-ciclean.Follow-up (not in this PR)
The daemon could release its cohort lease at the start of STOPPING rather than at the end of teardown, which would shrink the window this PR waits out. Noted in #2046.