Git delivery candidate cap is consumed by deleted paths and duplicate worktrees

[nichenke]

🤖 Agent-filed: investigated and written by Claude Code on behalf of @nichenke. Every code
citation was verified against 1a1b35a, and every count was measured on the reporter's machine.
Local paths and repository names are redacted.

git_delivery_candidates() selects the first 50 session project paths with no existence check and
no worktree resolution. On a machine with a long session history, that budget is largely spent on
paths that no longer exist and on multiple worktrees of the same repository, displacing live
repositories from the scan.

Mechanism

1. git_delivery_candidates() (token_meter/app.py:6926) walks session sources, dedupes by
   os.path.abspath, and breaks at a hardcoded 50 (:6940). It performs no filesystem check —
   delivery_project_label() only hashes the path.

2. A deleted root is not detected until _scan_candidate() (services/git_delivery.py:499) has
   already spawned git -C <root> rev-parse --show-toplevel (:503), which fails and returns
   repository_unavailable (:506).

3. The repo_key coalescing at :514 does not catch linked worktrees. repo_key is
   hash(rev-parse --show-toplevel) (:508), and in a linked worktree --show-toplevel returns
   that worktree's own root, not the main repository. Verified on this machine (paths redacted):

   worktree  <repo>/.worktrees/<branch>
     --show-toplevel   -> <repo>/.worktrees/<branch>
     --git-common-dir  -> <repo>/.git
   main repo <repo>
     --show-toplevel   -> <repo>
   

   Both report the same 12 refs/remotes, because remote-tracking refs live in the common dir. So
   each worktree of a repository re-reads the same shared reflogs (:539) under a distinct
   repo_key.

Impact

Measured on the reporter's machine — 152 Claude project directories and 181 Codex session files:

Session project paths (distinct cwd)	170
— path no longer exists	99
— live git path	62
— not a repository	9
Distinct underlying repositories (--git-common-dir)	41
Redundant paths (live paths − repositories)	21

Worst duplication: two repositories contribute 6 candidate paths each and a third contributes 4.

The live server is at the cap and reports both symptoms:

"coverage": {"repositories": 50, "measured": 11, "partial": 46,
             "codes": ["no_push_history", "no_remote_tracking_refs", "ready",
                       "repository_unavailable", "scan_limited"]}
"sources": 1188

Only 41 repositories exist in this machine's entire session history, so the 50-repository cap is not
protecting against repository count — it is being filled by stale session history and worktree
duplicates, and scan_limited is the result.

Per-pass git subprocess cost over the full live population, as an upper bound before
MAX_COMMITS_PER_SCAN truncates:

	reflog spawns
As-is	1895
Worktrees coalesced by common dir	1448
Redundant	447

Plus three fixed spawns per live candidate (rev-parse, config --get user.email, for-each-ref)
and one wasted rev-parse per deleted path.

Not instrumented: the actual mix of paths in any single 50-candidate pass, which depends on source
ordering. Measuring it would have written to the live delivery database, so the numbers above are
full-population bounds rather than an observed pass.

Secondary: the repository-limit signal is dead

repositories_limited = len(candidate_rows) > MAX_REPOSITORIES (services/git_delivery.py:626)
can never be true, because git_delivery_candidates() already truncated to 50 upstream. The
repository_limit code (:642) and its partial bump (:643) are therefore unreachable.
Reproduced against 1a1b35a:

sources: 120  candidates returned: 50  MAX_REPOSITORIES: 50
repositories_limited would be: False

The hardcoded 50 in app.py:6940 also duplicates MAX_REPOSITORIES in
services/git_delivery.py:17, so the two can drift.

Suggested fix

In git_delivery_candidates(), before applying the cap:

1. Drop roots that do not exist (os.path.isdir).
2. Key the dedupe on the repository rather than the working tree, so a repository's worktrees
   occupy one slot. git rev-parse --path-format=absolute --git-common-dir gives this, at the cost
   of one subprocess per surviving candidate — cheaper than the full reflog pass each duplicate
   currently triggers.
3. Import MAX_REPOSITORIES instead of the literal 50, and raise the limit signal where the
   truncation happens so repository_limit becomes reachable.

Design note on (1): the loop currently breaks as soon as it has 50 candidates. With an existence
filter it may walk the full source list — 1188 entries on this machine — so the cost becomes ~1188
isdir calls per pass. That is negligible against a single git spawn, but it is a change in
iteration shape worth stating.

Expected effect on this machine: 170 candidate paths reduce to 41, comfortably under the cap, and
scan_limited should stop appearing.

Related

• Git delivery watcher ignores its 5-minute interval and scans continuously #30 — the watcher's interval floor. Frequency, not pass composition; the two are independent.
• Fix Git delivery watcher interval floor #31 — fixes Git delivery watcher ignores its 5-minute interval and scans continuously #30. This report applies equally before and after it.

Environment

• Revision 1a1b35a, macOS 26.6.2 arm64, Python 3.9.6 (system Python, via the server LaunchAgent)

[bhavsarpratik]

@nichenke Thanks for the detailed measurements and for filing this. I pushed a fix in #46.

It removes deleted/non-Git paths before applying the cap, deduplicates repeated cwd entries before invoking Git, coalesces linked worktrees, preserves existing ledger evidence, and keeps Git discovery off dashboard reads.

Fresh clone:

git clone --branch codex/fix-git-delivery-candidates --single-branch https://github.com/splunk/token-meter.git token-meter-issue-33
cd token-meter-issue-33

Existing checkout, using a separate worktree:

git fetch origin codex/fix-git-delivery-candidates
git worktree add --detach ../token-meter-issue-33 origin/codex/fix-git-delivery-candidates
cd ../token-meter-issue-33

Then install and check:

./scripts/install
curl -fsS http://127.0.0.1:8722/health
curl -fsS http://127.0.0.1:8722/menubar
curl -fsS 'http://127.0.0.1:8722/git-delivery?range=7'

No sudo is needed. To restore your normal version afterward, rerun ./scripts/install from your usual Token Meter checkout.

If possible, please repeat the same seven-minute workload and share delivery-scan count, total Git subprocesses, CPU usage, repository coverage, and whether deleted paths or linked worktrees still crowd out live repositories.