perf: walker boundaries, batched worker dispatch, hot-path memos (+ deadcode fix) - #16
Merged
Merged
Conversation
The walker descended into subdirectories carrying their own `.git` (linked worktrees under .claude/worktrees/, vendored clones, submodules), so a repo with four worktrees indexed five copies of itself: 14 661 files instead of 2 924 on a real project, 4.0 s instead of ~0.9 s cold, a 68 MB cache instead of 13 MB. It also ignored `.git/info/exclude`, and `--ignore-dir` — which REPLACES the default list — pulled `.git/objects` into the index whenever the caller forgot to list `.git`. - A non-root directory whose listing contains `.git` (dir or gitfile) is skipped on the dirents already read, zero extra syscalls, and counted once in `excluded`. Structural: independent of `gitignore: false`. - `.git/info/exclude` is parsed ahead of the .gitignore chain (git's own precedence), following a gitfile and a worktree's `commondir`. - `.git` is ignored whatever `ignoreDirs` says, and a root gitfile is no longer indexed as a source file. Claude-Session: https://claude.ai/code/session_01YDMVcJNo7WZh9xFAGWs2F8
When several files export the same name, the caller index keeps the first-sorted definition under the bare name and the others under "name@file". findDeadCode consulted the bare key first, got the other file's entry, failed its own file check and reported a symbol that IS called as dead. Query "name@file" first, then fall back to the bare key. Claude-Session: https://claude.ai/code/session_01YDMVcJNo7WZh9xFAGWs2F8
…log n) lineAt
Single-threaded profile on a 2 900-file repo: blankComments was 5 % of
extraction — `split("")`/`join("")` allocated one string per byte of
every JS/TS file. Rewritten as one charCodeAt pass that copies kept
stretches as slices and comments as space runs (newlines inside block
comments kept, so offsets and lines survive). Verified byte-identical to
the previous implementation on 3 240 files plus edge cases.
collectAll's visitor ran five end-anchored regexes on every node's type;
a grammar has a few hundred types, so the verdicts are now memoized per
type as a bit set, and `startPosition` (a wasm marshal) is read only in
the branch that needs it.
extractReexports' lineAt re-split the whole prefix per re-exported
name; line starts are now computed once per file and binary-searched.
graph.json/symbols.json are byte-identical before/after on the fixture
repo and on a real 2 900-file repo; EXTRACTOR_VERSION untouched.
Claude-Session: https://claude.ai/code/session_01YDMVcJNo7WZh9xFAGWs2F8
…ommands Workers used to receive one static shard each. On a heterogeneous CPU (Apple's 4 performance + 6 efficiency cores) or a loaded machine the slowest shard sets the wall-clock: measured on a 2 358-file repo, eight equal shards ran at ~1 ms/file where one worker alone managed 0.45, and eight workers barely beat four. The main thread now hands out batches on demand from the path-sorted queue — sized remaining/(workers×4), never below 4 — with two batches in flight per worker so it never idles on the main thread's turn to deserialize and answer. Faster cores simply take more of the queue; the tail is one small batch. Records stay keyed by path and scanRepo orders them, so which worker built one never shows. Extraction wall-clock: ~355 → ~340 ms on an idle machine, larger under load; the byte-identity gate (--workers 0 vs 4) is unchanged. A worker's readiness message (an empty first batch) carries the grammar set it got ready; a mismatch, an error payload or a per-batch timeout still discards the whole run — and now terminates the other workers — before the sequential fallback. preloadSessionLazy scans through scanRepoParallel: with a persisted index whose code files drifted, a read command re-extracted every changed file on the main thread, so --workers / CODEINDEX_WORKERS meant nothing as soon as .codeindex/ existed. An unchanged index still loads no wasm and spawns nothing. betweennessOf allocates its per-source scratch once (typed arrays) and resets only the nodes the BFS visited, so a source costs its component rather than the whole graph — an import graph is mostly small ones. Same accumulation order, bit-identical scores. Not done from the plan, on measurement: the main thread's own grammar load costs 7–10 ms and worker boot ~35 ms on this machine, so skipping the former and overlapping the latter with the walk were not worth the extra tier-identity invariants. Claude-Session: https://claude.ai/code/session_01YDMVcJNo7WZh9xFAGWs2F8
Serial post-extraction work (buildArtifactsFromScan) on a 2 900-file repo: ~145 → ~122 ms steady state, all outputs byte-identical (graph.json, symbols.json, and 16 470 impactOf/neighborsOf/findReferences/literals/ deadcode answers compared against the previous bundle). - resolve.ts: JS/TS resolutions memoized per (importing dir, spec) in the ResolveContext — resolveJs only ever looks at the importer's directory, and a package's files import the same specifiers over and over; each hit skips ~20 normalize+probe rounds. Go/Java "first file in the directory" lists are built once per (ext, dir) instead of filter+sort per import. Callers get a copy of the cached record. - literals.ts: the holder filters that depend only on the symbol (kind, function-valued signature, span) run once per file rather than once per (literal × symbol); same order, same innermost-wins tie-break. - derived.ts: docs are tokenized once per scan (docMentionsFor) and shared by buildGraph's mention pass and computeSymbolRefs; the memo records whether the text was retained docText or re-read from disk so each consumer keeps its previous eligibility rule. identSetsFor gives findReferences a Set per file instead of Array.includes per query, and its doc RegExp is compiled once per query, not once per doc. - traverse.ts: dependents/adjacency maps (with pre-sorted neighbour lists) are built once per edge array and kept in a WeakMap, instead of rebuilt and re-sorted on every impactOf/neighborsOf — delta.ts calls impactOf once per changed file of every module. Left as-is on measurement: relations.ts's typeDefs/resolveRelations recompute (under 2 ms) and callers.ts's enclosingAmong (not on the index path). Claude-Session: https://claude.ai/code/session_01YDMVcJNo7WZh9xFAGWs2F8
github-actions Bot
pushed a commit
that referenced
this pull request
Sep 2, 2026
## [2.28.2](v2.28.1...v2.28.2) (2026-09-02) ### Performance Improvements * walker boundaries, batched worker dispatch, hot-path memos ([#16](#16)) ([39aa752](39aa752))
Contributor
|
🎉 This PR is included in version 2.28.2 🎉 The release is available on: Your semantic-release bot 📦🚀 |
maxgfr
added a commit
that referenced
this pull request
Sep 3, 2026
#17) Two regressions introduced by #16, found by an independent audit and reproduced against the published 2.28.1 bundle before being fixed. - fix(walk): the repository boundary triggered on the NAME `.git`, so a file named `.git` holding anything else — a truncated write, an unrelated file, a dangling symlink — silently dropped its entire subtree from the index. A `.git` DIRECTORY is now a git dir, and a `.git` FILE is a marker only when it opens with exactly `gitdir: `, which is git's own rule (read_gitfile_gently compares the first 8 bytes) — verified against real git for the no-space, leading-space and second-line spellings. Symlinks resolve through their target; the read is capped at 4 KiB. Not validating the TARGET stays a deliberate deviation, now documented as one: a stale gitfile from a pruned worktree still sits on a full checkout. - fix(traverse): the adjacency cache keyed on the edge array's identity and length, so retargeting an edge IN PLACE kept answering with the pre-edit graph. A snapshot of the fields a traversal reads is now checked on every hit. Measuring that check changed the design: reverseClosure is back to its pre-PR uncached shape (200 impactOf calls: 69 ms pre-PR, 70 ms now), while bfs keeps the checked cache where it still pays (161 ms → 41 ms). graph.json/symbols.json unchanged. 19 075 traversal answers match the pre-PR bundle on a real repo, and another 19 075 match after mutating every 7th edge in place — the case that used to go stale. Claude-Session: https://claude.ai/code/session_01YDMVcJNo7WZh9xFAGWs2F8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance pass over the indexing pipeline plus two correctness fixes, with every artifact kept byte-identical (
EXTRACTOR_VERSIONuntouched).fix(walk)): stop at nested repositories (a subdirectory with its own.git, i.e. linked worktrees, vendored clones, submodules), honor.git/info/exclude(following gitfiles and a worktree'scommondir), and never walk.giteven when--ignore-dirreplaces the default list. On a real repo with four worktrees under.claude/worktrees/the index went from 14 661 phantom-laden files to the 2 924 git actually tracks.fix(deadcode)): a called export whose name is also exported by another file was reported dead, because the bare caller-index key was consulted before the qualifiedname@filekey.perf(extract)): one-passblankComments(wassplit("")/join("")per byte), node-type class memo in the AST visitor,O(log n)lineAtfor barrels.perf(pool)): batches dispatched on demand with two in flight per worker instead of one static shard each (robust to heterogeneous P/E cores and load);--workers/CODEINDEX_WORKERSnow apply to read commands with a persisted index; betweenness scratch allocated once and reset per visited node only.perf(graph)): JS resolution memo per (dir, spec), Go/Java per-directory lists, once-per-file holder filter in literals, docs tokenized once and shared by graph and symbols.json, adjacency memo inimpactOf/neighborsOf,Setidents plus one compiled regex infindReferences.Measurements
codeindex indexoncode-du-travail-numerique(Apple M5, 4P+6E, Node 24, min of 3):--workers 1--workers 4main(default walker)mainwith.claudeexcluded by handIn-process phase timings on the same 2 924 files (8 workers): parallel extraction 388 → ~335 ms,
buildArtifactsFromScan197 → ~157 ms cold (145 → 122 ms steady state).Verification
pnpm typecheck,pnpm test(1 280 passed, 50 skipped),pnpm run check:buildclean.graph.json/symbols.jsonbyte-identical before/after steps 2–5 ontests/fixtures/mini-repoand on the real repo, and identical at--workers 1vs 8.blankCommentsold vs new: identical on 3 240 files plus edge cases.impactOf/neighborsOf/findReferences/findLiteralDuplications/findDeadCodeold bundle vs new: 16 470 answers, 0 mismatches.git ls-filesminus the walker's own binary/font/lockfile filters (the 6 remaining differences are NFC/NFD spellings of the same paths).tests/walk-nested.test.ts(nested repo boundary,info/excludeincl. negation and worktreecommondir,.gitunder--ignore-dir,excludedcount) and a homonym regression intests/phase2.test.ts.Not done from the plan, on measurement
relations.tstypeDefs/resolveRelations recompute (under 2 ms) andcallers.tsenclosingAmong(not on the index path).🤖 Generated with Claude Code
https://claude.ai/code/session_01YDMVcJNo7WZh9xFAGWs2F8