Silent index degradation — status:"indexed" but only ~500 nodes for 72k LOC Rust codebase

[nguiaSoren]

Title: Silent index degradation — status:"indexed" but only ~500 nodes for 72k LOC Rust codebase

Version: codebase-memory-mcp 0.6.0
Platform: macOS (Apple Silicon)
Project size: 72k LOC, 9 Rust files (single Cargo project)

Symptom

index_repository(mode="full") returns status:"indexed" but produces only ~500 nodes and ~500 edges for a 72k LOC Rust codebase. Expected ~5,000–10,000+ symbol nodes based on comparable smaller projects.

The status:"ready" / status:"indexed" response is misleading — the rebuild silently fails on something but reports success.

Reproduction

1. Kill any existing codebase-memory-mcp processes:
   ps aux | grep codebase-memory then kill <PID>
2. Delete cached index:
   rm ~/.local/share/codebase-memory-mcp/Users-soren-Desktop-new_simulator.db
3. Verify directory is empty of project cache
4. Restart Claude Code (spawns fresh MCP server process)
5. In Claude Code: index_repository(mode="full")
6. Run index_status — shows ~500 nodes / ~500 edges, status:"indexed"

Multiple fresh rebuilds produce the same low node count (469/462, 540/531, 572/555 across attempts).

Comparison with healthy project

Same machine, same MCP server binary:

• X project (~6k LOC): 6,436 nodes / 10,308 edges, 14 MB .db file ✓ healthy
• Y project (72k LOC): ~500 nodes / ~500 edges, 2 MB .db file ✗ degraded

The 2 MB file size is suspiciously exactly 2 × 1024 × 1024 bytes — suggests the database is being truncated or capped at a fixed boundary.

Known-symbol verification

search_graph(name_pattern="apply_knight_hero_effects") returns 0 results.

This function exists at src/hero_system.rs (verified via grep). Should be indexed as a function symbol but is missing from the graph.

Workaround

Direct file reads + grep work fine. The bug appears specific to the graph-based query path (search_graph, trace_path).

Diagnostic information requested

Happy to provide:

• Sample file from the codebase that may be triggering the issue (if I can isolate which file)
• Full index_repository output
• Any debug logging the indexer emits

[DeusData]

This looks like the same durability issue as #334 (note the near-identical counts — 469/540/572 — and 9-Rust-file setup). The varying low node count across fresh rebuilds points to non-deterministic write loss (uncheckpointed WAL under synchronous=OFF, often from a hard-killed sibling process), not a deterministic parser cap — I checked, there's no per-file size/node cap that would deterministically clip a large file.

v0.7.0 (#277) checkpoints the WAL on close and recovers a stale WAL on open, which should resolve the stranded-write loss. Could you retest on v0.7.0? If a clean full index still yields ~500 nodes for the 72k-LOC project, please share ls -l of the cache dir right after (so I can see if data is stranded in .db-wal) and, if possible, the repo or a representative large Rust file — then I can add a deterministic regression test. I'm also proposing an extracted-vs-persisted plausibility check over in #334 to make this fail loudly instead of silently; your input there welcome. Leaving open pending retest. 🙏

[DeusData]

Thanks for the data point. Important context here: Rust is currently a tree-sitter-only language in codebase-memory-mcp — it is not one of the hybrid-LSP languages (Python, TypeScript/JavaScript, PHP, C#, Go, C/C++). That means Rust gets structural definition extraction from the vendored tree-sitter grammar, but not the deeper LSP-backed symbol resolution and cross-file call-graph edges the hybrid languages get. So a lower node/edge count on a large Rust codebase is expected to a degree — though ~500 nodes for 72k LOC does sound lower than tree-sitter alone should yield, which points at gaps in the Rust extraction query (e.g. impl methods, trait methods, associated functions, macro-defined items).

This is a parsing/extraction completeness gap rather than the index-corruption class the rest of this umbrella tracks — the index isn't corrupt, Rust extraction is just shallow. To target it I'd like to know, on v0.7.0: roughly how many nodes you get now, and which symbol kinds are missing (free functions? impl methods? structs/enums? trait defs?). A pointer to the (public) Rust repo you tested would let me reproduce and tighten the tree-sitter query directly. Reclassifying this as a Rust-extraction-depth issue and keeping it open on that basis. Thanks!

[DeusData]

I dug into this empirically — built v0.7.0 and indexed a large pure-Rust OSS repo (ripgrep, 100 .rs files / 52k LOC) to test whether Rust extraction itself under-produces:

nodes=4581  edges=13509  errors=0   →  ~88 nodes/kLOC

Label breakdown was healthy: 2,039 Method, 697 Function, 356 Class (structs), 70 Enum, 608 Field, 7 Interface (traits). The single largest file (7,779 LOC) alone produced 985 nodes (~1 node / 8 LOC). So the Rust tree-sitter extraction is not generically broken — your ~500 nodes for 72k LOC is ~7 nodes/kLOC, 12× lower than a normal Rust repo, which points at something specific to your codebase rather than the parser.

Two likely causes, and one is easy to confirm:

1. File-discovery exclusion — if a .gitignore/.cbmignore pattern (or the workspace layout) is excluding most of your .rs files, discover only sees a few. Check your index log for pipeline.discover files=N — if N is small (tens, not hundreds), that's it.
2. Macro-heavy code — tree-sitter cannot see definitions that live inside macro invocations (it doesn't expand macros). I confirmed this on ripgrep itself: its macro-based test files (rgtest!(name, …)) extracted 0 functions despite 1,700+ LOC. If your codebase leans heavily on framework/proc macros (actix-web, diesel, rocket, sqlx, impl_trait!-style DSLs, generated code), large swaths become invisible and the node count collapses.

Could you paste these two lines from your index run? They pinpoint it immediately:

• pipeline.discover ... files=…
• parallel.extract ... done nodes=… errors=…

If files= is correct but nodes= is ~500, it's the macro blind spot (a known tree-sitter limitation we can discuss heuristics for). If files= is tiny, it's a discovery/ignore issue we can fix directly. Thanks — this'll let us nail it without your private repo.

[DeusData]

If its 2.: I already have the Rust LSP Hybrid in the works. Could potentially fix this, but we will need to double check, not sure though when I can ship this. Could take one or two weeks

[DeusData]

Update: the Rust LSP hybrid mentioned above shipped in v0.8.0 (current is v0.8.1) — Rust extraction now runs through the same hybrid-LSP path as TypeScript/Python/Go instead of tree-sitter-only.

Could you reindex the 72k-LOC Rust repo on v0.8.1 and post the resulting node/edge counts? If it still lands around ~500 nodes, the index log (pipeline.done and gbuf.dump nodes=… edges=… lines) would pin down where extraction stops. Thanks!

[nguiaSoren]

Retested on v0.8.1 (was still on 0.6.0 — sorry for the delay). Clean full index of the 72k-LOC Rust repo:

pipeline.discover files=948
gbuf.dump nodes=29594 edges=46487
pipeline.done ... status=indexed

So it's not reproducible — 29,594 nodes, and the symbol I reported missing (apply_knight_hero_effects) now resolves with full metadata.

Most useful data point: I think your first comment had it right — this was the WAL-durability class (#334), not extraction depth. Two reasons:

1. Before I even upgraded, the same v0.6.0 binary had already self-recovered to ~29.6k nodes (from the original ~500) with no code or version change. A deterministic extraction gap can't self-heal like that — but stranded-WAL write loss can, which matches the varying 469/540/572 counts you flagged.
2. I found 12 orphaned codebase-memory-mcp processes still running on macOS from rapid session churn — the exact killed-sibling / uncheckpointed-WAL scenario. Clearing them + a clean rebuild fixed it.

On the extraction-depth hypotheses from comments 2–3, for the record on this codebase: files=948 rules out discovery exclusion, and ground-truth grep -c 'fn ' vs the graph shows every production .rs file fully captured (graph symbols ≥ raw fn count in all of them — e.g. combat.rs 205 graph / 71 fns over 19.7k LOC; it's just a monolith of very large functions). It's also a free-function-heavy codebase (almost no impl blocks), so the impl/trait-method concern doesn't apply here.

Net: #387's WAL checkpoint-on-close looks like the correct fix. v0.8.1 node count is flat vs v0.6.0 (29,598 → 29,594) but edges jumped 36,233 → 46,487 (+28%) from the LSP hybrid's cross-file edges — nice bonus. Happy to keep an eye out for any recurrence under session churn now that the orphan-process pileup is cleared.

[DeusData]

This issue now has a first-class, durable reproduction in the cumulative bug-reproduction suite landing in #667 — a test that pins the exact failing behaviour, so it's tracked and guarded against silent regression. Where #667 also lands the root-cause fix, that's called out in the PR; otherwise the reproduction makes the eventual fix straightforward and verifiable.

The reproduction goes live once #667 merges. (Posted from the release branch's QA pass.)