task: Silent index corruption / incomplete index reported as success (3 issues)

[DeusData]

Scope

Umbrella tracker for cases where indexing reports status: "indexed" / ready but the resulting graph is catastrophically incomplete (single-digit % of expected node count, or empty). The reports cluster around macOS, large Rust projects, and rapid kill/restart cycles where the previous process didn't release WAL or hash-row state cleanly.

PR #387 ("checkpoint WAL on close and startup to prevent orphan accumulation") resolves the WAL subset; reporters asked to re-test on v0.7.0.

Sub-issues

• Indexing completes but DB is empty (all zeros) — macOS arm64 v0.6.0 #260 — Indexing completes but DB is empty (all zeros) — macOS arm64 v0.6.0 — awaiting v0.7.0 retest (WAL checkpoint-on-open/close + startup recovery shipped via fix(store): checkpoint WAL on close and startup to prevent orphan accumulation #387/a6ad401; not deterministically reproducible)
• Silent index degradation — status:"indexed" but only ~500 nodes for 72k LOC Rust codebase #333 — ~500 nodes for 72k LOC Rust — reclassified: Rust extraction-depth gap, not corruption. Rust is tree-sitter-only (not in the hybrid-LSP set), so extraction is shallow; asked reporter for v0.7.0 node count + missing symbol kinds + a public repo to tighten the tree-sitter query.
• Silent index corruption after rapid kill/restart cycles #334 — Silent index corruption after rapid kill/restart cycles — awaiting v0.7.0 retest (fix(store): checkpoint WAL on close and startup to prevent orphan accumulation #387 checkpoints WAL on open+close; crash-safety covered by bulk_crash_recovery / store_checkpoint tests)

Status (2026-05-31)

The shared WAL/checkpoint root cause (#260, #334) is addressed by #387 in v0.7.0 (checkpoint on open + close, startup WAL recovery), with existing crash-recovery regression tests. Neither is deterministically reproducible in CI (timing/platform-specific), so both await reporter confirmation on v0.7.0. #333 is a distinct Rust extraction-completeness gap (tree-sitter-only language), not corruption — tracked on that basis pending a reproducible Rust repo.

Acceptance

1. status: "indexed" is never returned when the index is materially incomplete — WAL stranding fixed (fix(store): checkpoint WAL on close and startup to prevent orphan accumulation #387); a plausibility-threshold gate (criterion 2) remains a candidate enhancement.
2. A plausibility threshold gates the success status — open (enhancement).
3. Recovery path is automatic — WAL is now checkpointed on open+close; no manual MCP reconnect needed.
4. Concurrent-writer / rapid-kill scenarios reproducible in a test — bulk_crash_recovery covers the committed/uncommitted invariants.

Why one task

All three reports share the surface symptom (success reported, graph materially incomplete); #260/#334 share a WAL-checkpointing root cause, while #333 turned out to be a separate Rust-extraction-depth gap.

Added 2026-06-29 (triage)

• cbm v0.8.1 silently deletes project DBs on "corrupt" detection — data loss with no recovery #557 — project DBs deleted after corrupt detection; data loss / recovery behavior

Added 2026-07-09 (triage)

• Project remains listed but becomes unqueryable after MCP restart with stale WAL/SHM sidecars #790 — project remains listed but unqueryable after MCP restart with stale WAL/SHM sidecars
• index_repository with relative repo_path (".") corrupts and auto-deletes the project DB every run #794 — index_repository with relative repo_path (".") corrupts and auto-deletes the project DB every run
• Fast artifact export raw-copies a live WAL-mode DB — torn graph.db.zst that import then installs unchecked #895 — fast artifact export raw-copies a live WAL-mode DB → torn graph.db.zst installed unchecked
• Row-scan loops swallow SQLITE_CORRUPT — queries return plausible truncated/empty results with no error #896 — row-scan loops swallow SQLITE_CORRUPT → plausible truncated/empty results with no error
• Sequential pipeline emits invalid JSON: unterminated "broker" on ASYNC_CALLS edges, raw non-JSON Route properties #898 — sequential pipeline emits invalid JSON (unterminated "broker" on ASYNC_CALLS, raw non-JSON Route properties)

(#897 — stale -wal/-shm replay over fresh DB — has an active fix PR #940 and is tracked there.)

[nwn900]

I have the following issue too