Populate the FTS5 index with comments and string literals

[mutoharohfiqhiabcd-source]

What

Populates code_text_fts, which has existed in the schema since day one but was never written to.

docs/development.md has carried this as an open TODO:

The FTS5 table schema exists but is not yet populated or exposed via MCP tools.

• Populate FTS table during indexing (extract strings/comments from tree-sitter AST)

The TextElement dataclass was declared for it too, but nothing ever produced one. The only statement touching the table was a DELETE, so every MATCH query returned nothing:

$ python -c "import sqlite3; ..."
code_text_fts : 0 rows
symbols       : 7597 rows

This wires up the missing path end to end:

extract -> cache -> worker -> sqlite writer -> code_text_fts

Extraction

• C / C++ / ObjC++ / Metal and Rust reuse the tree-sitter trees already built for symbol extraction, walking for comment and string_literal nodes. A matched node is not descended into, so a comment body cannot produce duplicate children.
• Python uses the stdlib tokenize module, since Python is parsed through ast here and no Python grammar is bundled as a dependency.
• Comment markers and surrounding quotes are stripped. Content is capped at 2000 chars and 5000 elements per file, so a generated header full of string blobs cannot bloat the index.

Caching

FileCache gains text_elements, with CACHE_VERSION bumped 6 → 7 so older caches are invalidated. Incremental runs reuse them exactly like symbols — a second run on an unchanged tree reports Cache: 600 cached, 0 parsed.

Measured

On a 600-file C++ project (583 C/C++ headers/sources, 17 Python):

symbols          7597
code_text_fts   27998     18267 comments, 9731 string literals
index time      ~20s, up from ~17s

Queries that previously returned nothing now work:

MATCH 'teleport'
  src\Lawn\BlinkSystem.cpp:160
  Lawn-mower protection: never teleport from the left of a ready mower

MATCH 'MimicZombie'
  src\Lawn\MimicZombie.cpp:1
  MimicZombie.h
  ... 4 more include sites

Known limitation

The existing schema uses tokenize='unicode61 remove_diacritics 2', which does not segment CJK. A run of Chinese characters becomes a single token, so substring queries against it match nothing:

MATCH '"植物"'  -> 0 rows     (the stored token is the whole sentence)

Switching to tokenize='trigram' would fix that and enable substring search generally, but it changes the schema documented in docs/architecture.md and docs/development.md. I left it out rather than smuggle a schema change into this PR — happy to follow up if you want it.

Test status

Unchanged by this commit. On Windows, tests/ reports 26 failed / 41 passed both with and without these changes — the failures are pre-existing and concentrated in the shell/git-hook tests (test_update_context.py, test_setup_permissions.py), not in indexing.

Note

This was developed on top of #5 (Windows compatibility) because that fix is what makes map.py runnable on Windows at all, then rebased onto main so the diff here is FTS-only. #5 is independent of this change; it's just a prerequisite for testing it on Windows.