Add a Go indexer service with a gRPC streaming contract

[ShreeBohara]

Extracts the walk-and-parse stage behind rpc IndexRepo(IndexRequest) returns (stream IndexEvent). Opt-in — IndexingService still uses the Python parser and nothing changes until enabled.

Why — and it is not speed

The obvious claim ("rewrote the parser in Go for performance") is false, and I measured it before writing any Go:

files parsed    : 900
sequential      : 1115.3 ms   (1.00x)
ThreadPool(4)   : 1092.5 ms   (1.02x)   <- threads give nothing
ProcessPool(4)  :  393.3 ms   (2.84x)

py-tree-sitter never releases the GIL, so threads are ~1.0x — which also means the obvious anyio.to_thread fix would not have worked. Only ~41% of the parse phase is C, so a perfect native rewrite ceilings near 2.4x, below the 2.84x that ProcessPoolExecutor on the existing Python already gives for half a day's work.

The real reason is a process and failure boundary: repos.py runs indexing by spinning a new event loop inside an anyio threadpool thread and blocking it for minutes, in the same process that serves chat.

Server streaming is why gRPC rather than one big response — progress becomes part of the contract instead of shared mutable state, the same bug class that made the SSE progress bar show only 0% or 100%.

Binding choice, verified not assumed

Uses the official tree-sitter/go-tree-sitter. Not smacker/go-tree-sitter, which is the trap — checked against the GitHub API:

	stars	last push	open issues
tree-sitter/go-tree-sitter	288	2025-11-16	15
smacker/go-tree-sitter	562	2024-08-27	42

The more popular one is the stale one, and it outranks the official binding in search results.

Two bugs not repeated

Both were fixed in the Python parser earlier; this gets them right from the start, with tests.

• .tsx uses LanguageTSX(), not LanguageTypescript() — the plain TS grammar cannot parse JSX.
• Chunk text comes from node.Utf8Text(source) over source bytes, because tree-sitter offsets are byte offsets.

had_parse_error is on the wire because tree-sitter returns a partial tree rather than failing — without it the Python caller cannot decide to fall back to raw indexing.

One bug fixed in the port

Python's _find_files max-files cap never worked: its break left only the inner filename loop, so os.walk continued into the next directory and kept appending. The Go walker uses fs.SkipAll and reports Truncated, so a partial index isn't mistaken for full coverage.

Verified end to end

Go server streaming to the Python client over real code (apps/api/src):

health: True v0.1.0
progress events : 5      chunk batches : 3      chunks : 509
summary : walked=59 parsed=58 chunks=509 errors=0 99ms
kinds   : {'module': 17, 'class': 121, 'function': 109, 'method': 262}
example : class 'Level' in api/graphql/schema.py L47-53   ← matches the file

Correctness cases:

tsx       function  'Widget'          had_parse_error=False
function  'after_unicode' L2-3  content: 'def after_unicode(x):'
class     'Café'         L4-6  content: 'class Café:'
method    'método'       L5-6  content: 'def método(self):'

That second block is the byte-offset test — the chunks come after a line containing héllo — wörld 日本語 🎉 and still slice correctly.

168 tests pass, ruff clean. Generated *_pb2*.py are excluded from ruff, since any fix there is overwritten by the next protoc run.

Not done, deliberately

The service is not wired into the pipeline. That swap needs a placement decision this PR doesn't make: the service must be co-located with the API on the shared volume, because clones live under ./data/repos/<owner>/<name> and that volume attaches to exactly one machine.

Also: no Go CI job (ci.yml has Python 3.11 and Node 20 only — no cgo toolchain, no grammar-compile caching, no cross-language contract test), and this build links 5 grammars against the Python side's 9. Health() reports which, so the caller never assumes parity.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
codebaseqa-web	Ready	Preview	Aug 11, 2026 9:10pm