Repository navigation
Parse the depth tier on WASM, so installing compiles nothing - #707
Conversation
The depth tier's grammars were native addons, and tree-sitter-kotlin has no prebuilt binary, so every `npm i -g` ran node-gyp (and failed without a C toolchain). The same extraction code now runs on web-tree-sitter, behind a thin adapter (src/graph/depth-parser.ts), with the same grammar versions: - go, java, php, r, typescript and tsx use the .wasm their package ships; python, kotlin and swift are compiled from their package's own src/ by scripts/build-grammar-wasm.mjs and committed in src/graph/grammars/. swift's scanner gets three wasm32 patches so it computes what the 64-bit addon computes. - The depth runtime is web-tree-sitter 0.22.2, the tree-sitter core the native addon vendors. On the current core, kotlin, swift and one typescript file parsed differently; on 0.22.2 all 10,963 corpus files give the native tree. - The grammar packages and tree-sitter move to devDependencies. They stay for test/depth-parity.ts, which builds real repos on both backends and diffs every node and edge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🌱 graft blast radius4 areas changed → 5 areas can be affected. 15 dependent symbols, depth 2. flowchart TB
A0(("Pull Request Review<br/>6 symbols"))
A1(("Workspace Graph Freshness<br/>4 symbols"))
A2(("Graph Engine<br/>2 symbols"))
A3(("Symbol Overlap Detection<br/>2 symbols"))
A4(("MCP Tool Invocation<br/>1 symbol"))
classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
class A0,A1,A2,A3,A4 reached;
Who knows this code — 1 person across 9 areas
Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no All 15 dependent symbols, grouped by areaPull Request Review — 6 symbols in 4 files
Workspace Graph Freshness — 4 symbols in 3 files
Graph Engine — 2 symbols in 1 file
Symbol Overlap Detection — 2 symbols in 1 file
MCP Tool Invocation — 1 symbol in 1 file
Test signal per changed area — 2 ✓ · 2 –Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.
35 test suites also reference this code41 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.
Open the interactive graph → — click an area to see its dependent symbols at file:line. |
What this does
npm i -g @trailhq/trail-codeno longer compiles anything. The depth tier (TypeScript/JavaScript, Python, Go, Java, Kotlin, Swift, PHP, R) used native tree-sitter addons, andtree-sitter-kotlinhas no prebuilt binary, so every install ran node-gyp, and failed on a machine without a C toolchain. The same extraction code now runs on WASM, and builds the same graph.Example: on okhttp (690 Kotlin and Java files) the native build and the WASM build produce the same 9,767 nodes and 17,913 edges, byte for byte, and the WASM build is 2.6x faster.
How
.wasmin their own npm package. python, kotlin and swift don't, soscripts/build-grammar-wasm.mjscompiles theirs from the package's ownsrc/(tree-sitter CLI 0.27.1, reproducible to the byte) and they are committed insrc/graph/grammars/.npm run buildcopies all nine intodist/vendor/grammars/.web-tree-sitter-depth), the tree-sitter core the native addon (tree-sitter0.21.1) vendors. The core decides how a grammar's conflicts and errors resolve. On the current core (0.27), 27 of 10,963 corpus files parsed differently: kotlinfun interfaceand@Tag("x") @Other class, swifttry? await, and one TypeScript file with deeply nested generics got an extra ERROR node. On 0.22.2, all 10,963 match. The breadth tier stays on 0.27.src/graph/depth-parser.tsowns parsing (warmDepthGrammars,depthReady,parseDepth) and aSyntaxNodeinterface that both runtimes satisfy.extract.tsandbindings.tsonly changed their node type name; the extraction logic is untouched. Builds already awaitedwarmGrammarsForbefore the parse loop, and that now warms the depth grammars too.unsigned long, andcalloc(0, …)returning a usable pointer. The native addon is also built at-O3, where an undefined1 << 32test vectorizes so that the scanner's!token is always dropped. Three source patches (documented in the build script) make the wasm compute what the addon computes. Without them, parses went wrong after the first file.tree-sitterand the eight grammar packages move to devDependencies. They stay for the parity check and to source the.wasmfiles.native.tsstays as the native backend behind the same interface (NATIVE_ONLYlists languages kept native; it is empty).Parity (the quality gate)
test/depth-parity.tsbuilds each repo twice, cold, in separate processes: once with every depth language on its native addon, once on WASM. Then it diffs the graphs: every node (all fields, including id, kind, path, span, signature, body_hash) and every edge (source, relation, target, confidence). Shallow clones of public repos, outside the repo:Times are whole-repo cold builds (all tiers), so a repo's languages share one number. WASM is faster everywhere (0.27x to 0.57x of native); nothing near the 2x limit. Breadth-tier files in these repos also matched.
Beyond the graph, I compared the parse trees themselves (every node's type, range and field, via TreeCursor) for all 10,963 depth-language files in the corpus: identical on all 9 grammars. One thing that's only visible through the old node API: native
node.typemislabels some comment nodes (it saysinterface_bodywhere the C tree sayscomment; 31 TS and 14 TSX files). WASM reports them correctly. It doesn't change any graph above.Nothing is kept native.
Size and install (
@trailhq/trail-code, packed withscripts/pack-trail.mjs)Fresh
npm i -g <tarball> --prefix <scratch> --cache <empty scratch cache> --foreground-scripts, Node 25.8, npm 11.11, darwin-arm64:npm warnlinestree-sitter-wasm(117 MB, breadth tier),openaiand@anthropic-ai/sdkare untouched, as planned for later PRs. The installed build was also smoke-tested withtrail buildon okhttp, Alamofire and dplyr: no fallback notice, same node counts as the parity run.Guards added
test/depth-grammars.test.ts:src/graph/grammars/grammars.jsonpins each of the nine.wasmfiles (package version, sha256, and the C source hash for built ones), so bumping a grammar package fails CI until the wasm is rebuilt and parity re-run. Also checks that the depth runtime stays at 0.22.2, that the grammar packages stay dev-only, and that the build vendors all nine byte for byte with their licenses.test/package-peers.test.tsnow reads the lockfile: no shipped dependency may have an install script, and every required peer must ship. On main this lists the 9 tree-sitter packages.test/graph-native-fallback.test.tscovers a depth.wasmthat won't load (broken install): that language falls back to the breadth stand-in, and the notice says to reinstall. The native backend's--allow-scriptsnotice is still covered.Fatal process out of memory: Zonefrom graft build OOMs with "Fatal process out of memory: Zone" on large monorepos (~2,900+ files); --max-old-space-size has no effect #122/test: pin Node 24 WASM compatibility #136) now also exercises every depth grammar (the probe file changed; the job itself didn't). Passes locally on Node 24.14 and Node 20.20. The job's step is still named "Exercise every breadth grammar": renaming it needs a token withworkflowscope, which this push didn't have.For reviewers
node scripts/build-grammar-wasm.mjs(fetches wasi-sdk, about 600 MB, on first use), thennode --import tsx test/depth-parity.ts <repos…>.npm ci(dev deps), same as today. The Dockerfile's build stage is unchanged for that reason.trace-shellfailed once under load and passed alone.🤖 Generated with Claude Code