Skip to content

Parse the depth tier on WASM, so installing compiles nothing - #707

Merged
shhdwi merged 1 commit into
mainfrom
feat/wasm-depth-parsers
Oct 27, 2026
Merged

shhdwi merged 1 commit into
mainfrom
feat/wasm-depth-parsers

Conversation

@shhdwi

@shhdwi shhdwi commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

What this does

npm i -g @trailhq/trail-code no longer compiles anything. The depth tier (TypeScript/JavaScript, Python, Go, Java, Kotlin, Swift, PHP, R) used native tree-sitter addons, and tree-sitter-kotlin has no prebuilt binary, so every install ran node-gyp, and failed on a machine without a C toolchain. The same extraction code now runs on WASM, and builds the same graph.

Example: on okhttp (690 Kotlin and Java files) the native build and the WASM build produce the same 9,767 nodes and 17,913 edges, byte for byte, and the WASM build is 2.6x faster.

How

  • Same grammars. Each language uses the exact grammar version the native addon used. go, java, php, r, typescript and tsx ship a .wasm in their own npm package. python, kotlin and swift don't, so scripts/build-grammar-wasm.mjs compiles theirs from the package's own src/ (tree-sitter CLI 0.27.1, reproducible to the byte) and they are committed in src/graph/grammars/. npm run build copies all nine into dist/vendor/grammars/.
  • Same core. The depth tier runs on web-tree-sitter 0.22.2 (dependency alias web-tree-sitter-depth), the tree-sitter core the native addon (tree-sitter 0.21.1) vendors. The core decides how a grammar's conflicts and errors resolve. On the current core (0.27), 27 of 10,963 corpus files parsed differently: kotlin fun interface and @Tag("x") @Other class, swift try? await, and one TypeScript file with deeply nested generics got an extra ERROR node. On 0.22.2, all 10,963 match. The breadth tier stays on 0.27.
  • Thin adapter. src/graph/depth-parser.ts owns parsing (warmDepthGrammars, depthReady, parseDepth) and a SyntaxNode interface that both runtimes satisfy. extract.ts and bindings.ts only changed their node type name; the extraction logic is untouched. Builds already awaited warmGrammarsFor before the parse loop, and that now warms the depth grammars too.
  • swift scanner patches. swift's scanner relies on two things that hold on 64-bit native and not in wasm32: a 64-bit unsigned long, and calloc(0, …) returning a usable pointer. The native addon is also built at -O3, where an undefined 1 << 32 test vectorizes so that the scanner's ! token is always dropped. Three source patches (documented in the build script) make the wasm compute what the addon computes. Without them, parses went wrong after the first file.
  • Deps. tree-sitter and the eight grammar packages move to devDependencies. They stay for the parity check and to source the .wasm files. native.ts stays as the native backend behind the same interface (NATIVE_ONLY lists languages kept native; it is empty).

Parity (the quality gate)

test/depth-parity.ts builds each repo twice, cold, in separate processes: once with every depth language on its native addon, once on WASM. Then it diffs the graphs: every node (all fields, including id, kind, path, span, signature, body_hash) and every edge (source, relation, target, confidence). Shallow clones of public repos, outside the repo:

repo language nodes edges node diffs edge diffs native ms wasm ms wasm/native
honojs/hono typescript 2787 4832 0 0 5050 1380 0.27
honojs/hono tsx 578 1057 0 0 5050 1380 0.27
expressjs/express javascript 271 170 0 0 1064 325 0.31
excalidraw/excalidraw typescript 3440 9769 0 0 9747 2878 0.30
excalidraw/excalidraw tsx 1517 7295 0 0 9747 2878 0.30
pallets/flask python 1705 2530 0 0 699 300 0.43
django/django python 46985 95957 0 0 23091 7968 0.35
gin-gonic/gin go 1653 3437 0 0 1164 411 0.35
google/gson java 4876 15990 0 0 2329 908 0.39
square/okhttp java 427 1332 0 0 6895 2607 0.38
square/okhttp kotlin 9340 16581 0 0 6895 2607 0.38
JetBrains/Exposed kotlin 11531 23002 0 0 11916 4464 0.37
composer/composer php 6771 17509 0 0 8145 2278 0.28
laravel/framework php 51035 102524 0 0 35422 10278 0.29
tidyverse/dplyr r 1192 2931 0 0 2500 747 0.30
Alamofire/Alamofire swift 2759 5950 0 0 3733 2146 0.57
vapor/vapor swift 3133 6016 0 0 3014 1238 0.41

Times are whole-repo cold builds (all tiers), so a repo's languages share one number. WASM is faster everywhere (0.27x to 0.57x of native); nothing near the 2x limit. Breadth-tier files in these repos also matched.

Beyond the graph, I compared the parse trees themselves (every node's type, range and field, via TreeCursor) for all 10,963 depth-language files in the corpus: identical on all 9 grammars. One thing that's only visible through the old node API: native node.type mislabels some comment nodes (it says interface_body where the C tree says comment; 31 TS and 14 TSX files). WASM reports them correctly. It doesn't change any graph above.

Nothing is kept native.

Size and install (@trailhq/trail-code, packed with scripts/pack-trail.mjs)

Fresh npm i -g <tarball> --prefix <scratch> --cache <empty scratch cache> --foreground-scripts, Node 25.8, npm 11.11, darwin-arm64:

before (0.26.0 from npm) after
tarball 3.96 MB 3.13 MB
install time (2 runs) 22.2 s, 14.2 s 8.2 s, 5.4 s
node_modules 304 MB 180 MB
installed package total 334 MB 203 MB
install scripts run 10 (9 tree-sitter addons + ours) 1 (our postinstall, no compile)
lines mentioning gyp 41 0
npm warn lines 0 0

tree-sitter-wasm (117 MB, breadth tier), openai and @anthropic-ai/sdk are untouched, as planned for later PRs. The installed build was also smoke-tested with trail build on okhttp, Alamofire and dplyr: no fallback notice, same node counts as the parity run.

Guards added

  • test/depth-grammars.test.ts: src/graph/grammars/grammars.json pins each of the nine .wasm files (package version, sha256, and the C source hash for built ones), so bumping a grammar package fails CI until the wasm is rebuilt and parity re-run. Also checks that the depth runtime stays at 0.22.2, that the grammar packages stay dev-only, and that the build vendors all nine byte for byte with their licenses.
  • test/package-peers.test.ts now reads the lockfile: no shipped dependency may have an install script, and every required peer must ship. On main this lists the 9 tree-sitter packages.
  • test/graph-native-fallback.test.ts covers a depth .wasm that won't load (broken install): that language falls back to the breadth stand-in, and the notice says to reinstall. The native backend's --allow-scripts notice is still covered.
  • The "Node 24 WASM regression" job (the V8 Turboshaft Fatal process out of memory: Zone from graft build OOMs with "Fatal process out of memory: Zone" on large monorepos (~2,900+ files); --max-old-space-size has no effect #122/test: pin Node 24 WASM compatibility #136) now also exercises every depth grammar (the probe file changed; the job itself didn't). Passes locally on Node 24.14 and Node 20.20. The job's step is still named "Exercise every breadth grammar": renaming it needs a token with workflow scope, which this push didn't have.

For reviewers

  • Rebuilding grammars: node scripts/build-grammar-wasm.mjs (fetches wasi-sdk, about 600 MB, on first use), then node --import tsx test/depth-parity.ts <repos…>.
  • Contributors still compile the native addons on npm ci (dev deps), same as today. The Dockerfile's build stage is unchanged for that reason.
  • Tests: full suite passes except the 2 known machine-only graft-to-trail tests. One timing test in trace-shell failed once under load and passed alone.

🤖 Generated with Claude Code

The depth tier's grammars were native addons, and tree-sitter-kotlin has no
prebuilt binary, so every `npm i -g` ran node-gyp (and failed without a C
toolchain). The same extraction code now runs on web-tree-sitter, behind a
thin adapter (src/graph/depth-parser.ts), with the same grammar versions:

- go, java, php, r, typescript and tsx use the .wasm their package ships;
  python, kotlin and swift are compiled from their package's own src/ by
  scripts/build-grammar-wasm.mjs and committed in src/graph/grammars/.
  swift's scanner gets three wasm32 patches so it computes what the 64-bit
  addon computes.
- The depth runtime is web-tree-sitter 0.22.2, the tree-sitter core the
  native addon vendors. On the current core, kotlin, swift and one
  typescript file parsed differently; on 0.22.2 all 10,963 corpus files
  give the native tree.
- The grammar packages and tree-sitter move to devDependencies. They stay
  for test/depth-parity.ts, which builds real repos on both backends and
  diffs every node and edge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trailhq-graft

trailhq-graft Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

🌱 graft blast radius

4 areas changed → 5 areas can be affected. 15 dependent symbols, depth 2.
Tests: 2 areas updated their tests.
Tag: @anirudhkumar-nanonets — 9 of 9 areas

flowchart TB
  A0(("Pull Request Review<br/>6 symbols"))
  A1(("Workspace Graph Freshness<br/>4 symbols"))
  A2(("Graph Engine<br/>2 symbols"))
  A3(("Symbol Overlap Detection<br/>2 symbols"))
  A4(("MCP Tool Invocation<br/>1 symbol"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2,A3,A4 reached;
Loading
Can be affected Symbols Nearest hop Reached from
Pull Request Review 6 src/app/review-worker.ts:L1-L110 review-worker.ts — imports, depth 1 Review Process, Graph Extraction
Workspace Graph Freshness 4 src/graph/check.ts:L59-L159 checkGraph — calls, depth 1 Graph Extraction
Graph Engine 2 src/engine.ts:L95-L105 graph — calls, depth 1 Graph Extraction
Symbol Overlap Detection 2 src/notes/edit-overlap.ts:L78-L93 symbolsIn — calls, depth 1 Graph Extraction
MCP Tool Invocation 1 src/mcp/tools.ts:L219-L247 callTool — calls, depth 2 Graph Extraction
Who knows this code — 1 person across 9 areas
Area Who knows it
Graph Extraction · changed @anirudhkumar-nanonets — 23 commits, last 5d ago
Review Process · changed @anirudhkumar-nanonets — 1 commit, last 2mo ago
Grammar Build · changed @anirudhkumar-nanonets — 1 commit, last 5d ago
CLI · changed @anirudhkumar-nanonets — 61 commits, last 5d ago
Pull Request Review · affected @anirudhkumar-nanonets — 11 commits, last 12d ago
Workspace Graph Freshness · affected @anirudhkumar-nanonets — 14 commits, last today
Graph Engine · affected @anirudhkumar-nanonets — 17 commits, last 14d ago
Symbol Overlap Detection · affected @anirudhkumar-nanonets — 1 commit, last 12d ago
…1 further area

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 15 dependent symbols, grouped by area

Pull Request Review — 6 symbols in 4 files

  • src/app/review-worker.ts:L1-L110 — review-worker.ts (imports, depth 1)
    9: * - `token` is handed over in the start message. This process never holds the
  • src/app/review.ts:L45-L99 — reviewPullRequest (calls, depth 1)
    51: const checkout = checkoutPullRequest({ owner: job.owner, repo: job.repo, number: job.number, baseRef: job.baseRef, token, log });
  • src/app/server.ts:L1-L159 — server.ts (imports, depth 1)
    70: const { id, token } = pages.put(j.owner, j.repo, j.number, html);
  • src/app/main.ts:L1-L66 — main.ts (imports, depth 2)
    2: * Entry point: read the environment, refuse to start half-configured, listen.
  • src/app/review-worker.ts:L67-L87 — run (calls, depth 2)
    84: // message is safe to put in the parent's log — which is where it is going.
  • src/app/server.ts:L31-L37 — AppSeams (references, depth 2)

Workspace Graph Freshness — 4 symbols in 3 files

  • src/graph/check.ts:L59-L159 — checkGraph (calls, depth 1)
    84: // Freshly extract Tier-1 nodes from the code on disk (same file set as build).
  • src/graph/refresh.ts:L233-L309 — ensureFreshGraph (calls, depth 1)
    248: // first build we lost the race for): a repo on trail gets its map built; any
  • src/graph/refresh.ts:L317-L343 — ensureFreshChildren (calls, depth 2)
    325: for (const child of children) {
  • src/graph/workspace.ts:L631-L656 — federateCheck (calls, depth 2)
    636: const lines = [`workspace check — ${wg.loaded.length + wg.missing.length} repo(s)`, ""];

Graph Engine — 2 symbols in 1 file

  • src/engine.ts:L95-L105 — graph (calls, depth 1)
    95: graph(dir: string, opts: GraphRunOptions = {}): Promise<GraphBuildResult> {
  • src/engine.ts:L86-L88 — checkGraph (calls, depth 2)

Symbol Overlap Detection — 2 symbols in 1 file

  • src/notes/edit-overlap.ts:L78-L93 — symbolsIn (calls, depth 1)
    80: await warmGrammarsFor([rel]);
  • src/notes/edit-overlap.ts:L189-L220 — functionOverlaps (calls, depth 2)
    190: repo: string,

MCP Tool Invocation — 1 symbol in 1 file

  • src/mcp/tools.ts:L219-L247 — callTool (calls, depth 2)
    245: return { text: err instanceof Error ? err.message : String(err), isError: true };
Test signal per changed area — 2 ✓ · 2 –

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

  • ✓ Graph Extraction — 15 of 124 reached · 12 test files changed here: test/ask-index.test.ts, test/depth-grammars.test.ts, test/depth-parity.ts, test/generic-node24-probe.ts, test/graph-bindings.test.ts, test/graph-extract-dedup.test.ts, test/graph-native-fallback.test.ts, test/graph-php.test.ts, test/graph-r-phase3.test.ts, test/graph-r-phase5.test.ts, test/graph-resolve-typed.test.ts, test/graph-swift.test.ts
    • not reached: defName, swiftDefName, rDefName, goReceiverVarOf, goReceiverTypeOf, isClassNode, collectBindings, collectAliases, …101 more
  • – Review Process — 1 test file changed here: test/app-review-process.test.ts
  • ✓ Grammar Build — 1 of 3 reached · 1 test file changed here: test/depth-grammars.test.ts
    • not reached: compile, sha256
  • – CLI — no function, method or class changed here
35 test suites also reference this code

41 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

  • test/app-review-process-probe.ts
  • test/app-server.test.ts
  • test/ask.test.ts
  • test/claude-prompt-private.test.ts
  • test/container-extract.test.ts
  • test/context-only-dir.test.ts
  • test/context.test.ts
  • test/covers.test.ts
  • test/generic-extract.test.ts
  • test/graph-go.test.ts
  • test/graph-incremental.test.ts
  • test/graph-invariants.test.ts
  • test/graph-java.test.ts
  • test/graph-languages.test.ts
  • test/graph-posix-paths.test.ts
  • test/graph-python.test.ts
  • test/graph-r-classes.test.ts
  • test/graph-r-phase4.test.ts
  • test/graph-r.test.ts
  • test/graph-references.test.ts
  • …15 more

⚠️ 6 changed files not in the graph (package-lock.json, package.json, src/graph/grammars/grammars.json, src/graph/grammars/tree-sitter-kotlin.wasm, src/graph/grammars/tree-sitter-python.wasm) — no parser claims the extension, or the index predates the file.

graft blast · origin/main...HEAD · depth 2 · 35 changed files

Open the interactive graph → — click an area to see its dependent symbols at file:line.

github-actions Bot added a commit that referenced this pull request Oct 27, 2026
@shhdwi
shhdwi merged commit 19865ff into main Oct 27, 2026
10 of 11 checks passed
github-actions Bot added a commit that referenced this pull request Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant