Skip to content

AST traversal stacks are allocated from the per-file result arena and retained for the whole index #2010

Description

@shafty023

Split out of #1997; the retention half is #1925.

ts_nstack_init (internal/cbm/extract_node_stack.h) cuts its stacks from ctx->arena, which is
result->arena. In the parallel path that result is stored into ec->result_cache[file_idx]
(pass_parallel.c:963) and freed only after parallel_resolve and the infra passes
(pipeline.c:1324), so a structure written for a one-file lifetime is in fact held for the whole
index, once per file, concurrently. The header says the opposite: "freed when the arena is
destroyed at end of file extraction".

The largest instance is channel extraction. CHAN_STACK_CAP is 4096 and sizeof(TSNode) is 32, so
scan_string_consts_js (extract_channels.c:104) and extract_channels_js
(extract_channels.c:376) take 128 KB each, per JavaScript or TypeScript file. On a 14k-file
TypeScript monorepo that is 3,168 MB across 12,673 files, and it produced 23 Channel nodes out of
164,684. ts_nstack_init has 55 call sites across internal/cbm/extract_*.c; channels is simply
where the constant cap is largest and the yield smallest.

Nothing in a CBMFileResult ever points into a traversal stack: all 55 stacks are function-local
and no .items pointer is dereferenced outside extract_node_stack.h. They are pure scratch that
happens to be cut from an arena with a much longer lifetime.

Measured on Linux aarch64, where mem.allocator.owned classes=all so mapped equals resident,
giving extraction a per-call scratch arena takes mem.collect post_extract rss_mb from 13,293 to
6,752 and mem.phase peak_mb from 14,840 to 8,565, with nodes, parse_partial_count and the
registry counts identical over five runs per side.

I have a PR ready for this.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    parsing/qualityGraph extraction bugs, false positives, missing edges

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions