Split out of #1997; the retention half is #1925.
ts_nstack_init (internal/cbm/extract_node_stack.h) cuts its stacks from ctx->arena, which is
result->arena. In the parallel path that result is stored into ec->result_cache[file_idx]
(pass_parallel.c:963) and freed only after parallel_resolve and the infra passes
(pipeline.c:1324), so a structure written for a one-file lifetime is in fact held for the whole
index, once per file, concurrently. The header says the opposite: "freed when the arena is
destroyed at end of file extraction".
The largest instance is channel extraction. CHAN_STACK_CAP is 4096 and sizeof(TSNode) is 32, so
scan_string_consts_js (extract_channels.c:104) and extract_channels_js
(extract_channels.c:376) take 128 KB each, per JavaScript or TypeScript file. On a 14k-file
TypeScript monorepo that is 3,168 MB across 12,673 files, and it produced 23 Channel nodes out of
164,684. ts_nstack_init has 55 call sites across internal/cbm/extract_*.c; channels is simply
where the constant cap is largest and the yield smallest.
Nothing in a CBMFileResult ever points into a traversal stack: all 55 stacks are function-local
and no .items pointer is dereferenced outside extract_node_stack.h. They are pure scratch that
happens to be cut from an arena with a much longer lifetime.
Measured on Linux aarch64, where mem.allocator.owned classes=all so mapped equals resident,
giving extraction a per-call scratch arena takes mem.collect post_extract rss_mb from 13,293 to
6,752 and mem.phase peak_mb from 14,840 to 8,565, with nodes, parse_partial_count and the
registry counts identical over five runs per side.
I have a PR ready for this.
Split out of #1997; the retention half is #1925.
ts_nstack_init(internal/cbm/extract_node_stack.h) cuts its stacks fromctx->arena, which isresult->arena. In the parallel path that result is stored intoec->result_cache[file_idx](
pass_parallel.c:963) and freed only afterparallel_resolveand the infra passes(
pipeline.c:1324), so a structure written for a one-file lifetime is in fact held for the wholeindex, once per file, concurrently. The header says the opposite: "freed when the arena is
destroyed at end of file extraction".
The largest instance is channel extraction.
CHAN_STACK_CAPis 4096 andsizeof(TSNode)is 32, soscan_string_consts_js(extract_channels.c:104) andextract_channels_js(
extract_channels.c:376) take 128 KB each, per JavaScript or TypeScript file. On a 14k-fileTypeScript monorepo that is 3,168 MB across 12,673 files, and it produced 23
Channelnodes out of164,684.
ts_nstack_inithas 55 call sites acrossinternal/cbm/extract_*.c; channels is simplywhere the constant cap is largest and the yield smallest.
Nothing in a
CBMFileResultever points into a traversal stack: all 55 stacks are function-localand no
.itemspointer is dereferenced outsideextract_node_stack.h. They are pure scratch thathappens to be cut from an arena with a much longer lifetime.
Measured on Linux aarch64, where
mem.allocator.owned classes=allso mapped equals resident,giving extraction a per-call scratch arena takes
mem.collect post_extract rss_mbfrom 13,293 to6,752 and
mem.phase peak_mbfrom 14,840 to 8,565, with nodes,parse_partial_countand theregistry counts identical over five runs per side.
I have a PR ready for this.