feat(metro-file-map): Lazily stat files and populate symlinks for Node crawled file trees

[kitten]

Summary

Note

This is part of the same pick/patch-set from an experiment on the Expo repo,
like #1676, #1677, and #1687, for metro-file-map (conflicts are expected, since changes were pulled out into separate PRs for clarity)

After #1677 the highest impact on initial crawling performance with the Node crawler are the individual lstat calls and readlink calls. The former happens for every file that's discovered and can hence get quite large for monorepos (or assets that aren't excluded with blocklists). The latter, readlinks, are expensive if we're in an isolated installation (pnpm & bun), which can contain a much larger amount of symlinks than was likely anticipated by the current code paths.

This is the last (and most minimal iteration) of a few attempts to reduce the cost.

The idea is to skip the lstat call for new files that aren't in the previous FS snapshot (entry doesn't exist). To make this safe, these files are also excluded from getDifference, i.e. when both the previous and new entries have 0 | null mtime values. In both paths 0 | null indicates that the file is skipped for lstat.

The mtime is later added via an fs.promises.lstat call in getOrComputeSha1 i.e. when the file is actually accessed. If the file isn't accessed the mtime value remains at null.

This cascades to require/allow us to lazily evaluate symlinks for mtime values of null. Their values aren't read until accessed in #resolveSymlinkTargetToNormalPath in TreeFS. This does introduce an update on read, but lazily populates the symlink value. This trades off the async readlink call on all symlinks for a sync readlink call on a subset of symlinks.

We've observed this to be an overall win, but this could be influenced further by this commit: expo/expo@af67435 (which could also be upstreamed but changes the getContent signature. This change can be avoided by awaiting first, but seems overall like a better change than awaiting getContent unconditionally before plugins)

Changelog: [Internal] Lazily stat Node crawled files and lazily populate symlink targets

Test plan

• Unit tests were added to demonstrate this
• This can further be tested E2E in the linked Expo PR, although not in isolation (if necessary I can create a new, isolated branch with a pnpm patch for these changes)

[kitten]

This (LLM-generated) branch of a squash of this PR and #1687 combined demonstrates the rough performance when both changes are combined: https://github.com/kitten/metro/tree/%40kitten/bench/metro-file-map/crawl-and-lookup

The goal (which seems to be reached, at least in the synthetic tests on the branch) is to keep performance roughly the same when all files are accessed. Performance should improve when a large amount of crawled files are never accessed.

[robhogan]

AI:

---

I've merged main into this, resolving the conflicts with #1677, #1687 and #1693 the way @expo/metro-file-map does it: symlinks with a known mtime are still read eagerly, and ones crawled without an lstat are read on first traversal, both stored as #1687's lexically resolved normal POSIX targets. #1677's crawler already checks the extension before lstat, so the crawler change here is now just skipping the lstat for a file with no previous mtime.

Benchmarked the current head (d7e1754) against main (d35efc4, including #1677), on an RN 0.87 app installed with pnpm (3,031 symlinks, 42k files). Both built from source, 30 rounds, each running both builds in separate processes in random order. Values are medians ± 95% bootstrap CI, and the change is the median per-round ratio. M5 Pro, Node 22.13.1, so macOS only.

The first ten rows are metro-file-map alone: the file map build is a cold FileMap.build() of the project root (crawl + TreeFS, no cache, no watch folder, not a bundle build), each lookup pass is ~10k lookups through symlinks (the link itself, <link>/package.json, a file via the link, and hierarchicalLookup for package.json), and warm start is deserialising a cached snapshot plus one lookup pass. The replay rows are end to end: all 12,990 resolveDependency calls from an iOS dev build of the same app, replayed through each build's own DependencyGraph with the app's config, cold (empty file map cache) and warm (the cache the cold run wrote). The first pass goes through the resolution cache as a build does, and the repeat pass calls the resolver directly. Resolutions were identical across every replay.

Metric	main	PR	Change	95% CI of change
File map build (crawl + TreeFS)	109.7 ± 2.4 ms	74.7 ± 1.1 ms	−31.7%	−32.8% to −30.5%
Warm start	19.4 ± 0.2 ms	31.5 ± 0.4 ms	+62.5%	+60.9% to +63.7%
First lookup pass	20.9 ± 0.2 ms	36.8 ± 0.3 ms	+76.9%	+72.7% to +78.7%
Later lookup passes	13.7 ± 0.1 ms	13.8 ± 0.1 ms	+1.6%	+0.9% to +2.2%
Lookup: link itself	990 ± 10 ns	1,005 ± 10 ns	+1.5%	+0.4% to +2.0%
Lookup: <link>/package.json	1,218 ± 10 ns	1,229 ± 12 ns	+1.2%	+0.6% to +2.0%
Lookup: deep file via link	1,319 ± 12 ns	1,323 ± 12 ns	+0.9%	+0.3% to +1.6%
hierarchicalLookup	1,855 ± 15 ns	1,889 ± 9 ns	+2.3%	+1.4% to +3.0%
Snapshot size	1,523 KB	1,019 KB	−33.1%	same every run
Heap after first pass	23.30 MB	22.61 MB	−3.0%	same every run
Replay, cold: startup	168.9 ± 1.8 ms	125.3 ± 1.6 ms	−26.4%	−26.9% to −25.5%
Replay, cold: first pass	206.8 ± 1.7 ms	208.8 ± 1.3 ms	+1.1%	+0.1% to +2.2%
Replay, cold: startup + first pass	375.6 ± 6.6 ms	334.7 ± 3.3 ms	−11.4%	−11.8% to −10.2%
Replay, cold: repeat pass	174.6 ± 1.0 ms	174.1 ± 0.8 ms	−0.3%	−0.6% to +0.7%
Replay, warm: startup	153.5 ± 2.7 ms	116.4 ± 3.1 ms	−23.4%	−24.6% to −21.7%
Replay, warm: first pass	202.9 ± 2.1 ms	203.1 ± 1.7 ms	+0.0%	−1.1% to +0.9%
Replay, warm: startup + first pass	355.9 ± 8.9 ms	319.9 ± 5.8 ms	−10.2%	−11.0% to −9.0%
Replay, warm: repeat pass	169.5 ± 2.1 ms	170.0 ± 3.0 ms	+0.2%	−0.8% to +1.5%

Bold where the CI excludes zero.

[robhogan]

AI:

---

I split this into its parts to see where the numbers above come from. Same app, benchmarks and stats as above, against main at d35efc4, which now includes #1677's extension check before lstat. Four builds, 30 rounds with all four in random order per round. All 240 replays resolved identically.

• main - d35efc4
• lazystat - this PR's crawler, symlinks read at crawl time
• lazylink - main's crawler, symlinks read on first traversal
• tip - this PR (d7e1754)

Changes are the median per-round delta against main, with 95% bootstrap CI. Bold where the CI excludes zero.

	main	lazystat	lazylink	tip
lstat per start	36,545	0	36,545	0
readlink per start	3,031 async	3,031 async	132 sync	132 sync
File map build (crawl + TreeFS)	109.7 ms	−22.6 ms [−22.4%, −19.3%]	−9.0 ms [−10.5%, −6.1%]	−34.7 ms [−32.8%, −30.5%]
Replay, cold: startup + first pass	375.6 ms	−33.3 ms [−9.5%, −7.6%]	−13.1 ms [−4.3%, −3.0%]	−43.1 ms [−11.8%, −10.1%]
Replay, warm: startup + first pass	355.9 ms	−39.8 ms [−12.2%, −10.6%]	+0.1 ms [−1.9%, +3.0%]	−37.2 ms [−11.0%, −9.0%]
Replay, cold: first pass	206.8 ms	+0.5 ms [−1.2%, +1.4%]	+1.2 ms [−0.2%, +1.1%]	+2.2 ms [+0.1%, +2.2%]
Warm start	19.4 ms	−0.0 ms [−0.4%, +0.9%]	+12.5 ms [+63.3%, +67.1%]	+12.1 ms [+60.9%, +63.7%]
First lookup pass	20.9 ms	+0.4 ms [−0.1%, +3.9%]	+15.7 ms [+72.3%, +75.6%]	+16.0 ms [+72.7%, +78.7%]
Later lookup passes	13.7 ms	+1.9% [+1.3%, +2.3%]	+0.8% [+0.3%, +1.2%]	+1.6% [+0.9%, +2.2%]
hierarchicalLookup	1,855 ns	+2.8% [+2.2%, +3.4%]	+0.6% [−0.1%, +1.7%]	+2.3% [+1.4%, +3.0%]
Snapshot size	1,523 KB	−17.2%	−15.9%	−33.1%

• Lazy stat is most of it. Of the −43 ms cold end to end, lazy stat is −33 ms and lazy symlinks −13 ms. Warm, it's all lazy stat (−40 ms) - lazy symlinks are within noise.
• In metro-file-map alone the build saving splits into lazy stat −23 ms and lazy symlinks −9 ms.
• Lazy symlinks trade 3,031 async reads at crawl for 132 sync reads during resolution - only the links on this build's resolution paths. That's +1-2 ms on the cold first pass.
• Lazy symlinks are also the whole of the metro-file-map warm start and first pass regressions (+12-16 ms). The snapshot is written before any link is read, so every new process reads them again, and those benchmarks traverse every link. End to end it's within noise.
• The +1-2% on repeat lookups comes with lazy stat (lazystat, tip), and not with performance: Improve metro-file-map's Node crawler hot-path #1677's crawler changes on main. I haven't pinned down why.
• The snapshot saving splits about evenly between null mtimes/sizes and unstored symlink targets.
• On a warm start the crawler calls getMtimeByNormalPath, a full lookup from the root, for every crawled file. Against the previous main, removing just that call from an otherwise identical build saved 29.0 ms on warm startup (−12.5%, CI −14.9% to −10.5%; 16 rounds), and this PR pays that cost on every warm start.

[robhogan]

High level thoughts having looked closely at this and rebased on two related, now merged ones:

• Lazy statting is worth doing, but since statting isn't relevant to resolution or the core function of TreeFS, I think we can lift stats into a lazy plugin, alongside SHA1, probably after metro-file-map: Lazy plugin workers, run on demand #1950. At a minimum, it needs to be through an abstraction compatible with custom file systems and should use the same processFile path as SHA1.
• Lazy symlinks are of less clear value - the time we save during start up is paid (and then some) during the first build, which is arguably worse (a bundle request is more often on a user's critical path). It also unavoidably brings synchronous I/O into lookups.

Parking this for now, but hopefully we can get the benefits of lazy statting soon.