Skip to content

fs: allocate FSReqPromise stat arrays lazily#63886

Open
MarshallOfSound wants to merge 1 commit into
nodejs:mainfrom
MarshallOfSound:perf/fsreq-promise-lazy-stats
Open

fs: allocate FSReqPromise stat arrays lazily#63886
MarshallOfSound wants to merge 1 commit into
nodejs:mainfrom
MarshallOfSound:perf/fsreq-promise-lazy-stats

Conversation

@MarshallOfSound

Copy link
Copy Markdown
Member

Description of Change

Every promise-based fs operation allocates two AliasedBuffers at request construction β€” an 18-field stats array and an 8-field statfs array β€” although only stat-family resolutions ever read the first and only statfs() reads the second. Each one costs an ArrayBuffer, a TypedArray and a strong v8::Global, so a plain fsp.writeFile() pays for two backing stores it can never use. The callback path has no equivalent cost: FSReqCallback resolves stats through a shared per-process array.

Under concurrency this allocation churn is the dominant overhead of the promises path β€” profiling 64 in-flight 1KB reads shows ~21% of CPU ticks in the malloc family for promises vs ~5% for callbacks issuing identical syscalls.

This PR constructs the arrays lazily in ResolveStat()/ResolveStatFs():

  • Non-stat operations (open, read, write, close, unlink, …) now allocate neither array; stat-family ops allocate one; only statfs allocates the statfs array.
  • Once created, lifetime is unchanged β€” the array is still owned by the request, so deferred promise continuations read from request-owned memory exactly as before. The reason the per-request copy exists (continuations, unlike callbacks, don't consume the result synchronously) is preserved.
  • MemoryInfo reports the arrays only when they exist.

benchmark/compare.js shows significant improvements on the concurrent configurations β€” writefile-promises +12.6…20.7%, readfile-promises +19.1…21.5% (unencoded), bench-stat-promise +2.2…3.5% β€” with sequential large-payload configs neutral (those are threadpool-latency-bound, as expected).

Benchmark results

benchmark/compare.js, 10 samples per configuration, Welch t-test (*** p<0.001, ** p<0.01, * p<0.05):

fs/bench-stat-promise.js statType='fstat' n=200000                          +3.51% ***
fs/bench-stat-promise.js statType='lstat' n=200000                          +2.20% *
fs/bench-stat-promise.js statType='stat' n=200000                           +2.88% ***
fs/readfile-promises.js c=1  len=1024     enc=''       +3.25% ***   enc='utf-8'  +4.21% ***
fs/readfile-promises.js c=1  len=512k     enc=''       +3.61%       enc='utf-8'  +4.38% **
fs/readfile-promises.js c=1  len=8M       enc=''       +8.34% **    enc='utf-8'  -2.35%
fs/readfile-promises.js c=1  len=16M      enc=''       +2.30%       enc='utf-8'  +1.41%
fs/readfile-promises.js c=1  len=32M      enc=''       -2.17%       enc='utf-8'  +0.43%
fs/readfile-promises.js c=10 len=1024     enc=''      +21.02% ***   enc='utf-8' +19.84% ***
fs/readfile-promises.js c=10 len=512k     enc=''       +7.07% *     enc='utf-8'  +0.94%
fs/readfile-promises.js c=10 len=4M       enc=''      +19.14% **    enc='utf-8'  +1.39% *
fs/readfile-promises.js c=10 len=8M       enc=''      +21.51% ***   enc='utf-8'  -0.25%
fs/readfile-promises.js c=10 len=16M      enc=''      +20.99% ***   enc='utf-8'  +0.99% *
fs/readfile-promises.js c=10 len=32M      enc=''       +5.16% **    enc='utf-8'  +0.76%
fs/writefile-promises.js c=1  size=2      asc +7.17%*** buf +6.00%*** utf +5.83%***
fs/writefile-promises.js c=1  size=1024   asc +5.19%*** buf +6.00%*** utf +5.06%***
fs/writefile-promises.js c=1  size=64k    asc +2.65%*** buf +4.00%*** utf +2.12%***
fs/writefile-promises.js c=1  size=1M     asc +0.88%    buf +0.93%*   utf -0.38%
fs/writefile-promises.js c=10 size=2      asc +19.80%*** buf +18.70%*** utf +20.71%***
fs/writefile-promises.js c=10 size=1024   asc +13.51%*** buf +18.77%*** utf +12.58%***
fs/writefile-promises.js c=10 size=64k    asc +4.58%***  buf +2.65%**   utf +11.02%***
fs/writefile-promises.js c=10 size=1M     asc +2.25%*    buf +0.15%     utf +0.06%

Additional ad-hoc microbenchmarks (tmpfs, 64 ops in flight): fsp.writeFile 1KB +53%, fsp.stat +26%, fsp.readFile 1KB +22% β€” with all callback variants unchanged (Β±3%), confirming attribution since callbacks never construct FSReqPromise. Scripts: https://gist.github.com/MarshallOfSound/7d22707e3e550a782e96f9da0081e9f1

Checklist

  • I have built and tested this change
  • test/parallel/test-fs-* pass (342 tests; 3 pre-existing environment failures fail identically on main)
  • Verified fsp.stat, fsp.statfs and fsp.readFile resolve correctly through the lazily-created-array paths

Every promise-based fs operation eagerly allocated two AliasedBuffers
(a stats array and a statfs array) at request creation, although only
stat-family resolutions ever read the first and only statfs() reads
the second. Each allocation is an ArrayBuffer, a TypedArray and a
strong v8::Global. The callback path has no equivalent cost since it
resolves through a shared global array.

Construct the arrays lazily in ResolveStat()/ResolveStatFs() instead.
Once created the lifetime is unchanged, so deferred continuations
still read from request-owned memory.

Improves fs/promises throughput under concurrency: writeFile +53%,
stat +26%, readFile +22% at 64 in-flight operations on tmpfs, with
callback paths unchanged.

Signed-off-by: Sam Attard <sattard@anthropic.com>
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels Jun 13, 2026
@MarshallOfSound MarshallOfSound marked this pull request as ready for review June 13, 2026 03:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants