vfs: fix readlink, realpath and mkdtemp encoding

[christianaurichzm]

On a mounted VFS, readlink(), realpath() and mkdtemp() only honor { encoding: 'buffer' }. The string form 'buffer' and other encodings like 'hex' are ignored, and so is a Buffer mkdtemp() prefix:

const fs = require('node:fs');
const path = require('node:path');
const vfs = require('node:vfs');

const myVfs = vfs.create();
myVfs.writeFileSync('/file.txt', 'x');
myVfs.symlinkSync('file.txt', '/link');
const link = path.join(myVfs.mount(), 'link');

fs.readlinkSync(link, 'buffer'); // 'file.txt' instead of a Buffer
fs.readlinkSync(link, 'hex');    // 'file.txt' instead of '66696c652e747874'

With RealFSProvider the options are passed on to the host fs, so readlink() with 'buffer' throws ERR_INVALID_ARG_TYPE and myVfs.realpathSync(p, 'hex') fails with EACCES.

The VirtualFileSystem readlink and realpath methods now get a string from the provider and apply the encoding with getOptions() and encodeRealpathResult(), as fs does. The helper moves from lib/fs.js to internal/fs/utils. The mkdtemp hooks do the same, which replaces the per-hook Buffer conversions for these methods in setup.js.

Since the VFS works with string paths, a RealFSProvider link target that isn't valid UTF-8 won't keep its bytes with 'buffer'. readdir() has a similar gap, which I'll leave for a separate PR.

The new test covers the sync, callback and promise variants on a memory and a real mount, and fails on main. The test-vfs-* tests and the fs realpath/readlink/mkdtemp tests pass on x64 Linux.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.36%. Comparing base (a2a064c) to head (878c5f6).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #66352      +/-   ##
==========================================
- Coverage   90.36%   90.36%   -0.01%     
==========================================
  Files         792      792              
  Lines      275386   275392       +6     
  Branches    52775    52773       -2     
==========================================
+ Hits       248843   248848       +5     
+ Misses      16979    16965      -14     
- Partials     9564     9579      +15     

Files with missing lines	Coverage Δ
lib/fs.js	97.32% <100.00%> (-0.01%)	⬇️
lib/internal/fs/utils.js	96.32% <100.00%> (+0.03%)	⬆️
lib/internal/vfs/file_system.js	99.63% <100.00%> (+<0.01%)	⬆️
lib/internal/vfs/setup.js	87.35% <100.00%> (+0.22%)	⬆️

... and 33 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.