Skip to content

test(fs): large-tree stress tests; stream rg's walk, cap ls -R at the stdout limit - #2820

Merged
chaliy merged 3 commits into
mainfrom
claude/project-thread-w2g6tc
Nov 3, 2026
Merged

chaliy merged 3 commits into
mainfrom
claude/project-thread-w2g6tc

Conversation

@chaliy

@chaliy chaliy commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What changed

  • New stress test binary (large_tree_stress_tests) runs walkers over a procedural tree (default 20k files, just stress = 2M files) with a counting allocator and asserts peak memory stays bounded: find, find|wc, du, tree -L 2, grep -rl, ls -R, rg.
  • rg walks a single root in path order and reads files as it finds them, instead of building the whole candidate list first. It stops reading once the input budget is spent.
  • ls -R stops producing output once it passes the stdout limit, instead of building the full listing and truncating it afterwards.
  • Knowledge: testing.md, threat model TM-DOS-140 rows (knowledge + public), vfs.md, log.md.

Why

Last part of the remote FS walk work: prove that a repo with millions of files can't push bashkit into unbounded memory.

Before / After

At 2M files (just stress, release):

command before after
find / du / tree / grep -r 0.2–2.3 MiB peak unchanged
ls -R 35.4 MiB ~7 MiB (capped at the stdout limit)
rg over budget (1 MiB budget) ~70 MB peak ~14 MB, reads stop early
rg within budget (200k files) 66 MB 34 MB

Known gap: rg within budget still keeps all searched text (158 B/file). Fixing it needs a streaming search; this is marked as TODO(TM-DOS-140) in the test and the docs.

Risk

  • Low/Medium. rg's file order and output are unchanged (still sorted by path). ls -R output is unchanged below the stdout limit.

Checklist

  • Tests added or updated
  • Backward compatibility considered

Generated by Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit bb10f8a Commit Preview URL

Branch Preview URL
Nov 03 2026, 07:04 AM

@bashkit-pr-reviewer bashkit-pr-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good mitigation, well-pinned by tests. Two gaps, one in the ls cap's granularity and one residual in rg's collect path. No P0.

Comment thread crates/bashkit/src/builtins/ls/list.rs Resolved
Comment thread crates/bashkit/src/builtins/rg/mod.rs Resolved
@bashkit-pr-reviewer

bashkit-pr-reviewer Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Review summary — #2820 (head bb10f8ad, 3rd push)

Verdict: The P1 is fixed: per-line output_limit breaks now guard every append loop (-l, one-per-line, and subdir headers), plus a ls_wide_directory_stops_at_the_stdout_limit regression test. Posted one new P2 on the narrowed residual; the rg multi-root P2 from before still stands.

Findings:

  • [P1, resolved] ls per-line cap — verified the new if output.len() > opts.output_limit { break; } checks in each loop and the wide-dir test (-l, -1, explicit paths).
  • [P2, new] names Vec / format_columns still unbounded (ls/list.rs:433): the non-long branch collects every entry name before the (now capped) output loop, and column mode formats the whole grid in one push — a wide dir in the default path still transiently holds ~50+ B per entry.
  • [P2, still open] rg multi-root / non-path sorts still collect-then-read (rg/mod.rs:3092): unchanged this push.

Context this round: refreshed checkout, confirmed rg/docs/stress hunks are untouched since the last head and the only implementation delta is the per-line ls breaks; read the new regression test. Still not built/run (no cargo in this shell).

Coverage (tarpaulin) build peaked at 17.1 MB vs 13.7 MB locally, over the
fixed 16 MiB bound. rg holds each read file until the budget trips, so
the bound now scales with reads (512 B/read), still far below the
pre-streaming ~1.5 KB/read.
…tory

Review follow-up: the cap was only checked between directories, so a
single directory with many entries still built its whole listing before
truncation. Also document rg's residual candidate list for several roots,
non-path sorts and --files (TM-DOS-140).

@bashkit-pr-reviewer bashkit-pr-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per-line caps look right and the wide-dir regression test covers -l/-1. One narrowed residual of the same issue remains in the non-long branch.

Comment thread crates/bashkit/src/builtins/ls/list.rs Resolved
@chaliy
chaliy merged commit 6d66d43 into main Nov 3, 2026
57 checks passed
@chaliy
chaliy deleted the claude/project-thread-w2g6tc branch November 3, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant