Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions crates/bashkit/src/builtins/walk_cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,30 @@ mod tests {
assert_eq!(cache.held(), 0);
}

/// The default walk visits directories in path order, the order tree,
/// ls -R and find consume them, so a batch cut short by `max_entries`
/// holds the next listings the caller needs instead of arbitrary ones.
#[tokio::test]
async fn default_walk_visits_directories_in_path_order() {
let fs = InMemoryFs::new();
for d in ["b", "a", "c", "a/y", "a/x", "c/z"] {
fs.mkdir(&PathBuf::from(format!("/t/{d}")), true)
.await
.unwrap();
}
let got: Vec<String> = fs
.walk(Path::new("/t"), WalkOptions::new(None))
.await
.unwrap()
.into_iter()
.map(|l| l.path.display().to_string())
.collect();
assert_eq!(
got,
["/t", "/t/a", "/t/a/x", "/t/a/y", "/t/b", "/t/c", "/t/c/z"]
);
}

#[tokio::test]
async fn missing_directory_reports_read_dir_error() {
let fs = tree(1, 1).await;
Expand Down
19 changes: 13 additions & 6 deletions crates/bashkit/src/fs/traits.rs
Original file line number Diff line number Diff line change
Expand Up @@ -539,12 +539,19 @@ pub async fn walk_with_read_dir<F: FileSystem + ?Sized>(
};
let child_depth = depth + 1;
if opts.max_depth.is_none_or(|m| child_depth < m) {
for entry in &entries {
if entry.metadata.file_type.is_dir()
&& !(opts.skip_hidden && entry.name.starts_with('.'))
{
pending.push((super::vfs_join(&path, &entry.name), child_depth, None));
}
// Visit children in name order (pushed reversed, popped first),
// the order tree/ls -R/find consume them: a batch cut short by

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] ls -R -t still consumes in mtime order while this prefetches in name order, so the "holds the next listings the caller needs" claim doesn't hold for that flag. Evidence: crates/bashkit/src/builtins/ls/list.rs:361-366 sorts sorted_entries by modified when sort_by_time, but list.rs:297 builds WalkCache::new(opts.recursive) unconditionally, and subdirs follows the time-sorted order. Correctness is unaffected (cache misses fall back to read_dir), just extra round trips on large ls -R -t runs. Consider scoping prefetch to the name-order case (e.g. WalkCache::new(opts.recursive && !opts.sort_by_time)) or softening this comment to "the order tree/ls -R (name order)/find consume them".

// `max_entries` then holds the next listings the caller needs.
let mut dirs: Vec<&str> = entries
.iter()
.filter(|e| {
e.metadata.file_type.is_dir() && !(opts.skip_hidden && e.name.starts_with('.'))
})
.map(|e| e.name.as_str())
.collect();
dirs.sort_unstable_by(|a, b| b.cmp(a));
for name in dirs {
pending.push((super::vfs_join(&path, name), child_depth, None));
}
}
held = held.saturating_add(entries.len());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -545,8 +545,10 @@ async fn tree_prefetch_stays_within_entry_budget() {
let r = bash.exec("tree tree").await.unwrap();
assert_ne!(r.exit_code, 0, "expected the entry budget to stop tree");
let c = fs.counts();
// 50 entries is a handful of directories; never the whole tree.
assert!(c.read_dir < 15, "{c}");
// 50 entries is a handful of directories. The walk prefetches in path
// order, the order tree prints, so it reads exactly the listings tree
// shows (before, arbitrary order read 8-15 depending on hashing).
assert!(c.read_dir <= 6, "{c}");
}

#[tokio::test]
Expand Down
8 changes: 5 additions & 3 deletions knowledge/foundations/vfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -417,9 +417,11 @@ and measured by `benches/remote_fs.rs` over the bashkit checkout:
the listings of `root` and the directories below it in one call, as a
bounded batch: it stops once the answer holds `max_entries` entries
(default 10k; at most one listing over, never a split listing). The
default is one `read_dir` per directory, depth first, reading a directory
only when popped (`walk_with_read_dir`); bulk backends override it with a
prefix query and a row limit. Wrappers forward it when they map the subtree one-to-one
default is one `read_dir` per directory, depth first in path order (the
order tree/ls -R/find consume, so a cut-short batch holds the next listings
they need), reading a directory only when popped (`walk_with_read_dir`);
bulk backends override it with a prefix query and a row limit, and should
answer in path order too. Wrappers forward it when they map the subtree one-to-one
(`FollowFs` rebases a linked root, `MountableFs` when no mount lies below the
root, `RootFs` outside system trees, `ReadOnlyFs`, per-execution wrappers);
otherwise they fall back to the default. Callers use it only when they would
Expand Down
1 change: 1 addition & 0 deletions knowledge/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## 2026-11-03

* **Walk order**: the default `walk` visits directories in path order. It used `read_dir` order (hash order for `InMemoryFs`), so a batch cut short by `max_entries` prefetched listings `tree` never printed and `tree_prefetch_stays_within_entry_budget` read 8-15 directories run to run. See [Virtual Filesystem](foundations/vfs.md#remote-backends-round-trips).
* **Large-tree stress tests**: `tests/large_tree_stress_tests.rs` measures heap peaks of tree walkers over a 200k-file procedural tree (2M with `BASHKIT_STRESS_FILES`). It found `rg` collecting every candidate path before reading (~70 MB for 200k files); its walk now streams in path order. See [Testing](operations/testing.md).
* **ls -R / tree bulk walk**: `ls -R` and `tree` list through bounded `walk` batches (`WalkCache`) like `find`/`du`. New `WalkOptions::skip_hidden` keeps the default walk out of dot directories they do not print; `tree` caps batches at its remaining entry budget and keeps `read_dir` under `-I`. See [Virtual Filesystem](foundations/vfs.md#remote-backends-round-trips).

Expand Down
Loading