perf: run cache scanners alongside the project walk and parallelize walking and sizing

docs/ARCHITECTURE.md

@@ -10,7 +10,7 @@ Scan → Classify → Filter/Sort → Output/Clean
 
 devclean is a monolithic Go CLI binary: every ecosystem scanner is built in, with no external scanner plugins.
 
-1. **Scan**: Registry batches all project (walk-based) scanners into a single filesystem pass, then runs stat scanners (fixed paths) sequentially. Real-time progress via spinner.
+1. **Scan**: Registry batches all project (walk-based) scanners into a single filesystem pass and runs stat scanners (fixed paths) sequentially alongside it. Results list the walk's first, then the stat scanners' in registry order. Real-time progress via spinner.
 2. **Classify**: activity status + gitignore-aware protection ([SPEC](SPEC.md#classification))
 3. **Filter/Sort**: CLI flags filter by ecosystem, category, status; sort by size/time/name; top N projects
 4. **Output**: colored table or JSON (`--json`)
@@ -39,13 +39,14 @@ Three invariants keep the figures honest:
 - **APFS clones** in a `CloneAware` artifact count once; clone groups split across artifacts go to `CloneShares` so `DedupedTotal` counts them once.
 - **Hugging Face shared blobs** referenced by several models go to `BlobShares` so `DedupedTotal` counts them once.
 
-Neither scanner family sizes inline. Each collects its artifacts first — the walk during its single pass, stat scanners via `stat` — then sizes them through one shared bounded worker pool (`sizePending`, `min(NumCPU, 8)`) so the tree-walk I/O overlaps.
+Neither scanner family sizes inline. Each collects its artifacts first — the walk during its single pass, stat scanners via `stat` — then sizes them through a worker pool (`sizePending`) so the tree-walk I/O overlaps. At most 8 sizing walks run at once process-wide, across the pools of both lanes: a worker waits for a slot before measuring an artifact, and `measure` passes the same slots to `fstree`, so a lone large artifact (a pnpm store) is not walked by one goroutine. Its goroutines share one hard-link and clone tally, so the totals do not depend on the split.
 
 ### Shared traversal (`fstree`)
 
 Finding, sizing and the pre-delete check walk through one package, so they cover the same files:
 
 - **Read each directory once.**
+- **Hand state down, not across**: a directory passes what its subdirectories need (the project walk's context stack) to them, so with `Slots` set subdirectories can be walked in other goroutines; hooks then run concurrently.
 - **Never follow symlinks.** Only the root is read through, and only for `--path` — [decision](decisions/symlink-no-follow.md)
 - **Stay on the root's filesystem** (like `du -x`); a directory on another device, of unknown device, or unreadable is returned as skipped — [decision](decisions/one-filesystem.md)
 - **Never read a dataless directory** (cloud placeholder whose content is not on disk); it is returned as skipped — [decision](decisions/skip-dataless.md)
@@ -60,7 +61,7 @@ A `walkEcosystem` table declares how one ecosystem participates in the shared wa
 | any-depth name | `__pycache__`, `.venv` | matches the directory name anywhere under the nearest project root |
 | any-depth suffix | `*.egg-info` | matches a directory-name suffix anywhere under the nearest project root |
 
-Per directory, the engine matches artifact rules against the **nearest** enclosing project root of each active ecosystem (table order, first match wins), emits the artifact and skips its subtree on a match, and otherwise reads the directory once to detect new project roots (pushed onto a recursion-scoped context stack) before descending into its child directories. Artifact matching runs **before** the hidden-directory check so compound rules ending in a hidden segment (`android/.gradle`) still match; unmatched hidden directories below the scan root are descended into only when an active ecosystem lists the name as an artifact.
+Per directory, the engine matches artifact rules against the **nearest** enclosing project root of each active ecosystem (table order, first match wins), emits the artifact and skips its subtree on a match, and otherwise reads the directory once to detect new project roots (added to the context stack it hands its subdirectories) before descending into its child directories. The walk runs on 8 walk slots of its own (separate from the sizing slots, so the walk is not starved while the stat scanners size); results are sorted by (table order, path) at the end, so they do not depend on the split. Artifact matching runs **before** the hidden-directory check so compound rules ending in a hidden segment (`android/.gradle`) still match; unmatched hidden directories below the scan root are descended into only when an active ecosystem lists the name as an artifact.
 
 Tables can also:
 

internal/fstree/fstree.go

@@ -12,7 +12,9 @@ import (
 	"io/fs"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
+	"sync"
 	"syscall"
 )
 
@@ -33,28 +35,32 @@ var deviceOf = func(_ string, info fs.FileInfo) (uint64, bool) {
 // cloud provider.
 var dataless = isDataless
 
-// Walker walks a tree depth-first, reading each directory once. Every hook is
-// optional.
-type Walker struct {
+// Walker walks a tree depth-first, reading each directory once. S is what a
+// directory hands its subdirectories (struct{} for nothing); the root gets the
+// zero S. Every hook is optional.
+type Walker[S any] struct {
 	// FollowRoot reads through a symlinked root, for a user-given location
 	// such as --path. Leave it off when the root is the item to delete:
 	// deleting a symlink removes only the link.
 	FollowRoot bool
+	// Slots, when set, bounds extra goroutines: a subdirectory is walked in
+	// its own goroutine while a slot is free. Hooks then run concurrently.
+	Slots chan struct{}
 	// Enter is called before a directory is read (and for a non-directory
-	// root, which is not read). Returning false skips it.
-	Enter func(dir string, info fs.FileInfo) bool
-	// Entries is called once a directory is read. Returning false skips its
-	// subdirectories and Leave.
-	Entries func(dir string, entries []fs.DirEntry) bool
-	// Leave is called after a read directory's subtree is walked.
-	Leave func(dir string)
+	// root, which is not read), with its parent's S. Returning false skips it.
+	Enter func(dir string, info fs.FileInfo, parent S) bool
+	// Entries is called once a directory is read, with its parent's S, and
+	// returns the S for its subdirectories. Returning false skips them. With
+	// Slots the subdirectories run concurrently on the same S, so neither hook
+	// may modify a parent S in place.
+	Entries func(dir string, entries []fs.DirEntry, parent S) (S, bool)
 }
 
 // Walk walks the tree under root and returns the directories it could not
-// check: on another filesystem, of unknown filesystem, dataless, or
+// check, sorted: on another filesystem, of unknown filesystem, dataless, or
 // unreadable. A root that does not exist is an empty tree. The only error is
 // ctx's.
-func (w Walker) Walk(ctx context.Context, root string) ([]string, error) {
+func (w Walker[S]) Walk(ctx context.Context, root string) ([]string, error) {
 	statRoot := os.Lstat
 	if w.FollowRoot {
 		statRoot = os.Stat
@@ -68,25 +74,36 @@ func (w Walker) Walk(ctx context.Context, root string) ([]string, error) {
 	}
 	rootDev, rootDevOK := deviceOf(root, info)
 
+	var mu sync.Mutex // guards skipped
 	var skipped []string
-	var visit func(dir string, info fs.FileInfo) error
-	visit = func(dir string, info fs.FileInfo) error {
-		if err := ctx.Err(); err != nil {
-			return err
+	skip := func(dir string) {
+		mu.Lock()
+		skipped = append(skipped, dir)
+		mu.Unlock()
+	}
+	var wg sync.WaitGroup
+	var visit func(dir string, info fs.FileInfo, parent S)
+	visit = func(dir string, info fs.FileInfo, parent S) {
+		if ctx.Err() != nil {
+			return
 		}
-		if w.Enter != nil && !w.Enter(dir, info) {
-			return nil
+		if w.Enter != nil && !w.Enter(dir, info, parent) {
+			return
 		}
 		if !info.IsDir() {
-			return nil
+			return
 		}
 		entries, err := readDir(dir)
 		if err != nil {
-			skipped = append(skipped, dir)
-			return nil
+			skip(dir)
+			return
 		}
-		if w.Entries != nil && !w.Entries(dir, entries) {
-			return nil
+		state := parent
+		if w.Entries != nil {
+			var ok bool
+			if state, ok = w.Entries(dir, entries, parent); !ok {
+				return
+			}
 		}
 		for _, e := range entries {
 			if !e.IsDir() { // false for symlinks: ReadDir does not follow them
@@ -97,30 +114,32 @@ func (w Walker) Walk(ctx context.Context, root string) ([]string, error) {
 			if errors.Is(err, fs.ErrNotExist) {
 				continue
 			}
-			if err != nil {
-				skipped = append(skipped, child)
-				continue
-			}
-			if dataless(childInfo) {
-				skipped = append(skipped, child)
+			if err != nil || dataless(childInfo) {
+				skip(child)
 				continue
 			}
 			if rootDevOK {
 				if dev, ok := deviceOf(child, childInfo); !ok || dev != rootDev {
-					skipped = append(skipped, child)
+					skip(child)
 					continue
 				}
 			}
-			if err := visit(child, childInfo); err != nil {
-				return err
+			select {
+			case w.Slots <- struct{}{}: // never ready when Slots is nil
+				wg.Go(func() {
+					defer func() { <-w.Slots }()
+					visit(child, childInfo, state)
+				})
+			default:
+				visit(child, childInfo, state)
 			}
 		}
-		if w.Leave != nil {
-			w.Leave(dir)
-		}
-		return nil
 	}
-	return skipped, visit(root, info)
+	var zero S
+	visit(root, info, zero)
+	wg.Wait()
+	slices.Sort(skipped)
+	return skipped, ctx.Err()
 }
 
 // UncheckedError reports a tree with directories Walk could not check.
@@ -138,7 +157,7 @@ func (e *UncheckedError) Error() string {
 // directories Walk could not check. A recursive delete does not stop at
 // them, so it would remove files that were never sized or shown.
 func CheckOneFilesystem(ctx context.Context, path string) error {
-	skipped, err := Walker{}.Walk(ctx, path)
+	skipped, err := Walker[struct{}]{}.Walk(ctx, path)
 	if err != nil {
 		return err
 	}