perf(rules): pre-filter walk by fixed glob filename - #203
Open
raysubham wants to merge 1 commit into
Open
Conversation
walkRoots indexes relative globs by their fixed final path component and walkOneRoot skips files whose name has no candidate before computing the relative path or running any regex. Any glob with a wildcard in its final segment disables the index and keeps the original flat matcher loop, so matcher order and matched_glob selection are unchanged. Absolute globs, evaluate, limits, and reporting are untouched. On a 200k-file Ubuntu corpus the walk dropped from 33.98s to 0.21s with byte-for-byte identical RuleScan output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The malicious-file scanner ran every relative glob regex against every regular file walked (47 full-path checks per file on the current bundle). Most files can never match because their filename matches no rule.
walkRootsbuilds a filename index from relative globs whose final path segment is fixed.walkOneRootlooks upd.Name()beforefilepath.Reland any regex; no candidates → skip. Candidates still go through the unchanged regex +evaluate.*/?in its final segment disables the index and falls back to the original flat matcher loop, preserving matcher order andmatched_globselection.evaluate, limits, telemetry shape unchanged.Reference measurement (200,000 non-matching files, 2,001 dirs, depth 10, Ubuntu VM): 33.98s → 0.21s,
model.RuleScanJSON byte-for-byte identical.Tests
TestScan_WildcardFilenameGlob: wildcard filename glob still matches, and first glob winsmatched_globwhen a wildcard and fixed glob overlap (fallback path).TestScanRegexMatch: same filename in wrong directory is rejected by the full-path check (indexed path).go test ./internal/detector/rules/,go vet,golangci-lint(0 issues),gofmt, andCGO_ENABLED=0 GOOS=linuxbuild all pass.Non-goals
No concurrency, caching, directory exclusion, schema, API, or limit changes. Benchmark / integration workflow deferred.