Repository navigation
Inspect: skip oversized files before reading; never parse plain JSON for TODOs (#1337) - #1340
Merged
Merged
Conversation
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Exclude plain JSON from TODO reads as not applicable. Reuse the 1 MiB search source limit, admit files before opening, and hold one of two shared permits across source reads, parsing and traversal. Surface oversized files through existing skipped coverage and invalidate persisted Tier-2 results. Skipped UTF-8 inputs are also omitted from scanned-file counts rather than counting unreadable inputs as analyzed. Size-limit wording remains pending Ufuk's approval as requested by the reviewer. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…lism (#1337) Replace the process-wide two-file cap with a 32 MiB stat-sized reservation budget. Keep reservations live through freshness reads, source reads, parsing and traversal; allow a lone file to proceed even if its clamped reservation exceeds a smaller budget. The delayed-parser regression now fills the byte budget with 32 simultaneous 1 MiB inputs and blocks the 33rd, replacing the previous two-file contract at the reviewer's request. Add small-file admission and single-file progress tests plus isolated Linux throughput/native tree-memory probes. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…e-size bump Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The automatic TODO scan could read and fully parse very large generated files, several at once, just to look for comments. On one reporter's machine, 2.1 GB of generated JSON drove the daemon from 0.8 GB to 56 GB of heap (#1337).
What changes
.jsonis no longer scanned for TODOs, because it can't contain comments..jsoncis still scanned.skipped_fileswith the reasonover the 1 MiB size limit, and the result reportscomplete: false.Measured on Linux, with 393 MB of synthetic JSON plus a small Rust source: peak RSS for the TODO scan fell from 625 MB to 73 MB, with the same TODO found. On AFT's own tree, the combined TODO and metrics scan is 10.7% faster than main (median of three interleaved rounds).
Gate: 520 inspect library tests, 377 inspect integration tests, 187 list-envelope tests, the inspect cache tests (after pinning version 43), formatting and the Windows check.
Not in this change: the JavaScript/TypeScript dead-code and unused-export readers (Oxc engine) still read whole files. That's a follow-up.
The skipped-file reason text was approved by the maintainer.
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Summary by cubic
Bounds memory use in the TODO, metrics, complexity and duplicate scanners so oversized generated files are skipped unread instead of fully parsed, which previously drove daemon heap from 0.8 GB to 56 GB on a 2.1 GB JSON fixture.
skipped_fileswith reasonover the 1 MiB size limit, and mark the resultcomplete: false..jsonis no longer scanned for TODOs since it can't contain comments;.jsoncstill is.Peak RSS for a TODO scan on Linux fell from 625 MB to 73 MB, and the combined TODO and metrics scan is about 10.7% faster than main on AFT's own tree. The JavaScript/TypeScript dead-code and unused-export readers still read whole files; bounding them is a follow-up.
Written for commit d0c9243. Summary will update on new commits.