Skip to content

Inspect: skip oversized files before reading; never parse plain JSON for TODOs (#1337) - #1340

Merged
ualtinok merged 7 commits into
mainfrom
land/todo-scan-bounds
Nov 3, 2026
Merged

ualtinok merged 7 commits into
mainfrom
land/todo-scan-bounds

Conversation

@aft-alfonso

@aft-alfonso aft-alfonso Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

The automatic TODO scan could read and fully parse very large generated files, several at once, just to look for comments. On one reporter's machine, 2.1 GB of generated JSON drove the daemon from 0.8 GB to 56 GB of heap (#1337).

What changes

  • Plain .json is no longer scanned for TODOs, because it can't contain comments. .jsonc is still scanned.
  • The TODO, metrics, complexity and duplicate scanners check a file's size before opening it. Files over 1 MiB are skipped unread and listed in skipped_files with the reason over the 1 MiB size limit, and the result reports complete: false.
  • Reads that grow between the size check and the read are cut at the limit.
  • These scanners share a 32 MiB budget for source being read, parsed or walked at once. Small files still run across the whole thread pool, and a single file is always admitted.
  • The inspect contribution cache version moves to 43, so results from the old unbounded scans aren't reused.

Measured on Linux, with 393 MB of synthetic JSON plus a small Rust source: peak RSS for the TODO scan fell from 625 MB to 73 MB, with the same TODO found. On AFT's own tree, the combined TODO and metrics scan is 10.7% faster than main (median of three interleaved rounds).

Gate: 520 inspect library tests, 377 inspect integration tests, 187 list-envelope tests, the inspect cache tests (after pinning version 43), formatting and the Windows check.

Not in this change: the JavaScript/TypeScript dead-code and unused-export readers (Oxc engine) still read whole files. That's a follow-up.

The skipped-file reason text was approved by the maintainer.


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Bounds memory use in the TODO, metrics, complexity and duplicate scanners so oversized generated files are skipped unread instead of fully parsed, which previously drove daemon heap from 0.8 GB to 56 GB on a 2.1 GB JSON fixture.

  • Files over 1 MiB are rejected before opening, reported in skipped_files with reason over the 1 MiB size limit, and mark the result complete: false.
  • Plain .json is no longer scanned for TODOs since it can't contain comments; .jsonc still is.
  • The scanners share a 32 MiB budget for source being read or parsed; small files still run across the whole thread pool and a single file is always admitted.
  • Reads that grow between the size check and the read are truncated at the limit.
  • The inspect contribution cache version moves to 43 so results from old unbounded scans aren't reused.

Peak RSS for a TODO scan on Linux fell from 625 MB to 73 MB, and the combined TODO and metrics scan is about 10.7% faster than main on AFT's own tree. The JavaScript/TypeScript dead-code and unused-export readers still read whole files; bounding them is a follow-up.

Written for commit d0c9243. Summary will update on new commits.

View guided diff Turn on auto-fix

ualtinok and others added 7 commits November 3, 2026 00:17
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Exclude plain JSON from TODO reads as not applicable. Reuse the 1 MiB search source limit, admit files before opening, and hold one of two shared permits across source reads, parsing and traversal. Surface oversized files through existing skipped coverage and invalidate persisted Tier-2 results.

Skipped UTF-8 inputs are also omitted from scanned-file counts rather than counting unreadable inputs as analyzed. Size-limit wording remains pending Ufuk's approval as requested by the reviewer.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…lism (#1337)

Replace the process-wide two-file cap with a 32 MiB stat-sized reservation budget. Keep reservations live through freshness reads, source reads, parsing and traversal; allow a lone file to proceed even if its clamped reservation exceeds a smaller budget.

The delayed-parser regression now fills the byte budget with 32 simultaneous 1 MiB inputs and blocks the 33rd, replacing the previous two-file contract at the reviewer's request. Add small-file admission and single-file progress tests plus isolated Linux throughput/native tree-memory probes.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…e-size bump

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
@ualtinok
ualtinok added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit ab821c8 Nov 3, 2026
93 of 94 checks passed
@ualtinok
ualtinok deleted the land/todo-scan-bounds branch November 3, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant