Match dependency and generated-output paths at any depth in FS_FORBIDDEN_READ

[shellygr]

Problem

FS_FORBIDDEN_READ (composer/spec/util.py) decides what the agent's source tools —
list_files / get_file / grep_files — are allowed to read. It had two defects.

1. Its directory rules were anchored at the project root (^node_modules/, ^lib/,
^test/, ^\.git, ^\.certora_internal). In a monorepo where the Foundry project lives in
a subdirectory, none of them match: the project's dependency tree sits at
<pkg>/<sub>/node_modules/.... There was also no rule for built web output.

A recent dev run died in the Custom Summaries phase with

AnthropicContextOverflowError: prompt is too long: 1285703 tokens > 1000000 maximum

A single grep_files(search_string='function consume', matching_lines=True) returned
2.53 MB in one tool result. Only 13 files matched — this is not a file-count problem.
~95% of the bytes came from five built frontend bundles, each inlining a minified JS payload
whose longest single line is 2.5–4.8 MB. grep_files reports whole matching lines, so one
match in such a file returns megabytes. Minified content also tokenizes at roughly two
characters per token, so 2.53 MB ≈ 1.28M tokens.

2. It withheld Solidity. Only node_modules had a .sol carve-out. Root lib/ and
test/ were excluded outright — so in a stock Foundry layout, every dependency contract and
every test contract was invisible to the agent.

Change

• Solidity is never withheld. The carve-out node_modules already had is hoisted into a
  single guard over every dependency, scaffolding and generated-output rule. Any .sol is
  readable wherever it sits.
• node_modules, lib, test and dist now match at any depth. With Solidity
  protected globally, lib/ and test/ no longer need to be root-only to keep dependency
  source reachable, so they are uniform with the rest.
• New rules for machine-generated output: dist/ trees, minified and bundled JS
  (.min.js / .bundle.js and the - / _ separator variants), source maps, packed
  data blobs.
• Prover working directories and VCS internals stay withheld whole — .certora_internal,
  .git, emv-*. This is the one place the Solidity carve-out deliberately does not reach:
  their .sol is a verbatim copy of a contract already readable at its canonical path, since
  each certoraRun invocation materializes its own inputs/.certora_sources/** (a tree can
  accumulate several), and analysis that wants a specific report's copy reaches it through a
  VFS scoped to that report rather than through this surface. Applying the carve-out here
  would hand grep_files N duplicate copies of every contract.

Measured effect

Applying the master and branch patterns to the source bundle of the run that failed:

	master	this PR
grep-visible files	914 (209.1 MB)	661 (9.5 MB)
visible .sol	597 / 597	597 / 597
longest visible line	4,802,095 B	6,406 B
the grep_files call that overflowed	2.53 MB (~1.27M tok)	1.5 KB

That project has no root lib//test/, so the Solidity invariant does not bite there. On a
stock Foundry layout it does:

path	master	this PR
lib/openzeppelin-contracts/contracts/token/ERC20.sol	hidden	readable
test/Widget.t.sol	hidden	readable
lib/forge-std/README.md	hidden	hidden
emv-1-.../inputs/.certora_sources/src/Widget.sol	hidden	hidden

Scope

This is path hygiene, and it is deliberately not the general fix. A path pattern cannot
express "this file has megabyte-long lines", so a sufficiently odd tree can still produce an
oversized tool result. The general fix is an output cap in graphcore's _grep_impl, which has
no bound of any kind today; that is a separate change in a separate repo and is not included
here. Two related weaknesses also remain out of scope: the summarization router reads token
usage off the preceding AI message and so cannot see a tool result that just landed, and the
summarizer re-sends the full history and silently swallows its own overflow.

Testing

• New tests/test_fs_forbidden_read.py — 6 tests covering the Solidity invariant (vendored,
  nested, root lib//test/), the any-depth rules, generated output, the withheld-whole
  prover directories, and false positives on hand-written files whose names end in
  bundle.js / min.js.
• pyright clean on the changed file.
• Locally the suite is not a clean signal: 6 modules cannot be collected (certora_cli
  missing) and test_rag_db needs a running Docker daemon for its testcontainers fixtures.
  Counts are identical with and without this change, so CI is the authoritative gate here.

🤖 Generated with Claude Code

[jtoman]

I made a mistake making this only a regex. The actual solution is to generalize the regex only to:

type GlobalExcludeArg = str | Callable[[pathlib.PurePath], bool] | None

which we already use for the global exclude predicate. This lets us, declaratively, write something like allow all .sol and express arbitrary properties instead of doing ... whatever tf is going on with these regexes. The machinery to turn GlobalExcludeArg into a predicate to apply within the tool implementations in vfs.py exists, we just need to use it for the forbidden read/write bits and push it up into this repo.

[shellygr]

Claude answers: agreed, and done in #120 — the regex is replaced by a predicate over the
path, with the type widened to GlobalExcludeArg at every site the filter flows through.
Two PRs because the machinery change lands in graphcore:

• Accept a predicate for forbidden_read / forbidden_write graphcore#29 — widens forbidden_read / forbidden_write from str | None to
  GlobalExcludeArg, and folds both surface forms into a single _make_exclude_pred so
  _make_checker and _make_global_include_pred differ only by the .git floor.
• Express the source-tool read filter as a predicate #120 — FS_FORBIDDEN_READ becomes fs_forbidden_read, and the four rule classes it
  distinguishes get names. allow all .sol is now literally if path.suffix == ".sol": return False. --forbidden-read still accepts a regex from the CLI, which is why the
  regex arm stays in graphcore.

The tests from this PR are carried over to #120 unchanged apart from the harness (predicate
instead of re.fullmatch), so their 6/6 green is the evidence the rewrite preserved the
behaviour they pin down. Two intentional deltas are called out in #120's description —
.gitignore becomes readable, since .git is now matched as an exact path component
rather than by prefix.

Leaving this PR open rather than closing it: a run is currently going against this
worktree's image. Happy to close it in favour of #120 once that finishes.