Skip to content

fix(verify): drop cached crates a run did not rebuild when their sources changed - #4419

Open
richiemcilroy wants to merge 2 commits into
mainfrom
fix/verify-cache-stale-units
Open

richiemcilroy wants to merge 2 commits into
mainfrom
fix/verify-cache-stale-units

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Pull requests that don't touch crates/enc-ffmpeg fail to compile in Engine verify on macOS, even when they are level with main:

error[E0433]: failed to resolve: could not find `decode_limit` in `cap_enc_ffmpeg`
   --> crates/video-decode/src/ffmpeg.rs:414:33

main has pub mod decode_limit; in crates/enc-ffmpeg/src/lib.rs. The cached build of cap-enc-ffmpeg that cargo reused does not.

Why

sequenceDiagram
    participant S1 as Saving run, day N
    participant S2 as Saving run, day N+1 (main)
    participant PR as Pull request run
    S1->>S1: builds a, and a with feature x
    S1->>S1: record: every file = its blob
    S2->>S2: a/src/lib.rs changed, apply leaves it new
    S2->>S2: builds only the suites it needs (a without x)
    S2->>S2: record: a/src/lib.rs = new blob
    Note over S2: a with feature x is still the day-N build
    PR->>PR: apply: a/src/lib.rs matches manifest, gets the old mtime
    PR->>PR: cargo trusts a with feature x as fresh
    PR-->>PR: compiles against day-N code
Loading
  • Each run builds only the crates its planned suites need (scripts/verify/plan.py), but saves the whole target/ci-verify, including units earlier runs built with other feature sets.
  • mtime.py record stamped every tracked file with its current blob id. Units the saving run never rebuilt then looked up to date to the next run.
  • When a public item is added, the result is a compile error like the one above. When behaviour changes, a pull request's suites silently run against older code for crates the pull request does not touch.

Fix (scripts/verify/mtime.py)

  • apply writes a stamp (<manifest>.applied) before the build.
  • record reads every rustc dep-info file under the target (deps/*.d and build/*/build_script_build-*.d). If a unit's dep-info predates the stamp, the run did not rebuild it. If it also lists a file whose blob changed since the restored manifest, record deletes the unit's .fingerprint/<pkg>-<hash> directory. The next run that needs that unit rebuilds it.
  • Units this run rebuilt or reused unchanged keep their fingerprints, so warm builds stay warm.
  • Manifests carry a # mtime manifest v2 header. A manifest without it is not trusted (every file keeps the checkout time). This flushes the units that are already stale in today's caches. Pull requests rebuild workspace crates until the first main run of a UTC day saves a v2 manifest.
  • record without a stamp writes an untrusted manifest.

On a 21 GB local target with 4,656 dep-info files, the extra record work takes about 3 s.

scripts/verify/test_mtime.py builds real three-crate workspaces through the same apply → build → record cycle CI runs:

Case Before After
Library unit with another feature set, not rebuilt by the saving run cannot find function added in crate a rebuilt, compiles
Build script not rerun by the saving run cannot find function added in crate a rerun, compiles
Unchanged dependency reused reused

The new test runs next to test_perf_build.py in Engine verify's planner checks.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

The PR prevents later verification runs from reusing cached crates built from older sources.

  • Engine verify rebuilds cached units when their sources change.

No new actionable issues found. The previous build-script input finding is addressed. The tests could not run here because Cargo is not installed.

Reviews (3) · Last reviewed commit: "fix(verify): drop build script runs whos..." · Reviewed by Greptile

@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 20, 2026 12:40am UTC

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

Comment thread scripts/verify/mtime.py Resolved
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@richiemcilroy
richiemcilroy marked this pull request as ready for review November 20, 2026 00:54
@sandchest-com

sandchest-com Bot commented Nov 20, 2026

Copy link
Copy Markdown

Sandchest

Sandchest reviewed b21293b: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · Review details

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant