Fixes #73: invalidate the incremental cache on extractor change, not just file change - #71
Open
learncoder4848 wants to merge 1 commit into
Open
Conversation
…hash Module.content_hash was a SHA-256 of the file's bytes alone, so nothing in the stored fingerprint reflected how the file was parsed -- no grammar identity, no digest of the language pack's query files, no extractor schema stamp. After a release that fixes an extractor (3.2.16 shipped both the Java method_reference capture and the C++ namespace-scoping fix), every existing index kept the old, wrong graph: unedited files still matched their stored hash, so they were skipped indefinitely unless the user knew to run index --full. LanguagePack now carries a fingerprint built from its entities.scm / relations.scm sources, grammar identity, custom edge definitions and a new EXTRACTOR_SCHEMA_VERSION; content_fingerprint folds that into the per-file hash. Invalidation is scoped per pack, so a Java query fix re-extracts Java files and leaves other languages cached. The skip gate in index_via_backend and extract_file each computed a file hash independently -- if the two ever disagree, nothing is skipped and every index silently becomes a full reindex -- so both now route through content_fingerprint. Grammar plugins are covered by plugin_fingerprint over plugin.toml, both ANTLR grammars and the extractor source. No schema migration: content_hash is already an opaque string in every backend. 13 new tests, including an end-to-end check that an upgraded extractor re-extracts unedited files and then settles rather than reindexing forever. Full workspace suite (1263 tests), fmt, clippy and --features remote all pass. Co-authored-by: Sarvesh Sawant <sarveshdev92@gmail.com>
learncoder4848
requested review from
WinterQuant,
johnintuit,
murari316 and
sandeep-mewara
as code owners
August 30, 2026 06:50
Author
|
@murari316 @johnintuit @sandeep-mewara @WinterQuant Can you guys review the changes ? |
learncoder4848
changed the title
Author
|
@murari316 @johnintuit @sandeep-mewara @WinterQuant @pradeepmouli Reminder |
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #73 .
Module.content_hashonly covers the file's bytes, soindex()skips any file whose contents haven't changed — even when the extractor that parsed it has. Extractor fixes therefore never reach an existing index. 3.2.16's Javamethod_referenceand C++ namespace-scoping fixes both silently missed anyone who didn't happen to runindex --full.LanguagePacknow carries a fingerprint built from itsentities.scm/relations.scmsources, the grammar's identity, its custom edge definitions, and a newEXTRACTOR_SCHEMA_VERSIONfor Rust-side changes.content_fingerprintfolds that into the per-file hash.It's scoped per pack, so a Java query fix re-extracts Java files and leaves Python and TypeScript cached. Bumping a single global version would also work, but would mean a full reindex on every release.
Two places computed the file hash independently — the skip gate in
index_via_backendandextract_file. Both now go throughcontent_fingerprint. If those ever drift apart nothing gets skipped and every index quietly becomes a full one, so there's a test pinning it.Grammar plugins get the same treatment via
plugin_fingerprint, over the plugin'splugin.toml, its.g4grammars and its extractor source.No schema migration —
content_hashis already an opaque string in every backend.One caveat: tree-sitter exposes no grammar version, so grammar identity is inferred from
abi_versionplus the node-kind / field / parse-state counts. An upgrade leaving all four unchanged would be missed. Rust-side extractor changes still need a manualEXTRACTOR_SCHEMA_VERSIONbump. Both are documented indocs/CODE-PARSING.md.Test plan
cargo test --all— 1263 passed, 0 failed (13 new)cargo fmt --all -- --checkandcargo clippy --all-targets -- -D warnings— cleancargo check --features remote -p infigraph-core -p infigraph-cli -p infigraph-mcp— clean (CI builds default features only; the release workflow uses--features remote)cargo test -p infigraph-core --test index_perf -- --ignored—test_incremental_index_skips_unchangedstill skips everything, confirming the skip gate and the stored hash agreeThe main new one is
test_extractor_change_reindexes_unedited_files: index a project, reopen with the same pack (nothing re-indexed), reopen with a changed pack (all re-indexed), reopen again with the changed pack (nothing). That last step is the one that matters — it proves this doesn't turn into a permanent reindex loop.Notes
Adding a private field to
LanguagePackis technically breaking for struct-literal construction downstream, so this probably wants a minor bump rather than a patch.