Repository navigation
fix(core): recover SQLite index rebuilds and large vector deletions - #1865
Merged
Merged
Conversation
… limit A single UPDATE with one 4-term AND predicate per pending chunk exceeds SQLITE_MAX_EXPR_DEPTH above ~995 deletions. Delete in batches of 200 and keep the pending-row match semantics. Add a 1200-row regression test. Signed-off-by: MbappeWU <MbappeWU@users.noreply.github.com> Signed-off-by: phernandez <paul@basicmachines.co>
Drop a leftover search_index_rebuilt before CREATE so an interrupted d2e3f4a5b6c7 upgrade can retry instead of failing with table already exists. Regression covers the empty leftover-table case from #1853. Signed-off-by: MbappeWU <MbappeWU@users.noreply.github.com> Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This was referenced Oct 25, 2026
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Two reported SQLite failures block reindexing or startup: deleting a large chunk manifest exceeds SQLite's expression-depth limit (#1850), and an interrupted script-ngram migration leaves a scratch table that prevents the next startup (#1853). Both reproduce on current main with the contributor's regression tests.
Fixes #1850. Fixes #1853.
What Changed
search_index_rebuiltscratch table before rebuilding the intact source index.Implementation Details
Deletion retains the existing project, key, source-hash and pending-status guards in one transaction. Batching limits expression depth and bound parameters without changing the concurrency protocol.
The migration rebuilds from
search_index; the scratch table has no authoritative state. Integration tests commit the interrupted state before opening a new transaction and verify preserved full-text matches afterward. They use Alembic's real Operations context and sqlite-vec without application mocks. Contributor workflow-evidence notes are omitted from the final diff.Testing
Expression tree is too largeandtable search_index_rebuilt already exists.uv run pytest tests/repository/test_sqlite_vector_search_repository.py tests/test_script_ngram_search_migration.py test-int/test_sqlite_index_recovery.py -q -o addopts= --tb=short: 46 passed.just fast-check: lint, format and full typecheck passed.just doctor: sandboxed write, index and search passed.git diff --check: passed.Risks / Follow-ups
Changes are SQLite-specific; PostgreSQL production paths are unchanged. The recovery assumes the original FTS table remains intact, matching the reported interrupted upgrade. #1861's watcher lifecycle changes are reviewed separately.