fix: reduce DiskANN index storage overhead - #279
Merged
lindixu6-hash merged 1 commit intoAug 30, 2026
Conversation
lindixu6-hash
approved these changes
Aug 30, 2026
lindixu6-hash
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed the new-shard index definition and direct-query coverage. The change is intentionally scoped to newly created content/tag indexes; existing-shard rebuild and VACUUM remain a separate migration concern. Local verification at 88221cd: targeted vector tests 3/3, full suite 447/447 with 1 skip, typecheck, and production build all pass. The approved GitHub package-smoke matrix is also green on Ubuntu, Windows, macOS Intel, and Apple Silicon.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
float8for both content and tag indexesmax_neighborsat 20 so high-dimensional embeddings do not inherit the much larger libSQL defaultvector_top_klookups through both tuned indexesScope
This deliberately changes newly created indexes only. Existing shards retain their current index definitions until a separate failure-safe rebuild/VACUUM migration is designed, so this patch does not risk leaving an existing shard without an index.
Addresses the new-shard portion of #277.
Validation
bun test tests/turso-vector-search.test.ts -t "inserts and searches memories"— 1 passbun run typecheck— passbun run build— passEBUSYfile rename in the legacy migrator test and the existing 30-second 200-vector ANN test timing out at 35.5 seconds. The latter was slower and eventually crashed Bun when repeated with the unmodified upstream index definition.