Skip to content

feat(index): record and carry the JSON index target data type - #9304

Open
wjones127 wants to merge 1 commit into
feat/json-index-details-protofrom
feat/json-index-details-target-data-type
Open

wjones127 wants to merge 1 commit into
feat/json-index-details-protofrom
feat/json-index-details-target-data-type

Conversation

@wjones127

@wjones127 wjones127 commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Populates JsonIndexDetails.target_data_type, added to the format in #9303, so a JSON index's persisted details describe the type its target index was trained on instead of leaving a rebuild to re-infer it.

The recorded type is read back when an index is loaded and copied forward when it is compacted or updated, rather than asked of the target index each time — only BTree targets can answer that, so re-deriving would keep losing the type for bitmap and every other target on every compaction, which is the case the issue is about. It is held as the raw enum value so a variant written by a newer build survives a compaction performed by this one, as the format requires.

Details written before the field existed are upgraded in passing wherever the type can be recovered: always on the update path, which already requires the target to report the type it was trained on, and for BTree targets on the remap path. derive_index_params reports the type so a rebuild reproduces it, and list_indices reports it too.

Not included

JsonIndex::training_data_type is deliberately unchanged, so update on a non-BTree target still fails as before. That guard is load-bearing: BitmapIndex::update takes its value type from the incoming stream and would merge mismatched keys rather than erroring.

uv run make lint was not run: uv sync builds the pylance extension, which needs a local python/Cargo.toml edit in this worktree. ruff and ruff-format were run over the diff via pre-commit instead.

Closes #9256

🤖 Generated with Claude Code

Stack created with GitHub Stacks CLI • Give Feedback 💬

@wjones127
wjones127 added this pull request to stack #9305 September 16, 2026 16:08
@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer enhancement New feature or request labels Sep 16, 2026
@wjones127
wjones127 marked this pull request as ready for review September 16, 2026 16:22
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
@wjones127
wjones127 force-pushed the feat/json-index-details-target-data-type branch from a0f6a06 to 25d428d Compare September 16, 2026 17:43
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026
Populate `JsonIndexDetails.target_data_type`, added to the format in the
previous PR, so a JSON index's persisted details describe the type its
target index was trained on instead of leaving a rebuild to re-infer it.

`JsonIndex` carries the recorded value, parsed from the details at load,
and `remap` and `update` copy it forward rather than re-deriving it:
`ScalarIndex::training_data_type` is implemented only by BTree, so
re-deriving would drop the type for every other target on every
compaction, which is the case the issue is about. It is held as the raw
enum value so a variant added by a later build survives a compaction
performed by this one, as the format requires. Details written before
the field existed are upgraded in passing where the type can be
recovered: always on the `update` path, which already requires the
target to report the type it was trained on, and for BTree targets on
the `remap` path.

`derive_index_params` reports the type, so a rebuild reproduces it, and
`details_as_json` reports it, so it is visible in `list_indices`. Both
spell it the same way.

`JsonIndex::training_data_type` is deliberately unchanged, so `update`
on a non-BTree target still fails as before: that guard is load-bearing,
since `BitmapIndex::update` takes its value type from the incoming
stream and would merge mismatched keys rather than erroring.

Closes #9256

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@wjones127
wjones127 force-pushed the feat/json-index-details-target-data-type branch from 25d428d to d2f44d9 Compare September 16, 2026 19:19
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The implementation patch is unchanged after the rebase. It records the trained target type, preserves raw values through load, remap, and update—including unknown future variants—and recovers legacy UNSPECIFIED only from the target index, matching the clarified format contract in #9303. Derived rebuild parameters carry the stored type forward.

#9303 remains the prerequisite and should land first.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer A-python Python bindings enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JsonIndexDetails cannot represent target_data_type, so details are insufficient to rebuild a JSON index

1 participant