Skip to content

Expose Datagen history reads and retain folded codec metadata - #252

Merged
beinan merged 1 commit into
lance-format:mainfrom
YangjunZ:datagen/read-history-codec-metadata
Sep 11, 2026
Merged

Expose Datagen history reads and retain folded codec metadata#252
beinan merged 1 commit into
lance-format:mainfrom
YangjunZ:datagen/read-history-codec-metadata

Conversation

@YangjunZ

Copy link
Copy Markdown
Contributor

Summary

  • Expose Python item_events / root_events through embedded and remote stores, preserving overwritten values, per-item ordering and lazy blobs. Encode fan-out IDs safely in HTTP paths.
  • Retain field_type / codec_version in folded fields. SET metadata follows the winning value; APPEND rejects mixed codecs instead of mislabeling accumulated values.
  • Refresh cached server handles before raw-history scans so external WAL merges cannot leave a stale, nonempty prefix. Add regression coverage and document read consistency.

Compatibility

No event-schema migration or changes to Context, Rollout, Generic, or shared MemWAL behavior. Python field dictionaries gain metadata keys; older server responses expose missing metadata as None, not an inferred codec.

Rust users must update DatagenFieldState variants, DTO literals and custom DatagenStoreApi implementations. Mixed-codec append histories now fail to fold. Deploy the updated server before using the new item-history endpoint.

Validation

  • Full Rust workspace: 375 passed, 25 ignored. Final server regressions passed after reproducing and fixing the stale-prefix case.
  • Full Python suite, including real HTTP and local/S3 persistence: 229 passed, 1 failed, 1 skipped, 1 xfailed.
  • Formatting, Ruff, workspace Clippy and default-feature Clippy passed.
  • Native Python module and server built locally; independent Opus 5 review and focused follow-up completed.

Existing baseline issues, both reproduced on pristine 8fe27b2 and left out of scope:

  • test_persistence.py::test_image_round_trip reads zero base-table rows after the image write.
  • With optional Pillow installed, Pyright reports the existing Image.get_format_mimetype access at api.py:210.

The Python skip is opt-in GCS integration; the expected failure is the existing MemWAL/base-version time-travel test. No pre-commit hook or configuration is installed in this repository.

🤖 Generated with MAI Agents

Expose embedded and remote raw event history, retain folded codec metadata, and reject mixed append codecs. Refresh raw-history server handles before scans to avoid truncated histories after external WAL merges. Add regression tests and compatibility documentation.

Generated with MAI Agents: https://github.com/infinity-microsoft/mai-agents

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a563b220-9ce6-4724-97e4-601ec8ffd89a
@beinan
beinan marked this pull request as ready for review September 11, 2026 17:01
@beinan
beinan merged commit 71316b3 into lance-format:main Sep 11, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants