feat: add distributed Lance column updates - #58
Draft
FANNG1 wants to merge 4 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
update_columns_dffor fragment-parallel updates driven by a prepared Daft DataFrameRewriteColumnstransactioncommit_lockin the final driver-side commitThis complements the SQL-expression update API proposed in #13 / #26. It covers values computed by arbitrary distributed Daft pipelines and writes them back to existing columns by row address.
Closes #57
Semantics
The input DataFrame must contain
_rowaddr,fragment_id, and every target column named incolumns. Workers reopen the driver-pinned snapshot, validate the rows, rewrite touched fragments, and return fragment metadata. The driver performs oneLanceOperation.Update(update_mode="rewrite_columns")commit against the pinned read version, using the caller-providedcommit_lockwhen present.Invalid, duplicate, deleted, or stale row addresses fail without committing. Empty input is a no-op. Values are cast to the target Arrow schema with
safe=True.Test plan
make check-formatuv buildCoverage includes partial multi-fragment updates, no-op behavior, validation and safe casting, deleted/stale addresses, stable row IDs, namespace tables, fixed-size lists, scalar-index correctness, physical file preservation, historical versions, and custom commit locks.
Known limitation
Current pylance bindings do not expose
updated_fragment_offsetsforRewriteColumns. Those offsets are required to advance_row_last_updated_at_versionand keep CDF metadata correct, so this API rejects stable-row-ID datasets before DataFrame execution or fragment writes. Support can be enabled once the binding exposes the required offsets.