Skip to content

feat: add distributed Lance column updates - #58

Draft
FANNG1 wants to merge 4 commits into
daft-engine:mainfrom
FANNG1:feat/update-columns-df
Draft

feat: add distributed Lance column updates#58
FANNG1 wants to merge 4 commits into
daft-engine:mainfrom
FANNG1:feat/update-columns-df

Conversation

@FANNG1

@FANNG1 FANNG1 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add update_columns_df for fragment-parallel updates driven by a prepared Daft DataFrame
  • validate unique live row addresses, safely cast update values, and commit one atomic Lance RewriteColumns transaction
  • support URI and namespace-addressed tables
  • reject stable-row-ID datasets before fragment writes until pylance can provide the metadata needed for correct CDF updates
  • honor commit_lock in the final driver-side commit

This complements the SQL-expression update API proposed in #13 / #26. It covers values computed by arbitrary distributed Daft pipelines and writes them back to existing columns by row address.

Closes #57

Semantics

The input DataFrame must contain _rowaddr, fragment_id, and every target column named in columns. Workers reopen the driver-pinned snapshot, validate the rows, rewrite touched fragments, and return fragment metadata. The driver performs one LanceOperation.Update(update_mode="rewrite_columns") commit against the pinned read version, using the caller-provided commit_lock when present.

Invalid, duplicate, deleted, or stale row addresses fail without committing. Empty input is a no-op. Values are cast to the target Arrow schema with safe=True.

Test plan

  • Native full suite: 366 passed, 5 skipped, 2 xfailed, 2 xpassed
  • Ray update, driver-side commit-lock, and stable-row-ID fail-fast scenarios: 3 passed
  • make check-format
  • strict mypy checks for changed files
  • uv build

Coverage includes partial multi-fragment updates, no-op behavior, validation and safe casting, deleted/stale addresses, stable row IDs, namespace tables, fixed-size lists, scalar-index correctness, physical file preservation, historical versions, and custom commit locks.

Known limitation

Current pylance bindings do not expose updated_fragment_offsets for RewriteColumns. Those offsets are required to advance _row_last_updated_at_version and keep CDF metadata correct, so this API rejects stable-row-ID datasets before DataFrame execution or fragment writes. Support can be enabled once the binding exposes the required offsets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support distributed DataFrame-driven updates of existing Lance columns

1 participant