feat(transaction): action-based transaction wire format (Transaction V2)

[wjones127]

Defines the wire format for action-based transactions (Transaction V2): a CompositeOperation that replaces the single legacy Operation with an ordered list of granular, composable Action deltas. This is the format change only — read-side fail-closed, no write path. Every protobuf change the implementation stack needs lands here, so the format is a single vote; the PRs stacked on top of it change no .proto file.

Layout

The V2 vocabulary lives in its own files under protos/transaction/:

• actions.proto — the V2 vocabulary (Ref, CompositeOperation, UserAction, Action + the action messages), with the design rationale summarized in an in-tree file header so it stands on its own.
• common.proto — shared building blocks (UpdateMap, KeyExistenceFilter, …) referenced by both legacy operations and V2 actions, promoted from nested Transaction messages to top-level to avoid a circular import. Wire-compatible: field numbers unchanged and none are Any-packed.
• transaction.proto — moved into the directory; keeps the Transaction envelope, legacy operations, and the new composite_operation oneof arm (field 116).

What's in the wire format

CompositeOperation { uuid, read_version, [UserAction] }              
UserAction    { description, [Action] }        // human-readable step; survives squash
Action        { oneof of 19 actions }
Ref           { committed: uint64 | local: uint32 }   // one ref type for field/fragment/base ids

Design principles (fully documented in actions.proto):

• Deltas, not post-images — actions record the change (AddDataFile, TombstoneFieldData), so compound commits and branch merge fall out uniformly.
• Minting vs. reference-stable — minted ids (field/fragment/base) carry a Local token via a single Ref and relocate on merge/rebase; reference-stable changes key off stable coordinates and keep a post-image with a derivable delta.
• Field-level schema — AddField / DropField / identity-preserving AlterField (per-facet, so a concurrent cast and nullability change on the same field commute). No wholesale SetSchema.
• Index segments — AddIndexSegment / RemoveIndexSegment / AdjustIndexCoverage (a logical index is the segments sharing a name; per-segment config duplication is a pre-existing limitation this draft doesn't fix).
• Assertions — AssertUniqueKeys carries merge-insert's non-derivable key-existence filter as an explicit precondition.
• Off the wire — computed conflict footprints and large derivable row-level deltas (deletion affected-rows, update matched-offsets) are recomputed at conflict time, not serialized.
• Additive-safe — message identities and core mutation fields are pinned; discoveries land as added optional fields.

data_change markers (Delta-style, for CDC skip of compaction) and index-coverage representation carry in-tree TODOs to finalize.

Settled by the implementation

These started out in the PRs stacked above and were pulled down here so the format is voted on once:

• UserOperation becomes CompositeOperation, naming what distinguishes it from the other operations, and loses its description — the per-step UserAction.description is what keeps history readable. UserAction keeps its name: it is the grouping a user recognizes.
• The field actions reference fields by Ref, so an action can name a field the same operation is minting.
• AddIndexSegment gains base, created_at, and dataset_version, and its covered_fragments becomes an optional wrapper rather than a bare repeated. A bare repeated field cannot distinguish "no coverage recorded" — what the MemWAL and fragment-reuse indices carry, and what the query path treats as "serve this segment" — from "covers no fragment", which it treats as "skip it". dataset_version is a correctness gate rather than provenance: an overlay committed at or before it counts as already folded into the index.
• RemoveIndexSegment and AdjustIndexCoverage carry the name of the logical index the segment belongs to, which is what conflict detection compares.
• The update_compacted_ss_tables oneof arm is named to match its UpdateCompactedSsTables message, which is what the action vocabulary keys the wire encoding off.

Library changes (minimal, fail-closed)

Read-side rejection only:

• One arm in TryFrom<pb::Transaction> returns Error::NotSupported for a CompositeOperation, with a comment forbidding lenient parsing (the try_collect contract aborts a concurrent V2 commit rather than silently skipping it).
• Regression test test_composite_operation_rejected_on_load.

No Operation enum variant, no apply / translation / conflict resolution, no write path, no Cargo feature gate, no Python/Java changes.

Verification

lance-table + lance build; new test passes; cargo fmt --all and cargo clippy -p lance-table -p lance --tests -- -D warnings clean.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (2)

• WIP
• Draft

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: d6c3e873-ac4a-40f7-89f6-e462d4db9be4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch will/oss-1530-draft-action-based-transaction

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

❌ Patch coverage is 87.75510% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-table/src/format/key_existence.rs	44.44%	5 Missing ⚠️
rust/lance-table/src/transaction/proto.rs	97.50%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The delta/action direction solves the real atomic-commit and N×N conflict-growth problem and is stronger than composing legacy post-images. The current proposal is not yet a self-contained experimental wire contract: its governing lifecycle and forward-reader prerequisite are not in force, and core payloads cannot yet be interpreted losslessly or unambiguously. Land the governance dependency first, keep unaware readers able to open affected datasets, use the outer transaction envelope as the single source of identity/version, and preserve covering-index semantics.

[lance-gatekeeper]

This new oneof arm is unknown to pre-v12 readers. When a composite transaction is inlined, prost drops the unknown arm and those releases propagate the resulting missing-operation error from Dataset::open. #9454 verifies the v1–v11 exposure, and the only matching fix is #7740 on main; no maintained-release backport PR exists. That fails #8304's prerequisite that feature writers not affect unaware readers. Before accepting this field, either make V2 transactions external-only until the compatibility floor moves (and specify/test that writer rule), or land the maintained-release backports first.

[wjones127]

Backports are unfortunately unlikely to be realistic. We have a lot of different packages that depend on Lance and bake it into their binaries, and many of them are size constrained on releases.

Realistically, I think the best solution we can do here is add a strong warning that the datasets written with this won't be backwards compatible for now. This is strictly an opt-in feature to use these. When we actually stabilize it, we'll make it a writer flag to use these. Maybe this bug means it will also be a reader flag.

[lance-gatekeeper]

An opt-in warning does not satisfy the forward-compatibility boundary this PR relies on: the proposed experimental policy requires writers using the feature not to affect unaware readers, while an inlined V2 transaction still prevents pre-v12 Dataset::open. The current head therefore still needs either an external-only writer rule or a compatible reader floor before this field can be accepted.

[wjones127]

Added, in ea4b6f3f and 0d8addd2. Agreed the backports are not realistic, so the docs now treat this as durable rather than transitional.

Four places a writer will actually hit it:

• the composite_operation oneof arm here, with the mechanism spelled out
• the actions.proto file header, short-form
• a !!! danger admonition in the transaction specification, stating that a table with one anywhere in its history has a minimum reader version of v12.0.0
• Rust: the action module's Stability section, a # Compatibility section on CommitBuilder::with_experimental_composite_operations, and the NotSupported message a caller gets without opting in -- that error is the first and possibly only place they read about it

All of them say the same thing explicitly: the version cannot be opened, not merely read as history.

On the writer/reader flag at stabilization -- agreed, and #9454 does look like it makes the reader-flag case. Worth settling on the discussion when we get there rather than now.

[lance-gatekeeper]

The warnings accurately document the break, but this finding is about the acceptance contract rather than discoverability. #8304 requires experimental writers not to affect unaware readers, while pre-v12 Dataset::open still fails. This therefore still needs an external-only writer rule or a compatible reader floor before merge.

[wjones127]

Well I think the only solution would be to create a reader feature flag. But that would also cause Dataset::open to fail, just with a nicer error message. Would adding that reader flag make things acceptable? One thing that makes me not like the idea of a reader flag is this is experimental and when we stabilize it we'll actually want the flag to mean "I understand the stable version of the protobuf". So on some level I'm fine with the experimental version just not being compatible. It's entirely opt-in: A user would have to manually construct a transaction with a CompositeOperation. So it's not unlike using an unstable file format version. I'm inclined to let it just fail as is while it's experimental, and add a reader flag upon stabilization.

[lance-gatekeeper]

A reader flag that still makes Dataset::open fail would not meet #8304's forward-compatibility prerequisite. Opt-in changes writer exposure, but the resulting stable table remains unreadable to an unaware reader. If the governing policy is changed and approved to permit that consequence, the acceptance contract changes; under the current proposed policy, this finding remains.

[lance-gatekeeper]

CompositeOperation repeats both Transaction.uuid and Transaction.read_version, so the wire format permits two transaction identities and two source versions without saying which copy is authoritative or requiring equality. The outer values already drive transaction-file naming and rebase checkout; an action implementation that reads the inner values can therefore apply against a different source state. This also conflicts with the protobuf rule to store each fact once. Remove the inner copies and make the outer Transaction authoritative—the existing alternative in #7737 already uses that invariant—rather than asking every implementation to reconcile contradictory valid messages.

[wjones127]

These are intentional. The idea is that if multiple transactions were squashed into a single one, you would still be able to derive the original identity and read versions.

[lance-gatekeeper]

The stated intent is not representable by this schema: squashing flattens each original CompositeOperation into a UserAction, but UserAction carries neither uuid nor read_version, and there is only one inner pair. The inner pair therefore cannot preserve multiple originals, while disagreeing inner and outer values remain valid but uninterpretable. Please encode per-original metadata at the repeated level or remove the duplicate pair.

[wjones127]

Oh, good point. I don't know why it's at this level. It should be at UserAction level then.

[lance-gatekeeper]

Agreed—that places identity and read version at the repeated level that represents each original operation. The current head still keeps the singleton pair on CompositeOperation while UserAction lacks it, so this finding remains until the fields move or the duplicate pair is removed.

[lance-gatekeeper]

Fixed in d1bc81a8: CompositeOperation no longer duplicates transaction identity or read version; the enclosing Transaction is now the sole source for both values.

[lance-gatekeeper]

AddIndexSegment cannot represent the current IndexMetadata.covering_fields declaration. fields contains keyed columns followed by carried columns, and covering_fields is the only state that splits them; losing it makes selection and maintenance treat payload columns as keys and can omit the covering-index reader feature flag. Add a non-duplicating representation (for example, the validated count of trailing covering fields) and round-trip a covering segment.

Reproducer

printf 'fields { committed: 1 } fields { committed: 2 } covering_fields { committed: 2 }\n' | \
  protoc -I protos --encode=lance.table.AddIndexSegment protos/transaction/actions.proto

The observed head fails with Message type "lance.table.AddIndexSegment" has no field named "covering_fields", while IndexMetadata accepts the equivalent keyed/carried declaration.

[wjones127]

This is a good point. Will address.

[lance-gatekeeper]

Thanks. The current head still cannot encode the covering-field split, so I am keeping this finding until the promised representation and round-trip evidence land.

[wjones127]

Addressed. AddIndexSegment now carries covering_fields (field 12) alongside fields, and it takes #9159's contract rather than main's: fields is the keyed columns only, covering_fields is independent of it, and the segment's dependency set is the union. The action has no historical encoding to stay compatible with, so there was no reason to be born with the legacy suffix convention.

The gap is that no release can store that contract yet — #9159 reserves FLAG_INDEPENDENT_COVERING_FIELDS with mark_supported(..., false), so no build writes or opens a manifest that sets it. Rather than block the action on that, apply lowers:

• Disjoint declaration (fields = [1], covering_fields = [2]) → legacy IndexMetadata { fields: [1, 2], covering_fields: [2] } under FLAG_COVERED_INDEX_METADATA. Commits today, and the lowering drops out once the flag is implemented.
• Overlapping declaration (fields = [1], covering_fields = [1, 2]) → rejected with NotSupported naming the flag. Flattening it would publish the weaker claim that the column is merely keyed, silently losing the fact that the segment serves its values.

The action's own shape does not change when the flag lands; only the lowering and the rejection come out.

The wire message and the format doc are in this PR (actions.proto, plus the covering paragraph in docs/src/format/table/transaction.md); the apply behaviour and its tests are in #8645.

[lance-gatekeeper]

Fixed in be2dd47e: AddIndexSegment now carries independent covering_fields, the format text defines the keyed/carried union contract, and the protobuf round-trip preserves both lists.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Author feedback clarified that original transaction identity and read version may be retained for squashed operations; that metadata must be carried per original operation, because the current single inner pair cannot represent multiple originals or resolve disagreement with the outer envelope. The remaining contract is unchanged: land #8304 first, keep experimental writers forward-compatible with unaware readers, and preserve covering-index field roles. A warning alone does not satisfy the explicit forward-compatibility prerequisite, and the promised covering-index representation is not yet in this head.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new warning now accurately states the pre-v12 open failure, but it documents rather than removes a compatibility break that the experimental policy forbids. This revision still needs the same self-contained wire contract: land #8304 first, keep unaware readers able to open affected datasets, encode identity and read version per original squashed operation instead of as one conflicting singleton pair, and preserve covering-index field roles. The delta/action design remains worthwhile once those constraints are met.

[wjones127]

Decision

AddIndexSegment declares keys and carried columns independently, per #9159: fields is the keyed columns only, covering_fields is not a subset of it, and the segment's dependency set is the union. Until a release implements FLAG_INDEPENDENT_COVERING_FIELDS, apply lowers a disjoint declaration to the legacy manifest form (fields = keys ++ carried, covering_fields = carried as the trailing suffix, fenced by FLAG_COVERED_INDEX_METADATA) and rejects an overlapping one with NotSupported.

Status

Settled 2026-09-21. Wire message and format doc on #7954; apply behaviour and tests on #8645 (lower_covering, two tests in add_index_segment.rs).

Options and criteria

Criteria, in order: (a) the action is a new encoding with no history, so it should carry the contract we want rather than the one IndexMetadata inherited; (b) nothing the action commits today may be unrepresentable in a manifest a current build can write; (c) no silent loss of a declared property.

1. Lower, reject overlap — chosen. Satisfies all three. The lowering and the rejection both delete themselves when the flag ships; the action's shape never changes.
2. Adopt main's contract (covering_fields a trailing subset of fields) — satisfies (b) and (c), fails (a): the action would be born with a convention feat(format): separate index keys from covering fields #9159 is already replacing, and would need a wire change to catch up.
3. Adopt feat(format): separate index keys from covering fields #9159's contract and write the flag — fails (b). feat(format): separate index keys from covering fields #9159 registers the bit with mark_supported(..., false), so no build can write or open such a manifest; the action would be able to express commits nothing can read.

Rejected, with the reason and when it was tried

• Flatten an overlapping declaration (drop the overlapping ids from covering_fields, keep them in fields) — rejected 2026-09-21, not implemented. It commits, but the manifest then claims only that the index is keyed on the column, silently dropping the writer's statement that the segment also serves its values. A query planner reading that manifest would decline a covering scan the index could actually answer. A rejection the caller sees is better than a downgrade it does not.

Open questions

• When FLAG_INDEPENDENT_COVERING_FIELDS becomes supported, the lowering and the rejection come out together and apply writes the union form directly. Nothing in the action or its wire encoding changes, so this needs no format revision — but it does need the flag's own vote to land first.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

1 fixed / 3 remain. The covering-field decision is now represented directly in AddIndexSegment and round-trips, so that blocker is closed. The proposal still needs an admissible experimental wire contract: land #8304 first, keep pre-v12 readers able to open V2-authored tables rather than only documenting the break, and encode identity and read version per original squashed operation instead of as a conflicting singleton duplicate. The delta/action design remains worthwhile once those constraints are met.

[wjones127]

Decision

CompositeOperation no longer carries uuid or read_version. The message is now just repeated UserAction actions = 1. Transaction identity lives on the enclosing Transaction alone.

Status

Done, in d1bc81a (proto) and 9abec50 on #8644 (the Rust conversions that existed only to populate them).

Options and criteria

The criterion was whether either field can ever carry information the envelope does not already have.

The justification for them was provenance across a future transaction squash. That justification does not survive contact with the squash design in this same file: squashing collapses each original CompositeOperation into one UserAction, so the message holding the identity is exactly the message squashing destroys. Pre-squash the fields duplicate the envelope; post-squash they are overwritten. There is no state in which they carry anything.

That was confirmed against the code rather than assumed. They were write-only — stamped from the envelope on the way out, dropped on the way in; the Rust struct never modelled them. No commit or conflict path consults a per-operation read version: ApplyState takes it from the Transaction (apply.rs:77), and check_action_txn compares only footprints.

Options weighed:

1. Move both to UserAction, optional, absent = inherit from the envelope. The level that survives squash, next to description, which serves the same readable-history purpose. This is the right shape if the fields are wanted.
2. Drop them now; add to UserAction when squashing exists. Chosen.
3. Keep as-is. Rejected outright — nothing reads them.

(2) over (1) because the feature that would justify the fields does not exist, and the current misplacement is itself evidence that they are hard to place correctly before it does. Adding an optional field later is cheap; voting.md commitment 3 permits breaking changes to experimental features without a separate vote, and this PR is not yet merged, so nothing has ever been written with them.

Rejected, with the reason and when it was tried

• Keeping the fields (option 3), 2026-09-21. Write-only in every path; no reader anywhere in rust/, python/, or java/ — the Java bindings reject CompositeOperation before any field access.
• Moving them to UserAction now (option 1), 2026-09-21. Correct placement, but speculative: it encodes a squash model that has no implementation, and a format vote should not be asked to ratify a provenance scheme whose consumer does not exist. Revisit when squashing lands — this is the shape to adopt then.
• reserved 1, 2; instead of renumbering actions to 1, 2026-09-21. Three for three, every removed field elsewhere in protos/ is reserved by number and name, so the convention does point that way. It does not apply here: reserved protects against old data using the number, and this message has never shipped. This file also states that its field numbers are not a stable contract. A gap at 1 and 2 in a message being read for a vote invites a question with no answer.

Open questions

• If per-step provenance does land on UserAction, read_version there is best-effort audit metadata, not a resolvable reference: the version it names may have been cleaned up, and the {read_version}-{uuid}.txn file with it. That needs saying in the proto comment, or a reader will try to dereference it. If we cannot say what a reader should do with it, that is an argument for not adding it at all.
• apply.rs:608 suggests squash provenance is expected to ride on per-action stamped fields (created_at_version_meta / last_updated_at_version_meta) rather than on any per-operation or per-step version. If that holds, option 1 may never be needed.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

3 fixed / 1 remains. The duplicate identity/read-version fields are gone, joining the already-closed governance and covering-field findings. The remaining compatibility contract is unchanged: an inlined V2 transaction makes pre-v12 readers unable to open the table, while the merged experimental-feature policy requires writers using the feature not to affect unaware readers. Keep V2 transactions external-only until the compatibility floor moves (and specify/test that writer rule), or establish an equivalent path that leaves those readers able to open the table. The delta/action design remains worthwhile once that prerequisite is met.

[wjones127]

I'm going to fix #9454 before marking this ready for review