Repository navigation
[core] Preserve append writer configuration per bucket across prepares - #1188
Conversation
JeonDaehong
left a comment
There was a problem hiding this comment.
Checked the "existing writers retain their settings" part against Java (paimon-bundle 2.1.0). It matches while the writer is alive. After a bucket sits idle for one committed round, Java drops the writer and the next write uses the current settings, while here the old settings stay. Details inline.
| .cloned() | ||
| .unwrap_or_else(|| self.append_write_config()); | ||
| let writer = self.create_append_writer(partition_path, bucket, &config)?; | ||
| self.append_write_configs.insert(key, config); |
There was a problem hiding this comment.
Nothing removes an entry from append_write_configs, so a partition/bucket keeps its first settings for the whole life of the TableWrite. Java only keeps them while the writer is alive. AbstractFileStoreWrite.prepareCommit drops a writer that wasn't modified since the last committed identifier, and the next write to that bucket creates a writer with the current writeType / BlobConsumer.
I checked with paimon-bundle 2.1.0, DE append table (pt, id, a, b) partitioned by pt:
- Write
(1, 1, 10, 100), thenprepareCommit+ commit. - Write
(9, 9, 9, 9)to another partition, thenprepareCommit+ commit. This is the round where pt=1 is idle. withWriteType(pt, id, a), write(1, 2, 20), commit.
Java reads back pt=1 id=2 a=20 b=null. With no idle round in between, Java fails in step 3 (ArrayIndexOutOfBoundsException), which matches what this PR keeps.
The same sequence here (with_write_type([pt, id]), write pt=1, commit, write pt=9, commit, with_write_type([pt, omitted]), write pt=1) fails with field name mismatch at index 1: expected 'id', actual 'omitted', after 0, 1 or 2 idle rounds.
Should a config be dropped once its bucket had no writes in a committed round, like the Java writer? That would also bound this map for a long-running writer that touches many partitions.
…col-unification # Conflicts: # crates/paimon/Cargo.toml # crates/paimon/src/table/internal.rs
Purpose
Keep one declared append write type throughout a
TableWritelifetime and preserve Java's Blob consumer lifecycle. Current Java callers configure their row type before writing; bucket writers capture that schema and consume rows positionally.Changes
with_write_typebefore the first effective write. Repeating the same ordered selection is a no-op; changing columns or order after writing is rejected, including after prepare or idle cleanup. Empty and ignored input does not lock the configuration._row_write_columns, its Python stubs and Core resolver, along with historical schema lookup and per-bucket write-column caches. Native rows encode the selected columns and required routing/event fields in one batch.prepare_commit()primitive unchanged. It conservatively retains fixed-bucket context without a checkpoint identifier; bindings forward their existing batch/stream identifiers through the internal bridge.internalmodule declaration.Verification
fulltext,vortex; formatting and diff checks.No Native extension was rebuilt or installed into the Python test environments. Matching PyPaimon adapter, Pure Python lifecycle, regression tests and documentation changes are in apache/paimon#10724.