feat(vector): add dedicated Vortex storage for append-only tables

[ChaomingZhangCN]

Purpose

Linked issue: None.

Add dedicated Vortex storage for a single top-level VECTOR column in append-only tables, building on the Vortex format support in #376. With vector.file.format=vortex, ordinary columns are written to Parquet main files and the VECTOR column is written to .vector.vortex files. Both files cover the same row range and are read together through the existing data-evolution reader.

The dedicated mode requires:

file.format=parquet
vector.file.format=vortex
bucket=-1
row-tracking.enabled=true
data-evolution.enabled=true

• Support full-row INSERTs with exactly one top-level VECTOR and at least one ordinary non-partition column.
• Roll main/VECTOR files together when either size target or the row-count target is reached; preserve matching row ranges and clean up both files on write/close failure.
• Add vector.target-file-size, defaulting to target-file-size.
• Reject unsupported dedicated-mode combinations: primary-key tables, custom/partial write schemas, multiple or nested VECTOR columns, and BLOB columns. Independent VECTOR file rolling is outside this implementation.

Separately, allow VECTOR columns in ordinary file.format=vortex tables. Validate visible VECTOR elements using ancestor validity before normalizing hidden NULL slots for Vortex import, so NULL STRUCT/LIST ancestors do not cause valid writes to fail. Visible NULL elements are still rejected for non-nullable VECTOR elements.

This is a scoped implementation of dedicated VECTOR storage; it does not claim complete feature parity with Paimon Java.

Tests

Java interoperability is tested using checked-in files actually written and read back by Paimon Java 2.2-SNAPSHOT at commit 7d6d146bbaf33a349dc83c3caf71334bd551e3d5 (Vortex JNI 0.73.0). The fixture contains two append commits, six rows including two NULL vectors, and two Parquet/Vortex file pairs. Its generator, runtime dependency POM, provenance, and reproduction commands are included.

DataEvolutionTableTest.TestReadJavaDedicatedVectorStore reads the fixture without rewriting it and verifies full rows, ordinary-column and VECTOR-only projections, row ranges crossing commits, and paired file metadata (firstRowId, row counts, and writeCols). VortexFileFormatTest.WriteThenReadVectorsWithNullAncestors covers NULL STRUCT/LIST parents, batch boundaries, multiple read batch sizes, and rejection of the same invalid elements when the ancestors become visible. The NULL-ancestor regression fails before the fix and passes afterward.

Local validation on macOS aarch64, C++17 Debug, with Vortex enabled:

Validation	Result
Core feature regression: rolling VECTOR writer, schema validation, options, append writer, row tracking, paths, BLOB rolling, and data evolution utilities	118 passed at feature commit dff3266
Complete paimon-data-evolution-table-test after the fix	294 passed, 12 skipped by format/parameter guards
Complete paimon-vortex-format-test after the fix	26 passed
paimon-write-and-read-inte-test --gtest_filter='Vortex*' after the fix	4 passed
Java fixture generation and Java readback assertions	Passed
Applicable changed-file pre-commit hooks and git diff --check	Passed

Whole-file clang-format was skipped to preserve unrelated existing formatting; new C++ files and modified lines were checked with clang-format 20.1.8. The existing local macOS build uses temporary dependency/link workarounds and still emits duplicate-library and deployment-target linker warnings. Linux CI, the repository-wide unittest target, sanitizers, and C++-written data read by Java have not been run.

API and Format

• Add exported option constants Options::VECTOR_FILE_FORMAT and Options::VECTOR_TARGET_FILE_SIZE in include/paimon/defs.h.
• Reuse existing .vector.vortex naming and data-evolution metadata (writeCols, firstRowId, and row counts); no new manifest version or protocol is introduced.
• Dedicated storage is selected explicitly by vector.file.format=vortex; schemas and write contexts are validated before using this writer.

Documentation

• Update docs/source/user_guide/data_types.rst with Vortex VECTOR support, dedicated-mode prerequisites, limitations, and paired rolling behavior.
• Add fixture provenance and reproducible Java generation instructions in test/test_data/vortex/dedicated_vector_java_compat.db/dedicated_vector_java_compat/README.md.

Generative AI tooling

Generated-by: OpenAI Codex (codex-cli 0.162.0-alpha.17.2).