feat(realtime): support user-defined-key deduplication for append tables

[lxy-9602]

Purpose

Linked issue: #158

Add a high-performance real-time deduplication mode for append tables.

The mode keeps append writes efficient while using deletion vectors to hide replaced or deleted rows in real time. Data files remain ordinary append-table files, so all predicates and file indexes can still be used for pruning and pushdown across data files, sealed segments, and building data.

Rows are deduplicated with last-write-wins semantics using a user-defined key, providing efficient real-time updates without introducing merge-tree read overhead.

This change adds:

• realtime.deduplicate-key-fields;
• key-to-offset indexes for memory and data files;
• deletion-vector generation for replaced or deleted rows;
• consistent reads across data files, sealed segments, and building data;
• committed snapshot refresh and memory reclamation.

Current limitations: one non-null key field, fixed buckets, no data evolution or real-time spill, and _REALTIME_OFFSET requires a bitmap file index.

Tests

Added unit and integration coverage for:

• schema and option validation;
• key/offset index lookup;
• INSERT, DELETE, and UPDATE semantics;
• deduplication across batches, segments, and committed files;
• read-view stability, partition/bucket isolation, and predicate pushdown.

API and Format

API impact: adds the deduplicate option and real-time store mode (realtime.deduplicate-key-fields).

Format impact: deduplicate tables persist _REALTIME_OFFSET, key-to-offset sidecars, and deletion vectors. Existing table modes are unchanged.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

[zjw1111]

Three issues in the realtime deduplication path.

[zjw1111]

One inline comment on key-offset index memory accounting.

[zjw1111]

Please add user-facing documentation for the realtime feature, covering append tables and primary-key tables, their behavior in realtime mode, and configuration and usage examples for enabling it. Please also document the additional schema and option requirements for realtime deduplication, including _REALTIME_OFFSET, realtime.deduplicate-key-fields, deletion vectors, and the offset bitmap index.

[lxy-9602]

Please add user-facing documentation for the realtime feature

This feature is still under development. We don’t expect external users to use it yet. Once it’s complete, we’ll add the corresponding user documentation.