[JingsongLi]

Purpose

Upserts, including the Python binding, currently plan a target scan without the incoming key predicates. Existing BTree indexes therefore cannot narrow key lookup. Push per-column key candidates into scan planning while retaining complete tuple matching for duplicates, NULLs, and partition identity.

Brief change log

• Use full scalar-index search for mutation lookups so partially indexed tables still match existing rows.
• Preserve index row ranges when reading keys and row IDs, without evaluating a large residual IN list for every row.
• Under IGNORE, compare relevant indexes' source snapshots against active files' key-column sequence numbers before pruning. Unchanged keys retain BTree acceleration after payload updates; changed or unverified keys use a complete lookup. Other current update policies bypass this freshness check. Inspection and lookup use one pinned snapshot.
• Keep freshness verification in global_index_freshness, independent of Arrow input and the upsert executor. The caller supplies match columns and canonical partition identities. Build a centered file-range tree once per relevant partition/bucket, then inspect only overlapping physical column files for each index shard. Cache file row ranges and canonical split partitions rather than traversing all files per shard; nested and long ranges retain O(log F + overlaps) lookup cost.
• Keep fallback for floating-point and BLOB keys to preserve the existing matcher's supported types and equality semantics.
• Keep partition matching canonical instead of pushing raw partition predicates, so Java-padded NULL partition encodings still match existing rows.
• Allow large linear IN domains through composite indexes. Keep the 256-entry budget for Cartesian expansion, retaining an already-bound left prefix and independent suffix indexes when expansion exceeds it. Batch complete tuple point lookups with query_in and cache normalized domain membership to avoid quadratic planning. Sort non-point prefix intervals and reuse their final data block, so even cache-disabled prefix queries read each target block once.

Tests

• Added regressions for actual BTree row ranges, partial/missing indexes, duplicate targets, partial updates, changed keys, deletion vectors, nullable composite indexes, partitions, floating-point/BLOB fallback, and stale key indexes retained by IGNORE.
• Verified IGNORE still prunes unchanged keys and payload-only updates, honors per-column compaction sequences with compact or explicit write-column metadata, falls back for untrusted source metadata, and resumes pruning after index rebuilds. THROW_ERROR and DROP_PARTITION_INDEX retain BTree pruning without requiring source metadata. Added a regression for Java-padded NULL timestamp partitions.
• Added composite regressions for 4096 unsorted/duplicate values, large-domain intersections, bounded Cartesian prefixes, suffix scalar indexes, NULL/NaN/signed-zero/decimal identity, and exact upsert updates/appends. A 257-key lookup narrows 4096 target rows to 256 existing matches; 4096 shuffled tuple points and prefix intervals require one read per target block with caching disabled, including suffix ranges that span multiple blocks.
• Added metadata-range regressions for inclusive endpoints, nested/duplicate ranges, per-column sequences and source baselines, irrelevant partitions/buckets, and unknown-range fallback. Deterministic work-count tests cover 5000 disjoint shards plus a covering file, and large crossing lists queried from either edge.
• BTree, upsert, key normalization, matcher, global-index freshness, and global-index scanner suites: 217 passed, 2 existing ignored tests. Command: cargo test -p paimon --no-default-features --features storage-memory,storage-fs --lib -- btree:: table::table_upsert table::key_normalization table::upsert_key_matcher table::global_index_freshness table::global_index_scanner.
• Formatting and git diff --check passed.

API and Format

No public API or storage-format changes.

Documentation

Existing upsert API behavior is preserved; the change optimizes key lookup.

[leaves12138]

LGTM