Add end-to-end Parquet VARIANT support

[xiangfu0]

Summary

Add an end-to-end VARIANT user journey for Apache Pinot:

• define a single-value VARIANT dimension retained in Pinot's raw forward index
• ingest top-level, non-repeated Apache Parquet VARIANT(1) values from unshredded and shredded files
• optionally materialize frequently queried scalar paths during ingestion while retaining the original value
• expose Spark-style parse, extraction, type, existence, null, and JSON-rendering functions
• carry Variant through Pinot schemas, segments, query plans, responses, DDL, Java/JDBC clients, and both query engines
• provide a packaged batch quickstart plus integration and mixed-version compatibility coverage

Raw Variant values are deliberately opaque to operations that require equality, hashing, or ordering. Queries must extract a typed scalar before comparing, grouping, joining, sorting, using set operations, or applying general aggregates. COUNT(raw_variant) remains supported.

The persisted format, null semantics, ownership boundaries, and rollout contract are documented in pinot-spi/VARIANT_DESIGN.md.

End-to-end flow

Define a table column with "dataType": "VARIANT", storage null handling enabled, dictionary disabled, and a raw forward index. The native Parquet reader reconstructs VARIANT(1) metadata/value pairs—including supported shredded layouts—into Pinot's versioned PVAR envelope.

Frequently queried fields can be materialized with ingestion transforms while the full Variant remains available:

SET enableNullHandling=true;

SELECT eventType, COUNT(*)
FROM variantEvents
GROUP BY eventType
ORDER BY eventType;

SELECT
  eventId,
  variant_get(payload, '$.user.id', 'STRING') AS userId,
  variant_get(payload, '$.amount', 'DOUBLE') AS amount
FROM variantEvents
WHERE eventType = 'checkout'
ORDER BY eventId;

SELECT eventId, variantToJson(payload)
FROM variantEvents
ORDER BY eventId;

Supported scope

• top-level, non-repeated Parquet VARIANT(1) columns
• single-value Pinot VARIANT dimensions without dictionaries
• raw forward-index retention and optional scalar materialization
• storage null handling enabled by schema or table configuration
• query-time Variant functions with SET enableNullHandling=true
• single-stage and multi-stage query execution

Nested/repeated Variant columns, streaming ingestion, and quoted path keys are outside the initial scope.

A JSON index is intentionally not accepted on a VARIANT column. The existing JSON index expects Pinot JSON text and its query rewrite semantics; treating the binary PVAR envelope as JSON would be incorrect. Indexing Variant paths should be introduced separately with an explicit physical/index contract. Materialized scalar columns are the supported indexed path for this PR.

The automatic Parquet reader selection preserves existing Avro-metadata precedence. Files containing Avro metadata must explicitly select the native Parquet reader to ingest Variant.

Production safety

• centralizes PVAR framing and validation in VariantEnvelope and freezes version-1 bytes with golden tests
• assigns protobuf wire value 24 to Variant while preserving UUID/UUID_ARRAY at 22/23
• rejects unsupported schemas, table/index configurations, query null-handling modes, and raw Variant operations
• validates Parquet decimal precision, scale, bounds, hostile exponents, malformed rows, and buffer ownership
• handles array-backed, direct, sliced, and read-only Parquet buffers without changing source positions or limits
• compiles constant paths/types once and reuses cursors in ingestion, single-stage, and multi-stage execution
• distinguishes SQL null, encoded Variant null, the string "null", missing paths, and conversion failures across query and client response formats
• keeps Parquet column/schema dependencies isolated to the input plugin

Rolling upgrades

Variant remains an explicit full-fleet activation feature: upgrade controllers, brokers, servers, clients, and external ingestion jobs before registering a Variant schema, and do not roll back while Variant tables are active.

Compatibility coverage verifies:

• all-old components reject the unknown wire type deterministically
• a new broker with old servers fails with upgrade guidance
• a mixed old/new server fleet rejects the whole query without returning partial rows
• an all-new fleet succeeds
• an old broker with upgraded servers remains queryable for the verified path
• capability checks, readiness retries, expected-error matching, and rollback are bounded and deterministic

Quickstart

./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar
build/bin/quick-start-variant-batch.sh

The quickstart registers variantEvents, ingests the committed five-row Parquet fixture, uploads the segment, and runs materialized-field, nested-extraction, JSON-rendering, and null-semantics queries. Its resources live under pinot-tools/src/main/resources/examples/batch/variantEvents.

Verification

• 19 VariantTypeTest scenarios across both query engines in the full integration-test reactor
• all-old, new/old, mixed-server, all-new, old-broker/new-server, and rollback compatibility flows
• post-rebase 53-module reactor covering protobuf compatibility, UUID/Variant serde, 229 planner compilation cases, Variant validation, 122 window cases, and 11 compatibility-verifier tests
• spotless, checkstyle, license formatting, and license checks across all 16 affected modules
• clean three-commit history on current upstream/master
• git diff --check

The JSON-versus-Variant performance harness and workstation measurements are intentionally excluded from this feature PR and will be proposed separately so the production API and compatibility contract can be reviewed without benchmark-suite scope.

Review areas

This PR changes public SPI and query-wire behavior. The main ownership reviews are:

• SPI, schema, and query-wire compatibility
• Parquet and input-format ingestion
• single-stage and multi-stage query semantics
• Java, JDBC, Arrow, JSON, and gRPC response behavior
• rolling-upgrade activation and rollback

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.38160% with 401 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.82%. Comparing base (c500e4f) to head (b1a2872).

Files with missing lines	Patch %	Lines
...va/org/apache/pinot/common/utils/VariantUtils.java	73.49%	118 Missing and 71 partials ⚠️
...n/inputformat/parquet/ParquetVariantConverter.java	80.98%	14 Missing and 13 partials ⚠️
...ransform/function/VariantGetTransformFunction.java	77.35%	12 Missing and 12 partials ⚠️
...uery/runtime/operator/operands/VariantOperand.java	82.95%	5 Missing and 10 partials ⚠️
...not/common/evaluator/InbuiltFunctionEvaluator.java	82.35%	4 Missing and 8 partials ⚠️
...pinot/query/runtime/operator/HashJoinOperator.java	47.82%	9 Missing and 3 partials ⚠️
...java/org/apache/pinot/common/utils/DataSchema.java	62.96%	4 Missing and 6 partials ⚠️
...n/inputformat/parquet/ParquetAvroRecordReader.java	75.00%	9 Missing and 1 partial ⚠️
...java/org/apache/pinot/spi/utils/PinotDataType.java	59.09%	8 Missing and 1 partial ⚠️
...e/pinot/common/function/TransformFunctionType.java	83.33%	5 Missing and 2 partials ⚠️
... and 32 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19101      +/-   ##
============================================
+ Coverage     65.60%   65.82%   +0.21%     
- Complexity     1423     1496      +73     
============================================
  Files          3450     3464      +14     
  Lines        218378   220337    +1959     
  Branches      34670    35015     +345     
============================================
+ Hits         143275   145028    +1753     
+ Misses        63538    63537       -1     
- Partials      11565    11772     +207     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	65.82% <80.38%> (+0.21%)	⬆️
temurin	65.82% <80.38%> (+0.21%)	⬆️
unittests	65.81% <80.38%> (+0.21%)	⬆️
unittests1	57.22% <76.23%> (+0.22%)	⬆️
unittests2	38.00% <37.18%> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adds end-to-end support for the VARIANT logical type across Pinot ingestion (notably Parquet VARIANT(1)), query planning/execution (single-stage + multi-stage), response encoders, and Java/JDBC client surfaces, while explicitly rejecting unsupported raw-VARIANT operations (e.g., ordering/grouping/join keys) with actionable errors.

Changes:

• Introduces VARIANT as a first-class Pinot data type, including schema/DDL handling, wire/proto support, and null-sentinel semantics.
• Adds Parquet VARIANT(1) ingestion support via the native Parquet reader and includes a packaged “variant batch” quickstart with resources and CI coverage.
• Adds query-time VARIANT functions (parse/get/existence/type/json rendering) and enforces operator-level restrictions where raw VARIANT lacks equality/hash/ordering semantics.

Reviewed changes

Copilot reviewed 174 out of 175 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pom.xml	Adds managed dependency for org.apache.parquet:parquet-variant.
pinot-tools/src/test/java/org/apache/pinot/tools/admin/command/QuickStartTest.java	Maps new quickstart command strings to VariantQuickStart.
pinot-tools/src/main/resources/examples/batch/variantEvents/*	Adds VARIANT batch quickstart schema/table config, ingestion spec, and documentation.
pinot-tools/pom.xml	Adds packaged quick-start-variant-batch program entry.
pinot-sql-ddl/src/main/, src/test/	Adds DDL compile/reverse/roundtrip support for VARIANT and validates default emission rules.
pinot-spi/src/main/, src/test/	Adds PinotDataType.VARIANT conversion/ingestion mapping and schema validation updates.
pinot-segment-spi/src/main/**	Handles VARIANT default-null sentinel persisted as empty string in segment metadata.
pinot-segment-local/src/main/, src/test/	Adds VARIANT validation/sanitization rules; disables min/max, ordering, dictionary creation where unsupported; adds targeted tests.
pinot-query-planner/src/main/, src/test/	Adds VARIANT type mapping (Calcite/plan nodes), plan validation, constant-folding guardrails, and serde coverage.
pinot-query-runtime/src/main/, src/test/	Enforces VARIANT restrictions at runtime operators (sort/window/set/group-by/etc.) and adds operator tests.
pinot-core/src/main/, src/test/	Enforces VARIANT restrictions in single-stage operators/aggregations/distinct/order-by and adds tests.
pinot-common/src/main/, src/test/	Adds VARIANT to encoders, datablock equality, proto enum, function canonicalization, and introduces scalar VariantFunctions.
pinot-clients/pinot-jdbc-client/src/main/, src/test/	Treats VARIANT as canonical JSON text (VARCHAR) while preserving encoded-variant-null vs SQL-null behavior.
pinot-clients/pinot-java-client/src/main/, src/test/	Preserves legacy getString() null behavior while distinguishing VARIANT nulls.
pinot-plugins/pinot-input-format/pinot-parquet/src/main/**	Enhances Parquet reader selection/metadata helpers and initializes native reader with schema-bound converters for VARIANT.
pinot-controller/src/main/, src/test/	Ensures stored-vs-compiled schema default comparisons handle VARIANT byte[] defaults by content.
pinot-compatibility-verifier/**	Adds file-contains operation and VARIANT mixed-version suite coverage including old-broker/new-servers phase.
.github/workflows/scripts/.pinot_quickstart.sh	Adds CI validation for the packaged VARIANT batch quickstart.