From 3802cff99efd1f19fec75a996972d63ad600003a Mon Sep 17 00:00:00 2001 From: comphead Date: Mon, 26 Oct 2026 14:13:07 -0700 Subject: [PATCH 1/5] Add blog post: nested TPC-H benchmark for Comet --- content/blog/2026-10-25-comet-nested-tpch.md | 210 +++++++ .../images/comet-nested-tpch/dashboard.html | 520 ++++++++++++++++++ .../images/comet-nested-tpch/intro-total.svg | 1 + .../comet-nested-tpch/results-queries.svg | 1 + 4 files changed, 732 insertions(+) create mode 100644 content/blog/2026-10-25-comet-nested-tpch.md create mode 100644 content/images/comet-nested-tpch/dashboard.html create mode 100644 content/images/comet-nested-tpch/intro-total.svg create mode 100644 content/images/comet-nested-tpch/results-queries.svg diff --git a/content/blog/2026-10-25-comet-nested-tpch.md b/content/blog/2026-10-25-comet-nested-tpch.md new file mode 100644 index 00000000..1b784cfc --- /dev/null +++ b/content/blog/2026-10-25-comet-nested-tpch.md @@ -0,0 +1,210 @@ +--- +layout: post +title: Announcing a derived TPC-H, a benchmark for nested data in Apache DataFusion Comet +date: 2026-10-25 +author: pmc +categories: [performance, benchmarks] +--- + + + +[TOC] + +Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark. Before the fixes this benchmark exposed, it took 1,088 s. + + + +Nested TPC-H is a TPC-H derived benchmark in the Comet repository ([#7044](https://github.com/apache/datafusion-comet/pull/7044)). It keeps the TPC-H data and questions and changes only the shape of the tables. The data and queries are engine neutral, so they can be used to compare Spark, Comet and other engines. + +We built it because our benchmarks and our users disagreed. Comet tracked its performance with TPC-H and TPC-DS, and the numbers were encouraging. Real user pipelines showed smaller gains, or none at all. What those pipelines had in common was nested data, which neither benchmark contains. + +So we started experimenting with nested benchmarks in Comet to see where the gaps were. The first runs showed them: some queries ran slower on Comet than on Spark, even though Comet accelerates the same queries on flat TPC-H. + +## Why: TPC-H is flat, real data is not + +[TPC-H](https://www.tpc.org/tpch/) is one of the most cited analytical benchmarks, and its schema is flat: eight normalized tables with primitive columns. Its 22 queries exercise well-understood choke points such as aggregation, join ordering and correlated subqueries (Boncz, Neumann and Erling, "TPC-H Analyzed", TPCTC 2013). + +Real data is often nested. Event logs, JSON, Avro and Protobuf records, and many data lake tables are full of structs and arrays, which Parquet stores with the record shredding scheme from Dremel (Melnik et al., VLDB 2010). Nested data adds work that flat benchmarks never exercise: + +- **Reading.** Rebuilding list offsets and validity bitmaps from repetition and definition levels, and decoding only the fields a query uses. +- **Planning.** Nested column pruning and predicate pushdown through `explode`. +- **Operators.** `explode`, field access, and sort, join and exchange over wide variable-size values. +- **Memory and spilling.** Variable-size values make memory use and spill volume harder to predict. +- **Conversion.** Moving nested values between a JVM and a native representation. + +Comet hit the shuffle case in [#5397](https://github.com/apache/datafusion-comet/issues/5397), where native shuffle writes of wide nested fields were two to three times slower than Spark's. The issue notes that it was "invisible for TPC-* benchmarks because of historical flat schema". + +We needed a benchmark that fills the gap between academic benchmarking and real user workloads. + +## How: an adapter that nests TPC-H + +We did not want to invent a new benchmark from scratch. TPC-H is well understood, widely published and easy to generate, and its business shape (customers, orders, line items, suppliers and parts) is what makes its results meaningful. The goal was to preserve that shape and those questions while making every query process nested data. + +The idea turned out to be surprisingly simple: a transformation of both the data and the queries that leaves the key and partition columns at the top level and nests everything else. The partition columns are `p_brand`, `c_mktsegment`, `o_orderdate` and `l_shipdate`, and every other non-key column moves into one `array` named `_data`. Each array holds exactly one element, so exploding it never changes the row count and the queries return the same answers as TPC-H. For `lineitem`: + +``` +-- flat -- nested +l_orderkey BIGINT l_orderkey BIGINT +l_partkey BIGINT l_partkey BIGINT +l_suppkey BIGINT l_suppkey BIGINT +l_linenumber INT l_shipdate DATE -- partition column +l_quantity DECIMAL(12,2) lineitem_data ARRAY> +``` + +Each of the 22 queries is rewritten to unnest only the arrays it needs. The queries are in [`benchmarks/nested/queries`](https://github.com/apache/datafusion-comet/tree/main/benchmarks/nested/queries). + +```sql +from (select l_orderkey, l_partkey, l_suppkey, l_shipdate, + explode(lineitem_data) as lineitem_data from lineitem) as lineitem +where lineitem_data.l_quantity < 24 +``` + +[`nest_tpch.scala`](https://github.com/apache/datafusion-comet/blob/main/benchmarks/nested/scripts/nest_tpch.scala) converts an existing flat TPC-H dataset to the nested layout. The source and the target can each be Parquet or Iceberg. The script runs under `spark-shell` on a laptop or a cluster, casts columns to fixed types, writes zstd-compressed Parquet with 512 MiB row groups, partitions by the columns above and checks row counts: + +```bash +spark-shell -i scripts/nest_tpch.scala \ + --conf spark.nesttpch.source=parquet:/data/tpch/sf10 \ + --conf spark.nesttpch.target=parquet:/data/tpch-nested/sf10 +``` + +The [README](https://github.com/apache/datafusion-comet/tree/main/benchmarks/nested) has the Iceberg variants and all settings. + +## Benchmark setup + +- **Data**: nested TPC-H, scale factor 1000 (1 TB). +- **Tables**: Apache Iceberg tables with zstd-compressed Parquet data files on S3, created with `nest_tpch.scala`. Comet reads them with its native Iceberg scan, which is built on [iceberg-rust](https://github.com/apache/iceberg-rust). +- **Cluster**: Kubernetes with 16-core executors. +- **Method**: one run per configuration, three iterations per query. Figures are the fastest iteration, and totals are sums of the per-query times. +- **Spark**: Spark 4.0 +- **Comet nested unoptimized**: Comet 1.1.0 snapshot (revision `a50fe3aea863`) +- **Comet nested optimized**: Comet 1.3.0 snapshot (revision `330b29cf`) + +## What nested benchmarks revealed + +The nested benchmark exposed issues that were invisible on flat TPC-H. In the unoptimized run, Comet was slower than Spark on q20 (32.9 s against 31.5 s) and q21 (490.6 s against 373.5 s), and barely faster on q02 (34.0 s against 34.9 s). Tracing those queries surfaced a lot of technical debt in how nested data is read, shuffled and spilled, in Comet and in the iceberg-rust reader underneath it. + +The issues fell into four groups: + +- **Spilling.** Wide nested values made q21's sort spill several hundred GiB per iteration, and Comet wrote spill files uncompressed. +- **Reading only what a query needs.** Comet's native Iceberg scan decoded every field of a nested column, even when Spark had pruned it to one field. This is why q20 was no faster than Spark. +- **Shuffle.** Native shuffle wrote wide nested fields two to three times slower than Spark. +- **Nested reads in iceberg-rust.** Nested fields read as NULL, some list types could not be read at all, and projection cost grew with the number of nested leaves. + +Spills and shuffle writes happen in flat workloads too. The nested schema changed the data volume or access pattern enough to expose them. + +Most of these issues are fixed. The issues and pull requests, with their measurements, are listed in [Issues and pull requests](#issues-and-pull-requests) at the end of the post. + +## Results + +### Total execution time + +Unoptimized Comet takes `1,087.5` s against `1,515.0` s for Spark (1.39x) and is slower on q20 and q21. Optimized Comet takes `777.2` s (1.95x), 29% less than unoptimized, and is faster than Spark on all 22 queries. + + + +### Execution time by query + +q21 is the longest query: 490.6 s on unoptimized Comet, 373.5 s on Spark and 256.6 s on optimized Comet. + + + +| Query | Spark (s) | Comet unoptimized (s) | Comet optimized (s) | +| ----- | ---: | ---: | ---: | +| q01 | 78.1 | 14.5 | 13.7 | +| q02 | 34.9 | 34.0 | 28.1 | +| q03 | 41.0 | 19.6 | 17.9 | +| q04 | 46.7 | 20.6 | 14.8 | +| q05 | 82.5 | 45.7 | 47.4 | +| q06 | 4.9 | 2.8 | 1.9 | +| q07 | 56.7 | 33.3 | 32.4 | +| q08 | 76.8 | 42.0 | 42.8 | +| q09 | 97.9 | 49.9 | 48.1 | +| q10 | 47.7 | 31.6 | 25.8 | +| q11 | 70.4 | 40.0 | 34.5 | +| q12 | 34.4 | 16.1 | 14.4 | +| q13 | 25.3 | 8.7 | 8.1 | +| q14 | 8.2 | 6.8 | 2.8 | +| q15 | 31.8 | 18.1 | 16.5 | +| q16 | 33.7 | 13.7 | 13.1 | +| q17 | 109.4 | 41.5 | 40.6 | +| q18 | 142.5 | 71.6 | 57.9 | +| q19 | 31.2 | 14.8 | 11.0 | +| q20 | 31.5 | 32.9 | 21.8 | +| q21 | 373.5 | 490.6 | 256.6 | +| q22 | 56.1 | 38.8 | 27.1 | +| **Total** | **1,515.0** | **1,087.5** | **777.2** | + +## What's next + +- **Nested TPC-DS**, with the same rule: keys and partition columns stay at the top level, the rest moves into a single-element `array`. Open questions are which columns count as keys (surrogate keys end in `_sk` and many are nullable), which columns to partition by, and how to rewrite `ROLLUP` and window queries next to `explode`. +- **Other nested shapes**: multi-element arrays, maps and deeper nesting, at the cost of changing the questions the queries answer. +- **More engines**: results for other engines on the same data. + +## Issues and pull requests + +Each item links to the pull request that fixed it, or to the open pull request or issue that tracks it. + +### In Comet + +- **q21 spills uncompressed ([#6467](https://github.com/apache/datafusion-comet/issues/6467), fixed by [#6698](https://github.com/apache/datafusion-comet/pull/6698)).** Spark pushes the `EXISTS` and `NOT EXISTS` predicates below the `explode`, so the whole unpruned `lineitem_data` column goes through the shuffle, sort and joins. The sort spilled several hundred GiB per iteration, and DataFusion writes spill files uncompressed by default. #6698 adds `spark.comet.exec.spill.compression.codec`, which defaults to `lz4`. This is behind q21's regression in the results above. The investigation in #6467 used a separate run on Spark 4, where q21 went from 380 s to 250 s against 367 s for Spark. +- **Nested columns are read in full ([#6724](https://github.com/apache/datafusion-comet/issues/6724), fixed by [#6725](https://github.com/apache/datafusion-comet/pull/6725)).** Comet's native Iceberg scan decoded every field of a nested column even when Spark pruned it. In q20 the `partsupp` scan read 26.4 GiB for a query that needs one field, and #6724 found that scan to be the main cost of Comet's q20. This is why q20 was no faster than Spark in the unoptimized run. #6725 passes the pruned schema, and a microbenchmark of that read went from 624 ms to 62 ms, against 413 ms for Spark. The follow-up [#7019](https://github.com/apache/datafusion-comet/issues/7019) covers a partition column nested inside a pruned struct. +- **Nested shuffle writes ([#5397](https://github.com/apache/datafusion-comet/issues/5397)).** The motivating issue above. Related work is tracked in [#6855](https://github.com/apache/datafusion-comet/issues/6855). + +### In iceberg-rust + +Comet's native Iceberg scan reads data files through iceberg-rust's Arrow reader, so the work on nested reads continued there. + +- **Nested fields of migrated files read as NULL ([apache/iceberg-rust#3374](https://github.com/apache/iceberg-rust/issues/3374), fixed by [#3384](https://github.com/apache/iceberg-rust/pull/3384)).** For data files without embedded field IDs, the reader applied the table's name mapping to top-level columns only. Fields inside a struct, list or map got no ID, so the projection never selected them. #3384 applies the mapping at every depth, as Iceberg Java does. +- **Projection mask cost grows with the number of leaves ([#3394](https://github.com/apache/iceberg-rust/issues/3394), fixed by [#3396](https://github.com/apache/iceberg-rust/pull/3396)).** The reader built a projection mask for every file scan task and hashed every Arrow leaf twice to do it. Nested columns add leaves, so this cost grows with the schema. For a file with 1,000 columns and 10 projected, one call went from 318 µs and 3,976 allocations to 95 µs and 178 allocations. +- **List view columns read as NULL ([#3393](https://github.com/apache/iceberg-rust/issues/3393), fix in review in [#3467](https://github.com/apache/iceberg-rust/pull/3467)).** When a file's embedded Arrow schema asks for `ListView`, the projection treats the whole list as one leaf. The column reads as NULL, and a list view of structs shifts every later column to the wrong Parquet leaf. #3467 reads list views as lists. +- **Fixed-size list columns fail to read ([#3471](https://github.com/apache/iceberg-rust/issues/3471), open).** This is the same kind of problem for `FixedSizeList`. The reader cannot convert it to Iceberg's `list` type. + +## Credits + +Issues, pull requests and results from other engines and scale factors are welcome. See the [Comet contributor guide](https://datafusion.apache.org/comet/contributor-guide/index.html). + +TPC-H is a trademark of the TPC. The queries are derived from TPC-H under the TPC Fair Use Policy, and results from them are not TPC-H results. diff --git a/content/images/comet-nested-tpch/dashboard.html b/content/images/comet-nested-tpch/dashboard.html new file mode 100644 index 00000000..97208ee2 --- /dev/null +++ b/content/images/comet-nested-tpch/dashboard.html @@ -0,0 +1,520 @@ + + + + + +Nested TPC-H SF1000: Comet vs Spark + + + +
+
+
+

Nested TPC-H at SF1000: Comet vs Spark

+

+
+ +
+ +
+ +
+ +
+ +
+

Unoptimized: total run time

+

+
+ +
+ +
+

Unoptimized: time by query

+

Best (fastest) of 3 iterations, in seconds. Hover a row for iterations and speedup.

+
+ +
+ +
+

Optimized: total run time

+

+
+ +
+ +
+

Optimized: time by query

+

Best (fastest) of 3 iterations, in seconds. Hover a row for iterations and speedup.

+
+ +
+ +
+

Comparison: total run time

+

Sum of the 22 per-query best times, each the fastest of 3 iterations. Whiskers extend to the sum of the slowest iterations.

+
+ +
+ +
+

Comparison: speedup over Spark by query

+

+
+
+ + + +
+
+ Table view +
+
+
+ +
+

Run details and caveats

+
    +
    + + + + + + diff --git a/content/images/comet-nested-tpch/intro-total.svg b/content/images/comet-nested-tpch/intro-total.svg new file mode 100644 index 00000000..e6ce31ff --- /dev/null +++ b/content/images/comet-nested-tpch/intro-total.svg @@ -0,0 +1 @@ +Total execution time of the 22 nested TPC-H queries at SF1000Sum of per-query fastest times: Spark, Comet nested unoptimized and Comet nested optimized.04008001,2001,600Execution time (s)1,515.0 sSpark1,087.5 sComet nested unoptimized777.2 sComet nested optimized diff --git a/content/images/comet-nested-tpch/results-queries.svg b/content/images/comet-nested-tpch/results-queries.svg new file mode 100644 index 00000000..ea739c53 --- /dev/null +++ b/content/images/comet-nested-tpch/results-queries.svg @@ -0,0 +1 @@ +Execution time by query at SF1000Fastest of three iterations for each of the 22 nested TPC-H queries: Spark, Comet nested unoptimized and Comet nested optimized.SparkComet nested unoptimizedComet nested optimized0100200300400500Execution time (s)01020304050607080910111213141516171819202122 From f45ee2d04c6afd1a262c2476f54101e931f919f7 Mon Sep 17 00:00:00 2001 From: ovoievodin Date: Tue, 27 Oct 2026 09:12:12 -0700 Subject: [PATCH 2/5] fixes --- .../images/comet-nested-tpch/dashboard.html | 520 ------------------ 1 file changed, 520 deletions(-) delete mode 100644 content/images/comet-nested-tpch/dashboard.html diff --git a/content/images/comet-nested-tpch/dashboard.html b/content/images/comet-nested-tpch/dashboard.html deleted file mode 100644 index 97208ee2..00000000 --- a/content/images/comet-nested-tpch/dashboard.html +++ /dev/null @@ -1,520 +0,0 @@ - - - - - -Nested TPC-H SF1000: Comet vs Spark - - - -
    -
    -
    -

    Nested TPC-H at SF1000: Comet vs Spark

    -

    -
    - -
    - -
    - -
    - -
    - -
    -

    Unoptimized: total run time

    -

    -
    - -
    - -
    -

    Unoptimized: time by query

    -

    Best (fastest) of 3 iterations, in seconds. Hover a row for iterations and speedup.

    -
    - -
    - -
    -

    Optimized: total run time

    -

    -
    - -
    - -
    -

    Optimized: time by query

    -

    Best (fastest) of 3 iterations, in seconds. Hover a row for iterations and speedup.

    -
    - -
    - -
    -

    Comparison: total run time

    -

    Sum of the 22 per-query best times, each the fastest of 3 iterations. Whiskers extend to the sum of the slowest iterations.

    -
    - -
    - -
    -

    Comparison: speedup over Spark by query

    -

    -
    -
    - -
    - -
    -
    - Table view -
    -
    -
    - -
    -

    Run details and caveats

    -
      -
      -
      - - - - - From b76957056ad3af95f84091eca1e60af8cc21703f Mon Sep 17 00:00:00 2001 From: Oleks V Date: Wed, 28 Oct 2026 16:13:57 -0700 Subject: [PATCH 3/5] Update content/blog/2026-10-25-comet-nested-tpch.md Co-authored-by: Tim Saucer --- content/blog/2026-10-25-comet-nested-tpch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/blog/2026-10-25-comet-nested-tpch.md b/content/blog/2026-10-25-comet-nested-tpch.md index 1b784cfc..77c487c4 100644 --- a/content/blog/2026-10-25-comet-nested-tpch.md +++ b/content/blog/2026-10-25-comet-nested-tpch.md @@ -27,7 +27,7 @@ limitations under the License. [TOC] -Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark. Before the fixes this benchmark exposed, it took 1,088 s. +Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark. Date: Wed, 28 Oct 2026 16:18:45 -0700 Subject: [PATCH 4/5] fixes --- content/blog/2026-10-25-comet-nested-tpch.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/content/blog/2026-10-25-comet-nested-tpch.md b/content/blog/2026-10-25-comet-nested-tpch.md index 1b784cfc..f1bddbeb 100644 --- a/content/blog/2026-10-25-comet-nested-tpch.md +++ b/content/blog/2026-10-25-comet-nested-tpch.md @@ -27,7 +27,7 @@ limitations under the License. [TOC] -Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark. Before the fixes this benchmark exposed, it took 1,088 s. +Query engines get fast at what their benchmarks measure, and the most widely used analytical benchmarks contain no nested data. Apache DataFusion Comet now runs all 22 queries of a nested TPC-H at scale factor 1000 (1 TB) in 777 s, against 1,515 s for Spark. Total execution time at SF1000: Spark 1,515.0 s, Comet nested unoptimized 1,087.5 s, Comet nested optimized 777.2 s -Nested TPC-H is a TPC-H derived benchmark in the Comet repository ([#7044](https://github.com/apache/datafusion-comet/pull/7044)). It keeps the TPC-H data and questions and changes only the shape of the tables. The data and queries are engine neutral, so they can be used to compare Spark, Comet and other engines. +Nested TPC-H is a TPC-H derived benchmark in the Comet repository ([#7044](https://github.com/apache/datafusion-comet/pull/7044)). The TPC-H data and the questions the queries answer are unchanged. The tables are reshaped to store most columns in arrays of structs, and the queries are rewritten to unnest those arrays. The data and queries are engine neutral, so they can be used to compare Spark, Comet and other engines. We built it because our benchmarks and our users disagreed. Comet tracked its performance with TPC-H and TPC-DS, and the numbers were encouraging. Real user pipelines showed smaller gains, or none at all. What those pipelines had in common was nested data, which neither benchmark contains. @@ -56,7 +56,7 @@ Real data is often nested. Event logs, JSON, Avro and Protobuf records, and many Comet hit the shuffle case in [#5397](https://github.com/apache/datafusion-comet/issues/5397), where native shuffle writes of wide nested fields were two to three times slower than Spark's. The issue notes that it was "invisible for TPC-* benchmarks because of historical flat schema". -We needed a benchmark that fills the gap between academic benchmarking and real user workloads. +We needed a benchmark that filled the gap between academic benchmarking and real user workloads. ## How: an adapter that nests TPC-H @@ -103,6 +103,8 @@ spark-shell -i scripts/nest_tpch.scala \ The [README](https://github.com/apache/datafusion-comet/tree/main/benchmarks/nested) has the Iceberg variants and all settings. +The script and queries in the Comet repository also serve as an example. You can customize them to test any nested complexity, such as multi-element arrays, maps or deeper nesting. + ## Benchmark setup - **Data**: nested TPC-H, scale factor 1000 (1 TB). @@ -115,7 +117,7 @@ The [README](https://github.com/apache/datafusion-comet/tree/main/benchmarks/nes ## What nested benchmarks revealed -The nested benchmark exposed issues that were invisible on flat TPC-H. In the unoptimized run, Comet was slower than Spark on q20 (32.9 s against 31.5 s) and q21 (490.6 s against 373.5 s), and barely faster on q02 (34.0 s against 34.9 s). Tracing those queries surfaced a lot of technical debt in how nested data is read, shuffled and spilled, in Comet and in the iceberg-rust reader underneath it. +The nested schema is deliberately simple: single-element arrays of structs, with no maps for now. Even that was enough to expose issues that were invisible on flat TPC-H. In the unoptimized run, Comet was slower than Spark on q20 (32.9 s against 31.5 s) and q21 (490.6 s against 373.5 s), and barely faster on q02 (34.0 s against 34.9 s). Tracing those queries surfaced a lot of technical debt in how nested data is read, shuffled and spilled, in Comet and in the iceberg-rust reader underneath it. The issues fell into four groups: From 229f8e9f246dc1ff73306e6e0a76560adcb47960 Mon Sep 17 00:00:00 2001 From: ovoievodin Date: Wed, 28 Oct 2026 16:22:47 -0700 Subject: [PATCH 5/5] fixes --- content/blog/2026-10-25-comet-nested-tpch.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/content/blog/2026-10-25-comet-nested-tpch.md b/content/blog/2026-10-25-comet-nested-tpch.md index f1bddbeb..5b51a887 100644 --- a/content/blog/2026-10-25-comet-nested-tpch.md +++ b/content/blog/2026-10-25-comet-nested-tpch.md @@ -117,7 +117,7 @@ The script and queries in the Comet repository also serve as an example. You can ## What nested benchmarks revealed -The nested schema is deliberately simple: single-element arrays of structs, with no maps for now. Even that was enough to expose issues that were invisible on flat TPC-H. In the unoptimized run, Comet was slower than Spark on q20 (32.9 s against 31.5 s) and q21 (490.6 s against 373.5 s), and barely faster on q02 (34.0 s against 34.9 s). Tracing those queries surfaced a lot of technical debt in how nested data is read, shuffled and spilled, in Comet and in the iceberg-rust reader underneath it. +The nested schema is: single-element arrays of structs, with no maps for now. Even that was enough to expose issues that were invisible on flat TPC-H. In the unoptimized run, Comet was slower than Spark on q20 (32.9 s against 31.5 s) and q21 (490.6 s against 373.5 s), and barely faster on q02 (34.0 s against 34.9 s). Tracing those queries surfaced a lot of technical debt in how nested data is read, shuffled and spilled, in Comet and in the iceberg-rust reader underneath it. The issues fell into four groups: @@ -128,7 +128,7 @@ The issues fell into four groups: Spills and shuffle writes happen in flat workloads too. The nested schema changed the data volume or access pattern enough to expose them. -Most of these issues are fixed. The issues and pull requests, with their measurements, are listed in [Issues and pull requests](#issues-and-pull-requests) at the end of the post. +The issues and pull requests, with their measurements and details, are listed in [Issues and pull requests](#issues-and-pull-requests) at the end of the post. ## Results @@ -145,8 +145,6 @@ alt="Total execution time at SF1000: Spark 1,515.0 s, Comet nested unoptimized 1 ### Execution time by query -q21 is the longest query: 490.6 s on unoptimized Comet, 373.5 s on Spark and 256.6 s on optimized Comet. -