Skip to content

Add Keyten native and Parquet benchmarks - #2353

Merged
alexey-milovidov merged 7 commits into
ClickHouse:mainfrom
singaraiona:add-keyten
Oct 3, 2026
Merged

alexey-milovidov merged 7 commits into
ClickHouse:mainfrom
singaraiona:add-keyten

Conversation

@singaraiona

@singaraiona singaraiona commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Add Keyten 0.1.53 through its Python dataframe API in two configurations: durable native storage (keyten) and direct scans of the single official Parquet file (keyten-parquet). Both use the shared benchmark driver and the same 43 query expressions.

The native load streams Parquet into the engine's native format without sorting or manual indexes. The loopback server opens its source on the first query after restart. The query client measures through receipt of the full JSON result, including planning, execution and serialization. Health checks do not read the dataset. The installer pins the published wheel.

Validation:

  • All 43 expressions checked against DuckDB on a 120,123-row synthetic dataset with the published wheel, including Unicode and temporal expressions.
  • Separate checks cover byte-length semantics, regex replacement when the pattern does not match, and delivery of more rows than a dataframe display would show.
  • All 43 full results from the official 99,997,497-row Parquet file checked against DuckDB using the source build at k10works/keyten@8a31be1869ef80dd31e6bb2c59da67fbabb2e94d, including aggregate values and valid LIMIT ties. This source build is distinct from the pinned published wheel.
  • Python compilation, shell syntax checks, and validate-results.py pass. The result validator reports only six existing Infobright warnings.

The full native load completed with 99,997,497 rows. Native validation passes 42 of 43 queries; Q24 fails because some empty-string payload values return as null. The mismatch reproduces without sorting and on a small Parquet-to-native fixture with both the source build and the pinned 0.1.53 wheel.

Draft blocked on resolving that native round-trip correctness failure, followed by the standard cold/hot run with the pinned release. No local diagnostic timings are submitted as official results.

@alexey-milovidov alexey-milovidov added the machine:all PR benchmark on every machine type label Oct 2, 2026
@alexey-milovidov alexey-milovidov self-assigned this Oct 2, 2026
@alexey-milovidov
alexey-milovidov deployed to benchmark-approval October 2, 2026 14:12 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

The run of keyten on c8g.4xlarge did not produce results.
The run of keyten on c8g.metal-48xl did not produce results.
The run of keyten-parquet on c8g.4xlarge did not produce results.
The run of keyten-parquet on c8g.metal-48xl did not produce results.

Logs:

@singaraiona singaraiona closed this Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Results for keyten-parquet are ready for: c6a.metal, c7a.metal-48xl.
This pull request is from a fork, so the automation cannot push to it; save this result as keyten-parquet/results/20261002/c7a.metal-48xl.json.
This pull request is from a fork, so the automation cannot push to it; save this result as keyten-parquet/results/20261002/c6a.metal.json.
Tick "Allow edits by maintainers" on the pull request to let the automation commit the results itself.

Logs:

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Results for keyten are ready for: c6a.4xlarge, c6a.metal, c7a.metal-48xl.
Results for keyten-parquet are ready for: c6a.2xlarge, c6a.4xlarge.
This pull request is from a fork, so the automation cannot push to it; save this result as keyten-parquet/results/20261002/c6a.2xlarge.json.
This pull request is from a fork, so the automation cannot push to it; save this result as keyten-parquet/results/20261002/c6a.4xlarge.json.
This pull request is from a fork, so the automation cannot push to it; save this result as keyten/results/20261002/c6a.metal.json.
This pull request is from a fork, so the automation cannot push to it; save this result as keyten/results/20261002/c7a.metal-48xl.json.
This pull request is from a fork, so the automation cannot push to it; save this result as keyten/results/20261002/c6a.4xlarge.json.
Tick "Allow edits by maintainers" on the pull request to let the automation commit the results itself.

Logs:

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Results for keyten are ready for: c6a.2xlarge.
The result files are committed as 0087498.
The run of keyten on t3a.small did not produce results.

Logs:

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Results for keyten-parquet are ready for: c6a.xlarge.
The result files are committed as 48fb6ec.

Logs:

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Results for keyten are ready for: c6a.xlarge.
The result files are committed as 6bc8a26.

Logs:

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Results for keyten-parquet are ready for: c6a.large.
The result files are committed as 94e870e.

Logs:

@alexey-milovidov

Copy link
Copy Markdown
Member

@singaraiona, this is interesting, and the benchmark was run successfully. Do you mind if we merge?

…e links

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@singaraiona

Copy link
Copy Markdown
Contributor Author

@singaraiona, this is interesting, and the benchmark was run successfully. Do you mind if we merge?

To be honest, I pushed it by accident - Keyten isn't quite in the shape I wanted it to be before making it public yet. So I'm actually a bit surprised everything passed :)

But sure, feel free to merge it! I'll send an update once Keyten is a bit more presentable.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Results for keyten are ready for: c6a.large.
The result files are committed as ec93b6e.
Removed manually added result files: keyten/results/20261002/c6a.4xlarge.json, keyten/results/20261002/c6a.metal.json, keyten/results/20261002/c7a.metal-48xl.json.
The run of keyten-parquet on t3a.small did not produce results.

Logs:

@alexey-milovidov
alexey-milovidov marked this pull request as ready for review October 3, 2026 17:13
@alexey-milovidov
alexey-milovidov merged commit d2e4fdb into ClickHouse:main Oct 3, 2026
1 of 2 checks passed

This branch is waiting to be deployed

1 waiting deployment
benchmark-approval — ec93b6e8 Waiting Oct 3, 2026 by robot-clickhouse via launch #633
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

machine:all PR benchmark on every machine type

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants