Repository navigation
perf: preserve dictionary encoding through regexp_replace - #26186
Merged
jayzhan211 merged 3 commits intoOct 25, 2026
Merged
Conversation
github-actions
Bot
added
sqllogictest
Rich-T-kid
force-pushed
the
rich-T-kid/regexp-replace-dict-encoding
branch
from
October 10, 2026 17:41
70cbf41 to
62e3ba4
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #26186 +/- ##
==========================================
+ Coverage 82.79% 82.89% +0.10%
==========================================
Files 1149 1154 +5
Lines 451645 458074 +6429
Branches 451645 458074 +6429
==========================================
+ Hits 373940 379731 +5791
- Misses 54929 55298 +369
- Partials 22776 23045 +269 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Keep Dictionary(_, Utf8/LargeUtf8/Utf8View) through regexp_replace instead of casting it to the inner string type. When the pattern, replacement, and flags args are all scalar the function runs on the dict values and re-attaches the keys; otherwise it falls back to a flat value-type output.
Rich-T-kid
force-pushed
the
rich-T-kid/regexp-replace-dict-encoding
branch
from
October 12, 2026 03:33
62e3ba4 to
7a2ee99
Compare
Contributor
Author
|
run benchmark tpch tpcds clickbench_partitioned wide_schema |
Contributor
Author
|
run benchmark clickbench_partitioned |
Rich-T-kid
marked this pull request as ready for review
October 12, 2026 03:52
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark wide_schema
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
Contributor
Author
forgot to flip the flag , re-running the last one |
Contributor
Author
|
run benchmark clickbench_partitioned |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "false"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
TODO fix the benchmark run. the baseline needs to be latest main with the flag enabled and the changed needs to be this branch with the flag enabled |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing rich-T-kid/regexp-replace-dict-encoding (7a2ee99) to 334f293 (merge-base) diff Run configurationrun benchmark wide_schema
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "32G"
baseline:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY: "true"CPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usagewide_schema — base (merge-base)
wide_schema — branch
File an issue against this benchmark runner |
Contributor
Author
|
actually no thats exactly what this does #26186 (comment), this is the exact query this PR targets https://github.com/apache/datafusion/blob/main/benchmarks/queries/clickbench/queries/q28.sql |
Contributor
Author
|
@kumarUjjawal would you mind taking a look at this as its related to #24111 this comment also provides some context #26186 (comment) |
Contributor
Thank you @Rich-T-kid for working on this. I'm currently on vacation. I will get to it when I return. |
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
jayzhan211
reviewed
Oct 24, 2026
Contributor
|
This PR appears to be waiting on the author to respond to reviewer feedback so marking as draft. Please mark ready to review when it is ready for another look |
alamb
marked this pull request as draft
October 24, 2026 10:04
Rich-T-kid
marked this pull request as ready for review
October 24, 2026 17:08
Contributor
|
Thanks @Rich-T-kid @kumarUjjawal |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
regexp_replacecurrently casts dictionary input to its underlying string type and materializes every row. When the pattern, replacement, and flags are scalar, it can apply the regex to the dictionary values once and reuse the existing keys.Filtered batches and shared Parquet dictionaries can contain many unreferenced values, so oversized dictionaries are compacted before applying the regex.
What changes are included in this PR?
regexp_replaceto the dictionary values and reuse the keys.What is the testing strategy for this PR?
The sqllogictest coverage checks dictionary inputs with
Utf8,LargeUtf8,Utf8View, andBinaryvalues, along with the non-literal fallback. The focusedregexp_replaceunit tests and sqllogictest pass.Are there any user-facing changes?
Yes.
regexp_replacenow preserves dictionary encoding when the other arguments are scalar. The returned values are unchanged.