opt: rewrite date_trunc equality into a range so it can prune (#403 item 1) - #739
Conversation
…tem 1)
From the ClickHouse VLDB 2024 paper's monotonicity traits: a predicate on a
function of the sort key cannot use the key's zone map, but the preimage of that
function is a range the zone map CAN use.
Measured before building anything, 500,000 rows clustered by time, 50 chunk
groups: `ts >= '2024-02-01' AND ts < '2024-02-02'` read 2 groups and removed 48;
`date_trunc('day', ts) = '2024-02-01'` read all 50 and removed 0, for the same
10,000 rows. It now removes 48, matching the explicit range exactly.
Dispatched on funcid in the scan-key builder beside the IN-list rewrite, so
adding a second function means adding its preimage rather than editing the
general comparison path -- the objection #369 raised against a special case.
Being straight about the shape: this inverts exactly one function, and the step
comes from a small table of unit names, because a month is not a fixed number of
seconds and has to be an interval applied by calendar arithmetic. The generality
is in where the rewrite happens, not yet in how many functions it knows.
timestamptz is deliberately absent: date_trunc(text, timestamptz) truncates in
the session TimeZone, so the interval depends on a GUC that can change after the
key is frozen. The suite asserts the timestamptz answer under two different
zones rather than asserting an absence.
TWO OF MY OWN DESIGN CLAIMS WERE REFUTED BY THEIR REMOVAL PROOFS, and the
comments now say what is true rather than what I assumed.
I wrote that accepting a non-truncated constant "would admit rows the original
excludes". It would not: the keys only prune, the executor re-applies the
clause, and the derived range is never NARROWER than the true matching set --
the empty set is contained in every range. Removing the check left the suite
green. It is defence in depth, kept because it is two lines and becomes
load-bearing if the keys are ever marked exact.
I wrote that the conservative marking was load-bearing against the batch fold.
It is not observable today: pgcolumnar_batch_type_ok admits only int and float,
so the fold refuses a timestamp column on the TYPE gate before exactness is
consulted. Marking the keys exact left the suite green. The arm that names the
fold now records that it passes for a different reason than its title suggests.
Removal proof: reverting the rewrite returns the work arm to 0 removed while
every answer arm stays green, which is the only thing that can move -- a
predicate rewrite that prunes correctly and one that does not prune at all
return identical rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN
…#403) The claim-1 correction went into the C comment and the design doc but not the test, and I said "the comment now says so" when it said so in two of three places. Caught in review. The suite's header bullet and its decline-section comment both still taught the belief my own removal proof had refuted: that emitting a range for a non-truncated constant "would admit rows the original excludes", and that the danger is "a rewrite that turns it into a range and returns rows". Neither is true while the keys are conservative and the executor re-applies the clause. A reader of the test would have come away believing the guard is load-bearing for correctness. Both now say what the C comment and the design doc say: the decline is defence in depth, established by removing it and watching the suite stay green, and it becomes load-bearing only if these keys are marked exact, since the batch fold would then use them as its only row filter with no recheck behind it. The decline arms now also state what they can and cannot show -- they assert the answer, and they are the arms that would catch it if exactness were claimed. Test comments only; no code change, and the suite is unchanged at 24 of 24. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN
The SUITES list is required to be sorted in C order and harness_selftest gates it. I inserted preimage_rewrite after pg19_vacuum_options, where it reads alphabetically but is not: in C order '1' (0x31) and '_' (0x5f) put pg19 before pg_dump, and "pr" sorts after "pg", "ph" and "pl", so it belongs between planner_choice_quality and projection_privilege, about fifteen entries later. Moving the one line; the set of suites is unchanged and the list is 215 entries before and after. This was in the ORIGINAL commit, so this PR has never been green on the full matrix. Caught by CI on the PG18 leg during review, not by me. The miss is the same one I made on #725 and wrote a rule about afterwards: I ran the suites the CODE touched and not the suite that owns the FILE I touched. A change that edits run_all_versions.sh is precisely the change harness_selftest exists for, and I have registered four suites this session -- running it after the first three and not the fourth is the shape of the mistake, not an oversight about which suite is relevant. Verified through the runner's own --list-suites rather than by re-parsing the array, which is what that flag exists for and what the file's own comment says to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN
OffgridwithJD
left a comment
There was a problem hiding this comment.
APPROVE — reviewed as ChronicallyJD. Verified on the merged tree (current main 43c7319 + this PR through 1cbfe05), pg17a assert + pg18_san.
Preimage-rewriting date_trunc(unit, timestamp) = const into a range on the column is a clean, well-scoped first cut: dispatched on funcid so the mechanism is a function property plus a rewrite, not a special case (the #369 objection), and it declines anything it cannot inarguably invert — declining costs a scan, never an answer.
The win, reproduced
On the time-clustered fixture the rewrite prunes 48 of 50 chunk groups — identical to the equivalent explicit range (reads 2, the full 25×), for the same rows. All oracle arms, both timestamptz zones, and every unit green; the general pushdown path is unaffected; ASAN+UBSAN clean over the pruning path.
The two self-corrected claims — reproduced independently
You flagged two design claims your own removal proofs had refuted, and asked me to check the corrections rather than the code. Both hold:
- The non-truncated-constant guard is defence in depth, not correctness. I removed it, rebuilt, and the suite stays green — "a non-truncated constant returns no rows" included — because the keys only prune and the executor re-applies the clause, so an empty true set is contained in any derived range.
- The conservative marking is not load-bearing against the fold today.
pgcolumnar_batch_type_okadmits only int2/4/8 and float4/8, so a timestamp column is refused on the type gate before exactness is consulted.
Keep both — because the cliff is named
On your question of whether to remove them: keep both. Check 1 guards a specific future — the moment these keys are marked exact, a non-truncated constant makes the fold over-count with no recheck behind it, and #715 is exactly the incident where that path was reached from the other side. Defence in depth against a future you can name is worth two lines; defence in depth in general is not. That distinction is the whole of it.
Two things this review turned up, both fixed
- The claim-1 correction had landed in the C comment and the design doc but not the test, which still taught the refuted framing in two places. Fixed in 2d25d96; all three artifacts now say the same thing, and the decline arms state what they can and cannot show.
run_all_versions.shregisteredpreimage_rewritein the wrong C-order slot (afterpg19_vacuum_options; it belongs afterplanner_choice_quality), which failedharness_selfteston the full matrix — present since the original commit, so the PR had never been green. Fixed in 1cbfe05, verified through the runner's own--list-suites. I own half of this one: I ran the feature's suites and not the suite that gates the file the PR edits.
Non-test code is byte-identical across both follow-up commits, so the feature and sanitizer verification stand. Feature correct, corrections honest, registration fixed, full matrix green.
Implements item 1 of #403, the item that issue names as the strongest and
recommends breaking out. From the ClickHouse VLDB 2024 paper's monotonicity
traits: a predicate on a function of the sort key cannot use that key's zone
map, but the preimage of the function is a range the zone map can use.
Measured before building anything
500,000 rows clustered by time, 50 chunk groups:
ts >= '2024-02-01' AND ts < '2024-02-02'date_trunc('day', ts) = '2024-02-01'— beforedate_trunc('day', ts) = '2024-02-01'— afterSame answer, 25x fewer groups read, now matching the explicit range exactly.
The issue is old, so its premises were re-checked first:
monoton|preimagestill finds only comments and no code, and the existing
date_trunchandling incolumnar_vector.cis a group-count estimate for the grouped node —unrelated machinery, though it showed the funcid-matching pattern.
Being straight about the generality
Dispatched on funcid in the scan-key builder beside the IN-list rewrite, so
adding a second function means adding its preimage rather than editing the
general comparison path — the objection #369 raised against a special case.
But this inverts exactly one function, and the step comes from a small table
of unit names, because a month is not a fixed number of seconds and has to be an
interval applied by calendar arithmetic. The generality is in where the rewrite
happens, not yet in how many functions it knows. Unlisted units decline.
timestamptzis deliberately absent:date_trunc(text, timestamptz)truncatesin the session
TimeZone, so the interval depends on a GUC that can changeafter the key is frozen at executor start. The suite asserts the timestamptz
answer under two different zones rather than asserting an absence.
Two of my own design claims were refuted by their removal proofs
Recorded because the comments now say what is true rather than what I assumed.
"A non-truncated constant would admit rows the original excludes." It would
not. The keys only prune, the executor re-applies the clause, and the derived
range is never narrower than the true matching set — the empty set is
contained in every range. Removing the check left the suite green. It is
defence in depth, kept because it is two lines and becomes load-bearing if these
keys are ever marked exact.
"The conservative marking is load-bearing against the batch fold." Not
observable today:
pgcolumnar_batch_type_okadmits only int and float, so thefold refuses a
timestampcolumn on the type gate before exactness isconsulted. Marking the keys exact left the suite green too. The arm that names
the fold now records that it passes for a different reason than its title
suggests.
Removal proof
Reverting the rewrite returns the work arm to 0 removed while every answer arm
stays green — which is the only thing that can move. A predicate rewrite that
prunes correctly and one that does not prune at all return identical rows, so
the pruning measurement is the only check that can see this change at all.
Suites
preimage_rewrite24/24, plus 15 green on PG17 covering every pruning path:pushdown_report,native_skip,native_zonemap,native_zonemap_narrow,zonemap_cost,native_saop_pushdown,native_param_pushdown,native_exact_selection,native_bloom,ungrouped_vector_agg,native_groupagg_batch,vector_agg_rescan_memory,native_decode_gating,differential,docs_style.🤖 Generated with Claude Code
https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN