Skip to content

opt: rewrite date_trunc equality into a range so it can prune (#403 item 1) - #739

Merged
jdatcmd merged 3 commits into
mainfrom
opt/403-preimage-rewrite
Aug 25, 2026
Merged

opt: rewrite date_trunc equality into a range so it can prune (#403 item 1)#739
jdatcmd merged 3 commits into
mainfrom
opt/403-preimage-rewrite

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Implements item 1 of #403, the item that issue names as the strongest and
recommends breaking out. From the ClickHouse VLDB 2024 paper's monotonicity
traits: a predicate on a function of the sort key cannot use that key's zone
map, but the preimage of the function is a range the zone map can use.

Measured before building anything

500,000 rows clustered by time, 50 chunk groups:

predicate groups read removed rows
ts >= '2024-02-01' AND ts < '2024-02-02' 2 48 10000
date_trunc('day', ts) = '2024-02-01' — before 50 0 10000
date_trunc('day', ts) = '2024-02-01' — after 2 48 10000

Same answer, 25x fewer groups read, now matching the explicit range exactly.

The issue is old, so its premises were re-checked first: monoton|preimage
still finds only comments and no code, and the existing date_trunc handling in
columnar_vector.c is a group-count estimate for the grouped node —
unrelated machinery, though it showed the funcid-matching pattern.

Being straight about the generality

Dispatched on funcid in the scan-key builder beside the IN-list rewrite, so
adding a second function means adding its preimage rather than editing the
general comparison path — the objection #369 raised against a special case.

But this inverts exactly one function, and the step comes from a small table
of unit names, because a month is not a fixed number of seconds and has to be an
interval applied by calendar arithmetic. The generality is in where the rewrite
happens, not yet in how many functions it knows. Unlisted units decline.

timestamptz is deliberately absent: date_trunc(text, timestamptz) truncates
in the session TimeZone, so the interval depends on a GUC that can change
after the key is frozen at executor start. The suite asserts the timestamptz
answer under two different zones rather than asserting an absence.

Two of my own design claims were refuted by their removal proofs

Recorded because the comments now say what is true rather than what I assumed.

"A non-truncated constant would admit rows the original excludes." It would
not. The keys only prune, the executor re-applies the clause, and the derived
range is never narrower than the true matching set — the empty set is
contained in every range. Removing the check left the suite green. It is
defence in depth, kept because it is two lines and becomes load-bearing if these
keys are ever marked exact.

"The conservative marking is load-bearing against the batch fold." Not
observable today: pgcolumnar_batch_type_ok admits only int and float, so the
fold refuses a timestamp column on the type gate before exactness is
consulted. Marking the keys exact left the suite green too. The arm that names
the fold now records that it passes for a different reason than its title
suggests.

Removal proof

Reverting the rewrite returns the work arm to 0 removed while every answer arm
stays green — which is the only thing that can move. A predicate rewrite that
prunes correctly and one that does not prune at all return identical rows, so
the pruning measurement is the only check that can see this change at all.

Suites

preimage_rewrite 24/24, plus 15 green on PG17 covering every pruning path:
pushdown_report, native_skip, native_zonemap, native_zonemap_narrow,
zonemap_cost, native_saop_pushdown, native_param_pushdown,
native_exact_selection, native_bloom, ungrouped_vector_agg,
native_groupagg_batch, vector_agg_rescan_memory, native_decode_gating,
differential, docs_style.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN

jdatcmd and others added 3 commits August 25, 2026 06:16
…tem 1)

From the ClickHouse VLDB 2024 paper's monotonicity traits: a predicate on a
function of the sort key cannot use the key's zone map, but the preimage of that
function is a range the zone map CAN use.

Measured before building anything, 500,000 rows clustered by time, 50 chunk
groups: `ts >= '2024-02-01' AND ts < '2024-02-02'` read 2 groups and removed 48;
`date_trunc('day', ts) = '2024-02-01'` read all 50 and removed 0, for the same
10,000 rows. It now removes 48, matching the explicit range exactly.

Dispatched on funcid in the scan-key builder beside the IN-list rewrite, so
adding a second function means adding its preimage rather than editing the
general comparison path -- the objection #369 raised against a special case.
Being straight about the shape: this inverts exactly one function, and the step
comes from a small table of unit names, because a month is not a fixed number of
seconds and has to be an interval applied by calendar arithmetic. The generality
is in where the rewrite happens, not yet in how many functions it knows.

timestamptz is deliberately absent: date_trunc(text, timestamptz) truncates in
the session TimeZone, so the interval depends on a GUC that can change after the
key is frozen. The suite asserts the timestamptz answer under two different
zones rather than asserting an absence.

TWO OF MY OWN DESIGN CLAIMS WERE REFUTED BY THEIR REMOVAL PROOFS, and the
comments now say what is true rather than what I assumed.

I wrote that accepting a non-truncated constant "would admit rows the original
excludes". It would not: the keys only prune, the executor re-applies the
clause, and the derived range is never NARROWER than the true matching set --
the empty set is contained in every range. Removing the check left the suite
green. It is defence in depth, kept because it is two lines and becomes
load-bearing if the keys are ever marked exact.

I wrote that the conservative marking was load-bearing against the batch fold.
It is not observable today: pgcolumnar_batch_type_ok admits only int and float,
so the fold refuses a timestamp column on the TYPE gate before exactness is
consulted. Marking the keys exact left the suite green. The arm that names the
fold now records that it passes for a different reason than its title suggests.

Removal proof: reverting the rewrite returns the work arm to 0 removed while
every answer arm stays green, which is the only thing that can move -- a
predicate rewrite that prunes correctly and one that does not prune at all
return identical rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN
…#403)

The claim-1 correction went into the C comment and the design doc but not the
test, and I said "the comment now says so" when it said so in two of three
places. Caught in review.

The suite's header bullet and its decline-section comment both still taught the
belief my own removal proof had refuted: that emitting a range for a
non-truncated constant "would admit rows the original excludes", and that the
danger is "a rewrite that turns it into a range and returns rows". Neither is
true while the keys are conservative and the executor re-applies the clause. A
reader of the test would have come away believing the guard is load-bearing for
correctness.

Both now say what the C comment and the design doc say: the decline is defence
in depth, established by removing it and watching the suite stay green, and it
becomes load-bearing only if these keys are marked exact, since the batch fold
would then use them as its only row filter with no recheck behind it. The
decline arms now also state what they can and cannot show -- they assert the
answer, and they are the arms that would catch it if exactness were claimed.

Test comments only; no code change, and the suite is unchanged at 24 of 24.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN
The SUITES list is required to be sorted in C order and harness_selftest gates
it. I inserted preimage_rewrite after pg19_vacuum_options, where it reads
alphabetically but is not: in C order '1' (0x31) and '_' (0x5f) put pg19 before
pg_dump, and "pr" sorts after "pg", "ph" and "pl", so it belongs between
planner_choice_quality and projection_privilege, about fifteen entries later.

Moving the one line; the set of suites is unchanged and the list is 215 entries
before and after.

This was in the ORIGINAL commit, so this PR has never been green on the full
matrix. Caught by CI on the PG18 leg during review, not by me.

The miss is the same one I made on #725 and wrote a rule about afterwards: I ran
the suites the CODE touched and not the suite that owns the FILE I touched. A
change that edits run_all_versions.sh is precisely the change harness_selftest
exists for, and I have registered four suites this session -- running it after
the first three and not the fourth is the shape of the mistake, not an oversight
about which suite is relevant.

Verified through the runner's own --list-suites rather than by re-parsing the
array, which is what that flag exists for and what the file's own comment says
to do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G8jGnYAbTgiAcoUqn7E9EN

@OffgridwithJD OffgridwithJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — reviewed as ChronicallyJD. Verified on the merged tree (current main 43c7319 + this PR through 1cbfe05), pg17a assert + pg18_san.

Preimage-rewriting date_trunc(unit, timestamp) = const into a range on the column is a clean, well-scoped first cut: dispatched on funcid so the mechanism is a function property plus a rewrite, not a special case (the #369 objection), and it declines anything it cannot inarguably invert — declining costs a scan, never an answer.

The win, reproduced

On the time-clustered fixture the rewrite prunes 48 of 50 chunk groups — identical to the equivalent explicit range (reads 2, the full 25×), for the same rows. All oracle arms, both timestamptz zones, and every unit green; the general pushdown path is unaffected; ASAN+UBSAN clean over the pruning path.

The two self-corrected claims — reproduced independently

You flagged two design claims your own removal proofs had refuted, and asked me to check the corrections rather than the code. Both hold:

  • The non-truncated-constant guard is defence in depth, not correctness. I removed it, rebuilt, and the suite stays green — "a non-truncated constant returns no rows" included — because the keys only prune and the executor re-applies the clause, so an empty true set is contained in any derived range.
  • The conservative marking is not load-bearing against the fold today. pgcolumnar_batch_type_ok admits only int2/4/8 and float4/8, so a timestamp column is refused on the type gate before exactness is consulted.

Keep both — because the cliff is named

On your question of whether to remove them: keep both. Check 1 guards a specific future — the moment these keys are marked exact, a non-truncated constant makes the fold over-count with no recheck behind it, and #715 is exactly the incident where that path was reached from the other side. Defence in depth against a future you can name is worth two lines; defence in depth in general is not. That distinction is the whole of it.

Two things this review turned up, both fixed

  • The claim-1 correction had landed in the C comment and the design doc but not the test, which still taught the refuted framing in two places. Fixed in 2d25d96; all three artifacts now say the same thing, and the decline arms state what they can and cannot show.
  • run_all_versions.sh registered preimage_rewrite in the wrong C-order slot (after pg19_vacuum_options; it belongs after planner_choice_quality), which failed harness_selftest on the full matrix — present since the original commit, so the PR had never been green. Fixed in 1cbfe05, verified through the runner's own --list-suites. I own half of this one: I ran the feature's suites and not the suite that gates the file the PR edits.

Non-test code is byte-identical across both follow-up commits, so the feature and sanitizer verification stand. Feature correct, corrections honest, registration fixed, full matrix green.

@jdatcmd
jdatcmd merged commit 43ee2b7 into main Aug 25, 2026
12 checks passed
@jdatcmd
jdatcmd deleted the opt/403-preimage-rewrite branch August 25, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants