Remove pgrust: the entry is cheating (per-query kernels and result caching)

[alexey-milovidov]

This removes the pgrust entry (added in #1163, updated to v0.4-preview in #2106) from the benchmark. The results it reports are produced by machinery written specifically for the 43 ClickBench queries, and its hot runs replay cached answers of the previous run. Both violate the rules: query result caches and caches near the end of the pipeline must be disabled, and the entry is tagged tuned: no while it is tuned for these exact queries.

Everything below was verified against the published v0.4-preview release binary (pgrust.com/downloads/v0.4-preview/, the one install downloads) and the public v0.3 source (github.com/malisper/pgrust). The v0.4-preview source is not published; the binary identifies itself as pgrust 0.3-beta.

One hand-written kernel per benchmark query

The aarch64 release binary embeds these source paths:

crates/gravity/backend/irprobe/src/q00.rs  q01.rs  q02.rs  q03.rs  q04.rs  q06.rs  q07.rs
q12.rs  q15.rs  q16.rs  q17.rs  q18.rs  q19.rs  q20.rs  q21.rs  q22.rs  q23.rs  q24.rs
q25.rs  q27.rs  q30.rs  q31.rs  q36.rs  tpch_q01.rs

The numbers are ClickBench query numbers. The plan matchers that dispatch to these kernels emit refusal messages named after the benchmark, for example:

clickbench group count: filter is not `k <> 0`
clickbench phrase-min: the third column is not count(*)
clickbench t3 distinct: min(text) / LIKE need one text key, count(*) and the count order
clickbench frame: date_trunc unit is not minute
tpch q1: column 6 is not sum(price * (1 - discount) * (1 + tax))

There is also an environment knob literally named PGRUST_GRAVITY_Q20_VARIANT.

The public v0.3 source confirms the design: crates/backend/executor/sqe/rig/plans/clickbench.ron is described in its header as "hand-lowered physical plans for all 43 ClickBench queries", and queries-clickbench.sql (the verbatim benchmark SQL) is compiled into the executor crate with include_str!. That file classifies Q0, Q1, Q2, Q3, Q6, Q19 and Q29 as MetadataAnswer (answered from per-part statistics, no scan), notes that Q19's "answer lines are reconstructed from the predicate constant", that Q29 is an algebraic rewrite SUM(x+k) = SUM(x) + k*COUNT followed by a metadata answer, and that Q18 depends on a persisted derived column "with baked bit widths 26/6/20". Q28's regular expression is a one-entry hardcoded table (HOST_EXTRACT_PATTERN = ^https?://(?:www\.)?([^/]+)/.*$), and a planner comment reads "The 43's only HAVING shape: COUNT(*) > k".

The hot runs are a result cache

For Q20, Q21 and Q23 the plan notes say: "hot rep = pure fold over cached verdicts — ZERO columns touched; the selection is the answer". EXPLAIN on the release binary reports SQE: engine (family: windowreplay) for Q20 and family: metaanswer for Q1 with the scan marked never executed.

Measured locally on a 96-core Graviton (Neoverse-V2) machine, using the entry's own scripts and configuration, with a true cold cycle (stop, drop caches, start) like the driver does:

query	cold	2nd run	3rd run	same shape, new constant, 1st run
Q20 URL LIKE '%google%'	10.8 s	11 ms	5 ms	'%rambler%': 283 ms, then 12 ms, 4 ms
Q17 GROUP BY UserID, SearchPhrase LIMIT 10	1.65 s	4 ms	15 ms
Q33 GROUP BY URL ORDER BY c DESC LIMIT 10	2.38 s	213 ms	108 ms

A second restart brought Q20 back to 11.2 s, then 3 ms. The engine's real warm speed for a substring scan is a few hundred milliseconds, which is about what ClickHouse does on the same hardware. The published hot numbers (7 to 23 ms for Q20) are the replay of the previous run's verdicts.

The v0.2 submission (#1163) disclosed a similar cache (pgrust.condition_cache = on) and offered to switch it off. The v0.4 update removed that README and the setting. No configuration disables the cache now: with pgrust.gravity_entry = off and pgrust.condition_cache = off, a repeated LIKE still returns in 3 to 4 ms.

Queries outside the 43 shapes are refused

The engine has no general execution path for the columnar table. Trivially perturbed benchmark queries fail with ERROR: sqe: statement over columnar relation "hits" is not served, and the error detail says "The sqe engine serves columnar shapes with no fallback; an unserved shape fails loudly instead of running slowly". 20 of 67 near-benchmark probes and 3 of 49 perturbed variants were refused, including:

SELECT MIN(URL) FROM hits;
SELECT COUNT(*) FROM hits WHERE URL = 'http://holodilnik.ru/';
SELECT AVG(UserID + 1) FROM hits;
SELECT SUM(ResolutionWidth * 2) FROM hits;
SELECT MAX(length(URL)) FROM hits;
SELECT COUNT(*) FROM hits WHERE strpos(URL, 'google') > 0;
SELECT COUNT(*) FROM (SELECT UserID FROM hits GROUP BY UserID) t;
SELECT SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase HAVING SUM(IsRefresh) > 10 ORDER BY c DESC LIMIT 10;
SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1', 'g') AS k, ... -- the Q28 regexp with a flag

INSERT, UPDATE and DELETE are not supported on the table either (pgrcolumnar2 does not support INSERT).

For the record

The answers to the 43 queries themselves are correct: compared against clickhouse local over the same hits.parquet, 30 match exactly and the remaining differences are valid tie-breaks on LIMIT queries (verified individually). The problem is not wrong results; it is that the results measure a lookup table of the benchmark, not a database.

Related: #1163
Related: #2106