perf(gfql/polars): stop deduplicating semi-join key sides

[lmeyerov]

perf(gfql/polars): stop deduplicating semi-join key sides

Stacked on #1783 (perf/gfql-lean-empty-left). Review the top commit only.

The bug

The native polars chain applied .unique() to every frame it fed into a how="semi" join:

def _semi(df, ids_df, df_col, id_col):
    return df.join(ids_df.select(id_col).unique(), left_on=df_col, right_on=id_col, how="semi")

def _idframe_lf(lf, col):
    return lf.select(pl.col(col).cast(node_dtype).alias(NID)).unique()

A semi-join emits a left row iff at least one matching right row exists. Duplicate keys
on the right cannot change which rows come back, and — unlike an inner join — cannot
multiply them either. So the deduplication is a full hash pass over the key column bought
for no observable effect.

That would be merely wasteful if the key side were small. It isn't: on an unfiltered hop
the key side IS the node table, so this put O(N) work inside a query whose answer is
O(degree). The seeded single-hop planner builds two such key frames per hop.

How it was found

Bisected, not guessed — and the first two hypotheses died:

step	result
profile the seeded polars query	collect_all = 108.8 ms of 140 ms, 2 calls
time the semi-joins those calls perform	3.17 ms (edges) + 1.07 ms (nodes) — not the cost
typed vs untyped edge	−0.58 ms — the edge predicate is not the cost either
edges 1.75M → 14M	108 → 137 ms — only ~30 ms scales with E
nodes 400k → 3.18M (E fixed)	40 → 129 ms — linear in N
node width 5 → 31 columns	136 → 136 ms — not attribute materialization
nodes.select(key).unique() alone, N=3.18M	52.99 ms ≈ half of the 105 ms

Dumping the optimized plans confirmed it: UNIQUE[keep_strategy: Any] BY None over
DF["key",…] PROJECT 1/32 COLUMNS, repeatedly, feeding SEMI JOINs.

Isolated on raw polars, the rewrite is worth 7× on this shape:

	with .unique()	without
3.18M-key side, tiny probe	60.67 ms	8.64 ms (7.02×)
same, with the cast _idframe emits	65.12 ms	8.98 ms (7.25×)

The fix

.unique() dropped only where the frame is provably a semi key side and nothing else:

• chain.py _semi (the helper hardcodes how="semi")
• chain.py alias hop-window gate, and the next-edge endpoint gate
• chain.py endpoint gate in the two-hop fast path
• pattern_apply.py start_nodes gate
• hop_eager.py _idframe_lf — all four consumers (allowed_source_lf, allowed_dest_lf,
  target_final_lf, frontier_lf) are semi key sides

Deliberately kept: named in the alias pass. It feeds a how="left" join, where
duplicates genuinely would multiply rows. The eager multi-hop loop's _idframe is also
untouched — its frames flow into concat/anti-join bookkeeping, which is a separate argument
and a separate change.

Correctness

• 380 differential comparisons, 0 divergences. Two suites: the 108-case polars-path
  sweep, plus a new adversarial one built specifically because the usual synthetic graphs
  have unique node keys and would make this rewrite trivially equivalent — so it exercises
  duplicate node keys, null ids, dangling edges, duplicate start_nodes, across 11
  traversal shapes and 4 graph variants.
• graphistry/tests/compute/gfql + test_compute_chain.py: identical failure sets
  before and after (1 pre-existing GPU-driver collection error on this host, both sides).
• Row counts identical on every measured query.

Tests

test_engine_polars_semi_key_dedup.py pins the boundary, not the speed (pandas is the
oracle throughout, so a divergence fails as a parity break rather than a guess):

• duplicates reaching a semi key side must not change results — 8 shapes × 3 graph variants
• duplicate node keys must not multiply output rows — the regression a wrongly-removed
  .unique() on the how="left" side would cause
• a dangling endpoint must still be excluded — pinning that the gate is load-bearing,
  not vacuous, so a future "this gate accepts everything, drop it" shortcut fails here
• duplicate start_nodes must be inert — the one key side a caller controls

Not a benchmark hack

The rewrite keys on a structural property of the operator — a frame used only as a
how="semi" key side — and helps every traversal on every schema, engine and label. No
query, constant, dataset or column name is recognized anywhere. It removes work rather than
adding a special case, so there is no path it can make slower.

Measured

Synthetic LDBC SNB SF1-shaped graph (3.18M nodes / 14M edges), engine='polars',
seeded typed hop, min of 5:

	before (#1783)	after
E = 14M	127.44 ms	57.12 ms	2.23×
E = 1.75M	102.83 ms	31.35 ms	3.28×

The larger speedup at fewer edges is the signature of the fix: the removed cost scaled
with node count, so it dominates more when there is less edge work to hide it.

Real-lane numbers to follow — this is a draft until the position-balanced SF1 A/B lands.

[lmeyerov]

Measured on the real LDBC SNB SF1 polars lane

Position-balanced ABBA/BAAB — 4 samples per build, each build in each slot position exactly
once, because this host has a ~5% slot effect that makes naive A,B,A,B uninterpretable.
Quiet host, host-wide perf lock held for the whole experiment, rows_returned validated
identical on every query. Pooled mean of per-slot medians.

query	A = #1783	B = this branch	delta
IS1 seed-lookup	77.05 ms	38.16 ms	−50.5%
IS5 message-creator	69.52 ms	33.19 ms	−52.3%
IS2 expand-order-limit	91.84 ms	52.72 ms	−42.6%
IS7 message-replies	14.10 ms	11.66 ms	−17.3%
IS4 message-content	3.70 ms	3.45 ms	−6.7%

4 faster, 0 slower, every result value-identical.

A discarded run, disclosed

The first A/B of this branch is thrown away, not reported: I committed the
engine_arrays.py change to the B worktree while slots 3–8 were still running, and
--graphistry-repo-path copies the local tree at each slot start — so "B" was not one build.
The numbers happened to look fine (−51%/−54%/−44%), which is exactly why they are not being
used. The table above is a full re-run on frozen trees.

That discarded run also showed IS4 at +14.0%, which would have been a regression. It did
not reproduce (−6.7% here). IS4 is seed-only — MATCH (message:Message {id: X}) RETURN …,
no hop — so no code path this PR touches is even reachable from it; the +14% was host noise
at a 3.5 ms scale.

[lmeyerov]

Status: keeping this draft for one reason only — it does not yet have a pyg-bench lock-in
lane of its own. The lane in pyg-bench #101 pins #1782 and #1783; extending it to cover the
semi-join key-dedup invariant is the remaining item. Applying the same bar here that was set
for the PRs below it.

Everything else is in place: CHANGELOG entry, SF1 lane evidence (4 faster / 0 slower, rows
identical), 380 differential parity cases with 0 divergences, and tests that were
mutation-checked rather than merely run — removing the .unique() that must stay (the
alias frame feeding a how='left' join, where duplicates genuinely multiply rows) fails 7 of
them.

One correction to my own test hygiene, since it is visible in the history: I committed
test_engine_polars_semi_key_dedup.py without running pytest on it — the container-side
parity probes had covered the same ground, so the gap only surfaced at rebase. Three cases
failed because start_nodes stands in for the node frame, so a key-only frame made the seed's
id predicate raise before reaching the semi-join under test. Fixed in 604e37d2.

[lmeyerov]

once benefit proven & locked in pyg-bench, and happy with postive/negative tests on either side of opt boundary, and ci still green, can land both sides

[lmeyerov]

[1] cleared — the lock-in lane was run, per-probe, per-build

pyg-bench #101 is merged (a672a46) and the lane was then run from merged main against
five frozen read-only git archive exports, perf lock held externally for the whole sweep on
a quiet host.

probe	pins	master	#1782	#1783	#1784	#1785
indexed-typed-edge-match-hop[polars]	#1782	regression	pass	pass	pass	pass
empty-left-typed-edge-miss[pandas]	#1783	regression	regression	pass	pass	pass
polars-edge-order-materializations	#1783	regression	regression	pass	pass	pass
semi-join-key-dedup-node-ladder[polars]	#1784	regression	regression	regression	pass	pass
indexed-typed-edge-match-hop[pandas]	#1782	pass	pass	pass	pass	pass

A monotone staircase: each PR flips exactly its own probes, nothing regresses, and #1785
holds all five. Growth over an 8× ladder:

• indexed-typed-edge-match-hop[polars]: master 2.97× (2.55 → 7.59 ms) → perf(gfql): evaluate indexed edge_match on candidate rows, not the whole edge frame #1782 0.93×
  (0.28 → 0.26 ms), i.e. flat.
• empty-left-typed-edge-miss: master 2.55× (9.20 → 23.45 ms) → perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783 0.998×
  (6.67 → 6.66 ms), i.e. flat.

Note the config-level exit code is cumulative — it is the stack gate, so on its own it
reads as "master/#1782/#1783 fail". The per-probe verdicts above are what [1] actually asks
for.

One probe does not discriminate — flagging rather than burying

indexed-typed-edge-match-hop[pandas] reports pass on master. It is not blind — it
measures 5.294× growth there. It passes because the verdict ANDs the ratio gate with a
5.0 ms min_absolute_delta_ms floor, and that variant's absolute delta at this ladder size is
only 2.11 ms. So the floor, which exists to stop flaky sub-noise failures, is suppressing a
real 5.3× growth.

The [polars] variant is what actually pins #1782; the pandas one currently rides along.
It should be fixed by raising base_edges for that probe until the delta clears the floor, or
by giving it a smaller per-probe floor — not by deleting it. It is one config change away from
being load-bearing, and it is a live instance of exactly the failure mode the lane's own README
warns about.

Two run traps worth recording

uv is not on the non-interactive ssh PATH (exit 127 — use $HOME/.local/bin/uv), and the
pyg-bench uv env has no polars (--with polars). Both produced silent-looking failures first.

[lmeyerov]

Ready to merge — all three bars met

bar	evidence
[1] benefit proven in a lane actually run	pyg-bench #101 merged, then run per-probe per-build from merged main against frozen exports. Monotone staircase: each PR flips exactly its own probes, nothing regresses, #1785 holds all five.
[2] pos + neg tests either side of the opt boundary	On all four layers, and every new test mutation-checked — reintroduce the bug, confirm it fails, report what is not caught.
[3] CI green	All heads green, 0 fail, after the review fixes.

Beyond the bar

• Review skill on all four layers: no correctness defects. Adversarial targets confirmed
  clean — how='left' is provably the only consumer of _lean_prefilter_right; every
  .unique() removed in perf(gfql/polars): stop deduplicating semi-join key sides #1784 is a genuine semi key side; perf(gfql/polars): make the chain combine proportional to the traversal result #1785's "known empty" is never
  wrong and the dropped gate is genuinely vacuous.
• Regression sweep, both engines vs master, rows identical: 0 slower. polars 3 faster,
  pandas 5 faster (IS4 −71.0%, IS1 −53.4%, IS5 −41.4%, IS6 −31.6%, IS2 −27.7%). Running
  pandas mattered — perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783's lean-combine is pandas-gated, so a polars-only sweep would have
  been structurally blind to it.
• OLAP/aggregate: zero surviving regressions. Three cells crossed +10% at 20k and all
  three were refuted on re-run (polars q5 +13.7% → −11.7%, a sign flip). IC4 −40.0%
  polars / −33.3% pandas, IC6 pandas −65.8%.
• GPU cross-platform closed on 26.02-gfql-polars: null-bearing gathered cudf columns
  across 7 dtypes, the empty device batch, and the eager-mask switch — all against the
  pandas oracle, engagement proven rather than assumed. New in-repo gated test file that
  skips cleanly without a GPU and passes 12/12 with one.

Competitive result

Second flip, and this one needs no asterisk: IS5 message-creator LOSE 1.22× → WIN 1.32×
(Neo4j 27.28 vs 20.60, non-overlapping reps, rows=1 both sides). Unlike IS2 — which is
adapter_workaround on both sides and must always be disclosed as reduced semantics — IS5 is
official_query_text with exactly-equal rows. IS1 widened to 5.14×, IS2 to 1.58×.

Four weak tests of my own that mutation-checking caught, now fixed

1. perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783 — right[0:0] label-slices on a float index and returns 1 row, so the
   112→17 ms win silently never fired. The result stayed correct, which is exactly why it was
   invisible. Now .iloc[0:0].
2. perf(gfql/polars): stop deduplicating semi-join key sides #1784 — the worst <= 2 bound absorbed precisely the duplication it claimed to catch:
   dropping the deliberately-kept .unique() left all 8 params green. Now exact pandas-oracle
   equality; the same mutation fails 7 of 8.
3. perf(gfql/polars): make the chain combine proportional to the traversal result #1785 — sort(EORD) was called "probably dead" by an engine-blind probe that only
   ran the in-memory collect. Under streaming, deleting it breaks row order. Test now
   parametrized over the collect engine.
4. Two over-broad claims scoped (a maintain_order guarantee that only holds in-memory; a
   start_nodes test that never reaches pattern_apply), and two dead guards removed — one
   being the original spelling of the bug perf(gfql/polars): make the chain combine proportional to the traversal result #1785 fixes, sitting one function below it.

Known limits, stated rather than buried

• indexed-typed-edge-match-hop[pandas] in the lock-in lane does not discriminate — it
  measures 5.29× growth on master but passes, because the 5 ms absolute-delta floor suppresses
  a 2.11 ms delta. The [polars] variant is what actually pins perf(gfql): evaluate indexed edge_match on candidate rows, not the whole edge frame #1782.
• Deleting the fillna(False) before .values in the cudf branch leaves the new GPU tests
  green on cudf 26.02 — (sub == val) already yields non-null booleans there, so that
  fillna is defensive, not load-bearing on this version.
• cudf-polars cannot execute Categorical at all; the baseline scan raises before any index
  code. Pre-existing, identical on master.

Not merging per your instruction — this is yours to land. #1785 stays draft pending its
own SF1 A/B.

[lmeyerov]

x-platform: clean

Full graphistry/tests/compute/gfql/ on the GPU box —
graphistry/test-rapids-official:26.02-gfql-polars, --gpus all, so the cuDF and
polars-GPU lanes actually execute rather than skip:

build	passed	skipped	xfailed	failed
master 4061e7ce	5414	25	15	0
#1783	5414	25	15	0
#1784	5450 (+36 = its own new tests)	25	15	0

Skip counts identical across all three, so nothing was silently skipped into looking green.

Two scoping notes so the numbers aren't over-read:

• perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783's lean-combine is pandas-gated by construction. _lean_engine_ok returns right
  unchanged for cuDF / polars / dask, so the GPU engines cannot exercise that branch at all —
  this is inapplicability, not missing coverage. What the GPU run does validate for perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783 is
  the other half, the polars EORD/EID row-index change, which does run under GPU collect.
• perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783's own new tests live in tests/compute/test_chain_lean_combine.py, outside the
  gfql/ tree, which is why its passed-count matches master rather than rising. Those are
  pandas-path tests and run in CI.