perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup by lmeyerov · Pull Request #1783 · graphistry/pygraphistry

lmeyerov · 2026-07-26T19:32:31Z

Stacked on #1782 (base = perf/gfql-indexed-edge-match-lazy). Two independent, provably-safe reductions of graph-sized work.

1. Empty-left merge no longer scans the right side

_lean_prefilter_right shrinks right to the keys in left before a how='left' merge, but declined when left was empty — the one case where shrinking is both maximally profitable and trivially correct (no left key can match, and a left merge keeps only matched right rows).

Instrumenting a single-node named query on a 3.18M-node / 14M-edge graph showed the dominant work was safe_merge(left=0x1, right=14000000x4, how='left') — a zero-row intermediate joined against every edge in the graph.

Returning right[0:0] keeps identical columns and dtypes, so the merge still yields the same schema. 112.64 → 17.29 ms (6.5×) on the affected shape.

2. Polars: reuse the synthetic edge id as the stable edge order

The native polars chain executor attaches EID via with_row_index, then later materializes a second with_row_index column (EORD) over that same frame. Both are 0..n-1 over the same frame in the same order — the second runs on the frame the first produced — so it is pure duplication of a graph-sized materialization. with_row_index was ~34 ms/query across the eager calls.

148.22 → 135.43 ms (−8.6%) on a seeded typed pattern with polars-engine indexes.

Parity

108 polars-path cases, 0 differ, 0 raised — 9 traversal shapes × 12 random seeds under engine='polars' with polars-engine indexes, hashed over column order, dtypes, row count and full content.
This is a new sweep, and it closes a real blind spot: the existing 650-case differential runs engine=auto, which resolves to pandas and never exercises the polars lazy executor. A regression in that executor would have shipped clean.
Index + lean-combine suites: no new failures.

Scope honesty

Neither change flips a competitor result. Fix 1 helps shapes with an empty intermediate; IS1/IS5 are full n→e→n patterns whose intermediates are non-empty. Fix 2 is −8.6% on the polars path. They are stacked here because they are correct and general, not because they close the gap.

lmeyerov · 2026-07-26T23:58:09Z


 ### Performance
+- **Seeded chain combine stops joining against the full frame when the intermediate is empty**: `_lean_prefilter_right` shrinks the big side of the combine's `how='left'` merge to the keys actually present on the left, but it declined to do so in the one case where shrinking is both maximally profitable and trivially correct — an **empty** left. A left merge keeps only the right rows that match, so a zero-row left yields a zero-row result whatever the right side holds; the merge was nonetheless materializing the whole graph-sized frame. It now hands back a zero-row slice of `right` (same columns and dtypes, so the merge still produces an identical schema). Measured: a single-node query whose 0-row intermediate was joined against 14M edges went **112.64 → 17.29 ms**. The shrink is used at exactly one call site, which is `how='left'`; a merge that retains unmatched right rows (`right`/`outer`) would NOT be safe to shrink this way, and the tests pin both directions — the empty-left case must return an empty result, and the non-empty cases must be byte-identical to the unshrunk merge.
+- **Native polars chain reuses the synthetic edge id as its stable row order**: when the executor had already added a synthetic edge-index column it also added a second, separate row-index column purely to restore input order at the end — two full `with_row_index` passes over the edge frame, and a redundant column carried through every intermediate. The existing synthetic id is already a contiguous, order-preserving row index, so it is now reused as the sort key and dropped once at the end. Only the pre-existing column is reused; when no synthetic id was added the separate order column is still created, so graphs that bring their own edge id are unaffected. Value-identical including row order (148.22 → 135.43 ms on a 3.18M-node / 14M-edge polars graph).


once benefit proven & locked in pyg-bench, and happy with postive/negative tests on either side of opt boundary, and ci still green, can land both sides

lmeyerov · 2026-07-27T01:44:36Z

`[1]` cleared — the lock-in lane was run, per-probe, per-build

pyg-bench #101 is merged (a672a46) and the lane was then run from merged main against
five frozen read-only git archive exports, perf lock held externally for the whole sweep on
a quiet host.

probe	pins	master	#1782	#1783	#1784	#1785
`indexed-typed-edge-match-hop[polars]`	#1782	regression	pass	pass	pass	pass
`empty-left-typed-edge-miss[pandas]`	#1783	regression	regression	pass	pass	pass
`polars-edge-order-materializations`	#1783	regression	regression	pass	pass	pass
`semi-join-key-dedup-node-ladder[polars]`	#1784	regression	regression	regression	pass	pass
`indexed-typed-edge-match-hop[pandas]`	#1782	pass	pass	pass	pass	pass

A monotone staircase: each PR flips exactly its own probes, nothing regresses, and #1785
holds all five. Growth over an 8× ladder:

indexed-typed-edge-match-hop[polars]: master 2.97× (2.55 → 7.59 ms) → perf(gfql): evaluate indexed edge_match on candidate rows, not the whole edge frame #1782 0.93×
(0.28 → 0.26 ms), i.e. flat.
empty-left-typed-edge-miss: master 2.55× (9.20 → 23.45 ms) → perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783 0.998×
(6.67 → 6.66 ms), i.e. flat.

Note the config-level exit code is cumulative — it is the stack gate, so on its own it
reads as "master/#1782/#1783 fail". The per-probe verdicts above are what [1] actually asks
for.

One probe does not discriminate — flagging rather than burying

indexed-typed-edge-match-hop[pandas] reports pass on master. It is not blind — it
measures 5.294× growth there. It passes because the verdict ANDs the ratio gate with a
5.0 ms min_absolute_delta_ms floor, and that variant's absolute delta at this ladder size is
only 2.11 ms. So the floor, which exists to stop flaky sub-noise failures, is suppressing a
real 5.3× growth.

The [polars] variant is what actually pins #1782; the pandas one currently rides along.
It should be fixed by raising base_edges for that probe until the delta clears the floor, or
by giving it a smaller per-probe floor — not by deleting it. It is one config change away from
being load-bearing, and it is a live instance of exactly the failure mode the lane's own README
warns about.

Two run traps worth recording

uv is not on the non-interactive ssh PATH (exit 127 — use $HOME/.local/bin/uv), and the
pyg-bench uv env has no polars (--with polars). Both produced silent-looking failures first.

…; pin edge order Review skill found no correctness defect (the adversarial question — is how='left' really the only consumer of _lean_prefilter_right — came back CONFIRMED: one non-test caller, the local is dead on the next line, and every other merge in chain.py builds its right side independently). These are the quality items. 1. `right[0:0]` is LABEL-based slicing on a float index — pandas routes those through slice_indexer — so it returned ONE row and the shrink silently did nothing. Now `right.iloc[0:0]`, positional on every index type. The RESULT was always correct (a 0-row left yields a 0-row how='left' merge either way), which is why this needed its own test: the bug was an invisible perf no-op, not a wrong answer. Mutation-checked — reverting to `[0:0]` fails the new test. 2. Docstring said "Only shrinks when `left` is materially smaller", which the new empty-left branch contradicts. Restated, and now records that the whole helper is pandas-only in practice (`_lean_engine_ok` is checked first), so there is no cross-engine exposure. 3. Removed `if added_edge_index and EID != EORD:` — dead by construction, since the only branch setting added_edge_index also sets EORD = EID. lint clean; 1022 lean-combine + polars-chain tests pass.

lmeyerov · 2026-07-27T05:27:05Z

Ready to merge — all three bars met

bar	evidence
[1] benefit proven in a lane actually run	pyg-bench #101 merged, then run per-probe per-build from merged `main` against frozen exports. Monotone staircase: each PR flips exactly its own probes, nothing regresses, #1785 holds all five.
[2] pos + neg tests either side of the opt boundary	On all four layers, and every new test mutation-checked — reintroduce the bug, confirm it fails, report what is not caught.
[3] CI green	All heads green, 0 fail, after the review fixes.

Beyond the bar

Review skill on all four layers: no correctness defects. Adversarial targets confirmed
clean — how='left' is provably the only consumer of _lean_prefilter_right; every
.unique() removed in perf(gfql/polars): stop deduplicating semi-join key sides #1784 is a genuine semi key side; perf(gfql/polars): make the chain combine proportional to the traversal result #1785's "known empty" is never
wrong and the dropped gate is genuinely vacuous.
Regression sweep, both engines vs master, rows identical: 0 slower. polars 3 faster,
pandas 5 faster (IS4 −71.0%, IS1 −53.4%, IS5 −41.4%, IS6 −31.6%, IS2 −27.7%). Running
pandas mattered — perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783's lean-combine is pandas-gated, so a polars-only sweep would have
been structurally blind to it.
OLAP/aggregate: zero surviving regressions. Three cells crossed +10% at 20k and all
three were refuted on re-run (polars q5 +13.7% → −11.7%, a sign flip). IC4 −40.0%
polars / −33.3% pandas, IC6 pandas −65.8%.
GPU cross-platform closed on 26.02-gfql-polars: null-bearing gathered cudf columns
across 7 dtypes, the empty device batch, and the eager-mask switch — all against the
pandas oracle, engagement proven rather than assumed. New in-repo gated test file that
skips cleanly without a GPU and passes 12/12 with one.

Competitive result

Second flip, and this one needs no asterisk: IS5 message-creator LOSE 1.22× → WIN 1.32×
(Neo4j 27.28 vs 20.60, non-overlapping reps, rows=1 both sides). Unlike IS2 — which is
adapter_workaround on both sides and must always be disclosed as reduced semantics — IS5 is
official_query_text with exactly-equal rows. IS1 widened to 5.14×, IS2 to 1.58×.

Four weak tests of my own that mutation-checking caught, now fixed

perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783 — right[0:0] label-slices on a float index and returns 1 row, so the
112→17 ms win silently never fired. The result stayed correct, which is exactly why it was
invisible. Now .iloc[0:0].
perf(gfql/polars): stop deduplicating semi-join key sides #1784 — the worst <= 2 bound absorbed precisely the duplication it claimed to catch:
dropping the deliberately-kept .unique() left all 8 params green. Now exact pandas-oracle
equality; the same mutation fails 7 of 8.
perf(gfql/polars): make the chain combine proportional to the traversal result #1785 — sort(EORD) was called "probably dead" by an engine-blind probe that only
ran the in-memory collect. Under streaming, deleting it breaks row order. Test now
parametrized over the collect engine.
Two over-broad claims scoped (a maintain_order guarantee that only holds in-memory; a
start_nodes test that never reaches pattern_apply), and two dead guards removed — one
being the original spelling of the bug perf(gfql/polars): make the chain combine proportional to the traversal result #1785 fixes, sitting one function below it.

Known limits, stated rather than buried

indexed-typed-edge-match-hop[pandas] in the lock-in lane does not discriminate — it
measures 5.29× growth on master but passes, because the 5 ms absolute-delta floor suppresses
a 2.11 ms delta. The [polars] variant is what actually pins perf(gfql): evaluate indexed edge_match on candidate rows, not the whole edge frame #1782.
Deleting the fillna(False) before .values in the cudf branch leaves the new GPU tests
green on cudf 26.02 — (sub == val) already yields non-null booleans there, so that
fillna is defensive, not load-bearing on this version.
cudf-polars cannot execute Categorical at all; the baseline scan raises before any index
code. Pre-existing, identical on master.

Not merging per your instruction — this is yours to land. #1785 stays draft pending its
own SF1 A/B.

`_lean_prefilter_right` shrinks `right` to the keys present in `left` before a how='left' merge. It declined to do so when `left` was EMPTY -- the one case where shrinking is both maximally profitable and trivially correct: no left key can match anything, and a left merge keeps only the right rows that DO match, so the result is empty whatever `right` holds. The cost of declining is graph-sized. Instrumenting a single-node named query on a 3.18M-node / 14M-edge graph showed the dominant work was: safe_merge(left=0x1, right=14000000x4, how='left') i.e. a zero-row intermediate joined against every edge in the graph. Returning a zero-row slice of `right` keeps the identical columns and dtypes -- so the merge still produces the same schema -- without materializing the frame. Measured on that graph: a named seed-only query goes 112.64 -> 17.29 ms (6.5x), same result shape. Patterns whose intermediates are non-empty are unaffected. Parity: 650 differential comparisons (pandas + polars x 13 shapes x 25 seeds, dense graph, null-carrying columns) -- 0 divergences. Index + lean-combine suites: no new failures (38 pre-existing GPU-image failures unchanged, 184 passed). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VARwnAqmvczhz5EmP7TYyZ

The native polars chain executor attaches a synthetic edge id (`EID`) via `with_row_index` when the graph has no edge binding, then — a few hundred lines later — materializes a SECOND `with_row_index` column (`EORD`) over that same frame to restore eager edge order for the fused combine. Both are 0..n-1 over the same frame in the same order, because the second runs on the frame the first produced, so the second is pure duplication of a graph-sized materialization. Reuse EID as EORD when it was attached. Both columns were already dropped before returning, so the single drop now covers it (guarded on `EID != EORD` for the case where the graph brought its own edge binding and EORD is still a separate column). Measured on a 3.18M-node / 14M-edge polars graph with polars-engine indexes, a seeded typed pattern: 148.22 -> 135.43 ms (-8.6%). `with_row_index` was ~34 ms/query across the eager calls before this. Parity: 108 polars-path cases (9 traversal shapes x 12 random seeds, engine='polars' with polars-engine indexes) hashed over column ORDER, dtypes, row count and full content -- 0 differ, 0 raised. This is a new sweep: the existing 650-case differential runs engine=auto, which resolves to pandas and never exercises this executor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VARwnAqmvczhz5EmP7TYyZ

…ine shrink Owner's merge conditions for this PR were a CHANGELOG entry, correctness tests in both directions, and a pyg-bench lane (the last is separate). Tests pin both sides of the new empty-left branch: - empty left shrinks to zero rows AND keeps columns/dtypes identical, so the downstream merge still produces the same schema; - the shrunk merge equals the unshrunk merge at the call site's actual how='left'; - non-empty lefts (1, 5, 999, 1000 rows — spanning both sides of the ratio gate) are byte-identical to the unshrunk merge, i.e. the new branch perturbs nothing; - a non-empty left whose keys miss entirely still yields one null-filled row per left row, rather than being collapsed; - end-to-end: a seed matching nothing gives lean-on == lean-off through chain(). Mutation-checked. Reverting the branch to `return right` fails the shrink test. A second mutation (also shrink when no keys overlap) is NOT caught — correctly: under how='left' both forms give null-filled rows, so there is no behaviour to distinguish. The docstring says so rather than implying coverage it does not have. Also records the safety precondition in the CHANGELOG: the shrink is used at exactly one call site, which is how='left'; a how='right'/'outer' merge retains unmatched right rows and would NOT be safe to shrink this way.

…; pin edge order Review skill found no correctness defect (the adversarial question — is how='left' really the only consumer of _lean_prefilter_right — came back CONFIRMED: one non-test caller, the local is dead on the next line, and every other merge in chain.py builds its right side independently). These are the quality items. 1. `right[0:0]` is LABEL-based slicing on a float index — pandas routes those through slice_indexer — so it returned ONE row and the shrink silently did nothing. Now `right.iloc[0:0]`, positional on every index type. The RESULT was always correct (a 0-row left yields a 0-row how='left' merge either way), which is why this needed its own test: the bug was an invisible perf no-op, not a wrong answer. Mutation-checked — reverting to `[0:0]` fails the new test. 2. Docstring said "Only shrinks when `left` is materially smaller", which the new empty-left branch contradicts. Restated, and now records that the whole helper is pandas-only in practice (`_lean_engine_ok` is checked first), so there is no cross-engine exposure. 3. Removed `if added_edge_index and EID != EORD:` — dead by construction, since the only branch setting added_edge_index also sets EORD = EID. lint clean; 1022 lean-combine + polars-chain tests pass.

lmeyerov · 2026-07-27T05:52:08Z

x-platform: clean

Full graphistry/tests/compute/gfql/ on the GPU box —
graphistry/test-rapids-official:26.02-gfql-polars, --gpus all, so the cuDF and
polars-GPU lanes actually execute rather than skip:

build	passed	skipped	xfailed
master `4061e7ce`	5414	25	15
#1783	5414	25	15
#1784	5450 (+36 = its own new tests)	25	15

Skip counts identical across all three, so nothing was silently skipped into looking green.

Two scoping notes so the numbers aren't over-read:

perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783's lean-combine is pandas-gated by construction. _lean_engine_ok returns right
unchanged for cuDF / polars / dask, so the GPU engines cannot exercise that branch at all —
this is inapplicability, not missing coverage. What the GPU run does validate for perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783 is
the other half, the polars EORD/EID row-index change, which does run under GPU collect.
perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup #1783's own new tests live in tests/compute/test_chain_lean_combine.py, outside the
gfql/ tree, which is why its passed-count matches master rather than rising. Those are
pandas-path tests and run in CI.

lmeyerov mentioned this pull request Jul 26, 2026

perf(gfql/polars): stop deduplicating semi-join key sides #1784

Merged

lmeyerov force-pushed the perf/gfql-lean-empty-left branch from 77e84b0 to 7a72a4c Compare July 26, 2026 23:24

lmeyerov mentioned this pull request Jul 26, 2026

perf(gfql): kill per-call O(E) terms in seeded Cypher/chain queries (#1755 slice 1) #1757

Closed

2 tasks

lmeyerov commented Jul 26, 2026

View reviewed changes

lmeyerov force-pushed the perf/gfql-lean-empty-left branch from 56c7949 to 5f4cade Compare July 27, 2026 00:14

lmeyerov mentioned this pull request Jul 27, 2026

perf(gfql/polars): make the chain combine proportional to the traversal result #1785

Merged

lmeyerov mentioned this pull request Jul 27, 2026

perf(gfql): evaluate indexed edge_match on candidate rows, not the whole edge frame #1782

Merged

lmeyerov and others added 4 commits July 26, 2026 22:41

lmeyerov force-pushed the perf/gfql-lean-empty-left branch from c08afb5 to 24ce01f Compare July 27, 2026 05:52

lmeyerov changed the base branch from perf/gfql-indexed-edge-match-lazy to master July 27, 2026 06:13

lmeyerov closed this Jul 27, 2026

lmeyerov reopened this Jul 27, 2026

lmeyerov merged commit 002bb91 into master Jul 27, 2026
68 checks passed

lmeyerov deleted the perf/gfql-lean-empty-left branch July 27, 2026 06:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup - #1783

perf(gfql): empty-left merge shrink + polars EORD/EID row-index dedup#1783
lmeyerov merged 4 commits into
masterfrom
perf/gfql-lean-empty-left

lmeyerov commented Jul 26, 2026

Uh oh!

lmeyerov Jul 26, 2026

Uh oh!

lmeyerov commented Jul 27, 2026

Uh oh!

lmeyerov commented Jul 27, 2026

Uh oh!

lmeyerov commented Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

lmeyerov commented Jul 26, 2026

1. Empty-left merge no longer scans the right side

2. Polars: reuse the synthetic edge id as the stable edge order

Parity

Scope honesty

Uh oh!

lmeyerov Jul 26, 2026

Choose a reason for hiding this comment

Uh oh!

lmeyerov commented Jul 27, 2026

[1] cleared — the lock-in lane was run, per-probe, per-build

One probe does not discriminate — flagging rather than burying

Two run traps worth recording

Uh oh!

lmeyerov commented Jul 27, 2026

Ready to merge — all three bars met

Beyond the bar

Competitive result

Four weak tests of my own that mutation-checking caught, now fixed

Known limits, stated rather than buried

Uh oh!

lmeyerov commented Jul 27, 2026

x-platform: clean

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`[1]` cleared — the lock-in lane was run, per-probe, per-build