fix(gfql): index AUTO preserves resident polars frames (kills the polars-hop O(E) tax)

[lmeyerov]

Problem (the C3 "polars hop is linear" mystery — root-caused)

g.gfql_index_all() (engine=auto) on a polars graph silently replaced the polars frames with pandas copies: resolve_engine(AUTO) maps polars input frames to Engine.PANDAS (legacy input-format policy), and create_index coerces the graph into the resolved engine before building.

Consequences for every later hop(engine='polars') call:

1. _coerce_input_formats re-converts the FULL nodes+edges frames pandas→polars per call — O(E), ~208ms of a 223ms hop at 4M edges (cProfile: pl.from_pandas new_str on the 4M-row string column + pandas _isna_string_dtype), ~1.7s at 32M.
2. The resident pandas-engine index can never serve: get_valid requires engine match AND frame identity, and the per-call conversion breaks both — so the hop was also a scan.

This is why the C3 ladder showed pandas flat at 0.16ms while polars scaled linearly 16.6→1751ms.

Fix

_resolve_index_engine: when the caller said AUTO and the resident frames are polars, the index API resolves POLARS and indexes the frames in place (the index layer is already engine-polymorphic: numpy sidecar arrays + polars row-gather). Explicit engines are honored unchanged. Deliberately scoped to the index API — this is NOT the global engine=auto routing-policy change (#1743, on hold).

Measured (dgx GB10, container polars 1.35.2, warm median of 30, parity vs pandas indexed oracle at every rung)

edges	polars seeded 1-hop before	after
0.25M	16.6ms	0.169ms
1M	(not measured pre-fix)	0.349ms
4M	221ms	0.800ms (276×)
16M	(not measured pre-fix; 32M was 1751ms)	0.834ms

Frames verified to remain polars after gfql_index_all(); parity OK at every rung.

Tests

test_index.py::TestIndexAutoPreservesPolarsFrames (file already in POLARS_TEST_FILES):

• AUTO keeps polars frames (identity preserved) + polars-engine index resident/valid
• index-engagement spy: resident polars index serves hop(engine='polars'), parity vs pandas indexed oracle
• explicit engine='pandas' still coerces (contract unchanged)

Full bin/test-polars.sh locally: failures identical to master baseline (pre-existing local polars-version drift in the cypher conformance matrix; zero new failures). Full index suite (non-GPU) 85/85.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

[lmeyerov]

can we split out autoindexing from enabling polars by default?

[lmeyerov]

Root cause measured, and it makes this PR not independent — needs a decision, not a restack

Measured today on a 3.18M-node / 14M-edge polars graph (LDBC SF1-shaped), same session, rows validated:

config	seeded point lookup
polars-engine indexes + engine='polars'	0.92 ms
pandas-engine indexes + engine='polars'	808.92 ms

An 879× cliff, silent, no diagnostic. That is the cost this PR is aimed at, and it is much larger than "housekeeping."

Why it happens

Engine.resolve_engine maps polars frames to Engine.PANDAS by design:

if 'polars' in str(type(g_or_df).__module__):
    if isinstance(g_or_df, (pl.DataFrame, pl.LazyFrame)):
        return Engine.PANDAS

So gfql_index_all() with a default engine builds pandas-engine indexes over polars frames. Those fail validation under engine='polars', every seam declines, and you get full scans.

The problem with landing this in isolation

If the index build alone resolves from the frames (this PR's stated fix — build polars indexes on polars frames), then AUTO queries still resolve to PANDAS (that routing is deliberately on hold, #1743). A pandas query would then find a polars index and decline it — making the default path worse for anyone not explicitly passing engine='polars'.

So the options are genuinely coupled:

1. Ship this together with perf(gfql): route engine=auto to native polars for polars-frame graphs #1743 (AUTO → native polars for polars-frame graphs) — the cliff disappears, but that's the policy change currently held.
2. Build indexes for both engines, or defer the engine choice to use time — more work, no policy change.
3. Interim: document + warn — building indexes without naming the engine on a polars graph silently yields an index a polars query cannot use.

I'd suggest not restacking this as-is until that call is made; option 3 is cheap and safe today.

Note on the benchmark lane

The SF1 lane is unaffected: it passes execution_engine: polars and builds its indexes with that engine, so it never hits the cliff. This bites ad-hoc users (and bit my own probes today).

[lmeyerov]

Rebased onto master 3f7f79236 and re-verified (2026-07-29)

Merged current master in (bbf068fa8). One conflict, in test_index.py: both sides appended a test block at the same point — kept both, since TestIndexAutoPreservesPolarsFrames (AUTO must not swap polars frames for pandas copies) and master's candidate-edge-mask shape pins test different things. No duplicate top-level names; AST parses.

Static (local): ./bin/lint.sh clean; MYPY_CMD="uvx mypy==2.3.0" ./bin/mypy.sh → no issues in 327 source files.

Tests (dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars, --gpus all --network none, polars 1.35.2 / pandas 2.3.3):

suite	result
graphistry/tests/compute/gfql/index/	309 passed
graphistry/tests/compute/ (whole tree)	9579 passed, 9 failed, 118 skipped, 19 xfailed

The 9 failures are pre-existing and not from this branch — pristine master 3f7f79236 was run in the same container and produced the identical 9 (7 × TestCpuOnlyPluginsCudfRoundTrip igraph/cudf round-trips, test_engine_coerces_cudf_to_pandas, test_chain_dask_edges). They are igraph/cudf/dask plugin issues in the image, untouched by this diff.

Diff is unchanged in substance: index/api.py +31/-2 (_resolve_index_engine), tests, CHANGELOG. Still deliberately scoped to the index API — not the global engine=auto routing change in #1743, which is on hold.

[lmeyerov]

Retracting my "ready to merge" — the 2026-07-26 objection above is unaddressed, and it is correct

Marking this draft. My previous comment reported a clean rebase and green tests and then called the PR ready. That was wrong: a rebase and a passing suite do not answer the coupling problem raised in this comment, and I skipped straight past it.

Restating the objection, because it is the blocker. With this fix, gfql_index_all() on a polars graph resolves POLARS and indexes in place. But AUTO query routing still resolves to PANDAS — that policy change is #1743, on hold. So for the user who never passes engine=:

	index built	frames	AUTO query resolves	index serves?
master today	pandas	coerced to pandas	PANDAS	yes
this PR	polars	stay polars	PANDAS	no

get_valid requires engine match and frame identity, so the default path plausibly goes from index-hit to scan-plus-per-call-conversion. I improve the explicit-engine='polars' case by 276× and may make the default case worse. Nobody has measured that second column, including me, and shipping on the strength of the first would be exactly the benchmark-shaped move this campaign is supposed to reject.

What I owe before this is reviewable, none of which is done:

1. Measure the AUTO-query path, before vs after, on a polars graph — the cell that decides whether this is a net win or a trade.
2. If it does regress, fix it generally rather than by policy. The most promising direction: the index layer is already engine-polymorphic (numpy sidecars + engine-specific row-gather), so get_valid's engine equality may simply be over-restrictive — letting a polars-built index serve a pandas query over the same unmutated frames would remove the cliff in both directions and would not touch perf(gfql): route engine=auto to native polars for polars-frame graphs #1743's routing policy. That has to be established, not asserted.
3. Lock the numbers in pyg-bench: a before/after lane with a committed runner and raw JSON, covering explicit-polars and AUTO, plus a non-regression arm for pandas-native graphs. Right now the 276× lives only in a PR description, which is not provenance.

The green CI in my last comment stands as a correctness result — 309/309 index tests, and the 9 compute-tree failures proven pre-existing against pristine master in the same container. It was never a perf or default-path result, and I presented it as if adjacent to one.

Not restacking or pushing further until (1) is measured.

[lmeyerov]

Measured: this PR regresses the default path 125–534×, and its win is already reachable on master. It should not land in this form.

dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars, --gpus all --network none, polars 1.35.2 / pandas 2.3.3. BEFORE = 3f7f79236 (master), AFTER = bbf068fa8 (this branch). One container per size × arm × state, 3 interleaved slots, 3 warmups + 15 timed reps, per-slot medians, TIE when slot ranges overlap. Load 0.33–1.08. Eager polars frames, int64 ids, 5 seeds, hop(hops=1, direction='forward').

The default path (gfql_index_all() then hop(), no engine argument anywhere)

E	BEFORE	AFTER
250K	0.133 ms (0.129–0.134)	16.582 ms (15.95–17.06)	125× slower
1M	0.131 ms (0.130–0.132)	32.823 ms (32.59–32.89)	251× slower
4M	0.137 ms (0.135–0.137)	73.007 ms (61.34–75.28)	534× slower

Ranges are nowhere near overlapping (BEFORE max 0.137, AFTER min 15.95). AFTER's time equals the un-indexed scan floor measured in the same run (15.8 / 33.2 / 67.8 ms), so this is precisely "the index build is paid for and then declined". It widens with E, because an index-served hop is O(degree) and flat at ~0.13 ms at every size while the scan is O(E).

Inspected, not inferred from timing — via index_trace() and show_indexes():

arm	BEFORE	AFTER
index_all() + engine='polars'	declined, index path not applicable -> scan	served, frontier below cost gate -> index
index_all() + no engine (default)	served, frontier below cost gate -> index	declined, index path not applicable -> scan
index_all('pandas') + engine='pandas'	served	served (0.133 → 0.137 ms, TIE)

Both conditions decline independently: the engine field mismatches (polars index vs pandas query), and _coerce_input_formats converts the whole node+edge frame polars→pandas on every call, which also breaks get_valid's source_ref is df identity check. Result values were byte-identical across all 88 measurements (per size, one signature for edges/nodes/distinct-dst/sum-dst/first-20-ids) — this is purely a performance change, not a correctness one.

And the win does not need this PR

The cell I had not measured, and should have before opening it. gfql_index_all(engine='polars') then hop(engine='polars') — the user simply naming the engine on both sides — on master:

E	BEFORE (master, explicit)	AFTER (this branch, explicit)	verdict
250K	0.154 ms (0.149–0.166)	0.153 ms (0.147–0.166)	TIE
1M	0.226 ms (0.219–0.229)	0.150 ms (0.150–0.173)	AFTER 1.5×
4M	0.224 ms (0.190–0.501)	0.213 ms (0.154–0.216)	TIE

Master already coerces to polars and builds a polars index when the engine is named — create_index does _coerce_input_formats(g, eng) on the resolved engine. So the fast path is already reachable on master, at 0.15–0.23 ms, without this diff. What this PR actually adds is making AUTO guess polars, and that guess is what breaks the default query path.

That reframes the trade: it is not "276× faster vs housekeeping". It is "saves the user from typing engine='polars', at the cost of 125–534× for everyone who types nothing at all." I also could not reproduce the 276× in the PR description on this synthetic uniform-random graph (arm 1 improves 6–15×); that figure came from a differently-shaped dataset and I am not claiming it is wrong, only that it is not what a reader should expect.

What the data does justify — and it is a real, generalizable bug on master today

The two mirror-image cases are the same pre-existing defect, and both are silent:

• master: AUTO index + engine='polars' query → declined → 2.6 / 4.2 / 8.7 ms instead of 0.15
• this branch: AUTO index + AUTO query → declined → scan floor

Worse, show_indexes() reports valid=True for every index in every cell above, including the ones being declined at query time. A user looking at their own index state is told it is fine while paying a 500× cliff. That is the actionable finding, it needs no policy change, and it fixes both directions rather than moving which caller pays.

Disposition

I am not pursuing this diff. Options 1 and 2 from the earlier comment both stand refuted or blocked: relaxing get_valid's engine equality would not help, because the dominant cost in the regressed arm is the per-call O(E) frame conversion, not the engine field — avoiding that means routing AUTO queries to polars, which is #1743, on hold. So the coupling identified 2026-07-26 is real and not fixable from the index layer.

Proposing instead to close this and replace it with the diagnostic: make an engine-mismatched decline loud (and stop show_indexes() reporting valid=True for an index the resolved query engine cannot use). That is generalizable, cannot regress any path, converts a silent 100–500× cliff into an actionable message, and leaves the eventual fix to #1743 where it belongs. Will open it separately rather than repurposing this branch.

Raw JSON, drivers and aggregators are kept; the arm-5 probe is arm5.py, run interleaved over both clones.