fix(datafusion): isolate pools for concurrent spilling executions

[lance-gatefixer]

Root cause

The original CI failure ran before #9183 sized the default memory pool by the effective DataFusion partition count. New production evidence on #9402 exposed a remaining failure: concurrent scalar BTree trainings use spilling sorts, but execute_plan reused a cached SessionContext and its single FairSpillPool. Each ExternalSorterMerge can need up to 40 MiB of non-spillable memory when it re-reserves after a spill. Several trainings can therefore exhaust the shared pool even when each sort fits by itself.

Fix

Give each spilling caller a fresh session and bounded pool; continue caching non-spilling sessions. Keep the named 40 MiB merge reservation limit and the partition headroom regression guard. The new regression holds three spilling contexts with the same 150 MiB configuration at once and reserves 100 MiB in each; the old shared-pool behavior fails on the second reservation.

Each spilling execution retains its configured memory-pool and temporary-disk budgets, but concurrent executions can consume multiples of LANCE_MEM_POOL_SIZE and LANCE_MAX_TEMP_DIRECTORY_SIZE. On resource-constrained hosts, limit concurrent index builds or size those per-execution budgets accordingly.

Validation

• cargo test -p lance-datafusion (183 unit tests and 5 doctests passed; 3 existing ignored doctests)
• cargo test -p lance --test scalar_index_spill (passed)
• cargo test -p lance --features slow_tests --test integration_tests query::primitives::test_query_integer (8 passed, including case_7_uint32)
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings
• RUSTDOCFLAGS='-D warnings' cargo doc --workspace --no-deps

Fixes #9402

[lance-gatefixer]

Blocked: The current-head rustdoc check fails on an inherited private intra-doc link from current main.

Remote head e785b2c contains current base tip 83d0084; rustdoc job 105692983695 reports that public documentation for with_k links to private Self::try_new_batch, and the base-tip rustdoc job 105689817498 reports the identical diagnostic. I fetched both refs, verified the base ancestry, compared the implicated source between base and repair head, and confirmed the repair patch does not modify it. The smallest next action is to merge a correction for the upstream #9334 documentation link into main and then update/rerun this PR; alternatively, explicitly authorize carrying that unrelated one-line documentation correction on this repair branch.

[lance-gatefixer]

No code change: the separate pools are intentional to prevent concurrent spilling sorts from exhausting one shared reservation budget. I updated the PR description to document that both memory and temporary-disk use can scale with concurrent builds, and to recommend limiting concurrency or tuning per-execution budgets on constrained hosts.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

Fresh spilling sessions prevent concurrent scalar index-training sorts from exhausting one shared pool. This remains a reasonable local remedy for reservation loss in pinned DataFusion 54.1.0; the upstream fix is not in that dependency.

The author confirmed that independent pools are intentional and documented the accepted consequence: concurrent builds can multiply memory and temporary-disk use. Each execution remains bounded; on constrained hosts, limit build concurrency or tune LANCE_MEM_POOL_SIZE and LANCE_MAX_TEMP_DIRECTORY_SIZE as per-execution budgets.

[lance-gatefixer]

Blocked: PR #9408 at remote head 1dca2f0 remains blocked by a failed current-head Windows check and required approval.

The Windows job failed in test_torch_index_with_nans[V3] with a PyTorch Inductor PermissionError opening a generated temporary file; the available evidence does not establish that this repair caused it. I re-read the issue, Reviews, and inline threads, verified the prior App disposition after the sole substantive review, and confirmed the later Gatekeeper Review only reconciles that finding. No new review-driven code change remains. The next step is a separate CI-failure pass to rerun the Windows job and investigate temporary-file access if it recurs, followed by maintainer approval. If the rerun passes, only the required approval remains.