Skip to content

RangeSetND: speed up multivariate folding - #656

Merged
thiell merged 1 commit into
clustershell:masterfrom
thiell:perf/rangesetnd-fold
Jul 30, 2026
Merged

RangeSetND: speed up multivariate folding#656
thiell merged 1 commit into
clustershell:masterfrom
thiell:perf/rangesetnd-fold

Conversation

@thiell

@thiell thiell commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #485 — rewrite of the nD folding internals: expand groups unique elements by leading coordinates in one pass (no per-element objects), and merge replaces pairwise vector comparison with hash-based grouping on the non-differing axes.

shape before after
x[2000-2073]c[0-7]s[0-7]b[0-1] from 9472 nodes (#485) 0.32s 0.06s
3000 scattered 2D points 26s 0.02s
4000 scattered 3D points 39s 0.05s
two overlapping 300x300 boxes 11.2s / 237MB 0.02s / 15MB
100x100 grid from 10000 nodes (already fast) 0.19s 0.02s
100k scattered 4D xnames, cluset -f end to end impractical 6.2s

Folded results always represent the same node set, but the decomposition of overlapping nD inputs may differ (often fewer vectors); test expectations updated, as previously done in ec14536.

Validation:

  • 44k+ differential fuzz cases vs master (element sets, length, padding, disjointness, idempotency): zero failures
  • output canonicity verified across build paths and hash seeds
  • identical fold output digests on Python 2.7.5, 3.9 and 3.13; test suite green on all three
  • no shape found where the new code is slower

Also adds a short "How nD folding works" overview to the docs.

@thiell thiell added this to the 1.11 milestone Jul 9, 2026
@thiell thiell self-assigned this Jul 9, 2026
@thiell
thiell requested a review from degremont July 9, 2026 18:17
@thiell
thiell force-pushed the perf/rangesetnd-fold branch from dfcb118 to 22c14b1 Compare July 29, 2026 22:35
@thiell

thiell commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator Author

Refreshed after rebasing onto master bcc8d83, which now includes #657 and #659 — the numbers in the description predate both.

Changes in the branch since the last push

  • the merge phase unions through the public update() instead of a raw set.update(), so the sorted-element cache added by RangeSet: cache sorted elements between mutations #657 is always invalidated; the two remaining raw set calls only fill freshly created vectors and carry a # fresh set marker
  • a new test rejects any other raw set mutation in RangeSet.py
  • test_folding_canonical: the same 13 nodes built three ways — as two overlapping boxes, as the same boxes in the opposite order, and as 13 individual nodes in scrambled order — must fold to an identical string, so the result depends only on the node set and not on how the caller assembled it
  • test_fold_update_fold gains a difference_update() step: that operation leaves an axis RangeSet referenced by more than one vector, so it is where an in-place mutation or a stale cached view would show up first

Fold, timed in isolation — CPython 3.13, input rebuilt for every repetition, min of 3, one process per measurement. Both columns come from the same run, so absolute values differ slightly from the table in the description.

input master bcc8d83 this branch speedup
x[2000-2073]c[0-7]s[0-7]b[0-1] from 9472 nodes (#485) 351 ms 81 ms 4.3x
100x100 grid from 10000 nodes 193 ms 24 ms 7.9x
3000 scattered 2D points 295 ms 9.6 ms 31x
4000 scattered 3D points 17.6 s 38 ms 458x
5000 scattered 4D points > 300 s 90 ms > 3300x
two overlapping 300x300 boxes 6.42 s, 248 MB peak 18 ms, 15 MB peak 353x, 16x less memory
3103 xnames of a 60%-populated cabinet 5.06 s 38 ms 134x

End to end on that last one, nodeset -f (cluset -f) over the 3103 names: 5.20 s → 0.25 s. Node name parsing is untouched and dominates the cheaper cases, so end-to-end gains are smaller than fold-only gains except where the old fold was the bottleneck.

Validated against master with a local differential harness (not part of this PR): a catalog of generated shapes — 1D to 6D, dense grids, sparse scatter, overlapping and nested boxes, diagonals, mixed padding, autostep, real cluster layouts — plus randomized fuzzing, all confirming that both trees fold to exactly the same node set, that the output is independent of hash seed and build order, and that no shape is measurably slower.

One correction to the description: it says the decomposition of overlapping nD input "often" has fewer vectors; on a larger sample it is closer to even, with roughly as many cases folding to fewer vectors as to more. The node set is identical either way.

@degremont degremont left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impressive. I did not validate the new algo.

OK. Now let's have cluster nodenames with 5 dimensions everywhere! 😁

Expand by grouping unique elements by leading coordinates and merge
vectors that differ by only one axis using hash-based grouping,
instead of pairwise vector comparisons. Folded results are equivalent
but may now use fewer vectors; tests updated. Add a brief overview of
nD folding to the docs.

Merging updates range sets through the public API to always drop their
cached sorted view; a new test keeps the remaining raw set mutations
restricted to freshly created sets.

Folding time (Python 3.13):
- x[2000-2073]c[0-7]s[0-7]b[0-1] as 9472 nodes (clustershell#485): 0.27s -> 0.06s
- 3000 scattered 2D points: 29s -> 0.02s
- 4000 scattered 3D points: 41s -> 0.04s
- two overlapping 300x300 boxes: 5.2s -> 0.01s, peak memory 16x lower
- 100x100 grid from 10000 nodes, already fast before: 0.17s -> 0.02s

No shape was found where the new code is slower, including with
adversarial search. In theory an input could require many merge
passes and slow down, but in practice merging always settles within
two passes.

Signed-off-by: Stephane Thiell <stephane@thiell.com>
@thiell
thiell force-pushed the perf/rangesetnd-fold branch from 22c14b1 to 3eff473 Compare July 30, 2026 12:19
@thiell
thiell added this pull request to the merge queue Jul 30, 2026
Merged via the queue into clustershell:master with commit 94a06fc Jul 30, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants