insert_rule: find the rule to replace through an index, not by a full scan

[tombouctou]

Fixes #1994.

When an unconditional rule is added, insert_rule looks for an existing rule with the same pattern so that it can be replaced. It does this by calling sameQ against every rule already stored. Building a definition of n rules therefore takes O(n²) time, and loading data files written by Save becomes impractical:

Clear[g];
Do[g[i, 2, 3] = i, {i, 1, 4000}];
AbsoluteTiming[Do[g[i, 2, 3] = i, {i, 4001, 5000}]]   (* 18.7 s on master *)

Change. Each Definition now keeps an index of the unconditional rules in each of its rule lists, keyed by a fingerprint of the rule's pattern.

• The fingerprint is the structure of the pattern expression, keeping integers, strings and symbols and putting every other atom in a single bucket. So patterns that are SameQ always get the same fingerprint (Reals of different precision may be SameQ, which is why they share one).
• The search: sameQ is called only on the rules in the new rule's bucket, and the rule found is then located in the list by binary search on its own precedence key.
• Keeping the index valid: an index belongs to the list object it was built from, and is rebuilt lazily when that list is replaced (ClearAll, DownValues[f] = {...}). Rules removed by Unset are dropped from it when they are met. Definitions pickled before this change get an index on first use.

Why not use the precedence key itself? I tried that first. The key of a left-hand side such as f[1] depends on the Flat/OneIdentity attributes of f, and of any inner heads, at the time the rule is created. So f[1] = a; SetAttributes[f, Flat]; f[1] = b would have kept both rules.

Behaviour is unchanged. I added a temporary check to insert_rule that compared, on every insertion, the rule found through the index with the one the old full scan finds. It reported no difference over the whole pytest suite and the doctests. The new tests in test/core/test_definitions.py cover replacement, replacement after the head or an inner head gets attributes, replacement after Unset, lists set as a whole in an order of their own, and conditional rules, which are never replaced. They pass on master as well. Another test counts sameQ calls when adding a rule to a definition of 500 rules; it fails on master.

Timings:

	master	this PR
1000 assignments after 4000 (above)	18.7 s	0.45 s
first 1000 assignments	2.3 s	0.43 s
a 10 MB file of ~90,000 assignments (written by Save)	not finished in 15 min	204 s, all 89,000 values as expected

pytest and the doctests give the same results as on master. The only failures are environmental ones, which also fail on master: XML/ZIP/JPEG FileFormat, and the CLI tests that need mathics on the PATH. mypy is clean on mathics/core/definitions.py.

Unrelated to this PR, but noticed while testing: DownValues[f] = {HoldPattern[f[1]] :> c} stores the left-hand side as HoldPattern[HoldPattern[f[1]]]. So a later f[1] = d adds a second rule instead of replacing it. This happens on master too.

🤖 Generated with Claude Code

[mmatera]

Notice that it would also happen that Condition wraps the pattern too.

[tombouctou]

Right. This PR only moves the old inline is_conditional out of insert_rule; which rules count as conditional is the same as on master. I checked the three places a Condition can sit, on master and on this branch, with the same results on both:

definitions	rules kept
f[x_ /; x > 0] = 1; f[x_ /; x > 0] = 2 (inside an argument)	1, replaced
f[x_] /; x > 0 := 1; f[x_] /; x > 0 := 2 (wrapping the whole left-hand side)	2
f[x_] := 1 /; x > 0; f[x_] := 2 /; x > 0 (on the right-hand side)	2

In the first case the condition is just part of the pattern, so sameQ and the fingerprint both see it, and a repeated definition replaces the old one. In the other two the rule is treated as conditional and never replaced, before and after this PR. As far as I know, WMA replaces a definition whose left-hand side and condition are both the same, so the last two rows may be worth changing. But that would change behavior, so I'd keep it out of this PR, which is meant to be a pure speed-up.

[mmatera]

With this PR, and #1994, I would like to wait a little bit. We have a problem with the ordering of rules and with the (lack of a) dispatch rules mechanism. I am working on that problem in a separate branch. However, it is worth getting some fresh ideas on this topic.

[tombouctou]

Sure, no hurry. Some background on where this and our other recent PRs come from.

We are trying to run Neil Strickland's Mathematica package for Toda's tables of unstable homotopy groups of spheres (strickland1.org/toda) unmodified in Mathics3:

• Toda.m builds a rule base of about 960 relations, adding them one by one.
• LambdaAlgebra.m computes in the lambda algebra.
• LambdaData.m is a 10 MB file written by Save, with about 90,000 assignments.

Each Mathics3 problem we hit we turn into a small, self-contained fix with its own tests, and we report it here.

This PR is the LambdaData.m case: with the full scan, adding the n-th rule costs O(n), and the file did not finish loading in 15 minutes; with the index it loads in about 200 s.

With all our open fixes applied, Toda.m now loads in full (962 relations, against 510 before) and agrees with an independent Python port:

• all 182 stored groups agree;
• 91 of 92 simplification queries agree, and the last one is the Flat matching bug fixed in Flat patterns: a blank can take several elements before a literal; no element is lost #2011.

If your dispatch branch replaces this mechanism, that's fine with us. We're glad to run these files against it as a benchmark when it's ready: they make a fairly demanding test for rule ordering and lookup. And if it helps, the fingerprint here (head/argument structure, keeping integers, strings and symbols) could be one building block for dispatching.

[rocky]

Sure, no hurry. Some background on where this and our other recent PRs come from.

We are trying to run Neil Strickland's Mathematica package for Toda's tables of unstable homotopy groups of spheres (strickland1.org/toda) unmodified in Mathics3:

• Toda.m builds a rule base of about 960 relations, adding them one by one.
• LambdaAlgebra.m computes in the lambda algebra.
• LambdaData.m is a 10 MB file written by Save, with about 90,000 assignments.

Each Mathics3 problem we hit we turn into a small, self-contained fix with its own tests, and we report it here.

This PR is the LambdaData.m case: with the full scan, adding the n-th rule costs O(n), and the file did not finish loading in 15 minutes; with the index it loads in about 200 s.

With all our open fixes applied, Toda.m now loads in full (962 relations, against 510 before) and agrees with an independent Python port:

• all 182 stored groups agree;
• 91 of 92 simplification queries agree, and the last one is the Flat matching bug fixed in Flat patterns: a blank can take several elements before a literal; no element is lost #2011.

If your dispatch branch replaces this mechanism, that's fine with us. We're glad to run these files against it as a benchmark when it's ready: they make a fairly demanding test for rule ordering and lookup. And if it helps, the fingerprint here (head/argument structure, keeping integers, strings and symbols) could be one building block for dispatching.

Thanks for the background. It is awesome that you are doing this; I am grateful you selected Mahics3.

If you or Dr. Strickland want to create a repository in the Mathics3 organization with any of you as an owner, I'd be happy to set that up.