Rule sort refactor

[mmatera]

This PR implements an alternative way to sort rules, which seems to be closer to the one in WMA. The rule order is not a total order but a partial order, where one rule comes before other rule if the first is more specific than the second. Two rules like F[x_Integer] and F[x_Real] are not comparable, so they are added to a set of rules (let's say, downvalues) in the order they are introduced.

This would solve #1978

[mmatera]

This rule must change, because otherwise would come before CubeRoot[n_Complex], just because WL rules come before FunctionRules.

[mmatera]

Now FunctionApplyRules come after RewriteRules.

[mmatera]

If the idea is right, this function would play the same role that now play the attribute pattern_precedence. By now it is implemented from "outside" to be able to compare behaviors.

[rocky]

I am glad to see this long-standing problem or black hole being understood. Thanks for putting us on that path.

I will be looking at the code in more detail later.

One thing that bothers me, because I thought or hoped we went over it, is the terminology. At minimum, it is misleading and may hide various algorithmic choices.

(A reason for defining concepts precisely is that if one is looking for pre-existing solutions and technology, as one should try to do, it is easier to find a fit if more precise and commonplace nomenclature is used.)

"Sorting" is something that is done on an entire set of elements. A sort key is used in a sort as a comparison function.

Here, there is no requirement to "sort" anything.

The requirement is that there be an ordering between patterns which determines which pattern to try first. The ordering here is a total ordering because for any pair of arbitrary patterns there is always one "right" answer.

The terms "order" and "precedence" are, I think, a better fit for what we have here: whenever there is a possibility of several pattern matches, one of these takes precedence over the others, or precedes the other possibilities. In the (hopefully rare) situation we have exhausted all possibilities, then yes, we will have done the work to sort the patterns.

But it is an algorithmic choice whether we want to sort elements eagerly or do this lazily.

Yes, ordering that is based solely on properties of a pattern and that does not consider the time or order at which the pattern was defined is a partial order. But once the time at which a pattern was defined is included, we have a total ordering.

It is also an algorithmic choice whether we want to break up the total ordering into components, some of which are partial orderings.

[mmatera]

I am glad to see this long-standing problem or black hole being understood. Thanks for putting us on that path.

I was working on this for a while, and this is just a basic prototype to discuss the general idea. What maybe is the most valueble thing here are the tests that I added and that I will add later.

I will be looking at the code in more detail later.

One thing that bothers me, because I thought or hoped we went over it, is the terminology. At minimum, it is misleading and may hide various algorithmic choices.

(A reason for defining concepts precisely is that if one is looking for pre-existing solutions and technology, as one should try to do, it is easier to find a fit if more precise and commonplace nomenclature is used.)

I agree with this. However, since I found some problems fitting this behaviour on the concepts I have in mind, I found it better to make my findings explicit, and then try to narrow them.

"Sorting" is something that is done on an entire set of elements. A sort key is used in a sort as a comparison function.

Here, there is no requirement to "sort" anything.

This difference is something that confuses me a little bit: in Spanish, we have a single verb for both words "ordenar", and I do not understand the nuance between "sort" and "order".

The requirement is that there be an ordering between patterns which determines which pattern to try first. The ordering here is a total ordering because for any pair of arbitrary patterns there is always one "right" answer.

From mathematics, a total order is something that decides which of two elements comes "first", let's say, real numbers have a total order because if you take two different real numbers x and y, x<y or y<x. In matrix analysis, the order between two matrices is a "partial order", defined by the condition X-Y>0 if X-Y has positive eigenvalues. So it could happen that X>Y, X<Y or the order is indecidible. In both cases, the order (or not) is an intrinsic property of the elements, not something external like the history of how these elements are presented.

The terms "order" and "precedence" are, I think, a better fit for what we have here: whenever there is a possibility of several pattern matches, one of these takes precedence over the others, or precedes the other possibilities. In the (hopefully rare) situation we have exhausted all possibilities, then yes, we will have done the work to sort the patterns.

Precedence is a closer concept: a b^c^d is interpreted as a*(b^(c^d)) because ot the operator precedence, and a grouping rule, which defines the "associativity". However, I am not sure if it is the same that the concept of "rule precedence": in this other case, the precedence order depends both on the partial order of the rules (r<s if r is "more specific than" s) and the time at which both rules are introduced in the definition. So, I was doing some research on how to reproduce what WL does using different algorithms (like bisect left, bisect right and the one I put here) and the one that matches better was this one.

But it is an algorithmic choice whether we want to sort elements eagerly or do this lazily.

Yes, ordering that is based solely on properties of a pattern and that does not consider the time or order at which the pattern was defined is a partial order. But once the time at which a pattern was defined is included, we have a total ordering.

Indeed, we always have that trick: add a timestamp. But I am not sure if this is the most sensible way to get it.

It is also an algorithmic choice whether we want to break up the total ordering into components, some of which are partial orderings.

So, in resume, yes, it should be better implementations, and probably there is an specific name for this particular way to assign precedence to rule transformations. But at some point, I would like to find something that works and can be described in clear terms. What this does is fix one issue, and avoid breaking almost anything of the existing tests (up to small adjustments on some particular rules)
The next step would be to make the _shape parameter behave closer to the pattern_precedence attribute, which is defined on each class, and see what is still wrong with the order in which the many, many rules produced in Rubi are stored.

[rocky]

I am glad to see this long-standing problem or black hole being understood. Thanks for putting us on that path.

I was working on this for a while, and this is just a basic prototype to discuss the general idea. What maybe is the most valueble thing here are the tests that I added and that I will add later.

Absolutely! Very useful and thanks!

...

"Sorting" is something that is done on an entire set of elements. A sort key is used in a sort as a comparison function.
Here, there is no requirement to "sort" anything.

This difference is something that confuses me a little bit: in Spanish, we have a single verb for both words "ordenar", and I do not understand the nuance between "sort" and "order".

I asked Gemini what Donald Knuth's Volume 3 book "Sorting and Searching" is called in Spanish. Let me know if what it reports makes sense:

me:
What is the title of Donald Knuth's book Sorting and Searching called in Spanish?

gemini:

The title of Donald Knuth's book Sorting and Searching (Volume 3 of The Art of Computer Programming) in Spanish is:

Clasificación y búsqueda

... While the word "ordenamiento" is often used in Latin America for "sorting" (e.g., Ordenamiento y búsqueda), the official published Spanish edition by Editorial Reverté uses Clasificación y búsqueda.

me

The requirement is that there be an ordering between patterns which determines which pattern to try first. The ordering here is a total ordering because for any pair of arbitrary patterns there is always one "right" answer.

mmatera

From mathematics, a total order is something that decides which of two elements comes "first", let's say, real numbers have a total order because if you take two different real numbers x and y, x<y or y<x. In matrix analysis, the order between two matrices is a "partial order", defined by the condition X-Y>0 if X-Y has positive eigenvalues. So it could happen that X>Y, X<Y or the order is indecidible. In both cases, the order (or not) is an intrinsic property of the elements, not something external like the history of how these elements are presented.

In OO languages, there is fluidity between what is stored in the object and what is outside of it. A pattern can have a time as part of the parameter. "Time:" is intended in the Lamport Timestamp sense (which was derived from the relativity time event sense, or a simulation time counter). It does not have to be, and probably would not be, a calendar date and time.

The terms "order" and "precedence" are, I think, a better fit for what we have here: whenever there is a possibility of several pattern matches, one of these takes precedence over the others, or precedes the other possibilities. In the (hopefully rare) situation we have exhausted all possibilities, then yes, we will have done the work to sort the patterns.

Precedence is a closer concept: a b^c^d is interpreted as a*(b^(c^d)) because ot the operator precedence, and a grouping rule, which defines the "associativity". However, I am not sure if it is the same that the concept of "rule precedence":

I see it as a direct extension of pattern operator precedence, which is definitely a thing. When I write in Python:

m = r"(?P<tag1>a)|(?P<tag2>.)|(?P<tag3>[a-z])")
print(m.group("tag1")
print(m.group("tag2")
print(m.group("tag3")
yy = r"(?P<tag3>[a-z])|(?P<tag1>a)|(?P<tag2>.)"
m2 = re.match(yy, "a")
print(m.group("tag3")

This feels very much the same thing that is happening in WMA.

A search for the word Precedence used this way turns up from 22.3.5.2 Operator Precedence from Oracle's SQL documentation describing pattern matching.

in this other case, the precedence order depends both on the partial order of the rules (r<s if r is "more specific than" s) and the time at which both rules are introduced in the definition. So, I was doing some research on how to reproduce what WL does using different algorithms (like bisect left, bisect right and the one I put here) and the one that matches better was this one.

Please give me some time to double check this.

But it is an algorithmic choice whether we want to sort elements eagerly or do this lazily.
Yes, ordering that is based solely on properties of a pattern and that does not consider the time or order at which the pattern was defined is a partial order. But once the time at which a pattern was defined is included, we have a total ordering.

Indeed, we always have that trick: add a timestamp. But I am not sure if this is the most sensible way to get it.

Again, to be clear, a timestamp in the Lamport sense. And I agree; I don't know if this is the best way to implement it. It could be done as the position that a pattern appears in a list.

But while that's an implementation choice which we may or may not use, the concept still is that there is a total order and the ordering function depends on the pattern as well as its timestamp, however it is implemented.

It is also an algorithmic choice whether we want to break up the total ordering into components, some of which are partial orderings.

So, in resume, yes, it should be better implementations,

I wasn't suggesting anything about the implementation. The first step is to understand the concepts clearly and cleanly and in a way that it is likely to be found elsewhere or generally used and understood.

and probably there is an specific name for this particular way to assign precedence to rule transformations. But at some point, I would like to find something that works and can be described in clear terms. What this does is fix one issue, and avoid breaking almost anything of the existing tests (up to small adjustments on some particular rules) The next step would be to make the _shape parameter behave closer to the pattern_precedence attribute, which is defined on each class, and see what is still wrong with the order in which the many, many rules produced in Rubi are stored.

I appreciate that it solves a specific problem while not breaking other things. I just want to make sure we don't get into the situation where we are trying to solve a Rubik's Cube by looking at one piece at a time when the problem is better thought of in terms of the subgroups of the cube.

[rocky]

I have been thinking more about the "sort" vs. "order" usage. In English, outside of computer and mathematical use, they mean about the same thing.

However, in mathematics, ordering with the "ing" refers to an existential property, while in computer science, "sort" is a materialization into a list of the existential property.