Skip to content

Add reflected set operators to AbstractSet - #16328

Open
afonsojanu wants to merge 3 commits into
python:mainfrom
afonsojanu:fix/abstractset-reflected-dunders
Open

Add reflected set operators to AbstractSet#16328
afonsojanu wants to merge 3 commits into
python:mainfrom
afonsojanu:fix/abstractset-reflected-dunders

Conversation

@afonsojanu

Copy link
Copy Markdown
Contributor

Closes #15532

collections.abc.Set defines __rand__, __ror__, __rsub__, and __rxor__ alongside their forward counterparts, but the stub only had __and__, __or__, __sub__, and __xor__. Without them, a Protocol expecting just the reflected method (as in the issue's example) rejects set()/frozenset() even though both satisfy it at runtime.

For __rand__/__ror__/__rxor__ this is exact: CPython's _collections_abc.Set assigns them as literal aliases of __and__/__or__/__xor__ (the same function object), which tracks with intersection, union, and symmetric difference not caring which operand comes first.

__rsub__ has its own separate body, though, since subtraction isn't commutative:

def __rsub__(self, other):
    if not isinstance(other, Set):
        if not isinstance(other, Iterable):
            return NotImplemented
        other = self._from_iterable(other)
    return self._from_iterable(value for value in other
                               if value not in self)

The result's elements come from whatever other contributes (filtered against self), not from self's own element type, so I typed it as other: Iterable[_T] returning AbstractSet[_T] rather than mirroring __sub__'s signature. The issue's suggested signature (matching __sub__'s shape) would have been wrong for this reason.

The issue also asks about __ror__/__rxor__ but says those weren't looked into yet since they seemed more complicated. Given they're literal aliases at the runtime level, I went ahead and added them too rather than leaving that gap, since the reasoning is the same as __and__/__or__.

Verified against the issue's own reproduction (a Protocol with just __rsub__/__rand__ failing to accept a plain set()) with both mypy and pyright, and confirmed the assignment error reappears if the fix is reverted. mypy_test.py and pyright_test.py both pass on stdlib/typing.pyi across all supported Python versions, and stubtest_stdlib.py shows no new errors for typing (the one __rsub__-related line in its output is for decimal.Decimal, unrelated and already there before this change).

collections.abc.Set defines __rand__, __ror__, __rsub__, and __rxor__
alongside their forward counterparts, but the stub only had __and__,
__or__, __sub__, and __xor__. Without them, a class that only
implements __rsub__ (etc.) won't satisfy a Protocol expecting it, even
though set and frozenset both do this at runtime, since the reflected
methods just fall back to the same underlying implementation.

For __rand__/__ror__/__rxor__ this is exact: CPython's _collections_abc.Set
assigns them as literal aliases of __and__/__or__/__xor__ (same function
object), which makes sense given intersection, union, and symmetric
difference don't care which operand comes first. __rsub__ has its own
separate body, though, since subtraction isn't commutative. Reading it
shows the result's elements come from whatever the *other* operand
contributes (the elements that survive filtering against self), not
from self's own element type, so its signature takes an Iterable[_T]
and returns AbstractSet[_T] rather than mirroring __sub__.

Verified against the reproduction from the issue (a Protocol with just
__rsub__/__rand__ failing to accept a plain set()) with both mypy and
pyright, and confirmed the assignment error reappears if the fix is
reverted. mypy_test.py and pyright_test.py both pass on stdlib/typing.pyi
across all supported Python versions, and stubtest_stdlib.py shows no
new errors for typing.
@github-actions

This comment has been minimized.

Adding __rsub__ to AbstractSet in the previous commit surfaces two
real Liskov substitution violations that already existed but were
invisible before: multiprocessing.managers._BaseSetProxy.__rsub__
returns a plain set instead of an AbstractSet-conformant value, and
boltons' IndexedSet.__rsub__ returns a distinct _RSub protocol type
that isn't a Set at all. Both subclasses were already inconsistent
with what Set.__rsub__ promises, this fix just makes that visible
for the first time.

Silenced both the same way this file already handles other set
operators with the same kind of mismatch (see __ior__, __ixor__,
__ror__, __rxor__ nearby), rather than trying to reshape either
class's actual runtime behavior to fit the protocol.
@github-actions

This comment has been minimized.

Adding __rsub__ (and the other reflected operators) to AbstractSet
meant set and frozenset now inherited it instead of relying purely
on their forward methods, and mypy's subclass-priority rule for
reflected binary operators started routing some calls through the
inherited abstract version instead of the concrete one whenever the
right operand's type was a subtype of the left operand's element
type. That changed inferred types from concrete set/frozenset to
plain AbstractSet in that case, which the regression test suite
catches (stdlib/@tests/test_cases/builtins/check_set.py).

Giving set and frozenset their own __rand__/__ror__/__rsub__/__rxor__
restores the previous concrete return types. The forward and
reflected operators on these two classes are still allowed to overlap
in a way mypy considers unsafe in the abstract, same as their
existing __iand__/__ior__/__isub__/__ixor__ neighbors already do, so
this follows that same established pattern rather than introducing a
new one.
@github-actions

Copy link
Copy Markdown
Contributor

Diff from mypy_primer, showing the effect of this PR on open source code:

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/server/services/task_run_recorder.py:146: error: Incompatible return value type (got "set[Any]", expected "frozenset[str]")  [return-value]

@srittau

srittau commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

I'm not sure why there is a primer hit now. Considering that frozenset.__sub__ is defined and returns frozenset, I'm not sure why mypy infers set[Any]:

https://github.com/PrefectHQ/prefect/blob/3a128c2b55d57bf30f9db28c2148bba5e5b6c145/src/prefect/server/services/task_run_recorder.py#L145-L161

    return (
        frozenset(
            column.name
            for column in db.TaskRun.__table__.columns
            if column.nullable
            and column.default is None
            and column.server_default is None
        )
        # the ON CONFLICT WHERE clause compares against this column, and
        # `x < NULL` is NULL, so a NULL-filled row would silently skip its update
        - {"state_timestamp"}
        # ON CONFLICT matches rows on these columns, and NULL matches nothing, so
        # a filled row would insert a duplicate instead of updating. Excluding them
        # also keeps `flow_run_id` out of the coalesce, so an event with no flow
        # run still clears it.
        - {column.key for column in db.orm.task_run_unique_upsert_columns}
    )

@afonsojanu

Copy link
Copy Markdown
Contributor Author

Thanks for digging into this, and sorry for the noise. I traced it down: the returned expression is a plain set literal ({"state_timestamp"}) subtracted via -, and with expected type: frozenset[str] propagating in from the return statement's own context, mypy resolves that binary op through set.__rsub__ rather than frozenset.__sub__, which is where the plain set[str] comes from instead of frozenset.

I could reproduce it in isolation: it only shows up when the set literal sits directly on the right of - inside a context that supplies an expected return type. Pre-binding the same literal to a local variable first (excluded = {"state_timestamp"}; return ... - excluded) makes it disappear entirely, so this really is about how mypy's reflected-operator priority interacts with inferred vs expected types on inline expressions, not a difference in the actual runtime types involved. frozenset.__sub__ was already correctly typed before and after this PR; nothing about its own signature changed.

The prefect code itself is unaffected at runtime either way (the subtraction still returns a real frozenset), this is purely a mypy-side inference quirk exposed by giving set/frozenset their own reflected operators for the first time. Given that, I'd rather leave the fix as is and treat this one narrow case as an acceptable mypy_primer hit than revert the reflected operators, but happy to hear if you'd rather I scope it down further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing reflected dunder methods from AbstractSet

2 participants