Incompatibility with pytest-forked

[duburcqa]

import time
import json

import pytest
import numpy as np


def test_TMP():
    np.random.seed(int(time.time()))
    if np.random.rand() > 0.2:
        qByUser = "1e"
        json.loads(qByUser).decode("utf-8")

pytest --reruns=10 --reruns-delay=1 --only-rerun=JSONDecodeError --forked test_tmp.py

============================================================================================================================= test session starts ==============================================================================================================================
platform linux -- Python 3.10.12, pytest-8.4.2, pluggy-1.6.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /mnt/home/duburcqa/workspace/src
plugins: dash-3.2.0, forked-1.6.0, print-1.2.0, random-order-1.2.0, rerunfailures-15.1, xdist-3.8.0, syrupy-5.0.0
collected 1 item

test_tmp.py F                                                                                                                                                                                                                                                            [100%]

=================================================================================================================================== FAILURES ===================================================================================================================================
___________________________________________________________________________________________________________________________________ test_TMP ___________________________________________________________________________________________________________________________________
def test_TMP():
        np.random.seed(int(time.time()))
        if np.random.rand() > 0.2:
             qByUser = "1e"
>           qUserData = json.loads(qByUser).decode("utf-8")

test_tmp.py:12:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.10/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <json.decoder.JSONDecoder object at 0x7313fb84efe0>, s = '1e', _w = <built-in method match of re.Pattern object at 0x7313fb9075e0>

    def decode(self, s, _w=WHITESPACE.match):
        """Return the Python representation of ``s`` (a ``str`` instance
        containing a JSON document).

        """
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        end = _w(s, end).end()
        if end != len(s):
>           raise JSONDecodeError("Extra data", s, end)
E           json.decoder.JSONDecodeError: Extra data: line 1 column 2 (char 1)

/usr/lib/python3.10/json/decoder.py:340: JSONDecodeError
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED test_tmp.py::test_TMP
============================================================================================================================== 1 failed in 0.13s ===============================================================================================================================

[The-Compiler]

This has nothing to do with either pytest-forked or pytest-rerunfailures. You're loading invalid JSON data (1e) and getting an exception that it is, indeed, invalid json data.

[duburcqa]

It does... Try running this example without --forked and it will pass. Please read my snippet more carefully. There is a probability to just do nothing (which obviously passes). Do you think I'm taking time to report issues without testing on my side before?

[The-Compiler]

My bad, I was indeed wrong there. That being said, it really doesn't help to have an overly complicated example, and a traceback that does not match the code in the example.

Here's a simpler example:

import random

def test_random():
    if random.random() > 0.2:
        1/0

It indeed fails without reruns with pytest --reruns=10 --forked test_tmp.py, but works without --forked.

[janmrow]

I spent some time looking on it from current master.

The issue is not fork() itself. The conflict happens at pytest_runtest_protocol: pytest-forked takes that hook first and runs one attempt, so the rerun loop in pytest-rerunfailures is never reached. A minimal fake plugin reproduces the same behavior without forking.

Simply reversing hook order is not a fix. Reruns come back, but pytest-forked is then bypassed and the attempts run in the parent process.

I also tried a simple prototype where pytest-rerunfailures delegates each attempt to pytest-forked. That can make the reruns fork correctly, but it loses exception state needed by features such as --only-rerun / --rerun-except, so it creates regression.

A proper fix would probably need a cleaner per-attempt API between the plugins, returning enough information for pytest-rerunfailures to make the rerun decision (before reports are published). That seems much larger than a small compatibility fix.