Introduce pytest integration out of the box

[aignas]

Currently users need to know about pytest-bazel or other ways to integrate rules_python with bazel.

Ideally the discovery and onboarding of new users should be as easy as we can make it to be without making rules_python unclude everything and the kitchen sink.

Some ideas how we could do this:

• Implement the default pypi hub idea Support a default hub repo concept in bzlmod #2094
• pre-declare the pytest and pytest-bazel dependencies, where we would generate entries in the hub repo with aliases. Maybe we could define a particular default version for everything to work out of the box.
• create a pytest_test rule where we are using the declared deps.
• the users can override the pytest and other included deps.
• use the main_module attribute to launch pytest-bazel. If pytest is not there users would get an error that it could not be imported.

Open questions:

• can we default to pytest-bazel in all py_test invocations if there is no main or main_module defined?
• Is "pypi" a good name to reserve as the default hub repo? How does the predeclaring work? We just accept the same args as whl_library but all optional?
• Who can pre-declare deps and if multiple versions are declared, which one do we chose? We have version marker parsing in rules_python and we can even do resolution based on version ranges, e.g. users declare that they need pytest 7.1 or above and we get the lowest available version that satisfies the conditions. We can also do this by python version since we have marker expression evaluation support.

Design goals

• no overhead in the number of targets, hence usage of main_module.
• little maintenance in rules_python.
• better out of the box experience. How would the users discover the feature?

[aignas]

@rickeylev anything to add? @keith, any requests as a user that we should think about?

[rickeylev]

default py_test to use pytest if there's no main/main_module

Oh, that's clever. Maybe? Eh. Well. I don't see how to do so without it being considered a breaking change. Today, you have to put if name == main: call_your_main(), so if we invoke it in a way where it isn't name==main, then its breaking.

I was more thinking, as step one, we introduce a pytest_test rule, then figure out how to transition.

is pypi a good name to reserve as the default hub repo

That's what I was thinking, yes. I'm pretty sure we can do so in a non-intrusive enough way.

Who can pre-declare deps

Anyone. The basic design I had in mind was: you call pip.dep(name), which makes @pypi//name available. If no name is seen, then it maps to a target that gives an error at execution phase saying "no implementation registered"

if multiple versions are declared, which one do we chose?

Yeah, it gets murky here. I was thinking something simple, e.g. order of pip.parse calls, after accounting for the more statically set platform settings (e.g. python_verison).

[aignas]

default py_test to use pytest if there's no main/main_module

Oh, that's clever. Maybe? Eh. Well. I don't see how to do so without it being considered a breaking change. Today, you have to put if name == main: call_your_main(), so if we invoke it in a way where it isn't name==main, then its breaking.

I was more thinking, as step one, we introduce a pytest_test rule, then figure out how to transition.

Another option would be to also control the default main_module value through a string flag. Have it set to nothing, but users could set it to whatever. Either unittest.main, pytest-bazel equivalent or similar.

Having default test deps is also something that would have to be done.

We could flip the default with some major version bump (doesn't have to be 2.0).

That said, stepping back a little, we want thebest OOB experience and the least surprising system behaviour. Adding flags may not make it easier to configure/discover features when setting up a prject from scratch.

[keith]

FWIW I don't think it's as important that we pre-supply the requirement("pytest") part. I think the primary value is from having a runner that does what users expect for pytest. Since python users are so used to providing requirements files, I think that part of the overhead, to reduce the risk of being too magical, is fine

[aignas]

It seems that Aspect is not doing anything super magical here: https://github.com/aspect-build/rules_py/blob/b87227504d6ec8f8fdde3dff3f467380e8198971/py/defs.bzl#L126

Just an idea before I logout:

• If the user has pytest in their requirements.txt, then we can also add pytest-bazel package which can be configured via pip.defaults.
• Make the main_module for py_test a string_flag that can be set to anything.
• The user can easily set it in the .bazelrc to be --@rules_python//python/config_settings:py_test_main_module=pytest_bazel
• Default pytest behaviour out of the box with very minimal effort and not too much magic?

[rickeylev]

Make the main_module for py_test a string_flag that can be set to anything.

How would this work? It's currently a plain string attribute.

[aignas]

OK, so the latest thinking is that #3837 and #3850 will help with this. We declare via pip.dep a dependency on pytest-bazel, then we create a pytest_test rule based on aignas/pytest-bazel#27

load("//python:py_test.bzl", "py_test")

def pytest_test(*, deps = [], data = [], **kwargs):
    """Use pytest-bazel for running pytest tests

    Args:
        deps: Deps to pass to the test.
        data: The data to pass to the test rule.
        **kwargs: Extra args passed to the py_test.
    """

    py_test(
        deps = (deps or []) + [
            "@pypi//pytest_bazel", # this is what is declared via the unified hub
        ],
        main_module = "pytest_bazel",
        data = (data or []) + [
            "//:pyproject.toml",
        ],
        **kwargs
    )

Should be a reasonably easy lift, I'll do that later this week.