tests/_support.py leaves bare stub modules in sys.modules, so tests/corekit + tests/project in one process fails three test_public_api tests

[JarryShaw]

Running tests/corekit/test_multidict.py and tests/project/test_public_api.py in one pytest process fails three tests that pass in either file alone. The cause is in tests/_support.py: bootstrap_core_modules() installs bare stub modules into sys.modules and never restores them, so a later test that introspects the real package sees the stubs instead.

Reproduced on untouched main

Measured on origin/main at 0c7f2b7c9 in a throwaway worktree with no changes of any kind, CPython 3.14.7, PYTHONSAFEPATH=1 and the worktree at PYTHONPATH:

$ python -m pytest tests/corekit/test_multidict.py tests/project/test_public_api.py -p no:cacheprovider -q
FAILED tests/project/test_public_api.py::PublicAPISurfaceTests::test_every_all_entry_resolves
FAILED tests/project/test_public_api.py::PublicAPISurfaceTests::test_every_public_package_declares_all
FAILED tests/project/test_public_api.py::PublicAPISurfaceTests::test_the_non_export_allowlist_is_tight
3 failed, 16 passed, 1 warning, 429 subtests passed in 1.47s
exit 1

Exit code read from a file, not from a pipeline.

The mechanism

tests/_support.py:187 bootstrap_core_modules() calls tests/_support.py:160 ensure_package(name, path), which leaves bare stub entries for pcapkit, pcapkit.corekit and pcapkit.utilities in sys.modules. Nothing removes them afterwards. test_public_api.py then imports the package to walk its __all__, gets the stubs, and finds no exports to check.

Why nobody has seen it

It is masked in a full-suite run, because some intervening test directory re-imports the real package between the two, repairing sys.modules by accident. It only surfaces when those two files are the whole selection — which is exactly what a developer does when narrowing to the area they are working on.

So the failure is real, is on main today, and is invisible to CI. It is also order-dependent, meaning it can appear or vanish from an unrelated change to the test selection, which is the worst property a test-infrastructure defect can have.

Not the defect reported in #660

#660 is the ProtocolBase stand-in being non-Generic (TypeError: type 'ProtocolBase' is not subscriptable), fixed by #662. That is a different mechanism and this is not fixed by #662 — the stub-restoration gap is separate from the Generic gap. Confirmed by reproducing with the test_multidict.py tests --deselected, and at untouched main.

Suggested fix

Restore sys.modules rather than only popping from it — a context manager or addCleanup that snapshots the affected keys and puts back exactly what was there, including absence. purge_modules currently pops without restoring, which is the same gap seen from the other side.

Found while working #661 / #667; not fixed there, since tests/_support.py is shared test infrastructure and the change wants its own review.

[JarryShaw]

Correction: the repro in this issue no longer reproduces, and the defect is still live

I filed this against main at 0c7f2b7c9. #662 has since merged and masks the symptom — the repro as written now passes. The mechanism is untouched.

tests/conftest.py:95 added an autouse fixture, restore_module_table, which snapshots and restores the pcapkit region around every test. It repairs the table after the polluting test whether or not the helper was fixed.

Re-measured on main at b34f132f6, three ways

$ pytest tests/corekit/test_multidict.py tests/project/test_public_api.py -q
19 passed, 432 subtests passed                                        exit 0   <- masked

$ pytest ... --noconftest -q
3 failed, 16 passed, 429 subtests passed                              exit 1   <- live

$ python -m unittest tests.corekit.test_multidict tests.project.test_public_api
Ran 19 tests ... FAILED (failures=3)                                  exit 1   <- live

The unittest form needs no flag. The stdlib runner reads no conftest.py, and running this suite that way is supported — so the defect is reachable without opting out of anything.

Why the masking matters rather than resolving it

A fixture that repairs the table after the fact leaves the binders owing restoration. The pollution still happens; it is merely cleaned up by something that happens to run afterwards. Any route that bypasses tests/conftest.py — --noconftest, unittest, a different runner, or a future conftest reorganisation — exposes it again, and CI cannot see it either way.

The blast radius was wider than this issue said

I named bootstrap_core_modules → ensure_package. Five modules leak, and two leak through load_module alone, never touching bootstrap_core_modules — so fixing only the helper I named would have been half a fix: test_multidict, test_io, test_exceptions_warnings, test_module, and test_compat. That inventory is #686's, verified by it rather than by me.

One stale premise of mine

This issue's analysis said tests/corekit/test_protochain.py:9-11 calls bootstrap_core_modules in setUp with no tearDown. True when filed; no longer true — it now calls isolate_modules(self) and was already safe before #686.

Fixed in #686, which is cross-reviewed GOOD TO GO.