Skip to content

test: gate test_isort_clean.py visibly and put isort on the test extra - #786

Merged
JarryShaw merged 1 commit into
mainfrom
fix-766-isort-visible-gate
Sep 25, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix-766-isort-visible-gate

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

What is the purpose of your pull request?

  • fix — corrects a defect
  • feat — adds a feature
  • perf — changes performance, not behaviour
  • refactor — changes neither behaviour nor performance
  • test — tests only
  • docs — documentation only
  • ci — workflows or build tooling
  • chore — anything else

Description of your pull request and other information

Closes #766. tests/project/test_isort_clean.py skipped on every CI leg, in the
try/except ImportError inside setUpClass shape _gates_of() cannot see — the third
instance of the hazard #745 exists to stop. Fixed with a module-level HAS_ISORT plus a
class-level @unittest.skipUnless, MODULE_PROVIDERS['isort'], and isort in the test
extra.

Install line, not an exclusion — and isort is not in lint.yml's install line. That
line is pip install -U vermin pylint mypy bandit (lint.yml:187), and the workflow's own
header records the omission as deliberate: isort appears only in cron-vendor.yml, as a
formatter that rewrites the generated constants, not as a check. So #779's argument for
declining mypy — a pytest leg would be a second copy of a check lint.yml already runs
(make mypy RUN=, lint.yml:223) — has no isort counterpart. Excluding it would have left
make isort's verdict, and the target-resolving globs wrong three times over (#757, #765,
#767), verified nowhere. It is cheap: pure-Python wheel, all four lines measure ~1.5s, and
the check passes on 0419c1c97 today. unit-tests.yml needed no edit — every pytest job's
install line already builds on .[test].

Gap report. dependency_gate_gaps() is 18 before and 18 after, which is the right
outcome but needs the middle state to be meaningful: gate + MODULE_PROVIDERS and no
test-extra entry gives 21 — three new HAS_ISORT gaps on test, engine-tests and
gate, exactly what #766 predicted a visible gate would do. Adding isort to the extra takes
it back to 18 with the gate now counted: gated_scopes() 250 → 251, so the guard sees the
gate and finds nothing to report, rather than not seeing it. HAS_ISORT therefore gets no
DEPENDENCY_GATE_EXCLUSIONS entry.

Tests. test_tier_guard.py 100 passed / 524 subtests → 102 / 529, identical under
pytest and python -m unittest. The +5 subtests: test_every_gated_flag_is_classified
250→251 (one per gate), test_the_suite_scan_only_looks_at_modules_pytest_collects 115→116
(one per gate-holding module), and 3 in the new coverage test. Both new tests fail against
0419c1c97. tests/project/test_isort_clean.py now runs — 1 passed / 4 subtests, was
OK (skipped=1). No pcapkit/ file is touched, so pcapkit line coverage is unchanged by
construction; what this raises is test execution.

tests/project/test_isort_clean.py gated on isort with a try/except
ImportError inside setUpClass, which tests/_dependency_gates.py cannot see
-- _gates_of() walks a decorator list and never a function body -- and isort
was in no pyproject.toml extra. So it reported OK (skipped=1) on every CI leg
and the dependency-gate guard could not tell (#766, the third instance of the
hazard #745 exists to stop).

* replace the setUpClass skip with a module-level HAS_ISORT and a
  class-level @unittest.skipUnless, so gated_scopes() counts the gate
* add MODULE_PROVIDERS['isort'] so the flag resolves instead of raising
* add isort to pyproject.toml's test extra, closing the gap rather than
  excluding it: lint.yml installs no isort and its own header says so, so
  unlike mypy's gate (#779) there is no existing check to defer to
* pin the outcome in tests/test_tier_guard.py, with a doctored_pyproject
  helper and a falsifiability test that drops the requirement on a copy and
  requires the three gaps back

No unit-tests.yml change needed: every pytest job's install line already
builds on .[test]. dependency_gate_gaps() is 18 before and after,
gated_scopes() 250 -> 251, and test_tier_guard.py goes 100 passed / 524
subtests to 102 / 529 under both pytest and unittest.
@JarryShaw JarryShaw added test Pull requests that add or correct tests (test: subject prefix) ci Pull requests that change CI or workflow configuration (ci: subject prefix) review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Sep 25, 2026
@JarryShaw

Copy link
Copy Markdown
Owner Author

Cross-review at b02f9ef9e (sonnet, a different model from the author): GOOD TO GO, no required changes. Every number reproduced, all four gap states rebuilt in scratch worktrees, and the merge checked.

The central asymmetry holds, and the reviewer sharpened why. Makefile's isort: target has zero --check-only/--diff — I confirmed that myself — and cron-vendor.yml:113,132,144-146 is the same, so isort in CI is a purely mutating formatter, never a checker. lint.yml:5-7,187 installs and runs mypy directly and deliberately omits isort. So #782's exclusion argument — "a pytest leg would pay for a second copy of a check lint.yml already runs" — genuinely does not transfer. One nuance worth keeping: make isort has no pass/fail verdict of its own to verify, so the accurate framing is "nothing checks isort-cleanliness anywhere", and the test's isort --check-only at test_isort_clean.py:341 is the proxy — which predates this PR, since only the docstring and the gate changed.

One correction to the PR's reasoning, which I verified. The body rejects a new lint extra because it "would have to carry mypy to mean anything — which would silently close the HAS_MYPY gap #779 decided to leave open." That misdescribes the mechanism. A gap closes at tests/_dependency_gates.py:1224 on if not (extras_providing(module) & available), where available = frozenset(job.extras) | {CORE} at :1215, and Job.extras comes from parsing each job's actual pip install -e '.[...]' line. So merely having a lint extra would close nothing — it would need adding to a pytest job's install line, which is a deliberate and visible edit, not a silent one. The choice of the test extra is still right (every relevant job already installs it, and the requests/beautifulsoup4 precedent from #507 is exactly this shape); only the stated reason needs a word changed. Not blocking, and not worth a revision on its own — but do not carry that sentence into a future rationale.

All four gap states reproduced exactly. A (0419c1c97) 250 gates / 18 gaps; B gate-only raises AssertionError: 'isort' has no MODULE_PROVIDERS entry…; B′ plus MODULE_PROVIDERS = 251 / 21 with HAS_ISORT × {test, engine-tests, gate} and extras_providing('isort') == frozenset(); C full = 251 / 18 with extras_providing('isort') == {'test'}. So 18→18 is identical because gated_scopes() moved 250→251 — the guard now counts a gate it could not see and finds nothing to report. contested_imports() == MUTUALLY_EXCLUSIVE_IMPORTS and ambiguous_satisfactions() == () at both ends.

The anti-rot claim was tested by injection, not assumed: adding a fake DEPENDENCY_GATE_EXCLUSIONS['HAS_ISORT'] on top of the fix makes test_each_exclusion_still_describes_a_gap_that_is_really_there fail with "the gap is closed, so delete the entry". So omitting an exclusion was right.

Class-level skipUnless is genuinely visible, confirmed by reading rather than only running: _gates_of is called on the ast.ClassDef node itself (:1158, :1169-1183) with func_name = None if isinstance(node, ast.ClassDef) else node.name. The file now reports 1 passed / 4 subtests against the previous OK (skipped=1).

The second new test is not a tautology, which I had asked about. Once its fixture assertion passes it goes on to assert real behaviour — gap.missing == ('isort',), the three specific gaps reappearing, and describe_gap's message contents. The fixture failure is the intended signal for "you ran this against pre-fix code".

Counts reproduced under both runners: 102 passed / 529 subtests, Ran 102 tests ... OK, tests/project/ 127 / 490, with the +5 attributed line by line (250→251, 115→116, and 3 from the new coverage test's reaching loop). Merge into 0419c1c97 clean, 229 passed / 1019 subtests on the result. CI now fully green — 27 success, 3 expected skips, including Engines Python 3.12, which was the reviewer's one open item.

@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Sep 25, 2026
@JarryShaw
JarryShaw merged commit ef859f7 into main Sep 25, 2026
31 checks passed
@JarryShaw
JarryShaw deleted the fix-766-isort-visible-gate branch September 25, 2026 14:24
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Sep 25, 2026
JarryShaw added a commit that referenced this pull request Sep 25, 2026
#812)

`make isort` runs its four recipe lines through `pipenv run`, which resolves
against `Pipfile`'s `[dev-packages]` -- and isort was not listed there, next to
the `mypy` and `pylint` it sits beside in the Makefile. So a `pipenv install
--dev` environment could not run the recipe, and #786's fix put isort on the
`test` extra only, leaving the two declarations disagreeing about a tool the
Makefile needs.

- Pipfile: add `isort = "*"` between `bandit` and `mypy`, bare like its
  neighbours. Additive only -- the `pyproject.toml` extra serves consumers of
  the published package, `Pipfile` this repo's dev environment.
- tests/project/test_isort_clean.py: its docstring recorded the gap as still
  open ("isort remains absent from Pipfile"), which this change makes stale.

Audited every executable the Makefile invokes against `[dev-packages]`: isort
was the only omission. build, twine, pytest, coverage, sphinx,
sphinx-autobuild, vermin, pylint, mypy and bandit are all present.

Closes #810
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Pull requests that change CI or workflow configuration (ci: subject prefix) test Pull requests that add or correct tests (test: subject prefix)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tests/project/test_isort_clean.py skips on every CI leg, invisibly to the dependency-gate guard

1 participant