HAS_PYPCAPFILE stays dark on 3.12-3.14 regardless of install line: pypcapfile is marker-limited

[JarryShaw]

Describe the bug

HAS_PYPCAPFILE's gated tests stay dark on Python 3.12, 3.13 and 3.14 no matter which job installs the extra, because the dependency is marker-limited:

pyproject.toml:201  PyPCAPFile = [ "pypcapfile; python_version < '3.12'" ]
pyproject.toml:225  "pypcapfile; python_version < '3.12'",          # inside `all`

The reason is recorded and legitimate (pyproject.toml:197): "pypcapfile 0.12.0's linklayer module imports imp, removed in Python 3.12". So .[...,PyPCAPFile] resolves to nothing on three of the five matrix legs, and the 15 gated methods report as skips there.

Expected behavior

Either the gated methods execute on every leg, or the fact that they cannot is visible to the dependency-gate guard rather than only to a reader of pyproject.toml.

Additional context

This is residual #738 scope. #738 is "seven more skipUnless gates have never executed on any CI path", and #774 closes the last of the seven by name — but HAS_PYPCAPFILE is only partially closed: engine-tests and pypcap-parity install the extra, so the methods run on 3.10 and 3.11 and are dark on 3.12-3.14. Filing separately so #774's Fixes #738. can close cleanly instead of burying this.

tests/_dependency_gates.py deliberately does not evaluate environment markers — its own docstring says so — so dependency_gate_gaps() reports no gap here and cannot. That is a defensible scope decision rather than a guard defect; the consequence is that this particular hole is invisible to the machinery built to find exactly this class of hole, which is what makes it worth its own issue.

Options, none chosen: pin a pypcapfile fork or a patched build that does not import imp; teach the guard to evaluate markers so the 3.12-3.14 darkness is reported as a gap and then explicitly excluded with this reason; or accept it and record it in DEPENDENCY_GATE_EXCLUSIONS as a marker-limited partial, which at least puts it where the next reader of the guard will find it. Surfaced by the cross-review of #774.

[JarryShaw]

Decision needed: is this worth doing, given most of it is already in place and the rest needs a larger change?

Measured at 83b58ebda. 15 methods, matching the current figure — 9 in tests/toolkit/test_pypcapfile_unit.py (class-level gate) and 6 in tests/foundation/engines/test_new_engine_parity_runtime.py (per-method), by AST walk cross-checked against grep and against the guard's own prose.

The version-bump route is dead, but only just. pypcapfile's latest PyPI release is 0.12.0, uploaded 2016-09-08 (18 releases, none since). Upstream has fixed it — kisom/pypcapfile commit 0aeb29e, 2024-05-29, swaps import imp for importlib, closing its own issue #56 — but no release was ever cut. pcapfile/linklayer.py imports imp at module scope and savefile.py imports linklayer at module scope, so it is unavoidable for pcapkit/foundation/engines/pypcapfile.py:165-167. Consuming the fix today would mean a VCS direct reference (pypcapfile @ git+…@0aeb29e) — materially unlike every other extra, which resolves from PyPI with version and marker specifiers only. It would add a GitHub-availability dependency to CI installs, pin an untagged commit of a project idle since 2024, and probably need handling in the guard's Requirement parser, which is built for PEP 508 name; marker rather than name @ url; marker.

Option 1 is already substantially in place — landed by #755, before this issue was filed. DEPENDENCY_GATE_EXCLUSIONS['HAS_PYPCAPFILE']'s reason already states in prose that engine-tests/pypcap-parity install the extra across the full 3.10-3.14 matrix while the marker only resolves on 3.10/3.11, and that the guard "cannot see that only two of five legs run for real … so that partial coverage is recorded here in prose rather than modelled". The module docstring at tests/_dependency_gates.py:46-51 separately names environment markers as deliberately unmodelled, using this exact extra as its worked example.

What is genuinely missing is the structured half: Exclusion.dark lists only test/integration/gate, not that engine-tests and pypcap-parity are partially dark on 3 of 5 legs each. And it cannot record that without matrix-awareness — pytest_jobs() returns one Job per job name with no per-Python-version dimension, so python_version < '3.12' is meaningless to it until the whole job × version matrix is modelled. That is a much larger widening than "teach the guard markers" suggests, and is precisely what the docstring's refusal was written to avoid.

So three ways, none obviously right:

1. Close as already-documented. The fact is recorded where a reader of the guard will find it; only the structured assertion is absent.
2. Extend dark to record the partial legs, accepting it is prose-adjacent rather than matrix-modelled. Needs coordination with what test_each_exclusion_still_describes_a_gap_that_is_really_there asserts about dark's shape — and ci(deps): install the vendor extra on engine-tests, closing HAS_VENDOR_DEPS (#738) #774 owns those files until it merges.
3. Model the job × Python-version matrix so markers become evaluable and this reports as a real gap. Correct, and the largest of the three.

Blocked on #774 either way for options 2 and 3.

[JarryShaw]

Not entirely sure the context but we definitely want full coverage (all possible Python versions being tested). And we should bump the deps version if new versions are available and fixing some blockers for us.

[JarryShaw]

Ruling recorded, in the maintainer's words: "Not entirely sure the context but we definitely want full coverage (all possible Python versions being tested). And we should bump the deps version if new versions are available and fixing some blockers for us."

That settles the preference and supersedes my three options — the answer is "fix the dependency", not "document the gap". But the two halves of the ruling collide, because there is no version to bump to. Measured:

pypcapfile        0.12.0   2016-09-08   requires_python=(unset)   <- the only release, 18 total
pcapfile          not on PyPI
python-pcapfile   not on PyPI
pypcapfile-ng     not on PyPI
pcapfile-py       not on PyPI

No newer release, and no maintained fork exists on PyPI either. Upstream has fixed it — kisom/pypcapfile commit 0aeb29e, 2024-05-29, swapping import imp for importlib and closing its own issue #56 — but no release was ever cut, and the project has had no activity since. pyproject.toml:199-200's existing comment already says exactly this.

So full coverage has exactly one path: a VCS direct reference, pypcapfile @ git+https://github.com/kisom/pypcapfile@0aeb29e. What that costs, stated plainly because it is a real change in kind:

• Every other extra in pyproject.toml resolves from PyPI with version and marker specifiers only. This would be the first direct reference.
• It adds a GitHub-availability dependency to every CI install that includes the extra.
• It pins an untagged commit of a project idle since 2024, with no release process to fall back on.
• The dependency-gate guard's Requirement parser is built for PEP 508 name; marker; name @ url; marker would likely need handling there too.
• PyPI rejects direct references in uploaded distributions, so this is only viable if the extra is never part of a published sdist/wheel's metadata — worth confirming against how pyproject.toml's extras are packaged before committing to it.

One decision left: take the VCS reference to get full coverage, or keep the marker and accept 3.10/3.11 only? The bump you asked for is not available, so those are the two real options. I have not implemented either. Re-applying needs: decision for that one question; everything else here is settled.

For completeness, the coverage at stake is 15 methods — 9 in tests/toolkit/test_pypcapfile_unit.py (class-level gate) and 6 in tests/foundation/engines/test_new_engine_parity_runtime.py — currently running on 2 of 5 legs.

[JarryShaw]

Oh in this case, let's keep marker and accept the limit of the 3P module's released versions.

[JarryShaw]

When i say "cover as much as possible", it only applies to actual releases, not git commits based distro.

[JarryShaw]

Ruling recorded, in the maintainer's words: "Oh in this case, let's keep marker and accept the limit of the 3P module's released versions." and "When i say 'cover as much as possible', it only applies to actual releases, not git commits based distro."

So: no VCS direct reference. pyproject.toml:201 keeps "pypcapfile; python_version < '3.12'" unchanged, and the 15 gated methods run on 3.10/3.11 only — 9 in tests/toolkit/test_pypcapfile_unit.py, 6 in tests/foundation/engines/test_new_engine_parity_runtime.py. The earlier "we definitely want full coverage" is scoped by this to coverage reachable from released distributions, which for pypcapfile means 0.12.0 (2016-09-08) and nothing more.

Closing as resolved. What this issue asked for was either the methods executing on every leg, or the reason they cannot being visible. The first is now explicitly declined, and the second is already in place — landed by #755, before this issue was filed: DEPENDENCY_GATE_EXCLUSIONS['HAS_PYPCAPFILE']'s reason states that engine-tests/pypcap-parity install the extra across the full 3.10-3.14 matrix while the marker only resolves on two legs, and tests/_dependency_gates.py:46-51 names environment markers as deliberately unmodelled using this exact extra as its worked example. pyproject.toml:197-200 records the cause and that upstream fixed it without releasing.

One thing deliberately not done, and it would be a different issue if wanted: Exclusion.dark still lists only test/integration/gate, so the partial darkness of engine-tests/pypcap-parity is narrated rather than asserted. That cannot be modelled as things stand — pytest_jobs() returns one Job per job name with no per-Python-version dimension, so python_version < '3.12' is meaningless to it until the whole job × version matrix is modelled, which is precisely what that docstring's refusal avoids. If the structured assertion is wanted later it is a guard-modelling issue, not this one.

Filing note for the record: I opened this issue on the premise that the fact was undocumented. It was already documented. The measurement that was genuinely new is that no maintained fork exists on PyPI — pcapfile, python-pcapfile, pypcapfile-ng and pcapfile-py all 404 — so there was never a bump available to take.