ci: engines not exercised across their claimed Python ranges — PyShark has no real-capture coverage, PyPCAPFile dark on 3 of 5 legs

[JarryShaw]

The maintainer asked whether every engine is exercised in CI across its supported Python
range. Audited: no. Five of eight are; three findings are not. Measured at 6216de505.
unit-tests.yml is the only workflow that runs pytest (grep -ln pytest .github/workflows/*.yml
→ one hit).

1. PyShark is never driven through a real tshark-parsed capture, on any Python version

The only engine='pyshark' extraction is tests/integration/test_engine_runtime.py:90,102.

• engine-tests installs pyshark and tshark — but its selection is
  --ignore=tests/integration --ignore-glob='*_runtime.py', so it cannot reach that file
  by either route.
• pypcap-parity has the fixture-tier selection that does reach it and installs pyshark,
  but installs no tshark (grep -ci tshark over its whole job log → 0). So
  PyShark.unsupported_reason() returns the missing-binary reason, the extractor falls back to
  the built-in parser, and line 104's assertGreater(extractor.length, 0) passes on the
  fallback's output.
• integration and gate install no pyshark at all.

A vacuous pass, not a skip — invisible in every skip count. The HAS_PYSHARK gates that
engine-tests does reach (tests/foundation/engines/test_pyshark_engine.py:137,223,245) are all
unsupported_reason/binary-detection. tests/toolkit/test_pyshark_unit.py is FakeLayer
doubles throughout.

Cheapest fix, two lines: add tshark to pypcap-parity's existing apt step — it already
has pyshark and the right selection — and make test_engine_runtime.py:102 assert
extractor._exnam == 'pyshark', as test_new_engine_parity_runtime.py:112-113 already does for
PyPCAP.

This is exactly why the pcapkit/toolkit/pyshark.py link-type defect in #843 survived: the
double sets layer_name='ethernet', which is a LinkType member, so the real-world eth
path was never exercised. Confirmed live here — pyshark reports eth, and PDML <proto name>
carries Wireshark filter names.

2. PyPCAPFile is on engine-tests's install line but resolves to nothing on 3 of 5 legs

Measured, run 36294992326:

Engines leg	pypcapfile installed	pytest
3.10	yes	1972 passed, 9 skipped
3.11	yes	1973 passed, 8 skipped
3.12	no	1965 passed, 16 skipped
3.13	no	1965 passed, 16 skipped
3.14	no	1965 passed, 16 skipped

Eight tests stop exactly at the python_version < '3.12' marker boundary.
tests/_dependency_gates.py:46-52 says it deliberately does not model environment markers,
so the repo's own guard cannot see this, and three legs read green on coverage they do not have.

Also: pypcapfile 0.12.0 installs cleanly on 3.12+ and only fails at use — a real user state
— so PYTHON_CEILING = (3, 12) firing is never reproduced. PyShark's (3, 14) ceiling is
covered, because pyshark carries no marker.

Cheapest fix: teach _dependency_gates.py to evaluate markers against each job's matrix
(it already retains Requirement.marker at :548-550) so marker-shadowed extras report as gaps.

3. Four of the six third-party engines sit only on non-required checks

Required checks (ruleset 23497679) are exactly Python 3.10-3.14, Integration Python 3.10-3.14, Compat Python 3.10-3.14 — and Compat runs no pytest at all. Neither
Engines Python X nor PyPCAP/PyPCAPFile parity Python X is required, so PyShark, PyPCAP,
PCAP_CT and PyPCAPFile are exercised on zero blocking legs; only DPKT and Scapy are.
Gate (full suite, Python 3.14) reports skipped on PRs by design.

That follows from #751's "per-engine coverage gets its own job" plus #738's "land it on a
non-blocking leg" — each defensible alone. Decision needed: promote Engines Python 3.10-3.14 into the required set, or record that engine coverage is advisory.

4. Versions claimed but in no matrix

requires-python = ">=3.6, <4" claims 3.6-3.9; no workflow runs any of them. 3.15 appears only
in python-compatibility.yml's schedule-only import check. Either the floor moves or the claim
is untested. PCAP_CT is the one engine that honestly excludes <3.10, via a marker.

Noticed in passing, not urgent

DPKT carries no marker but dpkt 1.9.8 classifies nothing above Python 3.9. PyShark carries
no marker but pyshark 0.6 classifies nothing above 3.10, while pcapkit runs it on 3.11-3.13 —
and pcapkit/foundation/engines/pyshark.py:89-93 admits 3.13 is inferred, not measured.

[JarryShaw]

Actually I think the engine tests should be like this: the test step is guarded with two dimension matrix:

one is the Python version (GHA current supported range 3.10-3.14; plus experimental prerelease but non-blocking, 3.15, meaning if it fails on 3.15, CI won't fail)

the other is 3p engine: DPKT, Scapy, PyShark, PyPCAPFile, PyPCAP, pcap-ct, etc.

for the engines, they are expected to throw on unsupported Python versions (if installable) - this is the test for them; and they are expected to complete their target test suites if on supported Python versions (unit and integ, if any).

[JarryShaw]

Recorded, verbatim:

Actually I think the engine tests should be like this: the test step is guarded with two
dimension matrix:

one is the Python version (GHA current supported range 3.10-3.14; plus experimental prerelease
but non-blocking, 3.15, meaning if it fails on 3.15, CI won't fail)

the other is 3p engine: DPKT, Scapy, PyShark, PyPCAPFile, PyPCAP, pcap-ct, etc.

for the engines, they are expected to throw on unsupported Python versions (if installable) -
this is the test for them; and they are expected to complete their target test suites if on
supported Python versions (unit and integ, if any).

That changes the shape of the fix, and it resolves the hardest part of finding 2 outright.
I had proposed teaching tests/_dependency_gates.py to model environment markers so a
marker-shadowed extra reports as a gap. Your design makes that unnecessary: if the engine is
installed anyway on an unsupported version and asserted to throw, the refusal path becomes a
tested behaviour rather than a modelling problem. The marker was what hid it.

Concretely, what your two-dimension matrix asks for:

• Python axis: 3.10-3.14 blocking; 3.15 present but non-blocking — continue-on-error: true
  on the 3.15 leg, so a failure reports without failing the run.
• Engine axis: one leg per engine — DPKT, Scapy, PyShark, PyPCAPFile, PyPCAP,
  PCAP_CT — rather than today's single job installing all of them at once.
• Two expected outcomes per cell, which is the part today's CI has never exercised:
  installable but unsupported → assert unsupported_reason() fires and names the interpreter;
  supported → the engine's unit and integration tests complete, with the assertion proving
  which engine actually ran.

The install-anyway requirement means CI must bypass the python_version markers in
pyproject.toml for the unsupported cells — pip install pypcapfile directly rather than
.[PyPCAPFile] — since the marker is precisely what stops it installing on 3.12+. Worth stating
explicitly in the workflow comment, or the next person will "fix" it back.

One measured caveat for the if installable clause: pypcap is a C extension and builds from
source, so on a cell where it does not compile there is nothing to assert a throw on. Its wheel
build is confirmed working on 3.10 (Created wheel for pypcap: pypcap-1.3.0-cp310-cp310-linux_x86_64.whl,
job 108552079089) but pcapkit's own comment records real 3.12/3.14 compile failures. So that
engine's unsupported cells are "not installable" rather than "installable and throws", and the
matrix needs to tolerate the difference rather than assert a throw that cannot happen.

I have redirected the in-flight worker: keep the two narrow fixes that remain correct under this
design (tshark on the job that can actually reach the pyshark extraction, and asserting
_exnam == 'pyshark' so a silent fallback stops passing), and stop the marker-modelling work.
The matrix rebuild is the larger piece and I will scope it separately rather than bolt it onto a
brief written before your ruling.

[JarryShaw]

Finding 1 landed as #846 — narrow on purpose: tshark on the one job whose selection can
reach the pyshark extraction, plus an assertion that the engine actually ran rather than the
fallback wearing its label. 106 tests pass.

Relabelled wip → blocked, because what remains here is waiting on you rather than on work:

# finding 3 — ruleset change, agents do not touch branch protection
gh api repos/JarryShaw/PyPCAPKit/rulesets/23497679 --jq '.rules[]|select(.type=="required_status_checks")'

• Finding 2 is superseded by your two-dimension matrix ruling and should be built as part of
  it, not as the marker-modelling patch I originally scoped. A worker had actually completed that
  patch — 262 lines in tests/_dependency_gates.py, correctly reporting the PyPCAPFile gap — before
  your ruling reached it. I had it reverted rather than landed, since the matrix rebuild changes the
  job structure it keys on and would likely rewrite it. Worth knowing it exists and worked, if the
  matrix design ends up wanting it after all.
• Finding 3 needs your call: promote Engines Python 3.10-3.14 into the required set, or record
  engine coverage as advisory. The job display names a ruleset would key on are unchanged by ci: give the pyshark engine real-capture coverage, and stop a fallback passing as a pass #846.
• The matrix rebuild itself is unstarted. One measured caveat for its if installable clause:
  pypcap is a C extension built from source, so on a cell where it does not compile there is
  nothing to assert a throw on — "installable and throws" and "not installable" are different
  outcomes the matrix has to tolerate rather than conflate.

[JarryShaw]

What exactly it's blocked by (which issue/pr)? And any decision needed from me?

[JarryShaw]

A three-way asymmetry the matrix design has to accommodate — measured, and it means
"install anyway and assert it throws" fits only two of the three.

grep -c PYTHON_CEILING over pcapkit/foundation/engines/ on origin/main:

engine	PYTHON_CEILING	fits "install anyway, assert it throws"?
PyShark	present	yes — already exactly this shape; unsupported_reason() declines on >= (3, 14) and CI already installs it there, so its refusal path is the one engine's that is tested today
PyPCAPFile	present	yes, in principle — the engine code is built for it, but its python_version < '3.12' pip marker stops it ever being installed on 3.12+, so the refusal never runs. Your design fixes this by installing past the marker.
PyPCAP	absent	no — structurally cannot
PCAP_CT	absent	resolves by backend identity, not version

Why PyPCAP cannot follow the model. Its failure mode is a build failure, not a runtime
one: no wheel, and the pre-generated pcap.c does not compile against the 3.12+ C API. So on the
cells where it is unsupported there is no installed engine to throw — pip install fails and the
cell has nothing to assert. Its unsupported_reason() deliberately distinguishes which competing
distribution is present (pypcap vs pcap-ct) rather than checking a version, and its own
docstring says a version check "would say nothing useful on 3.10 and 3.11, where both
distributions install happily."

So the matrix needs three cell outcomes, not two: supported → suite completes,
installable-but-unsupported → assert the decline, and not installable → assert the install
fails, or skip the cell deliberately. Collapsing the last two would make PyPCAP's 3.12+ cells
either red for the wrong reason or silently green.

Worth knowing the workflow already anticipates the third case in prose:
.github/workflows/unit-tests.yml:403-410 says that if pypcap-parity's install step goes red on
a clean run, the fix is deleting the job and returning HAS_PYPCAP to
DEPENDENCY_GATE_EXCLUSIONS — i.e. "not installable" is already a recognised outcome, just not a
matrix dimension.

[JarryShaw]

Recorded, verbatim: "What exactly it's blocked by (which issue/pr)? And any decision needed
from me?"

Fair challenge — I applied blocked without naming a blocker, which is exactly what I hold
every other item to. Correcting the labels and stating it plainly.

One decision needed from you, finding 3 — nothing else:

Promote Engines Python 3.10-3.14 (and PyPCAP/PyPCAPFile parity Python X) into ruleset
23497679's required checks, or record that engine coverage is deliberately advisory.

gh api repos/JarryShaw/PyPCAPKit/rulesets/23497679 \
  --jq '.rules[]|select(.type=="required_status_checks").parameters.required_status_checks[].context'
# -> Python 3.10..3.14, Integration Python 3.10..3.14, Compat Python 3.10..3.14

Agents do not touch branch protection, so this one cannot move without you. It is also the
finding with the widest effect: four of six third-party engines are currently exercised on zero
blocking legs.

The real blocker, for the remaining work only: #846 merged.

gh pr view 846 -R JarryShaw/PyPCAPKit --json merged -q .merged   # true == unblocked

Your two-dimension matrix rebuild edits .github/workflows/unit-tests.yml, and #846 already
modifies that file — so starting now means rebasing onto a workflow that changed underneath.
Nothing else in the repo blocks it.

State of the three findings, so this is legible without re-reading the thread:

	status
1 — PyShark never driven through a real capture	in PR #846, review: pending
2 — PyPCAPFile marker-shadowed on 3 of 5 legs	superseded by your matrix ruling; it becomes "install past the marker and assert the decline" rather than a gate-checker change
3 — four engines on non-required checks	awaiting your decision above
the matrix rebuild itself	unstarted, blocked on #846 merging

Labels corrected to needs: decision + blocked accordingly, rather than a bare blocked.

[JarryShaw]

3P engine tests should be a requirement same level as other builtin library tests. They're not advisory only.

[JarryShaw]

Recorded, verbatim: "3P engine tests should be a requirement same level as other builtin
library tests. They're not advisory only."

Done — applied to ruleset 23497679. Required checks went from 15 to 22:

+ Engines Python 3.10 .. 3.14
+ PyPCAP/PyPCAPFile parity Python 3.10, 3.11

enforcement=active and all four rule types (deletion, non_fast_forward, pull_request,
required_status_checks) verified intact afterwards. Backup of the pre-change ruleset is at
/tmp/ruleset-23497679.backup.json if it needs reverting.

I checked the one thing that could have gone wrong before applying it. A required check that
never reports blocks a PR permanently, so the question was whether these jobs run on every PR or
only some. Both are gated on if: ${{ inputs.gate-only != true }} —
.github/workflows/unit-tests.yml:332 and :434 — which is the identical condition the
already-required test and integration jobs carry (:51, :131). Confirmed empirically rather
than from the condition alone: #657, a docs-only PR, has all seven legs SUCCESS. So no new
blocking risk.

State of the three open PRs against the new gate:

PR	Engines 3.10-3.14	parity 3.10/3.11
#838	all SUCCESS	both SUCCESS
#657	all SUCCESS	both SUCCESS
#846	in progress	in progress

needs: decision removed. Finding 3 is closed.

What this does not cover, and is worth being explicit about: the legs are now required, but
Engines Python 3.12/3.13/3.14 still pass without PyPCAPFile installed at all — that is the
marker-shadowing your matrix ruling addresses. So the gate now blocks on a job whose coverage is
known-incomplete on three of its five legs. Making it required is still the right move; it just
means the matrix rebuild is now gating quality rather than merely reporting it.

Remaining here: the matrix rebuild, blocked on #846 merged (it edits the same workflow file).