ci: add a per-engine test job for Scapy, PyShark, PyPCAPFile and pcap-ct

[JarryShaw]

Please follow the guide below

• Searched for similar pull requests
• Followed the coding style (make pylint, make mypy, make isort) -- ran isort, mypy and pylint on the three touched Python files; clean except three pre-existing mypy findings and six pre-existing pylint line-length findings, none on a line this PR touches
• make test passes, and a test case covers the change -- the full suite is forbidden to run here; see the tests run below instead
• Added a changelog entry under docs/source/changelog/ and regenerated CHANGELOG.md, if the change is user-visible -- N/A, changelog centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657

What is the purpose of your pull request?

• ci — workflows or build tooling

---

Description of your pull request and other information

Fixes #751.
Fixes #762.

Per the maintainer's ruling on #751, third-party engine coverage (Scapy, PyShark,
PyPCAPFile, pcap-ct, PyPCAP) gets its own job(s) in unit-tests.yml rather than one
more install line on test, integration or gate — "so they dont intertwine
with the other major tests".

engine-tests (3.10-3.14): installs Scapy, PyShark, PyPCAPFile and PCAP_CT
together, plus a system libpcap via apt-get for PCAP_CT's ctypes loader. Closes
HAS_SCAPY (14 methods), HAS_PYSHARK (3 of 4), HAS_PYPCAPFILE (9 of 15) and
HAS_PCAP_CT (1), and closes HAS_RUNTIME (5 methods in
test_runtime_engines.py, which reuses that flag name for the four core
dependencies plus dpkt/scapy/pyshark) as a side effect of installing all
three together.

pypcap-parity (3.10-3.11 only, both extras' own version ceiling): a separate
job and venv with a C toolchain and libpcap headers, mirroring integration's
fixture-tier selection to reach test_new_engine_parity_runtime.py. This is the
"try to build [PyPCAP] and if the CI is not a good suit, then we ripe it"
instruction acted on directly — the job attempts the real pip install pypcap
build in CI rather than declining it up front. It also closes the other 6
HAS_PYPCAPFILE methods that live in the same module. Kept apart from
engine-tests because pypcap and pcap-ct both ship a top-level pcap module and
cannot share a venv — measured in
pcapkit/foundation/engines/_pcap_backend.py's own module docstring: with both
importable, pcap-ct's package always shadows upstream's extension.

tshark is now installed too — update, after review. #751 has a later
comment (19:16:10Z) extending the "try to build, rip it if CI is not a good
fit" instruction to tshark as well as PyPCAP; an earlier reading of this
thread had missed it. test_the_reason_tracks_the_running_interpreter used to
hard-assert tshark's absence, which would have flipped the assertion from a
pass to a failure the moment tshark was installed — that part of the original
reasoning was correct, but the fix was to remove the inconsistency, not to
decline the ruling. Its sibling
(test_this_host_really_has_no_tshark_so_the_check_is_not_vacuous) already
self-guards with shutil.which('tshark'); giving the first test the same
guard (one line, verified both branches locally with a fake tshark on
$PATH) makes tshark's presence a non-issue, so engine-tests now installs
it via apt-get (debconf pre-seeded to skip the postinst prompt). Confirmed
on real CI: all 5 engine-tests legs still pass with tshark installed.

HAS_PYPCAPFILE's marker stays unmodelled, by design. PyPCAPFile carries
python_version < '3.12', so it installs on 3.10/3.11 and skips cleanly on
3.12-3.14 even once taken. tests/_dependency_gates.py's guard cannot see that —
it does not evaluate markers, by its own module docstring — so this is recorded
in DEPENDENCY_GATE_EXCLUSIONS' prose rather than modelled in the guard itself;
teaching the guard markers is out of scope for a workflow-only change.

Cost, chosen rather than discovered: per-job venvs mean per-job installs, and
pypcap-parity additionally re-runs the whole fixture-tier suite (sample
regeneration plus integration's selection) a second time on two of the five
legs, since none of tests/_dependency_gates.py's three recognised selection
shapes offers a cheaper way to reach one fixture-dependent module. Restricting
that job's matrix to 3.10/3.11 — where its extras' own markers actually resolve —
is what keeps the duplication bounded to two legs instead of five.

What I measured

• AST-derived, tree-wide method counts per flag (confirming the issue's own
  figures exactly at PR creation): HAS_SCAPY 14, HAS_PYSHARK 3, HAS_PYPCAPFILE 15
  (9 unit + 6 parity), HAS_PYPCAP 4, HAS_PCAP_CT 1, HAS_RUNTIME 5 (scoped to
  test_runtime_engines.py). HAS_PYSHARK is now 4, not 3 -- this PR's own
  later regression test (test_a_config_ini_naming_an_off_path_tshark_is_not_read_as_missing)
  added a third in-file gated method; the fourth is
  tests/integration/test_engine_runtime.py:69, covered by integration/gate,
  not test/engine-tests.
• Skip delta, measured on a throwaway Python 3.10 venv (never the repo's real
  one), comparing a baseline matching test's current install line
  (.[test,DPKT,crypto,NGAP]) against engine-tests' full install line: 107
  passed / 27 skipped -> 134 passed / 0 skipped, 0 failed across every unit-tier
  file the new job's gates live in.
• The pypcap-parity half, same method: with PyPCAPFile installed and PyPCAP
  absent, test_new_engine_parity_runtime.py goes from 0 passed / 10 skipped
  to 6 passed / 4 skipped locally (a local attempt to also build PyPCAP on
  this non-Ubuntu dev box failed only on a nonstandard header search path, not on
  the compile itself). Confirmed on this PR's own CI run: both
  PyPCAP/PyPCAPFile parity Python 3.10/3.11 legs installed pypcap-1.3.0 for
  real (Successfully installed … pypcap-1.3.0 pypcapfile-0.12.0 …) and all 4
  HAS_PYPCAP methods executed rather than skipped — one of them logging a real
  AttributeWarning from the actual pypcap engine, not a mock. 172 passed / 2
  skipped / 0 failed on the 3.10 leg. So the "try to build, rip it if CI is not a
  good fit" instruction resolved in PyPCAP's favour: it builds cleanly on
  ubuntu-latest, nothing to rip.
• tests/test_tier_guard.py: 96 passed, 511 subtests (was 84 passed / 485
  subtests on main; the additional passes and subtests are from later
  cross-review rounds, not from this bullet's own skip-delta measurement).
  tests/foundation/engines/test_pyshark_engine.py, the third touched Python
  file: 9 passed, 8 subtests. Combined: 105 passed, 519 subtests.
  tests/_dependency_gates.DEPENDENCY_GATE_EXCLUSIONS'
  liveness check (test_each_exclusion_still_describes_a_gap_that_is_really_there)
  is what verified every dark job/package pair above; it was the primary check
  for this whole change.
• Ran isort --check-only, mypy and pylint against the three touched Python
  files: isort clean; mypy's three findings and pylint's line-length
  findings are all outside the lines this PR touches (verified against
  origin/main's diff hunks).
• No new tests are added — this makes existing gated tests execute rather than
  adding coverage, so the "raise coverage" rule is satisfied by the skip delta
  above rather than by new test cases.

Also fixed, after two cross-review rounds (both on a different model, Opus):
three comments (test, integration, gate) still describing the pypcapfile
bug #747/#748 already fixed as the reason PyPCAPFile isn't installed there;
a false "reaches all 14 HAS_SCAPY methods" claim in
tests/_dependency_gates.py (engine-tests reaches 10 of 14 — the other 4
are in tests/integration/ or match the *_runtime.py ignore-glob, already
covered elsewhere); and a stale module-docstring claim that neither PyPCAP
nor PCAP_CT was on any job's install line, both now are. That last one
surfaced a real, if currently harmless, gap in the dependency guard: it
cannot distinguish an extra satisfying HAS_PYPCAP from one satisfying
HAS_PCAP_CT (both resolve to needing top-level pcap), so a job that
installed one while reaching the other flag's gate would get a false
non-gap. Verified with job_reaches() directly that neither new job does
that today — confirmed, not assumed — and recorded the caveat in the
docstring rather than papering over it.

Not this PR

• HAS_VENDOR_DEPS (html5lib) is #738's own remaining scope; engine-tests
  is newly dark on it too (same reason test already is — its ignore-shape
  selection reaches the same gate), now recorded in the exclusion.
• Giving HAS_RUNTIME in test_runtime_engines.py its own flag name (so the
  skip reason names which dependency was missing) is left as a follow-up; this
  guard does not care what a flag is named, only whether the job that reaches it
  installs what it asks for, so it was not required to close this issue.

[JarryShaw]

❌ NEEDS CHANGES @ c2bbf68f3 — supersedes my ❌ at a4081b83e (opus, independent).

All three defects I raised are fixed and verified, and the tshark work is better than I suggested: the edited branch asserts reason is None rather than skipping, so it never goes vacuous. HAS_SCAPY's new "10 of 14 … other 4 in tests/integration/ or *_runtime.py, covered by integration/gate" is exactly right. Guard still 84 passed / 508 subtests; gate counts still 14/3/15/1/4; merges clean into 932cb48d1 with guard and pyshark green on the merge, so being behind is harmless — checked, not assumed.

The guard finding is confirmed, and it is two-way. extras_providing('pcap') and extras_providing('pcap._pcap') both return ['PCAP_CT', 'PyPCAP']. Doctoring one install line each way:

doctored job	reaches	guard reports	unexplained gaps
pypcap-parity installs PCAP_CT	4 HAS_PYPCAP gates	no gap	0
engine-tests installs PyPCAP	1 HAS_PCAP_CT gate	no gap	0

Liveness passes in both: HAS_PYPCAP's exclusion still lists integration+gate, both still real gaps, so the wrong non-gap is never examined. One correction — the flags do not "both read as needing pcap alone": flag_requirements() gives HAS_PYPCAP → {'pcap'} but HAS_PCAP_CT → {'pcap._pcap'}. The conflation is one level down, in extras_providing().

Prose is not sufficient, and a tripwire is expressible today — contra "it would need a real per-distribution model". Assert that every (job, gate) pair the guard deems satisfied is satisfied through an import name with exactly one providing extra, else require an allowlist entry. Same shape as the existing machinery, no per-distribution model needed, and it fails loudly on both rows above.

Three to fix:

1. tests/foundation/engines/test_pyshark_engine.py:22-25 is now false — "tshark is not installed here, so the 'missing binary' path is exercised for real … installing Wireshark was not an option". engine-tests installs it at unit-tests.yml:328. Third round running, same failure mode, in the file this round began editing.
2. The new branch uses the wrong oracle. It keys on shutil.which('tshark'), but PyShark.unsupported_reason()'s own docstring says it delegates to pyshark's get_process_path() because "the two are not equivalent and which would refuse setups that work". pyshark reads config.ini from Path.cwd() before PATH, so with a ./config.ini naming an off-PATH tshark: which→None, unsupported_reason()→None, 2 failed (this test, and the sibling at :204). Same on macOS via /Applications/Wireshark.app. Pre-existing in the sibling, now copied into a second test — use the oracle production uses.
3. unit-tests.yml:456 still reads "unlike the three jobs above"; there are now four (test, integration, engine-tests, pypcap-parity) — broken by this PR's own insertion, and flagged last round.

Noted, not blocking: with tshark present the sibling self-skips on all 5 engine-tests legs and is unreachable elsewhere, so the unmocked missing-binary path now runs on no per-PR leg — a swap rather than a loss, since the new else branch exercises the unmocked present path instead. The pair cannot both be vacuous (verified: 8 passed absent, 7 passed / 1 skipped present) — though per (2) they can both fail. Pre-existing nit: _dependency_gates.py:723 says HAS_DPKT reaches "none through a method decorator"; there are 8.

Labels ci+test still fit (test more so now that it edits a test); breaking correctly absent — no pcapkit/ file in the diff. Good for main 932cb48d1 once the three land.

❌ NEEDS CHANGES @ c2bbf68f3

[JarryShaw]

Measurement note from the a4081b83e round, still accurate and unaffected by c2bbf68f3.

The skip delta reproduces exactly on a throwaway 3.10 venv (never the repo venv): the precise 27-node subset goes 27 skipped / 0 passed → 27 passed / 0 skipped, and test_new_engine_parity_runtime.py goes 0 passed / 10 skipped → 6 passed / 4 skipped with all 4 remaining skips "pypcap not installed". unittest agrees (Ran 10 tests … OK (skipped=4)), so no subtest hiding. PCAP_CT's real backend passed for real rather than via unsupported_reason()'s degrade path.

Minor nit, not blocking: the absolute 107 passed → 134 passed pair matches no reconstructible selection — the whole 79-file bounded list gives 1098 → 1125, a narrower 9-file grouping 101 → 128. Every selection shows the same +27 / 0 failed, so the substance holds; only those two totals are unreproducible as written.

Current verdict is in the review above, re-run against c2bbf68f3.

[JarryShaw]

Confirmed the guard blind spot myself, and filed it as #762 — but it is being fixed here rather than deferred, since this PR adds the two jobs (engine-tests, pypcap-parity) whose gates it falsely clears, and a separate PR would contend on tests/_dependency_gates.py.

One precision point for the mechanism sentence, which is currently wrong. The two flags do not "both read as needing pcap alone" — flag_requirements() resolves them apart correctly:

('tests/foundation/engines/test_new_engine_parity_runtime.py', 'HAS_PYPCAP')  -> {'pcap'}
('tests/foundation/engines/test_pcap_ct_engine.py',            'HAS_PCAP_CT') -> {'pcap._pcap'}

The conflation is one level down, in extras_providing(), which maps either name to ['PCAP_CT', 'PyPCAP']. That matters because it says where the fix belongs.

A docstring is not enough here, contra "would need a real per-distribution model": asserting that every (job, gate) pair deemed satisfied resolves through an import name with exactly one providing extra — else an explicit allowlist entry — needs no new model and fails loudly on both doctored directions.

Also outstanding: unit-tests.yml:456 still says "the three jobs above" when this PR's own insertion made it four (flagged in round 1); the tshark branch probes shutil.which, which unsupported_reason()'s docstring explicitly says is not equivalent to what the code under test uses; and _dependency_gates.py:723's HAS_DPKT "none through a method decorator" undercounts.

[JarryShaw]

Cross-review verdict at e902ede37: NEEDS CHANGES. The guard rewrite is sound; the PR is red on CI from its own new test.

Blocker — all five Engines legs fail (22 other checks green):

>       self.assertIsNone(shutil.which('tshark'))
E       AssertionError: '/usr/bin/tshark' is not None
tests/foundation/engines/test_pyshark_engine.py:267

An intent/assertion mismatch, and the surrounding comment states the correct intent — "which genuinely cannot see the stand-in, because its directory was never put on PATH", which is true by construction. The assertion instead requires the host to have no tshark at all, and this PR's own apt-get install … tshark provides one. pypcap-parity passes only because it installs no tshark and never reaches that unit-tier file.

Four stale-prose sites, the same pattern as the three fixed last round: unit-tests.yml:316-317 and _dependency_gates.py:323 both claim the assertion "checks shutil.which('tshark') first" when both sites now use _tshark_missing() — the comments advertise the defect as the remedy; _dependency_gates.py:311 says "its 3 gated methods run nowhere" when HAS_PYSHARK has moved 3 → 4; and test_pyshark_engine.py:22-25 still says "tshark is not installed here".

What is confirmed good. The #762 fix landed well and the allowlist is not an escape hatch — rewriting an entry to name the wrong distribution fails, adding a vestigial one fails, and deleting one surfaces the real finding, so hiding the defect requires making the guard red. Both doctored install-swaps now yield the right AmbiguousProvider. unit-tests.yml:456's four-job enumeration and the HAS_DPKT 8-method/20-via-6-class counts re-derive exactly. Guard is 89 passed / 511 subtests, up from 84/508.

Worth recording: the reviewer withdrew its own proposed design in favour of the author's. Keying the ambiguity check on MODULE_PROVIDERS (distribution level, 2 real cases) rather than extras_providing() (extra level, 13 same-distribution false positives) was the author's correction to my brief, and it was right.

[JarryShaw]

Second cross-review at e902ede37 (opus, independent of the first): NEEDS CHANGES, and it overturns one of the first review's conclusions. I settled the disagreement myself; the second reviewer is right.

The allowlist can re-hide #762 with every check green. I had reported the opposite from the first review, and that was wrong. AMBIGUOUS_PROVIDER_ALLOWLIST ships two live entries:

{('pypcap-parity', 'HAS_PYPCAP'): 'pypcap', ('engine-tests', 'HAS_PCAP_CT'): 'pcap-ct'}

They are correct today, but the anti-rot test asserts only findings[(job, flag)].satisfied == frozenset({extra}) — that the table agrees with what the job installs. Nothing ties a flag to the distribution that is correct for it. So flipping a job's install line and its allowlist value together silences the finding and the test still passes. The first review's attack named a distribution the job does not install, which naturally fails; it never tried the mutation that matters. Consequence: ambiguous_satisfactions() returning 0 on the real workflow is 0-because-allowlisted, not 0-because-correct.

The allowlist is also unnecessary, because the discriminator already exists in the source:

HAS_PYPCAP  = _importable('pcap') and not _importable('pcap._pcap')
HAS_PCAP_CT = _importable('pcap._pcap')

Only pcap-ct ships pcap._pcap, so modelling the negated probe — or keying MODULE_PROVIDERS on the exact module path — makes both entries redundant and removes the hole rather than guarding it.

html5lib is not a two-distribution ambiguity, and the check false-positives on it. I previously called it a second genuine ambiguity; that was also wrong. pyproject.toml:203 declares beautifulsoup4[html5lib] and never bare html5lib — one distribution reached two ways, exactly as MODULE_PROVIDERS' own comment says. Doctoring test to gain vendor (i.e. closing #738's gap, which this PR's own "Not this PR" section names) produces a finding with no wrong half, whose recommended remedy walks the contributor into the allowlist hole above. Ambiguity should be declared explicitly rather than inferred from len(MODULE_PROVIDERS[...]) > 1; only pcap qualifies.

A new assertion is provably vacuous. Deleting the NON_DISTRIBUTION_FLAGS skip from ambiguous_satisfactions() leaves test_a_flag_no_extra_could_satisfy_is_not_reported_as_a_gap passing and the whole file green at 89/511. It is doubly vacuous — MODULE_PROVIDERS['pyshark'] has one provider, so the length filter excludes it anyway.

And the "exactly 4 failures / 4 tests" claim is wrong: stubbing to () gives 4 failures from 3 methods, because the allowlist test contributes two SUBFAILED lines — and pytest's -v prints PASSED for that method while its subtests fail. The known pytest-subtests hazard, live in this PR's own evidence.

Both reviews independently found the CI blocker at test_pyshark_engine.py:267, and both agree the tshark probe mechanism itself is right. The second also confirms unit-tests.yml:456's four-job enumeration is true of the jobs and not merely their count, the HAS_DPKT 8-plus-20 split, and no regression (89 passed / 511 subtests under both runners). One pre-existing error it turned up, not this PR's: _dependency_gates.py:111 says "Six of the other seven install .[all]" — only 5 of 7 do.

[JarryShaw]

Final cross-review at 01f456650: GOOD TO GO (opus, independent of the author's model). All five items landed and coverage is strictly wider than before.

The tripwire is genuinely derived, proven the hard way. contested_imports() reads only MODULE_PROVIDERS and declared_requirements() — the name MUTUALLY_EXCLUSIVE_IMPORTS does not appear in its body — and it is invariant under garbage values of the set it checks while remaining responsive to its real inputs:

contested_imports() == MUTUALLY_EXCLUSIVE_IMPORTS -> True      (both ['pcap'])
set emptied / set = ['html5lib'] / set = ['dpkt','pcap','zzz'] -> ['pcap'] unchanged
MODULE_PROVIDERS['pyshark'] made two-live -> ['pcap','pyshark']
MODULE_PROVIDERS['pcap'] narrowed to one  -> []

That is the exact inverse of the allowlist's failure mode: its anti-rot test read from the thing it was checking, so agreement was tautological. This one cannot.

The ordering bug is fixed and verified against the reviewer's own cases, not the author's:

unrelated flag with an unmapped negated probe -> ()          was: KeyError: 'zzz_unmapped_name'
contested flag, negation names an unmapped dotted path
  -> AssertionError naming the missing MODULE_PROVIDERS entry  was: silent false positive

Coverage widened rather than merely held. Deleting MODULE_PROVIDERS['pcap._pcap'] now fails 7 tests (was 5); emptying MUTUALLY_EXCLUSIVE_IMPORTS fails 5 (was 3); dropping the negation from HAS_PYPCAP's source still fails 2. Both #762 directions still produce the right finding with unexplained gaps=[], and the undoctored workflow produces none.

Counts: 105 passed / 519 subtests under pytest and Ran 105 tests … OK under python -m unittest, so nothing is hidden behind a passing parent node. CI 28 pass, 3 skipping, 0 fail. Merge with main 60bb62d8f is clean and green on the merge.

One non-blocking prose correction, for the next touch of the file rather than another round. Item 5's new docstring names four tests as covering the dotted-key deletion; two of them do not — test_a_negated_probe_is_not_a_requirement pins the string in a synthetic module and never touches MODULE_PROVIDERS, and test_removing_the_negation_or_the_exact_path_reopens_762 uses patch.dict, which creates the key when absent so the deletion is invisible to it. Four others it omits do catch it. So the conclusion is true and in fact understated — the deletion is caught seven ways, not four — and only the attribution is wrong.

Unpublished and awaiting the owner.