ci: four HAS_* gates run on no per-PR leg -- decide which install lines take them

[JarryShaw]

#745's guard (tests/_dependency_gates.py) lands with seven allowlisted gaps, each carrying its reason. Four need nothing from anyone: PyPCAP needs libpcap headers, PCAP_CT is pre-release-only and wants a system libpcap at run time, and HAS_VENDOR_DEPS is waiting on #738's own remaining scope — the non-blocking vendor leg from ruling (b).

Three want a ruling. Two of them are not new; what is new is that they now have exactly one place that fails if they are quietly re-widened.

gate	dark on	methods	status
HAS_SCAPY	test	10	#738 set it aside because integration/gate install Scapy. But all 10 are unit-tier, so integration never reaches them, and gate only runs where a caller passes gate-only: true — the three Saturday schedules and a v* tag, never a PR or a push to main. No per-PR leg runs them. The test job declined Scapy on cost.
HAS_PYSHARK	all three	3	one of #738's seven, left out of #740's cheap subset and never installed since. Two of the three assert what PyShark.unsupported_reason says, so they need the distribution importable; tshark is a separate decision.
HAS_RUNTIME	test, gate	5	new — absent from #738's inventory. tests/foundation/engines/test_runtime_engines.py reuses the name for the four core deps plus dpkt, scapy, pyshark, so its two classes skip. Unit-tier despite the name (the glob is *_runtime.py). Narrow fix: give that flag its own name so the skip says what was missing.

Separately, HAS_PYPCAPFILE (15 methods, not #738's 10 — #747 grew that class from 4 to 9): #747 and #748 fixed the two bugs that made installing the extra turn skips into failures, so the only remaining obstacle is the extra's own python_version < '3.12' marker. Adding it covers 3.10/3.11 and leaves three legs skipping, which the guard cannot see because it does not evaluate markers. Worth taking or not?

Fixing any of these means deleting its DEPENDENCY_GATE_EXCLUSIONS entry; leaving a stale entry fails the liveness check, so none can become invisible again.

[JarryShaw]

Verified the load-bearing figures myself before this sits on the board.

HAS_PYPCAPFILE is 15, and that supersedes the "10" that has been in circulation — including in
#738, in #745's body, and in briefs I wrote. My own AST sweep over tests/ on 5c0df9248 gives 9 in
tests/toolkit/test_pypcapfile_unit.py and 6 in
tests/foundation/engines/test_new_engine_parity_runtime.py. The count moved because #747 grew that
gated class from 4 methods to 9; nobody was wrong at the time, the tree changed underneath.

The requests/bs4 claim holds. Both are in the test extra (pyproject.toml:234-252), with a
comment naming exactly this reason — so HAS_VENDOR_DEPS's only real gap is html5lib, as stated.

Two scope notes so the other numbers are not misread as disagreeing with a whole-tree count: my sweep
gives 14 HAS_SCAPY methods against the 10 here, and 1,246 for HAS_RUNTIME against 5. Both
are reconcilable — 10 is the unit-tier subset dark on the test leg, and HAS_RUNTIME is a widely
reused flag name whose 5 is scoped to test_runtime_engines.py. Worth stating in the issue which scope
each figure is at, since that ambiguity has already produced one wrong correction on this repo.

On the HAS_PYPCAPFILE question you raise: taking it is worth it. #747 and #748 are on main, so
the extra no longer turns skips into failures, and 3.10/3.11 would cover 15 methods that run nowhere
today. The python_version < '3.12' marker means 3.12-3.14 keep skipping and the guard cannot see
that — which argues for the guard learning to evaluate markers later, not for declining the coverage
now.

[JarryShaw]

What about we have per-engine's tests covered by a separate test step and on all supported Python versions? so they dont intertwine with the other major tests.

[JarryShaw]

Ruling recorded: a separate per-engine test step, on all supported Pythons

Taking that as: the third-party engine tests get their own job in unit-tests.yml, installing the
engine extras, running the full 3.10-3.14 matrix, and not intertwined with the existing test,
integration and gate jobs. It answers all four gaps at once rather than one install line at a time,
which is the better shape — the test job declined Scapy on cost grounds and that reasoning does not
survive being asked four times.

What it covers, measured on 5c0df9248: HAS_SCAPY 14 methods, HAS_PYSHARK 3,
HAS_PYPCAPFILE 15, HAS_RUNTIME 5 in test_runtime_engines.py. That is the engine surface; the
HAS_CRYPTO / HAS_EMOJI / HAS_PYCRATE gates already run and are not affected.

Three things the new job has to decide, and they are cheaper to settle now than in review:

• pypcapfile is marked python_version < '3.12', so on a 3.10-3.14 matrix it installs on two legs
  and its 15 methods skip on three. That is correct behaviour, not a gap — but the guard cannot see it,
  since it does not evaluate markers. Either the guard learns markers or the allowlist gains an entry
  saying "partial by marker".
• pyshark may need a tshark binary, not just the distribution. Two of its three methods assert
  what PyShark.unsupported_reason says, so they need the import to exist; whether they need the binary
  is unverified and worth establishing before the job is written rather than after it goes red.
• HAS_PYPCAP and HAS_PCAP_CT need a C toolchain and system libpcap headers. If the intent is
  "all engines", say whether the new job installs those too — that is a materially bigger job — or
  whether they stay allowlisted.

Not blocking, but it changes what #738 asks of you: its remaining scope is 41 methods and one
missing package (html5lib, only from beautifulsoup4[html5lib]), not the "55 methods, mostly
network" this board has been recording. If the new per-engine job establishes the pattern, the
vendor-crawler leg is the same shape and the network-reachability concern from #518 is the only
remaining question there.

#745's guard lands independently of this and will fail loudly if a gate goes dark again, so the job and
the guard are not sequenced against each other — only the allowlist entries change as each gap closes.

[JarryShaw]

and since PyPCAP and pcap-ct are exclusive to each other. having them on separate steps making them using separate venvs and thus avoid the issue.

[JarryShaw]

Confirmed, and the repo already documents the mechanism you are describing

pcapkit/foundation/engines/_pcap_backend.py's own docstring states it: both distributions install a
top-level module named pcap — upstream pypcap as an extension module (pcap.cpython-*.so,
installable to 3.11) and pcap-ct as a package (pcap/__init__.py, 3.10+). Measured there on Python
3.10 with both present: the pcap/ package wins, upstream's extension is shadowed and unreachable, so
pcap-ct always takes the name.

The overlap is real and narrow: PyPCAP = [ "pypcap; python_version < '3.12'" ] (pyproject.toml:151)
against "pcap-ct>=1.3.0b3; python_version >= '3.10'" (:194) — so both resolve on 3.10 and 3.11
only. Outside that window there is nothing to separate: 3.12+ is pcap-ct alone, and pypcap has no leg
below 3.10 either.

So separate steps with separate venvs is right, and it is the only way to exercise both — one venv
cannot, by construction. Two consequences for the job:

• PyPCAP's step needs a C toolchain and libpcap headers; pcap-ct needs neither (:210). They are
  not symmetric, so a single matrix entry parameterised by extra will not do — the pypcap step wants its
  own apt-get/setup.
• Per-step venvs mean per-step installs, which is slower than one install reused across steps. Worth
  stating in the PR so the cost is chosen rather than discovered.

That leaves the two questions from my previous comment still open: whether pyshark needs a tshark
binary as well as the distribution (unverified), and whether HAS_PYPCAPFILE's
python_version < '3.12' marker gets an allowlist entry saying "partial by marker" or #745's guard
learns to evaluate markers.

[JarryShaw]

if one engine is genuinely not testable from GHA, then we dont have to force it.

[JarryShaw]

Recorded: an engine that is genuinely not testable from GHA stays allowlisted

That settles the three open sub-questions rather than deferring them, and it gives the guard's allowlist a
principle instead of a case list. Measured positions:

gate	testable from GHA?	disposition
HAS_SCAPY (14), HAS_PYSHARK (3), HAS_PYPCAPFILE (15)	yes — pure-Python wheels	take them on the new per-engine job
HAS_PCAP_CT	yes, with a system libpcap — apt-get install libpcap-dev on ubuntu-latest, no compiler needed (pyproject.toml:210)	take it, on its own step/venv
HAS_PYPCAP	needs a C toolchain plus libpcap headers, sdist-only, and 3.10/3.11 only	allowlist it unless you want the build step

So the honest shape is four engine steps, not five — and HAS_PYPCAP keeps an allowlist entry whose
reason is "sdist-only C extension, not forced per your ruling" rather than the vaguer "needs a toolchain".

Two caveats that survive the ruling and should not be lost in it:

• pyshark may need a tshark binary, not just the distribution. Unverified — two of its three methods
  assert what PyShark.unsupported_reason reports, so they need the import to exist, but whether they need
  the binary is untested. If they do, tshark is an apt-get away on ubuntu-latest, so this is probably
  "take it" rather than "allowlist it" — worth establishing before the job is written.
• HAS_PYPCAPFILE's python_version < '3.12' marker means it installs on two legs and skips on three
  even once taken. That is correct behaviour, but tests: nothing asserts each HAS_* dependency is installed on a leg that reaches its gated tests #745's guard cannot see it — it does not evaluate
  markers — so it needs an allowlist entry reading "partial by marker" or the guard needs to learn markers.
  Silently partial coverage is the exact failure tests: nothing asserts each HAS_* dependency is installed on a leg that reaches its gated tests #745 exists to prevent.

HAS_PYPCAP and HAS_PCAP_CT also cannot share a venv at all: both distributions install a top-level
pcap module and pcap-ct's package shadows upstream's extension, per
pcapkit/foundation/engines/_pcap_backend.py. Separate steps handle that regardless of which are taken.

[JarryShaw]

HAS_PYCAP - try to build and if the CI is not a good suit, then we ripe it. same for HAS_PYSHARK on tshark dependency.