From 748d46e3cee5afa5b9af36c8fe3bfc3173b8c8fa Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 16 Sep 2026 00:13:00 -0400 Subject: [PATCH] scapy: load the layer registry, so the engine actually dissects `engine='scapy'` returned every frame as `Raw` unless the calling program happened to have imported `scapy.all` itself, so the engine delivered none of the dissection it exists to provide -- silently. `scapy/__init__.py` is inert, so `from scapy import sendrecv` left `conf.l2types` with **0 entries** and `DLT 1 -> None`, and `PcapReader` could not map plain Ethernet. before, fresh process Raw x6, 0 TCP streams before, after import scapy.all Ether x6, 3 streams after, either way Ether x6, 3 streams That invariance to ambient import state *is* the fix; the test-order dependence was a symptom of it. The mechanism that hid this is worth recording: `pcapkit/toolkit/scapy.py`'s lazy `from scapy.layers.inet6 import IPv6ExtHdrFragment` **repairs** the registry mid-run -- measured, `l2types` 0 -> 14 and `DLT 1 -> Ether` -- but one call after `sniff()` has already decided how to parse. Documented there so it is not mistaken for the mechanism again. `scapy.all` chosen on correctness, not cost. The narrow alternative -- importing only the modules that register a DLT -- fixes the *link* layer and then silently truncates *payloads* on 6 of the 15 sample captures: dhcp.pcapng, dhcp_{big,little}_endian, test.pcapng, profile.pcapng narrow: Ethernet/IP/UDP/Raw broad: .../UDP/BOOTP/DHCP options test.pcap narrow: Ethernet/IP/UDP/Raw broad: .../UDP/DNS which is the "looks fixed and isn't" outcome, and it is unfixable by enumeration: it needs every payload binding, i.e. `conf.load_layers`, which scapy maintains and a hardcoded list here would silently rot against. There is also no cheaper complete option -- `scapy.layers.all`, the bare registry populator, measures 0.459s against `scapy.all`'s 0.46s, so the layer loading *is* the cost. The import stays lazy inside the engine, and that was verified rather than assumed: `import pcapkit` is unchanged at 0.51s with `scapy.all` absent from `sys.modules`; only engine construction moves, 0.108s -> 0.433s, once per process, paid solely by callers who ask for this engine. The `CryptographyDeprecationWarning` is deliberately not suppressed. It comes from `scapy.layers.dcerpc` pulling in the TLS layer, so *any* complete load emits it -- not `scapy.all` specifically -- and it subclasses `UserWarning`, so default filters show it. `pcapkit/utilities/warnings.py` states the house position that filters belong to the consumer, and silencing it here would hide the only notice that a future `cryptography` has broken scapy's TLS layer. The one-line filter is documented instead. Three tests encoded the wrong rationale and are corrected, each justified from the capture rather than re-baselined. `test_scapy_engine_finds_no_streams_for_this_capture` became `test_scapy_engine_matches_the_default_engine`, asserting three-axis parity with `assertNotIsInstance(frame[0], Raw)` **first**, so a regression names the cause instead of surfacing as `0 != 3`. `tests/integration/test_engine_runtime.py` asserted `'Raw'` where the two tests above it report the same frame as `Ethernet:IPv6:IPv6_ICMP` and `Ethernet:IP6:ICMP6`. New `tests/foundation/engines/test_scapy_engine.py` asserts in a **subprocess**, the only place ambient pollution cannot fool it; it fails on the unfixed engine with `scapy's conf.l2types was empty`. Sibling engines checked, and scapy was the only one broken: `dpkt` builds its registry in the package `__init__`, so narrow and full imports are byte-identical (74 modules, `_typesw`=11 both); `pyshark` has no registry and is already gated by `unsupported_reason()`; `pypcapfile` eagerly imports all four submodules it uses; `pypcap` is a single extension module; `pcap_ct` probes before importing. `unsupported_reason()` deliberately says nothing here -- a non-`None` return *swaps the engine for the default*, and scapy is installed, importable and fully working. The bug was ours. Verified: fixture-free CI 649 passed / 5 skipped against a 647/5 baseline, the delta being the two new tests; integration and the runtime/regression tiers byte-identical to baseline. Order dependence gone -- `tests/toolkit` then `test_misc.py` now 52 passed / 4 skipped where the baseline was 1 failed / 49 passed, and the reversed order is identical. --- pcapkit/foundation/engines/scapy.py | 75 ++++++++- pcapkit/toolkit/scapy.py | 18 ++ tests/foundation/engines/test_scapy_engine.py | 157 ++++++++++++++++++ tests/integration/test_engine_parity.py | 18 +- tests/integration/test_engine_runtime.py | 14 +- tests/interface/test_misc.py | 81 +++++++-- 6 files changed, 335 insertions(+), 28 deletions(-) create mode 100644 tests/foundation/engines/test_scapy_engine.py diff --git a/pcapkit/foundation/engines/scapy.py b/pcapkit/foundation/engines/scapy.py index a69fc22d41..8c893f5d19 100644 --- a/pcapkit/foundation/engines/scapy.py +++ b/pcapkit/foundation/engines/scapy.py @@ -9,6 +9,33 @@ .. _Scapy: https://scapy.net +.. note:: + + Constructing this engine imports :mod:`scapy.all`, which is what populates + `Scapy`_'s layer registries -- see :meth:`Scapy.__init__` for why anything + narrower silently returns undissected frames. + + One side effect is worth knowing about in advance: :mod:`scapy.all` loads + :mod:`scapy.layers.dcerpc`, which reaches `Scapy`_'s TLS layer and there + triggers a ``CryptographyDeprecationWarning`` from :mod:`cryptography` about + finite-field Diffie-Hellman. It is emitted by any complete registry load, not + by :mod:`scapy.all` in particular, and it concerns a key-exchange code path + :mod:`pcapkit` never executes -- but it subclasses :exc:`UserWarning`, not + :exc:`DeprecationWarning`, so Python's default filters show it. + + :mod:`pcapkit` deliberately does not filter it away. The warning is `Scapy`_'s + to emit and the consumer's to silence, on the same footing as every other + category (see :mod:`pcapkit.utilities.warnings`); hiding a third-party + deprecation notice from inside a constructor would suppress the only advance + warning that a future :mod:`cryptography` release breaks `Scapy`_'s TLS layer. + To silence it, filter it as usual:: + + import warnings + + from cryptography.utils import CryptographyDeprecationWarning + + warnings.filterwarnings('ignore', category=CryptographyDeprecationWarning) + """ from typing import TYPE_CHECKING, cast @@ -39,10 +66,10 @@ class Scapy(Engine['ScapyPacket']): """ if TYPE_CHECKING: - import scapy.sendrecv + import scapy.all #: Engine extraction package. - _expkg: 'scapy.sendrecv' + _expkg: 'scapy.all' #: Engine extraction temporary storage. _extmp: 'Iterator[ScapyPacket]' @@ -61,7 +88,41 @@ class Scapy(Engine['ScapyPacket']): ########################################################################## def __init__(self, extractor: 'Extractor') -> 'None': - from scapy import sendrecv as scapy # isort:skip + """Initialise the engine. + + Args: + extractor: :class:`~pcapkit.foundation.extraction.Extractor` instance. + + """ + # NOTE: :mod:`scapy.all`, not :mod:`scapy.sendrecv`, and the difference is + # load-bearing rather than cosmetic. Scapy dispatches on two registries that + # exist only as *import side effects* of its layer modules: ``conf.l2types``, + # mapping a capture's link type onto a link-layer class, and the + # ``bind_layers`` payload table. ``scapy/__init__.py`` fills in neither, so + # importing just the sniffing submodule leaves both empty -- + # :class:`~scapy.utils.PcapReader` then cannot map even link type 1 (plain + # Ethernet), writes ``unknown LL type [1]/[0x1]`` to stderr and returns every + # frame as one opaque :class:`~scapy.packet.Raw` layer. Nothing raises, so the + # engine used to deliver no dissection whatsoever and announce it only on + # stderr (#406). + # + # Naming the layer modules individually is not a cheaper way to the same + # place: it repairs the link layer and leaves the payload table short, so + # ``dhcp.pcapng`` still comes back as ``Ethernet / IP / UDP / Raw`` rather + # than ``... / UDP / BOOTP / DHCP options``. Enumerating them here would also + # go stale silently -- a link or payload type added by a later scapy would + # regress to ``Raw`` with no error -- whereas :mod:`scapy.all` defers to + # ``conf.load_layers``, which scapy itself keeps current. The cost is the + # layer loading, not the façade: importing :mod:`scapy.layers.all` alone + # measures the same, so there is no complete-but-cheaper option to prefer. + # + # This stays inside ``__init__`` rather than moving to module scope so that + # ``import pcapkit`` does not pay for it; only callers who actually select + # this engine do. And it is bound to :attr:`_expkg`, the attribute + # :meth:`run` calls ``sniff`` on, rather than left as a bare side-effecting + # import -- an import whose value is used cannot be dropped later as unused, + # which is how this class of bug comes back. + from scapy import all as scapy # isort:skip self._expkg = scapy self._extmp = cast('Iterator[ScapyPacket]', None) @@ -75,9 +136,11 @@ def __init__(self, extractor: 'Extractor') -> 'None': def run(self) -> 'None': """Call :func:`scapy.sendrecv.sniff` to extract PCAP files. - This method assigns :attr:`self._expkg ` - as :mod:`scapy.sendrecv` and :attr:`self._extmp ` - as an iterator from :func:`scapy.sendrecv.sniff`. + This method assigns :attr:`self._extmp ` as an iterator + from :func:`scapy.sendrecv.sniff`, reached through + :attr:`self._expkg ` -- which :meth:`__init__` binds to + :mod:`scapy.all`, since that is the import that populates the layer + registries ``sniff`` needs to dissect anything. Warns: AttributeWarning: If :attr:`self.extractor._exlyr ` diff --git a/pcapkit/toolkit/scapy.py b/pcapkit/toolkit/scapy.py index d5904c0063..3d79007647 100644 --- a/pcapkit/toolkit/scapy.py +++ b/pcapkit/toolkit/scapy.py @@ -15,6 +15,24 @@ This module requires installed `Scapy`_ engine. +.. note:: + + Several functions here import a `Scapy`_ layer module lazily, inside the + function body -- :func:`ipv6_reassembly` needs + :class:`scapy.layers.inet6.IPv6ExtHdrFragment`, for instance. Those imports are + for the *classes* they name and nothing more. They must not be mistaken for how + `Scapy`_'s layer registries get populated, even though they do populate them as + a side effect, because by the time any of these functions runs the engine has + already called ``sniff`` and every frame has already been dissected -- or not. + + That distinction is what made #406 hard to see. Reaching + :func:`ipv6_reassembly` repaired ``conf.l2types`` mid-run, one call too late to + affect the frames being reassembled, so whether a process dissected correctly + depended on what had imported `Scapy`_ earlier. Populating the registries before + ``sniff`` is + :class:`~pcapkit.foundation.engines.scapy.Scapy`'s job, and it does it by + importing :mod:`scapy.all` in its constructor. + """ import ipaddress import time diff --git a/tests/foundation/engines/test_scapy_engine.py b/tests/foundation/engines/test_scapy_engine.py new file mode 100644 index 0000000000..38353fd812 --- /dev/null +++ b/tests/foundation/engines/test_scapy_engine.py @@ -0,0 +1,157 @@ +"""Unit tests for :class:`pcapkit.foundation.engines.scapy.Scapy`'s import contract. + +`Scapy`_ builds its dissection tables as *import side effects* of its layer +modules. Two matter here, and neither is populated by ``scapy/__init__.py``: + +* ``conf.l2types`` maps a capture's link type onto a link-layer class. Without it + :class:`~scapy.utils.PcapReader` cannot tell what it is reading, writes + ``unknown LL type [1]/[0x1]`` to stderr and returns each frame as one opaque + :class:`~scapy.packet.Raw` layer. +* the ``bind_layers`` table maps each layer onto its payload. Without it a frame + dissects down as far as the loaded modules reach and then stops at ``Raw``. + +So this engine's *import statement* is load-bearing in a way an engine's usually +is not, and #406 is what happens when it is wrong: the engine imported only +:mod:`scapy.sendrecv`, and every frame of every capture came back as ``Raw`` -- +no exception, no :mod:`pcapkit` warning, and a stderr line from `Scapy`_ that a +library consumer never sees. ``follow_tcp_stream`` quietly reported no streams. + +**Why these tests use a subprocess.** The tables live on `Scapy`_'s process-wide +``conf`` singleton, and once any module has populated them they stay populated: no +public API resets them, and ``sys.modules`` surgery does not un-run a +``bind_layers`` call. So an in-process assertion here would pass whenever +something earlier in the run had imported a `Scapy`_ layer -- which is exactly +what happened before #406 was found. :mod:`tests.toolkit.test_scapy_unit` imports +:mod:`scapy.layers.l2`, :mod:`~scapy.layers.inet` and :mod:`~scapy.layers.inet6` +to build its fixtures, so ``pytest tests/toolkit tests/interface/test_misc.py`` +saw correct dissection while ``pytest tests/interface/test_misc.py`` alone did +not. A fresh interpreter is the only place the engine's own import can be held +responsible for the outcome, so that is where it is asserted. + +The end-to-end behaviour these guard is asserted through the public interface in +:meth:`tests.interface.test_misc.FollowTCPStreamTests.test_scapy_engine_matches_the_default_engine`; +this module pins the cause rather than the symptom. + +.. _Scapy: https://scapy.net + +""" +from __future__ import annotations + +import importlib.util +import json +import pathlib +import subprocess +import sys +import unittest + +from tests._support import sample_path + +#: Repository root, which the child interpreter needs on :data:`sys.path` to +#: import the :mod:`pcapkit` under test rather than an installed copy. +ROOT = pathlib.Path(__file__).resolve().parents[3] + +RUNTIME_DEPS = ('tbtrim', 'aenum', 'chardet', 'dictdumper') +HAS_RUNTIME = all(importlib.util.find_spec(name) is not None for name in RUNTIME_DEPS) +HAS_SCAPY = importlib.util.find_spec('scapy') is not None + +#: Marker the child prefixes its one JSON line with. `Scapy`_ writes to stderr on +#: an unmapped link type and :mod:`pcapkit` warns on EOF, so the result is picked +#: out by prefix rather than by assuming a clean stream. +RESULT = '@@RESULT@@' + +#: Run in a fresh interpreter, with nothing imported that could have populated +#: `Scapy`_'s tables beforehand. Reports what the engine produced *and* what the +#: link-type table looked like afterwards, so a failure distinguishes "the engine +#: did not import enough" from "the capture is not what the test thinks". +CHILD = f''' +import json, sys, warnings + +sys.path.insert(0, {str(ROOT)!r}) +warnings.simplefilter('ignore') + +import pcapkit + +extractor = pcapkit.extract(fin=sys.argv[1], engine='scapy', store=True, nofile=True) + +# Imported only now, and from ``scapy.config`` rather than ``scapy.all``, so that +# reading the table cannot be what filled it in. +from scapy.config import conf + +print({RESULT!r} + json.dumps({{ + 'frames': [type(frame).__name__ for frame in extractor.frame], + 'chain': extractor.frame[0].summary(), + 'has_tcp': any(frame.haslayer('TCP') for frame in extractor.frame), + 'l2types': len(conf.l2types.num2layer), + 'dlt1': getattr(conf.l2types.num2layer.get(1), '__name__', None), +}})) +''' + + +@unittest.skipUnless(HAS_RUNTIME, 'runtime dependencies not installed') +@unittest.skipUnless(HAS_SCAPY, 'scapy not installed') +class ScapyLayerRegistryTests(unittest.TestCase): + """What the Scapy engine dissects in an interpreter it did not warm up.""" + + def dissect(self, capture: str) -> dict: + """Extract ``capture`` with the Scapy engine in a fresh interpreter. + + Args: + capture: Absolute path to the capture to read. + + Returns: + The child's decoded report -- frame class names, frame 0's protocol + chain, whether any frame holds a TCP layer, and the size and DLT-1 + entry of `Scapy`_'s link-type table. + + """ + completed = subprocess.run( + [sys.executable, '-c', CHILD, capture], + capture_output=True, text=True, timeout=300, check=False, + ) + self.assertEqual( + completed.returncode, 0, + f'child interpreter failed\nstdout:\n{completed.stdout}\n' + f'stderr:\n{completed.stderr}', + ) + for line in completed.stdout.splitlines(): + if line.startswith(RESULT): + return json.loads(line[len(RESULT):]) + self.fail(f'child printed no result line\nstdout:\n{completed.stdout}\n' + f'stderr:\n{completed.stderr}') + + def test_engine_populates_the_link_type_table_by_itself(self) -> None: + # The engine's own import has to be sufficient. Asserted on the table + # rather than on the frames so that a regression names its cause: an empty + # ``l2types`` is the missing import, nothing else. + report = self.dissect(sample_path('in.pcap')) + + self.assertGreater( + report['l2types'], 0, + "scapy's conf.l2types was empty after the engine ran, so the engine " + 'imported no layer module -- see #406: it must import scapy.all', + ) + self.assertEqual( + report['dlt1'], 'Ether', + 'link type 1 (Ethernet) did not map to scapy.layers.l2.Ether, so ' + 'PcapReader cannot dissect even an ordinary Ethernet capture', + ) + + def test_frames_are_dissected_not_returned_as_raw(self) -> None: + # in.pcap is six Ethernet frames: two ICMPv6 neighbour-discovery, three + # TCP, one UDP. Under #406 all six came back as ``Raw``. + report = self.dissect(sample_path('in.pcap')) + + self.assertNotIn( + 'Raw', report['frames'], + f"scapy returned undissected Raw frames from in.pcap: {report['frames']} " + '-- the layer registry was not populated; see #406', + ) + self.assertEqual(report['frames'], ['Ether'] * 6) + self.assertTrue(report['chain'].startswith('Ether / IPv6 / ICMPv6ND_NS'), + f"unexpected chain for frame 0: {report['chain']}") + self.assertTrue(report['has_tcp'], + 'no TCP layer found, though in.pcap holds three TCP frames') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/integration/test_engine_parity.py b/tests/integration/test_engine_parity.py index 395bab755d..2f12dd8b51 100644 --- a/tests/integration/test_engine_parity.py +++ b/tests/integration/test_engine_parity.py @@ -111,10 +111,20 @@ def test_scapy_engine_returns_the_same_link_layer_bytes(self) -> None: self.assertEqual(len(native.frame), len(foreign.frame)) for number, (mine, theirs) in enumerate(zip(native.frame, foreign.frame), start=1): with self.subTest(frame=number): - # scapy does not know this capture's link type and hands back one - # opaque ``Raw`` layer holding the whole 60 octet frame, padded to - # the Ethernet minimum. Its first fourteen octets are the Ethernet - # header that the default engine parsed into a layer of its own. + # Re-serialising scapy's frame reproduces the whole 60 octet frame, + # padded to the Ethernet minimum, and its first fourteen octets are + # the Ethernet header the default engine parsed into a layer of its + # own. That is what makes this a bytes-level parity check: both + # engines account for every captured octet, whatever they call the + # layers they split it into. + # + # #406: this comment used to say scapy "does not know this capture's + # link type and hands back one opaque ``Raw`` layer". That was the + # missing-layer-registry bug, not a property of the capture -- + # arp.pcap is link type 1 and scapy now dissects it as + # ``Ethernet / ARP / Padding``. The assertions below are unchanged + # and still pass, because ``bytes()`` of a dissected frame is the + # same octet string as ``bytes()`` of an undissected one. captured = bytes(theirs) self.assertEqual(len(captured), 60) self.assertEqual(captured[:14], bytes(mine['Ethernet'].packet.header)) diff --git a/tests/integration/test_engine_runtime.py b/tests/integration/test_engine_runtime.py index 47a69cff32..ca63423596 100644 --- a/tests/integration/test_engine_runtime.py +++ b/tests/integration/test_engine_runtime.py @@ -43,14 +43,26 @@ def test_dpkt_engine_returns_dpkt_packets_and_toolkit_chain(self) -> None: @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') def test_scapy_engine_returns_scapy_packets(self) -> None: + # #406: this asserted ``Raw`` -- the engine imported only ``scapy.sendrecv``, + # so scapy's layer registries were empty and every frame came back + # undissected. ``Ether`` is what the capture actually holds: frame 1 of + # in.pcap is an Ethernet frame carrying an ICMPv6 neighbour solicitation over + # IPv6, which is exactly what the two tests above independently report for the + # same frame -- 'Ethernet:IPv6:IPv6_ICMP' from the default engine and + # 'Ethernet:IP6:ICMP6' from DPKT. The three chains are three libraries' + # spellings of one frame, so scapy agreeing here is parity, not a new claim. from pcapkit.interface import extract + from pcapkit.toolkit.scapy import packet2chain extractor = extract(fin=sample_path('in.pcap'), fout='/tmp/out', format='tree', store=True, nofile=True, engine='scapy') self.addCleanup(close_extractor, extractor) frame = extractor.frame[0] self.assertEqual(extractor.length, 6) - self.assertEqual(type(frame).__name__, 'Raw') + self.assertEqual(type(frame).__name__, 'Ether') + self.assertEqual(packet2chain(frame), + 'Ethernet:IPv6:ICMPv6 Neighbor Discovery - Neighbor Solicitation:' + 'ICMPv6 Neighbor Discovery Option - Source Link-Layer Address') self.assertGreater(len(bytes(frame)), 0) @unittest.skipUnless(HAS_PYSHARK, 'pyshark not installed') diff --git a/tests/interface/test_misc.py b/tests/interface/test_misc.py index 7f940ed824..59c8e363e8 100644 --- a/tests/interface/test_misc.py +++ b/tests/interface/test_misc.py @@ -22,8 +22,9 @@ #: TCP conversations the default (pcapkit-native) engine finds in the committed #: ``in.pcap`` capture. Measured against ``examples/captures/in.pcap``, and the -#: yardstick every other engine is judged by -- see the class docstring for why -#: the per-engine expected values are *not* uniformly this number. +#: yardstick every other engine is judged by: every engine reaching this module +#: is expected to agree on it, either by dissecting the capture itself or by being +#: redirected to the default engine. IN_PCAP_TCP_STREAMS = 3 @@ -38,10 +39,19 @@ class FollowTCPStreamTests(unittest.TestCase): crashed on DPKT frames (``AttributeError: 'dict' object has no attribute 'packet'``) and returned an empty, misleading result on Scapy frames. - The expected stream count is asserted per engine rather than as one shared - number, because engines with genuine dissection gaps legitimately differ: - DPKT dissects this capture fully and must match the default engine, whereas - Scapy cannot read its link layer at all and correctly finds nothing. + Both third-party engines that dissect this capture -- DPKT and Scapy -- must + therefore agree with the default engine on all three streams. Engines that do + no dissection at all (PyPCAP) or have no reassembly adapter (PyShark) are + redirected to the default engine instead, and are asserted separately below. + + #406: this class previously expected Scapy to find *zero* streams, on the + stated rationale that it could not read the capture's link layer -- "a + documented capability gap". That rationale was wrong. ``in.pcap`` is plain + Ethernet (link type 1), which Scapy dissects perfectly well; the zero came + from :class:`~pcapkit.foundation.engines.scapy.Scapy` importing only + :mod:`scapy.sendrecv`, which populates none of Scapy's layer registries, so + every frame came back as an undissected ``Raw``. Zero was the polluted + answer and three is the truthful one. """ def setUp(self) -> None: @@ -92,27 +102,64 @@ def test_dpkt_engine_matches_the_default_engine(self) -> None: [stream.conversations for stream in native]) @unittest.skipUnless(HAS_SCAPY, 'scapy not installed') - def test_scapy_engine_finds_no_streams_for_this_capture(self) -> None: - # Scapy does not recognise this capture's link-layer type and hands every - # frame back as one opaque ``Raw`` layer with no TCP inside, so it correctly - # finds no TCP stream to follow. Zero is therefore a capability gap, not the - # #399 bug -- and the second assertion pins the *cause*, so a future scapy - # that learns this link type fails here loudly instead of silently drifting. + def test_scapy_engine_matches_the_default_engine(self) -> None: + # Scapy dissects in.pcap in full -- its two ICMPv6-over-IPv6 frames, its + # three IPv4/TCP frames and its one UDP frame -- so, exactly like DPKT above, + # it must agree with the native engine rather than differ from it. + # This is the regression guard for #406: the engine imported only + # ``scapy.sendrecv``, so Scapy's ``conf.l2types`` registry was empty, + # ``PcapReader`` could not map link type 1, and all six frames arrived as + # ``Raw`` -- turning three streams into zero with nothing raised. import pcapkit with warnings.catch_warnings(): warnings.simplefilter('ignore') - streams = self._follow(engine='scapy') + native = self._follow() + foreign = self._follow(engine='scapy') extractor = pcapkit.extract(fin=sample_path('in.pcap'), engine='scapy', store=True, nofile=True) - self.assertEqual(len(streams), 0) - self.assertFalse( + # Asserted first, and on the frame type rather than on the stream count, + # because it names the *cause*. A regression of #406 fails here as "frame 0 + # came back Raw", which points straight at the engine's import, instead of + # surfacing downstream as an unexplained 0 != 3 stream-count mismatch. + self.assertNotIsInstance( + extractor.frame[0], self._scapy_raw(), + 'scapy returned frame 0 of in.pcap as an undissected Raw layer, so its ' + 'layer registry was not populated -- see #406: the engine must import ' + 'scapy.all, which loads the registries, not merely scapy.sendrecv', + ) + self.assertTrue( any(frame.haslayer('TCP') for frame in extractor.frame), - 'scapy dissected a TCP layer from in.pcap; the empty-stream assertion ' - 'above is no longer a capability gap and this test needs revisiting', + 'scapy dissected no TCP layer from in.pcap, which holds three TCP frames', ) + # Full parity with the native engine, on the same three axes the DPKT test + # above checks: count, per-stream frame counts, and reassembled conversations. + self.assertEqual(len(foreign), IN_PCAP_TCP_STREAMS) + self.assertEqual([len(stream.packets) for stream in foreign], + [len(stream.packets) for stream in native]) + self.assertEqual([stream.conversations for stream in foreign], + [stream.conversations for stream in native]) + + @staticmethod + def _scapy_raw() -> type: + """Scapy's undissected-payload class, :class:`scapy.packet.Raw`. + + Imported inside a helper rather than at module scope because Scapy is an + optional extra: this module is collected, and its other cases run, on + installs where ``import scapy`` would fail outright. + + :mod:`scapy.packet` is also the right submodule to reach ``Raw`` through + for a test about #406, because it is registry-inert -- measured: importing + it leaves ``conf.l2types`` empty -- so naming the class cannot itself + populate the registries and mask the defect being asserted against. + + """ + from scapy.packet import Raw + + return Raw + def test_pyshark_and_pypcap_fall_back_to_the_default_engine(self) -> None: # Neither engine can trace TCP flows (PyShark has no reassembly adapter, # PyPCAP does no dissection), so follow_tcp_stream redirects them to the