From 3f8f4d0e9f9fadb67df35882110a4fbe2408ecfd Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Tue, 15 Sep 2026 23:50:03 -0400 Subject: [PATCH 1/4] docs: prune the Help Wanted page, fix the docs extra, quiet 5 Sphinx warnings **`pep.rst` was soliciting help for work already done**, which is worse than saying nothing to a would-be contributor. Every claim was checked against the code before editing, not taken from a list. Removed the solicitations for SCTP (all 13 RFC 9260 chunk types, 8 parameters, 13 error causes, real CRC32c, PPID dispatch), the two new engines, the logging integration and the test suite -- all verifiably present. Narrowed rather than deleted where the work is only partly done, which is more useful than a stale "help wanted": * ESP -- the four missing algorithms confirmed genuinely absent, and sharpened with the real numbers: 5 encryption and 5 integrity algorithms are applied. * Mobility Header -- FMIPv6 *has* landed, so the vague "requires some help" became what actually remains: 10 of 24 message data types, 51 of 71 options, 3 of 4 CGA extensions. * SCTP -- narrowed to the registered-but-unimplemented surplus: 17 of 30 chunk types, 24 of 32 parameters, 10 of 23 error causes. * The `NotImplemented` list stays, with two corrections: NDP's stub is under `link/`, not `internet/`, and QUIC was dropped because no stub exists. Three places where the page was simply wrong: it claimed `BaseWarning` still calls `warnings.simplefilter` (removed in `487d3da82`, with tests); it said 84 test modules where there are 86; and it claimed the suite is "bundled with the distribution" when `[tool.setuptools.packages.find]` excludes `test*`, so an installed PyPCAPKit has no `tests` package -- now recorded as a still-wanted item rather than a boast. The page also claimed to mirror discussion 106 while carrying 4 of its 6 comments; the missing two are added. The `pcap-ct` paragraph said "a way out that has not been adopted yet". It has been adopted -- as its own `PCAP_CT` engine in #405 -- so it now describes what shipped, why a separate engine rather than a second backend (both distributions own the import name `pcap`, and with both installed pcap-ct wins while upstream is shadowed), and the two caveats that remain: both are betas, and a runtime `libpcap.so.1` is still required because the `libpcap` wheel's published config sets `LIBPCAP = None`. **The `[docs]` extra could not install at all.** It named `sphinx-opengraph`, which 404s on PyPI; the real package is `sphinxext-opengraph`, which is what `docs/source/conf.py:61` imports. `sphinxcontrib-mermaid` was missing entirely though `conf.py:65` loads it. So `pip install -e '.[docs]'` failed outright, which means nobody could build the docs from a clean checkout. Both fixed and verified to resolve. **Sphinx warnings 268 -> 263.** The TOC suspicion did not pan out, and that is worth recording: across 128 pages there are 0 orphans, 0 broken toctree entries, 0 `automodule` targets pointing at nothing, and 0 unresolvable `:doc:` targets. All 19 newly added modules resolve to a reachable page, and the const-enumeration convention is correctly followed -- `esp.rst` and `sctp.rst` live under `docs/source/pcapkit/const/`, and the ESP protocol page only cross-references them. What was fixed: `sphinx.ext.autodoc.typehints` removed from `extensions` (an internal submodule with no `setup()`, so it only ever warned; the real work is done by the third-party `sphinx_autodoc_typehints`), and the duplicate-object warnings on `esp.rst`/`context.rst`. Those two were the only pages in the protocols/corekit tree using `automodule`, and the cause needed two attempts: `:no-index:` alone removed only half, because the *module docstring itself* carries `.. module::`, as 362 pcapkit modules do. Dropping the redundant `.. module::` from the `.rst` and keeping `:no-index:` leaves the docstring as the single declaration. The 263 remaining are pre-existing and none are in the new modules: 158 ambiguous cross-references (short names like ``:class:`TransType``` matching both `const.*` and `vendor.*`), ~40 `TYPE_CHECKING`-only forward references, 38 duplicate object descriptions on established pages, 22 guarded-import failures that are environment-only, and 5 autodoc signature crashes on `__protocol_type__`/`__schema__`. Left alone deliberately: those pages render correctly today and fixing them means judgement calls about rendered output. --- docs/source/conf.py | 7 +- docs/source/pcapkit/corekit/context.rst | 7 +- .../source/pcapkit/protocols/internet/esp.rst | 7 +- docs/source/pep.rst | 461 +++++++++++------- pyproject.toml | 3 +- 5 files changed, 294 insertions(+), 191 deletions(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 0582bbcdb4..c06b691413 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -52,7 +52,12 @@ extensions = [ 'sphinx.ext.viewcode', 'sphinx.ext.intersphinx', - 'sphinx.ext.autodoc', 'sphinx.ext.autodoc.typehints', + # NB: ``sphinx.ext.autodoc.typehints`` is *not* listed here. It is an + # internal submodule of ``sphinx.ext.autodoc`` rather than an extension in + # its own right -- it exposes no ``setup()``, so loading it explicitly only + # earns a warning. The typehint rendering comes from the third-party + # ``sphinx_autodoc_typehints`` below. + 'sphinx.ext.autodoc', 'sphinx.ext.napoleon', 'sphinx.ext.todo', diff --git a/docs/source/pcapkit/corekit/context.rst b/docs/source/pcapkit/corekit/context.rst index 62a17bab09..e8b2954691 100644 --- a/docs/source/pcapkit/corekit/context.rst +++ b/docs/source/pcapkit/corekit/context.rst @@ -1,10 +1,15 @@ Parsing Context =============== -.. module:: pcapkit.corekit.context +.. Unlike its sibling pages, this one renders the module docstring through + ``automodule`` rather than repeating it as prose. That docstring already + carries its own ``.. module::`` directive -- as every module under + ``pcapkit`` does -- so there must be no second one here, and ``automodule`` + itself must not register a third: hence ``:no-index:``. .. automodule:: pcapkit.corekit.context :no-members: + :no-index: .. autoclass:: pcapkit.corekit.context.ProtocolContext :no-members: diff --git a/docs/source/pcapkit/protocols/internet/esp.rst b/docs/source/pcapkit/protocols/internet/esp.rst index faaac74bb9..8c4129aa57 100644 --- a/docs/source/pcapkit/protocols/internet/esp.rst +++ b/docs/source/pcapkit/protocols/internet/esp.rst @@ -1,10 +1,15 @@ ESP - Encapsulating Security Payload ==================================== -.. module:: pcapkit.protocols.internet.esp +.. Unlike its sibling pages, this one renders the module docstring through + ``automodule`` rather than repeating it as prose. That docstring already + carries its own ``.. module::`` directive -- as every module under + ``pcapkit`` does -- so there must be no second one here, and ``automodule`` + itself must not register a third: hence ``:no-index:``. .. automodule:: pcapkit.protocols.internet.esp :no-members: + :no-index: .. autoclass:: pcapkit.protocols.internet.esp.ESP :no-members: diff --git a/docs/source/pep.rst b/docs/source/pep.rst index ae0615bb86..b97259ece2 100644 --- a/docs/source/pep.rst +++ b/docs/source/pep.rst @@ -3,12 +3,17 @@ Help Wanted .. important:: - This is a copy of the `discussion thread `__ - started on the GitHub. The documentation is **only** used as a backup - reference to the original discussion thread. + This page mirrors the `discussion thread + `__ started on GitHub. + The thread holds the proposals as they were first raised, together with a + comment recording which of them have since landed; this page is the + maintained copy, kept in step with the code. So where a proposal has been + implemented it is described as implemented here, and only the work that is + genuinely still open is asked for. Please leave notes in the thread rather + than against this page. As PyPCAPKit reaches its *16k* lines of code and *800th* commit, I figure it -would be a better idea to record the project enchancement proposals here in +would be a better idea to record the project enhancement proposals here in the discussion thread. The proposals and/or notes will be documented and maintained here. @@ -21,188 +26,264 @@ Wish you enjoy **PyPCAPKit**!!! More Protocols, More!!! ----------------------- -.. note:: - - **SCTP** is now **done**. It is implemented as a first-class transport layer - protocol per :rfc:`9260`: the common header, all thirteen chunk types the - RFC defines, chunk parameters, error causes, and CRC32c checksum - verification. Chunk types, parameters and error causes that are registered - but not yet implemented fall through to the generic handlers rather than - failing the extraction. - - One note on how it differs from its siblings. The next layer is dispatched - on the DATA chunk's *payload protocol identifier* through - :func:`~pcapkit.foundation.registry.protocols.register_sctp`, not on port - numbers, so :func:`~pcapkit.foundation.registry.protocols.register_apptype` - deliberately does not fan out to it. - As you may have noticed, there are some protocol-named files under the ``NotImplemented`` folders. These protocols are what I planned to implement but not yet done. Namely, grouped by each TCP/IP layer and ordered by protocol name alphabetically, -* Link Layer: DSL, EAPOL, FDDI, ISDN, PPP -* Internet Layer: ECN, ICMP, ICMPv6, IGMP, NDP, Shim6 -* Transport Layer: DCCP, QUIC, RSVP +* Link Layer: DSL, EAPOL, FDDI, ISDN, NDP, PPP +* Internet Layer: ECN, ICMP, ICMPv6, IGMP, Shim6 +* Transport Layer: DCCP, RSVP * Application Layer: BGP, DHCP, DHCPv6, DNS, IMAP, LDAP, MQTT, NNTP, NTP, ONC/RPC, POP, RIP, RTP, SIP, SMTP, SNMP, SSH, Telnet, TLS/SSL, XMPP -**ESP** -- abandoned in the ``NotImplemented`` folder for years, because of -design flaws within PyPCAPKit at the time -- is now implemented, c.f. -:class:`~pcapkit.protocols.internet.esp.ESP`. It parses without keys, and -decrypts when a Security Association is supplied through the protocol keyed -:mod:`pcapkit.corekit.context` channel. What is still wanted there is wider -algorithm coverage: ChaCha20-Poly1305 [:rfc:`7634`], AES-CCM [:rfc:`4309`] and -AES-XCBC integrity [:rfc:`3566`] are not implemented, and neither are Extended -Sequence Numbers. +Each of those files is empty, so any one of them is a self-contained piece of +work: a schema class, a data class and a protocol class, as sketched in +`discussion #251 `__. +That thread asks for **NGAP** (5G, application layer), which is not on the list +above and has no stub, and the reply to it is the closest thing the project has +to a step-by-step guide for adding a protocol -- worth reading before starting +any of these. + +.. note:: -More over, :class:`~pcapkit.protocols.internet.mh.MH` requires some help to -implement all the *message data* types, you can find more information in the -specific file. + Two entries differ from the list in the discussion thread, and both are + corrections rather than progress. **NDP** is shown under the link layer + because that is where its stub actually lives, at + ``pcapkit/protocols/link/NotImplemented/ndp.py``; there is no + ``ndp.py`` under the internet layer. **QUIC** has been dropped because no + stub for it exists anywhere in the tree -- the thread lists it, but the file + was never created. + + **ESP** and **SCTP** have left the list because they are now implemented, + and neither left a stub behind. + +SCTP +~~~~ + +**Done.** :class:`~pcapkit.protocols.transport.sctp.SCTP` is a first-class +transport layer protocol implemented per :rfc:`9260`: the common header, all +thirteen chunk types the RFC defines, its eight chunk parameters, its thirteen +error causes, and CRC32c checksum verification. + +One note on how it differs from its siblings. The next layer is dispatched +on the DATA chunk's *payload protocol identifier* through +:func:`~pcapkit.foundation.registry.protocols.register_sctp`, not on port +numbers, so :func:`~pcapkit.foundation.registry.protocols.register_apptype` +deliberately does not fan out to it. + +What is still wanted is the registered-but-unimplemented type codes. IANA +registers considerably more than :rfc:`9260` defines, and the surplus falls +through to the generic handlers rather than failing the extraction -- so a +capture using one parses, but yields an opaque chunk instead of its fields. +As things stand that is 17 of the 30 registered chunk types, 24 of the 32 +chunk parameters and 10 of the 23 error causes; :doc:`pcapkit/const/sctp` +lists them all. + +ESP +~~~ + +**Done.** :class:`~pcapkit.protocols.internet.esp.ESP` -- abandoned in the +``NotImplemented`` folder for years, because of design flaws within PyPCAPKit +at the time -- now parses without keys, and decrypts when a Security +Association is supplied through the protocol keyed +:mod:`pcapkit.corekit.context` channel. + +What is still wanted there is wider algorithm coverage. The two enumerations +under :doc:`pcapkit/const/esp` carry every transform IANA has registered, but +only five encryption and five integrity algorithms are actually applied, as +listed by :data:`~pcapkit.protocols.internet.esp.CIPHER_SUITES` and +:data:`~pcapkit.protocols.internet.esp.INTEGRITY_SUITES`. Specifically not +implemented: ChaCha20-Poly1305 [:rfc:`7634`], AES-CCM [:rfc:`4309`], AES-XCBC +integrity [:rfc:`3566`], and Extended Sequence Numbers. + +Mobility Header +~~~~~~~~~~~~~~~ + +**Partly done**, and still the section of this page with the most work left in +it. :class:`~pcapkit.protocols.internet.mh.MH` has the FMIPv6 fast-handover +messages [:rfc:`5568`] -- Handover Initiate, Handover Acknowledge, FBU, FBack +and FNA -- along with the options they need. What remains is the rest of the +registry: + +* **10 of the 24 registered message data types**, namely Home Agent Switch, + Heartbeat, Binding Revocation, Localized Routing Initiation and + Acknowledgment, Update Notification and its Acknowledgement, Flow Binding, + Subscription Query and Subscription Response. +* **51 of the 71 registered options** -- broadly the PMIPv6, NEMO and + flow-binding block, including Home Network Prefix, Handoff Indicator, Access + Technology Type, Timestamp, GRE Key, Binding Identifier and the QoS options. +* **3 of the 4 CGA extensions**; only Multi-Prefix is implemented. + +Each of those falls through to a generic handler, so nothing breaks -- the +fields simply are not decoded. The ``# TODO`` markers in +``pcapkit/protocols/internet/mh.py`` sit at the exact dispatch tables that need +entries, and the file documents the shape each handler takes. + +PCAPNG Support +-------------- + +**Done.** The builtin default engine parses PCAP-NG files; +:class:`~pcapkit.protocols.misc.pcapng.PCAPNG` implements the format, with its +block and option enumerations under :doc:`pcapkit/const/pcapng`. This closes +the request in `#35 `__, which +the thread raised when only PCAP was supported. + +Maybe Even Faster? +------------------ + +**Still open.** Benchmarking put the builtin default engine at roughly 4x +Scapy and 10x DPKT, which is an acceptable price for what it decodes, but the +original proposal in the thread stands: fold consecutive ``_read_xxxxxx`` calls +into a single ``file.read`` so that the number of IO calls and the duplicated +:func:`struct.unpack` work both come down. + +Note that the parsing path has been rewritten since that was written. Protocols +no longer read fields inline; they declare a +:class:`~pcapkit.protocols.schema.schema.Schema` of field descriptors and let +:meth:`~pcapkit.protocols.schema.schema.Schema.unpack` drive it. The batching +idea still applies, but it belongs in the schema and field machinery now rather +than in each protocol's ``_read_`` methods, and the sketch in the thread no +longer maps onto the code. A measured benchmark showing where the time actually +goes would be the useful first contribution here. Logging Integration ------------------- -.. note:: - - Largely **done**. :mod:`pcapkit.utilities.logging` is no longer a single flat - logger with a hard-wired handler. It now provides: - - - a **logger hierarchy** rooted at ``pcapkit``, with every module logging - through its own child obtained from - :func:`~pcapkit.utilities.logging.get_logger`, so that a subtree such as - ``pcapkit.foundation.registry`` can be silenced independently of - ``pcapkit.foundation.extraction``; - - **library-safe defaults** -- importing :mod:`pcapkit` attaches only a - :class:`logging.NullHandler` and sets no level, leaving the destination and - verbosity to the application. :envvar:`PCAPKIT_DEVMODE` still bootstraps the - historical :obj:`sys.stderr` handler at :data:`logging.DEBUG`; - - a **runtime configuration API** -- - :func:`~pcapkit.utilities.logging.configure`, - :func:`~pcapkit.utilities.logging.reset` and - :func:`~pcapkit.utilities.logging.ensure_output` -- rather than a single - environment variable read once at import; - - **levels chosen deliberately**. Registration bookkeeping across - :mod:`pcapkit.foundation.registry` moved from ``info`` to ``debug``, since - a library announcing its own registry entries is not news to its consumer; - and the four :func:`print` calls that were marked - ``# pylint: disable=logging-fstring-interpolation`` are now real logger - calls; - - **``debug`` coverage of the extraction path** -- extractor construction, - engine selection and fallback, frame counts, cleanup, reassembly and - flow-tracing setup -- so that ``DEBUG`` explains what PyPCAPKit did with a - file without descending into per-field parsing. - - See :doc:`pcapkit/utilities/logging` for the configuration recipes, including - the one-line restore of the pre-existing :obj:`sys.stderr` output. - - What remains wanted is the two items called out there as deliberately out of - scope: :func:`pcapkit.utilities.warnings.warn` still double-reports every - warning through both :mod:`logging` and :mod:`warnings`, and - :class:`~pcapkit.utilities.warnings.BaseWarning` still mutates the global - warning filters with :func:`warnings.simplefilter`. - -Originally: as PyPCAPKit now has the :data:`pcapkit.utilities.logging.logger` in -place, I'm expecting to fully extend its functionality in the entire module. -Ideas and contributions are welcomed to integrate the logging system into -PyPCAPKit. +**Done.** :mod:`pcapkit.utilities.logging` is no longer a single flat logger +with a hard-wired handler. It now provides: + +- a **logger hierarchy** rooted at ``pcapkit``, with every module logging + through its own child obtained from + :func:`~pcapkit.utilities.logging.get_logger`, so that a subtree such as + ``pcapkit.foundation.registry`` can be silenced independently of + ``pcapkit.foundation.extraction``; +- **library-safe defaults** -- importing :mod:`pcapkit` attaches only a + :class:`logging.NullHandler` and sets no level, leaving the destination and + verbosity to the application. :envvar:`PCAPKIT_DEVMODE` still bootstraps the + historical :obj:`sys.stderr` handler at :data:`logging.DEBUG`; +- a **runtime configuration API** -- + :func:`~pcapkit.utilities.logging.configure`, + :func:`~pcapkit.utilities.logging.reset` and + :func:`~pcapkit.utilities.logging.ensure_output` -- rather than a single + environment variable read once at import; +- **levels chosen deliberately**. Registration bookkeeping across + :mod:`pcapkit.foundation.registry` moved from ``info`` to ``debug``, since + a library announcing its own registry entries is not news to its consumer; + and the four :func:`print` calls that were marked + ``# pylint: disable=logging-fstring-interpolation`` are now real logger + calls; +- **``debug`` coverage of the extraction path** -- extractor construction, + engine selection and fallback, frame counts, cleanup, reassembly and + flow-tracing setup -- so that ``DEBUG`` explains what PyPCAPKit did with a + file without descending into per-field parsing. + +See :doc:`pcapkit/utilities/logging` for the configuration recipes, including +the one-line restore of the pre-existing :obj:`sys.stderr` output. + +One item remains wanted, called out there as deliberately out of scope: +:func:`pcapkit.utilities.warnings.warn` still reports every warning twice, once +through :mod:`logging` and once through :mod:`warnings`, so an application that +has routed :mod:`warnings` into :mod:`logging` sees each one of them twice. New Engines ----------- -.. note:: - - **Done**, for both candidates. ``engine='pypcapfile'`` selects - :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` and - ``engine='pypcap'`` selects :class:`pcapkit.foundation.engines.pypcap.PyPCAP`; - each has a matching :mod:`pcapkit.toolkit` module - (:mod:`pcapkit.toolkit.pypcapfile`, :mod:`pcapkit.toolkit.pypcap`), a - ``pyproject.toml`` extra (``PyPCAPFile``, which ``all`` includes, and - ``PyPCAP``, which it deliberately does not -- see below), docs - under :doc:`pcapkit/foundation/engines/index`, and tests under - ``tests/foundation/engines/`` and ``tests/toolkit/``. Both were verified - end-to-end against the sample captures: each agrees with the ``default`` engine - on frame count, per-record capture length, timestamp and Ethernet header. - - Neither library, though, is usable straight from PyPI on a current Python, and - both of the following are worth knowing before reaching for them: - - * **pypcapfile** -- the released 0.12.0 imports the ``imp`` module, removed in - Python 3.12. Precisely, ``pcapfile/linklayer.py`` imports it at module scope - and ``pcapfile.savefile`` imports ``linklayer``, so while a bare - ``import pcapfile`` still succeeds on 3.12+, the two modules the engine - actually needs raise :exc:`ModuleNotFoundError`. Upstream ``master`` (0.12.1, - unreleased) has fixed this, and that is what the engine was verified against. - The extra therefore installs a version that only works on Python 3.10 and - 3.11 until 0.12.1 is published. - * **pypcap** -- the 1.3.0 sdist (the only distribution; there has been no wheel - since Python 2.7) ships a ``pcap.c`` pre-generated by Cython 0.29.32 and - compiles it verbatim -- its :file:`setup.py` never invokes Cython. That - generated C does not compile against the Python 3.12+ C API, and the cause is - three independent CPython removals rather than one: ``ob_digit`` and - ``PyThreadState.curexc_traceback`` went in 3.12, ``_PyLong_AsByteArray`` - gained a sixth parameter in 3.13, and ``PyDictObject.ma_version_tag`` went in - 3.14. **The ceiling is therefore Python 3.11**, measured: given - :program:`libpcap`, the shipped ``pcap.c`` builds unchanged on 3.10 and 3.11 - and fails on 3.12 and 3.14. Regenerating it needs Cython **3.0+** -- no - 0.29.x release, including the last, emits 3.12-compatible C. - - Separately, its :file:`setup.py` consults neither ``CFLAGS``/``LDFLAGS`` nor - :program:`pkg-config`; it searches a fixed prefix list (:file:`/usr`, - ``sys.prefix``, :file:`/opt/libpcap*`, :file:`../libpcap*`, - :file:`../wpdpack*`, the macOS SDKs). ``/opt/homebrew`` is absent from that - list, and Homebrew's ``libpcap`` is ``keg_only :provided_by_macos`` so it is - never symlinked into a searched prefix either -- meaning - ``brew install libpcap`` on the arm64 macOS runners would **not** have fixed - the original failure, on either count. It builds once :program:`libpcap` is - visible under ``sys.prefix`` *and* the interpreter is 3.11 or older. - - That is a packaging problem upstream rather than an engine problem -- and - upstream is unmaintained, with no code commit since the 1.3.0 release and its - Python 3.12 issue (`pynetwork/pypcap#116 - `__) open and uncommented - since May 2024. It does mean ``pip install pypcapkit[PyPCAP]`` can fail to - build. Since there is no wheel to fall back on, the extra is kept **out of** - ``all``: otherwise ``pip install pypcapkit[all]`` would demand a compiler and - the libpcap development files from every user, and it broke the docs, conda - and release workflows -- all of which install ``.[all]`` -- on the macOS - runner, where :file:`pcap.h` is present but no ``libpcap.dylib`` is. - - There is a way out that has not been adopted yet: `pcap-ct - `__ re-implements the ``pypcap`` API in - pure Python over :mod:`ctypes` and depends on `libpcap - `__, which bundles prebuilt - ``libpcap`` binaries for Linux, macOS and Windows. Both ship - ``py3-none-any`` wheels, so neither needs a compiler, :file:`pcap.h` or a - system ``libpcap``. Swapping the extra to ``pcap-ct`` was verified to drive - :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` on **Python 3.14**, - agreeing with the ``default`` engine on frame count. Both are still beta - releases and ``pcap-ct`` targets the ``pypcap`` 1.2.3 API, which is why this - is recorded as an option rather than done. - - Both engines support less than the ``default`` engine does, deliberately and - noisily: `pypcap`_ performs no protocol dissection, so it disables reassembly - *and* flow tracing; `pypcapfile`_ has no IPv6 decoder, so it disables IPv6 - reassembly while keeping IPv4 and TCP. Each gap is announced through an - :class:`~pcapkit.utilities.warnings.AttributeWarning` or an outright exception - rather than by silently returning nothing -- - :doc:`pcapkit/foundation/engines/index` tabulates them. - - The engine interface has since been refactored, so this no longer means adding - handler methods to :class:`~pcapkit.foundation.extraction.Extractor`. A new - engine subclasses :class:`pcapkit.foundation.engines.engine.Engine` and - implements just two methods, :meth:`~pcapkit.foundation.engines.engine.Engine.run` - and :meth:`~pcapkit.foundation.engines.engine.Engine.read_frame`; subclassing - registers it automatically. See :doc:`ext` for a worked example. What does - still apply is the unified auxiliary tools in :mod:`pcapkit.toolkit`, where - each engine has a matching module. - -Originally: although PyPCAPKit already has support for some popular PCAP parsing -libraries, I'm expecting to extend the list of supported engines furthermore. The -candidate engines include: - -- `pypcap `__ -- `pypcapfile `__ +**Done, for both candidates.** ``engine='pypcapfile'`` selects +:class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` and +``engine='pypcap'`` selects :class:`pcapkit.foundation.engines.pypcap.PyPCAP`; +each has a matching :mod:`pcapkit.toolkit` module +(:mod:`pcapkit.toolkit.pypcapfile`, :mod:`pcapkit.toolkit.pypcap`), a +``pyproject.toml`` extra (``PyPCAPFile``, which ``all`` includes, and +``PyPCAP``, which it deliberately does not -- see below), docs +under :doc:`pcapkit/foundation/engines/index`, and tests under +``tests/foundation/engines/`` and ``tests/toolkit/``. Both were verified +end-to-end against the sample captures: each agrees with the ``default`` engine +on frame count, per-record capture length, timestamp and Ethernet header. + +Neither library, though, is usable straight from PyPI on a current Python, and +both of the following are worth knowing before reaching for them: + +* **pypcapfile** -- the released 0.12.0 imports the ``imp`` module, removed in + Python 3.12. Precisely, ``pcapfile/linklayer.py`` imports it at module scope + and ``pcapfile.savefile`` imports ``linklayer``, so while a bare + ``import pcapfile`` still succeeds on 3.12+, the two modules the engine + actually needs raise :exc:`ModuleNotFoundError`. Upstream ``master`` (0.12.1, + unreleased) has fixed this, and that is what the engine was verified against. + The extra therefore installs a version that only works on Python 3.10 and + 3.11 until 0.12.1 is published. +* **pypcap** -- the 1.3.0 sdist (the only distribution; there has been no wheel + since Python 2.7) ships a ``pcap.c`` pre-generated by Cython 0.29.32 and + compiles it verbatim -- its :file:`setup.py` never invokes Cython. That + generated C does not compile against the Python 3.12+ C API, and the cause is + three independent CPython removals rather than one: ``ob_digit`` and + ``PyThreadState.curexc_traceback`` went in 3.12, ``_PyLong_AsByteArray`` + gained a sixth parameter in 3.13, and ``PyDictObject.ma_version_tag`` went in + 3.14. **The ceiling is therefore Python 3.11**, measured: given + :program:`libpcap`, the shipped ``pcap.c`` builds unchanged on 3.10 and 3.11 + and fails on 3.12 and 3.14. Regenerating it needs Cython **3.0+** -- no + 0.29.x release, including the last, emits 3.12-compatible C. + + Separately, its :file:`setup.py` consults neither ``CFLAGS``/``LDFLAGS`` nor + :program:`pkg-config`; it searches a fixed prefix list (:file:`/usr`, + ``sys.prefix``, :file:`/opt/libpcap*`, :file:`../libpcap*`, + :file:`../wpdpack*`, the macOS SDKs). ``/opt/homebrew`` is absent from that + list, and Homebrew's ``libpcap`` is ``keg_only :provided_by_macos`` so it is + never symlinked into a searched prefix either -- meaning + ``brew install libpcap`` on the arm64 macOS runners would **not** have fixed + the original failure, on either count. It builds once :program:`libpcap` is + visible under ``sys.prefix`` *and* the interpreter is 3.11 or older. + + That is a packaging problem upstream rather than an engine problem -- and + upstream is unmaintained, with no code commit since the 1.3.0 release and its + Python 3.12 issue (`pynetwork/pypcap#116 + `__) open and uncommented + since May 2024. It does mean ``pip install pypcapkit[PyPCAP]`` can fail to + build. Since there is no wheel to fall back on, the extra is kept **out of** + ``all``: otherwise ``pip install pypcapkit[all]`` would demand a compiler and + the libpcap development files from every user, and it broke the docs, conda + and release workflows -- all of which install ``.[all]`` -- on the macOS + runner, where :file:`pcap.h` is present but no ``libpcap.dylib`` is. + + This is now solved, though not by changing the ``PyPCAP`` extra. `pcap-ct + `__ re-implements the ``pypcap`` API in pure + Python over :mod:`ctypes`, and it is wired up as a **separate engine**, + :class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT`, selected with + ``engine='pcap_ct'`` and installed with ``pip install pypcapkit[PCAP_CT]``. It + works on every supported interpreter, verified on 3.10 and 3.14. + + A separate engine rather than a second backend because the two are **mutually + exclusive**: both distributions own the import name ``pcap``, and with both + installed ``pcap-ct`` wins the import while upstream's extension module is + shadowed and unreachable. Each engine detects which backend it actually got and + says so, rather than one ``engine=`` string silently meaning two different + implementations. + + Two caveats remain. Both distributions are still beta releases, and ``pcap-ct`` + targets the ``pypcap`` 1.2.3 API -- within what this engine uses, but worth + knowing. And despite the `libpcap `__ wheel + bundling prebuilt binaries, a runtime ``libpcap`` shared library is still + required: that wheel's published configuration sets ``LIBPCAP = None``, which + sends the loader to ``find_library("pcap")``. No compiler and no + :file:`pcap.h` are needed; a ``libpcap.so.1`` is. + +Both engines support less than the ``default`` engine does, deliberately and +noisily: `pypcap`_ performs no protocol dissection, so it disables reassembly +*and* flow tracing; `pypcapfile`_ has no IPv6 decoder, so it disables IPv6 +reassembly while keeping IPv4 and TCP. Each gap is announced through an +:class:`~pcapkit.utilities.warnings.AttributeWarning` or an outright exception +rather than by silently returning nothing -- +:doc:`pcapkit/foundation/engines/index` tabulates them. + +Adding a further engine no longer means adding handler methods to +:class:`~pcapkit.foundation.extraction.Extractor`, as the thread describes: the +engine interface has been refactored since. A new engine subclasses +:class:`pcapkit.foundation.engines.engine.Engine` and implements just two +methods, :meth:`~pcapkit.foundation.engines.engine.Engine.run` +and :meth:`~pcapkit.foundation.engines.engine.Engine.read_frame`; subclassing +registers it automatically. See :doc:`ext` for a worked example. What does +still apply is the unified auxiliary tools in :mod:`pcapkit.toolkit`, where +each engine has a matching module. .. _pypcap: https://github.com/pynetwork/pypcap .. _pypcapfile: https://github.com/kisom/pypcapfile @@ -210,21 +291,27 @@ candidate engines include: Test Cases ---------- -.. note:: - - Largely **done**. There is now a systematic unit test suite under ``tests/`` - (84 modules), bundled with the distribution, and it runs in CI against Python - 3.10 through 3.14 (see ``.github/workflows/unit-tests.yml``). The sample - captures the runtime, regression and integration tiers read are not tracked in - git, so ``examples/generators/make_samples.py`` (``make samples``) rebuilds them - from source. - - What remains wanted is coverage rather than infrastructure: the protocols and - the registered-but-unhandled type codes listed above have no tests because - they have no implementation yet. - -Originally: PyPCAPKit still does not have a systematic testing suite to be -bundled with it. The only test cases I have worked out are those in the -``/tests`` folder - mostly functional tests. As PyPCAPKit is growing bigger and -bigger, a comprehensive test suite is coming much more of demand for a more -reliable development process. +**Largely done.** There is now a systematic test suite under ``tests/`` -- 86 +modules matching ``test_*.py`` -- and it runs in CI against Python 3.10 +through 3.14, plus an allowed-to-fail 3.15 leg, per +``.github/workflows/unit-tests.yml``. + +The suite is split by what a test needs rather than by what it covers, and +``tests/_tiers.py`` enforces the split. The **unit** tier may read only +captures committed to the repository; the **fixture-dependent** tier -- which +is ``tests/integration/`` together with every ``*_runtime.py`` and +``*_regression.py`` module -- may also read the generated sample captures. +Those samples are not tracked in git, so +``examples/generators/make_samples.py`` (``make samples``) rebuilds them from +source, and the tier guard raises rather than letting a unit-tier module +quietly depend on a file that may not exist. + +Two things are still wanted: + +* **Coverage, rather than infrastructure.** The protocols and the + registered-but-unhandled type codes listed above have no tests because they + have no implementation yet. +* **Shipping the suite**, which was part of the original ask and is not done. + ``tests`` is excluded from the wheel by ``[tool.setuptools.packages.find]`` + in ``pyproject.toml``, so while the sdist carries the files, an installed + PyPCAPKit has no ``tests`` package and the suite cannot be run against it. diff --git a/pyproject.toml b/pyproject.toml index c8779cf2f0..7a206f8417 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -201,7 +201,8 @@ all = [ ] docs = [ "Sphinx>=6.1.3", "furo", - "sphinx-autodoc-typehints", "sphinx-opengraph", "sphinx-copybutton", + "sphinx-autodoc-typehints", "sphinxext-opengraph", "sphinx-copybutton", + "sphinxcontrib-mermaid", "typing-extensions", "mypy-extensions", ] test = [ From 6a62710be99da2ae06a0079c7a7ea0c1a0974ccb Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 16 Sep 2026 00:30:49 -0400 Subject: [PATCH 2/4] docs: stop the Help Wanted intro contradicting itself Copilot's review on #408 was right. The intro said proposals are recorded "here in the discussion thread" and then "maintained here", conflating the page with the thread -- and once this branch added an ``important`` block that carefully distinguishes the two, the paragraph directly contradicted the note three lines above it. Reworded so "the discussion thread" and "this page" are always distinct: the proposals were *raised* in the thread, and the page is where they are *kept up to date*. Also fixed the tense, since 16k lines and 800 commits are long past, and pointed the questions sentence at real links rather than a bare "this thread" that had no referent on a rendered page. --- docs/source/pep.rst | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/source/pep.rst b/docs/source/pep.rst index b97259ece2..c3e22f1f6a 100644 --- a/docs/source/pep.rst +++ b/docs/source/pep.rst @@ -12,14 +12,15 @@ Help Wanted genuinely still open is asked for. Please leave notes in the thread rather than against this page. -As PyPCAPKit reaches its *16k* lines of code and *800th* commit, I figure it -would be a better idea to record the project enhancement proposals here in -the discussion thread. The proposals and/or notes will be documented and -maintained here. - -Pull requests for the existing proposals and any new ideas are highly welcomed -and encouraged. Should you have any questions, please leave a note either in -this thread or under the `Q&A category discussions `__. +As PyPCAPKit reached its *16k* lines of code and *800th* commit, it seemed +better to record the project's enhancement proposals somewhere durable than to +leave them scattered. They were raised in the discussion thread, and this page +is where they are kept up to date. + +Pull requests for anything still open, and new ideas of your own, are very +welcome. For questions, leave a note in the `discussion thread +`__ or under the `Q&A +category `__. Wish you enjoy **PyPCAPKit**!!! From 984d5844342643755ebc9eadce3048d6641526ea Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 16 Sep 2026 01:41:16 -0400 Subject: [PATCH 3/4] docs: put esp and context back on the house convention, without automodule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit These two were the **only two of 128 pages** using `automodule`; every other page declares `.. module::` and then lists its members explicitly. Converted to match: the descriptive prose is carried in the `.rst`, and each class, function and data item gets its own entry. That also removes the `:no-index:` workaround this branch had added, for a better reason than suppressing a symptom -- the duplicate-object warnings were *created by* `automodule`, not by two `.. module::` directives coexisting. Which answers the question the workaround raised. 365 modules carry `.. module::` in their docstring and 122 of 128 pages carry one too, and that is not a duplication: **without `automodule`, Sphinx never parses the module docstring at all**, so the docstring's directive is inert text in a `.py` file and the page's is the only registration. `automodule` was what made them collide. Two traps the conversion had to handle, either of which would have silently degraded the page: * `esp.py`'s docstring defines the `|cryptography|` substitution and its link target, and **eight rendered member docstrings reference them** -- `load_cryptography`, `_CRYPTO`, `CipherSuite.requires_cryptography`, `ESPStatus.UNSUPPORTED`, `SecurityAssociation.decrypt`/`encrypt`. `automodule` had been supplying those definitions invisibly; dropping it without carrying them into the page would have broken every one. * Both pages were rendering their title **twice**, because `automodule` emitted the docstring's own title as a second heading. Now gone. Verified rather than assumed. Full builds before and after: 267 total, 263 warning lines, and a `diff` of the sorted warning sets is byte-for-byte identical -- zero new, zero removed. Neither page had a warning attributed to it either way. Member coverage checked by grepping every `id="pcapkit…"` out of the built HTML: `context.html` 13 -> 14, `esp.html` 74 -> 75, **nothing lost**; the additions are `_CT` and `_CRYPTO`, which `automodule` had skipped. The rendered visible text was diffed too: every sentence, table row and footnote survives. --- docs/source/pcapkit/corekit/context.rst | 64 ++++++- .../source/pcapkit/protocols/internet/esp.rst | 178 ++++++++++++++++-- 2 files changed, 219 insertions(+), 23 deletions(-) diff --git a/docs/source/pcapkit/corekit/context.rst b/docs/source/pcapkit/corekit/context.rst index e8b2954691..6b7fad0b6b 100644 --- a/docs/source/pcapkit/corekit/context.rst +++ b/docs/source/pcapkit/corekit/context.rst @@ -1,15 +1,57 @@ Parsing Context =============== -.. Unlike its sibling pages, this one renders the module docstring through - ``automodule`` rather than repeating it as prose. That docstring already - carries its own ``.. module::`` directive -- as every module under - ``pcapkit`` does -- so there must be no second one here, and ``automodule`` - itself must not register a third: hence ``:no-index:``. +.. module:: pcapkit.corekit.context -.. automodule:: pcapkit.corekit.context - :no-members: - :no-index: +:mod:`pcapkit.corekit.context` provides a *protocol keyed* channel for +caller supplied information that a protocol needs in order to parse a +packet, but that is **not** carried on the wire. + +Most protocols are self describing -- every length, offset and type that +:mod:`pcapkit` needs to walk a packet is present in the packet itself. A +few are not. :class:`~pcapkit.protocols.internet.esp.ESP` is the +motivating example: :rfc:`4303` deliberately leaves the payload length, +the position of the ``Pad Length`` / ``Next Header`` trailer and the +length of the ``Integrity Check Value`` to be derived from the Security +Association (SA), which is negotiated out of band and is therefore +knowable only to the caller. + +Rather than adding protocol specific keyword arguments to +:class:`~pcapkit.foundation.extraction.Extractor`, such information is passed +as a :class:`~pcapkit.corekit.context.ContextRegistry` -- a mapping of +protocol index ID (c.f. :meth:`Protocol.id +`) to a +:class:`~pcapkit.corekit.context.ProtocolContext` instance. The registry is +handed to :class:`~pcapkit.foundation.extraction.Extractor` once, and is then +propagated down the protocol stack by +:meth:`Protocol._import_next_layer `, +so that a protocol nested arbitrarily deep can reach it through +:meth:`Protocol._get_context `. + +Decoding an ESP tunnel end to end: + +.. code-block:: python + + >>> import pcapkit + >>> from pcapkit.protocols.internet.esp import (Cipher, ESPContext, + ... Integrity, SecurityAssociation) + >>> sa = SecurityAssociation( + ... spi=0x4321, + ... encryption=Cipher.AES_CBC, + ... encryption_key=bytes.fromhex('90d382b410eeba7ad938c46cec1a82bf'), + ... ) + >>> extraction = pcapkit.extract('esp.pcap', context=ESPContext(sa)) + +.. important:: + + A context object frequently holds secrets -- ESP encryption and + integrity keys, for instance. Contexts are therefore held as plain + instance attributes on the protocol object and are **never** written + into the protocol's data model, which is the only thing that reaches + :meth:`Info.to_dict ` and, + from there, the output dumpers. Implementations of + :class:`~pcapkit.corekit.context.ProtocolContext` are expected to keep + secrets out of their :meth:`~object.__repr__` as well. .. autoclass:: pcapkit.corekit.context.ProtocolContext :no-members: @@ -32,3 +74,9 @@ Parsing Context .. automethod:: __contains__ .. automethod:: __bool__ .. automethod:: __repr__ + +Type Variables +-------------- + +.. data:: pcapkit.corekit.context._CT + :type: pcapkit.corekit.context.ProtocolContext diff --git a/docs/source/pcapkit/protocols/internet/esp.rst b/docs/source/pcapkit/protocols/internet/esp.rst index 8c4129aa57..959213ac7a 100644 --- a/docs/source/pcapkit/protocols/internet/esp.rst +++ b/docs/source/pcapkit/protocols/internet/esp.rst @@ -1,15 +1,41 @@ ESP - Encapsulating Security Payload ==================================== -.. Unlike its sibling pages, this one renders the module docstring through - ``automodule`` rather than repeating it as prose. That docstring already - carries its own ``.. module::`` directive -- as every module under - ``pcapkit`` does -- so there must be no second one here, and ``automodule`` - itself must not register a third: hence ``:no-index:``. - -.. automodule:: pcapkit.protocols.internet.esp - :no-members: - :no-index: +.. module:: pcapkit.protocols.internet.esp + +:mod:`pcapkit.protocols.internet.esp` contains +:class:`~pcapkit.protocols.internet.esp.ESP` only, +which implements extractor for Encapsulating +Security Payload (ESP) [*]_, whose structure is +described as below: + +======= ========= ===================== ============================================== +Octets Bits Name Description +======= ========= ===================== ============================================== + 0 0 ``esp.spi`` Security Parameters Index (SPI) + 4 32 ``esp.seq`` Sequence Number + 8 64 ``esp.payload_data`` Payload Data (variable, encrypted) + ? ? Padding (0-255 bytes, encrypted) + ? ? ``esp.pad_len`` Pad Length (encrypted) + ? ? ``esp.next`` Next Header (encrypted) + ? ? ``esp.icv`` Integrity Check Value (ICV, variable) +======= ========= ===================== ============================================== + +Unlike every other protocol in :mod:`pcapkit`, ESP is **not** self +describing. :rfc:`4303` places the ``Pad Length`` and ``Next Header`` +fields *inside* the ciphertext, and leaves the length of the ``Integrity +Check Value`` to be determined by the Security Association (SA), which is +negotiated out of band. Therefore: + +* **Without** SA context, :class:`~pcapkit.protocols.internet.esp.ESP` + parses the ``SPI`` and ``Sequence Number``, reports the remainder as an + opaque encrypted payload, and says so through + :attr:`esp.status `. + It does *not* guess at the trailer, and it does not raise. +* **With** SA context, :class:`~pcapkit.protocols.internet.esp.ESP` splits + off the ICV, verifies integrity, decrypts, strips the padding using + ``Pad Length``, and dispatches the recovered plaintext to the next layer + using ``Next Header`` -- so an ESP tunnelled TCP segment decodes as TCP. .. autoclass:: pcapkit.protocols.internet.esp.ESP :no-members: @@ -34,8 +60,24 @@ ESP - Encapsulating Security Payload Security Associations --------------------- -SA context is supplied through the generic, protocol keyed channel of -:mod:`pcapkit.corekit.context`. +SA context is supplied through the generic, protocol keyed context channel +in :mod:`pcapkit.corekit.context`: + +.. code-block:: python + + import pcapkit + from pcapkit.protocols.internet.esp import (Cipher, ESPContext, Integrity, + SecurityAssociation) + + sa = SecurityAssociation( + spi=0x4321, + encryption=Cipher.AES_CBC, + encryption_key=bytes.fromhex('90d382b410eeba7ad938c46cec1a82bf'), + integrity=Integrity.HMAC_SHA2_256_128, + integrity_key=bytes.fromhex('00' * 32), + destination='192.168.123.100', # optional, disambiguates several tunnels + ) + extraction = pcapkit.extract('esp.pcap', context=ESPContext(sa)) .. autoclass:: pcapkit.protocols.internet.esp.SecurityAssociation :no-members: @@ -70,19 +112,102 @@ SA context is supplied through the generic, protocol keyed channel of Algorithm Registries -------------------- -The algorithm enumerations are the IANA IKEv2 transform ID registries, -generated into :mod:`pcapkit.const.esp` and re-exported here for convenience: +ESP has no algorithm registry of its own -- an SA's algorithms are negotiated +by IKEv2 -- so :class:`Cipher ` and +:class:`Integrity ` are generated from +the IKEv2 *transform ID* sub-registries and enumerate everything **IANA has +registered**: 3DES, AES-CTR, the AES-CCM and Camellia families, +ChaCha20-Poly1305, the implicit IV variants of :rfc:`8750`, the :rfc:`9227` +MGM suites, and the transforms long since deprecated. + +The enumerations themselves are generated into :mod:`pcapkit.const.esp` and +re-exported here for convenience: :class:`Cipher ` is :class:`pcapkit.const.esp.cipher.Cipher` and :class:`Integrity ` is :class:`pcapkit.const.esp.integrity.Integrity`. +Both enumerations additionally carry each transform's prefix-stripped spelling +as an alias, since that is how ESP and :rfc:`8221` name the algorithms, so +:attr:`Cipher.AES_CBC ` and +:attr:`Cipher.ENCR_AES_CBC ` +are the same member. The registry spells the "no integrity algorithm" +transform ``NONE`` rather than ``AUTH_NONE``, and :attr:`Integrity.NONE +` follows it. + Algorithm Support ----------------- A registry enumerates what IANA assigned an ID to, which is far more than -:mod:`pcapkit` implements. The tables below are the authority on what an SA may -actually name, and the two ``get`` methods refuse anything outside them. +:mod:`pcapkit` implements. **Registration is not support.** A member of either +enumeration says only that IANA assigned the transform an ID; what +:mod:`pcapkit` can actually apply is the separate, explicit +:data:`~pcapkit.protocols.internet.esp.CIPHER_SUITES` and +:data:`~pcapkit.protocols.internet.esp.INTEGRITY_SUITES` tables, and +:meth:`CipherSuite.get ` / +:meth:`IntegritySuite.get ` +refuse anything outside them rather than half-working: + +.. code-block:: python + + >>> Cipher.get('ENCR_3DES') # registered, so the enum has it + + >>> CipherSuite.get('ENCR_3DES') # but ESP cannot apply it + Traceback (most recent call last): + ... + pcapkit.utilities.exceptions.ProtocolError: unsupported ESP encryption + algorithm: ENCR_3DES; pcapkit implements ENCR_NULL, ENCR_AES_CBC, + ENCR_AES_GCM_8, ENCR_AES_GCM_12, ENCR_AES_GCM_16 + +Decryption requires the optional |cryptography|_ dependency +(``pip install pypcapkit[crypto]``). :mod:`pcapkit` imports and works +without it; an SA that names an AES suite simply degrades to the opaque +payload path, with a warning. + +.. |cryptography| replace:: ``cryptography`` +.. _cryptography: https://cryptography.io + +The supported set is anchored on the *mandatory to implement* algorithms of +:rfc:`8221`. The tables below are the authority on what an SA may actually +name; the rows marked ``yes`` are exactly the keys of +:data:`~pcapkit.protocols.internet.esp.CIPHER_SUITES`, and everything else the +registry lists is enumerated and rejected. + +============================ =================== ============ ================================== +Encryption :rfc:`8221` status Implemented Notes +============================ =================== ============ ================================== +``ENCR_NULL`` MUST yes :rfc:`2410`; needs no ``cryptography`` +``ENCR_AES_CBC`` MUST yes :rfc:`3602`; 128/192/256-bit keys +``ENCR_AES_GCM_16`` MUST yes :rfc:`4106`; 8-octet explicit IV +``ENCR_AES_GCM_8`` -- yes :rfc:`4106`, 8-octet ICV +``ENCR_AES_GCM_12`` -- yes :rfc:`4106`, 12-octet ICV +``ENCR_AES_CCM_8`` SHOULD **no** registered, not implemented +``ENCR_CHACHA20_POLY1305`` SHOULD **no** registered, not implemented +``ENCR_3DES`` SHOULD NOT **no** registered, deliberately omitted +DES, Blowfish, 3IDEA MUST NOT **no** registered, deliberately omitted +============================ =================== ============ ================================== + +"DES, Blowfish, 3IDEA" above covers ``ENCR_DES``, ``ENCR_DES_IV64``, +``ENCR_DES_IV32``, ``ENCR_BLOWFISH`` and ``ENCR_3IDEA``. + +Likewise, the rows marked ``yes`` below are exactly the keys of +:data:`~pcapkit.protocols.internet.esp.INTEGRITY_SUITES`: + +============================ =================== ============ ================================== +Integrity :rfc:`8221` status Implemented Notes +============================ =================== ============ ================================== +``NONE`` MUST (AEAD only) yes for AEAD suites +``AUTH_HMAC_SHA2_256_128`` MUST yes :rfc:`4868` +``AUTH_HMAC_SHA2_512_256`` SHOULD yes :rfc:`4868` +``AUTH_HMAC_SHA2_384_192`` -- yes :rfc:`4868` +``AUTH_HMAC_SHA1_96`` MUST- yes :rfc:`2404`; still widely captured +``AUTH_AES_XCBC_96`` SHOULD / MAY **no** registered, not implemented +``AUTH_AES_*_GMAC`` MAY **no** registered, not implemented +MD5, DES-MAC, KPDK-MD5 MUST NOT **no** registered, deliberately omitted +============================ =================== ============ ================================== + +"MD5, DES-MAC, KPDK-MD5" above covers ``AUTH_HMAC_MD5_96``, +``AUTH_HMAC_MD5_128``, ``AUTH_DES_MAC`` and ``AUTH_KPDK_MD5``. .. autoclass:: pcapkit.protocols.internet.esp.CipherSuite :members: @@ -100,6 +225,23 @@ actually name, and the two ``get`` methods refuse anything outside them. .. autofunction:: pcapkit.protocols.internet.esp._resolve +Known Limitations +----------------- + +* **Extended Sequence Numbers (ESN,** :rfc:`4303` **§2.2.1) are not + supported.** The high-order 32 bits of an ESN are not transmitted, and a + stateless parser cannot recover them; they are required both for the ICV + computation and for the AEAD associated data. An ESN protected packet + therefore fails the integrity check *cleanly* rather than being decoded. +* **Traffic Flow Confidentiality (TFC) padding (§2.4) is not detected.** + TFC padding is indistinguishable from real payload without inspecting the + inner protocol's own length field, so it is handed to the next layer as + part of the plaintext. +* **Anti-replay is not performed.** :mod:`pcapkit` is an analyser, not a + receiver; the sequence number is reported, never checked. +* The ICV is *verified* but a failure is reported rather than raised, so + that one bad packet does not abort a capture. + Processing Status ----------------- @@ -109,6 +251,8 @@ Processing Status .. autofunction:: pcapkit.protocols.internet.esp.load_cryptography +.. autodata:: pcapkit.protocols.internet.esp._CRYPTO + Header Schemas -------------- @@ -126,3 +270,7 @@ Data Models .. autoclass:: pcapkit.protocols.data.internet.esp.ESP :members: :show-inheritance: + +.. rubric:: Footnotes + +.. [*] https://en.wikipedia.org/wiki/IPsec From 873c869c99e927e571b7b6774e80afd225eee704 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 16 Sep 2026 09:19:47 -0400 Subject: [PATCH 4/4] docs: correct the test-module count, and frame shipping the suite as a decision Two corrections to the Test Cases section, both from measurement. The module count said 86; `find tests -name 'test_*.py'` counts **91**. The number has been wrong twice now (84 before this branch, 86 after), which is an argument for not quoting it at all -- but it is useful context for a contributor, so it is corrected rather than dropped. "Shipping the suite ... is not done" framed a deliberate choice as an omission. Built both artefacts to settle it: the **sdist carries all 91 modules**, the **wheel carries none**. So a distribution packager building from source has them, and `pip install pypcapkit` does not. Keeping the wheel lean is the right call, and the reason is stronger than "wheels don't usually ship tests": the suite *could not run* from an installed package anyway. The generated sample captures are not shipped, and `tests/_tiers.py` resolves paths from a repository root that an installed package does not have. Anyone who wants to run the tests wants the repository. Reworded to say that, rather than leaving a standing invitation to "fix" it by stuffing tests into the wheel. That left a single remaining item under a "Two things are still wanted" heading, so the list became a sentence. --- docs/source/pep.rst | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/source/pep.rst b/docs/source/pep.rst index c3e22f1f6a..a9cf98b541 100644 --- a/docs/source/pep.rst +++ b/docs/source/pep.rst @@ -292,7 +292,7 @@ each engine has a matching module. Test Cases ---------- -**Largely done.** There is now a systematic test suite under ``tests/`` -- 86 +**Largely done.** There is now a systematic test suite under ``tests/`` -- 91 modules matching ``test_*.py`` -- and it runs in CI against Python 3.10 through 3.14, plus an allowed-to-fail 3.15 leg, per ``.github/workflows/unit-tests.yml``. @@ -307,12 +307,15 @@ Those samples are not tracked in git, so source, and the tier guard raises rather than letting a unit-tier module quietly depend on a file that may not exist. -Two things are still wanted: - -* **Coverage, rather than infrastructure.** The protocols and the - registered-but-unhandled type codes listed above have no tests because they - have no implementation yet. -* **Shipping the suite**, which was part of the original ask and is not done. - ``tests`` is excluded from the wheel by ``[tool.setuptools.packages.find]`` - in ``pyproject.toml``, so while the sdist carries the files, an installed - PyPCAPKit has no ``tests`` package and the suite cannot be run against it. +What remains wanted is **coverage rather than infrastructure**: the protocols and +the registered-but-unhandled type codes listed above have no tests because they +have no implementation yet. + +The original ask also included **shipping the suite**, and that is now a +deliberate decision rather than an omission. ``tests`` is excluded from the wheel +by ``[tool.setuptools.packages.find]`` in ``pyproject.toml``; the sdist does carry +all 91 modules, so a distribution packager building from source has them. The +wheel stays lean because the suite could not run from an installed package +anyway: the generated sample captures are not shipped, and ``tests/_tiers.py`` +resolves paths from a repository root that an installed package does not have. +Anyone wanting to run the tests wants the repository, which is where they are.