From e0d8fa7d5c70fafb5d62142328cc7c6234d61965 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 16 Sep 2026 09:48:00 -0400 Subject: [PATCH 1/3] docs: fix 42 places where a page contradicts the code The house convention copies prose out of a module docstring into its `.rst` rather than pulling it in with `automodule`, which means a docstring edit does not follow into the page. A lot of source changed recently, so a lot of pages had drifted. This is the sweep for contradictions -- a page naming a parameter that does not exist, a default that changed, a limitation since fixed -- not for thinness. The one that was actively breaking the build: `engines/index.rst` declared `.. _libpcap:` **twice**, the only duplicate explicit target in the tree, so all three ```libpcap`_`` links were dead and the build emitted four errors. The C library reference now uses the page's own ``:manpage:`libpcap(3)``` idiom and the PyPI target is kept. Documentation for work that had none: `Engine.unsupported_reason` (new in #396, undocumented on `engines/engine.rst` while two other pages already cross-referenced it), the whole `_pcap_backend` module including `Probe` -- noted as a Mapping rather than a tuple, since it is an `Info` subclass now -- and `PyPCAPFile.PYTHON_CEILING`. The Scapy section still described the pre-#409 `scapy.sendrecv` import and omitted the `CryptographyDeprecationWarning` note that its docstring gained. Examples that could not have worked as written, all measured: `extract(strict=True)` raises `TypeError` -- the argument is `reasm_strict`; `python -m pypcapkit` has no such module, only the PyPI *name* is `pypcapkit` and the runnable one is `pcapkit`; three CLI transcripts relied on extension autocorrect that is gated behind `-a`, so they raised `FileNotFound`; and "export to a JSON file with no format specified" actually writes a *tree* dump into `out.json`, because `format=None` defaults to `'tree'`. The RFC 815 walkthrough on `reassembly/tcp.rst` had drifted from the implementation in six separate ways -- an exclusive `last` where the code is inclusive (`first + len - 1`), `ISN <- DSN` where a SYN spends a sequence number (`PSN = DSN + 1 if SYN`), an unconditional hole update the code guards with `if info.len > 0`, comparison operators that do not match, a `more_fragments` flag TCP does not have, and a four-tuple BUFID in the wrong order. And `reassembly/ip/ipv6.rst` still documented the reassembly key as the **flow label**, which is exactly the defect fixed in `3642dcaa9` -- the label is optional and routinely zero, so keying on it collapsed distinct datagrams. Nine docstrings were fixed rather than their pages, in the cases where the page was right and the docstring named something that does not exist -- `pcapkit.dumper.*`, `pcapkit.traceflow`, `pcapkit.protocols.null`, `Extrator`, `tractflow`, `TOS*` for `ToS*`, and `ipv6_opts.py` declaring itself as `hopopt`. Confined to that case deliberately, so the sweep does not leave a page and its docstring newly inconsistent by its own hand. Verified against a pristine baseline built from `git archive HEAD` in its own venv, after discarding a first attempt that had overlapped with in-flight edits: **278 warning lines and 4 errors before, 274 and 1 after** -- zero new, four eliminated, all four the `libpcap` family. The surviving error is pre-existing and untouched. All 2671 autodoc targets across all 128 pages resolve, and the rendered HTML was checked rather than assumed: `unsupported_reason`, `Probe`, `PCAP_CT.backend`, `PYTHON_CEILING` and `Backend Detection` all appear, and `index.html` now emits four working libpcap(3) links where all three were broken. --- docs/source/demo.rst | 23 ++-- docs/source/ext.rst | 2 + docs/source/pcapkit/const/ipv4.rst | 6 +- docs/source/pcapkit/dumpkit/index.rst | 6 +- docs/source/pcapkit/dumpkit/null.rst | 2 +- docs/source/pcapkit/dumpkit/pcap.rst | 2 +- .../pcapkit/foundation/engines/3rdparty.rst | 101 ++++++++++++++++++ .../pcapkit/foundation/engines/engine.rst | 2 + .../pcapkit/foundation/engines/index.rst | 10 +- docs/source/pcapkit/foundation/extraction.rst | 15 +-- docs/source/pcapkit/foundation/index.rst | 4 +- .../pcapkit/foundation/reassembly/index.rst | 6 +- .../pcapkit/foundation/reassembly/ip/ip.rst | 2 +- .../pcapkit/foundation/reassembly/ip/ipv4.rst | 10 +- .../pcapkit/foundation/reassembly/ip/ipv6.rst | 36 ++++--- .../foundation/reassembly/reassembly.rst | 2 +- .../pcapkit/foundation/reassembly/tcp.rst | 72 +++++++++---- .../pcapkit/foundation/traceflow/index.rst | 6 +- docs/source/pcapkit/interface/core.rst | 10 +- docs/source/pcapkit/protocols/index.rst | 3 +- docs/source/pcapkit/protocols/misc/null.rst | 6 +- docs/source/pcapkit/utilities/functools.rst | 2 +- docs/source/pcapkit/vendor/http.rst | 12 +-- docs/source/pcapkit/vendor/index.rst | 2 +- pcapkit/const/ipv4/__init__.py | 6 +- pcapkit/dumpkit/null.py | 2 +- pcapkit/dumpkit/pcap.py | 2 +- pcapkit/foundation/__init__.py | 4 +- pcapkit/foundation/traceflow/__init__.py | 2 +- pcapkit/protocols/internet/ipv6_opts.py | 2 +- pcapkit/protocols/misc/null.py | 6 +- pcapkit/vendor/esp/__init__.py | 9 +- pcapkit/vendor/ipv4/__init__.py | 6 +- 33 files changed, 272 insertions(+), 109 deletions(-) diff --git a/docs/source/demo.rst b/docs/source/demo.rst index ac43573dab..c7fb9580f7 100644 --- a/docs/source/demo.rst +++ b/docs/source/demo.rst @@ -40,8 +40,8 @@ its main interface. Several scenarios are shown as below. .. code-block:: python from pcapkit import HTTP, extract - # set strict to make sure full reassembly - extraction = extract(fin='in.pcap', store=False, nofile=True, reassembly=True, tcp=True, strict=True) + # set reasm_strict to make sure full reassembly + extraction = extract(fin='in.pcap', store=False, nofile=True, reassembly=True, tcp=True, reasm_strict=True) # print extracted packet if HTTP in reassembled payloads for datagram in extraction.reassembly.tcp: if datagram.packet is not None and HTTP in datagram.packet: @@ -58,7 +58,9 @@ The CLI (command line interface) of :mod:`pcapkit` has two different access. * through Python module - ``python -m pypcapkit [...]`` works exactly the same as above. + ``python -m pcapkit [...]`` works exactly the same as above. Note that the + module name is ``pcapkit``, even though the distribution on PyPI is named + ``pypcapkit``. Here are some usage samples: @@ -67,7 +69,7 @@ Here are some usage samples: .. code-block:: shell - $ pcapkit-cli in --format plist --verbose + $ pcapkit-cli in --auto-extension --format plist --verbose 🚨Loading file 'in.pcap' Frame 1: Ethernet:IPv6:IPv6_ICMP Frame 2: Ethernet:IPv6:IPv6_ICMP @@ -77,11 +79,18 @@ Here are some usage samples: Frame 6: Ethernet:IPv4:UDP:Raw 🍺Report file stored in 'out.plist' -2. export to a JSON file (with no format specified) +2. export to a JSON file + + .. note:: + + The output format is **not** presumed from the output file name, so + ``-f``/``--format`` (or the ``-j``/``--json`` switch) has to be given; + without it the default tree view is written into whatever file name was + supplied. .. code-block:: shell - $ pcapkit-cli in --output out.json --verbose + $ pcapkit-cli in --auto-extension --output out.json --format json --verbose 🚨Loading file 'in.pcap' Frame 1: Ethernet:IPv6:IPv6_ICMP Frame 2: Ethernet:IPv6:IPv6_ICMP @@ -95,7 +104,7 @@ Here are some usage samples: .. code-block:: shell - $ pcapkit-cli in --output out.txt --format tree --verbose + $ pcapkit-cli in.pcap --output out.txt --format tree --verbose 🚨Loading file 'in.pcap' Frame 1: Ethernet:IPv6:IPv6_ICMP Frame 2: Ethernet:IPv6:IPv6_ICMP diff --git a/docs/source/ext.rst b/docs/source/ext.rst index fb8826f2a2..46b73f34a5 100644 --- a/docs/source/ext.rst +++ b/docs/source/ext.rst @@ -55,6 +55,8 @@ The following table shows all available protocol classes in :mod:`pcapkit`: | | | | :class:`pcapkit.protocols.internet.ipsec.IPsec` | + + + IPsec Family +-------------------------------------------------------------+ | | | | :class:`pcapkit.protocols.internet.ah.AH` | ++ + + +-------------------------------------------------------------+ +| | | | :class:`pcapkit.protocols.internet.esp.ESP` | + +----------------+-----------------------+-------------------------------------------------------------+ | | :class:`pcapkit.protocols.internet.ipx.IPX` | + +----------------+-----------------------+-------------------------------------------------------------+ diff --git a/docs/source/pcapkit/const/ipv4.rst b/docs/source/pcapkit/const/ipv4.rst index b93869bc76..15a9745cc0 100644 --- a/docs/source/pcapkit/const/ipv4.rst +++ b/docs/source/pcapkit/const/ipv4.rst @@ -26,11 +26,11 @@ enumerations include: - ToS (DS Field) Delay * - :class:`IPv4_ToSECN ` - ToS ECN Field - * - :class:`IPv4_ToSPrecedence ` + * - :class:`IPv4_ToSPrecedence ` - ToS (DS Field) Precedence - * - :class:`IPv4_ToSReliability ` + * - :class:`IPv4_ToSReliability ` - ToS (DS Field) Reliability - * - :class:`IPv4_ToSThroughput ` + * - :class:`IPv4_ToSThroughput ` - ToS (DS Field) Throughput * - :class:`IPv4_TSFlag ` - TS Flag diff --git a/docs/source/pcapkit/dumpkit/index.rst b/docs/source/pcapkit/dumpkit/index.rst index bae7f259db..b394338fc5 100644 --- a/docs/source/pcapkit/dumpkit/index.rst +++ b/docs/source/pcapkit/dumpkit/index.rst @@ -33,8 +33,8 @@ of :mod:`pcapkit.dumpkit`: A --> Tree & XML & JSON subgraph pcapkit [PyPCAPKit Dumpers] - DumperBase --> Dumper --> PCAPIO & NotImplementedIO - Dumper --> E([user customisation ...]) + DumperBase --> PCAPIO & NotImplementedIO + DumperBase --> Dumper --> E([user customisation ...]) end A --> DumperBase @@ -49,4 +49,4 @@ of :mod:`pcapkit.dumpkit`: click DumperBase "/pcapkit/dumpkit/common.html#pcapkit.dumpkit.common.DumperBase" click Dumper "/pcapkit/dumpkit/common.html#pcapkit.dumpkit.common.Dumper" click PCAPIO "/pcapkit/dumpkit/pcap.html#pcapkit.dumpkit.pcap.PCAPIO" - click NotImplementedIO "/pcapkit/dumpkit/pcap.html#pcapkit.dumpkit.pcap.NotImplementedIO" + click NotImplementedIO "/pcapkit/dumpkit/null.html#pcapkit.dumpkit.null.NotImplementedIO" diff --git a/docs/source/pcapkit/dumpkit/null.rst b/docs/source/pcapkit/dumpkit/null.rst index f298ae7d25..04240c039d 100644 --- a/docs/source/pcapkit/dumpkit/null.rst +++ b/docs/source/pcapkit/dumpkit/null.rst @@ -1,7 +1,7 @@ Null Dumper =========== -.. module:: pcapkit.dumper.null +.. module:: pcapkit.dumpkit.null :mod:`pcapkit.dumpkit.null` is the dumper for :mod:`pcapkit` implementation, specifically for **NotImplemented** format, which is alike those described in diff --git a/docs/source/pcapkit/dumpkit/pcap.rst b/docs/source/pcapkit/dumpkit/pcap.rst index 7e781e2d0c..c44132092a 100644 --- a/docs/source/pcapkit/dumpkit/pcap.rst +++ b/docs/source/pcapkit/dumpkit/pcap.rst @@ -1,7 +1,7 @@ PCAP Dumper =========== -.. module:: pcapkit.dumper.pcap +.. module:: pcapkit.dumpkit.pcap :mod:`pcapkit.dumpkit.pcap` is the dumper for :mod:`pcapkit` implementation, specifically for PCAP format, which is alike those described in diff --git a/docs/source/pcapkit/foundation/engines/3rdparty.rst b/docs/source/pcapkit/foundation/engines/3rdparty.rst index 2f6e9f12e1..0c18f5bc21 100644 --- a/docs/source/pcapkit/foundation/engines/3rdparty.rst +++ b/docs/source/pcapkit/foundation/engines/3rdparty.rst @@ -12,6 +12,35 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`. .. _Scapy: https://scapy.net +.. note:: + + Constructing this engine imports :mod:`scapy.all`, which is what populates + `Scapy`_'s layer registries -- ``conf.l2types`` and the ``bind_layers`` + payload table, both of which exist only as import side effects of the layer + modules. Importing a narrower submodule leaves them empty, and + :class:`~scapy.utils.PcapReader` then returns every frame as one opaque + :class:`~scapy.packet.Raw` layer without raising, so the engine dissected + nothing at all and said so only on :data:`sys.stderr`. See + :meth:`Scapy.__init__` for why naming the layer modules individually is not a + cheaper route to the same place. + + One side effect is worth knowing about in advance: :mod:`scapy.all` loads + :mod:`scapy.layers.dcerpc`, which reaches `Scapy`_'s TLS layer and there + triggers a ``CryptographyDeprecationWarning`` from :mod:`cryptography` about + finite-field Diffie-Hellman. It concerns a key-exchange code path + :mod:`pcapkit` never executes, but it subclasses :exc:`UserWarning` rather + than :exc:`DeprecationWarning`, so Python's default filters show it. + + :mod:`pcapkit` deliberately does not filter it away -- it is `Scapy`_'s to + emit and the consumer's to silence, on the same footing as every other + category (see :mod:`pcapkit.utilities.warnings`):: + + import warnings + + from cryptography.utils import CryptographyDeprecationWarning + + warnings.filterwarnings('ignore', category=CryptographyDeprecationWarning) + .. autoclass:: pcapkit.foundation.engines.scapy.Scapy :no-members: :show-inheritance: @@ -19,6 +48,7 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`. .. autoattribute:: __engine_name__ .. autoattribute:: __engine_module__ + .. automethod:: __init__ .. automethod:: run .. automethod:: read_frame @@ -173,8 +203,12 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`. .. autoattribute:: __engine_name__ .. autoattribute:: __engine_module__ + .. autoattribute:: __engine_distribution__ + + .. automethod:: unsupported_reason .. autoproperty:: dlink + .. autoproperty:: backend .. automethod:: run .. automethod:: read_frame @@ -320,8 +354,12 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`. .. autoattribute:: __engine_name__ .. autoattribute:: __engine_module__ + .. autoattribute:: __engine_distribution__ + + .. automethod:: unsupported_reason .. autoproperty:: dlink + .. autoproperty:: backend .. automethod:: __init__ .. automethod:: run @@ -360,6 +398,9 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`. .. autoattribute:: __engine_name__ .. autoattribute:: __engine_module__ .. autoattribute:: LAYERS + .. autoattribute:: PYTHON_CEILING + + .. automethod:: unsupported_reason .. autoproperty:: dlink @@ -383,3 +424,63 @@ Internal Definitions .. automethod:: pcapkit.foundation.engines.pypcapfile.PyPCAPFile._get_decoder .. automethod:: pcapkit.foundation.engines.pypcapfile.PyPCAPFile._decode + +Backend Detection +================= + +.. module:: pcapkit.foundation.engines._pcap_backend + +Two unrelated PyPI distributions install a top-level module named :mod:`pcap` -- +`PyPCAP`_, a Cython binding shipped as a single extension module, and `pcap-ct`_, +a :mod:`ctypes` reimplementation shipped as a package. They therefore collide, +and ``import pcap`` resolves to whichever the import system finds first. +:class:`~pcapkit.foundation.engines.pypcap.PyPCAP` and +:class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` each have to know which one +they actually got rather than assume, and this module is the one place that +answers it -- deliberately shared, since the two engines must agree and two +copies of the detection would be two chances to disagree. It is *only* detection, +and it imports nothing from :mod:`pcapkit`, so it cannot introduce an import +cycle. + +.. autodata:: pcapkit.foundation.engines._pcap_backend.PYPCAP + +.. autodata:: pcapkit.foundation.engines._pcap_backend.PCAP_CT + +.. autodata:: pcapkit.foundation.engines._pcap_backend.DISTRIBUTIONS + +.. autodata:: pcapkit.foundation.engines._pcap_backend.ENGINE_NAMES + +.. autoclass:: pcapkit.foundation.engines._pcap_backend.Probe + :no-members: + :show-inheritance: + + .. note:: + + This is an :class:`~pcapkit.corekit.infoclass.Info` subclass, so it is a + :class:`~collections.abc.Mapping` rather than a :class:`tuple`: its fields + are reached by name, not by position, and it cannot be unpacked as a + sequence. + + .. autoattribute:: name + .. autoattribute:: version + .. autoattribute:: origin + .. autoattribute:: failure + .. autoattribute:: missing + .. autoattribute:: installed + + .. automethod:: describe + +.. autofunction:: pcapkit.foundation.engines._pcap_backend.probe + +.. autofunction:: pcapkit.foundation.engines._pcap_backend.identify + +.. autofunction:: pcapkit.foundation.engines._pcap_backend.installed_distributions + +.. autofunction:: pcapkit.foundation.engines._pcap_backend.wrong_backend_reason + +.. autofunction:: pcapkit.foundation.engines._pcap_backend.collision_reason + +Internal Definitions +-------------------- + +.. autofunction:: pcapkit.foundation.engines._pcap_backend._purge diff --git a/docs/source/pcapkit/foundation/engines/engine.rst b/docs/source/pcapkit/foundation/engines/engine.rst index 693c9d3be7..de7ef7107f 100644 --- a/docs/source/pcapkit/foundation/engines/engine.rst +++ b/docs/source/pcapkit/foundation/engines/engine.rst @@ -38,6 +38,8 @@ all engine support functionality. .. autoproperty:: extractor + .. automethod:: unsupported_reason + .. automethod:: run .. automethod:: read_frame .. automethod:: close diff --git a/docs/source/pcapkit/foundation/engines/index.rst b/docs/source/pcapkit/foundation/engines/index.rst index d6b7da480c..601ed7ee39 100644 --- a/docs/source/pcapkit/foundation/engines/index.rst +++ b/docs/source/pcapkit/foundation/engines/index.rst @@ -122,9 +122,10 @@ so it is worth knowing in advance. | | loop with :func:`asyncio.get_event_loop`, which raises from | | | 3.14 | +-----------------------------------------------------------------+---------------------------------------------------------------+ -| :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` | `libpcap`_ headers and library, a C compiler, and Python | -| | **3.11 or older** -- ``pypcap`` 1.3.0 publishes no wheel and | -| | its pre-generated :file:`pcap.c` does not compile on 3.12+ | +| :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` | :manpage:`libpcap(3)` headers and library, a C compiler, and | +| | Python **3.11 or older** -- ``pypcap`` 1.3.0 publishes no | +| | wheel and its pre-generated :file:`pcap.c` does not compile | +| | on 3.12+ | +-----------------------------------------------------------------+---------------------------------------------------------------+ | :class:`~pcapkit.foundation.engines.pcap_ct.PCAP_CT` | a system ``libpcap.so.1`` at *run* time -- nothing to build, | | | since ``pcap-ct`` and ``libpcap`` ship pure-Python wheels, | @@ -137,7 +138,7 @@ so it is worth knowing in advance. Every one of these constraints is also enforced in code rather than only documented: each engine overrides -:meth:`~pcapkit.foundation.engines.engine.EngineBase.unsupported_reason`, which +:meth:`~pcapkit.foundation.engines.engine.Engine.unsupported_reason`, which :meth:`Extractor.run ` consults *before* the import test, so asking for an engine that cannot run here produces one warning naming the actual cause and a clean fall back to the built-in parser. @@ -214,4 +215,3 @@ actually got, via .. _PyPCAP: https://github.com/pynetwork/pypcap .. _pcap-ct: https://pypi.org/project/pcap-ct/ .. _PyPCAPFile: https://github.com/kisom/pypcapfile -.. _libpcap: https://www.tcpdump.org diff --git a/docs/source/pcapkit/foundation/extraction.rst b/docs/source/pcapkit/foundation/extraction.rst index badd1acc53..07056decee 100644 --- a/docs/source/pcapkit/foundation/extraction.rst +++ b/docs/source/pcapkit/foundation/extraction.rst @@ -11,12 +11,13 @@ extracts parametres from a PCAP file. .. seealso:: - Engine support for |pypcap|_ and |pypcapfile|_ has since landed, as - :class:`pcapkit.foundation.engines.pypcap.PyPCAP` (``engine='pypcap'``) and - :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` - (``engine='pypcapfile'``). Both support less than the ``default`` engine - does; :doc:`engines/index` tabulates the gaps, and :doc:`../../index` - documents the installation prerequisites each of the two carries. + Engine support for |pypcap|_, |pcap-ct|_ and |pypcapfile|_ has since landed, + as :class:`pcapkit.foundation.engines.pypcap.PyPCAP` (``engine='pypcap'``), + :class:`pcapkit.foundation.engines.pcap_ct.PCAP_CT` (``engine='pcap_ct'``) + and :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` + (``engine='pypcapfile'``). All three support less than the ``default`` + engine does; :doc:`engines/index` tabulates the gaps, and :doc:`../../index` + documents the installation prerequisites each of the three carries. .. autoclass:: pcapkit.foundation.extraction.Extractor :no-members: @@ -91,5 +92,7 @@ Type Variables .. |pypcap| replace:: ``pypcap`` .. _pypcap: https://github.com/pynetwork/pypcap +.. |pcap-ct| replace:: ``pcap-ct`` +.. _pcap-ct: https://pypi.org/project/pcap-ct/ .. |pypcapfile| replace:: ``pypcapfile`` .. _pypcapfile: https://github.com/kisom/pypcapfile diff --git a/docs/source/pcapkit/foundation/index.rst b/docs/source/pcapkit/foundation/index.rst index 91b3d16c2e..9a67cd571f 100644 --- a/docs/source/pcapkit/foundation/index.rst +++ b/docs/source/pcapkit/foundation/index.rst @@ -5,8 +5,8 @@ Library Foundation :mod:`pcapkit.foundation` is a collection of foundations for :mod:`pcapkit`, including PCAP file extraction tool -:class:`~pcapkit.foundation.extraction.Extrator`, TCP flow tracer -:class:`~pcapkit.foundation.tractflow.TraceFlow`, registry management +:class:`~pcapkit.foundation.extraction.Extractor`, TCP flow tracer +:class:`~pcapkit.foundation.traceflow.traceflow.TraceFlow`, registry management APIs for :mod:`pcapkit`, and TCP/IP reassembly implementations. .. toctree:: diff --git a/docs/source/pcapkit/foundation/reassembly/index.rst b/docs/source/pcapkit/foundation/reassembly/index.rst index c65b133291..f236da9a64 100644 --- a/docs/source/pcapkit/foundation/reassembly/index.rst +++ b/docs/source/pcapkit/foundation/reassembly/index.rst @@ -41,9 +41,9 @@ diagram of the class hierarchy of :mod:`pcapkit.foundation.reassembly`: click C "/pcapkit/foundation/reassembly/reassembly.html#pcapkit.foundation.reassembly.reassembly.Reassembly" click D "/ext.html#reassembly-and-flow-tracing" - click IP "/pcapkit/foundation/reassembly/ip/index.html#pcapkit.foundation.reassembly.ip.IP" - click IPv4 "/pcapkit/foundation/reassembly/ip/ipv4.html#pcapkit.foundation.reassembly.ip.ipv4.IPv4" - click IPv6 "/pcapkit/foundation/reassembly/ip/ipv6.html#pcapkit.foundation.reassembly.ip.ipv6.IPv6" + click IP "/pcapkit/foundation/reassembly/ip/ip.html#pcapkit.foundation.reassembly.ip.IP" + click IPv4 "/pcapkit/foundation/reassembly/ip/ipv4.html#pcapkit.foundation.reassembly.ipv4.IPv4" + click IPv6 "/pcapkit/foundation/reassembly/ip/ipv6.html#pcapkit.foundation.reassembly.ipv6.IPv6" click TCP "/pcapkit/foundation/reassembly/tcp.html#pcapkit.foundation.reassembly.tcp.TCP" Auxiliary Data diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ip.rst b/docs/source/pcapkit/foundation/reassembly/ip/ip.rst index 8b74dff32a..2a8b2d978b 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ip.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ip.rst @@ -50,4 +50,4 @@ Type Variables -------------- .. data:: pcapkit.foundation.reassembly.data.ip._AT - :type: ipaddress.IPv4Address | ipaddress.IPv4Address + :type: ipaddress.IPv4Address | ipaddress.IPv6Address diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst index 79757bc5fd..6e6112530f 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst @@ -32,10 +32,10 @@ Terminology ipv4.src, # source IP address ipv4.dst, # destination IP address ipv4.id, # identification - ipv4.proto, # payload protocol type + ipv4.protocol, # payload protocol type ), num = frame.number, # original packet range number - fo = ipv4.frag_offset, # fragment offset + fo = ipv4.offset, # fragment offset, in octets ihl = ipv4.hdr_len, # internet header length mf = ipv4.flags.mf, # more fragment flag tl = ipv4.len, # total length, header includes @@ -57,7 +57,7 @@ Terminology | | |--> 'src' --> (IPv4Address) ipv4.src | | |--> 'dst' --> (IPv4Address) ipv4.dst | | |--> 'id' --> (int) ipv4.id - | | |--> 'proto' --> (EtherType) ipv4.proto + | | |--> 'proto' --> (TransType) ipv4.protocol | |--> 'index' : (tuple) packet numbers | | |--> (int) original packet range number | |--> 'header' : (bytes) IPv4 header @@ -69,7 +69,7 @@ Terminology | | |--> 'src' --> (IPv4Address) ipv4.src | | |--> 'dst' --> (IPv4Address) ipv4.dst | | |--> 'id' --> (int) ipv4.id - | | |--> 'proto' --> (EtherType) ipv4.proto + | | |--> 'proto' --> (TransType) ipv4.protocol | |--> 'index' : (tuple) packet numbers | | |--> (int) original packet range number | |--> 'header' : (bytes) IPv4 header @@ -91,7 +91,7 @@ Terminology | |--> ipv4.src | | |--> ipv4.dst | | |--> ipv4.id | - | |--> ipv4.proto | + | |--> ipv4.protocol | | |--> 'TDL' : (int) total data length | |--> 'RCVBT' : (bytearray) fragment received bit table | | |--> (bytes) b'\\x00' -> not received diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst index 00fab96ec1..e8882e96f8 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst @@ -31,18 +31,27 @@ Terminology bufid = tuple( ipv6.src, # source IP address ipv6.dst, # destination IP address - ipv6.label, # label + ipv6_frag.id, # identification ipv6_frag.next, # next header field in IPv6 Fragment Header ), num = frame.number, # original packet range number - fo = ipv6_frag.offset, # fragment offset + fo = ipv6_frag.offset, # fragment offset, in octets ihl = ipv6.hdr_len, # header length, only headers before IPv6-Frag mf = ipv6_frag.mf, # more fragment flag - tl = ipv6.len, # total length, header includes + tl = ipv6.hdr_len + + ipv6.raw_len, # total length, header includes header = ipv6.header, # raw bytes type header before IPv6-Frag payload = ipv6.payload, # raw bytearray type payload after IPv6-Frag ) + .. note:: + + The reassembly key is the Fragment header's *Identification* + (:rfc:`8200#section-4.5`), not the IPv6 header's *Flow Label*. The + label is optional and routinely zero, so keying on it collapses + every datagram between one address pair into a single buffer and + interleaves their fragments. + reasm.ipv6.datagram Data structure for **reassembled IPv6 datagram** (element from :attr:`IPv6.datagram ` @@ -56,24 +65,25 @@ Terminology | |--> 'id' : (Info) original packet identifier | | |--> 'src' --> (IPv6Address) ipv6.src | | |--> 'dst' --> (IPv6Address) ipv6.dst - | | |--> 'id' --> (int) ipv6.label - | | |--> 'proto' --> (EtherType) ipv6_frag.next + | | |--> 'id' --> (int) ipv6_frag.id + | | |--> 'proto' --> (TransType) ipv6_frag.next | |--> 'index' : (tuple) packet numbers | | |--> (int) original packet range number - | |--> 'payload' : (bytes) reassembled IPv4 packet + | |--> 'header' : (bytes) header before IPv6-Frag + | |--> 'payload' : (bytes) reassembled IPv6 payload | |--> 'packet' : (Protocol) parsed reassembled payload |--> (Info) data | |--> 'completed' : (bool) False --> not implemented | |--> 'id' : (Info) original packet identifier | | |--> 'src' --> (IPv6Address) ipv6.src | | |--> 'dst' --> (IPv6Address) ipv6.dst - | | |--> 'id' --> (int) ipv6.id - | | |--> 'proto' --> (EtherType) ipv6_frag.next + | | |--> 'id' --> (int) ipv6_frag.id + | | |--> 'proto' --> (TransType) ipv6_frag.next | |--> 'index' : (tuple) packet numbers | | |--> (int) original packet range number - | |--> 'header' : (bytes) IPv4 header - | |--> 'payload' : (tuple) partially reassembled IPv4 payload - | | |--> (bytes) IPv4 payload fragment + | |--> 'header' : (bytes) header before IPv6-Frag + | |--> 'payload' : (tuple) partially reassembled IPv6 payload + | | |--> (bytes) IPv6 payload fragment | | |--> ... | |--> 'packet' : (None) |--> (Info) data ... @@ -88,8 +98,8 @@ Terminology (dict) buffer --> memory buffer for reassembly |--> (tuple) BUFID : (dict) | |--> ipv6.src | - | |--> ipc6.dst | - | |--> ipv6.label | + | |--> ipv6.dst | + | |--> ipv6_frag.id | | |--> ipv6_frag.next | | |--> 'TDL' : (int) total data length | |--> RCVBT : (bytearray) fragment received bit table diff --git a/docs/source/pcapkit/foundation/reassembly/reassembly.rst b/docs/source/pcapkit/foundation/reassembly/reassembly.rst index 1d26acc188..c9785533e4 100644 --- a/docs/source/pcapkit/foundation/reassembly/reassembly.rst +++ b/docs/source/pcapkit/foundation/reassembly/reassembly.rst @@ -95,7 +95,7 @@ Type Variables Datagram data structure. .. data:: pcapkit.foundation.reassembly.reassembly._IT - :type: pcapkit.corekit.infoclass.Info + :type: tuple Buffer ID data structure. diff --git a/docs/source/pcapkit/foundation/reassembly/tcp.rst b/docs/source/pcapkit/foundation/reassembly/tcp.rst index 63045e47c8..a61f8e19d1 100644 --- a/docs/source/pcapkit/foundation/reassembly/tcp.rst +++ b/docs/source/pcapkit/foundation/reassembly/tcp.rst @@ -39,10 +39,12 @@ Algorithm +-------------+---------------------------+ | ``BUFID`` | Buffer Identifier | +-------------+---------------------------+ -| ``HDL`` | Hole Discriptor List | +| ``HDL`` | Hole Descriptor List | +-------------+---------------------------+ | ``ISN`` | Initial Sequence Number | +-------------+---------------------------+ +| ``PSN`` | Payload Sequence Number | ++-------------+---------------------------+ | ``src`` | source IP | +-------------+---------------------------+ | ``dst`` | destination IP | @@ -55,7 +57,12 @@ Algorithm .. code-block:: text DO { - BUFID <- src|dst|srcport|dstport|ACK; + BUFID <- src|srcport|dst|dstport; + + /* a SYN occupies a sequence number of its own, so payload sent by + or after it starts one octet later than the segment's DSN */ + PSN <- DSN + 1 IF (SYN is true) ELSE DSN; + IF (SYN is true) { IF (buffer with BUFID is allocated) { flush all reassembly for this BUFID; @@ -65,10 +72,19 @@ Algorithm IF (no buffer with BUFID is allocated) { allocate reassembly resources with BUFID; - ISN <- DSN; + ISN <- PSN; put data from fragment into data buffer with BUFID [from octet fragment.first to octet fragment.last]; - update HDL; + HDL <- [one hole from PSN + fragment.len to infinity]; + } ELSE { + put data from fragment into data buffer with BUFID + [from octet fragment.first to octet fragment.last]; + + /* a segment with no payload fills no hole, and its "last" lies + one below its "first", so it is not run through the algorithm */ + IF (fragment.len > 0) { + update HDL; + } } IF (FIN is true or RST is true) { @@ -82,24 +98,24 @@ Algorithm DO { select the next hole descriptor from HDL; - IF (fragment.first >= hole.first) CONTINUE. - IF (fragment.last <= hole.first) CONTINUE. + IF (fragment.first > hole.last) CONTINUE. + IF (fragment.last < hole.first) CONTINUE. delete the current entry from HDL; - IF (fragment.first >= hole.first) { + IF (fragment.first > hole.first) { create new entry "new_hole" in HDL; new_hole.first <- hole.first; new_hole.last <- fragment.first - 1; - BREAK. } - IF (fragment.last <= hole.last) { + IF (fragment.last < hole.last AND FIN is false AND RST is false) { create new entry "new_hole" in HDL; new_hole.first <- fragment.last + 1; new_hole.last <- hole.last; - BREAK. } + + BREAK. } give up until (no entry from HDL) } @@ -117,8 +133,9 @@ appeared in :rfc:`791`. And here is the process: new hole descriptor ``new_hole`` with ``new_hole.first`` equal to ``hole.first``, and ``new_hole.last`` equal to ``fragment.first`` minus one (``-1``). -6. If ``fragment.last`` is less than ``hole.last`` and - ``fragment.more_fragments`` is ``true``, then create a new hole +6. If ``fragment.last`` is less than ``hole.last`` and neither ``FIN`` + nor ``RST`` is set -- TCP has no *more fragments* flag, so the + termination flags take its place -- then create a new hole descriptor ``new_hole``, with ``new_hole.first`` equal to ``fragment.last`` plus one (``+1``) and ``new_hole.last`` equal to ``hole.last``. @@ -153,12 +170,17 @@ Terminology fin = tcp.flags.fin, # finish flag rst = tcp.flags.rst, # reset connection flag len = tcp.raw_len, # payload length, header excludes - first = tcp.seq, # this sequence number - last = tcp.seq + tcp.raw_len, # next (wanted) sequence number + first = tcp.seq, # first sequence number of payload + last = tcp.seq + tcp.raw_len - 1, + # last sequence number of payload header = tcp.packet.header, # raw bytes type header payload = tcp.raw, # raw bytearray type payload ) + Both ``first`` and ``last`` are absolute TCP sequence numbers and + both are **inclusive**, so a segment carrying no payload at all has + ``last`` one below ``first``. + reasm.tcp.datagram Data structure for **reassembled TCP datagram** (element from :attr:`TCP.datagram ` @@ -213,19 +235,22 @@ Terminology (dict) buffer --> memory buffer for reassembly |--> (tuple) BUFID : (dict) | |--> ip.src | - | |--> ip.dst | | |--> tcp.srcport | + | |--> ip.dst | | |--> tcp.dstport | | |--> 'hdl' : (list) hole descriptor list | | |--> (Info) hole --> hole descriptor - | | |--> "first" --> (int) start of hole - | | |--> "last" --> (int) stop of hole + | | |--> "first" --> (int) sequence number of the + | | | first missing octet + | | |--> "last" --> (int) sequence number of the + | | last missing octet, inclusive | |--> 'hdr' : (bytes) initial TCP header | |--> 'ack' : (dict) ACK list | |--> (int) ACK : (dict) | | |--> 'ind' : (list) list of reassembled packets | | | |--> (int) packet range number - | | |--> 'isn' : (int) ISN of payload buffer + | | |--> 'isn' : (int) sequence number of the octet + | | | held in raw[0] | | |--> 'len' : (int) length of payload buffer | | |--> 'raw' : (bytearray) reassembled payload, | | holes set to b'\x00' @@ -233,6 +258,15 @@ Terminology | |--> ... |--> (tuple) BUFID ... + The hole descriptor list is kept in **absolute TCP sequence numbers**, + once per ``BUFID``, whereas each ACK's payload buffer is indexed from + its own ``isn`` -- ``raw[n]`` holds the octet with sequence number + ``isn + n``, and ``isn`` is revised downwards whenever a segment turns + up below the data already buffered, so it is not necessarily the + connection's own initial sequence number. + :meth:`TCP.submit ` is the + one place that converts between the two. + Data Models =========== @@ -271,7 +305,7 @@ Type Variables ============== .. data:: pcapkit.foundation.reassembly.data.tcp._AT - :type: ipaddress.IPv4Address | ipaddress.IPv4Address + :type: ipaddress.IPv4Address | ipaddress.IPv6Address .. data:: pcapkit.foundation.reassembly.data.tcp.BufferID :type: typing.Tuple[_AT, int, _AT, int] diff --git a/docs/source/pcapkit/foundation/traceflow/index.rst b/docs/source/pcapkit/foundation/traceflow/index.rst index bee1f98c29..98ee0a9c65 100644 --- a/docs/source/pcapkit/foundation/traceflow/index.rst +++ b/docs/source/pcapkit/foundation/traceflow/index.rst @@ -11,8 +11,8 @@ Flow Tracing a approximate functionality of *Follow TCP Streams* in `Wireshark `__. -:mod:`pcapkit.traceflow` implements flow tracing functions for -:mod:`pcapkit` package. +:mod:`pcapkit.foundation.traceflow` implements flow tracing functions +for :mod:`pcapkit` package. .. seealso:: @@ -43,7 +43,7 @@ diagram of the class hierarchy of :mod:`pcapkit.foundation.traceflow`: click A "/pcapkit/foundation/traceflow/traceflow.html#pcapkit.foundation.traceflow.traceflow.TraceFlowMeta" click B "/pcapkit/foundation/traceflow/traceflow.html#pcapkit.foundation.traceflow.traceflow.TraceFlowBase" click C "/pcapkit/foundation/traceflow/traceflow.html#pcapkit.foundation.traceflow.traceflow.TraceFlow" - click D "/ext.html#traceflow-and-flow-tracing" + click D "/ext.html#reassembly-and-flow-tracing" click TCP "/pcapkit/foundation/traceflow/tcp.html#pcapkit.foundation.traceflow.tcp.TCP" diff --git a/docs/source/pcapkit/interface/core.rst b/docs/source/pcapkit/interface/core.rst index 8b75d0b2dd..eea5880948 100644 --- a/docs/source/pcapkit/interface/core.rst +++ b/docs/source/pcapkit/interface/core.rst @@ -66,10 +66,11 @@ Extration Engines .. note:: - These constants predate the `PyPCAP`_ and `PyPCAPFile`_ engines and no - equivalents were added for them, so those two are selected by their literal - ``engine=`` values -- ``'pypcap'`` and ``'pypcapfile'`` -- rather than through - a named constant. Any engine registered at runtime with + These constants predate the `PyPCAP`_, `pcap-ct`_ and `PyPCAPFile`_ engines + and no equivalents were added for them, so those three are selected by their + literal ``engine=`` values -- ``'pypcap'``, ``'pcap_ct'`` and + ``'pypcapfile'`` -- rather than through a named constant. Any engine + registered at runtime with :func:`~pcapkit.foundation.registry.foundation.register_extractor_engine` is likewise addressed by its string name. @@ -79,4 +80,5 @@ Extration Engines supports, and the installation prerequisites the third-party ones carry. .. _PyPCAP: https://github.com/pynetwork/pypcap +.. _pcap-ct: https://pypi.org/project/pcap-ct/ .. _PyPCAPFile: https://github.com/kisom/pypcapfile diff --git a/docs/source/pcapkit/protocols/index.rst b/docs/source/pcapkit/protocols/index.rst index e1878b92fc..9499763ccd 100644 --- a/docs/source/pcapkit/protocols/index.rst +++ b/docs/source/pcapkit/protocols/index.rst @@ -47,7 +47,7 @@ diagram of the class hierarchy of :mod:`pcapkit.protocols`: IP --> IPv4 & IPv6 & IPsec subgraph ipsec [IPsec Family] - IPsec --> AH + IPsec --> AH & ESP end end @@ -109,6 +109,7 @@ diagram of the class hierarchy of :mod:`pcapkit.protocols`: click Internet "/pcapkit/protocols/internet/internet.html#pcapkit.protocols.internet.Internet" click AH "/pcapkit/protocols/internet/ah.html#pcapkit.protocols.internet.ah.AH" + click ESP "/pcapkit/protocols/internet/esp.html#pcapkit.protocols.internet.esp.ESP" click HIP "/pcapkit/protocols/internet/hip.html#pcapkit.protocols.internet.hip.HIP" click HOPOPT "/pcapkit/protocols/internet/hopopt.html#pcapkit.protocols.internet.hopopt.HOPOPT" click IP "/pcapkit/protocols/internet/ip.html#pcapkit.protocols.internet.ip.IP" diff --git a/docs/source/pcapkit/protocols/misc/null.rst b/docs/source/pcapkit/protocols/misc/null.rst index 3eb24fb4c6..39da8bed4d 100644 --- a/docs/source/pcapkit/protocols/misc/null.rst +++ b/docs/source/pcapkit/protocols/misc/null.rst @@ -3,11 +3,11 @@ No-Payload Packet .. module:: pcapkit.protocols.misc.null -:mod:`pcapkit.protocols.null` contains -:class:`~pcapkit.protocols.null.NoPayload` only, which +:mod:`pcapkit.protocols.misc.null` contains +:class:`~pcapkit.protocols.misc.null.NoPayload` only, which implements a :class:`~pcapkit.protocols.protocol.Protocol` like object whose payload is recursively -:class:`~pcapkit.protocols.null.NoPayload` itself. +:class:`~pcapkit.protocols.misc.null.NoPayload` itself. .. autoclass:: pcapkit.protocols.misc.null.NoPayload :no-members: diff --git a/docs/source/pcapkit/utilities/functools.rst b/docs/source/pcapkit/utilities/functools.rst index 9f2d006937..b5b2114c90 100644 --- a/docs/source/pcapkit/utilities/functools.rst +++ b/docs/source/pcapkit/utilities/functools.rst @@ -34,7 +34,7 @@ Type Variables :type: pcapkit.protocols.protocol.ProtocolBase .. data:: pcapkit.utilities.decorators.R_prepare - :type: pcapkit.protocols.schema.schmea.Schema + :type: pcapkit.protocols.schema.schema.Schema Error Handling Utilities ======================== diff --git a/docs/source/pcapkit/vendor/http.rst b/docs/source/pcapkit/vendor/http.rst index cee3e8a962..1cc7edb28c 100644 --- a/docs/source/pcapkit/vendor/http.rst +++ b/docs/source/pcapkit/vendor/http.rst @@ -5,7 +5,7 @@ .. module:: pcapkit.vendor.http -This module contains all constant enumerations of +This module contains all vendor crawlers of :class:`~pcapkit.protocols.application.http.HTTP` implementations. Available vendor crawlers include: @@ -28,7 +28,7 @@ HTTP/2 Error Code .. module:: pcapkit.vendor.http.error_code This module contains the vendor crawler for **HTTP/2 Error Code**, -which is automatically generating :class:`pcapkit.+const+.http.error_code.ErrorCode`. +which is automatically generating :class:`pcapkit.const.http.error_code.ErrorCode`. .. autoclass:: pcapkit.vendor.http.error_code.ErrorCode :members: FLAG, LINK @@ -40,7 +40,7 @@ HTTP/2 Frame Type .. module:: pcapkit.vendor.http.frame This module contains the vendor crawler for **HTTP/2 Frame Type**, -which is automatically generating :class:`pcapkit.+const+.http.frame.Frame`. +which is automatically generating :class:`pcapkit.const.http.frame.Frame`. .. autoclass:: pcapkit.vendor.http.frame.Frame :members: FLAG, LINK @@ -52,7 +52,7 @@ HTTP Method .. module:: pcapkit.vendor.http.method This module contains the vendor crawler for **HTTP Method**, -which is automatically generating :class:`pcapkit.+const+.http.method.Method`. +which is automatically generating :class:`pcapkit.const.http.method.Method`. .. autoclass:: pcapkit.vendor.http.method.Method :members: LINK @@ -64,7 +64,7 @@ HTTP/2 Settings .. module:: pcapkit.vendor.http.setting This module contains the vendor crawler for **HTTP/2 Settings**, -which is automatically generating :class:`pcapkit.+const+.http.setting.Setting`. +which is automatically generating :class:`pcapkit.const.http.setting.Setting`. .. autoclass:: pcapkit.vendor.http.setting.Setting :members: FLAG, LINK @@ -76,7 +76,7 @@ HTTP Status Code .. module:: pcapkit.vendor.http.status_code This module contains the vendor crawler for **HTTP Status Code**, -which is automatically generating :class:`pcapkit.+const+.http.status_code.StatusCode`. +which is automatically generating :class:`pcapkit.const.http.status_code.StatusCode`. .. autoclass:: pcapkit.vendor.http.status_code.StatusCode :members: FLAG, LINK diff --git a/docs/source/pcapkit/vendor/index.rst b/docs/source/pcapkit/vendor/index.rst index 0b2f4eaeb1..78d8f7d655 100644 --- a/docs/source/pcapkit/vendor/index.rst +++ b/docs/source/pcapkit/vendor/index.rst @@ -5,7 +5,7 @@ Vendor Crawlers .. module:: pcapkit.vendor This module contains all web crawlers of :mod:`pcapkit`, which are -automatically generating from the :mod:`pcapkit.const` module's constant +automatically generating the :mod:`pcapkit.const` module's constant enumerations. Crawler Implementations diff --git a/pcapkit/const/ipv4/__init__.py b/pcapkit/const/ipv4/__init__.py index 31818463b6..f47d8918a2 100644 --- a/pcapkit/const/ipv4/__init__.py +++ b/pcapkit/const/ipv4/__init__.py @@ -25,11 +25,11 @@ - ToS (DS Field) Delay * - :class:`IPv4_ToSECN ` - ToS ECN Field - * - :class:`IPv4_ToSPrecedence ` + * - :class:`IPv4_ToSPrecedence ` - ToS (DS Field) Precedence - * - :class:`IPv4_ToSReliability ` + * - :class:`IPv4_ToSReliability ` - ToS (DS Field) Reliability - * - :class:`IPv4_ToSThroughput ` + * - :class:`IPv4_ToSThroughput ` - ToS (DS Field) Throughput * - :class:`IPv4_TSFlag ` - TS Flag diff --git a/pcapkit/dumpkit/null.py b/pcapkit/dumpkit/null.py index 4c7d647445..f5b77c9dfe 100644 --- a/pcapkit/dumpkit/null.py +++ b/pcapkit/dumpkit/null.py @@ -2,7 +2,7 @@ """Null Dumper ================= -.. module:: pcapkit.dumper.null +.. module:: pcapkit.dumpkit.null :mod:`pcapkit.dumpkit.null` is the dumper for :mod:`pcapkit` implementation, specifically for **NotImplemented** format, which is alike those described in diff --git a/pcapkit/dumpkit/pcap.py b/pcapkit/dumpkit/pcap.py index efa4f83f04..546bac7037 100644 --- a/pcapkit/dumpkit/pcap.py +++ b/pcapkit/dumpkit/pcap.py @@ -2,7 +2,7 @@ """PCAP Dumper ================= -.. module:: pcapkit.dumper.pcap +.. module:: pcapkit.dumpkit.pcap :mod:`pcapkit.dumpkit.pcap` is the dumper for :mod:`pcapkit` implementation, specifically for PCAP format, which is alike those described in diff --git a/pcapkit/foundation/__init__.py b/pcapkit/foundation/__init__.py index 1ec7115251..443f4b3603 100644 --- a/pcapkit/foundation/__init__.py +++ b/pcapkit/foundation/__init__.py @@ -7,8 +7,8 @@ :mod:`pcapkit.foundation` is a collection of foundations for :mod:`pcapkit`, including PCAP file extraction tool -:class:`~pcapkit.foundation.extraction.Extrator`, flow tracing -:mod:`~pcapkit.foundation.tractflow`, registry management +:class:`~pcapkit.foundation.extraction.Extractor`, flow tracing +:mod:`~pcapkit.foundation.traceflow`, registry management APIs for :mod:`pcapkit`, and TCP/IP reassembly implementations. """ diff --git a/pcapkit/foundation/traceflow/__init__.py b/pcapkit/foundation/traceflow/__init__.py index 0e202e3ec1..9e0d0ffdf5 100644 --- a/pcapkit/foundation/traceflow/__init__.py +++ b/pcapkit/foundation/traceflow/__init__.py @@ -5,7 +5,7 @@ .. module:: pcapkit.foundation.traceflow -:mod:`pcapkit.traceflow` implements flow tracing functions for +:mod:`pcapkit.foundation.traceflow` implements flow tracing functions for :mod:`pcapkit` package. .. note:: diff --git a/pcapkit/protocols/internet/ipv6_opts.py b/pcapkit/protocols/internet/ipv6_opts.py index cb1f55aced..f7f6ec692b 100644 --- a/pcapkit/protocols/internet/ipv6_opts.py +++ b/pcapkit/protocols/internet/ipv6_opts.py @@ -2,7 +2,7 @@ """IPv6-Opts - Destination Options for IPv6 ============================================== -.. module:: pcapkit.protocols.internet.hopopt +.. module:: pcapkit.protocols.internet.ipv6_opts :mod:`pcapkit.protocols.internet.ipv6_opts` contains :class:`~pcapkit.protocols.internet.ipv6_opts.IPv6_Opts` diff --git a/pcapkit/protocols/misc/null.py b/pcapkit/protocols/misc/null.py index bd05bc229b..6a13295aa9 100644 --- a/pcapkit/protocols/misc/null.py +++ b/pcapkit/protocols/misc/null.py @@ -4,11 +4,11 @@ .. module:: pcapkit.protocols.misc.null -:mod:`pcapkit.protocols.null` contains -:class:`~pcapkit.protocols.null.NoPayload` only, which +:mod:`pcapkit.protocols.misc.null` contains +:class:`~pcapkit.protocols.misc.null.NoPayload` only, which implements a :class:`~pcapkit.protocols.protocol.Protocol` like object whose payload is recursively -:class:`~pcapkit.protocols.null.NoPayload` itself. +:class:`~pcapkit.protocols.misc.null.NoPayload` itself. """ import io diff --git a/pcapkit/vendor/esp/__init__.py b/pcapkit/vendor/esp/__init__.py index 155c949a20..04847ea8ec 100644 --- a/pcapkit/vendor/esp/__init__.py +++ b/pcapkit/vendor/esp/__init__.py @@ -7,7 +7,7 @@ This module contains all vendor crawlers of :class:`~pcapkit.protocols.internet.esp.ESP` implementations. Available -enumerations include: +vendor crawlers include: .. list-table:: @@ -17,10 +17,9 @@ - Integrity Algorithm Transform IDs [*]_ ESP has no algorithm registry of its own: an SA's algorithms are negotiated by -IKEv2, so both enumerations are the corresponding IKEv2 *transform ID* -sub-registries. They live here rather than under an ``ikev2`` package because -:class:`~pcapkit.protocols.internet.esp.ESP` is the only thing in -:mod:`pcapkit` that consumes them. +IKEv2, so both crawlers pull the corresponding IKEv2 *transform ID* +sub-registries, which are published as separate CSV files from the IKEv2 +parameters page. .. [*] https://www.iana.org/assignments/ikev2-parameters/ikev2-parameters.xhtml#ikev2-parameters-5 .. [*] https://www.iana.org/assignments/ikev2-parameters/ikev2-parameters.xhtml#ikev2-parameters-7 diff --git a/pcapkit/vendor/ipv4/__init__.py b/pcapkit/vendor/ipv4/__init__.py index c97ccef2f8..673143ddf7 100644 --- a/pcapkit/vendor/ipv4/__init__.py +++ b/pcapkit/vendor/ipv4/__init__.py @@ -27,11 +27,11 @@ - ToS (DS Field) Delay * - :class:`IPv4_ToSECN ` - ToS ECN Field - * - :class:`IPv4_ToSPrecedence ` + * - :class:`IPv4_ToSPrecedence ` - ToS (DS Field) Precedence - * - :class:`IPv4_ToSReliability ` + * - :class:`IPv4_ToSReliability ` - ToS (DS Field) Reliability - * - :class:`IPv4_ToSThroughput ` + * - :class:`IPv4_ToSThroughput ` - ToS (DS Field) Throughput .. [*] https://www.iana.org/assignments/ip-parameters/ip-parameters.xhtml#ip-parameters-1 From 7b0b79018effa84658896acc0730339d8c26e183 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 16 Sep 2026 10:50:21 -0400 Subject: [PATCH 2/3] docs: align three field names with the code, drop two duplicate automethods Following the maintainer's decision that the octet tables exist to present the header structure as the RFCs define it, not to mirror the library's API -- but that the field names in them should still match what the code actually exposes. `` `tcp.opt` `` -> `` `tcp.options` `` (`data/transport/tcp.py:100`, `options: 'OrderedMultiDict[OptionNumber, Option]'`; nothing named `opt` exists) and `` `l2tp.ver` `` -> `` `l2tp.version` `` (`data/link/l2tp.py:44`, `version: 'int'`). The `tcp.options` cell needed re-padding afterwards, since the longer name pushed the description out of its column. `tcp.flags.ns` is a different case and stays. The bit *is* read off the wire -- `schema/transport/tcp.py:58` declares `ns: int` in the flags bitfield -- but the data model and the read site have been commented out in lockstep since `2e39aeb99` ("minor revision for TCP on typings", 2023-06-29): data/transport/tcp.py:52 #ns: 'bool' protocols/transport/tcp.py:447 #ns=bool(schema.offset['ns']), Left commented, deliberately: :rfc:`3540` was reclassified as Historic, and reviving a field nothing consumes adds surface for no gain. The table row is kept because the bit belongs to the header as the RFC defines it -- but it now carries a footnote saying the library does not surface it, so a reader does not go looking for `tcp.flags.ns` and find nothing. `mh.rst` listed `_read_opt_pad` and `_make_opt_pad` **twice each**, which is where two of the tree's duplicate-object warnings came from. Verified there is no `padn` sibling that the second line was meant to be -- Pad1 and PadN deliberately share one handler -- so they were simply stray copies. --- docs/source/pcapkit/protocols/internet/mh.rst | 2 -- docs/source/pcapkit/protocols/link/l2tp.rst | 2 +- docs/source/pcapkit/protocols/transport/tcp.rst | 8 ++++++-- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/source/pcapkit/protocols/internet/mh.rst b/docs/source/pcapkit/protocols/internet/mh.rst index 1f6545a95d..44e89b2e8f 100644 --- a/docs/source/pcapkit/protocols/internet/mh.rst +++ b/docs/source/pcapkit/protocols/internet/mh.rst @@ -77,7 +77,6 @@ Octets Bits Name Description .. automethod:: _read_mh_options .. automethod:: _read_opt_none .. automethod:: _read_opt_pad - .. automethod:: _read_opt_pad .. automethod:: _read_opt_bra .. automethod:: _read_opt_aca .. automethod:: _read_opt_ni @@ -100,7 +99,6 @@ Octets Bits Name Description .. automethod:: _make_mh_options .. automethod:: _make_opt_none .. automethod:: _make_opt_pad - .. automethod:: _make_opt_pad .. automethod:: _make_opt_bra .. automethod:: _make_opt_aca .. automethod:: _make_opt_ni diff --git a/docs/source/pcapkit/protocols/link/l2tp.rst b/docs/source/pcapkit/protocols/link/l2tp.rst index 37f2dee19d..38d18e3129 100644 --- a/docs/source/pcapkit/protocols/link/l2tp.rst +++ b/docs/source/pcapkit/protocols/link/l2tp.rst @@ -32,7 +32,7 @@ as below: ------- ----- --------------------- ------------------------------------------ 1 8 Reserved (must be zero ``x00``) ------- ----- --------------------- ------------------------------------------ - 1 12 ``l2tp.ver`` Version (``2``) + 1 12 ``l2tp.version`` Version (``2``) ------- ----- --------------------- ------------------------------------------ 2 16 ``l2tp.length`` Length (optional by ``len``) ------- ----- --------------------- ------------------------------------------ diff --git a/docs/source/pcapkit/protocols/transport/tcp.rst b/docs/source/pcapkit/protocols/transport/tcp.rst index 29b373aa49..79903343ac 100644 --- a/docs/source/pcapkit/protocols/transport/tcp.rst +++ b/docs/source/pcapkit/protocols/transport/tcp.rst @@ -18,7 +18,7 @@ Octets Bits Name Description 8 64 ``tcp.ack`` Acknowledgement Number (if ACK set) 12 96 ``tcp.hdr_len`` Data Offset 12 100 Reserved (must be ``\x00``) - 12 103 ``tcp.flags.ns`` ECN Concealment Protection (NS) + 12 103 ``tcp.flags.ns`` ECN Concealment Protection (NS) [*]_ 13 104 ``tcp.flags.cwr`` Congestion Window Reduced (CWR) 13 105 ``tcp.flags.ece`` ECN-Echo (ECE) 13 106 ``tcp.flags.urg`` Urgent (URG) @@ -30,7 +30,7 @@ Octets Bits Name Description 14 112 ``tcp.window_size`` Size of Receive Window 16 128 ``tcp.checksum`` Checksum 18 144 ``tcp.urgent_pointer`` Urgent Pointer (if URG set) - 20 160 ``tcp.opt`` TCP Options (if data offset > 5) + 20 160 ``tcp.options`` TCP Options (if data offset > 5) ======= ========= ========================= ======================================= .. autoclass:: pcapkit.protocols.transport.tcp.TCP @@ -532,3 +532,7 @@ Data Models .. rubric:: Footnotes .. [*] https://en.wikipedia.org/wiki/Transmission_Control_Protocol +.. [*] The NS bit is read off the wire but is **not** surfaced in + :class:`~pcapkit.protocols.data.transport.tcp.Flags`. :rfc:`3540` was + reclassified as Historic, so the field is left unexposed; the row is kept + because the bit is part of the header as the RFC defines it. From 2aa9efae2e261a5ea711261c1cdf385446cd7904 Mon Sep 17 00:00:00 2001 From: Jarry Shaw Date: Wed, 16 Sep 2026 12:42:09 -0400 Subject: [PATCH 3/3] docs: correct the reassembly snippets' attribute paths, and flag #415 Both Copilot findings on #413 hold up. The IPv4 and IPv6 reassembly glossary snippets named attributes that do not exist: - `ipv4.header` is not an attribute at all (`hasattr(IPv4, 'header')` is False, and `header` is not a field of the IPv4 data model). `toolkit/pcap.py` uses `ipv4.packet.header` and `bytearray(ipv4.packet.payload)`. - The IPv6 snippet's `ipv6.header` / `ipv6.payload` are really `ipv6_info.fragment.header` / `bytearray(ipv6_info.fragment.payload)`. Both snippets also elided the `.info` hop and wrote `tuple(a, b, c, d)`, which is not how `tuple` is called. They now mirror `toolkit/pcap.py` name for name, with a lead-in saying which object is which. The second finding's other half turned out to be a library defect rather than a comment error: `ihl` and `header` on this path *do* include the Fragment header, because `ipv6.py:341` adds each extension header's length before the Fragment-header check breaks the loop. `dpkt` and `scapy` exclude it and report 40 where this reports 48 for the same packet, and RFC 8200 s4.5 says the Fragment header is absent from a reassembled packet. Filed as #415; the doc now states what the code does today and warns that the value is not comparable across engines, rather than asserting semantics that are about to change. --- .../pcapkit/foundation/reassembly/ip/ipv4.rst | 29 ++++++------ .../pcapkit/foundation/reassembly/ip/ipv6.rst | 44 +++++++++++++------ 2 files changed, 46 insertions(+), 27 deletions(-) diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst index 6e6112530f..dd5524e746 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst @@ -23,24 +23,27 @@ Terminology reasm.ipv4.packet Data structure for **IPv4 datagram reassembly** (:meth:`IPv4.reassembly `) - is as following: + is as following, with ``ipv4`` the protocol instance + (``frame['IPv4']``) and ``ipv4_info`` its :attr:`~pcapkit.protocols.protocol.ProtocolBase.info` + -- the header fields come off the latter, the raw octets off the former: .. code-block:: python packet_dict = dict( - bufid = tuple( - ipv4.src, # source IP address - ipv4.dst, # destination IP address - ipv4.id, # identification - ipv4.protocol, # payload protocol type + bufid = ( + ipv4_info.src, # source IP address + ipv4_info.dst, # destination IP address + ipv4_info.id, # identification + ipv4_info.protocol, # payload protocol type ), - num = frame.number, # original packet range number - fo = ipv4.offset, # fragment offset, in octets - ihl = ipv4.hdr_len, # internet header length - mf = ipv4.flags.mf, # more fragment flag - tl = ipv4.len, # total length, header includes - header = ipv4.header, # raw bytes type header - payload = ipv4.payload, # raw bytearray type payload + num = frame.info.number, # original packet range number + fo = ipv4_info.offset, # fragment offset, in octets + ihl = ipv4_info.hdr_len, # internet header length + mf = ipv4_info.flags.mf, # more fragment flag + tl = ipv4_info.len, # total length, header includes + header = ipv4.packet.header, # raw bytes type header + payload = bytearray( + ipv4.packet.payload), # raw bytearray type payload ) reasm.ipv4.datagram diff --git a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst index e8882e96f8..d7e11ca34e 100644 --- a/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst +++ b/docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst @@ -23,27 +23,43 @@ Terminology reasm.ipv6.packet Data structure for **IPv6 datagram reassembly** (:meth:`IPv6.reassembly `) - is as following: + is as following, with ``ipv6_info`` the IPv6 + :attr:`~pcapkit.protocols.protocol.ProtocolBase.info` and + ``ipv6_frag_info`` the Fragment header's: .. code-block:: python packet_dict = dict( - bufid = tuple( - ipv6.src, # source IP address - ipv6.dst, # destination IP address - ipv6_frag.id, # identification - ipv6_frag.next, # next header field in IPv6 Fragment Header + bufid = ( + ipv6_info.src, # source IP address + ipv6_info.dst, # destination IP address + ipv6_frag_info.id, # identification + ipv6_frag_info.next, # next header field in IPv6 Fragment Header ), - num = frame.number, # original packet range number - fo = ipv6_frag.offset, # fragment offset, in octets - ihl = ipv6.hdr_len, # header length, only headers before IPv6-Frag - mf = ipv6_frag.mf, # more fragment flag - tl = ipv6.hdr_len - + ipv6.raw_len, # total length, header includes - header = ipv6.header, # raw bytes type header before IPv6-Frag - payload = ipv6.payload, # raw bytearray type payload after IPv6-Frag + num = frame.info.number, # original packet range number + fo = ipv6_frag_info.offset, # fragment offset, in octets + ihl = ipv6_info.hdr_len, # header length, IPv6-Frag included + mf = ipv6_frag_info.mf, # more fragment flag + tl = ipv6_info.hdr_len + + ipv6_info.raw_len, # total length, header includes + header = ipv6_info.fragment + .header, # raw bytes type header, IPv6-Frag included + payload = bytearray( + ipv6_info.fragment + .payload), # raw bytearray type payload after IPv6-Frag ) + .. warning:: + + ``ihl`` and ``header`` here **include** the 8-octet Fragment header, + because :attr:`IPv6.hdr_len ` + counts every extension header it has walked, the Fragment one included. + The ``dpkt`` and ``scapy`` adapters stop short of it and report 40 where + this one reports 48 for the same packet, so the value is not comparable + across engines -- and :rfc:`8200#section-4.5` says the Fragment header + is not present in a reassembled packet at all. Tracked as #415; expect + this line to change when that is fixed. + .. note:: The reassembly key is the Fragment header's *Identification*