Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ pyshark = "*"
dpkt = "*"
scapy = "*"
cryptography = "*"
# NB: ``pypcap`` is deliberately absent. It builds a C extension against
# libpcap, so listing it here would break ``pipenv install --dev`` -- and even
# ``pipenv lock``, which has to build its metadata -- on any machine without the
# libpcap development files. Install it by hand to work on that engine:
# ``pipenv run pip install pypcap``.
pypcapfile = "*"
beautifulsoup4 = {extras = ["html5lib"],version = "*"}
requests = {extras = ["socks"],version = "*"}
autopep8 = "*"
Expand Down
42 changes: 29 additions & 13 deletions docs/source/ext.rst
Original file line number Diff line number Diff line change
Expand Up @@ -310,23 +310,39 @@ the network packets for further processing.
The following table shows the available engines and the corresponding supported
file formats:

+---------------------+-----------------------------------------------------+-------------------------------------+
| Engine Type | Engine Class | Supported File Formats |
+=====================+=====================================================+=====================================+
| | :class:`pcapkit.foundation.engines.pcap.PCAP` | PCAP only |
+ Built-in Engines +-----------------------------------------------------+-------------------------------------+
| | :class:`pcapkit.foundation.engines.pcapng.PCAPNG` | PCAP-NG only |
+---------------------+-----------------------------------------------------+-------------------------------------+
| | :class:`pcapkit.foundation.engines.scapy.Scapy` | all formats supported by `Scapy`_ |
+ +-----------------------------------------------------+-------------------------------------+
| Third-party Engines | :class:`pcapkit.foundation.engines.dpkt.DPKT` | all formats supported by `DPKT`_ |
| +-----------------------------------------------------+-------------------------------------+
| | :class:`pcapkit.foundation.engines.pyshark.PyShark` | all formats supported by `PyShark`_ |
+---------------------+-----------------------------------------------------+-------------------------------------+
+---------------------+-----------------------------------------------------------+----------------------------------------+
| Engine Type | Engine Class | Supported File Formats |
+=====================+===========================================================+========================================+
| | :class:`pcapkit.foundation.engines.pcap.PCAP` | PCAP only |
+ Built-in Engines +-----------------------------------------------------------+----------------------------------------+
| | :class:`pcapkit.foundation.engines.pcapng.PCAPNG` | PCAP-NG only |
+---------------------+-----------------------------------------------------------+----------------------------------------+
| | :class:`pcapkit.foundation.engines.scapy.Scapy` | all formats supported by `Scapy`_ |
+ +-----------------------------------------------------------+----------------------------------------+
| | :class:`pcapkit.foundation.engines.dpkt.DPKT` | all formats supported by `DPKT`_ |
| +-----------------------------------------------------------+----------------------------------------+
| Third-party Engines | :class:`pcapkit.foundation.engines.pyshark.PyShark` | all formats supported by `PyShark`_ |
| +-----------------------------------------------------------+----------------------------------------+
| | :class:`pcapkit.foundation.engines.pypcap.PyPCAP` | PCAP only, and from a file on disk |
| +-----------------------------------------------------------+----------------------------------------+
| | :class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` | PCAP only |
+---------------------+-----------------------------------------------------------+----------------------------------------+

.. note::

An engine is free to support less than :class:`~pcapkit.foundation.extraction.Extractor`
offers, and several do. `PyPCAP`_ performs no protocol dissection whatsoever, so
it supports neither reassembly nor flow tracing; `PyPCAPFile`_ has no IPv6
decoder, so it supports IPv4 and TCP reassembly but not IPv6. What matters is
that the gap is *announced* -- each engine warns, or raises, for the capability
it cannot provide, rather than silently producing an empty result. See
:doc:`pcapkit/foundation/engines/index` for the full table.

.. _Scapy: https://scapy.net
.. _DPKT: https://dpkt.readthedocs.io
.. _PyShark: https://kiminewt.github.io/pyshark
.. _PyPCAP: https://github.com/pynetwork/pypcap
.. _PyPCAPFile: https://github.com/kisom/pypcapfile

Samples
~~~~~~~
Expand Down
105 changes: 105 additions & 0 deletions docs/source/pcapkit/foundation/engines/3rdparty.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,108 @@ support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. autoattribute:: _expkg
.. autoattribute:: _extmp

PyPCAP Support
==============

.. module:: pcapkit.foundation.engines.pypcap

This module contains the implementation for `PyPCAP`_ engine
support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. _PyPCAP: https://github.com/pynetwork/pypcap

.. important::

`PyPCAP`_ publishes no wheels, so installing it compiles a C extension and
needs both the :manpage:`libpcap(3)` headers and its shared library on the
system -- a header alone is not enough. It is therefore **not** part of the
``all`` extra, and is installed on its own once libpcap is available:

.. code-block:: shell

pip install pypcapkit[PyPCAP]

.. important::

`PyPCAP`_ is a :manpage:`libpcap(3)` binding aimed primarily at live capture.
Offline it performs **no protocol dissection**: each frame is the
``(timestamp, bytes)`` pair that :c:func:`pcap_next_ex` produced. Reassembly
and flow tracing are therefore unavailable and are disabled -- with an
:class:`~pcapkit.utilities.warnings.AttributeWarning` -- when requested.

The engine also reads PCAP savefiles only, and only from a file on disk.
PCAP-NG is rejected with a
:exc:`~pcapkit.utilities.exceptions.FormatError`, because
:manpage:`libpcap(3)` opens such a file without complaint and then yields no
frames at all; and a non-file input is rejected with an
:exc:`~pcapkit.utilities.exceptions.UnsupportedCall`, because
:c:func:`pcap_open_offline` opens a savefile by *name*.

.. autoclass:: pcapkit.foundation.engines.pypcap.PyPCAP
:no-members:
:show-inheritance:

.. autoattribute:: __engine_name__
.. autoattribute:: __engine_module__

.. autoproperty:: dlink

.. automethod:: run
.. automethod:: read_frame
.. automethod:: close

.. autoattribute:: _expkg
.. autoattribute:: _extmp
.. autoattribute:: _dlink
.. autoattribute:: _closed

PyPCAPFile Support
==================

.. module:: pcapkit.foundation.engines.pypcapfile

This module contains the implementation for `PyPCAPFile`_ engine
support, as is used by :class:`pcapkit.foundation.extraction.Extractor`.

.. _PyPCAPFile: https://github.com/kisom/pypcapfile

.. important::

`PyPCAPFile`_ is a pure Python savefile reader that decodes Ethernet, IPv4,
TCP and UDP and nothing else. IPv6 reassembly is therefore unavailable and is
disabled -- with an :class:`~pcapkit.utilities.warnings.AttributeWarning` --
when requested; IPv4 and TCP reassembly and TCP flow tracing remain
available. PCAP-NG is rejected with a
:exc:`~pcapkit.utilities.exceptions.FormatError`.

.. autoclass:: pcapkit.foundation.engines.pypcapfile.PyPCAPFile
:no-members:
:show-inheritance:

.. autoattribute:: __engine_name__
.. autoattribute:: __engine_module__
.. autoattribute:: LAYERS

.. autoproperty:: dlink

.. automethod:: run
.. automethod:: read_frame

.. autoattribute:: _expkg
.. autoattribute:: _extmp
.. autoattribute:: _dlink
.. autoattribute:: _declf

Internal Definitions
--------------------

.. autoclass:: pcapkit.foundation.engines.pypcapfile._NamedStream
:no-members:
:show-inheritance:

.. autoattribute:: name
.. automethod:: read

.. automethod:: pcapkit.foundation.engines.pypcapfile.PyPCAPFile._get_decoder
.. automethod:: pcapkit.foundation.engines.pypcapfile.PyPCAPFile._decode
31 changes: 30 additions & 1 deletion docs/source/pcapkit/foundation/engines/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Engine Support
:mod:`pcapkit.foundation.engines` is a collection of engines
support for :mod:`pcapkit`, including but not limited to the
built-in PCAP and `PCAP-NG`_ file support, `Scapy`_, `PyShark`_,
and `DPKT`_ 3rd party engine support.
`DPKT`_, `PyPCAP`_ and `PyPCAPFile`_ 3rd party engine support.

.. seealso::

Expand Down Expand Up @@ -44,6 +44,8 @@ class hierarchy of :mod:`pcapkit.foundation.engines`:
Scapy
DPKT
PyShark
PyPCAP
PyPCAPFile
end
B --> third-party

Expand All @@ -61,9 +63,36 @@ class hierarchy of :mod:`pcapkit.foundation.engines`:
click Scapy "/pcapkit/foundation/engines/3rdparty.html#pcapkit.foundation.engines.scapy.Scapy"
click DPKT "/pcapkit/foundation/engines/3rdparty.html#pcapkit.foundation.engines.dpkt.DPKT"
click PyShark "/pcapkit/foundation/engines/3rdparty.html#pcapkit.foundation.engines.pyshark.PyShark"
click PyPCAP "/pcapkit/foundation/engines/3rdparty.html#pcapkit.foundation.engines.pypcap.PyPCAP"
click PyPCAPFile "/pcapkit/foundation/engines/3rdparty.html#pcapkit.foundation.engines.pypcapfile.PyPCAPFile"

Not every engine can do everything :class:`~pcapkit.foundation.extraction.Extractor`
offers, and the ones that cannot say so rather than quietly doing less:

+-----------------------------------------------------------------+---------------------------------------------------------------+
| Engine | Gap, and how it is surfaced |
+=================================================================+===============================================================+
| :class:`~pcapkit.foundation.engines.pyshark.PyShark` | no reassembly -- disabled with an |
| | :class:`~pcapkit.utilities.warnings.AttributeWarning` |
+-----------------------------------------------------------------+---------------------------------------------------------------+
| :class:`~pcapkit.foundation.engines.pypcap.PyPCAP` | no protocol dissection at all, hence no reassembly and no |
| | flow tracing -- both disabled with an |
| | :class:`~pcapkit.utilities.warnings.AttributeWarning`; PCAP |
| | savefiles on disk only, otherwise |
| | :exc:`~pcapkit.utilities.exceptions.FormatError` / |
| | :exc:`~pcapkit.utilities.exceptions.UnsupportedCall` |
+-----------------------------------------------------------------+---------------------------------------------------------------+
| :class:`~pcapkit.foundation.engines.pypcapfile.PyPCAPFile` | no IPv6 decoder, hence no IPv6 reassembly -- disabled with an |
| | :class:`~pcapkit.utilities.warnings.AttributeWarning`, and |
| | :func:`~pcapkit.toolkit.pypcapfile.ipv6_reassembly` raises; |
| | PCAP savefiles only, otherwise |
| | :exc:`~pcapkit.utilities.exceptions.FormatError` |
+-----------------------------------------------------------------+---------------------------------------------------------------+

.. _PCAP-NG: https://wiki.wireshark.org/Development/PcapNg

.. _Scapy: https://scapy.net
.. _DPKT: https://dpkt.readthedocs.io
.. _PyShark: https://kiminewt.github.io/pyshark
.. _PyPCAP: https://github.com/pynetwork/pypcap
.. _PyPCAPFile: https://github.com/kisom/pypcapfile
82 changes: 82 additions & 0 deletions docs/source/pcapkit/toolkit/3rdparty.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,85 @@ Auxiliary Functions
-------------------

.. autofunction:: pcapkit.toolkit.pyshark.packet2dict

PyPCAP Tools
============

.. module:: pcapkit.toolkit.pypcap

:mod:`pcapkit.toolkit.pypcap` contains all you need for
:mod:`pcapkit` handy usage with `PyPCAP`_ engine. All reforming
functions returns with a flag to indicate if usable for
its caller.

.. _PyPCAP: https://github.com/pynetwork/pypcap

.. note::

`PyPCAP`_ performs no protocol dissection, so the reassembly and flow tracing
adapters below cannot be implemented. They are defined all the same, so that
reaching for one fails with an explanatory
:exc:`~pcapkit.utilities.exceptions.UnsupportedCall` rather than an
:exc:`ImportError`.

.. autofunction:: pcapkit.toolkit.pypcap.ipv4_reassembly

.. autofunction:: pcapkit.toolkit.pypcap.ipv6_reassembly

.. autofunction:: pcapkit.toolkit.pypcap.tcp_reassembly

.. autofunction:: pcapkit.toolkit.pypcap.tcp_traceflow

Auxiliary Functions
-------------------

.. autofunction:: pcapkit.toolkit.pypcap.packet2chain

.. autofunction:: pcapkit.toolkit.pypcap.packet2dict

PyPCAPFile Tools
================

.. module:: pcapkit.toolkit.pypcapfile

:mod:`pcapkit.toolkit.pypcapfile` contains all you need for
:mod:`pcapkit` handy usage with `PyPCAPFile`_ engine. All reforming
functions returns with a flag to indicate if usable for
its caller.

.. _PyPCAPFile: https://github.com/kisom/pypcapfile

.. note::

`PyPCAPFile`_ has no IPv6 decoder, so :func:`~pcapkit.toolkit.pypcapfile.ipv6_reassembly`
raises :exc:`~pcapkit.utilities.exceptions.UnsupportedCall` rather than returning
:data:`None` -- which would be indistinguishable from "this frame carries no
IPv6 fragment".

.. autofunction:: pcapkit.toolkit.pypcapfile.ipv4_reassembly

.. autofunction:: pcapkit.toolkit.pypcapfile.ipv6_reassembly

.. autofunction:: pcapkit.toolkit.pypcapfile.tcp_reassembly

.. autofunction:: pcapkit.toolkit.pypcapfile.tcp_traceflow

Auxiliary Functions
-------------------

.. autofunction:: pcapkit.toolkit.pypcapfile.packet2timestamp

.. autofunction:: pcapkit.toolkit.pypcapfile.ipv4_header

.. autofunction:: pcapkit.toolkit.pypcapfile.packet2chain

.. autofunction:: pcapkit.toolkit.pypcapfile.packet2dict

Internal Definitions
--------------------

.. autodata:: pcapkit.toolkit.pypcapfile.TCP_MIN_HEADER_LEN

.. autodata:: pcapkit.toolkit.pypcapfile.IPV4_FLAG_DF

.. autodata:: pcapkit.toolkit.pypcapfile.IPV4_FLAG_MF
58 changes: 51 additions & 7 deletions docs/source/pep.rst
Original file line number Diff line number Diff line change
Expand Up @@ -110,15 +110,49 @@ PyPCAPKit.
New Engines
-----------

Although PyPCAPKit already has support for some popular PCAP parsing libraries,
I'm expecting to extend the list of supported engines furthermore. The candidate
engines include:

- `pypcap <https://github.com/pynetwork/pypcap>`__
- `pycapfile <https://github.com/kisom/pypcapfile>`__

.. note::

**Done**, for both candidates. ``engine='pypcapfile'`` selects
:class:`pcapkit.foundation.engines.pypcapfile.PyPCAPFile` and
``engine='pypcap'`` selects :class:`pcapkit.foundation.engines.pypcap.PyPCAP`;
each has a matching :mod:`pcapkit.toolkit` module
(:mod:`pcapkit.toolkit.pypcapfile`, :mod:`pcapkit.toolkit.pypcap`), a
``pyproject.toml`` extra (``PyPCAPFile``, which ``all`` includes, and
``PyPCAP``, which it deliberately does not -- see below), docs
under :doc:`pcapkit/foundation/engines/index`, and tests under
``tests/foundation/engines/`` and ``tests/toolkit/``. Both were verified
end-to-end against the sample captures: each agrees with the ``default`` engine
on frame count, per-record capture length, timestamp and Ethernet header.

Neither library, though, is usable straight from PyPI on a current Python, and
both of the following are worth knowing before reaching for them:

* **pypcapfile** -- the released 0.12.0 imports the :mod:`imp` module, removed
in Python 3.12, so it cannot be imported at all on 3.12 or newer. Upstream
``master`` (0.12.1, unreleased) has fixed this, and that is what the engine
was verified against. The extra therefore installs a version that only works
on Python 3.10 and 3.11 until 0.12.1 is published.
* **pypcap** -- the 1.3.0 sdist ships a pre-generated ``pcap.c`` from an old
Cython, which no longer compiles against the Python 3.12+ C API
(``ob_digit``, ``_PyLong_AsByteArray``), and its ``setup.py`` searches a
fixed list of prefixes for ``pcap.h``. It builds once :program:`libpcap`'s
headers are visible under ``sys.prefix`` and ``pcap.c`` is regenerated with
Cython 3. That is a packaging problem upstream rather than an engine problem,
but it does mean ``pip install pypcapkit[PyPCAP]`` can fail to build. Since
there is no wheel to fall back on, the extra is kept **out of** ``all``:
otherwise ``pip install pypcapkit[all]`` would demand a compiler and the
libpcap development files from every user, and it broke the docs, conda and
release workflows -- all of which install ``.[all]`` -- on the macOS runner,
where :file:`pcap.h` is present but no ``libpcap.dylib`` is.

Both engines support less than the ``default`` engine does, deliberately and
noisily: `pypcap`_ performs no protocol dissection, so it disables reassembly
*and* flow tracing; `pypcapfile`_ has no IPv6 decoder, so it disables IPv6
reassembly while keeping IPv4 and TCP. Each gap is announced through an
:class:`~pcapkit.utilities.warnings.AttributeWarning` or an outright exception
rather than by silently returning nothing --
:doc:`pcapkit/foundation/engines/index` tabulates them.

The engine interface has since been refactored, so this no longer means adding
handler methods to :class:`~pcapkit.foundation.extraction.Extractor`. A new
engine subclasses :class:`pcapkit.foundation.engines.engine.Engine` and
Expand All @@ -128,6 +162,16 @@ engines include:
still apply is the unified auxiliary tools in :mod:`pcapkit.toolkit`, where
each engine has a matching module.

Originally: although PyPCAPKit already has support for some popular PCAP parsing
libraries, I'm expecting to extend the list of supported engines furthermore. The
candidate engines include:

- `pypcap <https://github.com/pynetwork/pypcap>`__
- `pypcapfile <https://github.com/kisom/pypcapfile>`__

.. _pypcap: https://github.com/pynetwork/pypcap
.. _pypcapfile: https://github.com/kisom/pypcapfile

Test Cases
----------

Expand Down
Loading