Skip to content

foundation: give IP reassembly the RFC timeout, and trace TCP flows bidirectionally #435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JarryShaw merged 8 commits into from
Sep 17, 2026
Merged

foundation: give IP reassembly the RFC timeout, and trace TCP flows bidirectionally #435

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5aebe48
foundation: give IP reassembly the RFC timeout and trace TCP flows bi…
JarryShaw Sep 17, 2026
7f7c6a7
traceflow: keep the __init__ stubs on one line, where their pylint di…
JarryShaw Sep 17, 2026
f5fd367
traceflow: a teardown is not the end of a flow; correct the IP loose-…
JarryShaw Sep 17, 2026
21e6118
docs: drop backticks from inside the buffer diagram, where they rende…
JarryShaw Sep 17, 2026
a90db24
docs: drop a duplicated paragraph from the flow-tracing bullet
JarryShaw Sep 17, 2026
7ce8cda
traceflow: wire the application layer in; stop the DPKT engine discar…
JarryShaw Sep 17, 2026
4daf2bf
traceflow: state what analyse=True actually returns, and pin it
JarryShaw Sep 17, 2026
c8e72b8
traceflow: the ACK bucket boundary is the peer's turn, not a message …
JarryShaw Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions docs/source/pcapkit/foundation/reassembly/ip/ipv4.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ Terminology
header = ipv4.packet.header, # raw bytes type header
payload = bytearray(
ipv4.packet.payload), # raw bytearray type payload
timestamp = float(
frame.info.time_epoch), # capture timestamp
)

reasm.ipv4.datagram
Expand All @@ -55,7 +57,7 @@ Terminology

(tuple) datagram
|--> (Info) data
| |--> 'completed' : (bool) True --> implemented
| |--> 'completed' : (Completion) COMPLETE --> reassembled in whole
| |--> 'id' : (Info) original packet identifier
| | |--> 'src' --> (IPv4Address) ipv4.src
| | |--> 'dst' --> (IPv4Address) ipv4.dst
Expand All @@ -67,7 +69,7 @@ Terminology
| |--> 'payload' : (bytes) reassembled IPv4 payload
| |--> 'packet' : (Protocol) parsed reassembled payload
|--> (Info) data
| |--> 'completed' : (bool) False --> not implemented
| |--> 'completed' : (Completion) PARTIAL or TIMEOUT --> incomplete
| |--> 'id' : (Info) original packet identifier
| | |--> 'src' --> (IPv4Address) ipv4.src
| | |--> 'dst' --> (IPv4Address) ipv4.dst
Expand Down Expand Up @@ -115,4 +117,20 @@ Terminology
| | |--> (int) packet range number
| |--> 'header' : (bytes) header buffer
| |--> 'datagram' : (bytearray) data buffer, holes set to b'\\x00'
| |--> 'timestamp' : (float) capture timestamp of the
| first-arriving fragment
|--> (tuple) BUFID ...

.. note::

A buffer is abandoned once the reassembly timeout elapses on the
*capture's* clock -- 60 seconds by default, per
:rfc:`1122#section-3.3.2` for IPv4 and :rfc:`8200#section-4.5` for
IPv6, counted from the first-arriving fragment. Its datagram is
reported with ``completed`` set to
:attr:`Completion.TIMEOUT <pcapkit.foundation.reassembly.data.data.Completion.TIMEOUT>`
rather than
:attr:`~pcapkit.foundation.reassembly.data.data.Completion.PARTIAL`,
which is what tells "these fragments are gone" apart from "these
fragments had not arrived yet". See
:meth:`~pcapkit.foundation.reassembly.reassembly.ReassemblyBase.expire`.
6 changes: 4 additions & 2 deletions docs/source/pcapkit/foundation/reassembly/ip/ipv6.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ Terminology
header = ipv6_info.fragment
.header[:hdr_len], # raw bytes type header before IPv6-Frag
payload = payload, # raw bytearray type payload after IPv6-Frag
timestamp = float(
frame.info.time_epoch), # capture timestamp
)

.. note::
Expand Down Expand Up @@ -77,7 +79,7 @@ Terminology

(tuple) datagram
|--> (Info) data
| |--> 'completed' : (bool) True --> implemented
| |--> 'completed' : (Completion) COMPLETE --> reassembled in whole
| |--> 'id' : (Info) original packet identifier
| | |--> 'src' --> (IPv6Address) ipv6.src
| | |--> 'dst' --> (IPv6Address) ipv6.dst
Expand All @@ -89,7 +91,7 @@ Terminology
| |--> 'payload' : (bytes) reassembled IPv6 payload
| |--> 'packet' : (Protocol) parsed reassembled payload
|--> (Info) data
| |--> 'completed' : (bool) False --> not implemented
| |--> 'completed' : (Completion) PARTIAL or TIMEOUT --> incomplete
| |--> 'id' : (Info) original packet identifier
| | |--> 'src' --> (IPv6Address) ipv6.src
| | |--> 'dst' --> (IPv6Address) ipv6.dst
Expand Down
5 changes: 5 additions & 0 deletions docs/source/pcapkit/foundation/reassembly/reassembly.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,11 @@ implements datagram reassembly of IP and TCP packets.

.. autoproperty:: count
.. autoproperty:: datagram
.. autoproperty:: timeout

.. automethod:: reassembly
.. automethod:: submit
.. automethod:: expire
.. automethod:: fetch
.. automethod:: index
.. automethod:: run
Expand All @@ -58,6 +60,8 @@ implements datagram reassembly of IP and TCP packets.
:no-value:
.. autoattribute:: _flag_n
:no-value:
.. autoattribute:: _timeout
:no-value:

.. autoattribute:: _buffer
:no-value:
Expand All @@ -69,6 +73,7 @@ implements datagram reassembly of IP and TCP packets.

.. autoattribute:: __protocol_name__
.. autoattribute:: __protocol_type__
.. autoattribute:: __timeout__

Internal Definitions
--------------------
Expand Down
19 changes: 17 additions & 2 deletions docs/source/pcapkit/foundation/reassembly/tcp.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,8 @@ Terminology
# last sequence number of payload
header = tcp.packet.header, # raw bytes type header
payload = tcp.raw, # raw bytearray type payload
timestamp = float(
frame.time_epoch), # capture timestamp
)

Both ``first`` and ``last`` are absolute TCP sequence numbers and
Expand All @@ -190,7 +192,7 @@ Terminology

(tuple) datagram
|--> (Info) data
| |--> 'completed' : (bool) True --> implemented
| |--> 'completed' : (Completion) COMPLETE --> reassembled in whole
| |--> 'id' : (Info) original packet identifier
| | |--> 'src' --> (tuple)
| | | |--> (IPv4Address) ip.src
Expand All @@ -206,7 +208,7 @@ Terminology
| |--> 'payload' : (bytes) reassembled payload
| |--> 'packet' : (Protocol) parsed reassembled payload
|--> (Info) data
| |--> 'completed' : (bool) False --> not implemented
| |--> 'completed' : (Completion) PARTIAL or TIMEOUT --> incomplete
| |--> 'id' : (Info) original packet identifier
| | |--> 'src' --> (tuple)
| | | |--> (IPv4Address) ip.src
Expand Down Expand Up @@ -256,8 +258,21 @@ Terminology
| | holes set to b'\x00'
| |--> (int) ACK ...
| |--> ...
| |--> 'timestamp' : (float) capture timestamp of the
| first segment buffered
|--> (tuple) BUFID ...

.. note::

TCP reassembly has **no** timeout by default: no specification gives
stream reassembly a deadline the way :rfc:`1122#section-3.3.2` and
:rfc:`8200#section-4.5` give IP fragmentation one, and an idle
connection is ordinary rather than pathological. ``timestamp`` is
recorded regardless, so passing ``timeout`` to
:class:`~pcapkit.foundation.reassembly.tcp.TCP` enables the same
eviction the IP reassemblers use -- see
:attr:`TCP.__timeout__ <pcapkit.foundation.reassembly.tcp.TCP.__timeout__>`.

The hole descriptor list is kept in **absolute TCP sequence numbers**,
once per ``BUFID``, whereas each ACK's payload buffer is indexed from
its own ``isn`` -- ``raw[n]`` holds the octet with sequence number
Expand Down
8 changes: 8 additions & 0 deletions docs/source/pcapkit/foundation/traceflow/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,11 @@ Auxiliary Data
.. autoclass:: pcapkit.foundation.traceflow.data.data.TraceFlowData
:members:
:show-inheritance:

.. autoclass:: pcapkit.foundation.traceflow.data.data.Deferred
:members:
:show-inheritance:

.. autoclass:: pcapkit.foundation.traceflow.data.data.DeferredPacket
:members:
:show-inheritance:
68 changes: 64 additions & 4 deletions docs/source/pcapkit/foundation/traceflow/tcp.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ TCP flows from a series of packets and connections.
.. autoproperty:: protocol

.. automethod:: dump
.. automethod:: make_bufid
.. automethod:: trace
.. automethod:: finish
.. automethod:: submit

.. autoattribute:: __protocol_name__
Expand All @@ -38,6 +40,12 @@ Terminology
frame=frame.info, # extracted frame info
syn=tcp.flags.syn, # TCP synchronise (SYN) flag
fin=tcp.flags.fin, # TCP finish (FIN) flag
rst=tcp.flags.rst, # TCP reset (RST) flag
seq=tcp.seq, # TCP sequence number
ack=tcp.ack, # TCP acknowledgement number
header=tcp.packet.header, # raw bytes type header
payload=bytearray(
tcp.packet.payload), # raw bytearray type payload
src=ip.src, # source IP
dst=ip.dst, # destination IP
srcport=tcp.srcport, # TCP source port
Expand All @@ -61,11 +69,40 @@ Terminology
| |--> ip.dst |
| |--> tcp.dstport |
| |--> 'fpout' : (dictdumper.dumper.Dumper) output dumper object
| |--> 'index': (list) list of frame index
| |--> 'index': (list) list of frame index, both directions
| | |--> (int) frame index
| |--> 'label': (str) flow label generated from ``BUFID``
| |--> 'label': (str) flow label generated from the packet
| | that opened the flow
| |--> 'origin': (tuple) (address, port) of the endpoint
| | that opened the flow
| |--> 'forward': (list) frame index sent by 'origin'
| |--> 'reverse': (list) frame index sent to 'origin'
| |--> 'fin': (set) endpoints seen to have sent a FIN
| |--> 'reset': (bool) whether a RST has been seen
| |--> 'reassembly': (Optional[TCP]) the flow's own
| reassembler, or None when
| analyse is off
|--> (tuple) BUFID ...

When tracing bidirectionally -- the default -- ``BUFID`` orders the two
endpoints canonically rather than as (source, destination), so both halves
of a conversation reduce to the same key. It stays a plain :obj:`tuple`
either way, because it is a :obj:`dict` key and an
:class:`~pcapkit.corekit.infoclass.Info` cannot be one --
:class:`collections.abc.Mapping` sets its ``__hash__`` to :data:`None`.

A teardown -- a FIN from each endpoint, or a RST from either -- is recorded
in ``fin`` and ``reset`` but does **not** finalise the flow. The four-way
close of :rfc:`9293#section-3.6` is FIN, ACK, FIN, ACK, so the
acknowledgement that completes it arrives after the second FIN; finalising
on that FIN would drop the ACK from the flow and let it open a fresh buffer
under the same ``BUFID``, which a later connection reusing those endpoints
would then merge into. The flow is finalised instead by proof that nothing
more can arrive -- a new connection's SYN on the same endpoints, or
:meth:`TCP.finish <pcapkit.foundation.traceflow.tcp.TCP.finish>` at the end
of the capture. Telling that SYN from the peer's SYN-ACK is what the
recorded teardown is for.

.. seealso:: :class:`pcapkit.foundation.traceflow.data.tcp.Buffer`

trace.tcp.index
Expand All @@ -78,11 +115,34 @@ Terminology
(tuple) index
|--> (Info) data
| |--> 'fpout' : (Optional[str]) output filename if exists
| |--> 'index': (tuple) tuple of frame index
| |--> 'index': (tuple) tuple of frame index, both directions,
| | in capture order
| | |--> (int) frame index
| |--> 'label': (str) flow label generated from ``BUFID``
| |--> 'label': (str) flow label generated from the packet that
| | opened the flow
| |--> 'forward': (tuple) frame index in the direction that
| | opened the flow
| |--> 'reverse': (tuple) frame index the other way; empty when
| | tracing unidirectionally
| |--> 'packet': (Optional[tuple]) one reassembled datagram per
| direction, or None when analyse is off
|--> (Info) data ...

``forward`` and ``reverse`` partition ``index``, so
``frame_number in flow.forward`` answers which way a packet went without
taking the label apart. ``forward`` is the direction of the packet that
opened the flow, whose endpoints the label names first.

``packet`` is the conversation's application layer: one reassembled datagram
per direction, present only when the tracer was constructed with
``analyse=True``. It is reassembled on the *first read*, and each datagram's
own :attr:`~pcapkit.foundation.reassembly.data.tcp.Datagram.packet` is
parsed later still, so a caller that wanted only frame numbers pays for
neither. The tracer does not reassemble the stream itself -- it feeds
:class:`~pcapkit.foundation.reassembly.tcp.TCP`, whose :rfc:`815` algorithm
handles the reordering and retransmission that concatenating payloads in
capture order would corrupt.

.. seealso:: :class:`pcapkit.foundation.traceflow.data.tcp.Index`

Data Structures
Expand Down
3 changes: 3 additions & 0 deletions docs/source/pcapkit/foundation/traceflow/traceflow.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ which is an abstract base class for all flow tracing classes.

.. automethod:: dump
.. automethod:: trace
.. automethod:: finish
.. automethod:: submit

.. autoattribute:: __output__
Expand All @@ -55,6 +56,8 @@ which is an abstract base class for all flow tracing classes.
:no-value:
.. autoattribute:: _stream
:no-value:
.. autoattribute:: _bidir
:no-value:

.. automethod:: __call__
.. automethod:: __init_subclass__
Expand Down
Loading
Loading