Skip to content

docs: record the protocol, reassembly and crypto gaps on the Help Wanted page - #419

Merged
JarryShaw merged 3 commits into
mainfrom
docs/record-known-gaps
Sep 16, 2026
Merged

JarryShaw merged 3 commits into
mainfrom
docs/record-known-gaps

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

Catalogues what pcapkit does not implement, so the gaps are trackable instead of rediscovered. Every entry is verified against the code — a file:line or a specific missing name and number, never an inference from the docs.

docs/source/pep.rst only, +205/−13. Docs build: 91 warning/error lines before, 91 after, the two sets byte-identical and none naming pep.rst.

One correction to the commit message. It says the after-count was "still running", because it was when the commit was written. The confirmed 91/91 above supersedes that line; correcting it in place would have needed a force-push, which is not worth it.

Protocols

  • DTLS has no dissector and no stub anywhere in the tree (find -iname '*dtls*' returns nothing; only TLS/SSL has one), yet it is named by SCTP chunk type 65, parameter 0x8006, error causes 100–103, and PPIDs 47/66/67/68/69/4242. All of it parses as opaque bytes. This is what blocks NGAP over PPID 66 (protocols: implement NGAP over SCTP, decoding aligned PER through pycrate #417), not NGAP itself.
  • Registry values with no dissector behind them, measured by importing the tree rather than counted by eye: LinkType 3 of 219 · TransType 15 of 151 · EtherType 6 of 160 · AppType 2 ports of 8182 · SCTP PPID 0 of 75 (SCTP.__proto__ is literally {} at transport/sctp.py:344-347, with no in-tree register_sctp caller).
  • Orphaned dissectors — OSPF and L2TP are implemented but bound to nothing (TransType 89 and 115 unbound, and both make __index__ raise). FTP_DATA exists with TCP port 20 unbound. HTTP is registered on port 80 only. EtherType 0x88A8 is unbound although VLAN already parses that shape.

Reassembly and analysis

  • SCTP reassembly is entirely absent — grep -rni sctp pcapkit/foundation/ returns nothing — even though every field a reassembler needs is already exposed at sctp.py:1143-1157. Today only the first DATA chunk of a bundle is dispatched (sctp.py:517-524), and the only readers of flags.B/flags.E are in _make_chunk_data.
  • I-DATA (RFC 8260, chunk type 64) is undissected: SCTP.__chunk__[Chunk(64)] is 'donone' and read() matches only Payload_Data, so an I-DATA-only packet yields no payload at all. FORWARD TSN 192 and I-FORWARD-TSN 194 likewise.
  • Registering a fourth reassembler is inert. register_reassembly accepts any name, but Extractor instantiates three literal keys and ReassemblyManager is a fixed-field Info — verified by the TypeError on sctp=. So SCTP reassembly needs plumbing, not just an algorithm.
  • Flow tracing is TCP-only, closes on FIN but not RST, and flows are unidirectional. No reassembly timeout or eviction exists anywhere and the buffer models carry no timestamp, while 72 KiB is preallocated per in-flight IP datagram id.

ESP cryptographic coverage

36 Cipher and 15 Integrity members against 5 suites each — and two of those ten are the no-ops ENCR_NULL and NONE. The page's existing "not implemented" list was incomplete; it now names AES-CTR, AES-CMAC-96, the AES-GMAC family, Camellia, RFC 8750 implicit-IV and RFC 9227 MGM, plus the two structural blockers: IntegritySuite stores its digest as a hashlib constructor name, and load_cryptography imports only the ciphers submodules.

Corrections to the page itself

The audit contradicted two things pep.rst previously asserted:

  • The MH # TODO markers are not at the dispatch tables. All six sit at the end of each handler block (mh.py:1510,2310,2422,2986,3757,3871). The counts themselves check out (24/14, 71/20, 4/1), read and make are symmetric at 37 each, and mh.py:32-78 already imports 32 const/mh sub-registry enums it never uses — so the enumerations for the missing options exist already.
  • Items checked and dismissed as already done rather than re-solicited: the 33-file stub list, the NDP/QUIC corrections, SCTP/ESP/PCAP-NG/logging/engines, and the module count. No read/make asymmetry exists anywhere in pcapkit/protocols/ — every UnsupportedCall there is a deliberate design decision.

Discussion posts

Comment Placement
-18470228 in-thread under More Protocols, More!!! — DTLS, registered-but-not-dissected, ESP/MH corrections
-18470233 new top-level, Reassembly Beyond IP and TCP
-18470238 #251 — DTLS blocks PPID 66

The new top-level comment is justified rather than lazy: #106's seven topics are protocols, PCAP-NG, performance, logging, engines, tests and a progress meta-comment, none of which covers reassembly or flow tracing, and the post says so in its first line. Everything else went in-thread, per convention.

Note the posts deep-link to #dtls, #registered-but-not-dissected and #reassembly-beyond-ip-and-tcp, which do not exist on the live docs until this merges and Pages redeploys — the links resolve to the page meanwhile, just not to the section.

Seven code defects found while auditing, none fixed here

Filed separately rather than smuggled into a docs PR. The first is the most consequential:

  1. __proto__ registry pollution — protocols/protocol.py:1300 indexes a defaultdict, so a lookup miss inserts the key. Reproduced: parsing one 24-byte IPv4 packet with proto=1 inserts TransType.ICMP -> Raw into the class-level, shared Internet.__proto__, after which register_transtype(TransType.ICMP, …) warns "already registered, overwriting". SCTP._import_next_layer overrides the base method precisely to avoid this and documents why at sctp.py:818-831 — a ready template for Frame, PCAPNG, Link, Internet, IPv6, TCP and UDP.
  2. beholder's Raw fallback on payload-less schemas — already fixed on protocols: implement NGAP over SCTP, decoding aligned PER through pycrate #417, so main only.
  3. register_apptype cannot register an SCTP-only service, raising RegistryError for ~41 AppType members whose proto is SCTP- or DCCP-only, while register_sctp is PPID-keyed.
  4. ESP AEAD dispatch falls through — anything not ENCR_NULL or ENCR_AES_CBC is treated as AES-GCM. Safe today; adding AES-CCM or ChaCha20 to CIPHER_SUITES without touching decrypt/encrypt would silently mis-process it.
  5. Cipher.get()/Integrity.get() invent members via extend_enum(…, -1) for an unknown string.
  6. vendor/pcapng/filter_type.py:25 has DATA = {} with a TODO URL, so the generated enum has no members.
  7. DPKT/Scapy trace output hits the mapping-vs-frame defect extraction.py:879-881 documents and excludes; interface: add the three missing engine constants, and guard dpkt/scapy tracing #412 is in flight over that area.

…ted page

Catalogues what PyPCAPKit does not yet implement, so the gaps are tracked
rather than rediscovered. Every entry below was verified against the code at
5e4378d; nothing here is inferred from documentation.

New sections in docs/source/pep.rst
-----------------------------------

* "DTLS" (under More Protocols, More!!!) -- no dissector and no stub anywhere
  in the tree; only TLS/SSL has a stub. The registries have moved ahead of it:
  SCTP DTLS chunk type 65 (pcapkit/const/sctp/chunk.py:77), DTLS Key Management
  chunk parameter 0x8006 (const/sctp/parameter.py:86), error causes 100-103
  (const/sctp/cause_code.py:62,66,70,74), and seven PPIDs -- 47, 66, 67, 68, 69
  and 4242 (const/sctp/payload_protocol_identifier.py:181,240,243,246,249,265).
  DTLS blocks NGAP over DTLS (PPID 66) independently of NGAP itself.

* "Registered, But Not Dissected" (under More Protocols, More!!!) -- registry
  values with no dissector behind them, all counts measured by importing the
  worktree copy:
    - LinkType 3 of 219 bound (ETHERNET 1, IPV4 228, IPV6 229), declared twice:
      pcapkit/protocols/misc/pcap/frame.py:89-96 and misc/pcapng.py:555-562
    - TransType 15 of 151 bound, pcapkit/protocols/internet/internet.py:89-107
    - EtherType 6 of 160 bound, pcapkit/protocols/link/link.py:74-87
    - AppType 2 port numbers of 8182 -- TCP 21, TCP 80 (transport/tcp.py:309-315)
      and UDP 80 (transport/udp.py:72-78)
    - SCTP PPID 0 of 75; SCTP.__proto__ ships empty, transport/sctp.py:344-347,
      and no in-tree caller of register_sctp exists
  Cheap wins named: OSPF (link/ospf.py:70) and L2TP (link/l2tp.py:76) are
  implemented but bound to nothing (TransType 89, 115 unbound; both make
  __index__ raise at ospf.py:229-237 / l2tp.py:231-239); FTP_DATA
  (application/ftp.py:195) exists but TCP 20 is unbound; HTTP is bound on port
  80 only; EtherType 0x88A8 unbound though VLAN parses that shape; LinkType
  NULL/LOOP/RAW carry bare IPv4/IPv6 which pcapkit dissects.

* "Reassembly Beyond IP and TCP" (new top-level section) -- SCTP has no
  reassembly; grep -rni sctp pcapkit/foundation/ returns nothing.
    - The DATA chunk fields a reassembler needs are all exposed already:
      tsn/stream_id/stream_seq/ppid/data and flags.B/flags.E at
      protocols/transport/sctp.py:1143-1157; the only readers of B/E are in
      _make_chunk_data (sctp.py:1725-1726), i.e. the construction path.
    - Only the first DATA chunk of a bundle is dispatched, sctp.py:517-524.
    - I-DATA (RFC 8260, chunk 64) is not dissected -- SCTP.__chunk__[Chunk(64)]
      is 'donone'; read() matches only Payload_Data (sctp.py:520), so an
      I-DATA-only packet yields no payload at all. FORWARD TSN 192 and
      I-FORWARD-TSN 194 likewise unhandled.
    - Where the work goes: pcapkit/foundation/reassembly/sctp.py, subclassing
      Reassembly and implementing reassembly() and submit() (the only two
      abstract methods; reassembly/reassembly.py:180-195). Registering it is
      inert today -- Extractor.register_reassembly accepts any name
      (extraction.py:396-416) but Extractor instantiates only three literal
      keys (extraction.py:840,848,856) and ReassemblyManager is a fixed-field
      Info container (reassembly/__init__.py:37-49). Also needs the read-side
      twin (reassembly/data/__init__.py:37-48), five toolkit adapters, five
      engine call sites, and interface/core.py:148-171.
    - Flow tracing is TCP only (Extractor.__traceflow__ has one key;
      TraceFlowManager one field). TCP tracer closes on FIN not RST -- rst is
      absent from traceflow/data/tcp.py:43-46 -- and flows are unidirectional
      (traceflow/tcp.py:99).
    - No reassembly timeout or eviction anywhere in pcapkit/foundation/; the
      buffer models carry no timestamp (reassembly/data/ip.py:29-51,
      data/tcp.py:29-61). 8191 + 65535 bytes preallocated per in-flight IP
      datagram id (reassembly/ip.py:82-88).
    - Engines: pyshark disables reassembly (engines/pyshark.py:207-213);
      pypcap (pypcap.py:295-306) and pcap_ct (pcap_ct.py:380-391) disable
      reassembly and traceflow; pypcapfile disables IPv6 (pypcapfile.py:225-229,
      toolkit/pypcapfile.py:348-350).

Corrections to text that was already on the page
------------------------------------------------

* ESP. "Only five encryption and five integrity algorithms" is right (verified:
  36 Cipher members, 15 Integrity, 5 entries each in CIPHER_SUITES at
  esp.py:405-426 and INTEGRITY_SUITES at esp.py:432-448) but two of the ten are
  the no-ops ENCR_NULL and NONE. The "not implemented" list was incomplete;
  added AES-CTR (RFC 3686), AES-CMAC-96 (RFC 4494), the AES-GMAC family
  (RFC 4543) with ENCR_NULL_AUTH_AES_GMAC, and the Camellia, RFC 8750
  implicit-IV and RFC 9227 MGM families. Added the two structural blockers:
  IntegritySuite records the digest as a hashlib constructor name
  (esp.py:360-362, 687-693) so a non-HMAC MAC needs the record widened, and
  load_cryptography imports only the ciphers submodules (esp.py:223-234).

* Mobility Header. The three counts are accurate (24 messages / 14 dispatched,
  71 options / 20, 4 CGA extensions / 1; tables at mh.py:527, 552, 583) but the
  claim that the "# TODO markers sit at the exact dispatch tables" was wrong --
  the six markers are at mh.py:1510, 2310, 2422, 2986, 3757, 3871, i.e. the end
  of each handler block. Both places need editing. Also noted that read and make
  are symmetric (37 each) so every gap is a gap in both directions, and that
  mh.py:32-78 imports 32 const/mh sub-registry enums it never uses -- the
  enumerations for the missing options are already generated.
  "NEMO" dropped from the option-group description: Mobile Network Prefix
  (option 6) is implemented, while DNS-UPDATE-TYPE (17), Vendor Specific (19)
  and Service Selection (20) are missing and belong to none of the named groups.

* Fixed one pre-existing rendering bug in the Logging Integration bullet list: a
  literal nested inside bold. RST has no nested inline markup and does not warn,
  so the backticks were rendering as visible characters.

Checked and confirmed still accurate, so deliberately left alone
----------------------------------------------------------------

* The NotImplemented stub list -- 33 files, all 0 bytes except
  link/NotImplemented/eapol.py which holds one comment line. Layer grouping in
  the page matches the tree exactly, including NDP being under link/ and QUIC
  having no stub.
* SCTP registered-but-unimplemented type codes: 30 chunk types / 13 in
  __chunk__, 32 parameters / 8, 23 cause codes / 13 -- so 17, 24 and 10 missing,
  exactly as the page says.
* Test suite: 91 modules matching tests/test_*.py.
* The warnings double-report item: pcapkit/utilities/warnings.py documents the
  two-channel emission as deliberate, and warn() does still log and then call
  warnings.warn, so an application routing warnings into logging sees each twice.
* ESP/SCTP/PCAPNG/logging/engines all genuinely done, as the page states.
* No read/make asymmetry anywhere: NotImplementedError appears only in
  utilities/exceptions.py definitions, and every UnsupportedCall in
  pcapkit/protocols/ is a deliberate design decision (extension headers with no
  payload of their own, container classes with no registry index, abstract
  bases).

Code defects found while auditing, not fixed here -- to be filed as issues
-------------------------------------------------------------------------

1. __proto__ registry pollution. protocol.py:1300 does self.__proto__[proto] on
   a defaultdict, so a lookup miss inserts the key; internet/internet.py:248 and
   internet/ipv6.py:409 repeat it. Reproduced: parsing one 24-byte IPv4 packet
   with proto=1 inserts TransType.ICMP -> Raw into the class-level (shared)
   Internet.__proto__, after which register_transtype(TransType.ICMP, ...) warns
   "protocol 1 already registered, overwriting". SCTP._import_next_layer
   overrides the base method precisely to avoid this and documents the bug at
   sctp.py:818-831 -- a ready-made template for Frame, PCAPNG, Link, Internet,
   IPv6, TCP and UDP.
2. beholder's Raw fallback breaks on payload-less schemas.
   utilities/decorators.py:137 calls self.__header__.get_payload(), which raises
   ProtocolUnbound (schema/schema.py:461) for SCTP, whose schema has no payload
   field. Any registered SCTP upper layer that throws takes the whole SCTP layer
   down to Raw. Already fixed on the open NGAP branch (PR #417), so main only.
3. register_apptype cannot register an SCTP-only service. It raises RegistryError
   (foundation/registry/protocols.py:633-634) for the ~41 AppType members whose
   proto is SCTP-only or DCCP-only, and register_sctp is PPID-keyed, so there is
   no alternative path. Intent unclear.
4. ESP AEAD dispatch falls through. esp.py:743-756 and 782-806 treat anything
   that is not ENCR_NULL or ENCR_AES_CBC as AES-GCM. Safe today because
   CIPHER_SUITES holds nothing else, but adding AES-CCM or ChaCha20-Poly1305 to
   the table without touching decrypt/encrypt would silently mis-process them.
5. Cipher.get() / Integrity.get() invent members for an unknown string via
   extend_enum(..., -1) (const/esp/cipher.py:229-233, integrity.py:118-122).
   ESP's own _resolve avoids them, so not reachable through ESP.
6. vendor/pcapng/filter_type.py:25 has DATA = {} with a TODO URL, so the
   generated enum has no members at all.
7. DPKT and Scapy trace output is affected by the mapping-vs-frame defect that
   extraction.py:879-881 documents and excludes. PR #412 is in flight over that
   area.

Docs build: 91 WARNING/ERROR lines before this change. The after-count is still
running at the time of this commit and is NOT yet confirmed; re-run
`PCAPKIT_SPHINX=1 PYTHONPATH=. python -m sphinx -b html docs/source /tmp/out`
and compare before treating this as verified.

Not done yet: the summary posts to discussion #106 (one reply under the "More
Protocols, More!!!" topic comment 2859991, one new top-level comment for the
reassembly theme, which no existing topic covers) and the DTLS note under #251.
Nothing has been posted to either discussion.
…answered it

"Maybe Even Faster?" closed by asking for "a measured benchmark showing where the
time actually goes". That benchmark now exists (#420) and cut extraction on a
1117-frame HTTP capture by ~46% with byte-identical output, so the section asks
for something already delivered.

Rewritten to record what was found instead: the four hot-path defects that were
fixed, the three larger wins deliberately left for their own review (the flow
dumper reopening its output per frame, `analyze()` running eagerly on
unfragmented frames, and every option being parsed twice), and the two
measurement traps that make a naive benchmark of this library meaningless --
`reassembly=`/`trace=` are no-ops without `ip=`/`tcp=`, and timing several
capture shapes in one process inflates them by up to 73%.

Two predictions the profiling contradicted are recorded as such, since both are
the sort of thing a contributor would otherwise assume and spend a day on:
`Info` construction, the usual suspect, is 0.0% of its own cost and 4.4% of a
parse; and the logging integration is 0.03% even in the heaviest shape. The
IO-batching idea the original thread proposed was never the bottleneck.

Whether unfragmented frames should be emitted as datagrams at all is left as an
open design question rather than answered here.

Clean build: 91 warning/error lines, unchanged from this branch's baseline, none
naming pep.rst.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Documentation-only changes appear internally consistent and well-formed in RST, with no verified issues requiring follow-up.

Pull request overview

Updates the Help Wanted / gaps documentation in pep.rst to explicitly catalogue missing protocol dissectors, reassembly limitations, and ESP crypto coverage constraints, with concrete code pointers so gaps are trackable and not rediscovered.

Changes:

  • Adds and expands sections describing SCTP reassembly absence, DTLS non-existence, and “registered-but-not-dissected” registry coverage gaps.
  • Rewrites the ESP algorithm-coverage subsection to enumerate the practical supported suites vs. the larger IANA registries, including structural blockers.
  • Updates performance/help-wanted notes (“Maybe Even Faster?”) with specific profiling findings and adds a dedicated “Reassembly Beyond IP and TCP” section.
File summaries
File Description
docs/source/pep.rst Expands the Help Wanted page with verified, code-referenced protocol/reassembly/crypto gaps and updated benchmarking notes.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@JarryShaw
JarryShaw merged commit 413e801 into main Sep 16, 2026
25 checks passed
JarryShaw added a commit that referenced this pull request Sep 16, 2026
Three review findings on #420.

`_detect_charset` lived in `corekit/fields/strings.py` and was imported into
`protocols/protocol.py` as a private cross-module name, which was the wrong shape
for something two callers share. It is now `pcapkit.utilities.chardet`, exported
as `detect_charset` and documented alongside the other utilities.

The module is named for the third-party package whose domain it covers, matching
`utilities/logging.py` and `utilities/warnings.py`, which are named for the
stdlib modules they extend. That does not shadow anything: absolute imports mean
`import chardet` inside it still resolves the real one (verified, 7.6.0). The now
unused `import chardet` in `strings.py` goes too.

`PROFILING-NOTES.md` is dropped rather than converted. `MANIFEST.in` carries
`global-include *.rst`, so renaming it to `.rst` at top level would newly ship
developer notes in every sdist, and `docs/` is pruned from the sdist but would
make it a Sphinx source needing a toctree entry. Its substance is already in
`docs/source/pep.rst`'s performance section, which #419 merged -- the findings,
the wins, the ruled-out suspects and the two measurement traps -- and the exact
call counts and per-capture figures are in #420's description permanently.

The prefix-caching suggestion is answered in the docstring rather than only in
the review thread, since the next reader will meet the bypass and wonder: keying
on the first N octets is unsound because `chardet` is statistical over the whole
sequence, measured at three disagreements in six realistic cases, each of which
would decode a non-ASCII body as ASCII.

Full suite 782 passed, 17 skipped.
@JarryShaw
JarryShaw deleted the docs/record-known-gaps branch September 17, 2026 01:06
JarryShaw added a commit that referenced this pull request Sep 17, 2026
…directionally

Both items are asked for on the Help Wanted page (docs/source/pep.rst, from #419)
and in discussion #106.

Reassembly timeout
- The clock is the capture's own timestamps, never the host's: an offline parser
  has no other notion of time passing, and keying on time.time() would make the
  same file reassemble differently on every run. Packet and Buffer models gained
  a `timestamp`, and Reassembly.expire() abandons a buffer whose first-arriving
  fragment is older than `timeout` seconds, swept when a packet is handed over --
  the only evidence capture time has advanced.
- 60s for IPv4 and IPv6. RFC 1122 s3.3.2 supersedes RFC 791's TTL-derived timer
  ("SHOULD be a fixed value, not set from the remaining TTL... between 60 and 120
  seconds") and RFC 8200 s4.5 mandates 60; RFC 791's 15s is an initial lower
  bound that MAX(TIMER,TTL) raises, not a deadline. TCP gets no timeout: no
  specification gives stream reassembly one, and an idle connection is ordinary.
- Datagram.completed widens from bool to Completion -- COMPLETE, PARTIAL,
  TIMEOUT -- so an abandoned datagram is distinguishable from one that was merely
  unfinished. Only COMPLETE is truthy, so `if datagram.completed` is unchanged.
- Fixes two cases where strict=False reported an incomplete datagram as
  *complete*: TCP passed off zero-filled holes as received data, and IP sliced
  datagram[:TDL] with TDL still -1, handing back 65534 octets of preallocated
  buffer. Both now report PARTIAL; the payload shape is unchanged, so
  follow_tcp_stream still reconstructs what it did.

Bidirectional flow tracing
- TCP.make_bufid() orders the two endpoints canonically, so both halves of a
  conversation share one buffer, label and output file. Index gained forward and
  reverse so per-direction ordering stays recoverable. A flow closes once *both*
  halves have FINed, since a connection is not over while one direction is still
  sending.
- Default; bidirectional=False (trace_bidirectional= on Extractor, extract() and
  follow_tcp_stream) restores per-direction flows and reproduces main exactly.
- Also fixes the Scapy adapter reporting time.time() as a flow timestamp, which
  put the moment of parsing into every label.

Verified against a git archive of the branch point: all 15 sample captures give
byte-identical tree, json and reassembly output with ip/tcp/reassembly enabled,
and no capture hits a timeout (1479 datagrams, all COMPLETE). Flow tracing goes
355 -> 234 flows over the corpus with traced frames unchanged at 1222, the drop
equalling the 121 two-way conversations exactly. make_samples.py regenerates
byte-identically. Suite 893 passed / 17 skipped; mypy 127 errors against 128 on
the branch point.
@JarryShaw JarryShaw added the docs Pull requests that change documentation only (docs: subject prefix) label Sep 22, 2026
@JarryShaw JarryShaw added this to the 1.5 milestone Oct 6, 2026
@JarryShaw JarryShaw moved this to Done in PyPCAPKit Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Pull requests that change documentation only (docs: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants