docs: record the protocol, reassembly and crypto gaps on the Help Wanted page - #419
Merged
Merged
Conversation
…ted page Catalogues what PyPCAPKit does not yet implement, so the gaps are tracked rather than rediscovered. Every entry below was verified against the code at 5e4378d; nothing here is inferred from documentation. New sections in docs/source/pep.rst ----------------------------------- * "DTLS" (under More Protocols, More!!!) -- no dissector and no stub anywhere in the tree; only TLS/SSL has a stub. The registries have moved ahead of it: SCTP DTLS chunk type 65 (pcapkit/const/sctp/chunk.py:77), DTLS Key Management chunk parameter 0x8006 (const/sctp/parameter.py:86), error causes 100-103 (const/sctp/cause_code.py:62,66,70,74), and seven PPIDs -- 47, 66, 67, 68, 69 and 4242 (const/sctp/payload_protocol_identifier.py:181,240,243,246,249,265). DTLS blocks NGAP over DTLS (PPID 66) independently of NGAP itself. * "Registered, But Not Dissected" (under More Protocols, More!!!) -- registry values with no dissector behind them, all counts measured by importing the worktree copy: - LinkType 3 of 219 bound (ETHERNET 1, IPV4 228, IPV6 229), declared twice: pcapkit/protocols/misc/pcap/frame.py:89-96 and misc/pcapng.py:555-562 - TransType 15 of 151 bound, pcapkit/protocols/internet/internet.py:89-107 - EtherType 6 of 160 bound, pcapkit/protocols/link/link.py:74-87 - AppType 2 port numbers of 8182 -- TCP 21, TCP 80 (transport/tcp.py:309-315) and UDP 80 (transport/udp.py:72-78) - SCTP PPID 0 of 75; SCTP.__proto__ ships empty, transport/sctp.py:344-347, and no in-tree caller of register_sctp exists Cheap wins named: OSPF (link/ospf.py:70) and L2TP (link/l2tp.py:76) are implemented but bound to nothing (TransType 89, 115 unbound; both make __index__ raise at ospf.py:229-237 / l2tp.py:231-239); FTP_DATA (application/ftp.py:195) exists but TCP 20 is unbound; HTTP is bound on port 80 only; EtherType 0x88A8 unbound though VLAN parses that shape; LinkType NULL/LOOP/RAW carry bare IPv4/IPv6 which pcapkit dissects. * "Reassembly Beyond IP and TCP" (new top-level section) -- SCTP has no reassembly; grep -rni sctp pcapkit/foundation/ returns nothing. - The DATA chunk fields a reassembler needs are all exposed already: tsn/stream_id/stream_seq/ppid/data and flags.B/flags.E at protocols/transport/sctp.py:1143-1157; the only readers of B/E are in _make_chunk_data (sctp.py:1725-1726), i.e. the construction path. - Only the first DATA chunk of a bundle is dispatched, sctp.py:517-524. - I-DATA (RFC 8260, chunk 64) is not dissected -- SCTP.__chunk__[Chunk(64)] is 'donone'; read() matches only Payload_Data (sctp.py:520), so an I-DATA-only packet yields no payload at all. FORWARD TSN 192 and I-FORWARD-TSN 194 likewise unhandled. - Where the work goes: pcapkit/foundation/reassembly/sctp.py, subclassing Reassembly and implementing reassembly() and submit() (the only two abstract methods; reassembly/reassembly.py:180-195). Registering it is inert today -- Extractor.register_reassembly accepts any name (extraction.py:396-416) but Extractor instantiates only three literal keys (extraction.py:840,848,856) and ReassemblyManager is a fixed-field Info container (reassembly/__init__.py:37-49). Also needs the read-side twin (reassembly/data/__init__.py:37-48), five toolkit adapters, five engine call sites, and interface/core.py:148-171. - Flow tracing is TCP only (Extractor.__traceflow__ has one key; TraceFlowManager one field). TCP tracer closes on FIN not RST -- rst is absent from traceflow/data/tcp.py:43-46 -- and flows are unidirectional (traceflow/tcp.py:99). - No reassembly timeout or eviction anywhere in pcapkit/foundation/; the buffer models carry no timestamp (reassembly/data/ip.py:29-51, data/tcp.py:29-61). 8191 + 65535 bytes preallocated per in-flight IP datagram id (reassembly/ip.py:82-88). - Engines: pyshark disables reassembly (engines/pyshark.py:207-213); pypcap (pypcap.py:295-306) and pcap_ct (pcap_ct.py:380-391) disable reassembly and traceflow; pypcapfile disables IPv6 (pypcapfile.py:225-229, toolkit/pypcapfile.py:348-350). Corrections to text that was already on the page ------------------------------------------------ * ESP. "Only five encryption and five integrity algorithms" is right (verified: 36 Cipher members, 15 Integrity, 5 entries each in CIPHER_SUITES at esp.py:405-426 and INTEGRITY_SUITES at esp.py:432-448) but two of the ten are the no-ops ENCR_NULL and NONE. The "not implemented" list was incomplete; added AES-CTR (RFC 3686), AES-CMAC-96 (RFC 4494), the AES-GMAC family (RFC 4543) with ENCR_NULL_AUTH_AES_GMAC, and the Camellia, RFC 8750 implicit-IV and RFC 9227 MGM families. Added the two structural blockers: IntegritySuite records the digest as a hashlib constructor name (esp.py:360-362, 687-693) so a non-HMAC MAC needs the record widened, and load_cryptography imports only the ciphers submodules (esp.py:223-234). * Mobility Header. The three counts are accurate (24 messages / 14 dispatched, 71 options / 20, 4 CGA extensions / 1; tables at mh.py:527, 552, 583) but the claim that the "# TODO markers sit at the exact dispatch tables" was wrong -- the six markers are at mh.py:1510, 2310, 2422, 2986, 3757, 3871, i.e. the end of each handler block. Both places need editing. Also noted that read and make are symmetric (37 each) so every gap is a gap in both directions, and that mh.py:32-78 imports 32 const/mh sub-registry enums it never uses -- the enumerations for the missing options are already generated. "NEMO" dropped from the option-group description: Mobile Network Prefix (option 6) is implemented, while DNS-UPDATE-TYPE (17), Vendor Specific (19) and Service Selection (20) are missing and belong to none of the named groups. * Fixed one pre-existing rendering bug in the Logging Integration bullet list: a literal nested inside bold. RST has no nested inline markup and does not warn, so the backticks were rendering as visible characters. Checked and confirmed still accurate, so deliberately left alone ---------------------------------------------------------------- * The NotImplemented stub list -- 33 files, all 0 bytes except link/NotImplemented/eapol.py which holds one comment line. Layer grouping in the page matches the tree exactly, including NDP being under link/ and QUIC having no stub. * SCTP registered-but-unimplemented type codes: 30 chunk types / 13 in __chunk__, 32 parameters / 8, 23 cause codes / 13 -- so 17, 24 and 10 missing, exactly as the page says. * Test suite: 91 modules matching tests/test_*.py. * The warnings double-report item: pcapkit/utilities/warnings.py documents the two-channel emission as deliberate, and warn() does still log and then call warnings.warn, so an application routing warnings into logging sees each twice. * ESP/SCTP/PCAPNG/logging/engines all genuinely done, as the page states. * No read/make asymmetry anywhere: NotImplementedError appears only in utilities/exceptions.py definitions, and every UnsupportedCall in pcapkit/protocols/ is a deliberate design decision (extension headers with no payload of their own, container classes with no registry index, abstract bases). Code defects found while auditing, not fixed here -- to be filed as issues ------------------------------------------------------------------------- 1. __proto__ registry pollution. protocol.py:1300 does self.__proto__[proto] on a defaultdict, so a lookup miss inserts the key; internet/internet.py:248 and internet/ipv6.py:409 repeat it. Reproduced: parsing one 24-byte IPv4 packet with proto=1 inserts TransType.ICMP -> Raw into the class-level (shared) Internet.__proto__, after which register_transtype(TransType.ICMP, ...) warns "protocol 1 already registered, overwriting". SCTP._import_next_layer overrides the base method precisely to avoid this and documents the bug at sctp.py:818-831 -- a ready-made template for Frame, PCAPNG, Link, Internet, IPv6, TCP and UDP. 2. beholder's Raw fallback breaks on payload-less schemas. utilities/decorators.py:137 calls self.__header__.get_payload(), which raises ProtocolUnbound (schema/schema.py:461) for SCTP, whose schema has no payload field. Any registered SCTP upper layer that throws takes the whole SCTP layer down to Raw. Already fixed on the open NGAP branch (PR #417), so main only. 3. register_apptype cannot register an SCTP-only service. It raises RegistryError (foundation/registry/protocols.py:633-634) for the ~41 AppType members whose proto is SCTP-only or DCCP-only, and register_sctp is PPID-keyed, so there is no alternative path. Intent unclear. 4. ESP AEAD dispatch falls through. esp.py:743-756 and 782-806 treat anything that is not ENCR_NULL or ENCR_AES_CBC as AES-GCM. Safe today because CIPHER_SUITES holds nothing else, but adding AES-CCM or ChaCha20-Poly1305 to the table without touching decrypt/encrypt would silently mis-process them. 5. Cipher.get() / Integrity.get() invent members for an unknown string via extend_enum(..., -1) (const/esp/cipher.py:229-233, integrity.py:118-122). ESP's own _resolve avoids them, so not reachable through ESP. 6. vendor/pcapng/filter_type.py:25 has DATA = {} with a TODO URL, so the generated enum has no members at all. 7. DPKT and Scapy trace output is affected by the mapping-vs-frame defect that extraction.py:879-881 documents and excludes. PR #412 is in flight over that area. Docs build: 91 WARNING/ERROR lines before this change. The after-count is still running at the time of this commit and is NOT yet confirmed; re-run `PCAPKIT_SPHINX=1 PYTHONPATH=. python -m sphinx -b html docs/source /tmp/out` and compare before treating this as verified. Not done yet: the summary posts to discussion #106 (one reply under the "More Protocols, More!!!" topic comment 2859991, one new top-level comment for the reassembly theme, which no existing topic covers) and the DTLS note under #251. Nothing has been posted to either discussion.
…answered it "Maybe Even Faster?" closed by asking for "a measured benchmark showing where the time actually goes". That benchmark now exists (#420) and cut extraction on a 1117-frame HTTP capture by ~46% with byte-identical output, so the section asks for something already delivered. Rewritten to record what was found instead: the four hot-path defects that were fixed, the three larger wins deliberately left for their own review (the flow dumper reopening its output per frame, `analyze()` running eagerly on unfragmented frames, and every option being parsed twice), and the two measurement traps that make a naive benchmark of this library meaningless -- `reassembly=`/`trace=` are no-ops without `ip=`/`tcp=`, and timing several capture shapes in one process inflates them by up to 73%. Two predictions the profiling contradicted are recorded as such, since both are the sort of thing a contributor would otherwise assume and spend a day on: `Info` construction, the usual suspect, is 0.0% of its own cost and 4.4% of a parse; and the logging integration is 0.03% even in the heaviest shape. The IO-batching idea the original thread proposed was never the bottleneck. Whether unfragmented frames should be emitted as datagrams at all is left as an open design question rather than answered here. Clean build: 91 warning/error lines, unchanged from this branch's baseline, none naming pep.rst.
There was a problem hiding this comment.
🟢 Approval recommended
Documentation-only changes appear internally consistent and well-formed in RST, with no verified issues requiring follow-up.
Pull request overview
Updates the Help Wanted / gaps documentation in pep.rst to explicitly catalogue missing protocol dissectors, reassembly limitations, and ESP crypto coverage constraints, with concrete code pointers so gaps are trackable and not rediscovered.
Changes:
- Adds and expands sections describing SCTP reassembly absence, DTLS non-existence, and “registered-but-not-dissected” registry coverage gaps.
- Rewrites the ESP algorithm-coverage subsection to enumerate the practical supported suites vs. the larger IANA registries, including structural blockers.
- Updates performance/help-wanted notes (“Maybe Even Faster?”) with specific profiling findings and adds a dedicated “Reassembly Beyond IP and TCP” section.
File summaries
| File | Description |
|---|---|
| docs/source/pep.rst | Expands the Help Wanted page with verified, code-referenced protocol/reassembly/crypto gaps and updated benchmarking notes. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
JarryShaw
added a commit
that referenced
this pull request
Sep 16, 2026
Three review findings on #420. `_detect_charset` lived in `corekit/fields/strings.py` and was imported into `protocols/protocol.py` as a private cross-module name, which was the wrong shape for something two callers share. It is now `pcapkit.utilities.chardet`, exported as `detect_charset` and documented alongside the other utilities. The module is named for the third-party package whose domain it covers, matching `utilities/logging.py` and `utilities/warnings.py`, which are named for the stdlib modules they extend. That does not shadow anything: absolute imports mean `import chardet` inside it still resolves the real one (verified, 7.6.0). The now unused `import chardet` in `strings.py` goes too. `PROFILING-NOTES.md` is dropped rather than converted. `MANIFEST.in` carries `global-include *.rst`, so renaming it to `.rst` at top level would newly ship developer notes in every sdist, and `docs/` is pruned from the sdist but would make it a Sphinx source needing a toctree entry. Its substance is already in `docs/source/pep.rst`'s performance section, which #419 merged -- the findings, the wins, the ruled-out suspects and the two measurement traps -- and the exact call counts and per-capture figures are in #420's description permanently. The prefix-caching suggestion is answered in the docstring rather than only in the review thread, since the next reader will meet the bypass and wonder: keying on the first N octets is unsound because `chardet` is statistical over the whole sequence, measured at three disagreements in six realistic cases, each of which would decode a non-ASCII body as ASCII. Full suite 782 passed, 17 skipped.
JarryShaw
added a commit
that referenced
this pull request
Sep 17, 2026
…directionally Both items are asked for on the Help Wanted page (docs/source/pep.rst, from #419) and in discussion #106. Reassembly timeout - The clock is the capture's own timestamps, never the host's: an offline parser has no other notion of time passing, and keying on time.time() would make the same file reassemble differently on every run. Packet and Buffer models gained a `timestamp`, and Reassembly.expire() abandons a buffer whose first-arriving fragment is older than `timeout` seconds, swept when a packet is handed over -- the only evidence capture time has advanced. - 60s for IPv4 and IPv6. RFC 1122 s3.3.2 supersedes RFC 791's TTL-derived timer ("SHOULD be a fixed value, not set from the remaining TTL... between 60 and 120 seconds") and RFC 8200 s4.5 mandates 60; RFC 791's 15s is an initial lower bound that MAX(TIMER,TTL) raises, not a deadline. TCP gets no timeout: no specification gives stream reassembly one, and an idle connection is ordinary. - Datagram.completed widens from bool to Completion -- COMPLETE, PARTIAL, TIMEOUT -- so an abandoned datagram is distinguishable from one that was merely unfinished. Only COMPLETE is truthy, so `if datagram.completed` is unchanged. - Fixes two cases where strict=False reported an incomplete datagram as *complete*: TCP passed off zero-filled holes as received data, and IP sliced datagram[:TDL] with TDL still -1, handing back 65534 octets of preallocated buffer. Both now report PARTIAL; the payload shape is unchanged, so follow_tcp_stream still reconstructs what it did. Bidirectional flow tracing - TCP.make_bufid() orders the two endpoints canonically, so both halves of a conversation share one buffer, label and output file. Index gained forward and reverse so per-direction ordering stays recoverable. A flow closes once *both* halves have FINed, since a connection is not over while one direction is still sending. - Default; bidirectional=False (trace_bidirectional= on Extractor, extract() and follow_tcp_stream) restores per-direction flows and reproduces main exactly. - Also fixes the Scapy adapter reporting time.time() as a flow timestamp, which put the moment of parsing into every label. Verified against a git archive of the branch point: all 15 sample captures give byte-identical tree, json and reassembly output with ip/tcp/reassembly enabled, and no capture hits a timeout (1479 datagrams, all COMPLETE). Flow tracing goes 355 -> 234 flows over the corpus with traced frames unchanged at 1222, the drop equalling the 121 two-way conversations exactly. make_samples.py regenerates byte-identically. Suite 893 passed / 17 skipped; mypy 127 errors against 128 on the branch point.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Catalogues what
pcapkitdoes not implement, so the gaps are trackable instead of rediscovered. Every entry is verified against the code — afile:lineor a specific missing name and number, never an inference from the docs.docs/source/pep.rstonly, +205/−13. Docs build: 91 warning/error lines before, 91 after, the two sets byte-identical and none namingpep.rst.Protocols
find -iname '*dtls*'returns nothing; only TLS/SSL has one), yet it is named by SCTP chunk type 65, parameter0x8006, error causes 100–103, and PPIDs 47/66/67/68/69/4242. All of it parses as opaque bytes. This is what blocks NGAP over PPID 66 (protocols: implement NGAP over SCTP, decoding aligned PER through pycrate #417), not NGAP itself.LinkType3 of 219 ·TransType15 of 151 ·EtherType6 of 160 ·AppType2 ports of 8182 · SCTP PPID 0 of 75 (SCTP.__proto__is literally{}attransport/sctp.py:344-347, with no in-treeregister_sctpcaller).OSPFandL2TPare implemented but bound to nothing (TransType 89 and 115 unbound, and both make__index__raise).FTP_DATAexists with TCP port 20 unbound. HTTP is registered on port 80 only. EtherType0x88A8is unbound althoughVLANalready parses that shape.Reassembly and analysis
grep -rni sctp pcapkit/foundation/returns nothing — even though every field a reassembler needs is already exposed atsctp.py:1143-1157. Today only the first DATA chunk of a bundle is dispatched (sctp.py:517-524), and the only readers offlags.B/flags.Eare in_make_chunk_data.SCTP.__chunk__[Chunk(64)]is'donone'andread()matches onlyPayload_Data, so an I-DATA-only packet yields no payload at all. FORWARD TSN 192 and I-FORWARD-TSN 194 likewise.register_reassemblyaccepts any name, butExtractorinstantiates three literal keys andReassemblyManageris a fixed-fieldInfo— verified by theTypeErroronsctp=. So SCTP reassembly needs plumbing, not just an algorithm.ESP cryptographic coverage
36
Cipherand 15Integritymembers against 5 suites each — and two of those ten are the no-opsENCR_NULLandNONE. The page's existing "not implemented" list was incomplete; it now names AES-CTR, AES-CMAC-96, the AES-GMAC family, Camellia, RFC 8750 implicit-IV and RFC 9227 MGM, plus the two structural blockers:IntegritySuitestores its digest as ahashlibconstructor name, andload_cryptographyimports only thecipherssubmodules.Corrections to the page itself
The audit contradicted two things
pep.rstpreviously asserted:# TODOmarkers are not at the dispatch tables. All six sit at the end of each handler block (mh.py:1510,2310,2422,2986,3757,3871). The counts themselves check out (24/14, 71/20, 4/1), read and make are symmetric at 37 each, andmh.py:32-78already imports 32const/mhsub-registry enums it never uses — so the enumerations for the missing options exist already.pcapkit/protocols/— everyUnsupportedCallthere is a deliberate design decision.Discussion posts
-18470228-18470233-18470238The new top-level comment is justified rather than lazy: #106's seven topics are protocols, PCAP-NG, performance, logging, engines, tests and a progress meta-comment, none of which covers reassembly or flow tracing, and the post says so in its first line. Everything else went in-thread, per convention.
Note the posts deep-link to
#dtls,#registered-but-not-dissectedand#reassembly-beyond-ip-and-tcp, which do not exist on the live docs until this merges and Pages redeploys — the links resolve to the page meanwhile, just not to the section.Seven code defects found while auditing, none fixed here
Filed separately rather than smuggled into a docs PR. The first is the most consequential:
__proto__registry pollution —protocols/protocol.py:1300indexes adefaultdict, so a lookup miss inserts the key. Reproduced: parsing one 24-byte IPv4 packet withproto=1insertsTransType.ICMP -> Rawinto the class-level, sharedInternet.__proto__, after whichregister_transtype(TransType.ICMP, …)warns "already registered, overwriting".SCTP._import_next_layeroverrides the base method precisely to avoid this and documents why atsctp.py:818-831— a ready template for Frame, PCAPNG, Link, Internet, IPv6, TCP and UDP.beholder'sRawfallback on payload-less schemas — already fixed on protocols: implement NGAP over SCTP, decoding aligned PER through pycrate #417, somainonly.register_apptypecannot register an SCTP-only service, raisingRegistryErrorfor ~41AppTypemembers whose proto is SCTP- or DCCP-only, whileregister_sctpis PPID-keyed.ENCR_NULLorENCR_AES_CBCis treated as AES-GCM. Safe today; adding AES-CCM or ChaCha20 toCIPHER_SUITESwithout touching decrypt/encrypt would silently mis-process it.Cipher.get()/Integrity.get()invent members viaextend_enum(…, -1)for an unknown string.vendor/pcapng/filter_type.py:25hasDATA = {}with a TODO URL, so the generated enum has no members.extraction.py:879-881documents and excludes; interface: add the three missing engine constants, and guard dpkt/scapy tracing #412 is in flight over that area.