Fix seven PCAP-NG parser defects (#341-#347) - #371
Conversation
- Custom Block padding was computed as `(4 - pkt['data'] % 4) % 4` on bytes, raising TypeError on every custom block. The field is dropped: `data` already spans the whole region between the PEN and the trailing length, which is what the data model and _make_block_cb assumed all along. - Interface Statistics Block sized its option area `length - 20` where the fixed fields occupy 24 octets, misparsing every capture with an ISB, Wireshark's own many_interfaces.pcapng included. - Name Resolution Block IPv6 records sized the name `length - 4`, copied from the IPv4 record, so a 16-octet address left the name reading 12 octets long. - Name Resolution Block options were read from past the end of the block. Fixed in OptionField, which now rewinds its unconsumed remainder as ForwardMatchField already does, so __option_padding__ means the same thing to the field reporting it and the field consuming it. - Obsolete Packet Block declared interface_id and drop_count 32-bit where the spec makes both 16-bit, and _read_block_packet reached for the linktype property before _info existed. Closes #341, #342, #343, #344, #345.
- IPv6InterfaceField.post_process parsed the trailing prefix-length octet as an ASCII decimal string, so /64 raised ValueError and /56 silently decoded as /8; only lengths 48-57 parsed at all, all of them wrongly. IPv4InterfaceField is symmetric and was not affected. - The engine rejected a Simple Packet Block in any section with more than one interface, which the format specification permits. The check belonged on the zero-interface case, and had to move before the block is parsed: a section with no interface description block died in _get_linktype first, which also left the EPB and Packet Block guards unreachable. Closes #346, #347.
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core schema unpacking semantics (OptionField rewind) and PCAP-NG engine control flow, which can have wide parsing impact beyond the specific fixtures even with added tests.
Pull request overview
This PR fixes seven PCAP-NG parsing/engine defects discovered while building deterministic PCAP-NG fixtures, and adds targeted regression tests to ensure the corrected block/option/field behaviors remain stable across byte orders and section/interface shapes.
Changes:
- Fixes multiple PCAP-NG schema layout/length bugs (ISB option sizing, NRB IPv6 record sizing, NRB option parsing/rewind, Custom Block data semantics, obsolete Packet Block 16-bit IDs, ns_dnsname padding).
- Fixes PCAP-NG engine section-rule handling for packet blocks (SPB allowed in multi-IDB sections; packet blocks rejected early when no IDB exists via peeking next block type).
- Fixes IPv6 interface option prefix-length decoding and adds exhaustive round-trip tests; adds multiple new unit and end-to-end regression tests.
File summaries
| File | Description |
|---|---|
| tests/protocols/misc/test_pcapng_unit.py | Adds schema-level regression tests for CB/ISB/NRB/PacketBlock parsing and OptionField remainder behavior. |
| tests/foundation/engines/test_runtime_engines.py | Updates engine test harness to use a peekable buffered reader, matching Extractor behavior. |
| tests/foundation/engines/test_pcapng_engine.py | Adds end-to-end section-rule tests via a synthetic PCAP-NG writer; expands engine unit coverage for peek/context checks. |
| tests/corekit/test_fields_ipaddress.py | Adds new exhaustive IPv6 interface prefix-length round-trip/spec tests and IPv4/IPv6 encoding non-interchangeability checks. |
| pcapkit/protocols/schema/schema.py | Rewinds underlying stream by OptionField unconsumed remainder so subsequent fields read the correct bytes. |
| pcapkit/protocols/schema/misc/pcapng.py | Fixes multiple PCAP-NG schema defects (NRB IPv6 sizing, NS_DNSName padding, NRB option sizing, ISB option sizing, CB data field, PacketBlock ID widths). |
| pcapkit/protocols/misc/pcapng.py | Fixes obsolete Packet Block linktype resolution; ensures Custom Block maker pads data/options to 32-bit boundaries. |
| pcapkit/foundation/engines/pcapng.py | Corrects SPB section rule; adds “peek next block type” guard to reject packet blocks before any IDB. |
| pcapkit/corekit/fields/ipaddress.py | Fixes IPv6 prefix-length octet parsing and adds explicit out-of-range validation; clarifies IPv4 vs IPv6 interface encoding notes. |
| examples/generators/pcapng.py | Updates generator documentation to reflect which constructs are now fixed vs still problematic. |
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The Notes block described a __padding_length__ key in packet, which nothing sets - the key is __option_padding__, as the paragraph below it and the OptionField rewind both use. Addresses Copilot's review comment on #371.
There was a problem hiding this comment.
🔵 Needs a closer look
The updated generator documentation still claims if_IPv6addr and multi-interface SPB are broken even though this PR fixes both, leaving the docs inconsistent with the new behavior.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
examples/generators/pcapng.py:86
- This section describes a Simple Packet Block multi-interface rejection as an active defect and says
test.pcapngcarries an SPB in a second section as a workaround. In this PR the engine guard is changed to only reject the zero-interface case and new tests cover multi-interface SPB parsing, so the documentation should be updated accordingly.
A further defect is worked around rather than avoided. A **Simple Packet Block**
in a section declaring more than one interface raises ``FormatError: PCAP-NG:
[SPB] invalid section with 2 interfaces`` at
``pcapkit/foundation/engines/pcapng.py:218``, which tests
``len(interfaces) != 1``. The format specification (section 4.4 of
examples/generators/pcapng.py:80
- The generator doc still states that
if_IPv6addrraises due to parsing the prefix-length octet as an ASCII string, but this PR changesIPv6InterfaceField.post_processto read the octet as a binary integer and adds tests verifying round-trips for all prefix lengths. This section should be updated so the documentation matches the new behavior.
- Files reviewed: 10/10 changed files
- Comments generated: 0 new
- Review effort level: Lite
The generator's docstring still described if_IPv6addr and a multi-interface simple packet block as live defects that no fixture could carry, which this branch fixes. Both are now covered rather than described: - every interface built by _interface_profile carries if_IPv6addr, using the /64 address from section 4.2 of the format specification, whose trailing 0x40 octet is exactly the byte that used to be read as ASCII '@'; - test.pcapng's simple packet block moved from the single-interface second section into the two-interface first section, which is the shape the engine used to reject. Verified on the regenerated fixture: test.pcapng parses to five frames, four enhanced packet blocks and the simple packet block. Addresses the two suppressed Copilot comments on #371.
There was a problem hiding this comment.
🔵 Needs a closer look
There are a couple of correctness/documentation issues in the changed code paths (notably float-based padding math in _make_block_cb and misleading __option_padding__ doc wording) that should be addressed before approval.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
pcapkit/protocols/misc/pcapng.py:3713
_make_block_cbpads usingmath.ceil(len(cb_data) / 4), which goes through floating-point arithmetic. For extremely largecb_datalengths (>= 2**53), float precision can produce an incorrect pad length. Using pure integer modulo avoids this edge case and is simpler.
pcapkit/protocols/schema/schema.py:587
- The docstring describes
__option_padding__as “the length of the padding field after an OptionField”, but it can also represent the unconsumed remainder of the option area (e.g., when a following field reads options out of that remainder). This wording is misleading given the behavior described immediately below and the use cases in PCAP-NG NRB parsing.
And an ``__option_padding__`` key in the ``packet`` to record the
length of the padding field after an
:class:`~pcapkit.corekit.fields.collections.OptionField`, which
is used to potentially determine the length of the remaining
padding field data.
- Files reviewed: 10/10 changed files
- Comments generated: 0 new
- Review effort level: Lite
_make_block_cb sized its 32-bit padding as math.ceil(n / 4) * 4 - n, which routes a length through a float. Past 2**53 that stops being exact: for n = 2**53 + 1 it yields -1, and bytes(-1) raises. -n % 4 is the octet count to the next boundary, zero when already there, and integer throughout. The same idiom appears three more times in this file, all predating this branch; left alone rather than widening the diff. Also reworded the __option_padding__ note, which called the remainder "the length of the padding field" when what follows the option area decides what it means -- usually padding to skip, but the name resolution block reads it as further options, which is the case this branch fixes. Addresses the two suppressed Copilot comments on the third review of #371.
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core schema unpacking and PCAP-NG engine control flow in ways that can have broad parsing impact beyond the covered fixtures, so a final human review is warranted.
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 0 new
- Review effort level: Lite
#580) An independent cross-review of #578 (which merged before the review finished) found the two new tests correct in every constructed byte, assertion, and pass/fail outcome, but flagged several prose errors in their docstrings: - 0x4f is TCP's Option.Reserved_79, not an "unassigned" kind (the UnassignedOption *schema class* handles it, but the wire code itself is reserved) -- fixed the TCP docstring's wording. - The IPv4 docstring attributed the option_padding rewind-and-reread- as-padding mechanism to "#431 machinery" inside OptionField.unpack. It is actually in Schema.unpack (schema.py:890), added by #371, and predates #431; #431's own contribution to OptionField.unpack is only the post-break progress check, which performs no rewind here. - The IPv4 docstring claimed declaring an 8-octet option area would trip IPv4's stricter length-sum check "before the accommodation under test is ever reached." The accommodation does run -- the short data field is read and left-padded -- the outer check just discards that result afterwards. Fixed to say so. - The TCP docstring attributed the "sizes by what it consumed, not by the declared length" measurement to OptionField.unpack; it is TCP._read_tcp_options itself (tcp.py:698, `len(schema)`). - The TCP docstring's opening line ("cut short mid-option") was wrong for the TCP fixture specifically: nothing is truncated there (hdr_len == len(raw), and the test asserts a full round-trip); the over-declaration is internal to the option, not the capture. Reworded. Left the IPv4 opening line as-is, since that fixture genuinely is short. - The TCP docstring's justification for checking length=32 alongside 12 ("the fix would reject both identically") argued for one case being enough; replaced with the actual distinction (pad width scales with the declared length: 24 zero octets vs 4). - Switched both tests' fixed 6-octet trailing literal to bytes.fromhex(), matching the surrounding files' idiom. No assertion, constructed byte, or test outcome changes. Both tests still pass; both still fail against a reject-on-any-shortfall guard with the FieldValueError text quoted in #572. Build/test: unit tier (pytest -q --ignore=tests/integration --ignore-glob='*_runtime.py' --ignore-glob='*_regression.py') green on this branch, same as before the docstring changes.
Draft. Seven defects found while building deterministic PCAP-NG fixtures for the test suite; each has a regression test that builds the block bytes in-test, and each was proven to fail before the fix and pass after.
Schema (
pcapkit/protocols/schema/misc/pcapng.py)(4 - pkt['data'] % 4) % 4onbytes, so every custom block raisedTypeError. The field is dropped rather than repaired with alen():dataalready spans the whole region between the PEN and the trailing block length, which is whatData_CustomBlockand_make_block_cbassumed all along._make_block_cbnow pads to 32 bits so the length it writes is a multiple of four.length - 20where the fixed fields occupy 24 octets. Unconditional, and reproduces on Wireshark's ownmany_interfaces.pcapng.length - 4, copied from the IPv4 record. On a real capture the names came back['v6.example', '', '', '', '', '', '', '', 'res']and the following record decoded as garbage withlength=29285.recordsspans both the record and option areas, andSchema.unpackadvances the file by each field's declared length, sooptionsbegan beyond the block. Fixed inOptionField(schema.py), which now rewinds its unconsumed remainder exactly asForwardMatchFieldalready does. That also removes a latent double-read behind every otherOptionFieldin the tree, TCP's and IPv4's included.interface_id/drop_countdeclared 32-bit where the spec makes both 16-bit, and_read_block_packetreaching for thelinktypeproperty before_infoexists. Field widths alone do not clear theAttributeError.Field and engine
if_IPv6addr—IPv6InterfaceField.post_processparsed the trailing prefix-length octet as an ASCII decimal string, so/64raisedValueErrorand/56silently decoded as/8. Only lengths 48–57 parsed at all, every one wrongly, and the field could not read what its ownpre_processwrote.IPv4InterfaceFieldwas checked and is genuinely unaffected — its encoding is symmetric — which a test now pins._get_linktypefirst, which also left the EPB and Packet Block bounds guards unreachable. The engine now peeks the next block type.Blast radius of the shared
OptionFieldchangeThe full
format='tree'dissection of all 15 fixtures was compared before and after. Onlytest.pcapngdiffers, and only by becoming correct: its threens_*options now appear where there were none, and one spuriouspacket length < 0warning is gone. No.pcapdissection changed.Tests
Nine tests across
tests/protocols/misc/test_pcapng_unit.py, a newtests/corekit/test_fields_ipaddress.py(exhaustive round trip over all 129 IPv6 prefix lengths, with an explicit case per length 48–57, since a test that only checked/64would have passed on the old code for/56), andtests/foundation/engines/test_pcapng_engine.py(SPB in multi-IDB sections in both byte orders;FormatErrorrather thanIndexErrorwhen a section has no IDB).Closes #341, #342, #343, #344, #345, #346, #347.