Skip to content

Fix seven PCAP-NG parser defects (#341-#347) - #371

Merged
JarryShaw merged 5 commits into
mainfrom
fix/pcapng-parser-defects
Sep 14, 2026
Merged

JarryShaw merged 5 commits into
mainfrom
fix/pcapng-parser-defects

Conversation

@JarryShaw

Copy link
Copy Markdown
Owner

Draft. Seven defects found while building deterministic PCAP-NG fixtures for the test suite; each has a regression test that builds the block bytes in-test, and each was proven to fail before the fix and pass after.

Schema (pcapkit/protocols/schema/misc/pcapng.py)

  • PCAP-NG Custom Block padding computed on bytes, not len(bytes) #341 Custom Block — padding was (4 - pkt['data'] % 4) % 4 on bytes, so every custom block raised TypeError. The field is dropped rather than repaired with a len(): data already spans the whole region between the PEN and the trailing block length, which is what Data_CustomBlock and _make_block_cb assumed all along. _make_block_cb now pads to 32 bits so the length it writes is a multiple of four.
  • PCAP-NG ISB option area sized 4 octets too long (length - 20) #342 Interface Statistics Block — option area sized length - 20 where the fixed fields occupy 24 octets. Unconditional, and reproduces on Wireshark's own many_interfaces.pcapng.
  • PCAP-NG NRB nrb_record_ipv6 sized as if the address were 4 octets #343 Name Resolution Block IPv6 records — name sized length - 4, copied from the IPv4 record. On a real capture the names came back ['v6.example', '', '', '', '', '', '', '', 'res'] and the following record decoded as garbage with length=29285.
  • PCAP-NG NRB options field reads past the end of the block #344 NRB options read past the end of the block — not a constant, as first suspected. records spans both the record and option areas, and Schema.unpack advances the file by each field's declared length, so options began beyond the block. Fixed in OptionField (schema.py), which now rewinds its unconsumed remainder exactly as ForwardMatchField already does. That also removes a latent double-read behind every other OptionField in the tree, TCP's and IPv4's included.
  • PCAP-NG obsolete Packet Block: 32-bit fields vs 32-octet arithmetic #345 obsolete Packet Block — two independent causes: interface_id/drop_count declared 32-bit where the spec makes both 16-bit, and _read_block_packet reaching for the linktype property before _info exists. Field widths alone do not clear the AttributeError.

Field and engine

  • if_IPv6addr prefix length parsed as an ASCII string, not an octet #346 if_IPv6addr — IPv6InterfaceField.post_process parsed the trailing prefix-length octet as an ASCII decimal string, so /64 raised ValueError and /56 silently decoded as /8. Only lengths 48–57 parsed at all, every one wrongly, and the field could not read what its own pre_process wrote. IPv4InterfaceField was checked and is genuinely unaffected — its encoding is symmetric — which a test now pins.
  • Simple Packet Block rejected in any multi-interface section #347 Simple Packet Block — rejected in any section with more than one interface, which §4.4 permits. The check belonged on the zero-interface case, and had to move before the block is parsed: a section with no IDB died in _get_linktype first, which also left the EPB and Packet Block bounds guards unreachable. The engine now peeks the next block type.

Blast radius of the shared OptionField change

The full format='tree' dissection of all 15 fixtures was compared before and after. Only test.pcapng differs, and only by becoming correct: its three ns_* options now appear where there were none, and one spurious packet length < 0 warning is gone. No .pcap dissection changed.

Tests

Nine tests across tests/protocols/misc/test_pcapng_unit.py, a new tests/corekit/test_fields_ipaddress.py (exhaustive round trip over all 129 IPv6 prefix lengths, with an explicit case per length 48–57, since a test that only checked /64 would have passed on the old code for /56), and tests/foundation/engines/test_pcapng_engine.py (SPB in multi-IDB sections in both byte orders; FormatError rather than IndexError when a section has no IDB).

Closes #341, #342, #343, #344, #345, #346, #347.

- Custom Block padding was computed as `(4 - pkt['data'] % 4) % 4` on bytes,
  raising TypeError on every custom block. The field is dropped: `data` already
  spans the whole region between the PEN and the trailing length, which is what
  the data model and _make_block_cb assumed all along.
- Interface Statistics Block sized its option area `length - 20` where the fixed
  fields occupy 24 octets, misparsing every capture with an ISB, Wireshark's own
  many_interfaces.pcapng included.
- Name Resolution Block IPv6 records sized the name `length - 4`, copied from
  the IPv4 record, so a 16-octet address left the name reading 12 octets long.
- Name Resolution Block options were read from past the end of the block. Fixed
  in OptionField, which now rewinds its unconsumed remainder as
  ForwardMatchField already does, so __option_padding__ means the same thing to
  the field reporting it and the field consuming it.
- Obsolete Packet Block declared interface_id and drop_count 32-bit where the
  spec makes both 16-bit, and _read_block_packet reached for the linktype
  property before _info existed.

Closes #341, #342, #343, #344, #345.
- IPv6InterfaceField.post_process parsed the trailing prefix-length octet as an
  ASCII decimal string, so /64 raised ValueError and /56 silently decoded as
  /8; only lengths 48-57 parsed at all, all of them wrongly. IPv4InterfaceField
  is symmetric and was not affected.
- The engine rejected a Simple Packet Block in any section with more than one
  interface, which the format specification permits. The check belonged on the
  zero-interface case, and had to move before the block is parsed: a section
  with no interface description block died in _get_linktype first, which also
  left the EPB and Packet Block guards unreachable.

Closes #346, #347.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core schema unpacking semantics (OptionField rewind) and PCAP-NG engine control flow, which can have wide parsing impact beyond the specific fixtures even with added tests.

Pull request overview

This PR fixes seven PCAP-NG parsing/engine defects discovered while building deterministic PCAP-NG fixtures, and adds targeted regression tests to ensure the corrected block/option/field behaviors remain stable across byte orders and section/interface shapes.

Changes:

  • Fixes multiple PCAP-NG schema layout/length bugs (ISB option sizing, NRB IPv6 record sizing, NRB option parsing/rewind, Custom Block data semantics, obsolete Packet Block 16-bit IDs, ns_dnsname padding).
  • Fixes PCAP-NG engine section-rule handling for packet blocks (SPB allowed in multi-IDB sections; packet blocks rejected early when no IDB exists via peeking next block type).
  • Fixes IPv6 interface option prefix-length decoding and adds exhaustive round-trip tests; adds multiple new unit and end-to-end regression tests.
File summaries
File Description
tests/protocols/misc/test_pcapng_unit.py Adds schema-level regression tests for CB/ISB/NRB/PacketBlock parsing and OptionField remainder behavior.
tests/foundation/engines/test_runtime_engines.py Updates engine test harness to use a peekable buffered reader, matching Extractor behavior.
tests/foundation/engines/test_pcapng_engine.py Adds end-to-end section-rule tests via a synthetic PCAP-NG writer; expands engine unit coverage for peek/context checks.
tests/corekit/test_fields_ipaddress.py Adds new exhaustive IPv6 interface prefix-length round-trip/spec tests and IPv4/IPv6 encoding non-interchangeability checks.
pcapkit/protocols/schema/schema.py Rewinds underlying stream by OptionField unconsumed remainder so subsequent fields read the correct bytes.
pcapkit/protocols/schema/misc/pcapng.py Fixes multiple PCAP-NG schema defects (NRB IPv6 sizing, NS_DNSName padding, NRB option sizing, ISB option sizing, CB data field, PacketBlock ID widths).
pcapkit/protocols/misc/pcapng.py Fixes obsolete Packet Block linktype resolution; ensures Custom Block maker pads data/options to 32-bit boundaries.
pcapkit/foundation/engines/pcapng.py Corrects SPB section rule; adds “peek next block type” guard to reject packet blocks before any IDB.
pcapkit/corekit/fields/ipaddress.py Fixes IPv6 prefix-length octet parsing and adds explicit out-of-range validation; clarifies IPv4 vs IPv6 interface encoding notes.
examples/generators/pcapng.py Updates generator documentation to reflect which constructs are now fixed vs still problematic.
Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread pcapkit/protocols/schema/schema.py
The Notes block described a __padding_length__ key in packet, which nothing
sets - the key is __option_padding__, as the paragraph below it and the
OptionField rewind both use. Addresses Copilot's review comment on #371.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The updated generator documentation still claims if_IPv6addr and multi-interface SPB are broken even though this PR fixes both, leaving the docs inconsistent with the new behavior.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

examples/generators/pcapng.py:86

  • This section describes a Simple Packet Block multi-interface rejection as an active defect and says test.pcapng carries an SPB in a second section as a workaround. In this PR the engine guard is changed to only reject the zero-interface case and new tests cover multi-interface SPB parsing, so the documentation should be updated accordingly.
A further defect is worked around rather than avoided. A **Simple Packet Block**
in a section declaring more than one interface raises ``FormatError: PCAP-NG:
[SPB] invalid section with 2 interfaces`` at
``pcapkit/foundation/engines/pcapng.py:218``, which tests
``len(interfaces) != 1``. The format specification (section 4.4 of

examples/generators/pcapng.py:80

  • The generator doc still states that if_IPv6addr raises due to parsing the prefix-length octet as an ASCII string, but this PR changes IPv6InterfaceField.post_process to read the octet as a binary integer and adds tests verifying round-trips for all prefix lengths. This section should be updated so the documentation matches the new behavior.
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@JarryShaw
JarryShaw marked this pull request as ready for review September 14, 2026 15:13
The generator's docstring still described if_IPv6addr and a multi-interface
simple packet block as live defects that no fixture could carry, which this
branch fixes. Both are now covered rather than described:

- every interface built by _interface_profile carries if_IPv6addr, using the
  /64 address from section 4.2 of the format specification, whose trailing
  0x40 octet is exactly the byte that used to be read as ASCII '@';
- test.pcapng's simple packet block moved from the single-interface second
  section into the two-interface first section, which is the shape the engine
  used to reject.

Verified on the regenerated fixture: test.pcapng parses to five frames, four
enhanced packet blocks and the simple packet block.

Addresses the two suppressed Copilot comments on #371.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

There are a couple of correctness/documentation issues in the changed code paths (notably float-based padding math in _make_block_cb and misleading __option_padding__ doc wording) that should be addressed before approval.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

pcapkit/protocols/misc/pcapng.py:3713

  • _make_block_cb pads using math.ceil(len(cb_data) / 4), which goes through floating-point arithmetic. For extremely large cb_data lengths (>= 2**53), float precision can produce an incorrect pad length. Using pure integer modulo avoids this edge case and is simpler.

pcapkit/protocols/schema/schema.py:587

  • The docstring describes __option_padding__ as “the length of the padding field after an OptionField”, but it can also represent the unconsumed remainder of the option area (e.g., when a following field reads options out of that remainder). This wording is misleading given the behavior described immediately below and the use cases in PCAP-NG NRB parsing.
            And an ``__option_padding__`` key in the ``packet`` to record the
            length of the padding field after an
            :class:`~pcapkit.corekit.fields.collections.OptionField`, which
            is used to potentially determine the length of the remaining
            padding field data.
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

_make_block_cb sized its 32-bit padding as math.ceil(n / 4) * 4 - n, which
routes a length through a float. Past 2**53 that stops being exact: for
n = 2**53 + 1 it yields -1, and bytes(-1) raises. -n % 4 is the octet count to
the next boundary, zero when already there, and integer throughout.

The same idiom appears three more times in this file, all predating this
branch; left alone rather than widening the diff.

Also reworded the __option_padding__ note, which called the remainder "the
length of the padding field" when what follows the option area decides what it
means -- usually padding to skip, but the name resolution block reads it as
further options, which is the case this branch fixes.

Addresses the two suppressed Copilot comments on the third review of #371.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core schema unpacking and PCAP-NG engine control flow in ways that can have broad parsing impact beyond the covered fixtures, so a final human review is warranted.

Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

JarryShaw added a commit that referenced this pull request Sep 21, 2026
#580)

An independent cross-review of #578 (which merged before the review
finished) found the two new tests correct in every constructed byte,
assertion, and pass/fail outcome, but flagged several prose errors in
their docstrings:

- 0x4f is TCP's Option.Reserved_79, not an "unassigned" kind (the
  UnassignedOption *schema class* handles it, but the wire code itself
  is reserved) -- fixed the TCP docstring's wording.
- The IPv4 docstring attributed the option_padding rewind-and-reread-
  as-padding mechanism to "#431 machinery" inside OptionField.unpack.
  It is actually in Schema.unpack (schema.py:890), added by #371, and
  predates #431; #431's own contribution to OptionField.unpack is only
  the post-break progress check, which performs no rewind here.
- The IPv4 docstring claimed declaring an 8-octet option area would
  trip IPv4's stricter length-sum check "before the accommodation
  under test is ever reached." The accommodation does run -- the short
  data field is read and left-padded -- the outer check just discards
  that result afterwards. Fixed to say so.
- The TCP docstring attributed the "sizes by what it consumed, not by
  the declared length" measurement to OptionField.unpack; it is
  TCP._read_tcp_options itself (tcp.py:698, `len(schema)`).
- The TCP docstring's opening line ("cut short mid-option") was wrong
  for the TCP fixture specifically: nothing is truncated there
  (hdr_len == len(raw), and the test asserts a full round-trip); the
  over-declaration is internal to the option, not the capture. Reworded.
  Left the IPv4 opening line as-is, since that fixture genuinely is short.
- The TCP docstring's justification for checking length=32 alongside 12
  ("the fix would reject both identically") argued for one case being
  enough; replaced with the actual distinction (pad width scales with
  the declared length: 24 zero octets vs 4).
- Switched both tests' fixed 6-octet trailing literal to bytes.fromhex(),
  matching the surrounding files' idiom.

No assertion, constructed byte, or test outcome changes. Both tests
still pass; both still fail against a reject-on-any-shortfall guard
with the FieldValueError text quoted in #572.

Build/test: unit tier (pytest -q --ignore=tests/integration
--ignore-glob='*_runtime.py' --ignore-glob='*_regression.py') green
on this branch, same as before the docstring changes.
@JarryShaw JarryShaw added the fix Pull requests that fix a defect (fix: subject prefix) label Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Pull requests that fix a defect (fix: subject prefix)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PCAP-NG Custom Block padding computed on bytes, not len(bytes)

2 participants