Repair the PCAP-NG write path (#365-#368)

[JarryShaw]

pcapkit could not write a valid PCAP-NG file at all — bytes() raised on 7 of the 11 block schemas, the Section Header Block among them. Four filed issues; one of them turned out to have a fifth cause the issue missed.

#366 — the write path

Five distinct causes, four sharing one shape: a length callback reading a key that only exists while unpacking.

1. padding_data was a BytesField, which Schema.pack never fills → now a PaddingField (Packet Block, Decryption Secrets Block).
2. Three options callbacks read len(pkt['padding_data']), but a PaddingField goes straight to __buffer__ and never into packet → padding recomputed inline (EPB, DSB, PB).
3. pre_pack was dead code — Schema.pack only ever called pre_unpack. It calls both now.
4. Eight pkt['__option_padding__'] subscripts → .get(..., 0), matching the existing ipv4.py / tcp.py precedent.
5. The cause the issue missed: NumberField.pre_process masked against the unsigned bit mask, so no signed field in the library could pack a negative value — Int8/16/32/64Field.pack(-1) all raised struct.error. With causes 1–4 fixed, the issue's own SHB repro (section_length=-1) still died. I verified this independently against main: Int32Field.pack(-1) and Int64Field.pack(-1) both raise there and produce ffffffff / ffffffffffffffff here, with unsigned packing unchanged.

Result: 12 of 12 schemas pack, each emitting exactly length - 4 octets. A full capture built entirely from bytes() — SHB with a comment, IDB, EPB, SPB, ISB — reads back through pcapkit.interface.extract with its options, interface linktype/snaplen, and both frames decoding as Ethernet:IPv4:UDP. That round trip is the test, not a "does not raise" assertion.

#365 — namespace='shb' everywhere

All eight _make_block_* passed it, and 'shb' is not a valid namespace anywhere — the SHB's own call site included. Confirmed rather than assumed that the damage is narrow: emitted bytes are unaffected (the raw-bytes branch appends verbatim) and duplicate detection still works (OptionType.__eq__/__hash__ key on opt_value alone), so it is misclassification plus permanent extend_enum pollution. Now opt/if/epb/ns/isb/dsb/opt/pack.

#367 — IndexError instead of FormatError

Confirmed the issue's own correction: the crash site is _get_timezone, not _get_linktype. A single _get_interface() guard now serves all four getters plus the two make-path snaplen lookups, and the three unreachable engine guards are removed. IndexError: list index out of range → FormatError: PCAP-NG: [EPB] invalid interface ID: 7.

#368 — SHB options read in the wrong byte order

In a big-endian section, dhcp_big_endian.pcapng reported one bogus opt_unknown (type 512, length 2304) swallowing 2240 zero bytes — the bytes 00 02 00 09 read backwards. The byte-order magic now writes packet['byteorder'] back, so the SHB's own options resolve from its own magic, and packing honours a requested order rather than always the host's. That file now reads Apple MBP, OS-X 10.10.5, pcap_writer.lua, test001, endofopt.

Two shared-machinery changes, and their blast radius

Schema.pack calling pre_pack, and the signed-field mask. Both are flagged deliberately:

• pre_pack is a no-op on the base class and SectionHeaderBlock is its only override in the tree, so every other schema is provably unaffected.
• Exactly four signed field declarations exist (section_length, if_tzone, if_tsoffset, PCAP's thiszone) and all four previously crashed on negatives, so the change can only turn a crash into correct bytes.

Checked by diffing the format='tree' dissection of all 15 fixtures in examples/captures/ before and after: exactly one file differs, dhcp_big_endian.pcapng, and the difference is precisely the #368 fix. The other 14 are byte-identical.

Verification

16 tests added, built from in-test bytes rather than fixtures. Every one fails against the pristine library and passes after, verified by restoring the five library files from origin/main. Two are deliberate controls that pass both before and after (they assert the read-side authority the #365 fix must agree with, and that the interface guard is a bound rather than a ban).

Suite 502 → 518 passed, 285 → 412 subtests. isort clean; pylint message-for-message identical to HEAD.

One existing test was rewritten rather than deleted: test_read_frame_rejects_invalid_interface_contexts asserted the three guards this PR removes and could only pass under a mock that skipped parsing. It now covers the one thing the engine still guards, and the three removed cases are replaced by stronger end-to-end coverage in both byte orders.

Reported, not fixed

• _get_linktype raises UnsupportedCall when ctx is None, where its siblings warn and return defaults — so constructing any packet block through the public API without a section context still fails on read-back. Pre-existing; previously masked because the pack died first.
• _make_block_dsb accepts a raw-bytes secrets_data the read-back cannot handle. Pre-existing, same masking.
• NumberField.post_process masks signed reads too, so a negative if_tzone parses as 4294967295. Deliberately not changed: SectionHeaderBlock.post_process depends on the masked form (== 0xFFFF_FFFF_FFFF_FFFF), so changing it would alter parsed output for existing fixtures. Wants its own change.
• Enum_OptionType.get mints a member for an unrecognised namespace instead of raising — which is what let All eight _make_block_* pass namespace='shb' to _make_pcapng_options #365 go unnoticed.

Closes #365, closes #366, closes #367, closes #368.

[Copilot]

🟢 Approval recommended

The changes are internally consistent, narrowly scoped (verified single pre_unpack override and single pre_pack override), and are backed by extensive regression and end-to-end tests covering the repaired failure modes.

Pull request overview

This PR repairs PCAP-NG writing/packing by fixing schema packing mechanics, correcting option namespace routing, and hardening interface-ID validation and SHB byte-order handling so that PCAP-NG files written via schemas can be read back correctly by the public extractor.

Changes:

• Fix schema packing prerequisites by invoking Schema.pre_pack() during packing, enabling SHB byte-order magic seeding.
• Correct PCAP-NG option handling: padding/length callbacks no longer rely on unpack-only keys, and _make_block_* uses the appropriate per-block option namespace.
• Improve robustness: out-of-range interface_id now raises FormatError (not IndexError), and SHB options respect the section byte order; plus add targeted regression tests.

File summaries

File	Description
tests/protocols/misc/test_pcapng_unit.py	Adds comprehensive regression tests for schema packing, signed numeric packing, option namespaces, interface-ID bounds, and SHB option byte-order.
tests/foundation/engines/test_pcapng_engine.py	Extends end-to-end engine tests for interface-ID bounds and updates context-validation expectations.
pcapkit/protocols/schema/schema.py	Ensures Schema.pack() calls pre_pack() (and still calls pre_unpack()) to support packing-time packet seeding.
pcapkit/protocols/schema/misc/pcapng.py	Fixes padding/option-length computations, makes option-padding lookups pack-safe, and corrects SHB byte-order propagation.
pcapkit/protocols/misc/pcapng.py	Adds _get_interface() guard, fixes option namespaces per block type, and routes snaplen lookups through the guard.
pcapkit/foundation/engines/pcapng.py	Removes unreachable post-parse interface-ID guards and documents why they cannot fire.
pcapkit/corekit/fields/numbers.py	Fixes signed numeric packing by remapping masked two’s-complement patterns back into the signed range.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.