pcap/pcapng: fix frame bytes, output filenames, and timestamp timezones

[JarryShaw]

Closes #357, #358, #361.

#357 — bytes(frame) returned the next frame's bytes

Verified on main before touching anything, using arp.pcap (176 bytes, records at raw[24:100] and raw[100:176]):

frame[0] len= 76 expected= 76 match=False     <- returned raw[84:100], record 2's header
frame[1] len= 16 expected= 76 match=False

Root cause. read() rewound -self.length (16) with SEEK_CUR to find the record's start — but by then the schema unpack had consumed the 16-octet record header and the incl_len payload octets, since Schema_Frame.packet is a PayloadField(length=incl_len). The rewind therefore landed incl_len octets late. Nested layers were unaffected because _import_next_layer reads the schema's payload rather than self._data, which is why this survived.

Fixed the way the PCAP-NG reader already does it: record an absolute _seek_set before the unpack and seek to it with SEEK_SET. The post-frame cursor works out to the same value as before in the well-formed case, so the engine's sequential read is unchanged.

PCAP-NG does not share it — verified across all 121 blocks in six samples: every stored block's bytes() lands on a real block boundary walked out of the raw file, 0 mismatches.

#358 — files=True wrote Frame 1..json

Extractor.__output__ spells extensions with the dot ('.json'), while make_name's own docstring promises them without — and all seven consumers compose f'{name}.{ext._fext}'. The existing engine tests already set _fext: 'json' bare, so the dotted value was the minority spelling.

make_name now honours its documented contract and supplies the dot itself in the non-files branch, so no engine module needed touching. PCAP-NG shared the idiom and is fixed by the same normalisation. TraceFlow keeps a separate __output__ and composes f'{label}{self._fdpext}' with no literal dot, so it still wants the dotted form and is deliberately left alone.

#361 — PCAP-NG timestamp_epoch shifted by the host timezone

The same frame of the same committed capture, read on main:

TZ	epoch	drift
UTC	1102274184.317453	—
Asia/Shanghai	1102302984.317453	+28800 s
America/New_York	1102259784.317453	−14400 s

Root cause. _read_timestamp added tzone.utcoffset(None) to the epoch it returned, and _get_timezone fell back to the reading host's zone when the capture named no if_tzone. A pcapng timestamp is an offset from the UNIX epoch, so it carries no timezone term at all (draft §4.3), and §4.2 says if_tzone "SHOULD NOT be used".

Worse: dhcp_big_endian.pcapng's first raw timestamp is 0, so it came back as -14400 and round-tripped to 18446744059309551616 — a 64-bit wraparound off the negative value.

The offset term is gone and the fallback is UTC, matching how _get_resolution and _get_offset fall back to format defaults; tzone survives only as the display datetime's tzinfo. _make_timestamp's arithmetic was already correct — only its docstring promised a conversion it never did, so the docstring was fixed rather than the code.

Why this hid so well: of six samples, the three carrying an explicit if_tzone=UTC were always right; only the three without one shifted.

Checked against if_tsresol, since two overlapping errors could mask each other: all 120 Enhanced Packet Blocks across six samples, at both 10⁶ and 10⁹ resolution, are exact for epoch, datetime and round trip under two timezones. Before the fix the same check failed on exactly the three no-if_tzone samples — so the timezone term and the resolution scaling were independent.

One change beyond the filed issues

Data_Frame.time for plain PCAP was a naive local datetime — but the same function's except ValueError branch has always returned aware UTC, and toolkit/pcapng.py:244 fills the same field with an aware datetime. So the field was naive on success and aware on failure:

try:
    _time = datetime.datetime.fromtimestamp(_irat[0] / _irat[1])        # naive local
except ValueError:
    _time = datetime.datetime.fromtimestamp(0, datetime.timezone.utc)   # aware UTC

That is not a contract, it is a bug — frame.time > x worked until it hit a malformed frame. It is now aware UTC throughout: the only self-consistent option without changing the error branch instead, consistent with PCAP-NG, and correct for a value that is epoch-relative and therefore an absolute instant. UTC specifically because the PCAP format has no usable timezone — its thiszone header field is deprecated and ignored.

This is a behaviour change for callers comparing or subtracting against a naive datetime (TypeError: can't compare offset-naive and offset-aware datetimes). Nothing in-tree does, so the suite would not have caught it; it is going into the 1.5.0 behaviour-changes list.

Verification

selection	baseline	after
fixture-free unit tier	575 passed, 14 skipped	579 passed, 14 skipped
tests/integration	70 passed, 3 skipped	74 passed, 3 skipped
protocol *_runtime.py/*_regression.py	46 passed	47 passed

Every delta is a test added. No pre-existing assertion was re-baselined — in particular no integration assertion pins a timestamp, so nothing there had to move.

New tests proven to fail against pristine copies of the three source files. The timezone tests are the ones worth noting: on unfixed code they pass under TZ=UTC and fail under Asia/Shanghai, America/New_York and Asia/Kolkata — the concrete demonstration that a UTC-only test would have been worthless here. They also skip with an explanatory message if the host resolves every zone to the same offset, so they cannot silently degrade.

Left alone, deliberately

• format='pcap'/'cap' raises TypeError: PCAPIO.__init__() missing 1 required keyword-only argument: 'protocol', and format='text' raises AttributeError: module 'dictdumper' has no attribute 'Text'. Both pre-existing and unfiled; excluded from the new test module with a note.
• PCAPNG._get_local_timezone is now unused by production code but kept, with a docstring warning that it must not drive a timestamp — it reports today's offset, so a December 2004 timestamp would get EDT rather than EST. Three existing tests reference it.

[Copilot]

🟡 Changes recommended

Extractor.make_name can still raise KeyError for unknown fmt instead of the documented FormatError, which can leak an unintended exception to callers.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR fixes several long-standing correctness issues in the PCAP/PCAPNG readers and the extractor’s output naming logic, and adds regression tests to lock the behavior in.

Changes:

• Fix PCAP Frame raw-byte capture (bytes(frame), frame.packet.*) by seeking from a recorded absolute record start, and make PCAP Frame.info.time consistently timezone-aware UTC.
• Fix PCAPNG timestamp epoch handling to be timezone-independent (UTC epoch), with UTC defaults when if_tzone is absent, plus targeted multi-timezone regression tests.
• Normalize dumper extensions to be bare (no leading dot) in Extractor.make_name to prevent files=True outputs like Frame 1..json, and add integration coverage for real on-disk filenames.

File summaries

File	Description
tests/protocols/misc/test_pcapng_unit.py	Updates/extends unit coverage for PCAPNG timezone/epoch behavior and adds multi-TZ regression tests.
tests/protocols/misc/pcap/test_header_frame_unit.py	Adds unit tests ensuring PCAP frames retain their own record bytes and timestamps are aware UTC.
tests/protocols/misc/pcap/test_frame_runtime.py	Adds runtime-path regression test through pcapkit.interface.extract for correct frame record bytes.
tests/integration/test_files_output_naming.py	New integration tests asserting files=True writes filenames with exactly one dot across formats/engines.
tests/integration/_helpers.py	Updates helper docstring now that doubled-dot output is fixed and covered elsewhere.
tests/foundation/test_extraction.py	Updates make_name expectations for bare extensions and broadens assertions to prevent regressions.
pcapkit/protocols/misc/pcapng.py	Removes timezone offset leakage into timestamp_epoch, defaults timezone to UTC, and clarifies timestamp docs.
pcapkit/protocols/misc/pcap/frame.py	Fixes PCAP frame raw-byte capture by using an absolute seek origin recorded pre-unpack; makes time UTC-aware.
pcapkit/foundation/extraction.py	Normalizes registered extensions to bare form and adjusts output filename composition to avoid doubled dots.
conda/build	Bumps conda build number.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.