PCAP-NG timestamp_epoch is shifted by the host timezone, not UTC

[JarryShaw]

PCAPNG._read_timestamp promises a UTC epoch and returns a value shifted by a timezone offset. When the capture carries no if_tzone option — which the pcapng draft says should be the normal case — the offset comes from the machine running pcapkit, so the same file parses to different absolute timestamps on different hosts.

file:line

• pcapkit/protocols/misc/pcapng.py:1230-1231 — the shift
• pcapkit/protocols/misc/pcapng.py:1187-1188 — the host-timezone fallback
• pcapkit/protocols/misc/pcapng.py:1218-1221 — the docstring it contradicts
• propagates to pcapkit/toolkit/pcapng.py:217 and :234

Offending expressions

# pcapkit/protocols/misc/pcapng.py:1230-1231
            ts_decimal = timestamp_epoch + decimal.Decimal(
                tzone.utcoffset(None).total_seconds())

# pcapkit/protocols/misc/pcapng.py:1187-1188  (_get_timezone)
        if tzone is None:
            return self._get_local_timezone()

while the docstring at :1218-1221 says the second element is a decimal.Decimal object "since UNIX-Epoch in UTC timezone".

Observable symptom

examples/captures/dhcp.pcapng frame 1. Parsing the raw block bytes directly (little-endian section, IDB carries only if_tsresol=6, no if_tzone and no if_tsoffset, EPB ts_high=256643 ts_low=892570125) gives raw 1102274184317453 / 10**6 = 1102274184.317453 = 2004-12-05T19:16:24.317453Z.

What pcapkit returns:

TZ	timestamp_epoch	delta vs truth	toolkit.block2frame → ts_sec
UTC	1102274184.317453	+0 s	1102274184
Asia/Shanghai	1102302984.317453	+28800 s (+8 h)	1102302984
America/New_York	1102259784.317453	-14400 s (-4 h)	1102259784

Two further symptoms:

1. The two return values contradict each other. ts_datetime is built from the unshifted timestamp_epoch at :1235, so frame.info.timestamp.astimezone(timezone.utc) is 2004-12-05T19:16:24.317453+00:00 under all three TZ values — correct — while the returned decimal is the shifted ts_decimal. One call, one correct value and one wrong one.

2. Read→write drifts by the same amount. _make_timestamp (:1283-1285) applies if_tsoffset but not the timezone, although its docstring at :1268-1269 promises "offset and timezone conversion". Under TZ=Asia/Shanghai:

raw ts field in file = 1102274184317453
_make_timestamp(timestamp_epoch) -> 1102302984317453   (drift +28800000000 units = +28800 s)

Also worth noting: _get_local_timezone snapshots today's offset via datetime.now(utc).astimezone().tzinfo, so America/New_York contributes a fixed -04:00 (EDT) to a 2004-12-05 timestamp that was actually in EST — the fallback is not even the host's offset at the capture's own instant.

Plain PCAP is not affected in the epoch, as suspected: pcapkit/protocols/misc/pcap/frame.py:220-224 computes ts_sec + ts_usec/1e6 with no offset, and test.pcap frame 1 gives time_epoch = 1500000000.000509 under all three TZ values. Its time field does vary (2017-07-14T10:40:00.000509 vs 2017-07-13T22:40:00.000509) because :228 calls datetime.fromtimestamp(...) with no tz argument — that is a naive local rendering of the correct instant, not a wrong instant.

Minimal reproduction

Sample captures are not tracked in git; generate them with python examples/generators/make_samples.py first.

import datetime
import os
import struct
import sys
import time

# pcapkit reads its timezone at parse time; set TZ before importing it
os.environ['TZ'] = sys.argv[1]
time.tzset()

import pcapkit

PATH = 'examples/captures/dhcp.pcapng'

# --- independent ground truth, straight out of the file bytes
buf = open(PATH, 'rb').read()
assert buf[8:12] == b'\x4d\x3c\x2b\x1a'          # little-endian section
shb_len = struct.unpack_from('<I', buf, 4)[0]
idb_len = struct.unpack_from('<I', buf, shb_len + 4)[0]
epb = shb_len + idb_len                           # first Enhanced Packet Block
hi, lo = struct.unpack_from('<II', buf, epb + 12)
true_epoch = ((hi << 32) | lo) / 10 ** 6          # if_tsresol=6, no if_tsoffset

blk = pcapkit.extract(fin=PATH, store=True, nofile=True).frame[0].info
print('TZ=%-20s local utcoffset=%s'
      % (sys.argv[1], datetime.datetime.now().astimezone().utcoffset()))
print('  true epoch      = %.6f  (%s)'
      % (true_epoch,
         datetime.datetime.fromtimestamp(true_epoch, datetime.timezone.utc).isoformat()))
print('  timestamp_epoch = %s' % blk.timestamp_epoch)
print('  DELTA           = %+.0f s' % (float(blk.timestamp_epoch) - true_epoch))
print('  timestamp.astimezone(utc) = %s  <- correct, disagrees with the epoch'
      % blk.timestamp.astimezone(datetime.timezone.utc).isoformat())

Output:

$ python repro.py Asia/Shanghai
TZ=Asia/Shanghai        local utcoffset=8:00:00
  true epoch      = 1102274184.317453  (2004-12-05T19:16:24.317453+00:00)
  timestamp_epoch = 1102302984.317453
  DELTA           = +28800 s
  timestamp.astimezone(utc) = 2004-12-05T19:16:24.317453+00:00  <- correct, disagrees with the epoch

$ python repro.py America/New_York
TZ=America/New_York     local utcoffset=-1 day, 20:00:00
  true epoch      = 1102274184.317453  (2004-12-05T19:16:24.317453+00:00)
  timestamp_epoch = 1102259784.317453
  DELTA           = -14400 s
  timestamp.astimezone(utc) = 2004-12-05T19:16:24.317453+00:00  <- correct, disagrees with the epoch

$ python repro.py UTC
TZ=UTC                  local utcoffset=0:00:00
  true epoch      = 1102274184.317453  (2004-12-05T19:16:24.317453+00:00)
  timestamp_epoch = 1102274184.317453
  DELTA           = +0 s
  timestamp.astimezone(utc) = 2004-12-05T19:16:24.317453+00:00  <- correct, disagrees with the epoch

What the spec says

draft-ietf-opsawg-pcapng-02:

• §4.3 (Enhanced Packet Block): the 64-bit timestamp, with if_tsoffset added, "represents the number of units of time that have elapsed since 1970-01-01 00:00:00 UTC". There is no timezone term.
• §4.2 (IDB options, Table 3, type 10) on if_tzone: it "has never been specified in greater detail" and would be "insufficient for converting between UTC and local time"; "Therefore, it SHOULD NOT be used" — "instead, the if_iana_tzname option SHOULD be used if time zone information is to be specified".
• §4.2 (type 14) on if_tsoffset: "This offset is not intended to be used as an offset between local time and UTC."

So the epoch is UTC by definition, if_tzone should not drive it, and deriving it from the host timezone when if_tzone is absent has no basis in the format at all.

What a fix would need to touch

• PCAPNG._read_timestamp (pcapkit/protocols/misc/pcapng.py:1209-1241) — drop the utcoffset term from ts_decimal; keep tzone only as the tzinfo handed to fromtimestamp for the display datetime.
• PCAPNG._get_timezone (:1168-1189) — return datetime.timezone.utc when if_tzone is absent, rather than self._get_local_timezone().
• PCAPNG._make_timestamp (:1259-1286) — either apply the same convention or fix the docstring at :1268-1269, so read and write agree.
• Two existing tests assert the current behaviour and must be updated: tests/protocols/misc/test_pcapng_unit.py:235 (ts_decimal == decimal.Decimal(3612), i.e. 12 + 3600) and :354 (epoch == decimal.Decimal(7200) + decimal.Decimal(2) + decimal.Decimal('0.000000001')).

Observed vs inferred

• Observed by running code: every number in this report — the raw-byte ground truth, all three timestamp_epoch values and their deltas, the astimezone(utc) agreement, the block2frame ts_sec values, the +28800 s _make_timestamp round-trip drift, and the test.pcap comparison.
• Inferred by reading code: behaviour on a capture that actually carries if_tzone — every measurement above exercises the fallback path, since dhcp.pcapng has no such option; that the two named unit tests will need editing under a fix (read, not run); and the suggested fix shape.

[JarryShaw]

Fixed by #403, verified on merged main at 89201dff6 — the same capture now reads identically regardless of the reading host's timezone:

TZ=UTC                epoch = 1102274184.317453
TZ=Asia/Shanghai      epoch = 1102274184.317453
TZ=America/New_York   epoch = 1102274184.317453

Before, those three were 1102274184.317453, 1102302984.317453 (+28800 s) and 1102259784.317453 (−14400 s).

Root cause: _read_timestamp added tzone.utcoffset(None) to the epoch it returned, and _get_timezone fell back to the reading host's zone when the capture named no if_tzone. A pcapng timestamp is an offset from the UNIX epoch and carries no timezone term (draft §4.3); §4.2 also says if_tzone "SHOULD NOT be used". The offset term is gone and the fallback is UTC, matching how _get_resolution and _get_offset fall back to format defaults.

A worse symptom found along the way: dhcp_big_endian.pcapng's first raw timestamp is 0, so it came back as -14400 and round-tripped to 18446744059309551616 — a 64-bit wraparound off the negative value.

Why it hid: of six samples, the three carrying an explicit if_tzone=UTC were always correct; only the three without one shifted. So a test on a UTC machine would have passed, and the new tests are driven under several TZ values for exactly that reason — they pass under TZ=UTC and fail under Asia/Shanghai, America/New_York and Asia/Kolkata against the unfixed code.

Checked against if_tsresol too, since a wrong resolution and a timezone shift could have masked each other: all 120 Enhanced Packet Blocks across six samples, at both 10⁶ and 10⁹, are exact under two timezones. The two errors were independent.

Related, fixed in the same PR: plain PCAP's Data_Frame.time was a naive local datetime while the same function's error branch returned aware UTC, so the field was naive on success and aware on failure. It is now aware UTC throughout.

Closing manually: #403 said Closes #357, #358, #361, but GitHub only auto-closes the first issue in a comma list.