fix(corekit): bound the total zero padding a parse may synthesise (#573)

CHANGELOG.md

@@ -39,6 +39,7 @@ The largest release since 1.0, and the first recorded here as it happened rather
 - **Fixed** -- 45 places where a documentation page contradicted the code (#413), ambiguous cross-references and five autodoc signature failures (#416), and `Extractor`'s documented exception plus 40 phantom or stale `Args:` labels (#501).
 - **Fixed** -- two more gaps the #514 keyword audit turned up, neither previously covered by a test: `StreamEOFError`'s docstring did not say that `@prepare` always raises it with `quiet=True` -- the same end-of-stream convention `StructError` follows via its own `eof=True` -- so nothing pinned that silence against a future regression; and `register_extractor_engine`'s real keyword, `name`, was not itself under test, only its already-corrected docstring, so a future rename could put the two out of step again exactly as quietly as before.
 - **Fixed** -- `FieldBase.unpack` zero-padded straight up to a field's declared `length` with `rjust()`, regardless of how little data `buffer` actually held; a ~40-octet PCAP-NG Decryption Secrets Block with a bogus inner length was enough to force a multi-gigabyte allocation, since `length` is frequently wire-derived and so attacker-controlled. A declared length past 262144 octets -- libpcap's own `MAXIMUM_SNAPLEN`, and this package's own default `snaplen` -- that the buffer cannot back now raises `FieldValueError` instead of padding for it; the option and list loops' own tolerance for a short read past a truncated area (#431) is far under that ceiling and is untouched (#554).
+- **Fixed** -- that ceiling bounds one field, and a packet holds many, so the *sum* of a parse's zero padding was still unbounded: a declared length just under 262144 octets is honoured however often it is declared. 200 minimal PCAP-NG Decryption Secrets Blocks -- 4,800 wire octets, each declaring a `secrets_length` of 262,142 against two supplied octets -- retained 50.0 MiB, an amplification of 10,922x per block, with the per-field guard never firing because every individual field was within it. `FieldBase.unpack` now keeps a running per-context ledger of octets supplied against octets synthesised, and raises `FieldValueError` once the total of shortfalls *past 65,536 octets* passes `262144 + 16 * supplied`. The same 200 blocks now retain 0.2 MiB, 54.6x rather than 10,922x. A shortfall of 65,536 octets or fewer is padded unconditionally and charged to nothing, and that band is the load-bearing part rather than a concession. 65,536 is the whole span of a 16-bit wire length -- how an IP header, an IPv6 payload, a TCP or IPv4 option and a PCAP-NG option all declare their size -- so no shortfall a capture cut short by its snapshot length can produce is subject to the budget at all, on the first frame or the ten-thousandth. Without that band, a running budget alone made the *same* legitimate 54-octet frame declaring an IPv4 total length of 65,535 parse to one result on 37 of 40 identical calls and to another on calls 26, 33 and 39, since whether it fit depended on what had been parsed before it; a guard whose answer moves with history is worse than the amplification it bounds. The worst legitimate single shortfall measured anywhere was 65,495 octets, from exactly that frame, and the worst from a truncated PCAP-NG option was 64,750. **What this does not close, measured rather than assumed**: the 16-bit band is deliberately untouched, so a length declared by a 16-bit wire field can still be repeated without limit. A crafted 80,048-octet PCAP-NG file of 2,000 Enhanced Packet Blocks, each carrying one option declaring 65,535 octets against four real ones, parses end to end through `Extractor(store=True)` and retains 125.00 MiB of synthesised zeros for 216.88 MiB of RSS -- 1,637x its own size, linear in the block count -- both before this change and after it. That is not an oversight in the bound: parsing a bare 40-octet IPv4 header declaring a total length of 65,535, which is what a legitimate capture of offload-sized segments truncated to its snapshot length looks like, amplifies by **the same 1,637x**. The two are not separable by any budget at this layer. Separating them needs the frame's own `incl_len`/`orig_len` -- a crafted block claims nothing was truncated while declaring more than it holds, and a snapshot-truncated frame says so on the wire -- which is knowable at the protocol layer and not here. Nor is the budget scoped per file: nothing in the package resets the ledger, so as shipped the bound is over everything a context has parsed rather than over one `Extractor` run. That is still proportionate to the octets that context was genuinely given, and tightening it is a one-line change at whichever layer owns a run. Verified against every capture in `examples/captures/`, every one of them truncated at some 8,700 offsets, 190 snapshot-length rewrites, and the synthetic offload shapes above. The comparison is of the instrumented padding and supplied-octet tallies *and* of each parse's outcome -- frame count, or exception type and message -- so a cut that changed from parsing to crashing would show rather than be swallowed; every one is identical to before (#573).
 - **Fixed** -- `TCP._make_mptcp_addaddr` could not build an `ADD_ADDR` option end to end: its `kind=`/`length=` arguments were rejected with `UnknownFieldWarning` and silently dropped, and `.pack()` then raised `KeyError: 'length'` from `port`'s own condition, `pkt['length'] in (10, 22)`. The cause was one layer up -- `MPTCP`, the base class every Multipath TCP subtype schema inherits, declared `kind` and `length` only under `typing.TYPE_CHECKING` rather than as real fields, unlike `Option`, which every non-Multipath TCP option schema inherits instead. That silently dropped `kind=`/`length=` for every `_make_mptcp_*` constructor, not only `ADD_ADDR`'s, so `MPTCP` now declares both for real, the same way `Option` already did (#541). The same missing fields broke parsing too: with no `kind`/`length` fields ahead of it, a Multipath TCP subtype schema's own leading field read the `kind` octet itself rather than the octet meant for it, an off-by-two in field alignment rather than a wire-format change -- a correct sender's octets were always right, only this library's reading of them was shifted. Spec-correct `ADD_ADDR` and `MP_PRIO` options failed to parse with `FieldError: TCP: [OptNo 30] 3 invalid IP version` and `KeyError: 'length'` respectively; both parse correctly now.
 - **Fixed** -- which exception a malformed TCP SACK option raised depended on unrelated process state: a clean interpreter raised `ProtocolError` as documented, but a process that had already popped `pcapkit.corekit.fields.misc` from `sys.modules` -- which the `#439` ABC-cache regression tests do in every case's `setUp`/`tearDown` -- raised `FieldValueError` instead, from a different layer entirely, before the documented check was even reached (#525). The cause was `ListField.unpack` resolving `SchemaField` through a function-local import re-run on every call; a module popped and reimported mid-process comes back as a second, distinct class, so `isinstance` against it silently misclassified the field and billed each item by its declared length instead of by what it actually consumed. **Any caller relying on the previously-observed** `FieldValueError` **for this case now gets** `ProtocolError` **instead, deterministically**, matching the method's own docstring. Fixed by importing at module level instead.
 - **Fixed** -- two dropped-keyword/wrong-cast defects flagged in review during this release and never filed until now: HIP's `_make_param_encrypted` passed `cipher=` to a schema with no such field, so the value was silently dropped and an AES-cipher `ENCRYPTED` parameter built through `make` packed without its IV; and IPv6-Route's `RPL.post_process`, which runs on every `Schema.pack` and not only after a parse, assumed `self.addresses` was still the concatenated `bytes` a parse leaves it as, and raised slicing the `list[bytes]` a `make`-built multi-address header actually holds there (#556).

docs/source/changelog/1.5.0.rst

@@ -354,6 +354,55 @@ pull requests between #326 and #509.
   ``FieldValueError`` instead of padding for it; the option and list loops'
   own tolerance for a short read past a truncated area (#431) is far under
   that ceiling and is untouched (#554).
+* **Fixed** -- that ceiling bounds one field, and a packet holds many, so the
+  *sum* of a parse's zero padding was still unbounded: a declared length just
+  under 262144 octets is honoured however often it is declared. 200 minimal
+  PCAP-NG Decryption Secrets Blocks -- 4,800 wire octets, each declaring a
+  ``secrets_length`` of 262,142 against two supplied octets -- retained 50.0 MiB,
+  an amplification of 10,922x per block, with the per-field guard never firing
+  because every individual field was within it. ``FieldBase.unpack`` now keeps a
+  running per-context ledger of octets supplied against octets synthesised, and
+  raises ``FieldValueError`` once the total of shortfalls *past 65,536 octets*
+  passes ``262144 + 16 * supplied``. The same 200 blocks now retain 0.2 MiB,
+  54.6x rather than 10,922x.
+  A shortfall of 65,536 octets or fewer is padded unconditionally and charged to
+  nothing, and that band is the load-bearing part rather than a concession. 65,536
+  is the whole span of a 16-bit wire length -- how an IP header, an IPv6 payload,
+  a TCP or IPv4 option and a PCAP-NG option all declare their size -- so no
+  shortfall a capture cut short by its snapshot length can produce is subject to
+  the budget at all, on the first frame or the ten-thousandth. Without that band,
+  a running budget alone made the *same* legitimate 54-octet frame declaring an
+  IPv4 total length of 65,535 parse to one result on 37 of 40 identical calls and
+  to another on calls 26, 33 and 39, since whether it fit depended on what had
+  been parsed before it; a guard whose answer moves with history is worse than the
+  amplification it bounds. The worst legitimate single shortfall measured anywhere
+  was 65,495 octets, from exactly that frame, and the worst from a truncated
+  PCAP-NG option was 64,750.
+  **What this does not close, measured rather than assumed**: the 16-bit band is
+  deliberately untouched, so a length declared by a 16-bit wire field can still be
+  repeated without limit. A crafted 80,048-octet PCAP-NG file of 2,000 Enhanced
+  Packet Blocks, each carrying one option declaring 65,535 octets against four
+  real ones, parses end to end through ``Extractor(store=True)`` and retains
+  125.00 MiB of synthesised zeros for 216.88 MiB of RSS -- 1,637x its own size,
+  linear in the block count -- both before this change and after it. That is not
+  an oversight in the bound: parsing a bare 40-octet IPv4 header declaring a total
+  length of 65,535, which is what a legitimate capture of offload-sized segments
+  truncated to its snapshot length looks like, amplifies by **the same 1,637x**.
+  The two are not separable by any budget at this layer. Separating them needs the
+  frame's own ``incl_len``/``orig_len`` -- a crafted block claims nothing was
+  truncated while declaring more than it holds, and a snapshot-truncated frame
+  says so on the wire -- which is knowable at the protocol layer and not here.
+  Nor is the budget scoped per file: nothing in the package resets the ledger, so
+  as shipped the bound is over everything a context has parsed rather than over
+  one ``Extractor`` run. That is still proportionate to the octets that context
+  was genuinely given, and tightening it is a one-line change at whichever layer
+  owns a run.
+  Verified against every capture in ``examples/captures/``, every one of them
+  truncated at some 8,700 offsets, 190 snapshot-length rewrites, and the synthetic
+  offload shapes above. The comparison is of the instrumented padding and
+  supplied-octet tallies *and* of each parse's outcome -- frame count, or
+  exception type and message -- so a cut that changed from parsing to crashing
+  would show rather than be swallowed; every one is identical to before (#573).
 * **Fixed** -- ``TCP._make_mptcp_addaddr`` could not build an ``ADD_ADDR``
   option end to end: its ``kind=``/``length=`` arguments were rejected with
   ``UnknownFieldWarning`` and silently dropped, and ``.pack()`` then raised