reassembly: resolve conflicting TCP overlaps first-write-wins, per RFC 9293 - #478
Conversation
…C 9293 (#443) Two segments claiming the same sequence range but carrying different bytes used to resolve last-write-wins in tcp.py's overlap branches (:194, :204), silently, with `completed` still reporting the datagram whole. RFC 9293 section 3.10 says the opposite -- "we reconstruct the segment to contain just the new data" -- so an already-received byte must win over a conflicting arriving one; section 3.10.7.4 agrees, trimming the duplicate portion off the incoming segment rather than the buffer. Repro from #443, confirmed on 5182ad0 before this change: completed=complete, payload=b'BBBBBBBBCCCC'. - pcapkit/foundation/reassembly/tcp.py: both overlap branches in reassembly() now merge through a new `_merge_overlap` helper, which compares the arriving segment against buffered bytes only where the hole descriptor list says the range was already received -- a position still a hole gets the arriving bytes outright (an ordinary gap fill), and a received position that disagrees keeps the buffered byte and records the range. Reproducing the issue now yields payload=b'AAAAAAAACCCC' -- the accepted, intended behaviour break (a conforming retransmission carries identical bytes, so nothing changes for it). - pcapkit/foundation/reassembly/data/tcp.py: additive `conflict` field on both `Fragment` (accumulator) and `Datagram` (public, tuple of absolute inclusive (first, last) ranges) -- `completed` is left alone, per the design decision recorded on the issue, since a caller can now ask a contested datagram apart from a clean one directly. - tests/foundation/reassembly/test_tcp.py: new `TCPReassemblyConflictTests` covering identical retransmission (uncontested), full and partial overlap on both the forward and reach-back branches, a three-way conflict, and a conflict that survives to a later completion; updated the existing overlap-merge test for the new first-write-wins arithmetic and hole-aware gap-fill. Reverting the fix and rerunning fails 8 of these (6 new + 2 updated), confirming the tests catch the regression. - tests/foundation/reassembly/data/test_models.py, docs/source/pcapkit/foundation/reassembly/tcp.rst: updated for the new `conflict` field/constructor argument. UDP has no reassembler in this codebase; IP fragment reassembly has an analogous unconditional overwrite but is a separate implementation under a different RFC (791/815, not 9293) -- filed as #477 rather than widening this PR. Full suite: 1026 passed, 17 skipped, at PYTHONPATH=<worktree>, interpreter 3.14.7. Reassembly suite alone: 62 passed.
Review of PR #478 (
|
…ist, for TCP overlap conflicts (#443) Review found a data-loss regression in the first version of this fix: _merge_overlap used Buffer.hdl -- one hole descriptor list shared by every ACK bucket under a BUFID -- as a proxy for "has this fragment already received a byte here". Buffer.ack is a dict of per-ACK Fragments with their own private raw buffers, so a different bucket's segment closing a hole in the shared hdl said nothing about this fragment's own receipt. Confirmed via the reviewer's repro on 274df06: bucket 1000's own real bytes for its own gap were discarded and replaced with never-received zero filler, and the discard was logged as a resolved conflict where bucket 1000 had never received anything to disagree with -- strictly worse than the pre-#443 last-write-wins, which was at least lossless. - pcapkit/foundation/reassembly/data/tcp.py: new Fragment.received, a bytearray mask aligned with raw, 1 where this fragment's own arriving segments placed a real byte and 0 where raw is still zero-fill placeholder for a gap this fragment has not received. Answers "has this fragment received a byte here" directly, per-Fragment, instead of inferring it from the buffer-wide hdl. - pcapkit/foundation/reassembly/tcp.py: reassembly() now maintains RCVD alongside RAW through every append/prepend/overlap branch. _merge_overlap's signature and body changed to consult and update the passed-in received mask instead of hdl, so a hole-fill or a conflict is now decided from this fragment's own history only. Reproducing the reviewer's repro now yields the same lossless bytes as origin/main, with conflict=() since bucket 1000 never actually disagreed with anything. Also fixed a wrong comment at the reach-back branch claiming a head-prepend and a tail-append "cannot both happen at once" -- they can, independently of each other; what is actually mutually exclusive is which one of the old tail or a genuinely new one is non-empty. - tests/foundation/reassembly/test_tcp.py: two new cases -- test_one_ack_buckets_hole_closing_does_not_leak_receipt_into_another (two ACK buckets under one BUFID; one bucket's fill closes the shared hdl hole while the other bucket's own real segment for the same range must survive with no spurious conflict) and test_a_genuine_gap_fill_through_the_overlap_merge_is_never_a_conflict (a hole filled via the overlap-merge path, flanked by already-received bytes that correctly match, records no conflict). Reverting just the two source files and rerunning fails the first of these (AssertionError: b'AAAA\x00\x00\x00\x00\x00\x00DDDDEEEE' != b'AAAACCCCCCDDDDEEEE'); the second passes either way, since it targets single-bucket correctness that neither version broke. - tests/foundation/reassembly/data/test_models.py, docs/source/pcapkit/foundation/reassembly/tcp.rst: updated for the new Fragment.received field/constructor argument. Full suite: 1028 passed, 17 skipped, at PYTHONPATH=<worktree>, interpreter 3.14.7. Reassembly suite alone: 64 passed.
|
Fixed the data-loss regression in Fix ( Matches Also added two tests (two-ACK-bucket cross-contamination, and a genuine gap-fill through the overlap-merge path asserting no conflict), fixed a wrong comment claiming a head-prepend and tail-append "cannot both happen at once" (they can), and re-ran the full suite: 1028 passed, 17 skipped. |
Re-review at
|
…omments (#443) Both at the owner's request on the PR; docs and comments only, no executable change. The `received` explanation had been inserted at 7-space indent in the middle of the 11-space ASCII art under the Terminology code-block, which split one literal block into two and broke its rendering -- the art resumed two lines later with 'timestamp' and ran on to 'BUFID'. Moved the paragraph to after the block ends, so the art is contiguous again and the prose follows it. The three payload-buffer bindings had their trailing comments at two different columns, because `RCVD = ...received` is longer than the `ISN` and `RAW` lines above it. All three now align at one column. tests/foundation/reassembly/: unchanged. No line exceeds the Makefile's --max-line-length=120.
…em silently (#482) `pcapkit/foundation/reassembly/ip.py:133` slice-assigned an arriving fragment's payload into the datagram buffer unconditionally, with no check of `RCVBT` beforehand and no record kept when two fragments claimed the same octets with different bytes. This is the IP analogue of #443 (fixed for TCP in unmerged PR #478), but is a separate implementation and, per research below, a different resolution. - pcapkit/foundation/reassembly/data/ip.py: add `Buffer.conflict` (list, mutable, accumulated) and `Datagram.conflict` (tuple, additive) -- `(first, last)` absolute, inclusive octet ranges, mirroring #478's TCP `conflict` field/convention. - pcapkit/foundation/reassembly/ip.py: add `IP._detect_conflicts`, called before the existing overwrite. It consults `RCVBT` block bits clipped by `TDL` (see below) to find which octets already held real data, then compares bytes directly against the arriving payload; a differing run is recorded, an equal one (a duplicate) is not. The overwrite itself is UNCHANGED -- last-write-wins, not first-write-wins. - tests/foundation/reassembly/test_ip.py: 7 new cases -- identical duplicate (uncontested), full block-aligned conflict, partial overlap extending each direction, a conflict confined to the final fragment's partial 8-octet block (the RCVBT-granularity edge case), a three-way conflict, and a conflict resolved before a later fragment completes the datagram. tests/foundation/reassembly/data/test_models.py updated for the new constructor field (IP section only). - docs/source/pcapkit/foundation/reassembly/ip/{ipv4,ipv6}.rst: document `conflict` on both Datagram and Buffer, and the RCVBT/TDL clipping. RFC 791 answers the "which fragment wins" question explicitly, and the answer is the OPPOSITE of TCP's RFC 9293 first-write-wins: "In the case that two or more fragments contain the same data either identically or through a partial overlap, this procedure will use the more recently arrived copy in the data buffer and datagram delivered." RFC 815's hole-descriptor algorithm says nothing about differing content directly, but its unconditional "copy the data from the fragment into the reassembly buffer" is consistent with the same last-write-wins resolution. So this fix keeps the existing overwrite direction and adds only the missing record of disagreement -- it does not port #478's first-write-wins merge logic, which would contradict RFC 791's own text. RCVBT records receipt in 8-octet blocks, coarser than an octet conflict needs. Every fragment but the last is required to be block-aligned (and FO is always a multiple of 8 -- it is wire-encoded in 8-octet units), so the only block that can be partially real is the one holding the final fragment's own tail. `_detect_conflicts` treats octet `pos` as genuinely already-received iff `rcvbt[pos // 8]` is set AND (`tdl < 0` or `pos < tdl`) -- `tdl` being `Buffer.TDL` as it stood before this fragment's own update. Verified against a case built for exactly this: a final fragment covering only 3 of an 8-octet block's octets, then a later fragment claiming the full block -- conflict is reported as (8, 10), not (8, 15); bytes 11-15 were never really sent by anyone and are correctly treated as new territory, not a conflict. Comparing bytes directly (per the brief's suggestion) was sufficient once combined with this TDL clip -- no new per-octet bytearray needed. Reproduced pre-fix through the public API: two conflicting 8-octet fragments at the same offset silently produced a complete datagram with the second fragment's bytes and `AttributeError` on `.conflict` (field did not exist). Post-fix: same payload (last-write-wins, per RFC 791), plus `conflict == ((0, 7),)`. Reverted the fix via a tagged stash and confirmed all 7 new boundary cases fail immediately with `AttributeError: 'Datagram' object has no attribute 'conflict'`, restored via `git stash apply` + `git stash drop`. Build/test: tests/foundation/reassembly/ -- 63 passed (7 new), 0 failed. tests/foundation/ -- 200 passed, 11 skipped. mypy pcapkit, mypy --config-file mypy.ini pcapkit, and the Makefile's --follow-imports=silent --ignore-missing-imports --show-column-numbers --show-error-codes pcapkit all report zero errors in the two files touched here (122 total errors in 39 files, all pre-existing and unrelated, e.g. toolkit/scapy.py, protocols/internet/ipv6.py). Accepted behaviour break, same reasoning as #478: a capture that hits this path is anomalous by definition, and callers reading `.conflict` on an existing `Datagram` for the first time is the point of the fix. Noticed but not fixed here (out of scope, filing separately): pcapkit/toolkit/scapy.py:160 sets `fo=ipv4.frag` with no `* 8`, but scapy's `IP.frag` is the raw 13-bit wire field in 8-octet units, not a byte offset -- confirmed by constructing `scapy.all.IP(frag=5)` and observing the raw fragment-offset field on the wire is `5`, not `40`. Every other adapter (`dpkt.py`, `pypcapfile.py`, `pcap.py`, `pcapng.py`, native `ipv4.py`) multiplies by 8 or uses an already-scaled `.offset`. This looks like a real defect in the scapy IPv4 fragment-offset handling, separate from #477 and not yet filed as its own issue.
…al list Per the owner's review comment on #478: "We've already used gap calculation in TCP reassembly logic. We should keep on that." Replaces the per-octet `Fragment.received: bytearray` receipt mask with `Fragment.gap: list[tuple[int, int]]`, absolute and inclusive sequence ranges still zero-fill in `raw` -- the same convention `conflict` already uses, reusing the GAP arithmetic `reassembly()` already computes at the two places a `bytearray(GAP)` filler ever enters `raw` (forward append and reach-back prepend) instead of adding a second bookkeeping concept. - Absolute coordinates make the alignment fixup the mask needed disappear entirely. The reach-back branch revises `isn` downward (`isn=PSN`); a per-octet mask aligned with `raw` had to be re-prefixed/shifted in lockstep with that revision, which is precisely where the earlier `274df0651` regression came from. An absolute interval needs no shifting when `isn` moves, so there is no fixup to get wrong. - `_merge_overlap` now takes and mutates `gap` in place (a Python list, the same object the caller holds) instead of taking and returning a `received` slice; a fill closes or trims the matching interval(s) via the same split-on-overlap shape the buffer-wide `hdl` list already uses. - A clean fragment now carries an empty list instead of a `raw`-sized mask: flat ~56 B regardless of payload size, against the old bytearray's one byte of mask per byte of payload. - Fixed docs/source/pcapkit/foundation/reassembly/tcp.rst, which still documented the removed `received` field in both the ASCII-art buffer diagram and the prose below it; both now describe `gap`. Verified conflict/payload output is byte-identical to fix-443-tcp-overlap-conflict@a1c50d4df across 8 hand-built scenarios, including three with two ACK buckets in flight at once (cross-bucket hole-closing, concurrent independent conflicts, concurrent reach-back). Memory: measured on real Fragment objects (Python 3.14.7), a 1,000,000 B payload costs 56 B at 0 gaps, 824 B at 10 gaps, 72,856 B at 1,000 gaps, against 1,000,057 B for the old mask at every gap count. tests/foundation/reassembly/: 66 passed / 218 subtests (64 passed / 18 subtests with the two new tests deselected, matching the pre-existing baseline). tests/foundation/: 203 passed, 11 skipped, 353 subtests.
…lpers Per the owner's "Let's get this addressed in this PR" on #478's complexity thread. The two halves of that thread are not equally real: - R0912 (too-many-branches) and R0915 (too-many-statements) belong to pylint's `design` checker, and the Makefile's `pylint:` target enables `R` and then later disables `design` -- the later disable wins, so neither can fire under `make pylint` at any commit. The project's actual gate score moves by -0.01 (8.57 -> 8.56), not the earlier-reported -0.26. Not chased here, per the owner's own follow-up comment refuting it. - Under *default* pylint (no Makefile flags), the complexity is real: on top of the gap-interval commit in this PR, `reassembly()` stood at 22 branches (limit 12) and 74 statements (limit 50) before this commit. That is what gets fixed. Extracts the forward/append direction, the reach-back/prepend direction (each already containing its own overlap-merge branch, so this is the "append, prepend and overlap" split the owner asked for), and the :rfc:`815` hole-descriptor update into three private helpers: `_reassemble_append`, `_reassemble_prepend`, `_update_hole_descriptors`. Each takes the `Fragment`/`Packet` objects directly rather than threading `BUFID`/`ACK`/`ISN`/`RAW`/`GAPS` through as separate parameters, mutates the fragment in place, and writes its own `raw`/`len`/`isn` back via `__update__` at the same point in the control flow the original code did. `reassembly()` is now the dispatch its own comments always described: pick a direction, or hand off to the hole-descriptor update. Pure extraction, verified two ways: - Default pylint on `reassembly()`: 22 branches / 74 statements before this commit, 0 findings (under 12 / under 50) after -- both measured with plain `pylint pcapkit/foundation/reassembly/tcp.py`, no Makefile flags. - conflict/payload output byte-identical to fix-443-tcp-overlap-conflict@a1c50d4df across the same 8 scenarios used for the previous commit, re-run after this one. tests/foundation/reassembly/: 66 passed / 218 subtests, unchanged from before this commit. tests/foundation/: 203 passed, 11 skipped, 353 subtests.
…pers Owner request on #478. The two extracted helpers each opened with a block whose trailing comments sat at three different columns -- the ISN/RAW/GAPS aliases at 29, the GAP computation at 36, and the guard that consumes it at 24 -- so the block read as ragged even though every comment was individually one space from its own code. Both blocks now align at column 33, which is two columns past the longest line in the run (the GAP expression), and the two helpers are now identical where they are parallel: _reassemble_prepend's guard had been left at 24 because an intervening __update__ call separated it from the run. Left alone deliberately: the lone comment on the past-the-buffered-end guard in _reassemble_append, which is a single-comment run with no counterpart in the sibling helper, so a shared column is vacuous for it; and the OFFSET/OVERLAP pairs and the __update__ keyword pairs, which are already internally aligned at 51 and 27. Comment-only -- verified by stripping trailing comments and whitespace from every added and removed line and comparing: the code is identical. tests/foundation/reassembly/test_tcp.py unchanged at 25 passed / 205 subtests.
|
Re-reviewed at 1. Is the split a pure extraction? Read 2.
All 6 cases matched to the byte; none diverged. Ran a 5000-trial randomized stress test on top of that (biased toward 3. Interval hygiene. 5000 randomized trials, checked for (a) any zero-length or inverted 4. Test adequacy -- enumerated. Exactly one test in 5. 6. Docs. Read the full 374-line Measurements I ran myself (fixtures generated via
CI. The PR moved twice during this review (the owner pushed follow-ups): Verdict. No code defect found in the split or the gap-list rewrite -- every adversarial case built to try to break it (spanning two gaps in one call, retouching a closed gap, both GOOD TO MERGE |
#478's re-review flagged the one real gap it found: no committed test drives the new Fragment.gap mechanism through two or more ACK buckets. Both tests that arrived with the gap rewrite are single-bucket, and the only existing multi-bucket test asserts an empty conflict for both buckets, so nothing covered one bucket genuinely recording a conflict while another, in flight at the same time, must not. That blind spot has bitten this code once already. The data loss #443 describes survived six boundary tests precisely because every one of them used a single ACK bucket, so none could observe state leaking across buckets. The representation has since changed from a per-octet `received` mask to an absolute gap-interval list, and the same gap had reopened against the new mechanism. Bucket 1000 takes a real conflicting retransmission and must report exactly its range; bucket 2000 is interleaved with it, never sees a conflicting byte, and must report nothing. Revert-proof: giving every Fragment one shared conflict list makes bucket 2000 report bucket 1000's range, failing with AssertionError: Tuples differ: ((2129461348, 2129461351),) != () and the test passes again once the source is restored. Test only; tests/foundation/reassembly/ 76 passed / 218 subtests.
Closes #443
Summary
Two segments claiming the same TCP sequence range but carrying different
bytes used to resolve last-write-wins, silently, in the two overlap
branches of
pcapkit/foundation/reassembly/tcp.py(:194and its mirroredreach-back case at
:204). The returnedDatagramstill reportedcompleted=complete, with nothing recording that two segments haddisagreed.
RFC 9293 (which obsoletes RFC 793) says the resolution should be the
opposite:
"Just the new data" means the already-received bytes are kept and the
overlapping portion of the arriving segment is discarded -- first-write-wins.
§3.10.7.4 agrees: trim any portion outside the window off the incoming
segment, never overwrite what's already buffered.
This is an accepted, intended behaviour break. The reproduction from the
issue now yields
payload=b'AAAAAAAACCCC'instead ofb'BBBBBBBBCCCC'--the current (pre-fix) output is not RFC-conformant, and a conforming sender
retransmits identical bytes, so nothing changes for a well-behaved stream.
What changed
pcapkit/foundation/reassembly/tcp.py: both overlap branches nowroute through a new
_merge_overlapstatic helper. It compares thearriving segment against buffered bytes only where the hole descriptor
list says the range was genuinely already received; a position still
marked as a hole gets the arriving bytes outright (an ordinary gap fill,
not a conflict), and a received position that disagrees keeps the
buffered byte and records the disagreement.
completedis untouched, perthe design decision on the issue -- it is no longer the only signal.
pcapkit/foundation/reassembly/data/tcp.py: additiveconflictfield -- a
list[tuple[int, int]]accumulator onFragment, exposed asan immutable
tuple[tuple[int, int], ...]of absolute, inclusive(first, last)sequence ranges onDatagram. Existing callers areunaffected; a caller can now ask a datagram after the fact whether it was
contested (precedent: Zeek's
rexmit_inconsistencyevent).docs/source/pcapkit/foundation/reassembly/tcp.rst: documents the newfield on both the datagram and buffer ASCII diagrams.
tests/foundation/reassembly/test_tcp.py: newTCPReassemblyConflictTestscovering identical retransmission(uncontested, unchanged), full and partial overlap on both the forward and
reach-back branches, a three-way conflict, and a conflict that survives to
a later completion. Updated the existing overlap-merge test
(
test_fragment_merging_covers_gaps_overlaps_new_acks_and_holes) for thenew first-write-wins arithmetic and hole-aware gap-fill.
tests/foundation/reassembly/data/test_models.py: updated for the newconstructor argument.
Explicitly out of scope
strictis not used to gate anything here -- it means "return alldatagrams, including those not implemented" (
self._flag_s), unrelated tothis path, and repurposing it would change the meaning of a public
keyword.
udp.pyunderfoundation/reassembly/).pcapkit/foundation/reassembly/ip.py) has astructurally similar unconditional overwrite on an overlapping fragment,
but it is a separate implementation governed by a different RFC
(791/815 IP fragment reassembly, not TCP's RFC 9293 §3.10) -- filed
separately as IP fragment reassembly overwrites already-received bytes on an overlapping fragment, unconditionally #477 rather than widening this PR.
Test plan
timestamp=0.0addedfor the now-required argument):
completed=complete,payload=b'BBBBBBBBCCCC',index=(1, 2, 3), on5182ad0cebefore anychange, and unaffected by
strict.payload=b'AAAAAAAACCCC',conflict=((100, 107),).partial overlap on the tail side (forward branch extending past the
buffered end), partial overlap on the head side (reach-back branch), a
three-way conflict, and a conflict recorded while the datagram was still
partial that survives its later completion.
touching the shared stash stack across other sessions) and rerunning fails
8 tests (6 new + 2 updated), confirming the tests catch the regression.
mypy/pylintagainst the two touched source files: no new findings(one pre-existing, unrelated
Deferredarg-type mypy error was confirmedpresent before this change too).