From 83f4f32e536984216aa193f8a78f329b3e01972f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 4 Sep 2026 23:31:11 +0000 Subject: [PATCH] Fix four claim-register drifts found by a fresh independent audit (#101) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran a structured, adversarially-verified audit against the live repo (mypy/pytest execution, YAML config reading, external API spot-checks, README<->CLAIMS.md cross-consistency) rather than trusting either file's prose. 20 of 24 checks held up exactly; 4 were genuine drift, not regressions: - docs/CLAIMS.md 5.3: coverage's statement count (1,935) had drifted from the live 2,091 as the codebase grew; the percentage looked stable only because the miss count never moved. pyproject.toml's own inline comment was still quoting an even older 1,441 - fixed too. - docs/CLAIMS.md 5.7: test_walk.py's test count (79) undercounted the live 85 - one test parametrized over the corpus's first 40 frames now collects its full 40 instances because the corpus grew past that slice, not because tests were added. - docs/CLAIMS.md 1.6 overstated its own claim about README.md ("in the same sentence"); the two figures are adjacent bullets there, not one sentence. - README's Pyodide bullet flatly said "pypacker was not tested", contradicted by CLAIMS.md 3.1's own simulation table two paragraphs away, which shows pypacker passing the identical CPython-side check used for dpkt; also softened the lead-in so it no longer implies the scapy-import failure was observed directly inside the real CI job rather than by source citation plus that simulation. README's CI-gating bullet lumped PyTCP-net_proto in with three libraries whose benchmark code targets the compared library; CLAIMS.md's own citation says PyTCP-net_proto's benchmark exercises its RX daemon instead - reworded rather than flattened. Also deep-linked the README -> CLAIMS.md §1.7 reference instead of pointing at the file root. Docs/comment-only change: uv run ruff check/format --check, mypy --strict and the full pytest suite all pass unchanged. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_011k6j6qzsttkU6RzZBgjjG7 --- CHANGELOG.md | 25 +++++++++++++++++++++++++ README.md | 27 ++++++++++++++++----------- docs/CLAIMS.md | 29 ++++++++++++++++++++++------- pyproject.toml | 8 ++++---- 4 files changed, 67 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c3e3042..70d95a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,6 +87,31 @@ methodology, and updated 1.1/1.2/1.6 figures. **released** marker 2.0.0 already carried. Issue #107, the source both docs draw this from, updated its own stated policy line with a dated correction rather than silently rewriting history. +- **A fresh independent audit for #101 found four more genuine drifts + and fixed them, none of them regressions — just the register falling + behind a codebase and a README that both kept moving.** `docs/CLAIMS.md` + §5.3's statement count (1,935) had drifted from the live count of + 2,091 as the codebase grew — the coverage percentage held steady only + because the miss count never moved, masking it; `pyproject.toml`'s own + inline comment was still quoting an even older 1,441 and is now + corrected too. §5.7's "79 tests" for `tests/test_walk.py` undercounted + the live 85: one test parametrized over the corpus's first 40 frames + now collects its full 40 instances because the corpus has since grown + past that slice, not because tests were added. §1.6 overstated its own + claim about README.md, saying the decode-depth figure travels with the + throughput figure "in the same sentence" there; they're adjacent + bullets, not one sentence — corrected. README's Pyodide/browser bullet + said outright that "pypacker was not tested", contradicted by + `docs/CLAIMS.md` §3.1's own simulation table two paragraphs away, which + shows pypacker importing cleanly under the identical CPython-side + check used for dpkt; reworded to say so, and to stop implying the + scapy-import failure was observed directly inside the real Pyodide CI + run rather than established by source citation plus that simulation. + README's CI-gating bullet lumped PyTCP-net_proto in with three + libraries whose benchmark code targets the library being compared; + `docs/CLAIMS.md` §1.7's own citation notes PyTCP-net_proto's benchmark + exercises its RX daemon instead — reworded to keep that distinction + rather than flattening it. ### Added - **Typed accessors for every enum-backed field.** `_enums.py` defines diff --git a/README.md b/README.md index 792ef82..b31db93 100644 --- a/README.md +++ b/README.md @@ -75,11 +75,14 @@ below, its reproduction command, and its caveats): auditing ten comparable Python packet libraries' CI configurations (dpkt, scapy, pypacker, construct, pcapkit, dnspython, pyshark, nfstream, stackforge, PyTCP-net_proto; full citations in - [docs/CLAIMS.md §1.7](docs/CLAIMS.md)) — none of them gate on one.** - Four of the ten (scapy, construct, stackforge, PyTCP-net_proto) ship - real benchmark code that simply never runs in CI (construct disables - it explicitly; stackforge's Criterion benches are never invoked); none - of the ten fails a build on a performance regression. + [docs/CLAIMS.md §1.7](docs/CLAIMS.md#17-decode-throughput-is-regression-gated-in-ci)) — none of them gate on one.** + Three of the ten (scapy, construct, stackforge) ship benchmark code + that targets the library being compared but never runs in CI + (construct disables it explicitly; stackforge's Criterion benches are + never invoked). A fourth, PyTCP-net_proto, ships a benchmark too, but + it exercises its RX daemon rather than the compared `net_proto` + package, and isn't wired to CI either. None of the ten fails a build + on a performance regression. - **Typed where the alternatives are not.** `mypy --strict` over the whole of `src/`, enforced in CI. scapy ships `py.typed` but enables strict checking on 89 of its files, 2 of the 121 under @@ -92,12 +95,14 @@ below, its reproduction command, and its caveats): GPL-3.0. The other permissive option, dpkt (BSD), last released 2022-08-18 and last committed 2024-05-05, still targets Python 2.7 and 3.5–3.9, and remains marked Beta after twelve years. -- **Runs where scapy cannot — including in the browser.** Verified - under a real Pyodide runtime in CI: netprotocols imports and decodes - the full corpus cleanly there, while scapy fails to import at all - (`from fcntl import ioctl` is unconditional in `scapy/arch/`). dpkt - was separately confirmed to import cleanly under the same missing - modules, though not inside this CI job; pypacker was not tested. +- **Runs where scapy cannot — including in the browser.** Under a real + Pyodide runtime in CI, netprotocols imports and decodes the full + corpus cleanly. Scapy cannot even be attempted there: it is absent + from Pyodide's package set, and `from fcntl import ioctl` is + unconditional in `scapy/arch/` — confirmed by that same CI run still + missing the module, plus the source citation. dpkt and pypacker were + both confirmed to import cleanly under the same missing modules, but + via a separate CPython-side simulation, not inside the CI job. - **The only one of these libraries a `match`/`case` statement dissects out of the box.** dpkt builds `__slots__` from a metaclass and generates no `__match_args__`; scapy routes fields through diff --git a/docs/CLAIMS.md b/docs/CLAIMS.md index 330c6ab..ae7773b 100644 --- a/docs/CLAIMS.md +++ b/docs/CLAIMS.md @@ -410,8 +410,11 @@ Reproduce: `uv run --group bench python scripts/benchmark.py --depth`. This is the control on every throughput figure in 1.1-1.3, not a footnote to them: a decoder that stops earlier has less to do, so "faster" only means something stated beside "and it decoded more". The -depth number travels with the speed number in the same sentence, -including in README.md. +depth number travels with the speed number as the very next point, +including in README.md, where it is the bullet immediately following +the throughput one (corrected 2026-09-04 from "the same sentence", +which overstated how tightly the two are joined there — an independent +audit for #101 caught the mismatch). ### 1.7 "Decode throughput is regression-gated in CI" **Status: VERIFIED** — including the comparative form, per the #124 audit below @@ -701,7 +704,15 @@ Reproduce a nightly-style run locally: ### 5.3 "99% test coverage" **Status: VERIFIED** -1,935 statements, 1 missed (99.95%). Enforced, not merely reported: +*Statement count refreshed 2026-09-04 (an independent audit for #101 +caught it): this section previously read "1,935 statements", which had +already drifted from 1,441 — still sitting, unfixed until now, in +`pyproject.toml`'s own inline comment above `fail_under`. The +percentage read as stable across both drifts by coincidence: the miss +count never moved while the codebase grew around it. Exactly the kind +of thing Rule 4 exists to catch.* + +2,091 statements, 1 missed (99.95%). Enforced, not merely reported: `[tool.coverage.report]` sets `fail_under = 98`, and the `test` CI job runs with `--cov-report=term` on every push and pull request, so a regression below the gate fails the build (#79). @@ -813,10 +824,14 @@ this, the README taught a hand-rolled loop, ARCHITECTURE.md showed it again, and the test suite carried a private copy — the most-used function in the library was the one it did not ship. -Reproduce: `uv run pytest tests/test_walk.py` — 79 tests, including one -that walks the corpus with a hand-rolled loop and asserts the shipped -walker agrees layer for layer, which is what made retiring the copies -safe. +Reproduce: `uv run pytest tests/test_walk.py` — 85 tests (up from the +79 recorded when this section was written, caught by an independent +audit for #101 on 2026-09-04; the file didn't gain tests, one already +there — `test_matches_a_hand_rolled_walk_over_the_corpus`, parametrized +over the first 40 corpus frames — now collects its full 40 instances +because the corpus has since grown past that slice). One of the 85 +walks the corpus with a hand-rolled loop and asserts the shipped walker +agrees layer for layer, which is what made retiring the copies safe. Four properties are the actual claim, since a walker by itself is eight lines: diff --git a/pyproject.toml b/pyproject.toml index 06ccab7..f824930 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -78,8 +78,8 @@ source = ["netprotocols"] [tool.coverage.report] show_missing = true -# The suite covers 99% of 1441 statements; the only misses are Packet's -# __repr__ and its __eq__ NotImplemented branch. Gate at 98 rather than -# at the current figure so a legitimately unreachable branch does not -# fail CI, and well above it so the bar is actually held. +# The suite covers 99% of 2091 statements; the only miss is Packet's +# __eq__ NotImplemented branch. Gate at 98 rather than at the current +# figure so a legitimately unreachable branch does not fail CI, and +# well above it so the bar is actually held. fail_under = 98