Skip to content

Close the decode-throughput regression against dpkt - #152

Merged
EONRaider merged 1 commit into
masterfrom
claude/new-session-ghy12u
Sep 4, 2026
Merged

Close the decode-throughput regression against dpkt#152
EONRaider merged 1 commit into
masterfrom
claude/new-session-ghy12u

Conversation

@EONRaider

Copy link
Copy Markdown
Owner

Summary

Re-measuring every held claim after #107/#124 closed surfaced a regression none of the five prior tiers had been watching for: decode throughput against dpkt had reversed from 1.16x faster (right after Tier 1) to ~11% slower, without anyone profiling why. This closes #147 by root-causing it with cProfile/pstats rather than guessing from the tier history, fixing what the profile actually showed, and fixing a benchmark-methodology bug the investigation surfaced along the way.

What's included

  • bytes_to_ipv6 sped up (src/netprotocols/_base.py). Added by Prove the browser claim with a Pyodide/WASM CI job #99 for Pyodide portability — not one of the three originally-suspected tiers (A public protocol registry #87/Ship decode_frame() — the chain walker belongs in the library #88/Structured parse diagnostics on ProtocolError #91) — it formatted its eight address words one f-string at a time through a generator, ~18% of total corpus decode time by profile. Reformatted to a single %-format call plus a split, verified byte-identical against glibc's inet_ntop via the existing test_bytes_to_ipv6_matches_glibc hypothesis test, 2.1x faster in isolation.
  • A Packet._from_decoded() fast-construction path (src/netprotocols/packet.py), used only by decode_frame() (src/netprotocols/walk.py). decode_frame()'s Packet(*layers, ...) call paid a redundant isinstance(layer, Protocol) check per layer through Protocol's ABC __instancecheck__ machinery — ~7-8% of decode_frame()'s own time — on a list already guaranteed homogeneous by decode_frame()'s own loop. The public Packet(...) constructor is untouched and still validates arbitrary caller-supplied arguments.
  • A public protocol registry #87 and Structured parse diagnostics on ProtocolError #91 ruled out directly, not re-guessed. Read every next_protocol() override (still hits its documented single-dict.get fast path) and every raise site Structured parse diagnostics on ProtocolError #91 touched (diagnostic fields populate only inside a raise, never on the happy path).
  • A benchmark methodology bug, fixed in the same pass (scripts/benchmark.py). decode_netprotocols()/_netprotocols_chain() had never called decode_frame(), the documented public chain-walking API Ship decode_frame() — the chain walker belongs in the library #88 shipped — they kept a hand-rolled copy of the pre-Ship decode_frame() — the chain walker belongs in the library #88 loop, so every decode-throughput figure this project has published described code real callers of the documented API never ran. Both now call decode_frame() directly; the now-dead walk() is deleted.
  • benchmarks/baseline.json refreshed — five tiers and this fix stale at a v1.3.0-era figure (114,388 f/s), now 98,007.5 f/s / 6.5677 normalized, matching the new methodology.
  • docs/CLAIMS.md — new "Re-measured after closing the dpkt-throughput regression" section with the full profiling writeup and an explicit old-vs-new-methodology breakdown (closing the gap under the old hand-rolled-loop measurement vs. what decode_frame()'s real, deliberate overhead costs under the new one); 1.1/1.2/1.6 re-measured.
  • README.md — headline scapy ratio updated (5.6x → 5.4x, dpkt gap direction unchanged).
  • CHANGELOG.md — new ## [Unreleased] entry.

Numbers

Old methodology (hand-rolled loop) closed the dpkt gap from 1.12x back to near parity (~1.02-1.03x) — the two accidental-overhead fixes alone did nearly all of that. New methodology (decode_frame(), what ships) measures more real work (bounded-depth check, full Packet construction), so the final published figure is a smaller but more honest gap:

frames/sec vs. netprotocols
netprotocols (via decode_frame()) 97,739
dpkt 1.9.8 106,619 1.09x (dpkt faster)
scapy 2.7.0 18,144 0.19x

Down from ~11% slower than dpkt to ~9% slower, on a workload that now measures more of what decode_frame() actually does. Decode depth unchanged (27/70 split vs dpkt).

Verification

  • uv run --frozen ruff check . and uv run --frozen ruff format --check . are clean
  • uv run --frozen mypy is clean (strict)
  • uv run --frozen pytest passes locally
  • uv run --frozen python scripts/benchmark.py --check --threshold 15 passes against the refreshed baseline
  • CHANGELOG.md has an entry under ## [Unreleased]
  • bytes_to_ipv6's existing glibc-differential hypothesis test passes against the new implementation

Notes

Tracked by #147, with sub-issues #148-#151 for the profiling/fix/re-measure/README-update pieces (deliberately lighter-weight than the full Tier-N epic apparatus the five prior tiers used — this is a smaller, investigative fix, not a new tier).

Does not bump pyproject.toml's version or touch release tagging.

Closes #147.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RzqEV89zgdGLb3QgnZy3vp


Generated by Claude Code

Re-measuring every held claim after #107/#124 closed surfaced a
regression none of the five prior tiers had been watching for: decode
throughput against dpkt had reversed from 1.16x faster (right after
Tier 1) to ~11% slower, without anyone profiling why.

cProfile against the corpus decode loop found two causes -- neither
one of the three tiers originally suspected:

- bytes_to_ipv6 (added by #99 for Pyodide portability, landing after
  all three originally-suspected tiers) formatted its eight address
  words one f-string at a time through a generator, ~18% of total
  corpus decode time. Reformatted to one %-format call plus a split,
  byte-identical output (verified against glibc's inet_ntop via the
  existing hypothesis test), 2.1x faster in isolation.
- decode_frame()'s Packet(*layers, ...) construction paid a redundant
  isinstance(layer, Protocol) check per layer through Protocol's ABC
  __instancecheck__ machinery, ~7-8% of decode_frame()'s own time --
  every element in that list is already guaranteed to be a Protocol
  instance by decode_frame()'s own loop. Packet gained a private
  _from_decoded() fast-construction path (object.__new__ plus direct
  attribute assignment, same shortcut _base.py already uses for
  Ethernet/ARP/IPv4), used only by decode_frame(). The public
  Packet(...) constructor is untouched.

#87's registry dispatch and #91's structured diagnostics were both
ruled out directly rather than re-guessed: every next_protocol()
override still hits its documented single-dict.get fast path, and
every raise site #91 touched only populates diagnostic fields inside
a raise, never on the happy path.

Also fixed a benchmark methodology bug this investigation surfaced:
scripts/benchmark.py never called decode_frame(), the documented
public chain-walking API #88 shipped -- it kept a hand-rolled copy of
the pre-#88 loop, so every decode-throughput figure this project has
published described code real callers never ran. decode_netprotocols()
and _netprotocols_chain() now call decode_frame() directly; the
now-dead walk() is deleted. benchmarks/baseline.json, five tiers stale
at a v1.3.0-era figure, is refreshed to match.

docs/CLAIMS.md 1.1/1.2/1.6 re-measured under the new methodology;
README's headline numbers updated to match. Full writeup, including
the old-vs-new-methodology breakdown, in CLAIMS.md's new
"Re-measured after closing the dpkt-throughput regression" section.

Closes #147.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RzqEV89zgdGLb3QgnZy3vp
@EONRaider
EONRaider merged commit 13fe17e into master Sep 4, 2026
8 checks passed
@EONRaider
EONRaider deleted the claude/new-session-ghy12u branch September 4, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Close the decode-throughput regression against dpkt

2 participants