Close the decode-throughput regression against dpkt - #152
Merged
Conversation
Re-measuring every held claim after #107/#124 closed surfaced a regression none of the five prior tiers had been watching for: decode throughput against dpkt had reversed from 1.16x faster (right after Tier 1) to ~11% slower, without anyone profiling why. cProfile against the corpus decode loop found two causes -- neither one of the three tiers originally suspected: - bytes_to_ipv6 (added by #99 for Pyodide portability, landing after all three originally-suspected tiers) formatted its eight address words one f-string at a time through a generator, ~18% of total corpus decode time. Reformatted to one %-format call plus a split, byte-identical output (verified against glibc's inet_ntop via the existing hypothesis test), 2.1x faster in isolation. - decode_frame()'s Packet(*layers, ...) construction paid a redundant isinstance(layer, Protocol) check per layer through Protocol's ABC __instancecheck__ machinery, ~7-8% of decode_frame()'s own time -- every element in that list is already guaranteed to be a Protocol instance by decode_frame()'s own loop. Packet gained a private _from_decoded() fast-construction path (object.__new__ plus direct attribute assignment, same shortcut _base.py already uses for Ethernet/ARP/IPv4), used only by decode_frame(). The public Packet(...) constructor is untouched. #87's registry dispatch and #91's structured diagnostics were both ruled out directly rather than re-guessed: every next_protocol() override still hits its documented single-dict.get fast path, and every raise site #91 touched only populates diagnostic fields inside a raise, never on the happy path. Also fixed a benchmark methodology bug this investigation surfaced: scripts/benchmark.py never called decode_frame(), the documented public chain-walking API #88 shipped -- it kept a hand-rolled copy of the pre-#88 loop, so every decode-throughput figure this project has published described code real callers never ran. decode_netprotocols() and _netprotocols_chain() now call decode_frame() directly; the now-dead walk() is deleted. benchmarks/baseline.json, five tiers stale at a v1.3.0-era figure, is refreshed to match. docs/CLAIMS.md 1.1/1.2/1.6 re-measured under the new methodology; README's headline numbers updated to match. Full writeup, including the old-vs-new-methodology breakdown, in CLAIMS.md's new "Re-measured after closing the dpkt-throughput regression" section. Closes #147. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RzqEV89zgdGLb3QgnZy3vp
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-measuring every held claim after #107/#124 closed surfaced a regression none of the five prior tiers had been watching for: decode throughput against dpkt had reversed from 1.16x faster (right after Tier 1) to ~11% slower, without anyone profiling why. This closes #147 by root-causing it with
cProfile/pstatsrather than guessing from the tier history, fixing what the profile actually showed, and fixing a benchmark-methodology bug the investigation surfaced along the way.What's included
bytes_to_ipv6sped up (src/netprotocols/_base.py). Added by Prove the browser claim with a Pyodide/WASM CI job #99 for Pyodide portability — not one of the three originally-suspected tiers (A public protocol registry #87/Ship decode_frame() — the chain walker belongs in the library #88/Structured parse diagnostics on ProtocolError #91) — it formatted its eight address words one f-string at a time through a generator, ~18% of total corpus decode time by profile. Reformatted to a single%-format call plus a split, verified byte-identical against glibc'sinet_ntopvia the existingtest_bytes_to_ipv6_matches_glibchypothesis test, 2.1x faster in isolation.Packet._from_decoded()fast-construction path (src/netprotocols/packet.py), used only bydecode_frame()(src/netprotocols/walk.py).decode_frame()'sPacket(*layers, ...)call paid a redundantisinstance(layer, Protocol)check per layer throughProtocol's ABC__instancecheck__machinery — ~7-8% ofdecode_frame()'s own time — on a list already guaranteed homogeneous bydecode_frame()'s own loop. The publicPacket(...)constructor is untouched and still validates arbitrary caller-supplied arguments.next_protocol()override (still hits its documented single-dict.getfast path) and every raise site Structured parse diagnostics on ProtocolError #91 touched (diagnostic fields populate only inside a raise, never on the happy path).scripts/benchmark.py).decode_netprotocols()/_netprotocols_chain()had never calleddecode_frame(), the documented public chain-walking API Ship decode_frame() — the chain walker belongs in the library #88 shipped — they kept a hand-rolled copy of the pre-Ship decode_frame() — the chain walker belongs in the library #88 loop, so every decode-throughput figure this project has published described code real callers of the documented API never ran. Both now calldecode_frame()directly; the now-deadwalk()is deleted.benchmarks/baseline.jsonrefreshed — five tiers and this fix stale at a v1.3.0-era figure (114,388 f/s), now 98,007.5 f/s / 6.5677 normalized, matching the new methodology.docs/CLAIMS.md— new "Re-measured after closing the dpkt-throughput regression" section with the full profiling writeup and an explicit old-vs-new-methodology breakdown (closing the gap under the old hand-rolled-loop measurement vs. whatdecode_frame()'s real, deliberate overhead costs under the new one); 1.1/1.2/1.6 re-measured.README.md— headline scapy ratio updated (5.6x → 5.4x, dpkt gap direction unchanged).CHANGELOG.md— new## [Unreleased]entry.Numbers
Old methodology (hand-rolled loop) closed the dpkt gap from 1.12x back to near parity (~1.02-1.03x) — the two accidental-overhead fixes alone did nearly all of that. New methodology (
decode_frame(), what ships) measures more real work (bounded-depth check, fullPacketconstruction), so the final published figure is a smaller but more honest gap:decode_frame())Down from ~11% slower than dpkt to ~9% slower, on a workload that now measures more of what
decode_frame()actually does. Decode depth unchanged (27/70 split vs dpkt).Verification
uv run --frozen ruff check .anduv run --frozen ruff format --check .are cleanuv run --frozen mypyis clean (strict)uv run --frozen pytestpasses locallyuv run --frozen python scripts/benchmark.py --check --threshold 15passes against the refreshed baselineCHANGELOG.mdhas an entry under## [Unreleased]bytes_to_ipv6's existing glibc-differential hypothesis test passes against the new implementationNotes
Tracked by #147, with sub-issues #148-#151 for the profiling/fix/re-measure/README-update pieces (deliberately lighter-weight than the full Tier-N epic apparatus the five prior tiers used — this is a smaller, investigative fix, not a new tier).
Does not bump
pyproject.toml's version or touch release tagging.Closes #147.
🤖 Generated with Claude Code
https://claude.ai/code/session_01RzqEV89zgdGLb3QgnZy3vp
Generated by Claude Code