Profile by self time: aenum.extend_enum is 16.7% of extraction, dispatch and imports under 1%

[JarryShaw]

Profiling pcapkit.extract() by self time rather than cumulative time puts the largest single cost in aenum.extend_enum, not in dispatch or imports. Filed so the next performance attempt starts from the measurement rather than from a guess.

Credit: this came out of profiling prompted by @Ts-Boom's #551 and #563. Both proposed optimisations that measurement did not support — struct.calcsize is already memoised inside CPython, and the import path's self time is under 1% — but the profiling those PRs prompted is what surfaced the figures below, and the suggestion to profile a real extract() run came from the #551 discussion.

The measurement

cProfile on main at 8cfd6ab01, http.pcap, 1117 frames, 2.09 s total, interpreter 3.14.7:

function	calls	tottime (self)	cumtime
_import_next_layer	3351	0.0121 s (0.58%)	1.888 s (90.2%)
_lookup_next_layer	4695	0.0028 s (0.14%)	0.005 s (0.2%)
_lookup_registry	8490	0.0019 s (0.09%)	0.002 s (0.1%)

Top self-time consumers, same run: aenum.extend_enum at 16.7% — the single largest item in the whole profile — then schema unpack, getattr at 976k calls, schema __new__, isinstance/abc.__instancecheck__, and field unpack.

The lesson for anyone reading a profile of this library: _import_next_layer is a recursive-descent dispatcher, so it sits in the call chain of every nested protocol layer and cumulative time attributes the entire parse beneath it to that one frame. Reading its 90% cumulative figure as a hotspot is the mistake #563's ~40% claim most plausibly came from.

Why extend_enum is plausibly addressable

It is called when a constant lookup meets a value the enumeration does not define and extends the enumeration at runtime — the _missing_ path. On a capture with many unrecognised codes that happens repeatedly. Whether the right answer is memoising the extension, avoiding it for values that will never be reused, or something else is exactly the design work this issue is for; it should not be guessed at.

Worth pairing with the getattr count: 976k calls on a 1117-frame capture is roughly 874 per frame, which suggests attribute access in a hot loop that may be hoistable.

What would make a PR here convincing

Self-time figures before and after on at least two captures — one ordinary such as http.pcap, one exercising the unrecognised-code path heavily such as many_interfaces.pcapng — plus wall-clock extract() timings with the repetition count and host load stated. Note that on a shared host single-digit-millisecond variance swamps microsecond-scale changes, so a claimed improvement smaller than the noise should be reported as unmeasurable rather than as a number. Byte-identical output across the sample captures, as #420 and #427 did.

[JarryShaw]

Reproduced, and the 16.7% and the 976k getattr turn out to be the same finding

Measured on this worktree at ecfe622a6 (a few commits past the 8cfd6ab01 in the issue), CPython 3.14.7, aenum 3.1.17, examples/captures/http.pcap, 1117 frames, extract(store=False, nofile=True, engine='default'), sorted by tottime. Tree provenance asserted in-process (PYTHONSAFEPATH=1, worktree root prepended to sys.path, assert pcapkit.__file__.startswith(<worktree>) before any other import).

Every figure in the issue reproduces:

function	issue: calls / tottime / %	reproduced: calls / tottime / %
aenum.extend_enum	— / — / 16.7%	111 / 0.3604 s / 16.90%
_import_next_layer	3351 / 0.0121 s / 0.58%	3351 / 0.0130 s / 0.61%
_lookup_next_layer	4695 / 0.0028 s / 0.14%	4695 / 0.0030 s / 0.14%
_lookup_registry	8490 / 0.0019 s / 0.09%	8490 / 0.0019 s / 0.09%
builtins.getattr	976k calls	976,700 calls

Three cold runs gave 16.90% / 17.41% / 16.67%, always exactly 111 calls, on total_tt of 2.13 / 2.08 / 2.10 s. So 16.7% is reproducible and not a fluke.

The profile only shows this on the first extract() in a process. A warm run — one extract() discarded, then profile the second — has no extend_enum in the profile at all, and aenum's aggregated self time falls from 17.9% to 1.19%. That distinction turns out to be the whole story.

All 111 calls come from one call site

=== callers of aenum.extend_enum ===
extend_enum: ncalls=111 tottime=0.3604 (16.90%) cumtime=0.5062
    <- pcapkit/const/reg/apptype.py:30584(get)    ncalls=111 cum=0.5062

AppType.get(port, proto=...), reached from the TCP/UDP source and destination port fields. The 111 are 111 distinct unregistered ports — ephemeral client ports, which have no IANA service name, so each one takes the extension path exactly once.

The 976k getattr calls are aenum's, not pcapkit's

The issue reads these as two findings and suggests "attribute access in a hot loop that may be hoistable". They are one finding:

• warm run (no extensions): 61,875 getattr calls
• cold run (111 extensions): 976,700 getattr calls
• difference: 914,825 = 111 × 8,242, and 8,242 is the average len(AppType._member_map_) across the run (8,182 → 8,293)

So 93.7% of all getattr calls in the profile are one loop inside aenum.extend_enum — _enum.py:2780, which does getattr(canonical_member, '_values_', ...) once per existing member. There is no separate pcapkit-side getattr win hiding here; hoisting attribute access in pcapkit's own loops addresses the remaining 6.3%.

The cost is O(members of the target enum)

One extension on each of pcapkit's own enums, 30 reps, same process:

enum	members	µs / extend	ns per member
hip.cipher.Cipher	5	25.7	5130
ipv4.option_number.OptionNumber	30	29.3	977
arp.hardware.Hardware	42	37.6	896
reg.transtype.TransType	151	55.7	369
reg.ethertype.EtherType	160	57.8	361
reg.linktype.LinkType	220	74.1	337
reg.apptype.AppType	8182	3069.6	375

Fits cost ≈ 24 µs + 0.372 µs × existing members. AppType is not special apart from its size — it is the IANA service-name registry, 8,182 members, ~37× the next largest enum.

The same thing shows up across captures. Identical function, per-call cost tracking enum size:

capture	frames	extend_enum calls	tottime	% of total	triggered from
http.pcap	1117	111	0.3604 s	16.90%	apptype.get ×111
many_interfaces.pcapng	64	15	0.0392 s	17.64%	apptype.get ×11, apptype._missing_ ×1, linktype._missing_ ×1, pcapng/option_type.get ×2
test.pcap	34	3	0.0106 s	8.53%	apptype.get ×3
options-transport.pcap	43	2	0.0072 s	12.06%	apptype.get ×2
http6.cap	26	2	0.0075 s	7.00%	apptype.get ×2
ipv4.pcap	4	1	0.0037 s	31.26%	apptype._missing_ ×1
options-internet.pcap	145	2	0.0001 s	0.04%	hip/cipher._missing_ ×1, mh/binding_error._missing_ ×1

Note the last two rows: one call into AppType costs 3.7 ms; two calls into small HIP/MH enums cost 0.1 ms combined — 37× per call, entirely explained by member count.

Also worth recording, since the issue suggests it as the heavy case: many_interfaces.pcapng is not heavier than http.pcap on this path in absolute terms (15 calls vs 111). The capture that exercises the unrecognised-code path hardest is the ordinary one, because what drives the count is distinct ephemeral ports.

"Memoising the extension" is already implemented, and is already ~7,300× effective

AppType.__new__ registers each member into AppType.__members_proto__[proto][port], and AppType.get consults that dict before extending. So the second lookup of the same port never extends:

• first lookup of an unregistered port: 3,070 µs
• second lookup of the same port: 0.42 µs

Wall clock, no profiler, http.pcap, 10 repetitions in one process, host load average 1.64 (16-core Linux, 62 GB):

run  1: 1.0265 s   AppType members 8182 -> 8293 (+111)
run  2: 0.5595 s   AppType members 8293 -> 8293 (+0)
...
run 10: 0.5324 s   AppType members 8293 -> 8293 (+0)

run 1 (cold enums) : 1.0265 s
runs 2..10 mean    : 0.5369 s   stdev 0.0081 s   min 0.5324  max 0.5595
cold-run excess    : 0.4896 s = 47.7% of the cold run

The +111 then +0 column is the direct evidence: the cost is strictly once per distinct value per process, and it is already cached. So of the three options the issue floats, "memoising the extension" is done.

The genuine risk is that it is quadratic, and here is the number

Each extension costs O(current members) and adds a member, so K distinct unknown ports cost K × 24 µs + 0.372 µs × (8182K + K²/2).

Measured, 200 ports per batch starting at 49152:

batch	members after	µs / call
0	8382	3069.8
11	10582	4019.1

1.31× the cost for 1.26× the members — linear per call, quadratic in total. 2,400 distinct unknown ports took 8.46 s; the model predicts 8.43 s (0.4% error), so the model is trustworthy. Extrapolating to a capture that touches the whole dynamic/private range 49152–65535 (K = 16384): ≈100 s inside extend_enum alone, and AppType ends up at ~24,500 members.

That, rather than the 16.7% on http.pcap, is the finding worth acting on.

97.8% of the call is two O(N) passes inside aenum, and one of them is dead code

Timed directly against AppType (8,182 members), per call:

aenum 3.1.17 site	what it does	µs	share of the 3069.6 µs call
_enum.py:2674	base_attributes = set([a for b in enumeration.mro() for a in b.__dict__])	273.7	8.9%
_enum.py:2780	linear alias scan over _member_map_.values()	2727.9	88.9%
	sum	3001.6	97.8%

Two things about those:

1. base_attributes is assigned and never read again anywhere in extend_enum. It is a pure O(N) set build whose result is discarded — 8.9% of every call, for nothing.
2. The alias scan asks "does any existing member already hold this value?". AppType._value2member_map_ answers exactly that question, is maintained by aenum itself, and answers it in 0.0252 µs — about 108,000× faster than the 2727.9 µs scan. (That map is keyed by value and so needs hashable values; aenum already keeps _value2member_seq_ for the unhashable and MultiValue cases, which is where a lookup would have to fall back to the existing scan. On AppType the values are plain str and _value2member_seq_ is empty, so the fast path applies.)

Decision: no pcapkit-side change, and here is why

Taking the issue's three candidate directions in turn:

• Memoise the extension — already done, already 7,300×, proven by the +111 / +0 column above. Nothing left to add.
• Avoid it for values that will never be reused — you cannot know that in advance, and the extension is the contract. Today:

  >>> m = AppType.get(61234, proto=TransportProtocol.get('tcp'))
  <AppType.unknown: 61234 [tcp]>
  >>> m.port, m.svc, m.proto, m.name
  (61234, 'unknown', <TransportProtocol.tcp: 1>, 'PORT_61234_tcp')
  >>> AppType['PORT_61234_tcp'] is m
  True
  

  An unrecognised wire value yields a usable, named, isinstance-correct member. Removing that is precisely the behaviour the issue says must be preserved.
• Something else — 97.8% of the call is inside aenum, and pcapkit passes it nothing it could pass more cheaply.

The one aenum knob that would work was measured, and it breaks the API

NoAlias in _settings_ is the only thing that makes extend_enum skip the alias scan (_enum.py:2774). AppType._settings_ is a plain set, so it can in fact be flipped at runtime, and the saving is large:

baseline (alias scan active)      3024.1 us/call
with NoAlias in _settings_         308.0 us/call     -> 9.8x faster, 90% saving

But it is fatal to the public API, for the whole class and not just for extended members:

>>> AppType(m._value_)
TypeError: NoAlias enumerations cannot be looked up by value      # aenum _enum.py:2248

So the 9.8× is real and unusable. Recorded here so nobody has to re-derive it.

Reimplementing member installation inside pcapkit to skip aenum's two passes would hard-code aenum's private internals (_member_map_, _member_names_, _value2member_map_, _sort_order_, the property redirect) into this library, where an aenum point release could break it silently. That is the speculative rewrite this issue was filed to prevent, so I am not proposing it.

The fix belongs upstream in aenum: drop the dead base_attributes build, and short-circuit the linear alias scan through the _value2member_map_ lookup aenum already maintains, keeping the scan only for the unhashable/MultiValue cases that need it. That removes ~97% of every extend_enum call for every aenum user with a large enumeration. If upstream declines, the next-best option for this library is to reduce K rather than the per-call constant — but K is the number of distinct unrecognised codes on the wire, which is data, not something pcapkit chooses.

Found while measuring: four unrelated defects on the same extension path

Not performance, and not fixed here — they want their own issue — but they surfaced while auditing every extend_enum call site, and all three are verified by running them, not inferred.

1. Command.get crashes on any non-uppercase FTP command. pcapkit/const/ftp/command.py:297-298 guards on key but registers key.upper():

if key not in Command._member_map_:
    return extend_enum(Command, key.upper(), default if default is not None else key)

FTP_REQUEST at pcapkit/protocols/application/ftp.py:36 is compiled with re.I, and ftp.py:99 passes the matched command through verbatim, so:

b'RETR file.txt\r\n' -> cmmd='RETR' -> OK -> <Command.RETR: Retrieve>
b'retr file.txt\r\n' -> cmmd='retr' -> TypeError: 'RETR' already in use as <Command.RETR: Retrieve>
b'Retr file.txt\r\n' -> cmmd='Retr' -> TypeError: 'RETR' already in use as <Command.RETR: Retrieve>

FTP commands are case-insensitive per RFC 959, so a capture containing retr fails extraction on the first occurrence.

2. Method.get has the same mismatch at pcapkit/const/http/method.py:175-176, and _RE_METHOD (httpv1.py:57) has no re.I — but re.match is a prefix match, and httpv1.py:292 passes the whole para1 rather than match1.group('method'):

para1=b'GET'  -> matched b'GET'  -> OK -> <Method.GET>
para1=b'Get'  -> matched b'G'    -> TypeError: 'GET' already in use as <Method.GET>
para1=b'Post' -> matched b'P'    -> TypeError: 'POST' already in use as <Method.POST>
para1=b'get'  -> no match (line is not treated as a request at all)

So this one needs two fixes: the guard/extend mismatch, and passing the captured group instead of the raw token.

For both 1 and 2, an unknown mixed-case value also crashes on the second occurrence (get('xfoo') succeeds as XFOO, then 'xfoo' not in _member_map_ is still true, so it re-extends XFOO and raises).

3. AppType.get with a combined TransportProtocol flag is a latent double-extend. AppType.__new__ registers __members_proto__ only under the single-bit values show_flag_values(proto) yields, so for proto=tcp|udp (whose .name is 'tcp|udp') the guard __members_proto__.get(proto, {}) stays empty forever and the second call re-extends 'PORT_59000_tcp|udp' → TypeError. Not currently reachable: the only production caller is Transport._make_port (pcapkit/protocols/transport/transport.py:165), and tcp.py/udp.py/sctp.py each pass a single-bit constant.

4. A structural collision between the two extension paths, lower severity than 1-3 because it needs a deliberate get() call rather than wire data, but it is in the standard template that most const/ modules share. get() guards on _member_map_ and extends with the name it was given; _missing_ generates 'Unassigned_%d' % value and extends with no name guard at all (e.g. pcapkit/const/arp/hardware.py:161-162 and :177). Both can mint the same name:

get('Unassigned_50', 999) -> <Hardware.Unassigned_50: 999>
then Hardware(50)         -> TypeError: 'Unassigned_50' already in use as property(<Hardware.Unassigned_50: 999>)

Reproduced on arp.hardware.Hardware, arp.operation.Operation and reg.linktype.LinkType. It needs the value to fall in a range _missing_ actually handles — EtherType(0x9123) raises ValueError: 37155 is not a valid EtherType instead, so it is not universal.

In the reverse order there is no crash but the caller's default is silently dropped, because get() falls through to return Cls[key]:

Hardware(1050)                      -> <Hardware.Unassigned_1050: 1050>
then get('Unassigned_1050', 12345)  -> <Hardware.Unassigned_1050: 1050>   # 12345 ignored

Also found, deliberately not fixed here

chardet is a larger cold-run self-time cost than extend_enum on small captures: run_pipeline is 42.8% of test.pcap and 50.2% of http6.cap (3.5% of http.pcap). pcapkit/utilities/chardet.py already wraps it in a bounded cache, so this is first-sight cache-miss cost, not a missing cache. Out of scope for this issue, but worth its own if small-capture latency matters.

[JarryShaw]

Owner decision: gated on #514

In the owner's words:

depending on how exactly it fits with #514. can fold it in and land after #514 if necessary.

So this does not land standalone yet. The #514 part-(c) investigation now running has a second question in its brief: whether 993ee4f65 conflicts with, duplicates, or depends on the registry design part (c) implies — and therefore whether it lands standalone now, is folded into part (c), or lands after it.

State of the work, so it is not lost: the fix exists as an unpushed commit 993ee4f65, "perf(apptype): stop minting an enum member per unregistered port (#575)", in the worktree /local/home/jarryx/GitHub/PyPCAPKit/.claude/worktrees/agent-ae91e21144e108370. Nothing else references it, so that worktree must not be reaped until this is resolved.

The profile reproduces in full against the figures in the issue body — CPython 3.14.7, aenum 3.1.17, examples/captures/http.pcap, 1117 frames, extract(store=False, nofile=True, engine='default'), sorted by tottime, tree provenance asserted in-process. The 16.7% for aenum.extend_enum and the 976k getattr calls turn out to be the same finding rather than two.

[JarryShaw]

Correcting an attribution I made earlier in this thread, because it points at the wrong cost centre.

I had implicated aenum.extend_enum as a meaningful contributor to import and suite time. It is not. Profiled over the suite: 245 calls, 0.010s self time — under 0.1%. Optimising or avoiding it would buy nothing measurable.

The real cost is EnumMeta class construction, which shows 76.7% cumulative under __build_class__, and the single largest contributor is pcapkit/const/reg/apptype.py — 32,936 lines declaring roughly 8,949 AppType members, all built eagerly at import. That file alone accounts for about 160ms of a 654ms cold import pcapkit.

So the lever here is lazy or deferred member construction for the very large registries, not the extend_enum call site. Flagging it so nobody spends effort on the part that is already free.

[JarryShaw]

Measured on current main (daa953d1b), 3.14.7, aenum==3.1.17. The memoisation option is dead and the issue's getattr diagnosis is wrong — but a better target fell out.

1. Repeated-work ratio is exactly 1.0. Instrumented aenum.extend_enum itself, http.pcap:

total calls	import-time	during extract()	distinct (enum, value)	ratio	pairs called twice
122	11	111	122	1.000	0

All 111 in-run calls target AppType. aenum registers the new member in _value2member_map_ permanently, so _missing_ is never re-entered for a value already seen — re-running extract() 7 more times in one process gives 0 further calls. A pcapkit-level memo cache would never hit.

2. The cost is O(current member count), concentrated in one enum. getattr calls per extend_enum, measured:

enum	pre-existing members	getattr	= members + 4
OptionType	40	44	44
LinkType	220	224	224
AppType	8182	8186	8186

So this is not "119 enums have a slow path" — it is one enum that ships 8182 static members, where every mutation costs ~40–200× a normal const enum.

3. Correction to this issue: the 976,720 getattr calls are not a hoistable pcapkit loop. 93.6% (914,529) are made inside aenum.extend_enum, invoked ~111 times — not 874 per frame in pcapkit code. The arithmetic is right, the attribution is wrong.

Self time reproduces unchanged: 15.85% ± 0.34pp over 5 fresh processes (http.pcap), 16.80% on many_interfaces.pcapng. The 16.7% at 8cfd6ab01 has not moved.

Separate, smaller finding: 55,846 getattr calls (~1.7% of the run) come from copy.copy() at pcapkit/corekit/fields/field.py:566, collections.py:71, misc.py:135,286. Real but unrelated — say if you want it filed; hoisting it risks mutable-default aliasing and needs its own look.

[JarryShaw]

Correction to my comment above: the extension happens in get, not in _missing_. Verified by recording the calling frame — on http.pcap, 111 of 111 in-run extend_enum calls come from AppType.get, zero from _missing_.

pcapkit/const/reg/apptype.py:30596-30624 — get calls _missing_ itself, and mints when _missing_ declines:

try:
    ret = AppType._missing_(key)
    if ret is None:
        raise ValueError
except ValueError:
    ret = extend_enum(AppType, 'PORT_%d_%s' % (key, proto.name), key, 'unknown', proto)

_missing_'s 766 range branches stop at 49149, so ephemeral ports 49152–65535 fall through all 766 comparisons, return None, and get extends at :30620. Each such lookup pays 766 Python comparisons, an exception round-trip, and the O(N) extension.

Three further corrections to figures in my comment:

claimed	actual
~119 enumerations	123 classes / 117 modules; 121 define _missing_; only 110 mint via extend_enum
getattr = members + 4	confirmed, and it is len(cls.__members__) + 4 — LinkType (220 members, 219 values) discriminates
self-time ±0.34pp over 5 runs	±0.43pp over 7 fresh processes (15.26–16.11%)

The 1.000 call ratio and the AppType-dominates-cost finding both reproduce unchanged.

Unrelated live bug found on the way, not filed separately: AppType.get('tcpmux') raises TypeError: 'tcpmux' already in use. get's string path tests key in AppType.__members_proto__, which is keyed by TransportProtocol, so a str never matches and the fall-through re-registers an existing name. Almost certainly meant __members__. Worth folding into #732 rather than replicating into N classes.

[JarryShaw]

Dropping blocked — this was never waiting on a dependency, and its evidence has since gone stale in two ways.

The cited code no longer exists. The measurement above points at pcapkit/const/reg/apptype.py:30596-30624. #754 deleted that file; AppType is now a package of per-transport registries (pcapkit/const/reg/apptype/), and the profile needs re-taking against 932cb48d1 before any fix is designed.

More usefully: the mechanism recorded here is the same code path as #758. This issue's finding was that 111 of 111 in-run extend_enum calls come from AppType.get and zero from _missing_, because get calls _missing_ itself and mints when _missing_ declines. #758 is that exact try/except ValueError — it swallows _missing_'s range guard and mints regardless. A live worker is fixing it now.

So the two interact, and it is worth stating which is which so neither gets credited with the other's work:

• const(apptype): AppType.get() swallows the range guard and mints out-of-range ports #758 is correctness — stop minting for a port that is out of range at all.
• This issue is cost — stop minting on the hot path for ports that are merely unassigned, which is where the 16.7% went.

Fixing #758 narrows the mint path but does not close this one, since an unassigned in-range port still mints. Two things worth measuring once #758 lands: whether extend_enum's share has moved at all post-split, and whether corekit/fields/numbers.py:540-552's existing argument — that not minting for ephemeral ports is the real lever — is now the cheapest fix.

No dependency, so no blocked. It needs a fresh profile, not a decision.