Registries mint a permanent member for every unrecognised value: 1,169 extend_enum sites across 113 registries

[JarryShaw]

Is your feature request related to a problem? Please describe.

Every registry under pcapkit/const/ turns an unrecognised value into a permanently registered enum member via aenum.extend_enum, so a process that parses many distinct unassigned values grows its registries without bound. #575 measured the cost on one path — 111 calls and ~13.5% of extraction self time on http.pcap — but the mechanism is repo-wide.

The maintainer's ruling, verbatim: "i think we should even apply to all other Enum's legit but unbounded values - so that we dont create registered enums out of unrecognised/unregistered values, unless user/caller explicitly created them."

Describe the solution you'd like

Return an unregistered member of the namespace class instead of minting. Measured as viable on AppType:

obj = str.__new__(TCP, 'unassigned')
obj._name_ = 'unassigned'; obj._value_ = 'unassigned'
obj.port = 54321; obj.svc = 'unknown'; obj.proto = TransportProtocol.tcp

repr             : <TCP.unknown: 54321 [tcp]>
isinstance TCP   : True
registry grew    : 6147 -> 6147 members, 6147 -> 6147 values

It satisfies isinstance, reprs identically to a declared member, and does not grow the registry. Note AppType is a StrEnum (TCP → AppType → StrEnum → str → ReprEnum), so it is str.__new__; an int-valued registry needs int.__new__ instead, and int.__new__(TCP, …) raises TypeError: TCP is not a subtype of int.

Scope, measured by AST over pcapkit/const/

extend_enum inside _missing_ : 1056 calls across 113 files
extend_enum inside get()     :  113 calls across 112 files

1,169 sites across 113 registries. Both paths mint per value — including the _missing_ declared-range branches, which is not obvious:

TCP.get(49100)   -> <TCP.unassigned: 49100 [undefined]>   members 6147 -> 6148
TCP.get(49101)   -> <TCP.unassigned: 49101 [undefined]>   members 6148 -> 6149
LinkType(60000)  -> <LinkType.Unassigned_60000: 60000>    members  220 ->  221

Those branches are bounded in which ranges they cover, not in how many members they create.

Additional context

#771 implements this for the four EnumField subclasses only — the parse path, where the measured cost is — and closes #575. This issue is the rest.

Three questions to settle before a wide change, none resolved: a value-lookup like TCP(54321) stays failing since the instance is absent from _value2member_map_; AppType.__eq__/__hash__ key on .port, so two unregistered instances for one port compare equal without being identical; and pickling or copying an unregistered member may not survive. Also open is whether this belongs in each registry's _missing_ — where aenum naturally hooks and where 1,056 of the sites are — rather than at the field layer. The ruling's "unless user/caller explicitly created them" means a direct get() call may still register; only paths that never asked for a name should get the unregistered member.

[JarryShaw]

Design ruled by the maintainer, verbatim: "for .get(), it's not an explicit call. I say we can probably add something like a .register(...) call that takes code, name, and any other required values to generate a proper enum as an explicit call. for _missing_, it is still scoped to those ranged bounded but not expanded branches, like for ports 100-200, they are all reserverd_nnn branch. this ruling must apply to all enums under pcapkit.const, not just the AppType classes."

So the shape is settled and wider than this issue was filed as:

1. Neither get() nor _missing_ may register. Both return an unregistered member of the namespace class. That covers all 1,169 sites — 1,056 in _missing_, 113 in get() — across 113 registries.
2. Registration becomes opt-in through a new .register(...), taking the code, the name and whatever else that registry's members need. Nothing named register exists under pcapkit/const/ today.
3. It applies to every enum under pcapkit.const, not only the AppType family.

That corrects the note above: I had said a direct get() call was explicit enough to keep registering. It is not — lookup never registers, only .register() does.

Two blockers to resolve before implementing, both measured:

.register() cannot have a single signature. Member shapes differ materially:

# pcapkit/const/reg/apptype/apptype.py:2282
def __new__(cls, value: 'int', name: 'str' = '<null>',
            proto: 'TransportProtocol' = TransportProtocol.undefined) -> 'Type[AppType]':
# pcapkit/const/reg/linktype.py:18
class LinkType(IntEnum):          # no custom __new__ at all

AppType members carry port/svc/proto; LinkType carries only name and value. So either .register() is declared per class with its own signature, or it takes **kwargs and each registry validates — the latter keeps one call site but converts a signature error into a runtime one. Worth choosing deliberately, since 113 registries inherit whichever it is.

Three tests currently depend on registration-through-lookup and must move to .register():

tests/const/test_const_apptype_split_unit.py:161  addCleanup(_purge_member, TCP, 'PORT_59001_tcp', 59001)
tests/const/test_const_apptype_split_unit.py:178  addCleanup(_purge_member, UDP, 'unit_test_80', 80)
tests/const/test_const_apptype_split_unit.py:272  addCleanup(_purge_member, TCP, 'PORT_51_tcp', 51)

This is breaking — a value parsed or looked up no longer appears in __members__ afterwards, and Registry[name] for a previously-minted name stops resolving. Staging matters: 113 registries in one change is not reviewable, and the three caveats from #771 still need answering first — value lookup stays failing for an unregistered instance, AppType.__eq__/__hash__ key on .port so two such instances compare equal without being identical, and pickling an unregistered member may not survive.

[JarryShaw]

Both blockers ruled, in the maintainer's words: "1. .register(...) lives per register with its own signature. 2. update the tests then."

So the design is fully specified. And measuring the implementation surface changes how large this is — it is generator work, not 113 hand-edits:

hand-written (non-generated) registries under pcapkit/const/ : 0
crawlers under pcapkit/vendor/ containing extend_enum        : 90
extend_enum emit sites in pcapkit/vendor/                    : 115
  default.py                                                :   3
  the other 90 crawlers                                      : 112

Every one of the 121 registries is generated, so the 1,169 sites in pcapkit/const/ are output, not source. The source is 115 emit sites across 91 files, of which the three in pcapkit/vendor/default.py are the shared template:

# default.py:96  -- get()'s fall-through
if key not in {NAME}._member_map_:
    return extend_enum({NAME}, key, default)

# default.py:364 -- the _missing_ range branches
miss.append(f"    return extend_enum(cls, '{self.safe_name(name)}_%d' % value, value)")

Editing default.py alone changes the behaviour of every registry that does not override it. The other 112 sites are per-crawler overrides and are the long tail.

Suggested staging, since 91 files in one PR is not reviewable:

1. default.py plus .register() on the base class, then regenerate. This is the whole design landing at once for every non-overriding registry, in one reviewable diff — plus the three test updates at tests/const/test_const_apptype_split_unit.py:161, :178, :272. Measure the extend_enum call count on a real extraction before and after; it should go to zero for those registries.
2. The 90 overriding crawlers, in batches. Each needs its own .register() signature per the ruling, so they cannot be mechanised as one sweep. Group by member shape rather than by protocol — registries sharing a __new__ shape share a signature.
3. The three caveats from fix(fields): fall through to the bounded pseudo-member instead of minting (#575) #771 still gate step 1 and none is answered: an unregistered instance stays unfindable by value lookup; AppType.__eq__/__hash__ key on .port, so two such instances compare equal without being identical; and pickling or copying one may not survive.

One consequence worth stating before anyone starts: this is breaking in a way that is invisible until runtime — a value parsed or looked up no longer appears in __members__, and Registry['PORT_54321_tcp'] stops resolving for a name that used to appear after parsing. The three tests above are the only in-tree dependants I found, but downstream callers are not measurable from here.

[JarryShaw]

Labelling this blocked, on a file conflict rather than on a decision — the design here is fully ruled and needs nothing further from the maintainer.

Step one regenerates every registry from pcapkit/vendor/default.py, which rewrites pcapkit/const/reg/apptype/apptype.py and pcapkit/vendor/reg/apptype/apptype.py. #777 owns both of those files (head 19b361770, review: pending), and #759 is already blocked on the same pair. Starting a 121-registry regeneration underneath it would clobber it, silently — the later write wins and nothing announces the earlier one happened.

#771 is the second hold, and a softer one: it is the bounded-fallback rule this issue generalises, and it is live in pcapkit/corekit/fields/numbers.py at review: needs-changes. Landing the generalisation before the specific case it generalises would mean writing .register() against a fallback that is still moving.

Checkable blocker: #777 and #771 both merged. Then step one goes out as one PR — default.py, the per-registry .register() signatures, the regeneration, and the three test updates at tests/const/test_const_apptype_split_unit.py:161,178,272 — followed by the 90 overriding crawlers batched by __new__ shape.

[JarryShaw]

Blocker moves from #777/#771 to #759. Both of the earlier holds have cleared — #771 merged as 9012f57f8 and #777 as fe80b8525 — so main now carries the bounded-fallback rule this issue generalises, and the .register() design can be written against it rather than against a moving target.

But step one regenerates all 121 registries, which rewrites pcapkit/const/reg/apptype/apptype.py and pcapkit/vendor/reg/apptype/apptype.py, and #759 is now in flight on exactly those two files with its _dispatch multi-bit fix. Regenerating underneath it would clobber it silently. Landing #759 first also means its fix is what the regeneration emits, instead of having to be re-applied afterwards.

Checkable blocker: #759 merged. Then step one goes out as one PR — pcapkit/vendor/default.py, the per-registry .register() signatures, the regeneration, and the three test updates at tests/const/test_const_apptype_split_unit.py:161,178,272 — followed by the 90 overriding crawlers batched by __new__ shape.

[JarryShaw]

#759 closed at 16:21:44Z, when #783 merged as f41e5f880. So the blocker this issue named has cleared — but the same reasoning now points at a different PR, so this stays blocked rather than opening.

I should have caught this two sweeps ago; the label was stale for roughly 25 minutes while I re-checked it against memory instead of against #759's state. Recording that rather than quietly re-blocking.

Blocker moves from #759 to #797. Step one of this issue regenerates all 121 registries, which rewrites pcapkit/const/reg/apptype/apptype.py and pcapkit/vendor/reg/apptype/apptype.py — and #797 (053c0889d, refactor/const/review: pending) is in flight on exactly those two files with #792's f-string conversion. Regenerating underneath it would clobber it silently: no error, the later write simply wins. Landing #797 first also means its six converted raises are what the regeneration emits, rather than having to be re-applied afterwards — the identical argument that held for #759's _dispatch fix.

Checkable blocker: #797 merged. gh pr view 797 -R JarryShaw/PyPCAPKit --json state,mergedAt.

Note this makes #797 the gate for two issues now — this one and #796 — so it is worth landing before anything else touches pcapkit/vendor/.

Unchanged from the previous comment: once clear, step one goes out as one PR — pcapkit/vendor/default.py, the per-registry .register() signatures, the regeneration, and the three test updates at tests/const/test_const_apptype_split_unit.py:161,178,272 — followed by the 90 overriding crawlers batched by __new__ shape. Those three test line numbers predate #783 and will have shifted; re-derive them rather than trusting the list.

[JarryShaw]

#797 merged at 18:35:36Z as 477ed00c4, so the blocker named here has cleared — but the same argument now points at the vendor work, so this stays blocked rather than opening.

Step one of this issue regenerates all 121 registries, which rewrites pcapkit/vendor/** and pcapkit/const/** wholesale. Two issues are live in exactly that tree:

• vendor: three bespoke templates and AppType's dunders still use %-formatting, so the f-string disable cannot drop #798 (wip) — the %-style guards in pcapkit/vendor/{tcp/flags,ftp/command,http/method}.py plus the AppType dunders.
• vendor: thirteen re.sub sites pass a regex flag as the positional count, capping substitutions at 8 #796 (blocked on vendor: three bespoke templates and AppType's dunders still use %-formatting, so the f-string disable cannot drop #798) — thirteen re.sub sites across pcapkit/vendor/, two of which vendor: three bespoke templates and AppType's dunders still use %-formatting, so the f-string disable cannot drop #798 also touches.

A regeneration underneath either would discard their edits silently — no error, the later write wins — and regenerating after them means their changes are what the crawl emits, rather than having to be re-applied by hand against a 12,341-member diff.

Checkable blocker: #796 merged, which is the last of the three since #796 is itself blocked on #798. That makes the whole chain: #798 → #796 → this issue.

gh issue view 796 -R JarryShaw/PyPCAPKit --json state

Two things to re-derive rather than trust when this starts, both of which have already bitten today:

• The three test line numbers in my earlier comment — tests/const/test_const_apptype_split_unit.py:161,178,272 — predate fix(const): refuse a multi-bit proto in AppType._dispatch rather than resolving it LSB-first (#759) #783, refactor(const): convert AppType's six %-style raises to f-strings (#792) #797 and everything since. Re-derive them.
• AppType.LINK fetches IANA's live CSV. A regeneration is not reproducible against a fixed input, so the diff will carry unrelated row churn. refactor(const): convert AppType's six %-style raises to f-strings (#792) #797 handled this by hand-editing and proving equivalence — expanding BASE and diffing the changed region for 0 diff lines. Step one here genuinely wants the crawl, so that trick does not apply; plan for reviewing a diff that mixes intended structural change with IANA drift, and say how you will separate them.

[JarryShaw]

#796 merged via #813, so the stated blocker has cleared — but this now waits on #815 instead. Re-blocked rather than opened.

Checkable blocker: #815 merged.

gh pr view 815 -R JarryShaw/PyPCAPKit --json state,mergedAt

Step one of this issue regenerates all 121 registries, which rewrites pcapkit/const/reg/apptype/{apptype,tcp,udp,sctp,dccp}.py and vendor/reg/apptype/apptype.py — every one of which #815 is rewriting right now, and #815 is breaking: it retypes AppType.proto to a single transport and changes the underlying str value of ~10,625 members. A regeneration landing first would discard that wholesale, with no conflict marker.

And #815 changes what this issue has to do. Its per-registry proto retyping removes the multi-bit values that this issue's "minting" analysis was written against, so re-derive the population figures after it lands rather than reusing any number from this thread. The three test line numbers cited in my earlier comment predate #783, #797, #803 and #813 — treat them as stale.

Unblocks when #815 merges, which also clears #807, #808 and #809.