Skip to content

fix(const,vendor): return the unregistered member instead of re-entering _missing_ (#866) - #867

Merged
JarryShaw merged 1 commit into
mainfrom
fix/866-crawler-missing-recursion
Sep 28, 2026
Merged

JarryShaw merged 1 commit into
mainfrom
fix/866-crawler-missing-recursion

Conversation

@JarryShaw

@JarryShaw JarryShaw commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Please follow the guide below

What is the purpose of your pull request?

  • fix — corrects a defect

Description of your pull request and other information

Fixes #866 — main is red right now: 12 tests across three files, all RecursionError. tests/const/test_const_enum_lookup.py, tests/const/test_const_enum_no_mint.py (RulingConversionDoesNotMintTests.test_converted_value_does_not_mint and test_repeated_lookup_does_not_grow_members, two registries each), and tests/protocols/misc/test_pcapng_unit.py (7 of the 12). Five Python 3.x legs plus Gate and Required checks passed, one defect.

pcapkit/vendor/pcapng/{record_type,secrets_type}.py rendered a two-line _missing_ body whose first line did not return:

miss = [
    "cls._unregistered_member(value, 'Unassigned')",
    'return cls(value)'
]

Harmless while line one was extend_enum(...), which registers the member so the following cls(value) found it. #861 replaced it with _unregistered_member, which deliberately does not register — so cls(value) missed again and re-entered _missing_.

#861 only half-corrected it. It fixed the two generated files and never touched the two crawlers — and in the generated files it left return cls(value) in place as unreachable dead code after the new return. That short-circuit is why #861's own CI and 70fa92010 were green (59 ok / 0 fail); e58618bdf regenerated from the stale crawlers and reintroduced it live. #861's cross-review verified byte-identical regeneration for 5 sampled crawlers of 82, and these two were not sampled.

Both crawlers collapse to the single returning line the other 101 registries use, and the two const files are regenerated from them. Note record_type.py has LINK = 'https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-03.html' and fetches it — only secrets_type.py is offline — so that regeneration is not reproducible without connectivity, though it came back byte-identical.

The guard is at the crawler layer deliberately: test_const_enum_lookup.py already sweeps every committed registry and is what caught this, so the uncovered layer was the source a crawler emits. The invariant it enforces is narrow and true — a hard-coded body must end in a return, and no emitted line may re-enter _missing_ for the same value. It is not "every emitted line must return": 53 crawlers emit 109 legitimately non-returning lines, the shape Vendor.process itself uses at default.py:341-343. And the guard sees only 35 of 96 crawlers; 61 are invisible (53 append-built, 8 returning no (enum, miss) pair) and are covered behaviourally instead.

Verified: 6 new tests pass; with the four source files reverted, 4 of 6 fail counted as methods — the self-recursive-lookup guard, the named-pair guard, and both behavioural lookups. The other two pass either way, the last-line check by design since #866's body ended on return cls(value). tests/vendor 92, tests/project 156, tests/const 190, test_pcapng_unit.py 93, all green on the fixed tree.

@JarryShaw JarryShaw added fix Pull requests that fix a defect (fix: subject prefix) const Regenerated IANA or vendor constant tables; members keep their numeric values review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Sep 27, 2026
@JarryShaw
JarryShaw force-pushed the fix/866-crawler-missing-recursion branch from c6d6d70 to 847451d Compare September 27, 2026 23:43
@JarryShaw

Copy link
Copy Markdown
Owner Author

NEEDS CHANGES on c6d6d7068 — cross-review (opus). The code passed clean, in the reviewer's words: "the 4-line code fix is correct and I would merge it as-is". All six required changes are prose or scoping, and three were false claims I wrote. Posting this late — it should have gone up when the verdict landed rather than living only in my session.

  1. The test's headline invariant is false. "Every line a crawler emits into a _missing_ body must return" — measured: 53 crawlers emit 109 legitimately non-returning lines, from the base class itself (pcapkit/vendor/default.py:341-343 emits an if header, a #: comment, then an indented return; three registries carry that shape today). So the assertion message instructed a future author to make valid code wrong.
  2. "all 6 fail with the four files reverted" is wrong — 5 of 6. test_the_sweep_size_is_pinned passes, correctly. I read FAILED (failures=6, errors=2) as six failing tests when it is six subTest records across three methods — the exact aggregation trap this PR's own body warns about two paragraphs earlier.
  3. "these two take a local DATA dict rather than a live fetch" is false for record_type.py: LINK = 'https://www.ietf.org/archive/id/draft-tuexen-opsawg-pcapng-03.html', fetched with no cache. Only secrets_type.py is offline.
  4. "refactor(const,vendor): apply the mint/unmint criterion ruling to 82 registries (#775) #861 corrected the two generated files" is wrong in a way that matters — it half-corrected them, leaving return cls(value) as unreachable dead code after the new return. That is why refactor(const,vendor): apply the mint/unmint criterion ruling to 82 registries (#775) #861 was green, and a statement after a return in generated output is itself a signal that the generator disagrees with the file.
  5. The guard's blind set is 61 of 96 crawlers, not the 2 named — 35 readable, 53 append-built, 8 returning no (enum, miss) pair. And the pin at 35 catches a crawler leaving the sweep but not a new one that was never in it.
  6. 12 failures across three files, not the one test cited — test_const_enum_lookup.py, test_const_enum_no_mint.py, and tests/protocols/misc/test_pcapng_unit.py (7 of the 12), the last outside the directories this PR says were verified.

One finding refutes the guard's own premise and is worth keeping: a "does every _missing_ path return" check would not have caught #866. The reviewer's AST sweep found zero fall-through bodies on main — the defect is a returning call to cls(value), a cycle rather than a missing return. So test_no_crawler_emits_a_self_recursive_lookup is the load-bearing test. Retargeting the other to "the last line must return" drops the honest fail-without-the-fix count to 4 of 6, which is better-scoped rather than weaker.

Confirmed alongside: scope is exactly 2 of 103 registries (re-derived by AST, not grep); regeneration byte-identical for both files, verified twice; tests/const 190, tests/vendor 92, tests/project 156, test_pcapng_unit.py 93 all green on the fixed tree; pylint 10.00/10 on the four changed files; isort clean; one pre-existing mypy [override] on vendor/pcapng/record_type.py:55, byte-identical on main. breaking correctly withheld — nothing that worked stops working.

A worker is applying all six now. No code changes, so this verdict carries to the amended head.

…ing _missing_ (#866)

- `pcapkit/vendor/pcapng/{record_type,secrets_type}.py` rendered a two-line
  `_missing_` body whose first line called `_unregistered_member` without
  returning it and whose second was `return cls(value)`. That was harmless
  while the first line was `extend_enum`, which registers the member so the
  following lookup found it; #861 replaced it with `_unregistered_member`,
  which deliberately does not register, so the lookup missed again and
  re-entered `_missing_` until `RecursionError`
- collapse both to the single returning line the other 101 registries already
  use, and regenerate `pcapkit/const/pcapng/{record_type,secrets_type}.py`
  from the fixed crawlers
- add `tests/vendor/test_vendor_missing_body_unit.py`, a crawler-layer guard:
  every constant line a crawler emits into `miss` must `return`, none may be
  `return cls(value)`, and the sweep size is pinned at 35 so a crawler joining
  or leaving is deliberate. #861 fixed the generated files and not the
  generators, so the defect was latent until `e58618bdf` regenerated

Build: 6 new tests pass. With the four source files reverted, 4 of the 6 fail
(counted as methods, not subTest records): the self-recursive-lookup guard, the
named-pair guard, and both behavioural lookups. `test_the_sweep_size_is_pinned`
and `test_the_hard_coded_body_ends_in_a_return` pass either way -- the latter by
design, since #866's body ended on `return cls(value)`, which returns.
`tests/const/test_const_enum_lookup.py` also fails on the reverted tree, which
is what caught this on main. tests/vendor 92 and tests/project 156 green.
@JarryShaw
JarryShaw force-pushed the fix/866-crawler-missing-recursion branch from 847451d to d22b71f Compare September 27, 2026 23:45
@JarryShaw JarryShaw added review: good-to-go Cross-review at the current head says ready; CI state is separate and removed review: pending No verdict for the current head - never reviewed, or the head moved since the last one labels Sep 27, 2026
JarryShaw added a commit that referenced this pull request Sep 27, 2026
Both `get`'s `cls(default)` sites -- the `str` branch's and the value
branch's -- reached `_missing_` for a `default` that fell inside a
still-minting registry's own range, growing the registry as a side
effect of resolving `default` rather than `key`.

- Replace both `cls(default)` calls with a plain `_value2member_map_`
  lookup (owner's ruling, option 1): `default` can no longer mint by
  construction, and a `default` naming no registered member now falls
  through to the same lookup error `key` itself would have raised.
- `key` resolution is unchanged: `EtherType.get(0x0888)` still mints
  `Xyplex_0x0888` through `_missing_`, per the ruling's own exception.
- Update the docstring's "It never mints" caveat, the now-false claim
  that an unregistered default still reaches `cls(default)`, and the
  Args/Raises text to match.
- Add `GetDefaultNoMintTests` pinning the issue's own repro, the
  preserved key-path mint, default resolving to a registered value
  (int and str), and the exception type per path. Update two
  `NoDefaultSentinelTests` assertions that asserted the old
  `cls(default)` failure mode for an unregistered `-1`/`-1.0` default.
- Update four assertions in `test_const_enum_get.py` and
  `test_const_enum_builtin_parity.py` that encoded the superseded
  contract (an unresolvable/unregistered default failing with its own
  named error, or resolving into a declared-but-unassigned range) --
  each retargeted to the new contract rather than weakened, with a
  registered-default check added back where the old one only proved
  the "genuinely consulted" half via a value that no longer resolves.

Owner's rulings, verbatim:
- "Take (b). Only register can mint. get should not mint unless it
  falls through the _missing_'s minted ranges."
- Choosing option 1 of three proposed: "I think 1 is correct mechanism
  we'd like."

Build/tests, under plain unittest: test_const_registry_protocol.py
74/74, test_const_enum_get.py 8/8, test_const_enum_builtin_parity.py
33/33, tests/vendor/ 86/86 -- no regressions. tests/const/ as a whole
carries 5 unrelated pre-existing failures (SecretsType/RecordType
recursion, filed as #866, fixed in #867) plus one further out-of-scope
failure in test_const_enum_no_mint.py (contended with #865, reported
separately rather than edited here).
@JarryShaw

Copy link
Copy Markdown
Owner Author

GOOD TO GO on d22b71f6d — this supersedes the NEEDS CHANGES above, which I posted late and against the superseded c6d6d7068. Apologies for the ordering: the six items were already fixed when that comment went up, and the label was flipped without this note, so the PR read as a rejection with a green label. This is the record that was missing.

All six applied, prose only:

  1. Invariant restated — "A hard-coded _missing_ body must end in a return, and no emitted line may re-enter _missing_ for the same value" — with the default.py:341-343 counter-example beneath it. test_every_emitted_line_returns retargeted and renamed to test_the_hard_coded_body_ends_in_a_return, checking only lines[-1], and its docstring now says outright that it does not catch main is red: two pcapng crawlers emit a self-recursive _missing_ body, causing RecursionError #866 alone and that the self-recursive-lookup test is the one that does.
  2. Blind set corrected in both places to 61 of 96 crawlers (35 readable, 53 append-built, 8 returning no (enum, miss) pair), plus the skip-by-construction cases and the "catches leaving, not joining" asymmetry.
  3. refactor(const,vendor): apply the mint/unmint criterion ruling to 82 registries (#775) #861's history corrected to half-corrected, with the before/after diff and the short-circuit explanation for why its CI was green.
  4. Blast radius corrected to 12 failures across three files, naming test_pcapng_unit.py for its 7.
  5. The commit message's Build: line corrected from "all 6 fail" to 4 of 6 counted as methods, naming which two pass either way and why.
  6. The PR body and main is red: two pcapng crawlers emit a self-recursive _missing_ body, causing RecursionError #866's body corrected — including dropping the false "local DATA dict" claim, since record_type.py fetches ietf.org.

Why the verdict carries rather than needing a fourth round: git diff c6d6d7068 d22b71f6d -- pcapkit/ is empty. The four source files are byte-identical to the sha the review passed clean, and the reviewer's position on them was explicit — "the 4-line code fix is correct and I would merge it as-is". Only tests/vendor/test_vendor_missing_body_unit.py moved, 175 → 239 lines, all prose.

Verified on the amended head: 6 tests pass; with the four source files reverted 4 of 6 fail as methods (the self-recursive-lookup guard, the named-pair guard, both behavioural lookups), exactly as predicted when the last-line check was narrowed. tests/vendor 92, tests/project 156, both green; isort clean.

Ready for you to merge — and it is the one that unblocks main, #865 and #868, all three of which are red for this single defect.

@JarryShaw
JarryShaw merged commit 3173c4f into main Sep 28, 2026
63 checks passed
@JarryShaw
JarryShaw deleted the fix/866-crawler-missing-recursion branch September 28, 2026 00:11
@JarryShaw JarryShaw removed the review: good-to-go Cross-review at the current head says ready; CI state is separate label Sep 28, 2026
JarryShaw added a commit that referenced this pull request Sep 28, 2026
Both `get`'s `cls(default)` sites -- the `str` branch's and the value
branch's -- reached `_missing_` for a `default` that fell inside a
still-minting registry's own range, growing the registry as a side
effect of resolving `default` rather than `key`.

- Replace both `cls(default)` calls with a plain `_value2member_map_`
  lookup (owner's ruling, option 1): `default` can no longer mint by
  construction, and a `default` naming no registered member now falls
  through to the same lookup error `key` itself would have raised.
- `key` resolution is unchanged: `EtherType.get(0x0888)` still mints
  `Xyplex_0x0888` through `_missing_`, per the ruling's own exception --
  landing in both lookup tables, exactly as `register` would leave it,
  since that is #775's own deliberately-kept-minting exception, not
  something `get` itself does.
- Update the docstring's "It never mints" caveat (now qualified: true
  for `default`, not for `key` on the three still-minting registries),
  the now-false claim that an unregistered default still reaches
  `cls(default)`, and the Args/Raises text to match.
- Add `GetDefaultNoMintTests` pinning the issue's own repro, the
  preserved key-path mint, default resolving to a registered value
  (int and str), and the exception type per path. Update two
  `NoDefaultSentinelTests` assertions that asserted the old
  `cls(default)` failure mode for an unregistered `-1`/`-1.0` default --
  their docstrings now say plainly that they coincidentally pass on the
  pre-#857 tree too, and point at the test that still discriminates it.
- Update four assertions in `test_const_enum_get.py` and
  `test_const_enum_builtin_parity.py` that encoded the superseded
  contract (an unresolvable/unregistered default failing with its own
  named error, or resolving into a declared-but-unassigned range) --
  each retargeted to the new contract rather than weakened. The
  FilterType test's three dropped resolve/identity assertions are
  restored through its unaffected `key` path and the method renamed to
  match; it and its comment cross-reference are updated together.

Owner's rulings, verbatim:
- "Take (b). Only register can mint. get should not mint unless it
  falls through the _missing_'s minted ranges."
- Choosing option 1 of three proposed: "I think 1 is correct mechanism
  we'd like."

Build/tests, under plain unittest, methods not subTest records:
test_const_registry_protocol.py 74/74, test_const_enum_get.py 8/8,
test_const_enum_builtin_parity.py 33/33, tests/vendor/ 86/86 -- no
regressions. tests/const/ as a whole carries 6 records: 3 methods
inherited from #866 (fixed in #867) plus 1 further out-of-scope failure
in test_const_enum_no_mint.py (contended with #865, handed over rather
than edited here).
JarryShaw added a commit that referenced this pull request Sep 28, 2026
…ertion

tests/const/test_const_enum_no_mint.py's
GetNoLongerMintsTests.test_unresolvable_string_key_with_default_in_unassigned_range
was contended with #865 (both touched pcapkit/corekit/enum.py-adjacent
const/vendor territory) and deferred to this follow-up commit; #865 has
now merged, so it lands here rather than being applied separately.

Hardware.get('Definitely-Not-A-Member', 40) used to resolve 40 -- a
value in Hardware's declared-but-unassigned 39-255 range, not a
registered member -- via cls(default) -> _missing_ to an unregistered
pseudo-member. #864 restricts default to a plain _value2member_map_
lookup, so it no longer resolves: the original KeyError for the key
propagates instead, exactly the accepted cost the owner's ruling names.

Re-measured fresh on the current tree (after #866/#867 and #862/#865
both landed): Hardware.get('Definitely-Not-A-Member', 40) raises
KeyError: 'Definitely-Not-A-Member', members unchanged at 42 before and
after. Checked for overlap with #865's own additions to this file
(ETHERTYPE_UNASSIGNED_PROBES, EtherTypeMixedMintTests, the retired
test_masked_old_xerox_row_converts_by_source) -- none: those are all
EtherType key-path minting-order probes, unrelated to Hardware's
default-path resolution this test covers. GetNoLongerMintsTests remains
the right class.
JarryShaw added a commit that referenced this pull request Sep 28, 2026
* fix(corekit): stop EnumRegistry.get's default from minting (#864)

Both `get`'s `cls(default)` sites -- the `str` branch's and the value
branch's -- reached `_missing_` for a `default` that fell inside a
still-minting registry's own range, growing the registry as a side
effect of resolving `default` rather than `key`.

- Replace both `cls(default)` calls with a plain `_value2member_map_`
  lookup (owner's ruling, option 1): `default` can no longer mint by
  construction, and a `default` naming no registered member now falls
  through to the same lookup error `key` itself would have raised.
- `key` resolution is unchanged: `EtherType.get(0x0888)` still mints
  `Xyplex_0x0888` through `_missing_`, per the ruling's own exception --
  landing in both lookup tables, exactly as `register` would leave it,
  since that is #775's own deliberately-kept-minting exception, not
  something `get` itself does.
- Update the docstring's "It never mints" caveat (now qualified: true
  for `default`, not for `key` on the three still-minting registries),
  the now-false claim that an unregistered default still reaches
  `cls(default)`, and the Args/Raises text to match.
- Add `GetDefaultNoMintTests` pinning the issue's own repro, the
  preserved key-path mint, default resolving to a registered value
  (int and str), and the exception type per path. Update two
  `NoDefaultSentinelTests` assertions that asserted the old
  `cls(default)` failure mode for an unregistered `-1`/`-1.0` default --
  their docstrings now say plainly that they coincidentally pass on the
  pre-#857 tree too, and point at the test that still discriminates it.
- Update four assertions in `test_const_enum_get.py` and
  `test_const_enum_builtin_parity.py` that encoded the superseded
  contract (an unresolvable/unregistered default failing with its own
  named error, or resolving into a declared-but-unassigned range) --
  each retargeted to the new contract rather than weakened. The
  FilterType test's three dropped resolve/identity assertions are
  restored through its unaffected `key` path and the method renamed to
  match; it and its comment cross-reference are updated together.

Owner's rulings, verbatim:
- "Take (b). Only register can mint. get should not mint unless it
  falls through the _missing_'s minted ranges."
- Choosing option 1 of three proposed: "I think 1 is correct mechanism
  we'd like."

Build/tests, under plain unittest, methods not subTest records:
test_const_registry_protocol.py 74/74, test_const_enum_get.py 8/8,
test_const_enum_builtin_parity.py 33/33, tests/vendor/ 86/86 -- no
regressions. tests/const/ as a whole carries 6 records: 3 methods
inherited from #866 (fixed in #867) plus 1 further out-of-scope failure
in test_const_enum_no_mint.py (contended with #865, handed over rather
than edited here).

* test(corekit): retarget the last #864 default-in-unassigned-range assertion

tests/const/test_const_enum_no_mint.py's
GetNoLongerMintsTests.test_unresolvable_string_key_with_default_in_unassigned_range
was contended with #865 (both touched pcapkit/corekit/enum.py-adjacent
const/vendor territory) and deferred to this follow-up commit; #865 has
now merged, so it lands here rather than being applied separately.

Hardware.get('Definitely-Not-A-Member', 40) used to resolve 40 -- a
value in Hardware's declared-but-unassigned 39-255 range, not a
registered member -- via cls(default) -> _missing_ to an unregistered
pseudo-member. #864 restricts default to a plain _value2member_map_
lookup, so it no longer resolves: the original KeyError for the key
propagates instead, exactly the accepted cost the owner's ruling names.

Re-measured fresh on the current tree (after #866/#867 and #862/#865
both landed): Hardware.get('Definitely-Not-A-Member', 40) raises
KeyError: 'Definitely-Not-A-Member', members unchanged at 42 before and
after. Checked for overlap with #865's own additions to this file
(ETHERTYPE_UNASSIGNED_PROBES, EtherTypeMixedMintTests, the retired
test_masked_old_xerox_row_converts_by_source) -- none: those are all
EtherType key-path minting-order probes, unrelated to Hardware's
default-path resolution this test covers. GetNoLongerMintsTests remains
the right class.
JarryShaw added a commit that referenced this pull request Sep 28, 2026
JarryShaw added a commit that referenced this pull request Sep 28, 2026
JarryShaw added a commit that referenced this pull request Sep 28, 2026
@JarryShaw JarryShaw added this to the 1.5 milestone Oct 6, 2026
@JarryShaw JarryShaw moved this to Done in PyPCAPKit Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

const Regenerated IANA or vendor constant tables; members keep their numeric values fix Pull requests that fix a defect (fix: subject prefix)

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

main is red: two pcapng crawlers emit a self-recursive _missing_ body, causing RecursionError

1 participant