feat(corekit,schema,utilities)!: enforce @final at runtime on Info and Schema (#778) - #788
Conversation
|
Labelled The ask, in the PR's own words: " Two independent halves are in this diff and only the second is in question:
One line either way is enough — "keep it raising" or "back to a warning". If the latter, the change is local: restore the Cross-review dispatched on a different model from the author; CI is still running (5 of 29 legs in at the time of writing). I will report both verdicts together and will not call this ready on CI alone. |
If it's something like, then it should only warns. But if it's something like then B should raise. |
|
Ruling recorded, in your words:
Measured against this PR's head Case 2 raises from So your ruling reduces to one change: re-decoration of the same class goes back to a warning. Two things fall out of it for free — Removing |
f85cc75 to
d471e86
Compare
|
Revised to The warned class still constructs correctly — that matters, because handing back a class without its generated The PR got smaller, not larger. Three test files are now byte-identical to One thing the downgrade genuinely makes worse, which the worker found rather than me. With a raise, keying the re-decoration check on Against- Unit tier unchanged at 1806 passed / 31 skipped / 0 failed; coverage 100%/99%/100%; mypy 3 pre-existing; pylint only the 2 pre-existing The earlier GOOD TO GO was at |
|
Delta cross-review at The mutation test I asked for is the part that matters, and it fired. It changed So Re-derived independently, both published numbers: Both runners agree at 7, so the corrected count in the body is right and It also confirmed live, not by reading: triple decoration with several real fields still warns and stays usable ( Two honest caveats it raised rather than smoothing over. Its R0801 count was 7 against the claimed 2, which it attributes to scan scope — duplicate-code detection is corpus-size dependent, and the #783 review hit the same nondeterminism. And a tree-proof trap worth recording: a bare CI is fully green at 27✅ / 3⏭ / 0❌ on this head. Setting |
| # back a class with no generated ``__init__`` -- silently, since this path | ||
| # only warns. Only the class the decorator was actually applied to carries | ||
| # the marker in its own namespace, and that is the class this is about. | ||
| if cls.__dict__.get('__final__'): |
There was a problem hiding this comment.
but what if the class if @final'ed but not @info_final'ed. we still need @info_final to finalize and generate the functions in the class. and my thoughts were @info_final implies @final but @final must not be used without @info_final.
e.g.,
@info_final
class A:... # all good, finalized and non-inheritable
@info_final
@final
class B:... # all good, finalized and non-inheritable
@final
@info_final
class C: ... # all good, finalized and non-inheritable
@info_final
@info_final
class D: ... # good but warn of finalized class
@final
class E: ... # raise, this is not finalized and not acceptable
same ruling applies to Schema's @schema_final.
There was a problem hiding this comment.
You are right, and case B is broken today — not merely unguarded. Measured on d471e863a, all five of your cases, with the generated __init__ exercised rather than just the marker checked:
A @info_final warns=[] constructs -> {'x': 1}
B @info_final @final warns=['InfoWarning'] CONSTRUCT FAILS: TypeError: 'int' object is not iterable
C @final @info_final warns=[] constructs -> {'x': 1}
D @info_final @info_final warns=['InfoWarning'] constructs -> {'x': 1}
E @final only warns=[] CONSTRUCT FAILS: TypeError: 'int' object is not iterable
B fails because decorators apply bottom-up. final(B) runs first and puts __final__ in B's own __dict__; info_final then sees it, warns "B: info class has been finalised; now skipping", and skips the generation. So B comes back marked final and with no generated __init__, and construction falls through to Info.__update__, which wants a mapping. C works only because the order puts info_final first.
E is worse — it is silent. No warning at all, and the class is equally unusable.
The root cause is exactly what you put your finger on: the re-decoration check keys on __final__, which conflates two different facts — "already finalised by info_final" and "merely marked by typing.final". They need distinguishing, and the marker for the first already exists in the tree (FinalisedState / __finalised__), which is what the check used before it was moved onto __final__.
So the ruling I will implement, in your terms:
| behaviour | |
|---|---|
@info_final |
finalise, silent |
@info_final @final / @final @info_final |
finalise, silent — order must not matter |
@info_final twice |
finalise once, warn |
@final alone on an Info subclass |
raise — marked final but never finalised |
That last one needs a hook that fires without info_final being called, so it lands in __init_subclass__/the metaclass rather than in the decorator. Same for Schema/schema_final.
Flipping #788 to review: needs-changes — I had set good-to-go on it, and this case was missed by me and by the cross-review alike: both of us exercised @info_final twice and bare subclassing, neither tried @info_final stacked on a bare @final. Routing it now.
d471e86 to
0d197c8
Compare
|
Revised to So case B was not a pre-existing defect — it was a regression this PR introduced at One correction to my own measurement: my Where the case-E guard went, and the cost. It raises before the auto-finalisation so Three limits it named in the ruling as stated, which are yours to weigh rather than mine:
Tests 13 → 21, with B and C asserted to agree with each other rather than separately, since order-independence is the property that broke. Against unmodified Cross-review dispatched on a different model at this head. Label stays |
0d197c8 to
9441619
Compare
|
Revised to The chosen fix for (1) is For (2), Flipping |
|
Correcting the cross-review, not the PR: its The review argued 100% is structurally impossible on one interpreter, since So "100%, unchanged from main" in the PR body is accurate and reproducible. The author recorded the disagreement plainly instead of deferring to the review — which is the right behaviour, and it is why this PR still says the true thing. Two of my own measurement errors while settling it, worth recording since the pattern keeps recurring: The other two numbers were handled correctly. The One thing the author could not verify and flagged rather than glossed: |
|
Merged So the overlap is benign: #795 edits the journal-export reader near line 1900 and #788 edits Worth noting one thing the merge surfaced that is not a defect: My own error while establishing this, recorded because it nearly produced a false all-clear: the first merge attempt printed |
|
Delta re-check verdict: NEEDS CHANGES — one narrow item, and it is a new defect this revision introduced rather than a leftover. Everything else from round 1 is closed. I reproduced the blocking finding myself on Mechanism: because Required: dedup the Why it is blocking despite a narrow blast radius: it is unbounded, it is on documented public API ( Everything else closed. Both round-1 blockers re-derived independently at the new head, not taken on trust. The attack battery is identical to round 1 — no new bypass, no new false positive ( Mutation D is properly closed — it survived round 1 and now fails ( One secondary item, your call. The reviewer disputes the replacement text in Flipping to |
|
Supplement: the Mutation: replace Four distinct methods catch it, across both families. The three-attempt failures on the My counts differ from the reviewer's ( Verdict unchanged: One methodological note the reviewer surfaced and I am recording because it is the inverse of the trap we keep hitting: its |
9441619 to
3edd196
Compare
|
Revised to Why it got worse: So the blocking item is closed and a different one is now visible. The fix that addresses the cause rather than the symptom is the short-circuit that was considered and rejected. The stated reason for rejecting it was that it "needs no new marker" and that the Please do that instead, and keep the two new bound tests — they are good and they will still pass, since a short-circuit bounds the length just as effectively. Add one asserting the per-call cost does not regress, or at minimum that Everything else in this revision is right. The two new tests are well built — prime once, snapshot, loop 50, assert length equals its own unique count — and were shown failing on the pre-fix code with Staying at One thing the author flagged honestly and I am carrying forward: CI's green run was against |
…d Schema (#778) `info_final` and `schema_final` both end `return final(cls)`, so every finalised class already carried `typing.final`'s `__final__` marker -- and nothing read it. `grep -rn '__final__' pcapkit/` found no hits at 110381b: the decorator was a promise to the type checker that the interpreter was free to ignore. Four shapes, four answers, `@schema_final`/`Schema` identically: - `@info_final` finalises silently. - `@info_final @final` and `@final @info_final` both finalise silently. Order cannot matter, so the re-entry check keys on `__finalised__` rather than on `__final__` -- only `__finalised__` records *the decorator* having run, and decorators apply bottom-up, so a `__final__` test reads a class marked by `final` an instant earlier as already finalised and skips the generation. - `@info_final` twice warns and hands back the finalised class, unchanged. - `@final` alone raises, at first construction: marked final, never finalised, so no generated `__init__` and nothing usable. `__init_subclass__` cannot catch it -- `final` is applied after class creation, so that hook has already returned -- and the check is nested inside the existing one-shot `FinalisedState.NONE` branch, so a finalised class pays nothing for it: `dis.dis(Info.__new__)` shows zero of the added bytecode executes on that path. - Deriving from a finalised class raises, from new `Info.__init_subclass__` and `Schema.__init_subclass__` hooks; neither existed, only `EnumSchema` had one. Every check reads the marker out of the class's own `__dict__`: both markers are ordinary class attributes and so inherit, and a class that merely descends from a finalised one has not been mismarked by anybody. That same inheritance had a second edge, caught by cross-review: `info_final`/`schema_final`'s own `_finalised=False` promotion wrote `BASE` onto `Info`/`Schema` themselves on a bare `Info()`/`Schema()` call, and every subclass declared afterwards inherited it -- silently defeating the bare-`@final` guard for the rest of the process. Fixed by skipping that promotion when `cls is Info`/`cls is Schema`, which on its own reopens the whole function to re-entry on every bare call rather than once -- closed with a second, own-`__dict__` marker, `__base_ready__`, that short-circuits the re-entry outright rather than making it idempotent. Two rounds of cross-review pinned this in turn: an unbounded `cls.__excluded__` first (a dedup'd `extend` fixed the growth but made the O(n) work O(n^2) and ran it on every call regardless), then the short-circuit above, both now pinned by tests -- one on the bound, one on the re-entry itself. `pcapkit.utilities.compat` takes `final` from `typing_extensions` below 3.11 rather than below 3.8, because `typing.final` only records `__final__` from 3.11 on -- the guards were otherwise a silent no-op on the 3.10 matrix leg. `EnumSchema.__init_subclass__` calls the base hook first, so a refused declaration cannot leave `__enum__` pointing at a discarded class; a second cross-review pass found `pcapkit/protocols/schema/misc/pcapng.py`'s `Option.__init_subclass__` did not follow this and reordered it to match -- otherwise a refused `Option` subclass still displaced a built-in schema first. Additive in tree: of `Info`'s 488 descendants 455 carry `__final__` and none is subclassed; of `Schema`'s 445, 408 do and none is subclassed; no class carries `__final__` without `FinalisedState.FINAL`, so the bare-`@final` guard cannot fire on the library either. `tests/test_final_enforcement.py` is the only test file this touches, plus the one-line `pcapng.py` reorder above. Unit tier (this tree, serial) 1819 passed / 15 skipped / 0 failed; mypy and pylint unchanged.
3edd196 to
84faf77
Compare
|
Revised to The The author caught its own probe artefact, and the mechanism is worth recording. Its first non-re-entry test snapshotted 31 passed, 10 subtests. Flipping Carried forward honestly by the author and worth your eye at merge time: mypy and pylint were not re-run against this sha (verified on |
|
Cross-review verdict at
Mutations A–F all caught, including the new F (remove the short-circuit → Audits unchanged — The Settling the So the Worth noting how long it took me to measure that correctly: my first two probes declared non-finalised subclasses, so One informational item worth a one-line docstring note rather than a change: hand-setting
Flipping to |
Please follow the guide below
You will be asked some questions, please read them carefully and answer honestly
Put an
xinto all the boxes [ ] relevant to your pull request (like that [x])Use Preview tab to see how your pull request will actually look like
Searched for similar pull requests
Followed the coding style (
make pylint,make mypy,make isort)make testpasses, and a test case covers the changeAdded a changelog entry under
docs/source/changelog/and regeneratedCHANGELOG.md, if the change is user-visible — N/A — changelog centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657What is the purpose of your pull request?
Tick the commit type your subject line carries.
fix— corrects a defectfeat— adds a featureperf— changes performance, not behaviourrefactor— changes neither behaviour nor performancetest— tests onlydocs— documentation onlyci— workflows or build toolingchore— anything elseDescription of your pull request and other information
Closes #778.
info_final/schema_finalboth endreturn final(cls), so every finalised class already carriedtyping.final's__final__— andgrep -rn '__final__' pcapkit/found nothing reading it. #778 is the ruling that it should be read.Four shapes, four answers (
@schema_final/Schema/SchemaError/SchemaWarningidentically):@info_final@info_final @finaland@final @info_final@info_finaltwiceInfoWarning), class returned usable@finalaloneInfoErrorat first constructionInfoErrorat declaration__finalised__, not__final__. The two record different facts and only__finalised__records the decorator having run. Decorators apply bottom-up, so@info_finalover@finalreaches the decorator with__final__already set by something that generated nothing — a__final__test there reads that as "already finalised", skips the generation, and returns a class with no__init__. That is what made row 2 order-dependent.@finalalone is caught at first construction, in__new__. It cannot be caught in__init_subclass__:finalis applied to the class object, after creation, so that hook has already returned. The check sits inside the existing one-shotFinalisedState.NONEbranch, so a finalised class never reaches it —dis.dis(Info.__new__)shows the branch compiles to onePOP_JUMP_IF_FALSEthat skips clean past it for aFINALclass, zero of the added bytecode executes, a property of the compiled branch rather than of any one timing run. It raises before the auto-finalisation, so the refusal repeats rather than firing once.__dict__. Both markers are ordinary class attributes and so inherit; a class that merely descends from a finalised one has not been mismarked by anybody. That inheritance had a second edge:info_final/schema_final's own_finalised=Falsepromotion wroteBASEontoInfo/Schemathemselves on a bareInfo()/Schema()call, which every later subclass then inherited — silently defeating the bare-@finalguard for the rest of the process. Fixed by skipping that promotion whencls is Info/cls is Schema, which on its own would re-enter the whole function on every bare call — closed with a second, own-__dict__marker (__base_ready__) that short-circuits the re-entry outright, so it costs nothing repeated and never touches__finalised__(two rounds of cross-review pinned this: an unbounded__excluded__first, then an O(n²) dedup papering over the re-entry instead of stopping it).compattakesfinalfromtyping_extensionsbelow 3.11, not 3.8 —typing.finalonly records__final__from 3.11 (gh-90500), so on the 3.10 matrix leg every guard here was a silent no-op.typing-extensionsis already a declared dependency forpython_version < '3.11'.EnumSchema.__init_subclass__calls the base hook first, so a refused declaration cannot leave__enum__pointing at a discarded class —pcapkit/protocols/schema/misc/pcapng.py'sOption.__init_subclass__did not follow this (it registered into__enum__before the base hook could raise) and is now reordered to match.Two honest notes. Row 2 was fine on
mainand was broken only by an earlier revision of this branch, which had moved the re-entry check onto__final__; the fix restoresmain's marker with an own-__dict__refinement. And the bare-@finalguard can only see a marker that was actually recorded, so on 3.10 it fires fortyping_extensions.finalbut not for a user'styping.final, which records nothing there. A@finalclass descending from aBASE-state ancestor also escapes the guard by inheritingBASEinstead ofNONE— documented as out of scope onInfo.__new__/Schema.__new__and pinned by a test, since closing it means re-finalising everyBASEdescendant on each subclassing.Additive:
Info488 descendants / 455 carry__final__/ 0 subclassed;Schema445 / 408 / 0; and no class in the tree carries__final__withoutFinalisedState.FINAL, so the bare-@finalguard cannot fire on the library either. Unit tier (serial, not-n auto) 1819 passed / 15 skipped / 0 failed in 28m07s; coverage ofinfoclass.pyandpcapng.py100%,schema.py99% (one pre-existing miss),compat.py100% viatests/utilities/test_compat.py's version-faking re-execution (unchanged frommain); mypy and pylint unchanged (identical findings, same files/kinds, before and after).tests/test_final_enforcement.pyis the only test file touched, plus a one-line reorder inpcapng.py— 31 tests, 12 shown failing againstmain.