diff --git a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md index f43cd71134..77490363fc 100644 --- a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md +++ b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md @@ -729,7 +729,7 @@ vocabulary key fails the smoke. | `relations.subsets` | superset vocabulary, excluded values, owners of the subset symbol | Owner symbols equal superset minus excluded | | `projections..mapping` | source value to target value or `null` | Keys equal the source vocabulary; mapped values match the owner function; `null` routes raise (I4) | | `schema_versions.` | constant name, value, owner modules | The only defining modules are the listed owners and all carry the value (I1) | -| `retirement_ledger..fields` | per-field Python and TypeScript module budgets | Actual module counts are at or below budget, and the field set and every budget match `RETIREMENT_ANCHOR` (I5) | +| `retirement_ledger..fields` | per-field Python and TypeScript budgets under two metrics: `*_module_budget` counts modules carrying the field token, `*_migration_surface` counts the modules that read, write or bind it | Actual counts are at or below both budgets; the field set and every token budget match `RETIREMENT_ANCHOR`, every surface budget matches `MIGRATION_SURFACE_ANCHOR`, and the five syntactic roles partition the token count exactly (I5) | | `dual_runtime_twins` | root and module budget | Tracked same-basename `.py`/`.ts` pair count is at or below budget; root and budget equal their code anchors (I5) | | `inventory_ratchets` | budgets for same-runtime fork names and definitions, conflicting names and definitions, schema-version forks, multi-value twins and forks, and the shared-vocabulary conflict and fork subsets | Inventory summary counts are at or below budget, and each budget equals its `BUDGET_ANCHOR` entry (I5, I9) | @@ -847,6 +847,9 @@ on the next full-tree scan; genuine shared-contract changes still need review. | Docs governance accepts the RFC pair | `python3 examples/docs-governance-smoke.py` | pass | Checks mirror, links, index | | Consumer roles are reported per site for vocabularies that declare a slot, with the unknown stated (B5, optional) | `uv run python scripts/generate_semantic_inventory.py --report --consumer-evidence` and `uv run --extra test python -m pytest tests/architecture/test_semantic_consumer_report.py` | the report names its coverage and the vocabularies it refused to analyse, then prints read/interpret/pass-through/unknown counts, both unknown shares, and every unknown reason with its site count; the tests pass | Advisory only, and optional in #4447: syntactic use over a two-root scan reach, never data flow and never a gate. Coverage is the 1 of 26 registered vocabularies that declares `literal_scan.field`; the other 25 are reported as `missing_slot_identity` and are not analysed | | Retirement budgets use standalone field tokens | `count_identifier_modules()` uses identifier boundaries for the six fields | `goal_boundary`: 30 Python modules under the new metric; the old substring metric was 35 | Conservative lexical measure; it removes compound-name false positives but does not prove semantic reader absence | +| The retirement metric separates readers from mentions (B3) | `check_reader_metric()` records every fact that holds of each module carrying one of the six field tokens -- reads, writes, binds, unresolved, mention only -- beside a single-label role partition | `goal_boundary`: 30 token modules resolve to 8 reading, 7 writing, 9 binding, 1 unresolved and 14 mention-only, so its migration surface is 16, not 30; `work_lane_contract` is 29 of 29 | Syntactic use, not data flow. The facts overlap, so they are asserted to *cover* the token population; the role partition is asserted to sum to it and is for ordering and printing only | +| A new reader of a legacy field fails the pull-request path | Add a module reading `payload["protocol_action_packet"]` beyond the budget | `check_reader_metric` fails naming the field and the count | Committed fixture in `tests/architecture/test_semantic_vocabulary_drift.py`; the anchor equality check is the same pattern as `RETIREMENT_ANCHOR` | +| A computed key stays unresolved rather than absent | Count Python mapping accessors whose first argument is not a literal, and TypeScript computed member accesses | 1711 Python sites and 400 TypeScript sites under `loopx/`; a field measured at zero readers is measured against that standing unknown | This is why zero readers cannot by itself authorize a removal (Q11). It is attributable to no field, so unlike a field-specific unknown it never enters a surface. Python subscripts with a computed key are excluded: `rows[index]` and `payload[key]` are the same syntax | | The module-local convention filter is a code edit | Widen `MODULE_LOCAL_CONVENTION` in `inventory.py` and scan | `*_semantic` budgets fall with no code change elsewhere | Known boundary; the regex is in code so the widening is a reviewed diff, and the unfiltered totals stay budgeted | | A registered value nobody produces fails (M0.5) | Run the production-form scan on the baseline | Fails naming `effective_action` and `skip`; passes after `skip` is removed or listed `compatibility_only` | First expected I12 failure; a compared-only value is not carried | | A producer of an unregistered value fails (M0.5) | Write `effective_action: "brand_new"` in a listed producer site | Fails naming the site and the value even though no consumer compares it | I13; production is stricter than comparison | @@ -963,7 +966,7 @@ with `npm ci --ignore-scripts` before running the TypeScript production scan. | M0.5b | `producers` and `compatibility_only` on `kernel` vocabularies; production-form scan with the two role checks (I12, I13); retirement budgets counted by identifier with all six anchors lowered in one diff (Q11); merge-order rule from Q9 written into Section 10 | M0.5a complete; Q9 decided or its interim rule accepted | Smoke green with I11 to I14 enforced; `skip` resolved; Section 9 producer rows green; `turn_route` persistence answered for Q2 | Remove producer fields and role checks; budgets return to the pre-M0.5b anchors | | M1 | `EffectiveAction` typed enum in one owner module; the replay observation and frontier slots split off (Q6); producers and consumers import it; registry `literal_scan` tightened to the enum | M0.5 merged; owner module chosen (Q3); slot split decided (Q6) | Smoke green; zero bare `effective_action` literals outside the owner; parity fixtures for status/should-run unchanged | Revert to literals; registry keeps the set | | M2 | Route-to-disposition projection, the `decide_loop_disposition` decision table, and the cross-runtime sets published through a shared contract with generated Python and TypeScript bindings, following the coordination contract generator | M1 merged; Q2 and Q7 decided | Generator `--check` and smoke green; `settlement.ts` and `transaction.py` read the generated set | Regenerate from prior contract | -| M3 | Per-field retirement of legacy should-run fields, one field per PR, budgets lowered to zero and the field removed | Field has zero external readers proven by producer/reader research | Schema-reduction record per `AGENTS.md`; Appendix B entry | Restore field from the last writer | +| M3 | Per-field retirement of legacy should-run fields, one field per PR, budgets lowered to zero and the field removed | Field's migration surface is emptied module by module, and the residual unresolved and dynamic-key evidence is reviewed; a zero count is not by itself the gate | Schema-reduction record per `AGENTS.md`; Appendix B entry | Restore field from the last writer | | M4 | Twin budget lowered with each replacement-first cutover from the migration RFC | Each cutover PR | Budget edit in the same diff | None needed; budget follows code | A ratchet without a target is a direction, not a plan. The table below is the @@ -980,7 +983,7 @@ vocabulary property the smoke can check. Rows marked *open* wait on a Section | Conflicting values, semantic | 2 names | `semantic-vocabulary-drift-smoke.py`: `conflicting_values_semantic` | 0 | baseline PRs | | Multi-value forks | 4 (1 misclassified) | `semantic-vocabulary-drift-smoke.py`: `multi_value_forks` and `multi_value_forks_semantic`. Only the count is printed today; #4614 adds `divergent_value_sets` to name the surviving forks | 0 after `scope` declares bounded-context names | M0.5 + baseline PRs | | Multi-value twins | 19 | `semantic-vocabulary-drift-smoke.py`: `multi_value_twins` | 0 | baseline PRs | -| Legacy should-run fields | 6 fields, 124 py / 10 ts module mentions | `semantic-vocabulary-drift-smoke.py`: one `.py` / `.ts` pair per field | 0 fields | M3, identifier-counted | +| Legacy should-run fields | 6 fields, 124 py / 10 ts module mentions | `semantic-vocabulary-drift-smoke.py`: one `.py` / `.ts` pair per field for the token count; one `retirement_role:` line per field and runtime under `--report` for the migration surface and its five roles | 0 fields | M3, gated on emptying the B3 migration surface; the token count stays budgeted until Q11 | | Merge-candidate groups | 32 unreviewed | `merge_candidate_groups()` in `loopx/semantics/inventory.py`; no command prints it today, and #4630 adds the CLI line. Read the reviewable count, not the raw one -- a registered cross-runtime vocabulary owns both its Python and TypeScript symbols, so those pairs are required by I3 rather than debt | every group classified; only `same_semantics` groups merged | classification PR, then per-group PRs | | Control-plane py/ts twins | 43 | `semantic-vocabulary-drift-smoke.py`: `independently_maintained` | follows the TypeScript migration RFC; no target here | M4 | @@ -1138,14 +1141,135 @@ introduce a competing target state. and generates their projection; it does not merge spellings. A future proposal to merge them must provide a dual-read/versioned migration and reader proof. Owner: Turn driver owner. Needed before M2 closes. -11. **Retirement budgets by identifier.** The six legacy-field budgets now use +11. **Retirement budgets by identifier.** The six legacy-field budgets use `count_identifier_modules()`, so `goal_boundary_repair` is not counted as `goal_boundary`. This is a conservative lexical metric, not proof of zero - semantic readers; computed accesses remain an evidence gap. Owner: kernel - maintainers. + semantic readers. B3 adds `check_reader_metric()` beside it, which records + every fact that holds of each module -- reads, writes, binds, unresolved, + mention only -- and budgets as the migration surface every module that is not + mention-only. That includes the field-specific unknowns: a module holding the + field name as data, or one the scan could not parse, is work someone must do + before this field can go, and its shape rather than its existence is what is + unknown. Both metrics are now checked. What stays open is whether the token + budget is retired once the surface budget has ordered a removal, and what + residual evidence a field at zero surface still owes given 1711 Python + computed-key sites and 400 TypeScript computed members, which belong to no + field and so can never be retired by emptying one. Owner: kernel maintainers. ## Appendix A: Execution ledger (non-normative) +### 2026-09-17 — B3: the retirement metric separates readers from mentions + +The six legacy should-run fields were budgeted by a token count: modules whose +text contains the standalone field name. That number answers "does this name +appear here", which is not the question a retirement asks. `check_reader_metric` +classifies the same modules by syntactic use and budgets every module that is +not a bare mention as the work owed before a field can be removed. The role +column below is a single label in a fixed precedence, for ordering and +printing; the counts a retirement reads are the overlapping facts beside it. + +| Field | Python token | reader | writer | binding | unresolved | mention | surface | TS token | surface | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `protocol_action_packet` | 5 | 1 | 4 | 0 | 0 | 0 | 5 | 2 | 2 | +| `external_evidence_observation` | 8 | 4 | 1 | 1 | 1 | 1 | 7 | 1 | 1 | +| `heartbeat_recommendation` | 17 | 8 | 4 | 1 | 0 | 4 | 13 | 1 | 1 | +| `execution_obligation` | 20 | 8 | 7 | 0 | 0 | 5 | 15 | 1 | 1 | +| `work_lane_contract` | 29 | 11 | 8 | 9 | 1 | 0 | 29 | 3 | 3 | +| `goal_boundary` | 30 | 8 | 4 | 3 | 1 | 14 | 16 | 2 | 1 | + +Three results the token count had hidden: + +- `goal_boundary` and `work_lane_contract` were within one module of each other + at 30 and 29, so the plan ordered them as equally expensive. Their real + surfaces are 16 and 29. Fourteen of `goal_boundary`'s modules are prompt prose + and module-path imports that no migration touches; `work_lane_contract` has no + such module at all. +- `protocol_action_packet` has one Python reader and four writers. It is the + cheapest first M3 removal, and the token count did not say so. +- 1711 mapping accessors under `loopx/` take a computed key, and 400 TypeScript + computed member accesses are that runtime's equivalent. No name-keyed scan, + lexical or syntactic, can attribute them to a field, so neither is part of any + field's surface and the smoke prints both numbers beside the per-field counts. + This is the measured form of "a zero count does not authorize a deletion"; + the residual obligation is Q11's. + +Five corrections were measured after the first implementation, all of which had +made the surface smaller than the work: + +- The TypeScript scan recognized a member access and nothing else, so the + constructs TypeScript actually uses were classified as prose: a destructuring + read, an object-literal write, and a declared property signature. Of twelve + equivalent accesses written for both runtimes, eight disagreed. `{field: x}` + was a writer in Python and a mention in TypeScript, so porting a dict literal + across the boundary shrank the surface with nothing migrated, and every one of + the six fields measured zero TypeScript writers. `execution_obligation.ts` read + as 0 modules to migrate while `turn_envelope.ts` declared its truncation + limits. Both runtimes are now asserted to classify the same access the same + way, which is the property that makes the surface safe to plan against. +- The TypeScript half of the standing unknown was drawn only from modules that + spelled a field, 4 of 145. Widening it to every tracked module raised the + count from 82 sites to 395 and cost 0.52s. +- The summary counted role labels. The label is single-valued, so a module that + both read and wrote the field was counted only as a reader and read/write + overlap could not appear at all. Each of the five facts -- reads, writes, + binds, unresolved, mention only -- is now counted separately and a module is + in every set it belongs to. The sets overlap, so the smoke asserts they + *cover* the token population rather than sum to it, and the partition keeps + its own sum-to-token assertion beside them. The overlap is large: + `heartbeat_recommendation` has 8 modules reading and 8 writing within a + surface of 13, which the labels reported as 8 readers and 4 writers. +- A module holding the field name as data was reported `unresolved` and then + excluded from the migration surface, on the reasoning that it was not *known* + to need migration. It is known to need investigation: nobody can say the field + is absent from that module without opening it, and that is field-specific work + either way. Excluding it also let the surface fall when a reader was rewritten + into a form the scan could not resolve. Field-specific unknowns now count; + `external_evidence_observation` 6 → 7, `goal_boundary` 15 → 16 and + `work_lane_contract` 28 → 29 follow, each one module. The repository-wide + computed-key totals stay outside every surface, because they belong to no + field and emptying one can never retire them. +- An unparseable tracked Python module was credited as a mention. It may hold + readers, so calling it prose shrank the surface on the strength of a parse + failure. It is now that field's unknown and stays in the surface. Raising + instead was considered and rejected: it fails the scan for a direct caller + working on a half-written tree, and unknown is the honest classification + rather than a louder one. + +The roles are asserted to partition the token count exactly, per field and per +runtime, on every run. The new metric therefore reclassifies one population +rather than measuring a smaller one, and this slice repays no debt: both +budgets are pinned at their measured values in the same diff. + +The partition assigns each module its first matching role, which answers "what +is this module mainly" and not "who writes this field". `work_lane_contract` +has 8 modules whose role is `writer` and 13 that write it; the other 5 also +read it and the partition calls them readers. A retirement looking for every +producer reads the overlapping `reads`/`writes`/`binds` counts printed beside +the partition, not the partition itself. + +The check costs 8.5s of a 36.9s guard, the median of five runs that time the +function inside the guard rather than subtracting two whole-guard runs; the five +spanned 8.4-9.2s of 35.8-39.8s, or 21-25% of the guard. Subtraction was tried +first and abandoned: on a shared machine it put the same cost anywhere +between 4s and 32s, because it differences two numbers that both move with +whatever else is running. The scan +must walk every tracked Python module, because the computed-key total is +repository-wide and a module that never names a field still contributes to it. +`parse_python` was factored out of `python_facts` so both scans raise the same +error on an unparseable source, and is deliberately left uncached: holding +roughly two million AST nodes for the rest of the run measured 0.7s worse +overall than parsing twice, and it slowed `check_inventory` from 2.4s to 5.4s, +which is the pass #4628 had just made cheaper. + +TypeScript reuses `scripts/semantic_production_scan.mjs` and its TypeScript +parser in one batch. AST property and literal-subscript accesses distinguish +reads, writes and compound updates, including optional access and template +interpolation; comments, quoted examples and regex literals cannot become +accesses or mask later code. Bare object/type keys remain mentions: syntax +alone does not establish that the object carries the retired payload field. +Computed-key data flow and external consumers remain outside this metric; +M3 must inspect those paths before removal. The token budgets are unchanged. + ### 2026-09-18 — Three producer-scan answers that were confidently wrong Normative for what the producer scan may report as complete. No budget, floor or @@ -1474,7 +1598,6 @@ produces it. matches the code after the code moves. For the `cross_runtime` tier there is no producer scan to check it against, which is the same gap F1/F2's domain bounds already disclose. - ### 2026-09-17 — Formula, role and enforcement claims separated; formal signature mutated Normative for the enforcement-lane wording; the checks are unchanged except for @@ -1540,7 +1663,6 @@ result on the current tree; what changes is what the invariants claim. consistent false claim, because moving an invariant's `enforcement` and its policy lane together stays self-consistent. That grounding gap is separate. - ### 2026-09-16 — B2 pilot: one re-export hop bound in the Python producer scanner - **Trigger:** after M2 moved the three Turn owners into @@ -1569,7 +1691,6 @@ result on the current tree; what changes is what the invariants claim. - Clarify candidate dispositions as advisory metadata; no per-candidate runtime store or enforcement is delivered by this schema. - ### 2026-09-15 — M0 opened with the RFC - **Baseline:** `1dc6ad8d8` @@ -1722,6 +1843,8 @@ result on the current tree; what changes is what the invariants claim. | --- | --- | --- | --- | --- | | 2026-09-16 | Q9: compute the full inventory on demand; retire the committed census | Implementation for [maintainer feedback](https://github.com/huangruiteng/loopx/pull/4360#issuecomment-5692062394); PR review pending | Committed snapshot with post-merge regeneration; diff-only scan rejected | 1, I6, 3, 5, 9, 10, 12 | | 2026-09-16 | B2: bind one unrenamed re-export hop in the Python producer scanner | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2; PR review pending | Require every consumer to import the owner module (fragile; failed silently in M2); unbounded multi-hop resolution rejected | 5, Appendix A | +| 2026-09-17 | B3 repair: count all five orthogonal use facts rather than the role labels, and put field-specific unknowns inside the migration surface | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B3; raises `goal_boundary` 15 to 16, `work_lane_contract` 28 to 29 and `external_evidence_observation` 6 to 7, each being one module that carries the field name as data, with token budgets and carrier counts unmoved; per Section 5 **kernel-maintainer approval required, not yet given** | Keep counting the role labels and add an overlap column (rejected: the label is single-valued, so any count built from it under-reports whichever fact sorted second, and one more column would not change that); leave `unresolved` outside the surface as "not known to need migration" (rejected: it is known to need investigation, and a budget that excludes it falls when a reader is rewritten into a form the scan cannot resolve); raise on an unparseable module (rejected: it fails the scan for a direct caller working on a half-written tree, and unknown is the honest classification, not a louder one); rely on the cross-runtime equivalence test alone (rejected: it asserts agreement, and two runtimes that both read a destructuring as prose agree) | 11, Appendix A, Appendix B | +| 2026-09-17 | B3: budget the migration surface beside the token count; keep both until Q11 | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B3; PR review pending | Replacing the token budget outright (rejected: the token count is the anchor that proves the new roles partition the same population, and dropping it in the same diff that introduces them would make the smaller number unauditable); counting Python computed-key subscripts as unresolved reads (rejected: `rows[index]` and `payload[key]` are one syntax, and the unknown would stop carrying information; TypeScript has no mapping-accessor convention, so its computed member access is counted separately and stated as an upper bound) | 5, 9, 11, 12 | | 2026-09-16 | B1 rename invariance: add the name-keyed divergence advisory; state the limit it does not close | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B1; PR review pending | Keying the budget on value sets (rejected: `CONFIDENCE_LEVELS` and `EDGE_CASE_COMPLEXITIES` share `high/low/medium` with different meanings); a committed name ledger (rejected at M0: Q9 retired the committed census). The advisory lists surviving forks by name; it was first described as catching a one-sided rename, which measurement disproved, so both mirrors state the limit as it behaves | 9 | | 2026-09-17 | B2: bind same-module call results, ordered local rebinding and key-precise container writes; reclassify the TypeScript residue rather than shrink it | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2; PR review pending | Bind cross-module calls and object fields (rejected: a separate bounded form with its own blast radius, not this slice); count a field-named keyword as production (rejected: it makes the obligation tautological, Section 5); leave `typescript_dynamic` as one catch-all (rejected: eight sites shared one reason, so the residue was not actionable); bind the callee's parameters to the call-site arguments (rejected: the answer would depend on the caller and could not be memoised, and a wrong binding would invent evidence) | 5, 9, Appendix A | | 2026-09-17 | B5 (optional): report consumer roles per site only for vocabularies that declare `literal_scan.field`, and state the unknown share instead of classifying everything | Implementation, Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B5; PR review pending | Fall back to the vocabulary id when no slot is declared (rejected on review: it analysed 25 of 26 vocabularies against a field name nobody declared, and every row downstream inherited the guess; they are now named under `missing_slot_identity` and not analysed); register consumers in the registry (rejected: the tracking issue forbids blanket consumer registration, and a declared list is a claim rather than evidence); make the report a merge gate (rejected: F3 is the advisory lane, and a 90.2% unknown share cannot gate anything); report only the sites the grammar resolves (rejected: the tables would read as complete, so an unrecognized mention and a computed-key read are printed rows with reasons) | 9, Appendix A, Appendix B, Appendix C | @@ -1755,6 +1878,9 @@ result on the current tree; what changes is what the invariants claim. | E21 | F1/F2 were unconditional but verified over one tier | `3ca868193` | `check_producers`' skip predicate, and the producer scan roots, read from the tree | 6 of 26 vocabularies declare `producers`, exactly the `tier: kernel` ones; the 20 skipped are all `cross_runtime`; the scan reaches 432 of 1203 tracked `loopx/**/*.{py,ts}` files (35.9%), the uncovered bulk being capabilities 285, other control-plane 192, extensions 83 | Counts from the registry and the tracked tree; the reach denominator moves with any new module, so it is reported, not pinned | | E22 | Fifteen reported unresolved sites can never become evidence | `3ca868193` | smoke report `unresolved_producer_blockers` | 41 unresolved sites, of which `argument_name_only` 10 and `annotation_only` 5 are a field-named keyword argument and a bare declaration; the other 26 are dynamic or interprocedural | Label-keyed; the two labels are code-owned in the scanner, so the floor moves only by a code edit | | E23 | F4 as written could not be violated | `3ca868193` | read `check_scope_declarations` against the F4 statement | Scope is declared and never inferred, so `conflict := collision ∧ scope_overlap` is a definition; what is enforced is that a declaration names every defining module exactly once, over 1 declaration and 4 contexts | Judgement from reading the check; value-set disjointness across contexts is deliberately *not* the property, because `SOURCE_SURFACES` legitimately reuses one name in four contexts (E19) | +| E24 | The retirement budget counted mentions as readers | B3 integration tree | `check_reader_metric()` over the six legacy fields; the five facts asserted to cover `count_identifier_modules()` and the role labels to partition it; both runtimes asserted to classify one access identically | 109 py token modules resolve to 85 surface modules; `goal_boundary` 30 → 16, `work_lane_contract` 29 → 29, `protocol_action_packet` 5 → 5 with one module reading it; 8 of 12 equivalent accesses had disagreed across the runtimes, and all six fields had measured zero TypeScript writers | Syntactic use, not data flow; 1711 Python computed-key accessors and 400 TypeScript computed members stay unattributable and belong to no field, so zero surface is not zero readers; the role partition is not a producer count | +| E27 | Agreement between runtimes does not by itself keep an access in the surface | B3 repair tree | Seven spellings of one TypeScript access, each scanned alone and each required to land in a named class that is inside the migration surface: dotted read, subscript read, shorthand and aliased destructuring, object-literal key, type property, name carried to a subscript | All seven are in the surface; none is a mention. The cross-runtime equivalence test alone would have passed with both runtimes reading a destructuring as prose, which is the shape the first implementation actually had | A counterexample set, not a proof of completeness: an eighth spelling nobody wrote down is still unmeasured, which is why an unprovable use lands in `unresolved` rather than in a confident default | + | E26 | B5 consumer evidence covers 1 of 26 registered vocabularies | `d8e7af141` | `scripts/generate_semantic_inventory.py --report --consumer-evidence`, cross-read against the drift smoke's `literal_scan_fields` coverage | 1 vocabulary declares `literal_scan.field` (`effective_action`) and is analysed; 25 are reported `missing_slot_identity` and are not analysed. 713 rows over 485 of 1213 tracked sources: 0 `read`, 61 `interpret`, 9 `pass_through`, 643 `unknown` (90.2%); of the 111 rows attributed to `effective_action`, 41 are unknown (36.9%). The removed vocabulary-id fallback had reported 921 rows at a 78.9% unknown share | Coverage is a registry property, not a code one: it moves only when a vocabulary declares a slot. Rows are syntactic use over a two-root reach, never data flow, and the scan is advisory — it refuses to run without `--report` | | E13 | The conflict budget mostly measured local naming | `1dc6ad8d8` | `MODULE_LOCAL_CONVENTION` applied to `conflicting_values` and `same_runtime_forks` names | 16 of 18 conflicts and 7 of 25 forks are module-local conventions; the semantic subsets are 2 and 18 | Classification is a name pattern, documented in the scanner and pinned by a fixture test | diff --git a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md index c700f55261..090c332e53 100644 --- a/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md +++ b/docs/architecture/rfcs/semantic-vocabulary-convergence-v0.zh-CN.md @@ -585,7 +585,7 @@ external_input | compatibility_only | unknown | `relations.subsets` | 超集词表、排除值、子集符号的 owner | owner 符号等于超集减排除值 | | `projections..mapping` | 源值到目标值或 `null` | 键等于源词表；映射值与 owner 函数一致；`null` 路由抛出（I4） | | `schema_versions.` | 常量名、值、owner 模块 | 唯一的定义模块就是列出的 owner 且都携带该值（I1） | -| `retirement_ledger..fields` | 每字段的 Python 与 TypeScript 模块预算 | 实际模块数不超过预算，且字段集合与每个预算与 `RETIREMENT_ANCHOR` 一致（I5） | +| `retirement_ledger..fields` | 每字段在两种指标下的 Python 与 TypeScript 预算：`*_module_budget` 统计携带该字段 token 的模块，`*_migration_surface` 统计真正读、写或以形参/局部名承载它的模块 | 两个实测值都不超过各自预算；字段集合与每个 token 预算与 `RETIREMENT_ANCHOR` 一致，每个迁移面预算与 `MIGRATION_SURFACE_ANCHOR` 一致，且五种句法角色恰好划分 token 计数（I5） | | `dual_runtime_twins` | 根目录与模块预算 | 同名 `.py`/`.ts` 对数不超过预算（I5） | | `inventory_ratchets` | 同运行时分叉的名字数与定义数、冲突的名字数与定义数、schema 版本分叉数、多值孪生与分叉数，以及共享词表冲突与分叉子集的预算 | 清单摘要计数不超过预算，且每个预算必须等于其 `BUDGET_ANCHOR` 条目（I5、I9） | @@ -682,6 +682,9 @@ owner 符号集合的组：`EffectiveAction` 与 `EFFECTIVE_ACTIONS` 是同一 | 文档治理接受这对 RFC | `python3 examples/docs-governance-smoke.py` | 通过 | 检查镜像、链接、索引 | | 对声明了槽位的词表按位点报告消费者角色，并写明未知量（B5，可选项） | `uv run python scripts/generate_semantic_inventory.py --report --consumer-evidence` 与 `uv run --extra test python -m pytest tests/architecture/test_semantic_consumer_report.py` | 报告先写明自己的覆盖面以及它拒绝分析的词表，再打印 read/interpret/pass-through/unknown 计数、两个未知占比，以及每种未知原因及其位点数；测试通过 | 仅为参考，且在 #4447 中属可选项：在两个根目录的扫描范围内度量语法使用，既不是数据流，也永远不是闸门。覆盖面是 26 个已注册词表中声明了 `literal_scan.field` 的那 1 个；其余 25 个作为 `missing_slot_identity` 上报，不做分析 | | 退休预算按子串而非标识符计数 | 分别以 `in file.text` 与 `\bgoal_boundary\b` 统计 `goal_boundary` | 基线上 35 对 30 个 Python 模块 | 已知边界；M3 的零读者门需要标识符计数，见第 12 节 | +| 退休指标把读者与提及分开（B3） | `check_reader_metric()` 为携带六个字段 token 的每个模块记录一切成立的事实——读、写、承载、未定、仅提及——并在旁边保留单标签的角色划分 | `goal_boundary`：30 个 token 模块解析为 8 读、7 写、9 承载、1 未定、14 仅提及，迁移面是 16 而非 30；`work_lane_contract` 是 29 中的 29 | 度量的是句法使用，不是数据流。事实相互重叠，因此断言它们**覆盖** token 样本；角色划分断言相加等于它，且只用于排序与打印 | +| 旧字段新增读者会在 PR 路径上失败 | 让一个模块读 `payload["protocol_action_packet"]` 从而超出预算 | `check_reader_metric` 失败并点名该字段与计数 | `tests/architecture/test_semantic_vocabulary_drift.py` 内的提交测试；锚点等值检查与 `RETIREMENT_ANCHOR` 同一套模式 | +| 计算式键保持「未定」而非「不存在」 | 统计首参数不是字面量的 Python mapping 访问器，以及 TypeScript 计算式成员访问 | `loopx/` 下 Python 1711 处、TypeScript 400 处；某字段读者计为零时，是对着这个公开的未知数计零 | 这正是零读者本身不能授权删除的原因（Q11）。它不可归属到任何单个字段，因此与字段专属的未知不同，永远不进入任何迁移面。Python 计算式下标不计入：`rows[index]` 与 `payload[key]` 是同一种语法 | | 模块局部约定过滤器是一次代码修改 | 扩宽 `inventory.py` 的 `MODULE_LOCAL_CONVENTION` 并重新生成 | `*_semantic` 预算下降而别处无代码改动 | 已知边界；正则在代码里，扩宽是可评审的 diff，未过滤总数仍在预算内 | | 无人生产的注册值失败（M0.5） | 在基线上运行生产形式扫描 | 失败并点名 `effective_action` 与 `skip`；删除 `skip` 或列入 `compatibility_only` 后通过 | 第一个预期的 I12 失败；只被比较的值不算已携带 | | 生产未注册值失败（M0.5） | 在某个已列生产位点写 `effective_action: "brand_new"` | 即使无消费者比较它也失败，并点名位点与值 | I13；生产比比较更严 | @@ -779,7 +782,7 @@ TypeScript effective-action 绑定与[术语表](../../reference/glossary.md)通 | M0.5b | `kernel` 词表的 `producers` 与 `compatibility_only`；带两条角色检查（I12、I13）的生产形式扫描；退休预算改按标识符计数并在一个 diff 里调整六个锚点（Q11）；Q9 的合并序规则写入第 10 节 | M0.5a 完成；Q9 已决或其临时规则被接受 | smoke 在 I11 到 I14 强制下全绿；`skip` 已处理；第 9 节生产者行全绿；为 Q2 回答 `turn_route` 是否持久化 | 删除生产者字段和角色检查；预算回到 M0.5b 前的锚点 | | M1 | 单一 owner 模块中的 `EffectiveAction` 类型化枚举；replay observation 与 frontier 槽位拆出（Q6）；生产者与消费者 import 它；注册表 `literal_scan` 收紧到枚举 | M0.5 合入；owner 模块已定（Q3）；槽位拆分已决（Q6） | smoke 绿；owner 之外零裸 `effective_action` 字面量；status/should-run 的 parity fixture 不变 | 回退为字面量；注册表保留集合 | | M2 | route 到 disposition 的投影、`decide_loop_disposition` 决策表与跨运行时集合通过共享契约发布，生成 Python 与 TypeScript 绑定，效仿协调契约生成器 | M1 合入；Q2 与 Q7 已决 | 生成器 `--check` 与 smoke 绿；`settlement.ts` 与 `transaction.py` 读取生成集合 | 从上一版契约重新生成 | -| M3 | 逐字段退休旧 should-run 字段，每个 PR 一个字段，预算降到零并删除字段 | 经生产者/读者调研证明该字段外部读者为零 | 按 `AGENTS.md` 的 schema 缩减记录；附录 B 条目 | 从最后一个写方恢复字段 | +| M3 | 逐字段退休旧 should-run 字段，每个 PR 一个字段，预算降到零并删除字段 | 逐模块清空该字段的迁移面，并评审残留的 unresolved 与计算式键证据；计数归零本身不构成这道门 | 按 `AGENTS.md` 的 schema 缩减记录；附录 B 条目 | 从最后一个写方恢复字段 | | M4 | 随迁移 RFC 的每次 replacement-first 切换调低孪生预算 | 每个切换 PR | 同 diff 中的预算修改 | 无需；预算跟随代码 | 没有目标的棘轮只是方向，不是计划。下表是本 RFC 完成时的状态；每一行都是一个 @@ -795,7 +798,7 @@ TypeScript effective-action 绑定与[术语表](../../reference/glossary.md)通 | 冲突值（语义） | 2 个名字 | `semantic-vocabulary-drift-smoke.py`：`conflicting_values_semantic` | 0 | 基线窄 PR | | 多值分叉 | 4（1 个误分类） | `semantic-vocabulary-drift-smoke.py`：`multi_value_forks` 与 `multi_value_forks_semantic`。今天只打印计数；#4614 增加 `divergent_value_sets` 以按名字列出存活的分叉 | `scope` 声明有界上下文名字后为 0 | M0.5 + 基线窄 PR | | 多值孪生 | 19 | `semantic-vocabulary-drift-smoke.py`：`multi_value_twins` | 0 | 基线窄 PR | -| 旧 should-run 字段 | 6 个字段，124 py / 10 ts 模块提及 | `semantic-vocabulary-drift-smoke.py`：每个字段一对 `<字段>.py` / `<字段>.ts` | 0 个字段 | M3，按标识符计数 | +| 旧 should-run 字段 | 6 个字段，124 py / 10 ts 模块提及 | token 计数：`semantic-vocabulary-drift-smoke.py` 每个字段一对 `<字段>.py` / `<字段>.ts`；迁移面与五种角色：`--report` 下每字段每运行时一行 `retirement_role:` | 0 个字段 | M3，以清空 B3 迁移面为门；token 计数在 Q11 决策前继续计入预算 | | 合并候选组 | 32 组未评审 | `loopx/semantics/inventory.py` 的 `merge_candidate_groups()`；今天没有任何命令打印它，#4630 增加该 CLI 行。读可评审数而非原始数——注册的跨运行时词表本就同时拥有 Python 与 TypeScript 两个符号，这类配对是 I3 的要求而不是债务 | 每组已分类；只合并 `same_semantics` 的组 | 分类表 PR，随后逐组 PR | | 控制面 py/ts 孪生 | 43 | `semantic-vocabulary-drift-smoke.py`：`independently_maintained` | 跟随 TypeScript 迁移 RFC；本 RFC 不设目标 | M4 | @@ -915,13 +918,108 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 Q2 建议保留两者。Q2 的实际写入及读回证据证明 `turn_route` 已持久化，因此 实现保留三套不同值集并生成投影，不合并拼法。未来合并提案须提供双读或带版本 的迁移及读者证据。Owner：Turn driver owner。 -11. **退休预算使用独立字段 token。** 六个旧字段预算现在使用 +11. **退休预算使用独立字段 token。** 六个旧字段预算使用 `count_identifier_modules()`，因此 `goal_boundary_repair` 不会被算作 - `goal_boundary`。这是保守的词法指标，不等于证明不存在语义读者；计算式访问 - 仍然是证据缺口。Owner：内核维护者。 + `goal_boundary`。这是保守的词法指标，不等于证明不存在语义读者。B3 在它旁边 + 加入 `check_reader_metric()`：为每个模块记录一切成立的事实——读、写、承载、 + 未定、仅提及——并把所有非「仅提及」的模块作为迁移面纳入预算。这包含字段专属 + 的未知：把字段名当数据持有的模块，以及扫描无法解析的模块，都是删除该字段前 + 必须有人处理的工作，未知的是工作形态而不是工作是否存在。两个指标现在都在 + 检查。仍然未决的是：当迁移面预算已经能排序删除工作后，是否退役 token 预算； + 以及在 1711 处 Python 计算式键访问与 400 处 TypeScript 计算式成员之下，迁移 + 面为零的字段还欠哪些残余证据——它们不属于任何字段，因此清空任何一个字段都 + 无法退役它们。Owner：内核维护者。 ## 附录 A：执行账本（非规范） +### 2026-09-17 — B3：退休指标把读者与提及分开 + +六个旧 should-run 字段此前按 token 计数计入预算：文本里出现该独立字段名的模块 +数。这个数字回答的是「这个名字在这里出现过吗」，而不是退休所问的问题。 +`check_reader_metric` 把同一批模块按句法使用分类，并把所有非「仅提及」的模块 +作为字段删除前欠下的工作纳入预算。下表的 role 列是固定优先级下的单一标签，用 +于排序与打印；退休时真正要读的计数是它旁边那些相互重叠的事实。 + +| 字段 | Python token | reader | writer | binding | unresolved | mention | 迁移面 | TS token | 迁移面 | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `protocol_action_packet` | 5 | 1 | 4 | 0 | 0 | 0 | 5 | 2 | 2 | +| `external_evidence_observation` | 8 | 4 | 1 | 1 | 1 | 1 | 7 | 1 | 1 | +| `heartbeat_recommendation` | 17 | 8 | 4 | 1 | 0 | 4 | 13 | 1 | 1 | +| `execution_obligation` | 20 | 8 | 7 | 0 | 0 | 5 | 15 | 1 | 1 | +| `work_lane_contract` | 29 | 11 | 8 | 9 | 1 | 0 | 29 | 3 | 3 | +| `goal_boundary` | 30 | 8 | 4 | 3 | 1 | 14 | 16 | 2 | 1 | + +token 计数掩盖掉的三个结果： + +- `goal_boundary` 与 `work_lane_contract` 是 30 与 29，只差一个模块，于是计划 + 把两者当作同等代价排序。它们真实的迁移面是 16 与 29。`goal_boundary` 有十四 + 个模块是提示词散文与模块路径导入，迁移根本不会碰到；`work_lane_contract` + 一个这样的模块都没有。 +- `protocol_action_packet` 只有一个 Python 读者和四个写方。它是代价最低的首个 + M3 删除对象，而 token 计数说不出这一点。 +- `loopx/` 下有 1711 处 mapping 访问器使用计算式键，另有 400 处 TypeScript + 计算式成员访问是该运行时的对应形态。任何按名字的扫描——词法的还是句法的—— + 都无法把它们归属到某个字段，因此两者都不进入任何字段的迁移面，smoke 把这两 + 个数字与各字段计数一起打印。这就是「计数归零不授权删除」的可测形式；残余 + 义务归 Q11。 + +首个实现落地后又实测出五处更正，它们都曾把迁移面做得比实际工作量小： + +- TypeScript 扫描只认成员访问，于是 TypeScript 真正使用的写法被归成了散文： + 解构读取、对象字面量写入、以及已声明的属性签名。为两个运行时各写一遍的十二 + 种等价访问中，有八种给出不同结论。`{field: x}` 在 Python 是 writer、在 + TypeScript 是 mention，因此把一个 dict 字面量移到边界另一侧就能让迁移面缩小 + 而没有迁移任何东西；六个字段在 TypeScript 侧全部实测为零个写入者。 + `execution_obligation.ts` 读作「0 个模块要迁移」，而 `turn_envelope.ts` 正在 + 那里声明它的截断上限。现在断言两个运行时对同一种访问给出同一个结论——这正是 + 让迁移面可以据以排期的性质。 +- 标准不确定的 TypeScript 那一半此前只取自拼出过字段名的模块，145 个里的 4 个。 + 扩到每个被跟踪模块后，计数从 82 处升到 395 处，代价 0.52s。 +- 汇总统计的是角色标签。该标签单值，于是一个既读又写该字段的模块只被算作读者， + 读写重叠根本无法出现。现在五项事实——读、写、承载、未定、仅提及——各自单独 + 统计，模块会计入一切对它成立的集合。集合相互重叠，因此 smoke 断言它们**覆盖** + token 样本而不是相加等于它，划分则在旁边保留自己「相加等于 token 计数」的断言。 + 重叠很大：`heartbeat_recommendation` 迁移面 13 个模块里有 8 个读、8 个写，而 + 标签把它报成 8 个读者、4 个写方。 +- 把字段名当数据持有的模块被报为 `unresolved`，然后以「并不**已知**需要迁移」 + 为由排除在迁移面之外。它是已知需要调查的：不打开那个模块，没人能说该字段不 + 在其中，而无论结论如何那都是该字段专属的工作。排除它还会让迁移面在有人把读者 + 改写成扫描无法解析的形式时下降。字段专属的未知现在计入；随之 + `external_evidence_observation` 6 → 7、`goal_boundary` 15 → 16、 + `work_lane_contract` 28 → 29，各为一个模块。全仓范围的计算式键总数仍在所有迁 + 移面之外：它们不属于任何字段，清空任一字段也永远退役不了它们。 +- 不可解析的被跟踪 Python 模块被记成了提及。它可能含有读者，把它记为散文等于凭 + 一次解析失败缩小迁移面。它现在是该字段的未知，位于迁移面之内。曾考虑改为抛错 + 并否决：那会让在半写状态树上工作的直接调用方整个扫描失败，而「未知」是诚实的 + 分类，不是更响的那一种。 + +每次运行都会按字段、按运行时断言这些角色恰好划分 token 计数。因此新指标是对同 +一批模块的重新分类，而不是换了一批更小的样本；本切片也不偿还任何债务：两个预算 +都在同一 diff 里钉在各自的实测值上。 + +这个划分给每个模块分配它首个命中的角色，回答的是「这个模块主要是什么」，不是 +「谁写这个字段」。`work_lane_contract` 有 8 个模块角色为 `writer`，而实际写它的 +有 13 个；另外 5 个同时也读它，于是划分把它们算作读者。退休时要找齐生产者，读 +的是打印在划分旁边的 `reads`／`writes`／`binds` 重叠计数，而不是这个划分本身。 + +这道检查占 36.9s 守卫中的 8.5s，是五次实测的中位数；这五次都在守卫内部对该函数计 +时，而不是相减两次整体守卫耗时，跨度为 35.8–39.8s 中的 8.4–9.2s，即守卫的 +21%–25%。相减法先试过并被放弃：在共享机器上它把同一份成本给到 4s 到 32s 之间， +因为它相减的两个数都会随机器上其他任务一起波动。扫描必须遍历每个被跟踪的 +Python 模块：计算式键总数是全仓范围的，一个从不提及任何字段的模块同样计入它。 +`parse_python` 从 `python_facts` 中析出，使两处扫描对不可解析的源抛出同一个错误； +它刻意不加缓存——把约两百万个 AST 节点留到运行结束，整体实测比解析两次还慢 +0.7s，并且会把 `check_inventory` 从 2.4s 拖到 5.4s，而那正是 #4628 刚刚变快的 +那一趟。 + +TypeScript 一次批量复用 `scripts/semantic_production_scan.mjs` 的 TypeScript +解析器，按 AST 属性和字面量下标区分读、写及复合更新，覆盖可选访问与模板插值。 +注释、引用示例与正则字面量不会变成访问，也不会遮住后续代码。对象字面量键按 +Python 的 `dict_literal_key` 同样计为写入，类型／接口属性签名计为承载，解构 +计为读取，未被任何键位消费的字段名字符串计为未定——每一条都与 Python 侧同构。 +计算式键的数据流与外部消费者仍在本指标之外，M3 删除前必须另行核查；token 预算 +保持不变。 + ### 2026-09-18 — 生产者扫描三处自信而错误的答案 对"扫描可以把什么报成完整"是规范性的。没有移动任何预算、下限或锚点： @@ -1161,7 +1259,6 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 - 本次未处理：备注是散文，没有任何机制检查其真伪。没有任何机制保证所述的产生条件曾经 成立，或在代码移动之后仍然成立。对 `cross_runtime` 层而言并不存在可供比对的 producer 扫描，这与 F1/F2 的值域边界已经披露的是同一个缺口。 - ### 2026-09-17 — 分离公式、角色与强制性声明；对形式签名做突变 强制层级的表述是规范性变更；除新增一条规则外，检查本身不变。#4447 Track B 的 B0 切片。 @@ -1216,7 +1313,6 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 - 本次未处理：`check_formal_model` 仍会接受一个自洽的错误声明，因为同时挪动某条 不变量的 `enforcement` 与它的 policy 层级仍然自洽。那个接地缺口是另一件事。 - ### 2026-09-16 — B2 试点：Python producer 扫描器绑定一跳再导出 - **触发：** M2 把三个 Turn owner 迁入 `turn_contract_generated.py` 后，仍经 @@ -1239,7 +1335,6 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 不把生产值预先定义为合法值。 - 明确候选处置为建议性元数据；本 schema 不交付逐候选运行时存储或执行门禁。 - ### 2026-09-15 — 随 RFC 开启 M0 - **基线：** `1dc6ad8d8` @@ -1362,6 +1457,8 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 | --- | --- | --- | --- | --- | | 2026-09-16 | Q9：全树按需计算；移除已提交结构清单 | 根据[维护者反馈](https://github.com/huangruiteng/loopx/pull/4360#issuecomment-5692062394)实现，PR 评审待完成 | 取代合并后补再生成；拒绝只扫描 diff | 1、I6、3、5、9、10、12 | | 2026-09-16 | B2：Python producer 扫描器绑定一跳未改名再导出 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2；PR 评审待完成 | 要求每个消费者都从 owner 模块导入（脆弱；M2 中已静默失效）；拒绝无界多跳解析 | 5、附录 A | +| 2026-09-17 | B3 修复：统计全部五项相互正交的使用事实而不是角色标签，并把字段专属的未知纳入迁移面 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B3；将 `goal_boundary` 由 15 上调至 16、`work_lane_contract` 由 28 上调至 29、`external_evidence_observation` 由 6 上调至 7，三处上调各自只涉及一个把字段名当数据携带的模块，token 预算与 carrier 计数均未变动；按第 5 节**需要内核维护者批准，尚未给出** | 继续统计角色标签并加一列重叠数（否决：该标签单值，任何由它构造的计数都会漏报排序靠后的那个事实，多加一列并不改变这一点）；让 `unresolved` 留在迁移面之外，理由是「并不已知需要迁移」（否决：它是已知需要调查的，而把它排除在外的预算会在有人把读者改写成扫描无法解析的形式时下降）；对不可解析模块直接抛错（否决：这会让在半写状态树上工作的直接调用方整个扫描失败，而「未知」是诚实的分类，不是更响的那一种）；只依赖跨运行时等价性测试（否决：它断言的是一致，而两个运行时都把解构读成散文也是一致的） | 11、附录 A、附录 B | +| 2026-09-17 | B3：在 token 计数旁边为迁移面设预算；Q11 决策前两者都保留 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B3；PR 评审待完成 | 直接用新指标取代 token 预算（否决：token 计数正是证明新角色划分同一批模块的锚，在引入角色的同一个 diff 里把它删掉会让更小的数字无法复核）；把 Python 计算式下标也计为未定读取（否决：`rows[index]` 与 `payload[key]` 是同一种语法，未知数会大到不再携带信息；TypeScript 没有 mapping 访问器约定，其计算式成员访问单独计数并声明为上界） | 5、9、11、12 | | 2026-09-16 | B1 改名不变性：新增按名字归组的分歧报告；写明它未闭合的边界 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B1；PR 评审待完成 | 把预算改按值集归组（否决：`CONFIDENCE_LEVELS` 与 `EDGE_CASE_COMPLEXITIES` 共享 `high/low/medium` 而含义不同）；提交名字账本（M0 否决：Q9 已退役提交式清单）。该报告列出仍然存在的分叉；初稿称它能抓住单侧改名，实测证否，故两份镜像按真实行为写明边界 | 9 | | 2026-09-17 | B2：绑定同模块调用结果、局部变量有序重绑定与按键精确的容器写入；对 TypeScript 残量做重新归类而非缩减 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B2；PR 评审待完成 | 绑定跨模块调用与对象字段（拒绝：那是另一条有自己影响面的有界形式，不属于本切片）；把与字段同名的关键字算作生产（拒绝：会使该义务变成同义反复，见第 5 节）；保留 `typescript_dynamic` 作为单一兜底（拒绝：八个位点共用一个原因，残量无法被行动）；把被调方形参绑定到调用点实参（拒绝：结果会依赖调用方而无法记忆化，且一次错误绑定会凭空造出证据） | 5、9、附录 A | | 2026-09-17 | B5（可选项）：只对声明了 `literal_scan.field` 的词表按位点报告消费者角色，并写明未知占比，而不是把一切都分类 | 实现，Refs [#4447](https://github.com/huangruiteng/loopx/issues/4447) B5；PR 评审待完成 | 没有声明槽位时回退到词表 id（评审中否决：这把 26 个词表中的 25 个按一个谁也没声明过的字段名去分析，下游每一行都继承了该猜测；它们现在列在 `missing_slot_identity` 下，不做分析）；在注册表中登记消费者（否决：跟踪 issue 明令禁止全量消费者登记，且一份声明清单是主张而非证据）；把该报告做成合并闸门（否决：F3 属于参考层级，90.2% 的未知占比也不足以闸住任何东西）；只报告语法能解析的位点（否决：这会让表格读起来像是完整的，因此未识别的提及与计算键读取都作为带原因的行打印出来） | 9、附录 A、附录 B、附录 C | @@ -1395,6 +1492,9 @@ PR review 保留这些层级。普通改动记录检查范围和理由，无共 | E21 | F1/F2 写成无条件，但只在一个层上被验证 | `3ca868193` | 从源码树读 `check_producers` 的跳过谓词与 producer 扫描根目录 | 26 个词表中 6 个声明了 `producers`，恰好是 `tier: kernel` 那几个；被跳过的 20 个全部是 `cross_runtime`；扫描触及 1203 个已跟踪 `loopx/**/*.{py,ts}` 中的 432 个（35.9%），未覆盖部分主要是 capabilities 285、其余控制面 192、extensions 83 | 计数来自注册表与已跟踪源码树；分母会随任何新模块移动，所以只上报、不钉住 | | E22 | 15 个被上报的未解析位点永远不可能成为证据 | `3ca868193` | smoke 报告的 `unresolved_producer_blockers` | 41 个未解析位点，其中 `argument_name_only` 10 个、`annotation_only` 5 个分别是以字段名命名的关键字参数和裸声明；其余 26 个是动态或跨过程的 | 按标签归组；这两个标签在扫描器里由代码持有，因此这个下界只能靠改代码移动 | | E23 | F4 写法本身不可能被违反 | `3ca868193` | 对照 F4 表述阅读 `check_scope_declarations` | 作用域是声明的、从不推断，所以 `conflict := collision ∧ scope_overlap` 是一条定义；真正被强制的是一份声明必须恰好枚举每个定义模块，范围是 1 份声明、4 个上下文 | 阅读检查后的判断；各上下文值集互斥故意*不*作为该性质，因为 `SOURCE_SURFACES` 正是合理地在四个上下文复用同一个名字（E19） | +| E24 | 退休预算把提及算成了读者 | B3 integration tree | 对六个旧字段运行 `check_reader_metric()`；断言五项事实覆盖 `count_identifier_modules()`、角色标签划分它；断言两个运行时对同一种访问给出同一结论 | 109 个 py token 模块解析为 85 个迁移面模块；`goal_boundary` 30 → 16，`work_lane_contract` 29 → 29，`protocol_action_packet` 5 → 5 且只有一个模块读它；十二种等价访问中曾有 8 种跨运行时结论不一致，六个字段在 TS 侧曾全部实测为零写入者 | 度量句法使用而非数据流；1711 处 Python 计算式键访问与 400 处 TypeScript 计算式成员仍无法归属且不属于任何字段，因此迁移面为零不等于读者为零；角色划分不是生产者计数 | +| E27 | 两个运行时结论一致，并不能凭此把某种访问留在迁移面里 | B3 repair tree | 把同一个 TypeScript 访问的七种写法各自单独扫描，每种都必须落到一个具名类别且位于迁移面之内：点号读、下标读、简写解构、别名解构、对象字面量键、类型属性、名字经局部变量进入下标 | 七种全部在迁移面内，没有一种是 mention。仅靠跨运行时一致性测试，在两个运行时都把解构读成散文时同样会通过——而这正是首个实现的实际形态 | 这是一组反例，不是完备性证明：没人写下来的第八种写法仍未被度量，这正是无法证明的使用要落到 `unresolved` 而不是落到一个自信默认值的原因 | + | E26 | B5 消费者证据覆盖 26 个已注册词表中的 1 个 | `d8e7af141` | `scripts/generate_semantic_inventory.py --report --consumer-evidence`，并与 drift smoke 的 `literal_scan_fields` 覆盖面互相印证 | 1 个词表声明了 `literal_scan.field`（`effective_action`）并被分析；25 个作为 `missing_slot_identity` 上报、不做分析。在 1213 个已跟踪源文件中的 485 个上共 713 行：`read` 0、`interpret` 61、`pass_through` 9、`unknown` 643（90.2%）；归属 `effective_action` 的 111 行中有 41 行未知（36.9%）。被删除的“回退到词表 id”实现当时报出 921 行、未知占比 78.9% | 覆盖面是注册表的性质而非代码的性质：只有当某个词表声明了槽位时它才会变化。各行是两个根目录范围内的语法使用，不是数据流；该扫描仅为参考——缺少 `--report` 时会拒绝运行 | | E13 | 冲突预算主要在度量局部命名 | `1dc6ad8d8` | 对 `conflicting_values` 与 `same_runtime_forks` 名字应用 `MODULE_LOCAL_CONVENTION` | 18 个冲突中 16 个、25 个分叉中 7 个是模块局部约定；语义子集分别为 2 与 18 | 分类是名字模式，已在扫描器中说明并由夹具测试钉住 | diff --git a/examples/semantic-vocabulary-drift-smoke.py b/examples/semantic-vocabulary-drift-smoke.py index c1f34306aa..77700d9d84 100755 --- a/examples/semantic-vocabulary-drift-smoke.py +++ b/examples/semantic-vocabulary-drift-smoke.py @@ -39,6 +39,9 @@ PRODUCER_FILES, PRODUCER_ROOTS, ) from loopx.semantics.python_production import scan_python_production # noqa: E402 +from loopx.semantics.field_use import ( # noqa: E402 + FACTS, ROLES, field_use_summary, lexical_module_count, render_field_uses, scan_field_uses, +) from scripts.generate_semantic_bindings import build_artifacts # noqa: E402 from loopx.canary.maintainability_ratchet import evaluate_maintainability_findings # noqa: E402 @@ -239,6 +242,23 @@ "goal_boundary": (30, 2), "protocol_action_packet": (5, 2), } +# B3 migration surface: modules that read, write or bind the legacy field, plus +# the modules holding its name unresolved, which have to be investigated before +# anyone can say the field is gone. Anchored like RETIREMENT_ANCHOR so the +# registry and this literal move in one diff. This does not replace the token +# budget above; Q11 owns that decision, and until it lands both are checked. +MIGRATION_SURFACE_ANCHOR = { + "execution_obligation": (15, 1), + "heartbeat_recommendation": (13, 1), + "work_lane_contract": (29, 3), + "external_evidence_observation": (7, 1), + "goal_boundary": (16, 1), + "protocol_action_packet": (5, 2), +} +RETIREMENT_FIELD_KEYS = { + "python_module_budget", "typescript_module_budget", + "python_migration_surface", "typescript_migration_surface", +} RATCHET_KEYS = ( "same_runtime_forks", "same_runtime_fork_definitions", @@ -809,6 +829,10 @@ def check_retirement_budgets(registry: dict[str, Any], sources: list[SourceFile] ledger = registry["retirement_ledger"]["should_run_legacy_decision_fields"]["fields"] require(set(ledger) == set(RETIREMENT_ANCHOR), f"retirement ledger fields are {sorted(ledger)}; the anchored set is {sorted(RETIREMENT_ANCHOR)}") for field, budgets in ledger.items(): + require( + set(budgets) == RETIREMENT_FIELD_KEYS, + f"retirement ledger {field} carries {sorted(budgets)}; expected {sorted(RETIREMENT_FIELD_KEYS)}", + ) for suffix, key, anchored in ( (".py", "python_module_budget", RETIREMENT_ANCHOR[field][0]), (".ts", "typescript_module_budget", RETIREMENT_ANCHOR[field][1]), @@ -830,15 +854,91 @@ def count_identifier_modules(field: str, suffix: str, sources: list[SourceFile]) This is intentionally a conservative lexical metric. It removes the known ``goal_boundary_repair`` false positive without claiming to prove that every remaining occurrence is a reader or that computed accesses are absent. + ``check_reader_metric`` splits this same population by syntactic role. """ - pattern = re.compile( - rf"(? tuple[list[str], list[str]]: + """Check the B3 syntactic metric against the ledger, and report its roles. + + Four obligations, all measured rather than assumed: + + * the five roles partition the token count exactly, so the new metric is a + reclassification of the same modules and not a different population that + happens to be smaller. The partition assigns each module its first + matching role, so it answers "what is this module mainly", not "who + writes this field": the fact counts beside it overlap on purpose and are + the ones a retirement reads to find every producer; + * the five orthogonal fact sets cover that same population. They overlap, + so they are not required to sum to it; what is required is that every + module carrying the token owes at least one fact and that no module is + invented. A module that both reads and writes is in both counts, which + the single role label hides; + * the migration surface stays within its anchored budget, so a new reader + of a legacy field fails the PR path that adds it; + * the unresolved populations stay visible, and the two kinds stay apart. A + module holding the field name unresolved, or one the scan could not + parse, is that field's own work and is inside its surface. + ``dynamic_mapping_key_sites`` and ``typescript_dynamic_member_sites`` are + not: they count computed-key accesses anywhere under ``loopx/``, belong + to no field, and while either is nonzero a field measured at zero readers + is not thereby proven dead. The smoke says so in its own output. + """ + ledger = registry["retirement_ledger"]["should_run_legacy_decision_fields"]["fields"] + summary = field_use_summary(ledger, sources) + report: list[str] = [] + detail: list[str] = [] + for field in sorted(ledger): + entry = summary["fields"][field] + for suffix, runtime, anchored in ( + (".py", "python", MIGRATION_SURFACE_ANCHOR[field][0]), + (".ts", "typescript", MIGRATION_SURFACE_ANCHOR[field][1]), + ): + classified = sum(entry[f"{runtime}_{role}_modules"] for role in ROLES) + carriers = entry[f"{runtime}_token_modules"] + require( + classified == carriers, + f"{field}{suffix}: roles classify {classified} modules but the token metric finds {carriers}; " + "the syntactic metric must reclassify the same modules, not a smaller population", + ) + covered = entry[f"{runtime}_classified_modules"] + facts = sum(entry[f"{runtime}_{label}_modules"] for _, label in FACTS) + require( + covered == carriers, + f"{field}{suffix}: the fact sets cover {covered} modules but the token metric finds " + f"{carriers}; every module carrying the token owes at least one fact", + ) + require( + facts >= covered, + f"{field}{suffix}: {facts} facts over {covered} modules; the fact sets overlap by " + "construction and can never total less than the population they cover", + ) + budget = ledger[field][f"{runtime}_migration_surface"] + actual = entry[f"{runtime}_migration_surface"] + require( + actual <= budget, + f"legacy field {field} now needs {actual} {suffix} modules migrated; budget is {budget}", + ) + require( + budget == anchored, + f"legacy field {field} {suffix} migration surface budget is {budget} but " + f"MIGRATION_SURFACE_ANCHOR pins {anchored}; the registry and the anchor move together in one diff", + ) + detail.append( + f"{field}{suffix} surface={actual}/{budget} " + + " ".join(f"{role}={entry[f'{runtime}_{role}_modules']}" for role in ROLES) + + " | " + " ".join( + f"{label}={entry[f'{runtime}_{label}_modules']}" for _, label in FACTS + ) + + f" carriers={carriers}" + ) + report.append( + f"dynamic_mapping_key_sites={summary['dynamic_mapping_key_sites']} " + f"typescript_dynamic_member_sites={summary['typescript_dynamic_member_sites']} " + "(computed keys, unattributable)" ) + return report, detail def check_dual_runtime_twins(registry: dict[str, Any]) -> str: @@ -930,11 +1030,13 @@ def main() -> int: check_projections(registry) check_schema_version_owners(registry, sources) budgets = check_retirement_budgets(registry, sources) + reader_metric, reader_detail = check_reader_metric(registry, sources) twins = check_dual_runtime_twins(registry) print("semantic-vocabulary-drift-smoke: ok") print(" " + coverage) print(" " + ratchets) print(" " + " ".join(budgets)) + print(" " + " ".join(reader_metric)) print(" " + twins) permanent = count_permanently_unresolvable(unknown_producers) print(f" unresolved_producer_sites={len(unknown_producers)} (not proven safe; " @@ -946,6 +1048,12 @@ def main() -> int: if '--report' in sys.argv[1:]: for site in unknown_producers: print(f" unknown_producer: {site}") + for line in reader_detail: + print(f" retirement_role: {line}") + ledger_fields = registry["retirement_ledger"]["should_run_legacy_decision_fields"]["fields"] + uses, _ = scan_field_uses(ledger_fields, sources) + for line in render_field_uses(uses): + print(f" field_use: {line}") return 0 diff --git a/loopx/semantics/field_use.py b/loopx/semantics/field_use.py new file mode 100644 index 0000000000..2b545757de --- /dev/null +++ b/loopx/semantics/field_use.py @@ -0,0 +1,418 @@ +"""Classify how each module uses a named payload field (RFC B3). + +The retirement ledger used to count modules whose text contains the field +token. That metric answers "does this name appear here", which is not the +question retirement asks. Removing a legacy field requires knowing which +modules *read* it, which modules *write* it, and which merely mention it in +prose -- three populations the token count folds into one number. + +What is measured here is syntactic use, not data flow. A module is a reader +when it performs a recognized literal-key read (``payload["legacy_field"]``, +``payload.get("legacy_field")``, ``"legacy_field" in payload``, or the +TypeScript property read). It is a writer when it performs a recognized literal +write (subscript store, dict-literal key, keyword argument, attribute store, +``setdefault``). Reader and writer are not exclusive: a projection module that +reads the legacy field and re-emits it is both. + +These limits are part of the metric, not caveats around it: + +* A computed key is **unresolved**. ``payload.get(name)`` may read any field, so + no name-keyed scan -- lexical or syntactic -- can prove a module is not a + reader. ``dynamic_mapping_key_sites`` counts those sites repository-wide, so a + field measured at zero readers is measured against a stated unknown rather + than declared dead, and a module that carries the field name as a bare string + is reported as unresolved rather than as a mention. Subscripts with a computed + key are deliberately *not* counted: ``rows[index]`` and ``payload[key]`` are + the same syntax, and counting sequence indexing as an unresolved mapping read + would inflate the unknown until it stopped carrying information. +* Same-prefix identifiers are different fields. ``legacy_field_repair`` is not + a use of ``legacy_field``; both language AST scans compare whole keys. +* A field this module measures must not be spelled out here. The scan reads + tracked sources under ``loopx/``, this file is one of them, and a field name + in a docstring would add a mention to that field's own budget. The examples + above use ``legacy_field`` for that reason; the real names live in the + registry and in the smoke's anchors, outside the scanned Python/TypeScript + sources. +* A mention is evidence of nothing. Prompt prose and module paths carry the + token without a recognized access. Locals and parameters are instead bindings: + they may carry the value through a signature that a migration must inspect. +""" + +from __future__ import annotations + +import ast +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path +import re +from typing import Any, Iterable + +from .inventory import SourceFile, parse_python +from .production import run_typescript_scan + +# ``dict``/``Mapping`` accessors whose first literal argument names a field. +MAPPING_READ_CALLS = frozenset({"get", "pop"}) +MAPPING_WRITE_CALLS = frozenset({"setdefault"}) + +READ_FORMS = frozenset({ + "subscript_read", "mapping_call_read", "membership_read", "attribute_read", + "property_read", "destructured_read", +}) +WRITE_FORMS = frozenset({ + "subscript_write", "mapping_call_write", "dict_literal_key", "keyword_argument", + "attribute_write", "property_write", "object_literal_key", +}) +# The module names the field as a parameter, a local or its own definition. It +# handles the value without a recognized key access -- a pass-through consumer +# in the RFC's role hierarchy, and a signature the migration has to change. +BINDING_FORMS = frozenset({ + "local_binding", "local_reference", "parameter", "definition", + "property_signature", +}) +# This module's use of the field is not proven either way. ``name_constant`` is +# the field name travelling as data: a string constant that no recognized key +# position consumed -- a name in a field list a loop will index with, or a label +# in an emitted record. ``unparsed_module`` is a module the scan could not parse +# at all. Both are field-specific: someone has to open *this* module and decide +# before *this* field can be removed, which is why they join the migration +# surface rather than being counted as mentions. The repository-wide computed-key +# totals in ``UnresolvedKeySites`` are the other kind and stay out of it. +UNRESOLVED_FORMS = frozenset({"name_constant", "unparsed_module"}) +MENTION_FORMS = frozenset({"module_import", "prose"}) +USE_FORMS = READ_FORMS | WRITE_FORMS | BINDING_FORMS | UNRESOLVED_FORMS | MENTION_FORMS + + +@dataclass(frozen=True) +class UnresolvedKeySites: + """Computed-key accesses that no name-keyed scan can attribute to a field. + + The two runtimes are counted apart because their exclusions differ, and a + single total would hide that. ``python_mapping_calls`` counts only + ``mapping.get(name)``-shaped calls with a computed argument, because + ``rows[index]`` and ``payload[key]`` are the same subscript syntax in + Python. TypeScript has no mapping-accessor convention, so the equivalent + read *is* the computed member access: ``typescript_members`` counts those + with a non-numeric argument and is therefore an upper bound that includes + indexing an array by a variable. + """ + + python_mapping_calls: int = 0 + typescript_members: int = 0 + + +@dataclass(frozen=True) +class FieldUse: + """One module's recognized uses of one field.""" + + field: str + module: str + forms: frozenset[str] + + @property + def reads(self) -> bool: + return bool(self.forms & READ_FORMS) + + @property + def writes(self) -> bool: + return bool(self.forms & WRITE_FORMS) + + @property + def unresolved(self) -> bool: + return bool(self.forms & UNRESOLVED_FORMS) + + @property + def binds(self) -> bool: + return bool(self.forms & BINDING_FORMS) + + @property + def mention_only(self) -> bool: + """True when nothing but prose or an import carries the name here.""" + return not (self.reads or self.writes or self.binds or self.unresolved) + + @property + def role(self) -> str: + """A single label for ordering and printing -- never a fact set. + + Facts overlap: a projection module both reads and writes. This label + keeps only the first of ``reader > writer > binding > unresolved > + mention``, so it can order a work queue and print one line per module, + and it is the wrong thing to count a population with. ``reads``, + ``writes``, ``binds``, ``unresolved`` and ``mention_only`` are the + orthogonal facts; ``field_use_summary`` counts those beside it. + """ + if self.reads: + return "reader" + if self.writes: + return "writer" + if self.binds: + return "binding" + if self.unresolved: + return "unresolved" + return "mention" + + @property + def in_migration_surface(self) -> bool: + """True when removing the field requires work in this module. + + Readers, writers and bindings have to change. ``unresolved`` modules + have to be *investigated*: the field name is here as data, or the module + did not parse, and nobody can say the field is absent without opening + it. That is field-specific work, so it is counted. The repository-wide + ``UnresolvedKeySites`` totals are not: they are attributable to no + single field, so they stay a standing unknown beside every field's + budget and can never authorize a deletion on their own. + """ + return self.reads or self.writes or self.binds or self.unresolved + + +def _literal_key(node: ast.AST) -> str | None: + if isinstance(node, ast.Constant) and isinstance(node.value, str): + return node.value + return None + + +def python_module_scan(tree: ast.AST, fields: frozenset[str]) -> tuple[dict[str, set[str]], int]: + """Recognized forms per field, and the module's computed-key site count. + + Both come from one walk. The scan runs over every tracked Python module on + every pull request that touches ``loopx/``, so a second traversal is a cost + paid by everyone. Parsing errors share the inventory's ``parse_python`` + boundary; ASTs are deliberately not retained in a cache. + """ + found: dict[str, set[str]] = {} + # Constants consumed as a literal key. Whatever is left over is the field + # name travelling as data, which is how a computed access is written. + keyed: set[int] = set() + seen_constants: list[tuple[int, str]] = [] + dynamic_sites = 0 + + def record(field: str, form: str) -> None: + found.setdefault(field, set()).add(form) + + for node in ast.walk(tree): + key = _literal_key(node) + if key is not None: + if key in fields: + seen_constants.append((id(node), key)) + continue + if isinstance(node, ast.Subscript): + key = _literal_key(node.slice) + if key in fields: + keyed.add(id(node.slice)) + record(key, "subscript_write" if isinstance(node.ctx, (ast.Store, ast.Del)) else "subscript_read") + elif isinstance(node, ast.Call): + function = node.func + if isinstance(function, ast.Attribute) and node.args: + accessor = function.attr in MAPPING_READ_CALLS or function.attr in MAPPING_WRITE_CALLS + key = _literal_key(node.args[0]) + if accessor and key is None: + dynamic_sites += 1 + elif key in fields and function.attr in MAPPING_READ_CALLS: + keyed.add(id(node.args[0])) + record(key, "mapping_call_read") + elif key in fields and function.attr in MAPPING_WRITE_CALLS: + keyed.add(id(node.args[0])) + record(key, "mapping_call_write") + for keyword in node.keywords: + if keyword.arg in fields: + record(keyword.arg, "keyword_argument") + elif isinstance(node, ast.Dict): + for key_node in node.keys: + key = _literal_key(key_node) if key_node is not None else None + if key in fields: + keyed.add(id(key_node)) + record(key, "dict_literal_key") + elif isinstance(node, ast.Compare): + key = _literal_key(node.left) + if key in fields and any(isinstance(op, (ast.In, ast.NotIn)) for op in node.ops): + keyed.add(id(node.left)) + record(key, "membership_read") + elif isinstance(node, ast.Attribute): + if node.attr in fields: + record(node.attr, "attribute_write" if isinstance(node.ctx, (ast.Store, ast.Del)) else "attribute_read") + elif isinstance(node, ast.arg): + if node.arg in fields: + record(node.arg, "parameter") + elif isinstance(node, ast.Name): + if node.id in fields: + record(node.id, "local_binding" if isinstance(node.ctx, (ast.Store, ast.Del)) else "local_reference") + elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + if node.name in fields: + record(node.name, "definition") + elif isinstance(node, ast.ImportFrom): + parts = set((node.module or "").split(".")) + for field in fields: + if field in parts or any(alias.name == field for alias in node.names): + record(field, "module_import") + elif isinstance(node, ast.Import): + for alias in node.names: + parts = set(alias.name.split(".")) + for field in fields & parts: + record(field, "module_import") + for identity, key in seen_constants: + if identity not in keyed: + record(key, "name_constant") + return found, dynamic_sites + + +@lru_cache(maxsize=256) +def _token_pattern(field: str) -> re.Pattern[str]: + return re.compile(rf"(? int: + """The pre-B3 metric: modules whose text contains the standalone token.""" + token = _token_pattern(field) + return sum(1 for source in sources if source.suffix == suffix and token.search(source.text)) + + +def scan_field_uses( + fields: Iterable[str], sources: Iterable[SourceFile], +) -> tuple[list[FieldUse], UnresolvedKeySites]: + """Classify every module's use of each field, beside the standing unknown.""" + wanted = frozenset(fields) + uses: list[FieldUse] = [] + dynamic_sites = 0 + materialized = list(sources) + # Every TypeScript module is scanned, not only those that name a field: + # the standing unknown is repository-wide on both sides, and narrowing it + # to modules that spell the field would measure a zero-reader field against + # an unknown that excludes the modules most able to hide a reader. The + # scanner records a form only for a requested field, so the wider + # population costs one parse and adds no per-field work. Measured on 145 + # tracked modules: 0.86s narrowed against 0.52s more for all of them, and + # the unknown it reports rises from 82 sites to 395. + ts_sources = [source for source in materialized if source.suffix == ".ts"] + # One AST request for the complete TS population, not one Node process per + # module/field. Reuse the producer scanner's parser and safe error boundary. + ts_rows = {row["path"]: row for row in run_typescript_scan( + Path(__file__).resolve().parents[2], ts_sources, + {"mode": "field_uses", "fields": sorted(wanted)}, + )} + ts_dynamic_sites = sum(int(row.get("dynamic_member_sites") or 0) for row in ts_rows.values()) + for source in materialized: + # Every Python module contributes to the computed-key total whether or + # not it names a field, so the cheap substring filter only narrows the + # per-field work, never the standing unknown. + present = frozenset(field for field in wanted if field in source.text) + if source.suffix == ".py": + try: + tree = parse_python(source) + except (SyntaxError, ValueError): + # An unparseable tracked module is a measurement gap, and a gap + # may hold readers. Calling it a mention would shrink the + # migration surface on the strength of a parse failure, so it is + # recorded as this field's unknown and stays in the surface + # until someone reads the module. Failing closed here is worse: + # a direct caller scanning a work-in-progress tree would get an + # exception instead of a measurement. The inventory scan rejects + # such a module first, so the drift smoke never reaches this. + for field in wanted: + if lexical_module_count(field, ".py", [source]): + uses.append(FieldUse(field=field, module=source.path, + forms=frozenset({"unparsed_module"}))) + continue + forms, module_dynamic_sites = python_module_scan(tree, present) + dynamic_sites += module_dynamic_sites + elif source.suffix == ".ts": + row = ts_rows.get(source.path) or {} + forms = {field: set(observed) for field, observed in (row.get("fields") or {}).items()} + else: + continue + for field in present: + recognized = forms.get(field, set()) + if not recognized and _token_pattern(field).search(source.text): + # The token is present but no recognized form carries it: a + # comment, a docstring, or prompt prose. + recognized = {"prose"} + if not recognized: + continue + unclassified = recognized - USE_FORMS + if unclassified: + # A form with no role would disappear from the role counts while + # still carrying the token, breaking the partition the ledger + # check relies on. Fail where the form was added, not there. + raise ValueError(f"unclassified field-use form(s): {sorted(unclassified)}") + uses.append(FieldUse(field=field, module=source.path, forms=frozenset(recognized))) + return sorted(uses, key=lambda use: (use.field, use.module)), UnresolvedKeySites( + python_mapping_calls=dynamic_sites, typescript_members=ts_dynamic_sites, + ) + + +# Print and ordering labels. Single-valued by construction, so they partition +# the classified population -- useful for a work queue, useless for asking how +# many modules read the field. +ROLES = ("reader", "writer", "binding", "unresolved", "mention") +# The orthogonal facts: each is a property of ``FieldUse``, and a module is +# counted in every one that holds of it. They overlap on purpose, so their +# counts do not sum to the population; only ``mention_only`` is disjoint from +# the rest. The summary key for each is ``{runtime}_{key}_modules``. +FACTS = ( + ("reads", "reads"), + ("writes", "writes"), + ("binds", "binds"), + ("unresolved", "unresolved_use"), + ("mention_only", "mention_only"), +) + + +def field_use_summary(fields: Iterable[str], sources: Iterable[SourceFile]) -> dict[str, Any]: + """Per-field use counts and migration surface, beside the old token count. + + ``migration_surface`` is the number of modules that must be changed or at + least investigated before the field can be removed: every reader, writer + and binding, plus the field-specific unknowns. Only mentions -- prose and + imports -- are outside it. The repository-wide ``UnresolvedKeySites`` + totals are reported beside the budgets and are part of no field's surface, + because they are attributable to no field and emptying one can never retire + them. + + Two families of count are reported per runtime and they answer different + questions. + + ``*_reads_modules``, ``*_writes_modules``, ``*_binds_modules``, + ``*_unresolved_use_modules`` and ``*_mention_only_modules`` are the + orthogonal facts, one per entry in ``FACTS``: a module is counted in every + set it belongs to, because a retirement has to fix every site it has. They + overlap, so they do not sum to the population; ``*_classified_modules`` is + their union and equals the token count. + + ``*_{role}_modules`` is instead a partition by the first role in ``ROLES`` + that a module matches, so those five counts do sum to the token count and + the ledger can assert that the roles reclassify that population rather than + sample a smaller one. A reader that also writes is a ``reader`` there and + invisible in ``writer``, which is why the partition must not be read as a + producer count. + """ + materialized = list(sources) + ordered = sorted(fields) + uses, unknown = scan_field_uses(ordered, materialized) + summary: dict[str, Any] = { + "fields": {}, + "dynamic_mapping_key_sites": unknown.python_mapping_calls, + "typescript_dynamic_member_sites": unknown.typescript_members, + } + for field in ordered: + entry: dict[str, Any] = {} + for suffix, runtime in ((".py", "python"), (".ts", "typescript")): + selected = [use for use in uses if use.field == field and use.module.endswith(suffix)] + roles = [use.role for use in selected] + for role in ROLES: + entry[f"{runtime}_{role}_modules"] = roles.count(role) + for fact, label in FACTS: + entry[f"{runtime}_{label}_modules"] = sum(1 for use in selected if getattr(use, fact)) + # One FieldUse per (field, module), and every use carries at least + # one fact, so this is the union of the overlapping sets above. + entry[f"{runtime}_classified_modules"] = len(selected) + entry[f"{runtime}_migration_surface"] = sum(1 for use in selected if use.in_migration_surface) + entry[f"{runtime}_token_modules"] = lexical_module_count(field, suffix, materialized) + summary["fields"][field] = entry + return summary + + +def render_field_uses(uses: Iterable[FieldUse]) -> list[str]: + """One reviewable line per module, for ``--report``.""" + return [ + f"{use.field} {use.role} {use.module} [{','.join(sorted(use.forms))}]" + for use in uses + ] diff --git a/loopx/semantics/inventory.py b/loopx/semantics/inventory.py index f88299b1da..b44b306170 100644 --- a/loopx/semantics/inventory.py +++ b/loopx/semantics/inventory.py @@ -116,6 +116,21 @@ def _assigned_name(node: ast.AST) -> tuple[str, ast.AST] | None: return None +def parse_python(source: SourceFile) -> ast.AST: + """Parse one tracked Python source, rejecting an unparseable one by path. + + Deliberately not cached. Caching the trees would hold roughly two million + AST nodes for the rest of the run, and measured on this tree that costs + ``check_inventory`` more than sharing the parse with the retirement scan + saves. ``python_facts`` caches its small fact dicts instead, so the + inventory still parses each file once per scan. + """ + try: + return ast.parse(source.text) + except SyntaxError as error: + raise ValueError(f"cannot inventory invalid Python source: {source.path}:{error.lineno}") from None + + @lru_cache(maxsize=4096) def python_facts(source: SourceFile) -> dict[str, list[dict[str, Any]]]: """Inventory the Python facts of one source file. @@ -131,10 +146,7 @@ def python_facts(source: SourceFile) -> dict[str, list[dict[str, Any]]]: "literal_aliases": [], "string_constants": [], } - try: - tree = ast.parse(source.text) - except SyntaxError as error: - raise ValueError(f"cannot inventory invalid Python source: {source.path}:{error.lineno}") from None + tree = parse_python(source) for node in tree.body: if isinstance(node, ast.ClassDef) and _is_enum_class(node): values: list[str] = [] diff --git a/loopx/semantics/production.py b/loopx/semantics/production.py index 97eedc2c7a..ea0686202c 100644 --- a/loopx/semantics/production.py +++ b/loopx/semantics/production.py @@ -106,11 +106,23 @@ def _typescript_scan( ) -> list[Production]: if not ts_sources or not field: return [] - rows = [] + rows = run_typescript_scan(root, ts_sources, {'field': field, 'return_functions': returns, 'mode': mode}) + # The parser names the same blocker vocabulary as the Python scanner; + # ``typescript_dynamic`` stays the fallback for a form it cannot classify. + return [Production(r['site'], r['line'], r['form'], frozenset(r['values']), r['unresolved'], + (r.get('blocker') or 'typescript_dynamic') if r['unresolved'] else None) + for r in rows] + + +def run_typescript_scan( + root: Path, ts_sources: list[SourceFile], request: dict[str, Any], +) -> list[dict[str, Any]]: + """Run repository semantic AST analysis with one bounded error boundary.""" + if not ts_sources: + return [] completed = subprocess.run( ['node', str(root / 'scripts/semantic_production_scan.mjs')], - input=json.dumps({'field': field, 'sources': [{'path': s.path, 'text': s.text} for s in ts_sources], - 'return_functions': returns, 'mode': mode}), + input=json.dumps({**request, 'sources': [{'path': s.path, 'text': s.text} for s in ts_sources]}), capture_output=True, text=True, encoding="utf-8", timeout=60, check=False, ) if completed.returncode: @@ -126,12 +138,7 @@ def _typescript_scan( and isinstance(error.get('line'), int) and error['line'] > 0): raise ValueError(f"{error['path']}:{error['line']}: invalid TypeScript source; repair syntax before semantic scanning") raise ValueError('TypeScript production parser failed; run npm ci --ignore-scripts and check the Node runtime') - # The parser names the same blocker vocabulary as the Python scanner; - # ``typescript_dynamic`` stays the fallback for a form it cannot classify. - rows.extend(Production(r['site'], r['line'], r['form'], frozenset(r['values']), r['unresolved'], - (r.get('blocker') or 'typescript_dynamic') if r['unresolved'] else None) - for r in json.loads(completed.stdout)) - return rows + return json.loads(completed.stdout) def collect_literal_uses(root: Path, field: str, sources: list[SourceFile]) -> dict[str, set[str]]: diff --git a/loopx/semantics/vocabulary_v0.json b/loopx/semantics/vocabulary_v0.json index 160fa33fe7..9ce1406301 100644 --- a/loopx/semantics/vocabulary_v0.json +++ b/loopx/semantics/vocabulary_v0.json @@ -1069,31 +1069,43 @@ }, "retirement_ledger": { "should_run_legacy_decision_fields": { - "meaning": "Decision fields the should-run documentation already calls legacy. Budgets count modules under loopx/ that still mention the field.", + "meaning": "Decision fields the should-run documentation already calls legacy. module_budget counts modules under loopx/ whose text carries the standalone field token; migration_surface counts the modules that read, write or bind the field, plus the modules holding its name unresolved and so owing an investigation, all of which is work owed before removal. The repository-wide computed-key totals are attributable to no single field and are never part of a surface. Both budgets are checked; the token count stays until Q11 decides whether the syntactic metric replaces it.", "fields": { "execution_obligation": { "python_module_budget": 20, - "typescript_module_budget": 1 + "typescript_module_budget": 1, + "python_migration_surface": 15, + "typescript_migration_surface": 1 }, "heartbeat_recommendation": { "python_module_budget": 17, - "typescript_module_budget": 1 + "typescript_module_budget": 1, + "python_migration_surface": 13, + "typescript_migration_surface": 1 }, "work_lane_contract": { "python_module_budget": 29, - "typescript_module_budget": 3 + "typescript_module_budget": 3, + "python_migration_surface": 29, + "typescript_migration_surface": 3 }, "external_evidence_observation": { "python_module_budget": 8, - "typescript_module_budget": 1 + "typescript_module_budget": 1, + "python_migration_surface": 7, + "typescript_migration_surface": 1 }, "goal_boundary": { "python_module_budget": 30, - "typescript_module_budget": 2 + "typescript_module_budget": 2, + "python_migration_surface": 16, + "typescript_migration_surface": 1 }, "protocol_action_packet": { "python_module_budget": 5, - "typescript_module_budget": 2 + "typescript_module_budget": 2, + "python_migration_surface": 5, + "typescript_migration_surface": 2 } } } diff --git a/scripts/semantic_production_scan.mjs b/scripts/semantic_production_scan.mjs index a7e28b78fd..7dae91dcee 100644 --- a/scripts/semantic_production_scan.mjs +++ b/scripts/semantic_production_scan.mjs @@ -114,6 +114,81 @@ for (const source of request.sources) { } return values(node).values; }; + if (request.mode === 'field_uses') { + const fields = new Set(request.fields); + const forms = new Map(); + // A computed member access may name any field, so it is the standing + // unknown a name-keyed scan cannot resolve. It is reported apart from the + // Python mapping-call count because the two exclusions differ: `rows[i]` + // and `payload[key]` are one syntax here, so this is an upper bound. + let dynamicSites = 0; + // String literals a key position consumed. A field-named literal left over + // is the name travelling as data -- the Python scan's `name_constant`. + const keyed = new Set(); + const constants = []; + const record = (key, form) => { + if (!fields.has(key)) return; + if (!forms.has(key)) forms.set(key, new Set()); + forms.get(key).add(form); + }; + const markKeyed = name => { + if (!name) return; + keyed.add(name); + if (ts.isComputedPropertyName(name)) keyed.add(unwrap(name.expression)); + }; + const visit = node => { + if (ts.isPropertyAccessExpression(node) || ts.isElementAccessExpression(node)) { + const property = ts.isPropertyAccessExpression(node); + const key = property ? node.name.text : staticName(node.argumentExpression); + if (!property) { + const argument = unwrap(node.argumentExpression); + if (key === null && argument && !ts.isNumericLiteral(argument)) dynamicSites += 1; + else if (key !== null) keyed.add(argument); + } + let target = node, parent = node.parent; + while (parent && unwrap(parent) === target) { target = parent; parent = parent.parent; } + const assignment = ts.isBinaryExpression(parent) && parent.left === target && + parent.operatorToken.kind >= ts.SyntaxKind.FirstAssignment && + parent.operatorToken.kind <= ts.SyntaxKind.LastAssignment; + const update = (ts.isPrefixUnaryExpression(parent) || ts.isPostfixUnaryExpression(parent)) && + [ts.SyntaxKind.PlusPlusToken, ts.SyntaxKind.MinusMinusToken].includes(parent.operator); + const deletion = ts.isDeleteExpression(parent); + const prefix = property ? 'property' : 'subscript'; + if (assignment || update || deletion) record(key, `${prefix}_write`); + if ((!assignment && !deletion) || update || + (assignment && parent.operatorToken.kind !== ts.SyntaxKind.EqualsToken)) { + record(key, `${prefix}_read`); + } + } else if (ts.isBindingElement(node) && ts.isObjectBindingPattern(node.parent)) { + // `const {field} = payload` and `const {field: alias} = payload` read + // the field exactly as `payload.field` does, in a declaration or in a + // parameter list. `propertyName` is present only when the binding + // renames the key; otherwise the bound name is the key. + const name = node.propertyName ?? node.name; + markKeyed(name); + record(named(name), 'destructured_read'); + } else if (ts.isPropertyAssignment(node) || ts.isShorthandPropertyAssignment(node)) { + // An object literal key is where TypeScript writes the field. The + // Python scan counts the same construct as `dict_literal_key`, so + // porting a dict literal to TypeScript must not shrink the surface. + markKeyed(node.name); + record(named(node.name), 'object_literal_key'); + } else if (ts.isPropertySignature(node) || ts.isPropertyDeclaration(node)) { + // A declared contract surface: the type a retirement has to change. + markKeyed(node.name); + record(named(node.name), 'property_signature'); + } else if (ts.isStringLiteral(node) || ts.isNoSubstitutionTemplateLiteral(node)) { + if (fields.has(node.text)) constants.push(node); + } + ts.forEachChild(node, visit); + }; + visit(tree); + for (const node of constants) if (!keyed.has(node)) record(node.text, 'name_constant'); + result.push({path: source.path, dynamic_member_sites: dynamicSites, fields: Object.fromEntries( + [...forms].map(([key, observed]) => [key, [...observed].sort()]), + )}); + continue; + } function walk(node, scope) { if (ts.isFunctionDeclaration(node) || ts.isMethodDeclaration(node)) scope = scope === '' ? named(node.name) : `${scope}.${named(node.name)}`; else if (ts.isArrowFunction(node) || ts.isFunctionExpression(node)) { diff --git a/tests/architecture/test_semantic_field_use.py b/tests/architecture/test_semantic_field_use.py new file mode 100644 index 0000000000..4051d26d81 --- /dev/null +++ b/tests/architecture/test_semantic_field_use.py @@ -0,0 +1,398 @@ +"""Pin the B3 retirement metric against finite counterexamples. + +The metric replaces a token count with a syntactic role, so the tests that +matter are the ones that separate populations the token count merged: a reader +from a writer, a same-prefix identifier from the field, prose from an access, +and a computed key from an absent one. +""" + +from __future__ import annotations + +import ast + +import pytest + +from loopx.semantics.field_use import ( + FACTS, + ROLES, + field_use_summary, + python_module_scan, + scan_field_uses, +) +from loopx.semantics.inventory import SourceFile + +FIELD = "goal_boundary" +FIELDS = frozenset({FIELD}) + + +def python(text: str) -> dict[str, set[str]]: + return python_module_scan(ast.parse(text), FIELDS)[0] + + +def source(text: str, suffix: str = ".py", path: str = "loopx/probe") -> SourceFile: + return SourceFile(path + suffix, suffix, text) + + +def role(text: str, suffix: str = ".py") -> str: + uses, _ = scan_field_uses([FIELD], [source(text, suffix)]) + assert len(uses) == 1, uses + return uses[0].role + + +@pytest.mark.parametrize("text", [ + 'value = payload["goal_boundary"]', + 'value = payload.get("goal_boundary")', + 'value = payload.pop("goal_boundary", None)', + 'present = "goal_boundary" in payload', + 'value = route.goal_boundary', +]) +def test_literal_key_reads_are_readers(text: str) -> None: + assert role(text) == "reader" + + +@pytest.mark.parametrize("text", [ + 'payload["goal_boundary"] = built', + 'payload = {"goal_boundary": built}', + 'emit(goal_boundary=built)', + 'payload.setdefault("goal_boundary", {})', + 'route.goal_boundary = None', +]) +def test_literal_key_writes_are_writers(text: str) -> None: + assert role(text) == "writer" + + +def test_a_module_that_reads_and_writes_counts_as_a_reader_and_stays_in_the_surface() -> None: + text = 'def project(payload):\n payload["goal_boundary"] = payload.get("goal_boundary")\n' + uses, _ = scan_field_uses([FIELD], [source(text)]) + assert uses[0].reads and uses[0].writes + assert uses[0].role == "reader" and uses[0].in_migration_surface + + +@pytest.mark.parametrize("text", [ + 'def build(goal_boundary):\n return goal_boundary\n', + 'goal_boundary = collect()\nreturn goal_boundary\n', +]) +def test_named_parameters_and_locals_are_bindings_in_the_migration_surface(text: str) -> None: + uses, _ = scan_field_uses([FIELD], [source(text)]) + assert uses[0].role == "binding" + assert uses[0].in_migration_surface, "a signature carrying the field still has to change" + + +@pytest.mark.parametrize("text", [ + '"""The quota guard publishes goal_boundary for the agent."""', + '# goal_boundary is described in the should-run docs', + 'note = "read quota.goal_boundary.capabilities before spending"', +]) +def test_prose_is_a_mention_and_never_enters_the_migration_surface(text: str) -> None: + uses, _ = scan_field_uses([FIELD], [source(text)]) + assert uses[0].role == "mention" + assert not uses[0].in_migration_surface + + +def test_a_field_name_carried_as_data_is_unresolved_rather_than_absent() -> None: + text = 'LEGACY = ["goal_boundary", "work_lane_contract"]\nfor field in LEGACY:\n payload.get(field)\n' + uses, _ = scan_field_uses([FIELD], [source(text)]) + assert uses[0].role == "unresolved" + assert "name_constant" in uses[0].forms + assert uses[0].in_migration_surface, ( + "the field name is in this module as data; someone has to read it before the field " + "can be removed, and that is field-specific migration work whichever way it resolves" + ) + + +def test_the_same_prefix_identifier_is_not_a_use_of_the_field() -> None: + text = ( + 'value = payload["goal_boundary_repair"]\n' + 'other = payload.get("repair_goal_boundary")\n' + 'route.goal_boundary_repair = None\n' + ) + assert python(text) == {} + uses, _ = scan_field_uses([FIELD], [source(text)]) + assert uses == [], "goal_boundary_repair must not be counted as a reader of goal_boundary" + + +def test_computed_keys_are_counted_as_the_standing_unknown() -> None: + tree = ast.parse('payload.get(name)\npayload.get("goal_boundary")\nrows[index]\npayload.pop(key, None)\n') + assert python_module_scan(tree, FIELDS)[1] == 2, ( + "literal keys are attributable and sequence indexing is not a mapping access" + ) + + +def test_computed_keys_are_counted_in_modules_that_never_name_the_field() -> None: + uses, unknown = scan_field_uses([FIELD], [source("payload.get(name)\n", path="loopx/other")]) + assert uses == [] and unknown.python_mapping_calls == 1, ( + "the standing unknown is repository-wide; narrowing it to modules that name the " + "field would make a zero-reader field look proven" + ) + + +def test_an_unparseable_module_is_an_unknown_rather_than_a_mention() -> None: + uses, _ = scan_field_uses([FIELD], [source('def broken(:\n "goal_boundary"\n')]) + assert [use.role for use in uses] == ["unresolved"] + assert uses[0].forms == frozenset({"unparsed_module"}) + assert uses[0].in_migration_surface, ( + "a module the scan could not read may hold readers; calling it a mention would " + "shrink the surface on the strength of a parse failure" + ) + + +def test_an_unparseable_module_does_not_fail_the_scan_for_its_callers() -> None: + uses, _ = scan_field_uses( + [FIELD], + [source('def broken(:\n "goal_boundary"\n', path="loopx/broken"), + source('value = payload["goal_boundary"]', path="loopx/reader")], + ) + assert [use.role for use in uses] == ["unresolved", "reader"] + + +@pytest.mark.parametrize("text, expected", [ + ("const source = object(payload.goal_boundary);", "reader"), + ('const source = payload["goal_boundary"];', "reader"), + ("capsule.goal_boundary = projection;", "writer"), + ('capsule["goal_boundary"] = projection;', "writer"), + (" goal_boundary: JsonObject;", "mention"), + ('report("decision.goal_boundary", value);', "mention"), + ("// goal_boundary is projected downstream", "mention"), + ("const repaired = payload.goal_boundary_repair;", None), +]) +def test_typescript_forms_are_classified_by_the_bounded_ast_scan(text: str, expected: str | None) -> None: + uses, _ = scan_field_uses([FIELD], [source(text, ".ts")]) + assert [use.role for use in uses] == ([expected] if expected else []) + + +def test_typescript_string_paths_do_not_become_property_reads() -> None: + assert role('log("decision.goal_boundary");', ".ts") == "mention" + + +@pytest.mark.parametrize("text, expected", [ + ('// payload["goal_boundary"]', "mention"), + ('/* payload["goal_boundary"] = value; */', "mention"), + ('''const note = 'payload["goal_boundary"]';''', "mention"), + ('const note = `payload["goal_boundary"]`;', "mention"), + ('const url = "https://example.test"; const value = payload.goal_boundary;', "reader"), + ('const marker = "/*"; payload["goal_boundary"] = value;', "writer"), + ('// "unclosed quote\nconst value = payload["goal_boundary"];', "reader"), + ('const pattern = /["\\\']+/; const value = payload.goal_boundary;', "reader"), + ('const value = `${payload.goal_boundary}`;', "reader"), + ('const pattern = /payload.goal_boundary/;', "mention"), + ('const value = payload?.["goal_boundary"];', "reader"), + ('payload.goal_boundary += 1;', "reader"), +]) +def test_typescript_lexical_context_preserves_real_accesses(text: str, expected: str) -> None: + assert role(text, ".ts") == expected + + +def test_commented_write_does_not_change_a_real_reader_into_a_writer() -> None: + text = 'const value = payload["goal_boundary"]; // payload["goal_boundary"] = other;' + uses, _ = scan_field_uses([FIELD], [source(text, ".ts")]) + assert uses[0].reads and not uses[0].writes + + +def test_roles_partition_the_token_count_so_the_metric_reclassifies_one_population() -> None: + sources = [ + source('value = payload["goal_boundary"]', path="loopx/reader"), + source('payload["goal_boundary"] = built', path="loopx/writer"), + source('def build(goal_boundary):\n return goal_boundary\n', path="loopx/binding"), + source('LEGACY = ["goal_boundary"]', path="loopx/unresolved"), + source('# goal_boundary', path="loopx/mention"), + source('value = payload["goal_boundary_repair"]', path="loopx/unrelated"), + ] + summary = field_use_summary([FIELD], sources)["fields"][FIELD] + classified = sum(summary[f"python_{role}_modules"] for role in ROLES) + assert classified == summary["python_token_modules"] == 5 + # Reader, writer, binding and the unresolved name carrier; only the prose + # mention is outside. The unresolved module is work whose shape is unknown, + # not work that is known to be absent. + assert summary["python_migration_surface"] == 4 + + +# One access, written the way each runtime writes it. The metric exists to say +# where a retirement has to work, so the answer has to come from what the code +# does, not from which syntax it happens to use. A runtime that reads its half +# differently lets a port manufacture progress: move a dict literal to +# TypeScript and the migration surface shrinks with nothing migrated. +EQUIVALENT_ACCESSES = [ + ("read", 'value = payload["goal_boundary"]', 'const value = payload["goal_boundary"];'), + ("read", 'value = payload.get("goal_boundary")', "const value = payload.goal_boundary;"), + ("read", 'value = payload["goal_boundary"]', "const {goal_boundary} = payload;"), + ("read", 'value = payload["goal_boundary"]', "const {goal_boundary: renamed} = payload;"), + ("read", "def f(payload):\n return payload['goal_boundary']\n", + "function f({goal_boundary}: Payload) { return goal_boundary; }"), + ("write", 'out = {"goal_boundary": built}', "const out = {goal_boundary: built};"), + ("write", 'out["goal_boundary"] = built', "out.goal_boundary = built;"), + ("declare", "def goal_boundary():\n return 1\n", + "interface Decision { goal_boundary: JsonObject; }"), + ("unknown", 'LEGACY = ["goal_boundary"]', 'const legacy = ["goal_boundary"];'), +] + + +@pytest.mark.parametrize("meaning, python_source, typescript_source", EQUIVALENT_ACCESSES) +def test_an_equivalent_rewrite_does_not_change_the_answer( + meaning: str, python_source: str, typescript_source: str, +) -> None: + """Both runtimes must classify the same access the same way. + + This is the test that makes the surface safe to plan against. Without it + the metric measures TypeScript idiom rather than TypeScript behaviour: + destructuring is how TypeScript reads a payload and an object literal is + how it writes one, and both once counted as prose. + """ + python_role = role(python_source) + typescript_role = role(typescript_source, ".ts") + assert python_role == typescript_role, ( + f"{meaning}: python reads this as {python_role} and typescript as " + f"{typescript_role}; a port between the runtimes would move the surface" + ) + + +@pytest.mark.parametrize("text, expected", [ + ("const {goal_boundary} = payload;", "reader"), + ("const {goal_boundary: renamed} = payload;", "reader"), + ("const {outer: {goal_boundary}} = payload;", "reader"), + ("function build({goal_boundary}: Decision) { return goal_boundary; }", "reader"), + ("const out = {goal_boundary: built};", "writer"), + ("const out = {goal_boundary};", "writer"), + ('const out = {"goal_boundary": built};', "writer"), + ("interface Decision { goal_boundary: JsonObject; }", "binding"), + ("type Decision = { goal_boundary: JsonObject };", "binding"), + ("class Decision { goal_boundary: JsonObject; }", "binding"), + ('const names = ["goal_boundary"];', "unresolved"), + ('const key = "goal_boundary"; const value = payload[key];', "unresolved"), +]) +def test_typescript_recognizes_the_forms_its_own_idiom_uses(text: str, expected: str) -> None: + assert role(text, ".ts") == expected + + +def test_a_typescript_computed_member_is_the_standing_unknown_its_runtime_has() -> None: + """A computed member read is TypeScript's `mapping.get(name)`. + + Python excludes `payload[key]` because `rows[index]` is the same syntax; + TypeScript has no mapping accessor, so the computed member *is* the access + and the count is an upper bound. Counting nothing at all was the worse + error: it let the smoke claim a stated unknown that covered one runtime. + """ + _, unknown = scan_field_uses([FIELD], [source("const value = payload[key];", ".ts")]) + assert unknown.typescript_members == 1 + _, indexed = scan_field_uses([FIELD], [source("const first = rows[0];", ".ts")]) + assert indexed.typescript_members == 0, "a numeric literal index is not a mapping read" + + +def test_the_typescript_unknown_covers_modules_that_never_name_the_field() -> None: + """The standing unknown has to be repository-wide on both sides. + + Scanning only the modules that spell a field would measure a zero-reader + field against an unknown drawn from the modules least likely to hide a + reader. Python already counts every tracked module; this is the same + obligation for the runtime whose population the substring filter used to + narrow from 145 modules to 4. + """ + _, unknown = scan_field_uses( + [FIELD], [source("const value = payload[key];", ".ts", path="loopx/other")], + ) + assert unknown.typescript_members == 1 + + +def test_a_module_that_reads_and_writes_is_counted_in_both() -> None: + """The role partition answers a different question than the producer count. + + A projection module reads the legacy field and re-emits it. The partition + calls it a reader, because reader is first in ROLES, and it disappears from + the writer count -- so a retirement looking for every producer would miss + it. The overlapping counts are the ones that answer that. + """ + text = ('value = payload["goal_boundary"]\n' + 'out = {"goal_boundary": value}\n') + summary = field_use_summary([FIELD], [source(text, path="loopx/projection")])["fields"][FIELD] + assert summary["python_reader_modules"] == 1 and summary["python_writer_modules"] == 0 + assert summary["python_reads_modules"] == 1 and summary["python_writes_modules"] == 1 + assert summary["python_migration_surface"] == 1 + + +def test_the_roles_still_partition_the_token_count_over_the_typescript_forms() -> None: + """The new TypeScript forms must reclassify carriers, not add or drop any.""" + sources = [ + source("const {goal_boundary} = payload;", ".ts", path="loopx/reader"), + source("const out = {goal_boundary: built};", ".ts", path="loopx/writer"), + source("interface D { goal_boundary: JsonObject }", ".ts", path="loopx/binding"), + source('const names = ["goal_boundary"];', ".ts", path="loopx/unresolved"), + source("// goal_boundary", ".ts", path="loopx/mention"), + ] + summary = field_use_summary([FIELD], sources)["fields"][FIELD] + classified = sum(summary[f"typescript_{role}_modules"] for role in ROLES) + assert classified == summary["typescript_token_modules"] == 5 + assert summary["typescript_migration_surface"] == 4, ( + "reader, writer, binding and the unresolved name carrier; only the comment is out" + ) + + +def test_the_five_fact_sets_overlap_and_their_union_is_the_whole_population() -> None: + """Every module carrying the token owes at least one fact, and may owe several. + + The role label is single-valued, so counting labels under-reports whichever + fact sorted second. The fact sets are the answer to "how many modules do + X", they overlap, and only their union has to match the token count. + """ + sources = [ + source('payload["goal_boundary"] = payload.get("goal_boundary")', path="loopx/projection"), + source('value = payload["goal_boundary"]', path="loopx/reader"), + source('payload["goal_boundary"] = built', path="loopx/writer"), + source('def build(goal_boundary):\n return goal_boundary\n', path="loopx/binding"), + source('LEGACY = ["goal_boundary"]', path="loopx/unresolved"), + source('# goal_boundary', path="loopx/mention"), + ] + entry = field_use_summary([FIELD], sources)["fields"][FIELD] + assert entry["python_reads_modules"] == 2 and entry["python_writes_modules"] == 2 + assert entry["python_reader_modules"] == 2 and entry["python_writer_modules"] == 1, ( + "the role label keeps only the first fact, which is why it cannot be the count" + ) + # Overlapping sets over-count the population; that is what makes them facts. + assert sum(entry[f"python_{label}_modules"] for _, label in FACTS) == 7 + assert entry["python_classified_modules"] == entry["python_token_modules"] == 6 + assert entry["python_migration_surface"] == 5 + + uses, _ = scan_field_uses([FIELD], sources) + members = {fact: {use.module for use in uses if getattr(use, fact)} for fact, _ in FACTS} + assert members["reads"] & members["writes"] == {"loopx/projection.py"} + assert set().union(*members.values()) == {use.module for use in uses} + assert not members["mention_only"] & ( + members["reads"] | members["writes"] | members["binds"] | members["unresolved"] + ) + + +# One logical access spelled every way TypeScript spells it, plus the type +# declaration that is not an access. `test_an_equivalent_rewrite_does_not_change_ +# the_answer` pins the two runtimes to the same answer; this pins what that +# answer may be. Agreement alone is not enough: if both runtimes read a +# destructuring as prose they would agree and the surface would still shrink +# every time someone reformatted a reader. +EQUIVALENT_TYPESCRIPT_WRITINGS = [ + ("dotted_read", "const value = payload.goal_boundary;", "reader"), + ("subscript_read", 'const value = payload["goal_boundary"];', "reader"), + ("destructured_read", "const {goal_boundary} = payload;", "reader"), + ("aliased_destructured_read", "const {goal_boundary: bound} = payload;\nuse(bound);", "reader"), + ("object_literal_write", "const outbound = {goal_boundary: value};", "writer"), + ("type_declaration", "type Payload = {goal_boundary: JsonObject};", "binding"), + ("name_carried_to_a_subscript", 'const key = "goal_boundary";\nconst value = payload[key];', "unresolved"), +] + + +@pytest.mark.parametrize("name, text, expected", EQUIVALENT_TYPESCRIPT_WRITINGS) +def test_equivalent_writings_do_not_manufacture_retirement_progress( + name: str, text: str, expected: str, +) -> None: + uses, _ = scan_field_uses([FIELD], [source(text, ".ts")]) + assert len(uses) == 1, (name, uses) + assert uses[0].role == expected, (name, sorted(uses[0].forms)) + assert uses[0].role != "mention", ( + f"{name} degraded to a mention; rewriting an access would shrink the migration " + "surface with nothing migrated" + ) + assert uses[0].in_migration_surface, (name, sorted(uses[0].forms)) + + +def test_every_equivalent_writing_of_one_access_is_still_one_module_of_work() -> None: + text = "\n".join(text for _, text, _ in EQUIVALENT_TYPESCRIPT_WRITINGS) + uses, _ = scan_field_uses([FIELD], [source(text, ".ts")]) + assert len(uses) == 1, uses + assert uses[0].reads and uses[0].writes and uses[0].binds and uses[0].unresolved + assert uses[0].role == "reader" and uses[0].in_migration_surface diff --git a/tests/architecture/test_semantic_vocabulary_drift.py b/tests/architecture/test_semantic_vocabulary_drift.py index 9e7b4e40fc..212b1db2a5 100644 --- a/tests/architecture/test_semantic_vocabulary_drift.py +++ b/tests/architecture/test_semantic_vocabulary_drift.py @@ -720,6 +720,75 @@ def test_inventory_report_discloses_budget_slack(monkeypatch): assert "slack=conflicting_values=2" in line, line +def _retirement_registry(python_surface: int, typescript_surface: int) -> dict: + """A one-field ledger shaped like the real one, for the B3 budget checks.""" + return {"retirement_ledger": {"should_run_legacy_decision_fields": {"fields": { + "protocol_action_packet": { + "python_module_budget": 5, + "typescript_module_budget": 2, + "python_migration_surface": python_surface, + "typescript_migration_surface": typescript_surface, + }, + }}}} + + +def test_a_new_reader_of_a_legacy_field_exceeds_its_migration_surface() -> None: + smoke = runpy.run_path(str(SMOKE)) + readers = [ + smoke["SourceFile"](f"loopx/probe_{index}.py", ".py", 'value = payload["protocol_action_packet"]') + for index in range(6) + ] + with pytest.raises(smoke["Drift"], match="modules migrated"): + smoke["check_reader_metric"](_retirement_registry(5, 2), readers) + + +def test_migration_surface_budget_cannot_move_without_its_anchor() -> None: + smoke = runpy.run_path(str(SMOKE)) + with pytest.raises(smoke["Drift"], match="MIGRATION_SURFACE_ANCHOR"): + smoke["check_reader_metric"](_retirement_registry(6, 2), []) + + +def test_the_reader_metric_reports_all_five_facts_beside_the_role_labels() -> None: + smoke = runpy.run_path(str(SMOKE)) + sources = [smoke["SourceFile"]( + "loopx/projection.py", ".py", + 'payload["protocol_action_packet"] = payload.get("protocol_action_packet")', + )] + _, detail = smoke["check_reader_metric"](_retirement_registry(5, 2), sources) + line = next(item for item in detail if item.startswith("protocol_action_packet.py")) + assert "reader=1 writer=0" in line, line + assert "reads=1 writes=1" in line, ( + "one module both reads and writes the field; the role label keeps only the first, " + f"so the overlapping facts have to be reported beside it: {line}" + ) + for label in ("binds", "unresolved_use", "mention_only"): + assert f"{label}=" in line, f"{label} missing from the reported facts: {line}" + + +def test_an_unresolved_name_carrier_is_inside_the_migration_surface() -> None: + smoke = runpy.run_path(str(SMOKE)) + sources = [smoke["SourceFile"]( + "loopx/carrier.py", ".py", + 'LEGACY = ["protocol_action_packet"]\nfor name in LEGACY:\n emit(name)\n', + )] + _, detail = smoke["check_reader_metric"](_retirement_registry(5, 2), sources) + line = next(item for item in detail if item.startswith("protocol_action_packet.py")) + assert "surface=1/5" in line and "unresolved=1" in line, ( + "the field name is in this module as data; that is work to investigate before the " + f"field can go, so it belongs to the surface: {line}" + ) + + +def test_prose_and_same_prefix_identifiers_do_not_consume_the_migration_surface() -> None: + smoke = runpy.run_path(str(SMOKE)) + sources = [ + smoke["SourceFile"]("loopx/prose.py", ".py", '"""protocol_action_packet is published downstream."""'), + smoke["SourceFile"]("loopx/prefix.py", ".py", 'value = payload["protocol_action_packet_v2"]'), + ] + report, detail = smoke["check_reader_metric"](_retirement_registry(5, 2), sources) + assert any("surface=0/5" in line and "mention=1" in line for line in detail), detail + assert any("dynamic_mapping_key_sites=0" in line for line in report), report + def _with_unexecuted_projection(registry: dict) -> dict: registry = copy.deepcopy(registry) registry["projections"]["unexecuted_probe"] = {