From 680fefc003546c8e527cc0ea250fcf54b544f960 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 14:02:49 +0000 Subject: [PATCH 1/2] =?UTF-8?q?process(testing):=20three=20check=20tiers?= =?UTF-8?q?=20=E2=80=94=20every=20exec,=20module=20close,=20release=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maintainer decision: only fast checks run during development; heavy checks run at module close or at the release gate. testing-shape.md §3 now holds the tier table as the single source, beside the unchanged lanes and re-run events. Unbuilt checks are marked (planned: #N): #348, #349 (PR #355), #351, #352, #353, #354. The runbook's Stage 6 paragraph points at it in one line. Baseline words 75,028 -> 75,261 (budget 80,000). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 1 + skills/conversion-runbook.md | 3 ++- skills/testing-shape.md | 14 ++++++++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ff60ea83..b10c9c44 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- process(testing): **three check tiers — every exec (seconds), module close (minutes), release gate before UAT or production.** Only fast checks run during development; heavy checks wait for module close or the release gate. `testing-shape.md` §3 holds the tier table as the single source, beside the existing lanes and re-run events, which are unchanged. Checks not built yet are marked `(planned: #N)`: the static STOP-row scan (#349, PR #355), the Studio Pro warning ratchet (#351) and human-locked baselines (#348) for every exec; new page/action has a test (#354) and per-module server-side security tests (#352) at module close; page × role smoke on the production DB engine (#353), the full security report (#352) and language/RTL layout (#354) at the release gate. The runbook's Stage 6 paragraph points at it in one line. Field evidence: none yet. — MendixMau (from external reviewer feedback) - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/skills/conversion-runbook.md b/skills/conversion-runbook.md index 1f229619..8bddd642 100644 --- a/skills/conversion-runbook.md +++ b/skills/conversion-runbook.md @@ -705,7 +705,8 @@ nothing about the seams. **Read `testing-shape.md` before writing or judging any test here.** It defines what testing a module means, so it is the same thing every time, and it carries the **false-green register**: the -confirmed ways a test reports green while the feature is broken. Alongside it, `measured-claims.md` +confirmed ways a test reports green while the feature is broken. Its §3 check tiers name what the +release gate owes beyond module close. Alongside it, `measured-claims.md` holds the rule that gives evidence its force — *a claim not measured in that register may not be cited as evidence.* diff --git a/skills/testing-shape.md b/skills/testing-shape.md index 925a7f48..cf338191 100644 --- a/skills/testing-shape.md +++ b/skills/testing-shape.md @@ -169,6 +169,20 @@ their trigger is. | Project close / cutover | The project-wide suite, once. | | The user asks for a full run | Run it, that message is the trigger. | +**Check tiers — which checks each of those moments owes.** Fast checks run during development; +heavy checks wait for module close or the release gate. This sorts checks by *when*, not what +testing a module means: the §1 shape and the two lanes are unchanged, and this is not the numbered +ladder §1 dropped. A row marked *(planned: #N)* is not built yet and claims nothing until it lands. + +| Tier | Budget | What runs | +|---|---|---| +| **Every exec** | seconds | What `exec.sh` already runs: the exec lock, the mxbuild delta gate, the lint ratchet. Static STOP-row scan before the apply (planned: #349, PR #355); Studio Pro warning ratchet (planned: #351); human-locked gate baselines (planned: #348). | +| **Module close** | minutes | The module's full suite and journey set (row above). Every new page or action has a test (planned: #354); that module's server-side security tests, run as each demo user (planned: #352). | +| **Release gate** — before UAT or production | as needed | The project-wide suite (row above). Page × role smoke on the production database engine (planned: #353); the full security report (planned: #352); language/RTL layout when the app has several languages (planned: #354). | + +A heavy check never moves into the every-exec tier because a change "felt wide" — that is the +over-testing measured below. + > **Why scoped (#274):** the old unscoped "re-run the suite as regression" re-proved each UI-only > change 5–10× (other modules' journeys, the reviewer re-running them, a full `mxcli test` at every > batch close), and with no Unit default set, **143 minutes** of microflow tests found stale strings, From 4933b1c0059aa192e3d8d794afcd4ac51ea9afee Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 14 Oct 2026 14:20:37 +0000 Subject: [PATCH 2/2] =?UTF-8?q?process(testing):=20review=20fixes=20for=20?= =?UTF-8?q?#364=20=E2=80=94=20check=20moments,=20exec=20chain,=20one=20rel?= =?UTF-8?q?ease=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename "tiers" to "check moments" so it does not collide with §1's "No tiers". - Every-exec row lists the real exec.sh chain (lock, module-brief advisory, mxcli check, snapshot, apply, mxbuild delta gate, auto-restore on a new error, lint ratchet), verified against project-bin/exec.sh's header; budget ~1 minute (mxbuild 22-65 s per script, iterative-build-loop.md Gate cost). - The re-run event row "Project close / cutover" is renamed "Release gate (before UAT/production)", so the project-wide suite has one name and runs once; the moment row points at it. - Drop the closing sentence that repeated the "felt wide" rule; shorter planned markers. Baseline words 75,261 -> 75,225. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VJgWP5vEoAsNsJYqCDMGNw --- CHANGELOG.md | 2 +- skills/conversion-runbook.md | 2 +- skills/testing-shape.md | 19 ++++++++----------- 3 files changed, 10 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b10c9c44..32f82cee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased -- process(testing): **three check tiers — every exec (seconds), module close (minutes), release gate before UAT or production.** Only fast checks run during development; heavy checks wait for module close or the release gate. `testing-shape.md` §3 holds the tier table as the single source, beside the existing lanes and re-run events, which are unchanged. Checks not built yet are marked `(planned: #N)`: the static STOP-row scan (#349, PR #355), the Studio Pro warning ratchet (#351) and human-locked baselines (#348) for every exec; new page/action has a test (#354) and per-module server-side security tests (#352) at module close; page × role smoke on the production DB engine (#353), the full security report (#352) and language/RTL layout (#354) at the release gate. The runbook's Stage 6 paragraph points at it in one line. Field evidence: none yet. — MendixMau (from external reviewer feedback) +- process(testing): **three check moments — every exec (~1 minute), module close (minutes), release gate before UAT or production.** Only fast checks run during development; heavy checks wait for module close or the release gate. `testing-shape.md` §3 holds the check-moment table as the single source, beside the existing lanes and re-run events; the event row "Project close / cutover" is renamed "Release gate (before UAT/production)" so the project-wide suite has one name and runs once. The every-exec row lists the chain `exec.sh` already runs. Checks not built yet are marked `planned: #N`: the static STOP-row scan (#349), the Studio Pro warning ratchet (#351) and human-locked baselines (#348) for every exec; new page/action has a test (#354) and per-module server-side security tests (#352) at module close; page × role smoke on the production DB engine (#353), the full security report (#352) and language/RTL layout (#354) at the release gate. The runbook's Stage 6 paragraph points at it in one line. Field evidence: none yet. — MendixMau (from external reviewer feedback) - process(build): **Stage 5 drafts the ready set in parallel and applies serially.** The runbook dispatch row said "sequential within a module", which serialised drafting even for independent scripts; it now points to `iterative-build-loop.md` → Gate cost, the single statement of the rule. The unit is the build-plan row: every `BUILD` row whose "Depends on" rows are all applied, minus overlapping writers (two rows that replace, alter or drop+create the same document never share a ready set, and no row is drafted while it overlaps a queued unapplied one; `brd-to-build-plan.md` has Produces name every document a row writes), runs at once, its scripts in order, one `mdl-agent` per script. Applies stay one at a time through `bin/exec.sh`, and the ready set is recomputed after each. A queued draft made before a sibling apply landed is re-checked with `mxcli diff --format struct` (preflight STOP rule 23); any removal line (checked against `describe` per BUG-146) or STOP rule 21 target means re-draft, never patch, because a stale restatement of a shared document is exactly rule 21's silent seed-call loss and passes `check --references`. Report `ready set N, drafted N, applied k`; the `mdl-agent` template says a drafter never applies and never depends on an unapplied draft. Field evidence: none yet. — MendixMau - fix(platform-link): **CLI commits now show their changed documents in Studio Pro History.** Studio Pro's History → Model changes pane reads a `ModelChanges` array from the commit's `refs/notes/mx_metadata` note. Trap 6c's hand-written template set it to `[]`, so every mxcli/agent commit showed "Model changes 0" (field project: 270 of 270 notes empty). New `project-bin/mx-metadata-note.py` (stdlib only) computes the array from the commit's changed `.mxunit` files, in Studio Pro's entry shape: Status, UnitID, UnitType, UnitName and Module (walked up the `.mpr` Unit tree). It has `changes` / `write` / `backfill` / `sync ` subcommands. `install-project-hooks.sh` now also installs post-commit and post-rewrite hooks that write the note and never block, and `--check` covers them, so `sync-project.sh` rolls them out to wired projects. Notes are published with `mx-metadata-note.py sync ` after the branch push. Push config is never touched. With mxcli v0.26+ in the project root the hooks call `mxcli git note --write` instead (mendixlabs/mxcli#1337); the Python script is the fallback for older pins, and on 160 field commits both wrote the same 3,594 entries. Trap 6c rewritten. Measured against Studio Pro's own notes, 51 commits in 5 apps: 3387/3389 entries exact on all five fields. Field run on a 357-commit Mendix 11.15.0 app: 158 notes backfilled and pushed, and Studio Pro History lists the documents per revision. Review fixes: an earlier revision added `HEAD` + the notes ref to `remote..push`. It was dropped because any push refspec replaces `push.default`. A branch tracking a differently named upstream got a new same-named branch on Team Server where `simple` refuses, a missing local notes ref failed the whole push, and nobody knows whether Studio Pro's own push reads that config. The installer now only fetches the remote's notes (non-forced). It removes that exact `HEAD` + notes pair if present, and leaves and reports any other entry. `sync` handles a remote with no notes ref (it used to exit 1) and retries a rejected push once after re-merging. A note that already lists changes (Studio Pro's) is never overwritten by `write`/`backfill` without `--force`, and `sync` keeps a colleague's non-empty note over ours. The mxcli probe greps `git note --help` for `--write`. Hooks and docs call the script through `resolve_py`. Fixture `tests/wave2/test-mx-metadata-note.sh` (40 cases, over real captured template units; single- and two-tree layouts, amend, backfill and `--force`, push exit codes identical to a no-hooks clone, leftover-refspec cleanup, sync to an empty remote and its retry, notes fetch at install, a diverged notes merge; green under `init.defaultBranch` master and main). — field project (Mendix 11.15 app), MendixMau - learn(fixture-seeding): **explore the data structure first, choose rows by use case, scan actions for resets before any run.** Two seeding runs on public template apps: seeding raised journey passes 33 -> 58 and 20 -> 51 of the same sets (FAIL 93 -> 68, 163 -> 132; INVALID unchanged), and a "Reset all demodata" journey wiped accounts so 104 of 127 journeys reported `The username or password you entered is incorrect`, identical on empty and seeded databases. Skill gains three questions a seed must answer, a use-case checklist, a destructive-action scan, a generic traps table with symptoms, measured expectations and a denominator checklist; routing trigger now fires on seed/test/demo data and empty pages. Independent review fixes: object ids follow the Mendix bigint pattern with a business-key guard (no uuids), associations are seeded where the page's path stores them, four new traps (enum names, inheritance chain, calculated attributes, view/external entities), and the port-offset and container-label traps moved to `harness-architecture.md`. — MendixMau diff --git a/skills/conversion-runbook.md b/skills/conversion-runbook.md index 8bddd642..069920ef 100644 --- a/skills/conversion-runbook.md +++ b/skills/conversion-runbook.md @@ -705,7 +705,7 @@ nothing about the seams. **Read `testing-shape.md` before writing or judging any test here.** It defines what testing a module means, so it is the same thing every time, and it carries the **false-green register**: the -confirmed ways a test reports green while the feature is broken. Its §3 check tiers name what the +confirmed ways a test reports green while the feature is broken. Its §3 check moments name what the release gate owes beyond module close. Alongside it, `measured-claims.md` holds the rule that gives evidence its force — *a claim not measured in that register may not be cited as evidence.* diff --git a/skills/testing-shape.md b/skills/testing-shape.md index cf338191..43f5bac8 100644 --- a/skills/testing-shape.md +++ b/skills/testing-shape.md @@ -166,22 +166,19 @@ their trigger is. | A new script within agreed scope | Re-run its subject set. Extend the suite only for a *behaviour* the coverage checklist doesn't already cover. | | A new behaviour / scope change | The coverage checklist changes — the existing drift-sync path. The suite extends with it. | | Module close | The full suite, once (`module-review.md` PROVE). | -| Project close / cutover | The project-wide suite, once. | +| Release gate (before UAT/production) | The project-wide suite, once. | | The user asks for a full run | Run it, that message is the trigger. | -**Check tiers — which checks each of those moments owes.** Fast checks run during development; +**Check moments — which checks each of those events owes.** Fast checks run during development; heavy checks wait for module close or the release gate. This sorts checks by *when*, not what -testing a module means: the §1 shape and the two lanes are unchanged, and this is not the numbered -ladder §1 dropped. A row marked *(planned: #N)* is not built yet and claims nothing until it lands. +testing a module means: the §1 shape and the two lanes are unchanged. A *planned: #N* check is not +built yet and claims nothing until it lands. -| Tier | Budget | What runs | +| Moment | Budget | What runs | |---|---|---| -| **Every exec** | seconds | What `exec.sh` already runs: the exec lock, the mxbuild delta gate, the lint ratchet. Static STOP-row scan before the apply (planned: #349, PR #355); Studio Pro warning ratchet (planned: #351); human-locked gate baselines (planned: #348). | -| **Module close** | minutes | The module's full suite and journey set (row above). Every new page or action has a test (planned: #354); that module's server-side security tests, run as each demo user (planned: #352). | -| **Release gate** — before UAT or production | as needed | The project-wide suite (row above). Page × role smoke on the production database engine (planned: #353); the full security report (planned: #352); language/RTL layout when the app has several languages (planned: #354). | - -A heavy check never moves into the every-exec tier because a change "felt wide" — that is the -over-testing measured below. +| **Every exec** | ~1 minute (mxbuild 22–65 s per script, `iterative-build-loop.md` → Gate cost) | `exec.sh` includes: lock → module-brief advisory → `mxcli check` → snapshot → apply → mxbuild delta gate → auto-restore on a new error → lint ratchet. Static STOP-row scan (planned: #349); Studio Pro warning ratchet (planned: #351); human-locked baselines (planned: #348). | +| **Module close** | minutes | The module's full suite and journeys (row above). New page or action has a test (planned: #354); that module's server-side security tests (planned: #352). | +| **Release gate** | as needed | The project-wide suite, once (row above). Page × role smoke on the production DB engine (planned: #353); full security report (planned: #352); language/RTL layout, multi-language apps only (planned: #354). | > **Why scoped (#274):** the old unscoped "re-run the suite as regression" re-proved each UI-only > change 5–10× (other modules' journeys, the reviewer re-running them, a full `mxcli test` at every