process(testing): per-project test strategy at CAC-5, unit tests off by default and enforced

[MendixMau]

Fast-tracked by the maintainer on 2026-10-16 because users report tests firing on every change. The baseline-text A/B is waived; the end-to-end A/B (E3) follows after merge as a check, not a gate.

Direct lane

What changed and why (one paragraph):
Split out of draft #409 (third of three; #413 is step timing, #416 is affected-tests.sh, both merged). Per change we test the gate plus what the change affects; a module's own journeys at module done; everything once at build complete and before every later deploy; everything any time the user says "run everything". The strategy is picked per project with the developer at CAC-5 and recorded in PROJECT.md ## Test strategy (Status CONFIRMED, or ASSUMED in auto mode, no new question). Build complete is declared by the developer, never by the agent alone. Unit tests are OFF by default and are on only for modules named at CAC-5 (calculation, money/date, mapping, REST/Java logic or a fixed bug); writing them is separate from running them.

Field test of the release candidate (#417 + #421 + #427) found "unit tests off by default" was not enforced. Findings: (1) Unit tests default: was written but read by no script; (2) affected-tests.sh defaulted to unit=affected and printed a run ./bin/test-scratch.sh ... .test.mdl row; (3) a grant execute on microflow X that describe appends to a microflow script counted as an access change and selected every unit test of the owning module; (4) the CAC-5 table put unit tests in "Every change"; (5) wording in testing-shape.md module close, brd-to-build-plan.md PROVE and existing-app-change.md could still send an agent to run tests per change. Fixes in this PR:

• affected-tests.sh reads Unit tests default:; no or the line absent means unit is off for every module not flipped, printed as one unit off (...) line and never a run ... .test.mdl row. yes keeps the old behaviour. Build complete and "run everything" (--moment deploy / --moment all) run every existing unit test.
• The ## Test strategy Unit cell is a scope only and never turns unit on: at the change and module moments the default and the per-module switches always filter it (a row recorded with the Offer defaults runs no unit for a module that is off; a row none plus a switched-on module still runs that module). The Offer table shows "modules on only". Unit modules: strips # and ( ... ) comments, reads a - bullet, none and any case, and warns on stderr for a name matching no module.
• Turn unit on: whole project Unit tests default: yes in PROJECT.md; per module Unit modules: Shop, Admin in PROJECT.md (decided at CAC-5) or the line Unit tests: yes (or the Declared-shape row | Unit (...) | yes |) in the module brief.
• A grant/revoke on a microflow or page the same script creates or modifies is part of that subject and selects nothing by module. A standalone grant (entity, module role) still counts.
• Docs: Offer table renamed to Every change / Module done / Build complete with the "run everything" sentence and the developer-declared definition; testing-shape.md events table, module-done and build-complete rows; module-review.md PROVE now runs --moment module --module <M>; existing-app-change.md and the Offer name --module <M> (the bare call selected nothing). REST skills unchanged on purpose.
• Review of the first head (CHANGES): HELD removed from title, CHANGELOG and test header; CHANGELOG conflict with master resolved; waiver in the Impact block.

Impact

Row	Answer
Process	YES. (1) Unit tests are off per project unless turned on (new enforcement in affected-tests.sh); (2) the full run moves from module close to build complete / before each later deploy; module done is module-scoped; (3) new stage artifact in PROJECT.md ## Test strategy, written inside the existing CAC-5 Offer (no new question; the end-of-build "full run now?" is a confirmation); (4) module-review.md PROVE gains one step. Nothing new blocks.
Dependencies	project-bin/affected-tests.sh reads PROJECT.md (Unit tests default:, Unit modules:, ## Test strategy) and module briefs (architecture/modules/<M>/module-brief.md, <M>-brief.md); exec.sh lane text; producers are the CAC-5 Offer and the module brief. Absent means unit off. sync-project.sh does not write PROJECT.md.
Context cost	bin/render-routing.sh --check, origin/master then this head: baseline 75,225 to 75,219 words (budget 80,000). Stage 5 69,795 to 69,789 (test role 21,667 to 21,666, gate 22,731 to 22,729, review 31,435 to 31,434); stage 6 45,835 to 45,832; stage 4 33,487 to 33,482; stage 0 28,247 to 28,245. Every stage and role is at or below master.
Behaviour	NOT RUN, waived by the maintainer on 2026-10-16: users report tests firing after every change and that cost outweighs waiting. The baseline-text A/B and the end-to-end A/B (E3, master vs this branch, scripted changes on a lab copy) run after merge as a check. If E3 shows escapes at build complete that a module-done full would have caught, the default goes back in a follow-up PR.
Runtime	n/a: advisory script reads two more files; no gate change.

Field evidence: a five-app lab project (Mendix 11.15, mxcli v0.26.0): one exec 28 to 41 s, a full mxcli test about 6 min. The RC field test: a logic fix selected a unit test by module through a trailing grant line. Named risk: a defect the full module-close run would have caught may now surface only at build complete.

• No client data anywhere in the diff
• Test tier: T1. tests/wave2/test-affected-tests.sh 79/0 on this head; the new cases (default absent, no, yes, module flip by line, table row and PROJECT.md Unit modules:, describe-style microflow with trailing grant, standalone entity grant, row is scope only, Unit modules: parsing) give 71 pass and 8 fail against the previous affected-tests.sh (origin/master). tests/wave2/test-test-strategy-docs.sh 15/0. exec change-class, subjects and step-time tests pass.
• CHANGELOG.md line under Unreleased
• Categorisation: kind idea (process change, maintainer decision) plus fix (field-test defect) · lane Direct · regression risk medium · toolkit change, not an upstream mxcli bug

🤖 Generated with Claude Code