From b005ca1daa640aeb0537b2c4aa62f3d4a935fffb Mon Sep 17 00:00:00 2001 From: MendixMau Date: Fri, 16 Oct 2026 18:29:22 +0800 Subject: [PATCH] learn(load-test): record the real page, replay it, say what the number does not prove New on-demand skill from a field implementation (warehouse demo app, Mendix 11, 16 runs, bursts of 1000 and 2000 concurrent calls). Playwright records the page's /xas/ calls, a Jaeger probe maps each runtimeOperation to its microflow, and the chosen call is replayed with the session's own cookie and CSRF token, so nothing is correlated by hand. Ten field traps and a four-line report. Routing row, one owner-table row in testing-shape.md, roadmap line reworded. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 1 + ROUTING.md | 1 + agents/review-agent.md | 1 + agents/test-agent.md | 1 + bin/gate-check.sh | 4 +- bin/lib/skill-routing.tsv | 1 + docs/toolkit-roadmap.md | 6 +- skills/load-test.md | 143 ++++++++++++++++++++++++++++++++++++++ skills/testing-shape.md | 1 + 10 files changed, 156 insertions(+), 4 deletions(-) create mode 100644 skills/load-test.md diff --git a/CHANGELOG.md b/CHANGELOG.md index d76e227f..5648e73c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m ## Unreleased +- learn(load-test): **new on-demand skill for load testing a page without correlating Mendix's client protocol by hand: Playwright records the real `/xas/` calls, a probe maps each opaque `runtimeOperation` to its microflow through Jaeger, and the chosen call is replayed with the session's own cookie and CSRF token.** Ten field traps (HTTP 200 on a failed action, caught REST errors under a green microflow span, the wrong call hammered, a recording that still held the previous page, a tracing check that failed open, a harness that throttled itself, harness vs app limits, requests vs users, sampled trace counts, empty data) and a four-line report that says what the run does not prove. Routed ondemand for lead, test and review; no instrument ships yet and the roadmap's "no load testing" line is reworded. Field evidence: a warehouse demo app, Mendix 11, 16 runs 2026-09-09 to 2026-10-08, bursts of 1000 and 2000 concurrent calls on one session — MendixMau - learn(skills): **five lab findings land as compact rows, each paid for by trimming prose in the same files, so always-loaded words do not grow (baseline 75,225 to 75,204, budget 80,000).** Enum value `New` and `AutoCreatedDate`/`AutoOwner` (detection gaps), `[%CurrentDateTime%]` not `currentDateTime()`, expression return types (Decimal from date diffs and `div`, absolute `secondsBetween`, `formatDecimal` on pages), CE2729 from the default home page and page microflow grants, Atlas selector carriers for list views, page body and timelines, `-small` spacing suffixes do not exist, `display: contents` rows paint nothing, CSS pseudo text is invisible to `innerText` — lab field runs, Refs #276 (TK-020, TK-021, TK-022, TK-024) - fix(design-check): **a design system whose palette lives only in scoped `:root.` blocks no longer reads as "0 of N tokens carry the design's value".** Pass 2b compared every token against a bare `:root` of the built sheet, so a themed-only palette always warned, a false warning that teaches agents to ignore the check. A token the design declares only in scoped blocks is now compared block for block against the same-named block in the built sheet, and a wrong value is still named with its block (knob binding stays bare-only: a knob bound inside a theme the app is not wearing is not bound). Synthetic fixtures `ds-themed.css`, `built-themed.css`, `built-themed-wrong.css`; 5 new cases in `test-design-reaches-app.sh` (50 total) — TK-015 part (b) - fix(tests): **`test-exec-hash-verify.sh` case K passes on macOS.** `$TMPDIR` ends in `/`, so the bare `mktemp` path held `//` and the exact `MXPATCH_BACKUP_DIR` path assert could not match; `WORK` is now `cd`-ed into and read back with `pwd -P`. Test only, no script change. 42 pass / 1 fail before, 43 / 0 after — #384 review diff --git a/README.md b/README.md index 05c99b68..f2714749 100644 --- a/README.md +++ b/README.md @@ -670,6 +670,7 @@ Every mxcli project has a `.ai-context/skills/` directory (bundled by `mxcli ini | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | `skills/journey-proof.md` | | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | `skills/unhappy-path-testing.md` | | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | `skills/monkey-test.md` | +| Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) — record the real /xas/ calls with Playwright and replay them instead of correlating tokens by hand, and report what the number does NOT prove (requests vs users, empty data, whose limit was hit) | `skills/load-test.md` | | UX audit and screenshot-loop discipline | `skills/learned-skill-ux-audit.md` | | Tracking scope delta between the BRD and the built state | `skills/learned-skill-scope-delta.md` | | Writing or reading docs/report.json — the append-only contract every instrument writes to and every renderer reads; open BEFORE building a new instrument or a second renderer | `skills/report-schema.md` | diff --git a/ROUTING.md b/ROUTING.md index fbe25354..9cd59930 100644 --- a/ROUTING.md +++ b/ROUTING.md @@ -222,6 +222,7 @@ picks the row up. That is the whole procedure — there is no second list to rem | Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red | `skills/journey-proof.md` | test,review | 5,6 | ondemand | | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test | `skills/unhappy-path-testing.md` | test,review | 5,6 | ondemand | | Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works | `skills/monkey-test.md` | test,review | 5,6 | ondemand | +| Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) — record the real /xas/ calls with Playwright and replay them instead of correlating tokens by hand, and report what the number does NOT prove (requests vs users, empty data, whose limit was hit) | `skills/load-test.md` | lead,test,review | 5,6 | ondemand | | UX audit and screenshot-loop discipline | `skills/learned-skill-ux-audit.md` | review | 6 | ondemand | | Tracking scope delta between the BRD and the built state | `skills/learned-skill-scope-delta.md` | review | 6 | ondemand | | Writing or reading docs/report.json — the append-only contract every instrument writes to and every renderer reads; open BEFORE building a new instrument or a second renderer | `skills/report-schema.md` | test,review,gate | 5,6 | ondemand | diff --git a/agents/review-agent.md b/agents/review-agent.md index f8590697..7cdb9a5a 100644 --- a/agents/review-agent.md +++ b/agents/review-agent.md @@ -70,6 +70,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/journey-proof.md` | Proving a module's user journey end-to-end | | `skills/unhappy-path-testing.md` | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green | +| `skills/load-test.md` | Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) | | `skills/e2e-locators-that-lie.md` | An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) | | `skills/learned-skill-ux-audit.md` | UX audit and screenshot-loop discipline | | `skills/learned-skill-scope-delta.md` | Tracking scope delta between the BRD and the built state | diff --git a/agents/test-agent.md b/agents/test-agent.md index f2d30d89..6633ca7a 100644 --- a/agents/test-agent.md +++ b/agents/test-agent.md @@ -55,6 +55,7 @@ This table is your whole list. The baseline table in the project's CLAUDE.local. | `skills/journey-proof.md` | Proving a module's user journey end-to-end | | `skills/unhappy-path-testing.md` | Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) | | `skills/monkey-test.md` | Running the fuzz/crash net on a module whose journeys are already green | +| `skills/load-test.md` | Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) | | `skills/e2e-locators-that-lie.md` | An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) | | `skills/company-brain.md` | Setting up or wiring a COMPANY BRAIN | | `skills/close-the-loop.md` | Cutover and retrospective | diff --git a/bin/gate-check.sh b/bin/gate-check.sh index 4885ae39..529025ac 100755 --- a/bin/gate-check.sh +++ b/bin/gate-check.sh @@ -1771,8 +1771,8 @@ stage_protocol_paths() { 2) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-brd.md skills/checkpoints/checkpoint-architecture.md skills/dispatching.md skills/image-transcription.md skills/small-project-tier.md skills/kb-generation.md skills/brd-generation.md skills/brd-validation.md" ;; 3) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-design.md skills/dispatching.md skills/small-project-tier.md skills/mendix-best-practices-index.md skills/decompose-app.md skills/mendix-service-seam.md skills/layering-review.md skills/architecture-blueprint.md skills/modularize-domain.md skills/design-artifacts.md skills/interaction-model.md skills/brd-to-build-plan.md skills/workflow-structure-rules.md skills/rest-between-mendix-apps.md skills/learned-mdl-cannot-express.md" ;; 4) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-build.md skills/agent-roles.md skills/dispatching.md skills/small-project-tier.md skills/module-brief.md skills/module-folder-convention.md skills/brd-to-build-plan.md skills/coverage-ledger.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/rest-between-mendix-apps.md skills/learned-constants-and-secrets.md" ;; - 5|build-ready) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/agent-roles.md skills/dispatching.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/decompose-app.md skills/mendix-service-seam.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; - 6) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/dispatching.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/decompose-app.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/e2e-locators-that-lie.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/demo-recording-polish.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 5|build-ready) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/agent-roles.md skills/dispatching.md skills/module-brief.md skills/learned-mdl-preflight.md skills/module-folder-convention.md skills/learned-microflow-patterns.md skills/microflow-preflight.md skills/mendix-best-practices-index.md skills/ui-preflight-pages.md skills/design-spacing.md skills/learned-stylegallery.md skills/ui-loop.md skills/module-review.md skills/testing-shape.md skills/decompose-app.md skills/mendix-service-seam.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/iterative-build-loop.md skills/mdl-cookbook-microflows.md skills/build/mdl/oneshot-mdl-method.md skills/learned-page-patterns.md skills/oneshot-page-structure-patterns.md skills/mendix-agents.md skills/mendix-agent-ui.md skills/mendix-agent-setup.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/load-test.md skills/e2e-locators-that-lie.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/learned-workflow-patterns.md skills/workflow-structure-rules.md skills/rest-integration-first-time-right.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/learned-sidebar-collapse-icons.md skills/learned-popup-navigation.md skills/learned-datagrid-customcontent-binding.md skills/learned-popup-feedback-pattern.md skills/learned-mdl-cannot-express.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-dg2-patterns.md skills/learned-file-upload-widget.md skills/security-is-not-a-later-script.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/preview-over-hub-tunnel.md skills/walking-skeleton.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; + 6) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/dispatching.md skills/mendix-best-practices-index.md skills/module-review.md skills/testing-shape.md skills/existing-app-assurance.md skills/decompose-app.md skills/app-analysis.md skills/module-dependency-review.md skills/microflow-loop-antipatterns.md skills/rehost-app.md skills/qa-loop-goal-pattern.md skills/mendix-agent-setup.md skills/e2e-harness-base.md skills/learned-db-assertions.md skills/fixture-seeding.md skills/journey-proof.md skills/unhappy-path-testing.md skills/monkey-test.md skills/load-test.md skills/e2e-locators-that-lie.md skills/learned-skill-ux-audit.md skills/learned-skill-scope-delta.md skills/report-schema.md skills/harness-architecture.md skills/process-coherence-pass.md skills/e2e-evidence-report.md skills/record-demo-video.md skills/demo-recording-polish.md skills/share-demo-package.md skills/lint-that-actually-runs.md skills/improvement-register.md skills/journey-examples.md skills/wiring-sweep.md skills/workflow-structure-rules.md skills/rest-between-mendix-apps.md skills/bug-submission-checklist.md skills/empty-widget-triage.md skills/anonymize-client-app-for-demo.md skills/learned-css-that-never-applied.md skills/learned-detection-gaps.md skills/learned-local-db-confusion.md skills/full-harness-audit.md skills/test-result-audit.md skills/finding-disposition.md skills/handoff-to-studio-pro.md skills/preview-over-hub-tunnel.md skills/platform-link.md skills/teamserver-alignment.md skills/learned-constants-and-secrets.md" ;; 7) echo "skills/interview-protocol.md skills/change-intake.md skills/change-intake-detail.md skills/grill-mode.md skills/checkpoints/checkpoint-template.md skills/checkpoints/checkpoint-cutover.md skills/close-the-loop.md skills/share-demo-package.md skills/handoff-to-studio-pro.md skills/platform-link.md skills/teamserver-alignment.md skills/deploy-to-sandbox.md" ;; *) echo "" ;; # diff --git a/bin/lib/skill-routing.tsv b/bin/lib/skill-routing.tsv index 94ade35c..936acf86 100644 --- a/bin/lib/skill-routing.tsv +++ b/bin/lib/skill-routing.tsv @@ -173,6 +173,7 @@ fixture-seeding skills/fixture-seeding.md Seed data, test data, demo data or fix journey-proof skills/journey-proof.md Proving a module's user journey end-to-end — the deep form of step 3 PROVE; use whenever an instrument reports green and you cannot say what would have made it red test,review 5,6 ondemand verify unhappy-path-testing skills/unhappy-path-testing.md Writing or running e2e cases beyond the happy journeys (blank/whitespace required fields, over-length, duplicates, cancel, double submit, wrong state, wrong role, backend down, stale write, empty search) — every case cites a requirement first, UNSPECIFIED is re-scored against house rules, GAP counts carry the denominator; runs after journey-proof and before monkey-test test,review 5,6 ondemand verify monkey-test skills/monkey-test.md Running the fuzz/crash net on a module whose journeys are already green — and reading the result, which is NOT evidence the module works test,review 5,6 ondemand verify +load-test skills/load-test.md Load, performance or stress testing a page ("will it hold 1000 users", concurrent users, k6, JMeter) — record the real /xas/ calls with Playwright and replay them instead of correlating tokens by hand, and report what the number does NOT prove (requests vs users, empty data, whose limit was hit) lead,test,review 5,6 ondemand verify e2e-locators-that-lie skills/e2e-locators-that-lie.md An e2e rung reports a UI defect (widget never appeared, click did nothing, screen contradicts the data, fixture count wrong) — check the locator before blaming the app: :visible, click order, poll, derived fixtures test,review 5,6 ondemand diagnose learned-skill-ux-audit skills/learned-skill-ux-audit.md UX audit and screenshot-loop discipline review 6 ondemand verify learned-skill-scope-delta skills/learned-skill-scope-delta.md Tracking scope delta between the BRD and the built state review 6 ondemand verify diff --git a/docs/toolkit-roadmap.md b/docs/toolkit-roadmap.md index 8447ee62..a0649274 100644 --- a/docs/toolkit-roadmap.md +++ b/docs/toolkit-roadmap.md @@ -321,8 +321,10 @@ Ideas with a reason and no design. Listed so they are not rediscovered. forbidden by the coverage skill, and generating a picture of a screen that already exists is exactly that. Wireframe what has no implementation yet, never draw one backwards from running code. -- A load testing capability. Measured as theatre without production data volumes, and the static - index audit answers the same question more honestly and in about a minute. +- A shipped load testing instrument. The method is now a skill (`skills/load-test.md`, added + 2026-10-16 from a field implementation); a generic `project-tests/e2e/loadtest.js` waits until + a second project runs the recipe. The theatre objection stands for empty databases, which is + why the skill makes the report state the data volume. - A randomised browser walker. The client has documented traps that make an untuned walker thrash rather than explore. The model level equivalent above is the version worth building. - Anything requiring a graph database or a vector store. The activity level schema is a good idea diff --git a/skills/load-test.md b/skills/load-test.md new file mode 100644 index 00000000..6ba064e9 --- /dev/null +++ b/skills/load-test.md @@ -0,0 +1,143 @@ +# Skill: load-test — record the real page, replay it, and say what the number does not prove + +**Applies to:** any mxcli project with a runnable app, pipeline or not. On request only: a +load test is never a build-plan step and never a gate. +**Purpose:** answer "does this page hold up under N at once" without correlating Mendix's +client protocol by hand, and without a report that overclaims. +**Source:** a field implementation on a warehouse demo app (Mendix 11, `mxcli run --local` +with OTel to Jaeger), 16 recorded runs from 2026-09-09 to 2026-10-08, bursts of 1000 and +2000 concurrent calls. Every trap below was hit in those runs. + +**Instrument:** none ships in `project-tests/` yet. Build it in the project's `tests/e2e/` on +top of the shipped `helpers.js` (login, nav) and `otel.js` (`capture`, `liveness`). This file +holds the method and every judgement; the script fetches facts only (`skills-over-scripts.md`). + +--- + +## Why record and replay, not hand-written requests + +A Mendix page does not call a public API. The client posts to `/xas/`, and in Mendix 11 every +client-invoked microflow is action `runtimeOperation` with an opaque `operationId`. Writing +that payload by hand is a guess, and it changes between versions. Classic protocol tools hit +the same wall: dynamic ids and the CSRF token must be correlated, again after every release. + +The recipe skips correlation entirely: + +``` +1. Playwright logs a real persona in and opens the page (helpers.js login, navTo) +2. record every POST to /xas/ while the page loads (page.on('request')) +3. probe: fire each recorded call once, read Jaeger (otel.js capture) + → which microflow did it run, does it write? +4. replay the chosen call with that session's cookie and the + token from mx.session.getCSRFToken() (raw node:http) +5. ramp → burst → optional ladder → one lone request after +``` + +After a release you re-record, one command. Nothing to re-correlate. + +A k6 script fed from a HAR export works too, but the HAR carries one live session cookie: +replayed as-is, 1000 virtual users are one session, and the cookie expires mid-run. Each VU +must log in itself. Delete the HAR afterwards; it holds a live session. + +## When to run it + +| Situation | Run? | +|---|---| +| The user asks "will this hold N users", before a go-live or a sizing talk | **Yes**, on the page they name | +| Journeys for that page are red | No. A broken page under load measures the breakage. | +| As evidence a module is done | **No.** Load is not correctness. | +| Against a shared or production environment | Only with the owner's written yes. A burst is indistinguishable from an attack. | +| The chosen call writes (create, commit, move, submit, import) | **Refuse** unless the user explicitly allows writes on a throwaway database | + +## The traps, each paid for + +**1. HTTP 200 on a failed action.** Mendix answers 200 and puts the failure in the JSON body +under `error`. Counting status codes reported 0% errors over a broken service. Count +transport errors and application errors separately; parse every body. + +**2. A caught failure leaves the microflow green.** A handler that catches an upstream REST +error lets the page render and the microflow span end OK; only its activity spans are ERROR. +Walk activity-level spans, not just the microflow span. + +**3. The wrong call gets hammered.** Every recorded call reads `runtimeOperation`, so a label +regex cannot tell the grid's data fetch from a keep-alive. Measured: the most frequent call +was a 771-byte no-op; replaying it measured the HTTP dispatcher and reported it as the page. +Choose by the probe (which microflow ran), and when the probe finds nothing, by the biggest +response, and say in the report that the choice was not evidence-backed. + +**4. The tape holds the previous page.** Login and the page you parked on fire microflows too. +Measured: without clearing, six dashboard microflows topped the candidate list of a list-page +test. Park on another page, clear the recording, navigate, then prove you landed (a ready +selector, not caption text that sits in the nav on every page). + +**5. Tracing looks on when it is off.** Jaeger keeps a service name in its list long after the +process is gone. Two full runs printed `serverSide: null` because the check was "the name is +listed". The gate is: spans for this service arrived after the page was opened. No spans after +a real navigation is an **instrument FAULT** (exit 2), not a clean run. Usually the cause is a +`--trace-service` override; mxcli's default (the .mpr name) already matches. + +**6. The harness throttles itself.** Node's `fetch` pools sockets, so "1000 concurrent" became +"N concurrent, queued" and the app got the credit. Use `http.Agent({ maxSockets: Infinity })` +and fire the whole burst before awaiting any answer. No socket timeout: a 90 s answer is a +finding, a 30 s cut-off relabels it as a harness error. + +**7. Whose limit was hit.** On the ladder, classify every stop: + +| Error | Who broke | +|---|---| +| `ECONNRESET`, `ECONNREFUSED`, `EPIPE`, 5xx, error body | **the app** | +| `EADDRNOTAVAIL`, `EMFILE`, `ENFILE` | **the harness** (~28k ports per destination, TIME_WAIT ~60 s carries across rungs) | + +After each rung send one lone request. A rung that answered thousands and then cannot answer +one is a server that fell over late, and the rung's percentiles called it a pass. + +**8. Requests are not users.** One recorded cookie replayed by every worker is N concurrent +*requests* on one session. For users: log in K separate sessions and round-robin their +cookie and token pairs, from K **distinct** accounts. Two live sessions on one account can +degrade each other (measured: both lost their navigation menu), which reads as "the app got +slow". End every session afterwards through the logout action; closing the browser context +leaves it alive against the runtime's session cap. + +**9. The trace store samples.** Jaeger's query caps at a few hundred traces; one burst makes +far more. Read server-side percentiles, never counts, and never read "0 ERROR spans" as "no +request failed" above the cap. + +**10. An empty database is a different app.** A p95 on 50 rows says nothing about 5 million. +State the row counts of the entities the page reads, and look at the query plan or missing +indexes before blaming the runtime. This is the honest half of "load testing is theatre". + +## Reading the result + +| Run says | Verdict | +|---|---| +| 0 transport, 0 app errors, lone request after answers, spans arrived | **PASS for this call, this data, this host**: nothing more | +| App errors > 0 or 5xx | **FAIL**, name the microflow from the probe | +| Stopped on `EADDRNOTAVAIL`/`EMFILE` | **Harness limit**, not a finding about the app | +| No spans after navigation | **FAULT**, no server-side claim may be made | +| Target picked by size, not by probe | Report it, with that caveat in the first line | + +Field numbers, for scale: bursts of 1000 on one session gave p95 between 0.2 s and 4.6 s +across runs with 0 errors; 2000 gave 1.2 to 11.5 s. All local, one account, one replayed call. +That is "the runtime absorbs 1000 calls at once on a laptop", not "1000 people can use it". + +## Reporting + +Four lines, every time: + +``` +load · → · · + burst : p50/p95/p99 <..> · transport · app errors · lone request after: + sessions: on distinct accounts [or: one session, so concurrent requests, not users] + Read as: . Not: production capacity. +``` + +Write the full run to `tests/e2e/artifacts/loadtest-.json`, with the recorded +calls, the probe mapping (operationId to microflow) and the chosen target, so a reader can +check what was hammered. + +## Related + +- `skills/testing-shape.md`: the module shape this sits outside of +- `skills/monkey-test.md`: the same "what a clean run is not allowed to mean" discipline +- `skills/e2e-harness-base.md`: login, persona and the no-unapproved-fix rule +- `skills/tool-output-is-not-ground-truth.md`: why a quiet run is not a passing one diff --git a/skills/testing-shape.md b/skills/testing-shape.md index b9defd0f..becb0491 100644 --- a/skills/testing-shape.md +++ b/skills/testing-shape.md @@ -21,6 +21,7 @@ and the guard for each. Job 2 is why this skill exists; if you shorten it, do no | Why `$?` and tool output are not evidence | `tool-output-is-not-ground-truth.md` (shared toolkit) | | Does the whole journey hang together, not just each piece — a different axis, not a fifth rung | `process-coherence-pass.md` (shared toolkit) | | The full build→gate→prove→look→confirm→next loop this shape plugs into, incl. monkey testing | `module-review.md` (shared toolkit) | +| "Will this page hold N users": on request only, outside the module shape | `load-test.md` (shared toolkit) | > **Promotion note, closed 2026-08-18.** All three rows above now have a shared-toolkit copy, > so §4 — the load-bearing section — no longer cites anything a fresh project cannot open.