Goal
Make WrightKit-wide policy explicitly prohibit durable documentation from using test, fixture, snapshot, corpus, or testcase counts as evidence of quality, compatibility, support, or project progress.
Context
The current testing policy states that counts such as 26/26 fixtures passed or 124 tests passed are not correctness claims, but still describes them as useful execution summaries. This leaves room for agents to persist volatile counts into README files, compatibility documents, evidence records, PR narratives, and other long-lived documentation.
Absolute testcase inventory is not a meaningful capability contract. It becomes stale as the suite evolves and encourages quantity-based reporting instead of describing the semantic behavior, scenarios, boundaries, regressions, and evidence actually covered.
Execution totals may still exist transiently in test-runner output, CI logs, or generated CI summaries where they describe that specific run. They must not be promoted into durable documentation or capability claims.
Scope
- Strengthen
docs/testing-policy.md so test/fixture/corpus counts are prohibited in durable documentation and long-lived capability/status evidence.
- Make the agent-facing routing/invariants strong enough that documentation work does not introduce these counts even when the task is not primarily a testing task.
- Require documentation and review evidence to describe semantic coverage, protected contracts, known gaps, and relevant scenarios instead of testcase quantity.
- Define the narrow execution-time exception for ephemeral test-runner/CI output.
- Establish a cleanup expectation for existing WrightKit documentation that contains volatile test/fixture/corpus inventory counts.
Non-goals
- Do not suppress test-runner output or CI metadata that reports counts for a single execution.
- Do not forbid numeric values that are themselves part of a product, protocol, semantic, performance, or resource contract.
- Do not introduce a brittle repository-wide numeric-text linter that would reject legitimate numbers.
- Do not change the actual composition of test suites or corpora solely to alter their size.
Acceptance Criteria
docs/testing-policy.md no longer characterizes testcase counts as suitable durable summaries and explicitly bans agents/contributors from writing them into long-lived documentation or capability claims.
- Workspace agent guidance routes or states the prohibition clearly enough to apply during documentation work.
- The policy distinguishes ephemeral execution metadata from durable documentation.
- The preferred replacement is explicit: name semantic capabilities, scenarios, contracts, regressions, evidence states, and known gaps rather than testcase quantity.
- Known existing occurrences in core WrightKit repositories are identified for owning-repository cleanup without duplicating authoritative policy text.
Dependencies
- None for the policy change.
- Owning repositories should perform their own documentation cleanup after this policy is accepted.
Goal
Make WrightKit-wide policy explicitly prohibit durable documentation from using test, fixture, snapshot, corpus, or testcase counts as evidence of quality, compatibility, support, or project progress.
Context
The current testing policy states that counts such as
26/26 fixtures passedor124 tests passedare not correctness claims, but still describes them as useful execution summaries. This leaves room for agents to persist volatile counts into README files, compatibility documents, evidence records, PR narratives, and other long-lived documentation.Absolute testcase inventory is not a meaningful capability contract. It becomes stale as the suite evolves and encourages quantity-based reporting instead of describing the semantic behavior, scenarios, boundaries, regressions, and evidence actually covered.
Execution totals may still exist transiently in test-runner output, CI logs, or generated CI summaries where they describe that specific run. They must not be promoted into durable documentation or capability claims.
Scope
docs/testing-policy.mdso test/fixture/corpus counts are prohibited in durable documentation and long-lived capability/status evidence.Non-goals
Acceptance Criteria
docs/testing-policy.mdno longer characterizes testcase counts as suitable durable summaries and explicitly bans agents/contributors from writing them into long-lived documentation or capability claims.Dependencies