Skip to content

Ban test and fixture counts from durable documentation #11

Description

@Teakowa

Goal

Make WrightKit-wide policy explicitly prohibit durable documentation from using test, fixture, snapshot, corpus, or testcase counts as evidence of quality, compatibility, support, or project progress.

Context

The current testing policy states that counts such as 26/26 fixtures passed or 124 tests passed are not correctness claims, but still describes them as useful execution summaries. This leaves room for agents to persist volatile counts into README files, compatibility documents, evidence records, PR narratives, and other long-lived documentation.

Absolute testcase inventory is not a meaningful capability contract. It becomes stale as the suite evolves and encourages quantity-based reporting instead of describing the semantic behavior, scenarios, boundaries, regressions, and evidence actually covered.

Execution totals may still exist transiently in test-runner output, CI logs, or generated CI summaries where they describe that specific run. They must not be promoted into durable documentation or capability claims.

Scope

  • Strengthen docs/testing-policy.md so test/fixture/corpus counts are prohibited in durable documentation and long-lived capability/status evidence.
  • Make the agent-facing routing/invariants strong enough that documentation work does not introduce these counts even when the task is not primarily a testing task.
  • Require documentation and review evidence to describe semantic coverage, protected contracts, known gaps, and relevant scenarios instead of testcase quantity.
  • Define the narrow execution-time exception for ephemeral test-runner/CI output.
  • Establish a cleanup expectation for existing WrightKit documentation that contains volatile test/fixture/corpus inventory counts.

Non-goals

  • Do not suppress test-runner output or CI metadata that reports counts for a single execution.
  • Do not forbid numeric values that are themselves part of a product, protocol, semantic, performance, or resource contract.
  • Do not introduce a brittle repository-wide numeric-text linter that would reject legitimate numbers.
  • Do not change the actual composition of test suites or corpora solely to alter their size.

Acceptance Criteria

  • docs/testing-policy.md no longer characterizes testcase counts as suitable durable summaries and explicitly bans agents/contributors from writing them into long-lived documentation or capability claims.
  • Workspace agent guidance routes or states the prohibition clearly enough to apply during documentation work.
  • The policy distinguishes ephemeral execution metadata from durable documentation.
  • The preferred replacement is explicit: name semantic capabilities, scenarios, contracts, regressions, evidence states, and known gaps rather than testcase quantity.
  • Known existing occurrences in core WrightKit repositories are identified for owning-repository cleanup without duplicating authoritative policy text.

Dependencies

  • None for the policy change.
  • Owning repositories should perform their own documentation cleanup after this policy is accepted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions