[merefield]

Why

Make duplication measurable and prevent production duplication from growing in future PRs. Reports should tell both maintainers and Codex exactly where to investigate, rather than merely reporting an opaque percentage.

Changes

Independent CI gate

• Add Duplication ratchet as a discrete, top-level CI job, parallel with existing Go and web tests, with its own status, logs, summary and report artifact.
• Pin jscpd 5.4.2 in a separate development-only npm package. No application/runtime dependency, LLM calls, embeddings or billable model tokens.
• Gate clone count, duplicated lines and duplicated source tokens, independently, across four production scans: exact and consistently renamed/literal-normalised matches at 50 tokens/5 lines and 100 tokens/10 lines.
• Every gated metric must remain equal or improve. Percentages are informational; deleting unique code must not fail the gate merely by increasing a ratio.
• Measure test-source duplication separately, report-only, without discouraging useful regression fixtures. Exclude generated code, dependencies and embedded build output.

First-run baseline and automatic ratcheting

• Introduce a versioned pending-bootstrap baseline: the first measurement establishes the benchmark and passes the duplication gate without arbitrary initial limits. Tooling/scanner failures still fail.
• After all ordinary CI jobs pass on main, a separate main-only Update duplication baseline job commits the initial numerical baseline, then only tightens it on subsequent improvements. Equal measurements do not rewrite provenance.
• Main owns the baseline: compare future PRs using the checker, policy and baseline from their exact main-branch base commit, never a PR-supplied ceiling. Reject any PR baseline-file edit. Take the lower limit of that committed ceiling and fresh measurements of the same actual base, so a delayed baseline update cannot hide a regression.
• Ordinary application PRs use one authoritative comparison. Only changes to maintained duplication tooling, policy, package or lock files trigger extra validation: the candidate checker must reproduce main-policy metrics, limits and provenance using main's scanner. A no-op or mismeasuring candidate fails; documentation/test-only changes do not incur a second production scan.
• Deliberate rule/scanner upgrades require an explicit migration record with the exact old policy, target policy hash and reason. CI retains the old-rule gate and separately compares head/base under the new rules/scanner. Only successful main CI migrates the inherited baseline; the PR never edits it. Push checks use the pre-push main SHA rather than comparing main with itself.
• Keep PR/fork repository permissions read-only. Scope write permission to the post-success main writer, stage only the baseline, skip outdated main commits and never force-push. Preserve the measured baseline artifact if repository rules prevent automatic commits.

Actionable output

• Put human-readable actual/limit and PASS/FAIL/BOOTSTRAP tables in the GitHub job summary, plus a clearly marked test-source report-only table. Do not repeat these tables in CI logs; local runs still print them.
• Upload structured results with metric deltas, policy/commit provenance, clone paths, both line ranges, exact source fragments, native new-clone fingerprints and input manifests. Add bounded source annotations when a gated scan fails.
• Document installation, reproduction against the exact PR base, bootstrap/ratchet behaviour, exclusions, report consumption and repository setup in the contributor guide.

Initial production measurements

These are observed results, not manually imposed ceilings. The initial successful main run persists its own measurements.

Scan	Clones	Duplicated lines	Duplicated tokens
exact-50	34	316	2,466
renamed-50	152	1,576	11,451
exact-100	2	28	274
renamed-100	12	225	2,043

The full local base/head/test scan takes approximately 6–8 seconds; runner checkout, npm setup and artifact upload add CI overhead.

Validation

• Ten tooling tests pass, including real native-scanner fixtures exercising bootstrap, subsequent regression detection, new-clone source locations/fragments, persisted baseline creation, unchanged-baseline stability, improvement ratcheting and attempted PR baseline edits/reset rejection.
• Native fixtures also verify that baseline-enabled scans retain all known clones and full absolute totals, report paths resolve from the artifact root, unchanged tooling skips extra scans, honest checker changes match main, no-op/stale-output and mismeasuring checkers fail, and an explicit policy migration passes old/new checks before main writes the migrated baseline.
• Tests also cover independent metric limits, stale-ceiling protection, invalid baselines, tracked/generated/test source selection, symlink rejection, workflow-command escaping and report tables.
• Full check against current default branch passes; production results unchanged.
• CI-mode check confirms tables are written to the job summary rather than repeated in logs; expected regression fixtures do not emit real GitHub error annotations.
• npm audit --prefix tools/duplication --audit-level=high: zero known vulnerabilities.
• Tooling formatting, workflow YAML structure and git diff --check pass.
• make screenshots-check: all seven existing terminal screenshots verified unchanged for v0.27.2.

Repository setup after merge

Make Duplication ratchet a required branch check and require up-to-date branches (or merge-queue validation) if it should block merges. A workflow alone cannot enable branch protection. Actions and repository rules must permit the narrowly scoped main-only baseline writer. If the push is blocked, the writer fails and preserves a measured artifact for diagnosis; fix repository configuration and rerun it rather than editing the PR's baseline or granting broad bypass rights.

No TUI/web behaviour, release version or application dependency changes.

[Copilot]

🟡 Changes recommended

Active critical and moderate findings remain in the CI gate and duplication checker, including bypassable candidate tooling changes, an invalid migration path, broken report resolution, and incomplete baseline-derived metrics.

4 open issues

• PR changes to the checker and policy bypass the production gate · New
• Report path is resolved from the wrong directory · New
• Baseline-filtered scan undercounts candidate duplication metrics · New
• No passing path exists for policy or tool-version migrations · New

1 file not reviewed

• tools/duplication/package-lock.json: Generated file

What changed in this PR

Adds a standalone, pinned jscpd-based duplication ratchet for production sources, with report-only test scanning, baseline bootstrapping, automatic main-branch ratcheting, actionable reports, and contributor documentation.

Changes:

• Adds duplication policies, independent metrics, baseline validation, and ratcheting.
• Adds CI jobs for production gating, reporting, artifacts, annotations, and baseline updates.
• Adds tooling tests, isolated package metadata, exclusions, and local reproduction guidance.

This pull request requires changes.

File	Description
tools/​duplication/​README.md	Documents duplication checks, lifecycle, exclusions, reports, and repository setup.
tools/​duplication/​policy.json	Defines scanner settings, source cohorts, exclusions, and gated scans.
tools/​duplication/​package.json	Adds the isolated development-only jscpd package and scripts.
tools/​duplication/​package-lock.json	Pins the duplication tooling dependency tree.
tools/​duplication/​metrics.mjs	Implements metric extraction, baseline validation, limits, ratcheting, and output escaping.
tools/​duplication/​check.mjs	Selects sources, runs scans, generates reports, evaluates gates, and emits CI summaries.
tools/​duplication/​check.test.mjs	Tests scanning, baselines, regressions, source selection, reports, and security handling.
tools/​duplication/​baseline.json	Adds the initial pending-bootstrap baseline sentinel.
README.md	Adds contributor-facing duplication-check documentation.
.gitignore	Ignores local duplication dependencies and generated reports.
.github/​workflows/​ci.yml	Adds the standalone duplication gate and main-only baseline update workflow.

_{Review effort: Lite}