Repository navigation
test(evidence): separate coverage index integrity from executed test evidence #4051
Description
Parent: #4045
Purpose
Source substring checks accept a named skipped test or command preserved in comments. They establish index integrity, not actual executed command coverage.
Owner and scope
All six test/integration/smoke-*-coverage.test.ts index guards; existing test/integration/command-coverage/evidence.ts and Android android-emulator-e2e/live-coverage-report.ts are the execution-evidence owners. iOS/tvOS extension only after the bounded decision.
Required behavior
Inventory all six source/index guards and rename their test names/messages to state the fact they check. Keep public-command completeness assertions. Reuse Android's existing executed-tier report/evidence owner for a bounded proof of declared/executed/skipped/missing evidence; evaluate extending it to iOS/tvOS or explicitly retain the distinction without extension. Do not design new receipt storage, telemetry or broad CI wiring.
API or evidence example
Reuse existing command-coverage evidence tiers and Android report. Source presence remains a declaration fact; executed claims require existing named-SHA runner evidence.
Observable completion
- Index-only guard names/messages accurately describe source/declaration integrity across the six examined platform files.
- A bounded existing-owner experiment distinguishes skipped/missing evidence from passed execution at a named SHA.
- A decision states whether to extend existing executed-tier evidence to iOS/tvOS, names source/artifacts and implementation scope, or explicitly retains current reporting. No invented receipt infrastructure.
Dependencies and sequencing
Blocked by: none.
Validation
Node smoke guard tests and existing provider-integration scenario inspection; test skipped/comment-only cases in isolated experiments. No blanket deletion of completeness tables.
Base audit SHA: 144bd59890277fd4717995588ad817597c9d6552. Re-read the issue and all Decision: comments before work. Read applicable AGENTS.md routes. Keep one owner boundary per PR. Independent review and serialized exact-head validation are coordinated by the umbrella orchestrator.
Activity
thymikee commentedon Oct 29, 2026
Decision: accept all three independent review findings at
391c44d6aa. Corrected in6291af80fd: shared helper wording refers to the appropriate runner and scenario completion, not only the Android/iOScommandEvidencefield; Android completeness title includes command/behavior evidence and completed scenarios. The retain decision now scopes each runner owner and requires the report to be retained with external source/run identity because no SHA is embedded. Lint, typecheck and full format pass; no tests run yet.The proposed fabricated-context test is unit evidence only. Parent will separately drive the existing harness with an actual successful host CLI command plus an unrun scenario, retain the existing-owner report and missing-evidence output at the named head, and state explicitly that this does not prove Android/iOS device execution. Source-comment/skip experiments remain separate. The issue stays open until those bounded proofs and selected validation run.
thymikee commentedon Oct 29, 2026
Current state: head
6291af80fd62a3064c39b78e3020253ce69f554c, 52 guard tests passed. Accepted review corrections scope execution claims to the appropriate runner and scenario completion. Parent isolated proof (fresh frozen install/build): tvOS index baseline7passed, skipped provider test7passed, comment-only close command7passed. Thus source/index results cannot establish execution. macOS index baseline7passed; removing the replayisaction yielded2failed /5passed with missing-command diagnostics, a positive detection control.Named-SHA existing-owner experiment: built CLI
help --jsonactually executed at this head through createLiveDeviceHarness/runScenario/runStep, displayed agent-device usage and exited0. verifyCommand recorded help and runScenario recordedbounded:helpcompletion. Selected-scenario completeness passed. A deliberately unrunbounded:unrunscenario requiring snapshot yielded missingCommands:[snapshot], missingScenarios:[bounded:unrun], and full completeness rejected. No commandEvidence was fabricated. The existing coverage-report.json, step-history.json, missing.json and external run-identity.json are retained under remaining-proof/b3-artifacts; the source SHA is external because the standard report has no SHA field. This is host CLI/harness evidence, not Android/iOS or live-device execution. No receipt system or CI change was introduced. All experiment checkouts removed and reviewed SHA/status unchanged. Exact-head affected validation/publication remain pending.Decision: retain the source/execution distinction; Android and iOS already share the same live-device coverage owner, and tvOS has no live-device lane. Do not add report infrastructure or extend a nonexistent lane.