Round-trip harness reports a vacuous pass when a spec carries no metadata

[nathanacurtis]

Problem

The round-trip harness reports a pass when it has nothing to check. Its configuration-mismatch comparison returns "no mismatch" when a spec carries no recorded conventions or settings at all, which is indistinguishable in the output from a spec whose conventions matched.

A spec authored by hand, or one written by a catalogue run that records its facts once in a separate file, has no per-spec conventions block. Round-tripping it therefore reads as a clean pass while the check that would have caught a convention drift never ran.

Potential solution(s)

• Distinguish "nothing to compare" from "compared and matched", and report the first as a skip rather than a pass
• Name which specs were skipped and why, so a run of all skips cannot read as a run of all passes
• Where the run's facts live in a sibling metadata file, load them before comparing rather than treating their absence as agreement

Acceptance criteria

• A spec with no recorded conventions or settings is reported as skipped, not passed
• The run names every spec it skipped and the reason
• A run whose every spec was skipped does not report overall success
• A spec whose facts are recoverable from a sibling run-metadata file is compared rather than skipped

Case data

• Workspace: specs-testing
• Territory: testing
• Size: s

Notes

This survives #669, which fixes the product's fallback for a spec carrying no metadata. The harness would still report a vacuous pass for any spec that legitimately carries none.

Related observation worth folding in: a spec copied away from the sibling file holding its run facts loses the naming convention a render needs, and render then fails to resolve instance elements for a reason unconnected to whatever is being tested. A harness that loads the sibling facts would not produce that false signal.

---

Implementation details are tracked internally.

[nathanacurtis]

Fixed in the round-trip diff harness.

The comparison now distinguishes three states instead of two: compared and matched, compared and differing, and not compared. A spec carrying no metadata.conventions or metadata.settings on either side falls into the third, where it used to fall into the first.

Before, a spec with nothing to compare produced:

**Identical.**

Now:

**Identical.** Excluded from that count: config not compared — see below.

## Config not compared

Neither spec records `metadata.conventions` or `metadata.settings`, so the
conventions this spec was generated under could not be checked against the
round-trip. …it means a convention drift would not have been caught here.

And the run says so at the top level, with the all-skipped case called out specifically:

⚠ Config NOT compared on 1/1 component(s): c — neither side records
metadata.conventions/settings, so a convention drift would not have been caught.
Skipped, not passed. EVERY component skipped: this run verified nothing about conventions.

• a spec with no recorded conventions or settings is reported as skipped, not passed
• the run names every spec it skipped and the reason
• a run whose every spec was skipped does not report overall success for that check
• configSkipped is carried in the machine-readable summary beside configMismatch, so a report builder can tell them apart

One criterion I did not implement: loading the facts from the sibling run-metadata file so the comparison can run rather than skip. That is the better end state, but it changes what the harness reads rather than how it reports, and the reporting fix is what stops a false signal today. Worth its own pass — the skip message now points at exactly where it would apply.

A real case confirming the distinction holds: comparing an authored spec against its captured round-trip reports a genuine config mismatch, because one side carries the facts and the other does not. Only when neither does is it a skip.