Skip to content

Warnings on procedures that pass but should probably change - #95

Merged
jadamcrain merged 3 commits into
mainfrom
procedure-warnings
Oct 15, 2026
Merged

jadamcrain merged 3 commits into
mainfrom
procedure-warnings

Conversation

@jadamcrain

Copy link
Copy Markdown
Member

A procedure can now record a warning on its result, next to its notes. A warning means the device passes the procedure but departs from the standard, or should probably change something. A note stays information, such as an answer the procedure allows as an alternative. Neither changes the outcome.

Report

  • Each warning appears under a passed procedure's title as "⚠ Warning: …", in amber, before any notes.
  • The counts at the top add "passed with warnings". "Passed" now counts only clean passes, so the counts still add up to the total.

CLI

  • A run ends by listing each procedure that passed with warnings, with its warnings, even when a later procedure fails.
  • Warnings don't change the exit code.

Results and library

  • results.properties stores warnings as warning.1, warning.2, … as it does notes.
  • The procedure's log shows each warning at warning level.
  • ProcedureResult has a warnings field (default empty), and RunSummary.passedWithWarnings lists the procedures that passed with warnings. A pass keeps its notes and warnings in the partial summary of a run the harness or plugin aborts after it.

Reclassified (3.1r1)
These notes become warnings, with their text unchanged:

  • 8.2.6: a select echoed with status 0 (complies with IEEE 1815-2012, not 1815-202x).
  • 8.14.2j/m and 8.15.2: reads of events with time rejected with IIN2.1 (complies with 1815-2012 only).
  • 8.24: analog output status kept only in 32 bits (not conformant at subset level 1 or 2).

All other notes stay notes. Where a description said such a result is "noted", it now says the procedure passes with a warning; that wording is the only change to -describe output.

Docs: automation.md, manual-testing.md and the 0.2.0 changelog entry.

Tests: results round-trip, the report's warning text, order, escaping and counts, the end-of-run list, a pass whose plugin then fails keeping its warnings, and each reclassified site recording a warning and not a note.

A procedure can now record a warning on its result, beside its notes: the
device passes the procedure but departs from the standard, or should
probably change something. A note stays information, such as an answer
the procedure allows as an alternative.

- ProcedureReporter.warning(text) logs the warning at warning level and
  keeps it with a pass; ProcedureResult.warnings carries it, and
  RunSummary.passedWithWarnings lists the procedures that passed with
  warnings. results.properties stores them as warning.1, warning.2, ...
- The report shows each warning under a passed procedure's title as
  "⚠ Warning: ...", before its notes, and counts the passes with warnings
  apart from the clean passes, so each procedure is in one count.
- A run ends by listing each procedure that passed with warnings, with its
  warnings, even when a later procedure fails.
- Three 3.1r1 notes become warnings: 8.2.6's echo with status 0 and the
  reads of events with time rejected with IIN2.1 (both compliant with
  IEEE 1815-2012 but not 1815-202x), and 8.24's analog output status kept
  only in 32 bits. Their step descriptions don't change.
@jadamcrain
jadamcrain added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit ec44373 Oct 15, 2026
4 checks passed
@jadamcrain
jadamcrain deleted the procedure-warnings branch October 15, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant