Repository navigation
Warnings on procedures that pass but should probably change - #95
Merged
Merged
Conversation
A procedure can now record a warning on its result, beside its notes: the device passes the procedure but departs from the standard, or should probably change something. A note stays information, such as an answer the procedure allows as an alternative. - ProcedureReporter.warning(text) logs the warning at warning level and keeps it with a pass; ProcedureResult.warnings carries it, and RunSummary.passedWithWarnings lists the procedures that passed with warnings. results.properties stores them as warning.1, warning.2, ... - The report shows each warning under a passed procedure's title as "⚠ Warning: ...", before its notes, and counts the passes with warnings apart from the clean passes, so each procedure is in one count. - A run ends by listing each procedure that passed with warnings, with its warnings, even when a later procedure fails. - Three 3.1r1 notes become warnings: 8.2.6's echo with status 0 and the reads of events with time rejected with IIN2.1 (both compliant with IEEE 1815-2012 but not 1815-202x), and 8.24's analog output status kept only in 32 bits. Their step descriptions don't change.
…t's pill like the others
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A procedure can now record a warning on its result, next to its notes. A warning means the device passes the procedure but departs from the standard, or should probably change something. A note stays information, such as an answer the procedure allows as an alternative. Neither changes the outcome.
Report
CLI
Results and library
warning.1,warning.2, … as it does notes.ProcedureResulthas awarningsfield (default empty), andRunSummary.passedWithWarningslists the procedures that passed with warnings. A pass keeps its notes and warnings in the partial summary of a run the harness or plugin aborts after it.Reclassified (3.1r1)
These notes become warnings, with their text unchanged:
All other notes stay notes. Where a description said such a result is "noted", it now says the procedure passes with a warning; that wording is the only change to
-describeoutput.Docs: automation.md, manual-testing.md and the 0.2.0 changelog entry.
Tests: results round-trip, the report's warning text, order, escaping and counts, the end-of-run list, a pass whose plugin then fails keeping its warnings, and each reclassified site recording a warning and not a note.