diff --git a/CHANGELOG.md b/CHANGELOG.md index ea4bbb5..c46ec67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ - **DNP3 IED Certification Procedure 3.1 rev 1**, subset levels 1 to 3, alongside 2.7. Choose the revision with `-rev`; `-revs` lists them. - **Serial and UDP**, as well as TCP, including 3.1 rev 1's network, multi-drop and collision avoidance tests. - **Exclusions:** leave a procedure out of a run with a reason, e.g. for a known gap in the device. The report shows it as not run, apart from skipped procedures, with its reason. -- **Reports** say why each procedure was skipped, and name the revision and subset level. +- **Reports** say why each procedure was skipped, flag the procedures that pass with warnings, where the device should probably change something, and name the revision and subset level. - The runnable jar and the CLI zip list the licenses of the dependencies they bundle. ### Upgrading from 0.1.0 @@ -14,7 +14,7 @@ - `outstation.subset.level` is required in the configuration. - Results are kept by revision and level, e.g. `results/2.7/L2/`. - Some configuration keys were renamed or removed; an old key is reported as an unknown option. `-write` writes a file with every current option. -- Library: `Harness` takes the revision to run, e.g. `new Harness(options, plugin, Revisions.v2_7)`, and so does `Harness.writeReport(options, revision)`. `ProcedureResult` has a third field, `notes`. +- Library: `Harness` takes the revision to run, e.g. `new Harness(options, plugin, Revisions.v2_7)`, and so does `Harness.writeReport(options, revision)`. `ProcedureResult` has two more fields, `notes` and `warnings`. - Integration plugins: `IntegrationPlugin`'s per-type methods take the kind of point, and runs can report a procedure as `ProcedureOutcome.Excluded`. See [Automating the tests](https://github.com/stepfunc/dnp3c/blob/0.2.0/docs/automation.md). ## 0.1.0 diff --git a/docs/automation.md b/docs/automation.md index 3c72f64..a57dff9 100644 --- a/docs/automation.md +++ b/docs/automation.md @@ -72,6 +72,14 @@ if (!summary.succeeded || summary.exclusions.nonEmpty || !reported) sys.exit(1) An exclusion replaces a procedure's earlier result only when a run reaches it, as running it does: run the procedure, its section or `all`. `writeReport` never changes results. +### Warnings and notes + +A procedure that passes can say more about the pass. A warning means the device passes the procedure but departs from the standard, or should probably change something: e.g. it complies with IEEE 1815-2012 but not 1815-202x. A note is information, such as an answer the procedure allows as an alternative. A passing result has them in `ProcedureResult.warnings` and `ProcedureResult.notes`, and neither changes its outcome. `summary.passedWithWarnings` lists the procedures that passed with warnings, in order. For a CI job that must fail on them too: + +```scala +if (!summary.succeeded || summary.passedWithWarnings.nonEmpty || !reported) sys.exit(1) +``` + The plugin runs in the same JVM as dnp3c. It can control a device elsewhere, e.g. through a test fixture, an API or a simulator, or it can host the device itself. A software outstation can be started in `startProcedure` and stopped in `endProcedure`, so each procedure starts from a known state. ### When things fail diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 186e84c..ea2d8dd 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -104,7 +104,7 @@ Some questions come after a procedure's steps, where stopping for your answer wo If you miss a time limit, the procedure fails with a message saying so; rerun it. A limit that's too short to work to by hand is usually a timeout on the DUT that you can lengthen, along with its option here. -A run stops at the first procedure that fails, and the procedures after it don't run. The exit code is 0 if every procedure passed, was skipped or was excluded and the report was written, and nonzero otherwise. +A run stops at the first procedure that fails, and the procedures after it don't run. It ends by listing each procedure that passed with warnings, with its warnings (see [Results](#results)). The exit code is 0 if every procedure passed, was skipped or was excluded and the report was written, and nonzero otherwise. ## Results @@ -118,13 +118,13 @@ results/ report.css 8.6.5.4/ log.html every step, frame and check, with the outcome - results.properties the result (pass, fail, skip or excluded), the revision and subset level, and why it failed, was skipped or was excluded + results.properties the result (pass, fail, skip or excluded), the revision and subset level, and why it failed, was skipped or was excluded, or a pass's warnings and notes dump.pcap the packet capture, if capture is on ``` A procedure that won't run, because the DUT lacks a feature it needs, it doesn't apply at the subset level, or it isn't implemented, is recorded as skipped without connecting to the DUT. -The report is rebuilt after every run, including runs that fail. It names the revision and the subset level, and shows every procedure: the ones you've run with their result (and, for one that was skipped or excluded, why, or for one that passed, anything it noted, such as a device that complies with IEEE 1815-2012 but not 1815-202x), and the rest as not run, with no saved result. Excluded procedures are shown as "NOT RUN — EXCLUDED", and if there are any, the report says at the top that conformance hasn't been demonstrated for them. To rebuild it without running anything, use `dnp3c -rev 2.7 -config device.ini -report`. +The report is rebuilt after every run, including runs that fail. It names the revision and the subset level, and shows every procedure: the ones you've run with their result (and, for one that was skipped or excluded, why, or for one that passed, its warnings and notes), and the rest as not run, with no saved result. A warning, shown under the procedure's title as "⚠ Warning: …", before any notes, means the DUT passes the procedure but departs from the standard, or should probably change something: e.g. it complies with IEEE 1815-2012 but not 1815-202x. A note, shown as "Note: …", is information, such as an answer the procedure allows as an alternative. The counts at the top count a pass with warnings as "passed with warnings", apart from the clean passes, so each procedure is in exactly one count. Excluded procedures are shown as "NOT RUN — EXCLUDED", and if there are any, the report says at the top that conformance hasn't been demonstrated for them. To rebuild it without running anything, use `dnp3c -rev 2.7 -config device.ini -report`. The report counts only results saved under its revision and level. It lists anything else it finds at the top and leaves it out: a result saved under another revision or level, or a procedure whose last run didn't finish (it has a log but no result). diff --git a/src/main/resources/report.css b/src/main/resources/report.css index 16f5ab6..549b41c 100644 --- a/src/main/resources/report.css +++ b/src/main/resources/report.css @@ -12,6 +12,8 @@ --sf-wash: #f5f7f8; --pass: #1f7a35; --pass-bg: #e3f6e1; + --warning: #a34f00; /* amber, apart from the skips' */ + --warning-bg: #fff1e0; --fail: #b02a37; --fail-bg: #fbe4e6; --skip: #8a6400; @@ -100,6 +102,7 @@ h1 { } .total.pass { color: var(--pass); } +.total.warning { color: var(--warning); } .total.fail { color: var(--fail); } .total.skip { color: var(--skip); } .total.excluded { color: var(--excluded); } @@ -252,6 +255,13 @@ span.exclusion { font-weight: 600; } +/* what a passing procedure warned the device should probably change, under its title */ +span.result-warning { + color: var(--warning); + font-size: 12px; + font-weight: 600; +} + /* what a procedure noted about a pass, under its title */ span.note { color: var(--pass); diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala index 4a7ed85..1bbe46d 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala @@ -190,21 +190,7 @@ object Main { var ran = false var reported = false try { - ran = - try { - val summary = harness.run(id) - if (!summary.succeeded) Colors.error.printLine("Abort due to test failure") - exclusionWarning(summary).foreach(Colors.warn.printLine) - summary.succeeded - } catch { - case ex: HarnessExecutionException => - Colors.error.printLine(ErrorText.of(ex)) - exclusionWarning(ex.partial).foreach(Colors.warn.printLine) - false - case NonFatal(ex) => - Colors.error.printLine(ErrorText.of(ex)) - false - } + ran = concludeRun(harness.run(id)) } finally { // the report covers every result so far, however the run ended reported = writeReport(harness.writeReport()) @@ -215,6 +201,37 @@ object Main { } } + // runs, then tells how the run ended, however it ended: whether it failed, the procedures that passed with warnings, + // and how many it excluded; whether it succeeded + private[conformance] def concludeRun(run: => RunSummary): Boolean = { + def tell(summary: RunSummary): Unit = { + warningLines(summary).foreach(Colors.warn.printLine) + exclusionWarning(summary).foreach(Colors.warn.printLine) + } + try { + val summary = run + if (!summary.succeeded) Colors.error.printLine("Abort due to test failure") + tell(summary) + summary.succeeded + } catch { + case ex: HarnessExecutionException => + Colors.error.printLine(ErrorText.of(ex)) + tell(ex.partial) + false + case NonFatal(ex) => + Colors.error.printLine(ErrorText.of(ex)) + false + } + } + + // how a run's end lists the procedures that passed with warnings, each warning on a line of its own + private[conformance] def warningLines(summary: RunSummary): List[String] = summary.passedWithWarnings match { + case Nil => Nil + case warned => + val count = if (warned.size == 1) "1 procedure" else s"${warned.size} procedures" + s"$count passed with warnings:" :: warned.flatMap(r => r.warnings.map(w => s" ${r.id}: $w")) + } + // how a run's end tells of the procedures it excluded, which it didn't run private[conformance] def exclusionWarning(summary: RunSummary): Option[String] = Option.when(summary.exclusions.nonEmpty)(Report.exclusionWarning(summary.exclusions.size)) diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala index 2a0478c..7d8cf1e 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala @@ -23,10 +23,15 @@ object ProcedureOutcome { /** * A procedure's outcome * - * @param notes what the procedure noted about a pass, e.g. that the device complies with IEEE 1815-2012 but not - * 1815-202x; empty otherwise + * @param notes what the procedure noted about a pass, e.g. an answer it allows as an alternative; empty otherwise + * @param warnings what the device should probably change, though the procedure passed it, e.g. that it complies with + * IEEE 1815-2012 but not 1815-202x; empty otherwise */ -case class ProcedureResult(id: String, outcome: ProcedureOutcome, notes: List[String] = Nil) +case class ProcedureResult( + id: String, + outcome: ProcedureOutcome, + notes: List[String] = Nil, + warnings: List[String] = Nil) /** * The procedures a run reached, in order, and the ones it didn't because it stopped at a failure @@ -41,6 +46,10 @@ case class RunSummary(results: List[ProcedureResult], unreached: List[String]) { /** the procedures the configuration excluded, in order, which the device hasn't been shown to conform to */ def exclusions: List[ProcedureResult] = results.filter(_.outcome.isInstanceOf[ProcedureOutcome.Excluded]) + + /** the procedures that passed with warnings, in order: what the device should probably change */ + def passedWithWarnings: List[ProcedureResult] = + results.filter(r => r.outcome == ProcedureOutcome.Passed && r.warnings.nonEmpty) } /** diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TimeSyncOptions.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TimeSyncOptions.scala index bbaa910..e6aa23c 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TimeSyncOptions.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TimeSyncOptions.scala @@ -45,8 +45,8 @@ object TimeSyncOptions extends OptionGroup { s"${input}ChangeWithTimeRejected", false, s"For a DUT without $name Change with Time events: does it answer a read of g${eventGroup}v2 with IIN2.1, as " + - "IEEE 1815-2012 allows? 3.1r1 then notes it as compliant with 1815-2012 but not 1815-202x, and skips its " + - "other with-time tests" + "IEEE 1815-2012 allows? 3.1r1 then passes it with a warning that it complies with 1815-2012 but not 1815-202x, " + + "and skips its other with-time tests" ) // 3.1r1: likewise for events with relative time val withRelativeTimeRejected: BooleanOption = OptionTypes.boolean( @@ -54,8 +54,8 @@ object TimeSyncOptions extends OptionGroup { s"${input}ChangeWithRelativeTimeRejected", false, s"For a DUT without $name Change with Relative Time events: does it answer a read of g${eventGroup}v3 with " + - "IIN2.1, as IEEE 1815-2012 allows? 3.1r1 then notes it as compliant with 1815-2012 but not 1815-202x, and " + - s"skips $relativeTimeSkipped" + "IIN2.1, as IEEE 1815-2012 allows? 3.1r1 then passes it with a warning that it complies with 1815-2012 but not " + + s"1815-202x, and skips $relativeTimeSkipped" ) } diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/api/package.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/api/package.scala index 9792bac..b0c18a4 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/api/package.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/api/package.scala @@ -42,6 +42,14 @@ package object api { driver.reporter.note(text) } + /** + * a step that records a warning, `text`, on the procedure's result, which the report shows even when it passes: what + * the device should probably change, though the procedure passes it + */ + def warning(text: String): TestStep = TestStep(List(Comment(s"Record a warning on the result: $text"))) { driver => + driver.reporter.warning(text) + } + def repeat(count: Int)(step: => TestStep): TestStep = new TestStep { assert(count >= 0) diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/application/SavedResult.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/application/SavedResult.scala index 96da8f9..7fd3f3e 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/application/SavedResult.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/application/SavedResult.scala @@ -17,13 +17,16 @@ import scala.util.control.NonFatal * * @param revision the revision it was run under; None for a result saved before results recorded one (0.1.0) * @param level the subset level it was run at; None for a result saved before results recorded one - * @param notes what the procedure noted about a pass, e.g. that it's to IEEE 1815-2012 but not 1815-202x + * @param notes what the procedure noted about a pass, e.g. an answer it allows as an alternative + * @param warnings what the device should probably change, though the procedure passed it, e.g. that it's to IEEE + * 1815-2012 but not 1815-202x */ case class SavedResult( outcome: ProcedureOutcome, revision: Option[String], level: Option[Int], - notes: List[String] = Nil) + notes: List[String] = Nil, + warnings: List[String] = Nil) object SavedResult { @@ -38,6 +41,14 @@ object SavedResult { private val levelKey = "subsetLevel" private val reasonKey = "reason" private val noteKey = "note" + private val warningKey = "warning" + + // a list saved as key.1, key.2, ... + private def list(props: Properties, key: String): List[String] = + Iterator.from(1).map(i => props.getProperty(s"$key.$i")).takeWhile(_ != null).toList + + private def setList(props: Properties, key: String, values: List[String]): Unit = + values.zipWithIndex.foreach((v, i) => props.setProperty(s"$key.${i + 1}", v)) def read(file: File): SavedResult = { val stream = new FileInputStream(file) @@ -58,7 +69,8 @@ object SavedResult { outcome, Option(props.getProperty(revisionKey)), Option(props.getProperty(levelKey)).flatMap(_.toIntOption), - Iterator.from(1).map(i => props.getProperty(s"$noteKey.$i")).takeWhile(_ != null).toList + list(props, noteKey), + list(props, warningKey) ) } finally { stream.close() @@ -87,7 +99,8 @@ object SavedResult { } saved.revision.foreach(props.setProperty(revisionKey, _)) saved.level.foreach(l => props.setProperty(levelKey, l.toString)) - saved.notes.zipWithIndex.foreach((n, i) => props.setProperty(s"$noteKey.${i + 1}", n)) + setList(props, noteKey, saved.notes) + setList(props, warningKey, saved.warnings) props.store(stream, "test result") } Files.move(temporary, file, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE) diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/application/TestRunner.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/application/TestRunner.scala index 283109e..eed25be 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/application/TestRunner.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/application/TestRunner.scala @@ -97,9 +97,9 @@ object TestRunner { def failed: Boolean = results.exists(_.outcome.isInstanceOf[ProcedureOutcome.Failed]) def unreachedAfter(count: Int): List[String] = procedures.drop(count).map(_.id) - def aborted(p: TestProcedure, outcome: Option[ProcedureOutcome], error: Throwable): HarnessExecutionException = { - val partial = results.toList ++ outcome.map(ProcedureResult(p.id, _)) - new HarnessExecutionException(p.id, RunSummary(partial, unreachedAfter(results.size + 1)), error) + def aborted(p: TestProcedure, ex: ProcedureAborted): HarnessExecutionException = { + val partial = results.toList ++ ex.outcome.map(ProcedureResult(p.id, _, ex.notes, ex.warnings)) + new HarnessExecutionException(p.id, RunSummary(partial, unreachedAfter(results.size + 1)), ex.error) } elements.iterator.takeWhile(_ => !failed).foreach { @@ -109,16 +109,22 @@ object TestRunner { try { results += runProcedure(revision, level, p, options, integrationPlugin, environment) } catch { - case ex: ProcedureAborted => throw aborted(p, ex.outcome, ex.error) - case NonFatal(ex) => throw aborted(p, None, ex) + case ex: ProcedureAborted => throw aborted(p, ex) + case NonFatal(ex) => throw aborted(p, new ProcedureAborted(None, ex)) } } RunSummary(results.toList, unreachedAfter(results.size)) } - // the harness or the plugin failed, after the procedure reached the outcome, if any - private class ProcedureAborted(val outcome: Option[ProcedureOutcome], val error: Throwable) extends Exception(error) + // the harness or the plugin failed, after the procedure reached the outcome, if any, with the notes and warnings a + // pass keeps + private class ProcedureAborted( + val outcome: Option[ProcedureOutcome], + val error: Throwable, + val notes: List[String] = Nil, + val warnings: List[String] = Nil) + extends Exception(error) // why a procedure is skipped: it isn't implemented, it doesn't apply at the subset level, or the device lacks a // feature it needs @@ -142,11 +148,8 @@ object TestRunner { val resultsFile = path.resolve(ResultFiles.result) val title = s"${procedure.id} ${procedure.description} (revision ${revision.id}, subset level $level)" - // a pass keeps the notes the procedure made about it - def save(result: ProcedureOutcome, notes: List[String] = Nil): Unit = { - val kept = if (result == ProcedureOutcome.Passed) notes else Nil - SavedResult.write(SavedResult(result, Some(revision.id), Some(level), kept), resultsFile) - } + def save(result: ProcedureOutcome, notes: List[String] = Nil, warnings: List[String] = Nil): Unit = + SavedResult.write(SavedResult(result, Some(revision.id), Some(level), notes, warnings), resultsFile) // decided before anything is opened, so a procedure that's skipped or excluded never touches the device or the // plugin @@ -204,17 +207,20 @@ object TestRunner { throw cleanUp(cleanup, Some(ex)).getOrElse(ex) } + // a pass keeps the notes and warnings the procedure made about it, even if what follows it fails + val passed = if (outcome.contains(ProcedureOutcome.Passed)) reporter.toList else Nil + val (notes, warnings) = (passed.flatMap(_.notes), passed.flatMap(_.warnings)) + cleanUp(cleanup, failure).foreach { - case NonFatal(ex) => throw new ProcedureAborted(outcome, ex) + case NonFatal(ex) => throw new ProcedureAborted(outcome, ex, notes, warnings) case ex => throw ex } // the procedure, and everything around it, completed, so the outcome is known val result = outcome.get - val notes = if (result == ProcedureOutcome.Passed) reporter.toList.flatMap(_.notes) else Nil - try save(result, notes) - catch { case NonFatal(ex) => throw new ProcedureAborted(outcome, ex) } - ProcedureResult(procedure.id, result, notes) + try save(result, notes, warnings) + catch { case NonFatal(ex) => throw new ProcedureAborted(outcome, ex, notes, warnings) } + ProcedureResult(procedure.id, result, notes, warnings) } // records a procedure that's skipped or excluded: its log says why, and nothing is opened diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Classes.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Classes.scala index 69847f8..d0db26d 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Classes.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Classes.scala @@ -12,6 +12,7 @@ object Classes { def level(i: Int) = Clazz(s"level$i") val pass = Clazz("pass") + val warning = Clazz("warning") val fail = Clazz("fail") val skip = Clazz("skip") val excluded = Clazz("excluded") diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Report.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Report.scala index d2757f5..ccb1e47 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Report.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/application/report/Report.scala @@ -63,14 +63,15 @@ case class Report(revision: Revision, level: Int, sections: Iterable[Report.Sect case None => Classes.noResult } - // why the procedure was skipped or excluded, or what it noted about a pass, under its title + // why the procedure was skipped or excluded, or what it warned of and noted about a pass, under its title def skipReason: String = p.outcome match { case Some(ProcedureOutcome.Skipped(reason)) if reason.nonEmpty => "
Skipped: " + Tags.escape(reason) + "" case Some(ProcedureOutcome.Excluded(reason)) => "
Excluded: " + Tags.escape(reason) + "" case Some(ProcedureOutcome.Passed) => - p.notes.map(n => "
Note: " + Tags.escape(n) + "").mkString + p.warnings.map(w => "
⚠ Warning: " + Tags.escape(w) + "").mkString + + p.notes.map(n => "
Note: " + Tags.escape(n) + "").mkString case _ => "" } @@ -98,11 +99,15 @@ case class Report(revision: Revision, level: Int, sections: Iterable[Report.Sect private def excluded: Int = implemented.count(_.outcome.exists(_.isInstanceOf[ProcedureOutcome.Excluded])) - // e.g. "146 passed", counting implemented procedures by result + // e.g. "146 passed", counting implemented procedures by result: each in one count, so they add up to the total, with + // a pass that has warnings counted apart from a clean pass private def totals: Iterable[(Clazz, String)] = { def count(counted: Option[ProcedureOutcome] => Boolean) = implemented.count(p => counted(p.outcome)) + def passed(withWarnings: Boolean) = + implemented.count(p => p.outcome.contains(ProcedureOutcome.Passed) && p.warnings.nonEmpty == withWarnings) List( - (Classes.pass, s"${count(_.contains(ProcedureOutcome.Passed))} passed"), + (Classes.pass, s"${passed(withWarnings = false)} passed"), + (Classes.warning, s"${passed(withWarnings = true)} passed with warnings"), (Classes.fail, s"${count(_.exists(_.isInstanceOf[ProcedureOutcome.Failed]))} failed"), (Classes.skip, s"${count(_.exists(_.isInstanceOf[ProcedureOutcome.Skipped]))} skipped"), (Classes.excluded, s"$excluded excluded"), @@ -169,7 +174,8 @@ object Report { description: String, outcome: Option[ProcedureOutcome], hasPcaps: Boolean, - notes: List[String] = Nil) + notes: List[String] = Nil, + warnings: List[String] = Nil) extends Element /** @param children its sections and procedures, in the tree's order */ @@ -218,7 +224,9 @@ object Report { procedure.description, counted.map(_.outcome), hasPcaps, - counted.toList.flatMap(_.notes)) + counted.toList.flatMap(_.notes), + counted.toList.flatMap(_.warnings) + ) } val sections = revision.sections.map(convert) diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatus.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatus.scala index 50b31cf..9893edf 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatus.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatus.scala @@ -15,7 +15,7 @@ import io.stepfunc.dnp3c.dnp3.app.objects.CommandStatus /** * 3.1r1's 8.2.6, No Control When Status Code is Non-Zero (TB2014-002): a select to binary output 0 whose status field * is 126 is echoed with a non-zero status. One echoed with status 0 complies with IEEE 1815-2012, though not with - * 202x, so it passes, and the result notes it. + * 202x, so it passes, with a warning on the result. */ object ControlStatus { @@ -27,8 +27,8 @@ object ControlStatus { private[v31r1] def echoesWithAnyStatus(select: ObjectHeader, onZero: => Unit): MessageValidation = MessageValidation.check( "Verify that the response echoes the select, but for its status code", - "Verify that the status code isn't zero; a DUT that answers with zero complies with IEEE 1815-2012, which is " + - "noted, but not with IEEE 1815-202x", + "Verify that the status code isn't zero; a DUT that answers with zero passes with a warning: it complies with " + + "IEEE 1815-2012, but not with IEEE 1815-202x", "Verify that the IIN field has no error bits but IIN2.1 and IIN2.2, which are optional" ) { message => message.headers match { @@ -55,7 +55,7 @@ object ControlStatus { var zero = false val checks = expectSingleResponse(Sequence.zero, check { zero = true }).run(driver) if (zero) - driver.reporter.note( + driver.reporter.warning( "the DUT echoes a select whose status field isn't zero with status code 0: it complies with IEEE " + "1815-2012, but not with IEEE 1815-202x") checks diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/InputEvents.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/InputEvents.scala index b16f90e..ee5aa88 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/InputEvents.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/InputEvents.scala @@ -119,17 +119,18 @@ object InputEvents { } // a read of events with time, absolute or relative, answered with IIN2.1, as a device configured to reject it may: - // that passes, noted as compliant with 1815-2012 but not 1815-202x; the steps ask for IIN2.1, and other error bits - // aren't ruled out + // that passes, with a warning that it's compliant with 1815-2012 but not 1815-202x; the steps ask for IIN2.1, and + // other error bits aren't ruled out private[v31r1] def rejectedWithIIN21(seq: Sequence, time: String): TestStep = expectSingleResponse(seq, Validations.isEmpty && Validations.containing(IIN.OBJECT_UNKNOWN)) + - noting(s"the DUT answers a read of events with $time with IIN2.1: it complies with IEEE 1815-2012, but not " + + warning(s"the DUT answers a read of events with $time with IIN2.1: it complies with IEEE 1815-2012, but not " + "with IEEE 1815-202x") /** * With time, qualifier 0x06 (8.14.2j). A device without events with time may answer with a Null Error Response - * with IIN2.1, if it's configured to: that passes, noted as compliant with 1815-2012 but not 1815-202x, and its other - * with-time tests are skipped. Otherwise the events, with time or not, are reported as the rules require. + * with IIN2.1, if it's configured to: that passes, with a warning that it's compliant with 1815-2012 but not + * 1815-202x, and its other with-time tests are skipped. Otherwise the events, with time or not, are reported as the + * rules require. */ final class AllQualifierWithTime(input: InputType) extends EventTest(input) { import input._ @@ -220,8 +221,8 @@ object InputEvents { /** * With relative time, qualifier 0x06 (8.14.2m), in three phases: after a restart, while the clock is * unsynchronized; after a time write, synchronized; and after the synchronization expires. A device without events - * with relative time may answer with IIN2.1, if it's configured to: that passes, noted as compliant with 1815-2012 - * but not 1815-202x, and the rest of 8.14 and 8.15 are skipped. + * with relative time may answer with IIN2.1, if it's configured to: that passes, with a warning that it's compliant + * with 1815-2012 but not 1815-202x, and the rest of 8.14 and 8.15 are skipped. */ final class AllQualifierWithRelativeTime(input: InputType) extends EventTest(input) { import input._ diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariations.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariations.scala index c7a2ec9..42aa888 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariations.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariations.scala @@ -105,7 +105,7 @@ object StaticVariations { var first: Option[Int] = None var expectedFlags: Option[Int] = None var abnormal = false - // an analog output status point that comes back in 32 bits for a 16-bit request, which step 5 has noted + // an analog output status point that comes back in 32 bits for a 16-bit request, which step 5 has warned of var only32Reported: Set[Int] = Set.empty // the flags the first point came back with once one was set abnormal, which the result notes var reported: Option[Int] = None @@ -236,7 +236,7 @@ object StaticVariations { val notedOnly32 = only32Reported val checks = expect.run(driver) (only32Reported -- notedOnly32).toList.sorted.foreach(i => - driver.reporter.note( + driver.reporter.warning( s"analog output status point $i comes back only in 32 bits: the DUT isn't conformant at subset " + "level 1 or 2")) if (!noted) diff --git a/src/main/scala/io/stepfunc/dnp3c/dnp3/TestReporter.scala b/src/main/scala/io/stepfunc/dnp3c/dnp3/TestReporter.scala index e029319..0e8a07a 100644 --- a/src/main/scala/io/stepfunc/dnp3c/dnp3/TestReporter.scala +++ b/src/main/scala/io/stepfunc/dnp3c/dnp3/TestReporter.scala @@ -134,10 +134,11 @@ class TestReporter(val logger: Logger) { final class ProcedureReporter(logger: Logger, val plugin: IntegrationPlugin) extends TestReporter(logger) { private val noted = scala.collection.mutable.ListBuffer.empty[String] + private val warned = scala.collection.mutable.ListBuffer.empty[String] /** - * Note something about the result that a reader of the report should see, e.g. that a passing device complies with - * IEEE 1815-2012 but not 1815-202x. The note goes in the log now, and with the result if the procedure passes. + * Note something about the result that a reader of the report should see, e.g. an answer the procedure allows as an + * alternative. The note goes in the log now, and with the result if the procedure passes. */ def note(text: String): Unit = { noted += text @@ -145,4 +146,16 @@ final class ProcedureReporter(logger: Logger, val plugin: IntegrationPlugin) ext } def notes: List[String] = noted.toList + + /** + * Warn of something about a pass that the device should probably change, though the procedure passes it, e.g. that + * it complies with IEEE 1815-2012 but not 1815-202x. The warning goes in the log now, and with the result if the + * procedure passes. + */ + def warning(text: String): Unit = { + warned += text + warn("Warning on the result: " + text) + } + + def warnings: List[String] = warned.toList } diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/HarnessTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/HarnessTest.scala index 3f910b5..ea2fc5e 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/HarnessTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/HarnessTest.scala @@ -95,6 +95,61 @@ class HarnessTest extends AnyFunSuite with Matchers { "Note: compliant with 1815-2012 only") } + test("the report shows a pass's warnings before its notes, and counts it apart from the clean passes") { + val directory = output + val folder = directory.resolve("2.7").resolve("L2") + def result(id: String, saved: SavedResult): Unit = { + Files.createDirectories(folder.resolve(id)) + SavedResult.write(saved, folder.resolve(id).resolve("results.properties")) + } + def plain(outcome: ProcedureOutcome) = SavedResult(outcome, Some("2.7"), Some(2)) + result("8.1.2", plain(ProcedureOutcome.Passed)) + result( + "8.3.2", + SavedResult( + ProcedureOutcome.Passed, + Some("2.7"), + Some(2), + notes = List("an allowed answer"), + warnings = List("complies with <1815-2012> & not 202x", "a second warning") + ) + ) + result("8.4.1.2.1", plain(ProcedureOutcome.Skipped("disabled"))) + result("8.4.1.2.2", plain(ProcedureOutcome.Failed("no response"))) + result("6.1.2", plain(ProcedureOutcome.Excluded("a known gap"))) + val report = Files.readString(Harness.writeReport(options(directory), revision)) + + val warning = report.indexOf("⚠ Warning: complies with <1815-2012> & not 202x") + warning should be >= 0 + report should not include "<1815-2012>" + report.indexOf("⚠ Warning: a second warning") should be > warning + report.indexOf("Note: an allowed answer") should be > report.indexOf("⚠ Warning: a second warning") + report should include("""⚠ Warning: """) + + // each implemented procedure in exactly one count + val totals = """\s*(\d+) ([^<]*?)\s*""".r + .findAllMatchIn(report).map(m => (m.group(1), m.group(3)) -> m.group(2).toInt).toList + totals.map(_._1) should equal(List( + ("pass", "passed"), + ("warning", "passed with warnings"), + ("fail", "failed"), + ("skip", "skipped"), + ("excluded", "excluded"), + ("no-result", "with no saved result") + )) + totals.map(_._2).take(5) should equal(List(1, 1, 1, 1, 1)) + totals.map(_._2).sum should equal(revision.flatMapProcedures(p => Option.when(!p.incomplete)(p)).size) + } + + test("the report counts no passes with warnings when none has any") { + val directory = output + save(directory.resolve("2.7").resolve("L2").resolve("8.1.2"), Some("2.7")) + val report = Files.readString(Harness.writeReport(options(directory), revision)) + report should include regex """\s*1 passed\s*""" + report should include regex """\s*0 passed with warnings\s*""" + report should not include "⚠" + } + test("the report shows each exclusion's reason, counts exclusions apart from skips, and warns of them") { val directory = output val folder = directory.resolve("2.7").resolve("L2") diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala index 8f0e6b4..314516d 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala @@ -7,7 +7,10 @@ import java.io.{ ByteArrayOutputStream, File, PrintStream } import java.nio.charset.StandardCharsets import java.nio.file.{ Files, Path } -import io.stepfunc.dnp3c.conformance.api.{ BooleanOption, OptionGroups, Options, ProcedureOutcome, TestProcedure } +import io.stepfunc.dnp3c.conformance.api.{ + BooleanOption, HarnessExecutionException, OptionGroups, Options, ProcedureOutcome, ProcedureResult, RunSummary, + TestProcedure +} import io.stepfunc.dnp3c.conformance.api.options.{ OutstationOptions, SupportOptions, TestOptions } import io.stepfunc.dnp3c.conformance.application.SavedResult import io.stepfunc.dnp3c.conformance.application.Console @@ -285,4 +288,52 @@ class MainTest extends AnyFunSuite with Matchers { aborted should not equal 0 told.indexOf(oneExcluded) should be > told.indexOf("can't start") } + + private val warned = ProcedureResult( + "8.2.6", + ProcedureOutcome.Passed, + notes = List("an allowed answer"), + warnings = List("complies with IEEE 1815-2012 only")) + + test("a run's end lists each procedure that passed with warnings, with its warnings, even after a failure") { + val summary = RunSummary( + List( + ProcedureResult("8.1.2", ProcedureOutcome.Passed, notes = List("only a note")), + warned, + ProcedureResult("8.24.4", ProcedureOutcome.Passed, warnings = List("point 0 in 32 bits", "point 1 in 32 bits")), + ProcedureResult("8.3.2", ProcedureOutcome.Failed("no response")) + ), + List("8.4.1") + ) + val (code, printed) = output(if (Main.concludeRun(summary)) 0 else 1) + code should equal(1) + val listed = printed.indexOf("2 procedures passed with warnings:") + listed should be > printed.indexOf("Abort due to test failure") + // each warning, in order, after the heading + val lines = List( + " 8.2.6: complies with IEEE 1815-2012 only", + " 8.24.4: point 0 in 32 bits", + " 8.24.4: point 1 in 32 bits").map(printed.indexOf(_)) + (listed :: lines) shouldBe sorted + lines.distinct.size should equal(3) + printed should not include "8.1.2" + printed should not include "note" + + // or when the plugin fails around a later procedure + val partial = RunSummary(List(warned), List("8.3.2")) + val (aborted, told) = + output(if (Main.concludeRun(throw new HarnessExecutionException("8.3.2", partial, new Exception("can't start")))) + 0 + else 1) + aborted should equal(1) + told.indexOf("1 procedure passed with warnings:") should be > told.indexOf("can't start") + told should include(" 8.2.6: complies with IEEE 1815-2012 only") + } + + test("a run's end lists no warnings when no procedure passed with any") { + val summary = RunSummary(List(ProcedureResult("8.1.2", ProcedureOutcome.Passed, notes = List("a note"))), Nil) + val (code, printed) = output(if (Main.concludeRun(summary)) 0 else 1) + code should equal(0) + printed should not include "warnings" + } } diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/application/SavedResultTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/application/SavedResultTest.scala index e85a37e..f44cf27 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/application/SavedResultTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/application/SavedResultTest.scala @@ -36,6 +36,24 @@ class SavedResultTest extends AnyFunSuite with Matchers { List(results, directory).foreach(Files.delete) } + test("a pass keeps its warnings and notes, each in order, as warning.N and note.N") { + val directory = Files.createTempDirectory("dnp3c") + val results = directory.resolve("results.properties") + val saved = SavedResult( + ProcedureOutcome.Passed, + Some("3.1r1"), + Some(2), + notes = List("a note"), + warnings = List("complies with 1815-2012 only", "point 0 only in 32 bits")) + SavedResult.write(saved, results) + val written = Files.readString(results) + written should include("warning.1=complies with 1815-2012 only") + written should include("warning.2=point 0 only in 32 bits") + written should include("note.1=a note") + SavedResult.read(results.toFile) should equal(saved) + List(results, directory).foreach(Files.delete) + } + test("an excluded result keeps its reason, and can't be read without one") { val directory = Files.createTempDirectory("dnp3c") val results = directory.resolve("results.properties") diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala index 5887f33..0a52ee4 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala @@ -487,23 +487,56 @@ class TestRunnerTest extends AnyFunSuite with Matchers { environment.policies.map(_.all) should equal(Some(List(rule))) } - test("a pass keeps the notes its procedure made") { - val run = new Run() - val noting = TestProcedure( - "a", - "a", + // a procedure that notes and warns, then fails if told to + private def remarking(name: String, fail: Failures): TestProcedure = + TestProcedure( + name, + name, new ProcedureDefinition { override protected def steps(options: OptionSource): Iterable[LabeledStep] = List(LabeledStep( - "note", + "remark", new TestStep { def description: List[Comment] = Nil - def run(driver: TestDriver): List[DeferredCheck] = { driver.reporter.note("compliant with 2012 only"); Nil } + def run(driver: TestDriver): List[DeferredCheck] = { + driver.reporter.note("an allowed alternative") + driver.reporter.warning("compliant with <2012> only") + fail(s"step $name") + Nil + } } )) } ) - run(noting).results.map(_.notes) should equal(List(List("compliant with 2012 only"))) - run.saved("a").map(_.notes) should equal(Some(List("compliant with 2012 only"))) + + test("a pass keeps the notes and warnings its procedure made, and logs a warning as one") { + val run = new Run() + val summary = run(section(remarking("a", run.fail), procedure("b", run.fail))) + summary.results.map(r => (r.id, r.notes, r.warnings)) should equal(List( + ("a", List("an allowed alternative"), List("compliant with <2012> only")), + ("b", Nil, Nil))) + summary.passedWithWarnings.map(_.id) should equal(List("a")) + run.saved("a").map(s => (s.notes, s.warnings)) should equal( + Some((List("an allowed alternative"), List("compliant with <2012> only")))) + val log = Files.readString(run.folder.resolve("a").resolve("log.html")) + log should include regex + """\s*[^<]*\s*[^<]*\s*\s*Warning on the result: compliant with <2012> only""" + } + + test("a pass whose plugin then fails to end it keeps its notes and warnings in the partial summary") { + val run = new Run("end") + val ex = intercept[HarnessExecutionException](run(remarking("a", run.fail))) + ex.getCause.getMessage should equal("end failed") + ex.partial.results.map(r => (r.outcome, r.notes, r.warnings)) should equal( + List((Passed, List("an allowed alternative"), List("compliant with <2012> only")))) + ex.partial.passedWithWarnings.map(_.id) should equal(List("a")) + } + + test("a failure keeps no notes or warnings, and isn't a pass with warnings") { + val run = new Run("step a") + val summary = run(remarking("a", run.fail)) + summary.results.map(r => (r.notes, r.warnings)) should equal(List((Nil, Nil))) + summary.passedWithWarnings should equal(Nil) + run.saved("a").map(s => (s.notes, s.warnings)) should equal(Some((Nil, Nil))) } test("a saved result records its revision, and why the procedure was skipped") { diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/CommonTimeTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/CommonTimeTest.scala index 9947118..71c1e96 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/CommonTimeTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/CommonTimeTest.scala @@ -31,12 +31,12 @@ class CommonTimeTest extends AnyFunSuite with Matchers { describes(CommonTime.relativeTimeSupported, relative) should not include "DUT supports Binary Input Change" } - test("8.15.2 accepts IIN2.1 from a device configured to reject reads of relative time events, and notes it") { + test("8.15.2 accepts IIN2.1 from a device configured to reject reads of relative time events, and warns of it") { val rejecting = Map( TimeSyncOptions.binaryInput.withRelativeTime.id -> "false", TimeSyncOptions.binaryInput.withRelativeTimeRejected.id -> "true") - describes(CommonTime.readWithRelativeTime, rejecting) should include("Note the result") - describes(CommonTime.readWithRelativeTime, relative) should not include "Note the result" + describes(CommonTime.readWithRelativeTime, rejecting) should include("Record a warning on the result") + describes(CommonTime.readWithRelativeTime, relative) should not include "Record a warning on the result" } test("8.15.3 runs where 8.15.2 gets events with relative time: with time recorded, and the read not rejected") { diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatusTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatusTest.scala index 017b8a7..0b6090a 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatusTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/ControlStatusTest.scala @@ -3,9 +3,10 @@ package io.stepfunc.dnp3c.conformance.v31r1 +import io.stepfunc.dnp3c.conformance.Run import io.stepfunc.dnp3c.conformance.api.{ Index, Options } import io.stepfunc.dnp3c.conformance.api.options.BinaryOutputOptions -import io.stepfunc.dnp3c.dnp3.Validate +import io.stepfunc.dnp3c.dnp3.{ LinkOverride, RespondingDriver, Validate } import io.stepfunc.dnp3c.dnp3.app.Message import io.stepfunc.dnp3c.dnp3.app.objects.CommandStatus import org.junit.runner.RunWith @@ -13,7 +14,8 @@ import org.scalatestplus.junit.JUnitRunner import org.scalatest.matchers.should.Matchers import org.scalatest.funsuite.AnyFunSuite -// 3.1r1's 8.2.6: a select whose status field is non-zero is echoed with a non-zero status, or with 0, which is noted +// 3.1r1's 8.2.6: a select whose status field is non-zero is echoed with a non-zero status, or with 0, which is warned +// of @RunWith(classOf[JUnitRunner]) class ControlStatusTest extends AnyFunSuite with Matchers { @@ -31,7 +33,7 @@ class ControlStatusTest extends AnyFunSuite with Matchers { (passes, zero) } - test("an echo with a non-zero status passes, as does one with status 0, which is noted") { + test("an echo with a non-zero status passes, as does one with status 0, which is flagged") { result(response(126)) should equal((true, false)) result(response(4)) should equal((true, false)) result(response(0)) should equal((true, true)) @@ -44,4 +46,23 @@ class ControlStatusTest extends AnyFunSuite with Matchers { // a different control code result(response(4, change = bytes => bytes.updated(7, (bytes(7) ^ 0x01).toByte)))._1 should equal(false) } + + // a DUT that echoes each select with the status given + private class Echoing(status: Int) extends RespondingDriver() { + override def writeMessage(message: Message, options: LinkOverride): Unit = + enqueue(response(status, change = _ => message.headers.head.toBytes.init :+ status.toByte)) + } + + test("8.2.6 passes an echo with status 0 with a warning that it complies with 1815-2012 only, and not a note") { + val zero = new Echoing(0) + Run.passes(ControlStatus.nonZeroStatus, zero) should equal(true) + zero.reporter.warnings should equal(List( + "the DUT echoes a select whose status field isn't zero with status code 0: it complies with IEEE 1815-2012, but " + + "not with IEEE 1815-202x")) + zero.reporter.notes should equal(Nil) + + val nonZero = new Echoing(4) + Run.passes(ControlStatus.nonZeroStatus, nonZero) should equal(true) + nonZero.reporter.warnings should equal(Nil) + } } diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariationsTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariationsTest.scala index ceb4cd1..c151a97 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariationsTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/StaticVariationsTest.scala @@ -4,10 +4,12 @@ package io.stepfunc.dnp3c.conformance.v31r1 import io.stepfunc.dnp3c.conformance.Run -import io.stepfunc.dnp3c.conformance.api.options.{ CounterOptions, OutstationOptions } +import io.stepfunc.dnp3c.conformance.api.options.{ AnalogOutputOptions, CounterOptions, OutstationOptions } import io.stepfunc.dnp3c.dnp3._ import io.stepfunc.dnp3c.dnp3.app._ -import io.stepfunc.dnp3c.dnp3.app.objects.{ Group20Var1, Group20Var2, Group20Var5, Group20Var6 } +import io.stepfunc.dnp3c.dnp3.app.objects.{ + Group20Var1, Group20Var2, Group20Var5, Group20Var6, Group40Var1, Group40Var2 +} import io.stepfunc.dnp3c.protocol.Hex import org.junit.runner.RunWith import org.scalatestplus.junit.JUnitRunner @@ -85,9 +87,11 @@ class StaticVariationsTest extends AnyFunSuite with Matchers { test("counters in the variations requested, the abnormal one with flags, pass, and the flags are noted") { val plugin = new Plugin - val (passed, notes) = run(plugin, dut(plugin)) + val d = dut(plugin) + val (passed, notes) = run(plugin, d) passed should equal(true) notes.exists(_.contains("came back with flags 0x03")) should equal(true) + d.reporter.warnings should equal(Nil) plugin.abnormal should equal(None) // set back to normal at the end } @@ -108,9 +112,11 @@ class StaticVariationsTest extends AnyFunSuite with Matchers { test("a level 1 or 2 device may answer with IIN2.1 and skip the test; a level 3 one may not") { val plugin = new Plugin - val (passed, notes) = run(plugin, dut(plugin, nullError = true), level = 2) + val d = dut(plugin, nullError = true) + val (passed, notes) = run(plugin, d, level = 2) passed should equal(true) notes.exists(_.contains("as a subset level 1 or 2 device may")) should equal(true) + d.reporter.warnings should equal(Nil) run(new Plugin, dut(new Plugin, nullError = true), level = 3)._1 should equal(false) } @@ -120,4 +126,27 @@ class StaticVariationsTest extends AnyFunSuite with Matchers { val plugin2 = new Plugin(says = 0x02) run(plugin2, dut(plugin2))._1 should equal(false) } + + // analog output status points 0 and 1, flags 0x01, each in the variation requested, but point 0 always in 32 bits + private def analogOutputs = new ScriptedDut( + StubDriver.reporter(new Plugin), + request => { + val requested = request.headers.head.variation + val headers = List(0, 1).flatMap { i => + val v = if (i == 0) Group40Var1 else requested + val value = if (v == Group40Var1) "01 05 00 00 00" else "01 05 00" + Message.parseObjectHeaders(Hex.from("28 %02X 00 %02X %02X %s".format(v.id, i, i, value))) + } + (IIN.empty, headers) + } + ) + + test("analog output status kept only in 32 bits passes with a warning that it isn't conformant at level 1 or 2") { + val d = analogOutputs + val options = Map(OutstationOptions.subsetLevel.id -> "2", AnalogOutputOptions.statusOnly32Bit.id -> "0") + Run.passes(StaticVariations.test(StaticVariations.analogOutputStatus), d, options) should equal(true) + d.reporter.warnings should equal(List( + "analog output status point 0 comes back only in 32 bits: the DUT isn't conformant at subset level 1 or 2")) + d.reporter.notes should equal(Nil) + } } diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/WithTimeTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/WithTimeTest.scala index b26c33c..6e429da 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/WithTimeTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/v31r1/WithTimeTest.scala @@ -31,11 +31,12 @@ class WithTimeTest extends AnyFunSuite with Matchers { TimeSyncOptions.binaryInput.withTime.id -> "false", TimeSyncOptions.binaryInput.withTimeRejected.id -> "true")) - test("8.14.2j passes a device declared to reject reads with time, noting it complies with 1815-2012 only") { + test("8.14.2j passes a device declared to reject reads with time, warning it complies with 1815-2012 only") { val reporter = StubDriver.reporter(new Plugin) val dut = new ScriptedDut(reporter, answer) Run.passes(InputEvents.binaryAllWithTime, dut, rejecting) should equal(true) - reporter.notes.head should include("IEEE 1815-2012, but not with IEEE 1815-202x") + reporter.warnings.head should include("IEEE 1815-2012, but not with IEEE 1815-202x") + reporter.notes should equal(Nil) } test("8.14.2k and l are skipped for that device") { @@ -54,11 +55,12 @@ class WithTimeTest extends AnyFunSuite with Matchers { TimeSyncOptions.binaryInput.withRelativeTime.id -> "false", TimeSyncOptions.binaryInput.withRelativeTimeRejected.id -> "true")) - test("8.14.2m passes a device declared to reject reads with relative time, noting it complies with 1815-2012") { + test("8.14.2m passes a device declared to reject reads with relative time, warning it complies with 1815-2012") { val reporter = StubDriver.reporter(new Plugin) val dut = new ScriptedDut(reporter, answer) Run.passes(InputEvents.binaryAllWithRelativeTime, dut, rejectingRelative) should equal(true) - reporter.notes.head should include("relative time with IIN2.1") + reporter.warnings.head should include("relative time with IIN2.1") + reporter.notes should equal(Nil) } test("8.14.2n to p are skipped for that device, and p for any device without relative time") {