diff --git a/README.md b/README.md index 5d39688..1755f70 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ dnp3c -rev 2.7 -help # list the commands and test se Results, including `report.html`, are written to the revision's folder of the directory set by `test.output.directory`, e.g. `results/2.7/L2/`. The configuration must set `outstation.subset.level`, the DNP3 subset level the device is configured for: it has no default. See [Testing a device by hand](docs/manual-testing.md) for the whole workflow. -A procedure the device can't pass yet, e.g. for a known gap, can be excluded in the configuration, with the reason: `test.exclude.8.26.2 = Assign Class isn't supported yet`. It isn't run, and the report shows it as not run, with the reason, apart from skipped procedures, which don't apply to the device. See [Exclude procedures](docs/manual-testing.md#exclude-procedures). +A procedure the device can't pass yet, e.g. for a known gap, can be excluded in the configuration, with the reason: `test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet`. It isn't run, and the report shows it as not run, with the reason, apart from skipped procedures, which don't apply to the device. Each exclusion names its revision, so one configuration can carry exclusions for several revisions. See [Exclude procedures](docs/manual-testing.md#exclude-procedures). ## Library diff --git a/docs/automation.md b/docs/automation.md index 25a982b..b1a2906 100644 --- a/docs/automation.md +++ b/docs/automation.md @@ -57,10 +57,10 @@ A procedure that won't run is skipped before any of that: one that isn't impleme A device with a known gap would stop every run at the procedure that tests it. Exclude the procedure in the configuration, with the reason, and the run carries on past it: ``` -test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480) +test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480) ``` -[Testing a device by hand](manual-testing.md#exclude-procedures) has the rules: one property per procedure of the revision, each with a reason, checked before anything runs, and a procedure that's skipped stays skipped. Exclude a procedure for a feature the device has; turn off a `supports.*` option only for one it lacks. +[Testing a device by hand](manual-testing.md#exclude-procedures) has the rules: one property per procedure, naming its revision, each with a reason, checked before anything runs, and a procedure that's skipped stays skipped. A run excludes only its own revision's procedures, so one configuration can serve every revision. Exclude a procedure for a feature the device has; turn off a `supports.*` option only for one it lacks. An excluded procedure's outcome is `Excluded(reason)`, which is new: a match over `ProcedureOutcome` must handle it. Unlike `Skipped`, it means the procedure applies to the device but didn't run, so the device hasn't been shown to conform to it. The report shows it as not run, with the reason, and says that conformance hasn't been demonstrated. diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 7620b0b..1b40fe0 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -72,10 +72,11 @@ Procedures for features the DUT doesn't support are optional. If a `supports.*` A procedure that applies to the DUT, but that you can't run yet, e.g. for a known gap in the DUT that's being fixed, can be left out of runs by excluding it in the configuration, with the reason: ``` -test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480) +test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480) ``` -- **One property per procedure,** `test.exclude.`, and each needs a reason. Only procedures can be excluded, not sections or `all`. Every exclusion must name a procedure of the revision given with `-rev`, even one outside what you run: `-run`, `-describe` and `-disabled` check them all, and fail on any that doesn't. +- **One property per procedure,** `test.exclude./`, and each needs a reason. Only procedures can be excluded, not sections or `all`. +- **Each names its revision,** as given to `-rev`, then a `/` and the procedure, since the revisions number their procedures differently: 8.23 is Device Attributes in 2.7 and Specific Point Ranges in 3.1r1. A run excludes only its own revision's procedures, so one configuration can exclude procedures of several revisions. Every exclusion must name a revision and a procedure of it, even one for another revision or outside what you run: `-run`, `-describe` and `-disabled` check them all, and fail on any that doesn't. - **An excluded procedure isn't run.** dnp3c doesn't connect to the DUT, capture or prompt for it, and the run carries on to the next procedure. It's recorded as excluded, with a log saying so, replacing the procedure's earlier result, log and capture. - **Excluded isn't skipped.** A skipped procedure doesn't apply to the DUT or its configuration, and the DUT can still be conformant. An excluded one does apply, so the DUT hasn't been shown to conform to it. The report shows it as not run, with the reason, and counts it apart from passed, failed and skipped procedures. A run that excludes procedures still succeeds, but it ends by saying how many it excluded. - **A procedure that's skipped stays skipped,** e.g. one that doesn't apply at the DUT's subset level, even if it's excluded too. diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/Harness.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/Harness.scala index 3652ccf..3ae81ed 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/Harness.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/Harness.scala @@ -34,14 +34,15 @@ final class Harness(options: Options, plugin: IntegrationPlugin, revision: Revis * Run the procedures of a procedure or section, or "all" of them, in order, stopping at the first that fails. * * A procedure fails when one of its steps or validations does, including a plugin action or verification that - * throws. One the configuration excludes (`test.exclude.`) isn't run: it's recorded as excluded, without - * touching the device or the plugin, and the run carries on. + * throws. One the configuration excludes for the revision (`test.exclude./`) isn't run: it's + * recorded as excluded, without touching the device or the plugin, and the run carries on. * * @param id a procedure or section of the revision, e.g. "8.6.5.4" or "8.6", or "all" * @throws IllegalArgumentException if the revision has no procedure or section with the id, before anything runs * @throws api.ConfigurationException if the configuration has no subset level, or one the revision doesn't test, or - * excludes something that isn't a procedure of the revision, or lacks what the - * revision, or a procedure that will run, needs configured, before anything runs + * excludes something that isn't a procedure of the revision it names, or lacks + * what the revision, or a procedure that will run, needs configured, before + * anything runs * @throws HarnessExecutionException if the harness or the plugin fails around a procedure, e.g. when * `startProcedure` throws or the capture can't be opened; it holds the * results so far. An interruption propagates as it is. diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala index 4f9396a..4a7ed85 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala @@ -52,7 +52,7 @@ object Main { Revisions.find(id) match { case Some(revision) => printHelp(Some(revision)) case None => - Colors.error.printLine(unknownRevision(id)) + Colors.error.printLine(Revisions.unknown(id)) return -1 } } @@ -109,7 +109,7 @@ object Main { Revisions.find(id) match { case Some(r) => r case None => - Colors.error.printLine(unknownRevision(id)) + Colors.error.printLine(Revisions.unknown(id)) return -1 } } @@ -131,7 +131,7 @@ object Main { val level = try { val level = revision.subsetLevel(options) - // the exclusions -disabled and -describe show must name procedures of the revision + // the exclusions -disabled and -describe show, and every other revision's, must name procedures of them if (config.listDisabled || config.describe.nonEmpty) revision.checkExclusions(options) level } catch { @@ -282,9 +282,6 @@ object Main { Colors.text.printLine("") } - private def unknownRevision(id: String): String = - s"Unknown revision: $id. The revisions are: ${Revisions.list.map(_.id).mkString(", ")}" - def listDisabled(options: OptionSource, revision: Revision, level: Int): Unit = { Colors.text.printLine("Listing the test procedures the configuration skips or excludes: ") Colors.text.printLine("") diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/Revision.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/Revision.scala index d3db011..cbda903 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/Revision.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/Revision.scala @@ -100,23 +100,36 @@ final class Revision private[conformance] ( private[conformance] def decide(procedure: TestProcedure, options: OptionSource, level: Int): Decision = skip(procedure, options, level) match { case Some(s) => Decision.Skip(s) - case None => options.get(TestOptions.exclude).get(procedure.id).fold(Decision.Run)(Decision.Exclude(_)) + case None => + val exclude = TestOptions.exclude + options.get(exclude).get(exclude.key(id, procedure.id)).fold(Decision.Run)(Decision.Exclude(_)) } - // every exclusion must name a procedure of this revision, whatever procedures a run selects - private def exclusionProblems(options: OptionSource): List[ConfigProblem] = - options.get(TestOptions.exclude).toList.sorted.flatMap { (procedure, reason) => - val key = s"${TestOptions.exclude.id}.$procedure" - elements.get(procedure) match { - case Some(_: TestProcedure) => None - case Some(_) => - Some(ConfigProblem(key, reason, "a section: exclude its procedures, each with its own property")) - case None => Some(ConfigProblem(key, reason, s"revision $id has no procedure $procedure")) + // every exclusion must name a revision and a procedure of it, whatever revision and procedures a run selects, so one + // configuration serves every revision; a test's revision isn't listed, but its own exclusions name it + private def exclusionProblems(options: OptionSource): List[ConfigProblem] = { + val exclude = TestOptions.exclude + val revisions = (Revisions.list :+ this).distinctBy(_.id) + options.get(exclude).toList.sorted.flatMap { (excluded, reason) => + def problem(message: String) = Some(ConfigProblem(s"${exclude.id}.$excluded", reason, message)) + exclude.split(excluded) match { + case None => problem(s"name the revision: ${exclude.syntax}. The revisions are: ${Revisions.ids}") + case Some((rev, procedure)) => + revisions.find(_.id == rev) match { + case None => problem(Revisions.unknown(rev)) + case Some(revision) => + revision.elements.get(procedure) match { + case Some(_: TestProcedure) => None + case Some(_) => problem("a section: exclude its procedures, each with its own property") + case None => problem(s"revision $rev has no procedure $procedure") + } + } } } + } /** - * Check that every exclusion in the configuration names a procedure of this revision + * Check that every exclusion in the configuration names a revision and a procedure of it * * @throws ConfigurationException listing every one that doesn't */ @@ -127,8 +140,8 @@ final class Revision private[conformance] ( /** * Check that the configuration is one this revision can test: it gives a subset level the revision tests, its - * exclusions name procedures of the revision, and it has what the revision needs configured, whatever procedures - * run, and what each procedure selected needs, if it will run (see [[decide]]). + * exclusions name revisions and procedures of them, and it has what the revision needs configured, whatever + * procedures run, and what each procedure selected needs, if it will run (see [[decide]]). * * @param selected the procedures a run will run; all of them unless given * @return the subset level @@ -227,4 +240,9 @@ object Revisions { val list: List[Revision] = List(v2_7, v3_1r1) def find(id: String): Option[Revision] = list.find(_.id == id) + + private[conformance] def ids: String = list.map(_.id).mkString(", ") + + // for an id that isn't one of them, as -rev or an exclusion gives it + private[conformance] def unknown(id: String): String = s"Unknown revision: $id. The revisions are: $ids" } diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala index c50aecd..2a0478c 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala @@ -14,8 +14,8 @@ object ProcedureOutcome { case class Skipped(reason: String) extends ProcedureOutcome /** - * The configuration left it out (`test.exclude.`), for this reason. Unlike a skipped procedure, it applies to - * the device, so the device hasn't been shown to conform to it. + * The configuration left it out (`test.exclude./`), for this reason. Unlike a skipped procedure, + * it applies to the device, so the device hasn't been shown to conform to it. */ case class Excluded(reason: String) extends ProcedureOutcome } diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/api/TestOption.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/api/TestOption.scala index 3b123d4..7762768 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/api/TestOption.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/api/TestOption.scala @@ -50,15 +50,32 @@ trait IndexListOption extends TypedTestOption[List[Index]] trait RequiredIntegerOption extends TypedTestOption[Option[Int]] { final override def required: Boolean = true } -// procedures to leave out of a run, by id, each with the reason: a family of properties, one per procedure, under the -// option's id, e.g. "test.exclude.8.26.2 = " +// procedures to leave out of a run, by revision and id, each with the reason: a family of properties, one per +// procedure, under the option's id, e.g. "test.exclude.3.1r1/8.26.2 = ". Each is keyed by the rest of its key, +// e.g. "3.1r1/8.26.2", which the revisions check: the options don't know them trait ExclusionsOption extends TypedTestOption[Map[String, String]] { // a property of the family, for the configurations written and listed, commented out def example: String - // the procedure id and reason of a property of the family, given the rest of its key after the option's id and "." - def entry(procedure: String, value: String): (String, String) + // the revision and procedure ids and reason of a property of the family, given the rest of its key after the + // option's id and "." + def entry(excluded: String, value: String): (String, String) + + // between an exclusion's revision id and procedure id, which both have dots + final def separator: String = "/" + + // an exclusion's key in the map, the rest of its property's key + final def key(revision: String, procedure: String): String = revision + separator + procedure + + // the revision and procedure ids of an exclusion's key, split at the first separator, if it has one + final def split(excluded: String): Option[(String, String)] = excluded.indexOf(separator) match { + case -1 => None + case i => Some(excluded.take(i) -> excluded.drop(i + separator.length)) + } + + // how to write an exclusion, for messages + final def syntax: String = s"$id.${key("", "")} = " } object OptionTypes { @@ -122,13 +139,13 @@ object OptionTypes { // the family has no value of its own override def parse(s: String): Map[String, String] = - throw new Exception(s"exclude one procedure per property, with the reason: $id. = ") + throw new Exception(s"exclude one procedure per property, with the reason: $syntax") - override def entry(procedure: String, value: String): (String, String) = { - if (procedure.isEmpty) throw new Exception(s"name the procedure: $id. = ") + override def entry(excluded: String, value: String): (String, String) = { + if (excluded.isEmpty) throw new Exception(s"name the procedure: $syntax") val reason = value.trim - if (reason.isEmpty) throw new Exception(s"give the reason $procedure is excluded") - procedure -> reason + if (reason.isEmpty) throw new Exception(s"give the reason $excluded is excluded") + excluded -> reason } } diff --git a/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TestOptions.scala b/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TestOptions.scala index d67d92b..df29e8d 100644 --- a/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TestOptions.scala +++ b/src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TestOptions.scala @@ -30,10 +30,12 @@ object TestOptions extends OptionGroup { val exclude: ExclusionsOption = OptionTypes.exclusions( this, "exclude", - "Procedures to leave out of a run, one property each, with the reason: test.exclude. = . " + + "Procedures to leave out of a run, one property each, with the reason: " + + "test.exclude./ = . A run leaves out only its own revision's, so one " + + "configuration can exclude procedures of several revisions. " + "An excluded procedure applies to the DUT but isn't run, and is reported as not run, so the DUT isn't shown to " + "conform to it. A procedure for a feature the DUT lacks is skipped by its supports.* option instead", - "test.exclude.8.26.2 = Assign Class isn't supported yet" + "test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet" ) def prefix: String = "test" diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala index 7d1d07f..8f0e6b4 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/MainTest.scala @@ -7,7 +7,7 @@ import java.io.{ ByteArrayOutputStream, File, PrintStream } import java.nio.charset.StandardCharsets import java.nio.file.{ Files, Path } -import io.stepfunc.dnp3c.conformance.api.{ BooleanOption, OptionGroups, Options, ProcedureOutcome } +import io.stepfunc.dnp3c.conformance.api.{ BooleanOption, OptionGroups, Options, ProcedureOutcome, TestProcedure } import io.stepfunc.dnp3c.conformance.api.options.{ OutstationOptions, SupportOptions, TestOptions } import io.stepfunc.dnp3c.conformance.application.SavedResult import io.stepfunc.dnp3c.conformance.application.Console @@ -146,8 +146,8 @@ class MainTest extends AnyFunSuite with Matchers { test("-disabled lists an exclusion with its reason, and a procedure that's skipped anyway as skipped") { val options = Options.from(Map( OutstationOptions.subsetLevel.id -> "2", - s"${TestOptions.exclude.id}.8.26.2" -> "Assign Class isn't supported yet", - s"${TestOptions.exclude.id}.8.26.3" -> "nor no-class assignment" + s"${TestOptions.exclude.id}.3.1r1/8.26.2" -> "Assign Class isn't supported yet", + s"${TestOptions.exclude.id}.3.1r1/8.26.3" -> "nor no-class assignment" )) val lines = Main.disabledLines(options, Revisions.v3_1r1, 2) lines.sliding(2).toList should @@ -156,6 +156,7 @@ class MainTest extends AnyFunSuite with Matchers { lines.mkString should not include "nor no-class assignment" } + // excluding a procedure of a revision, given its key, e.g. "3.1r1/8.26.2" private def withExclusion(id: String, reason: String = "Assign Class isn't supported yet") = level2 + s"${TestOptions.exclude.id}.$id = $reason\n" @@ -169,39 +170,77 @@ class MainTest extends AnyFunSuite with Matchers { private def command31r1( contents: String, + revision: String = "3.1r1", describe: Option[String] = None, disabled: Boolean = false, run: Option[String] = None, plugin: IntegrationPlugin = new DefaultIntegrationPlugin()): (Int, String) = output(withConfig(contents)(config => Main.execute( - command(describe, None, Some(config), revision = Some("3.1r1"), run = run, listDisabled = disabled), + command(describe, None, Some(config), revision = Some(revision), run = run, listDisabled = disabled), plugin))) test("-disabled and -describe show an exclusion's reason, without the procedure's steps") { - val (disabled, listed) = command31r1(withExclusion("8.26.2"), disabled = true) + val (disabled, listed) = command31r1(withExclusion("3.1r1/8.26.2"), disabled = true) disabled should equal(0) listed should include("Procedure: 8.26.2") listed should include("NOT RUN — EXCLUDED: Assign Class isn't supported yet") - val (described, procedure) = command31r1(withExclusion("8.26.2"), describe = Some("8.26.2")) + val (described, procedure) = command31r1(withExclusion("3.1r1/8.26.2"), describe = Some("8.26.2")) described should equal(0) procedure should include("NOT RUN — EXCLUDED 8.26.2: Assign Class isn't supported yet") procedure should not include "step" - val (section, tree) = command31r1(withExclusion("8.26.2"), describe = Some("8.26")) + val (section, tree) = command31r1(withExclusion("3.1r1/8.26.2"), describe = Some("8.26")) section should equal(0) tree should include("Test Procedures [NOT RUN — EXCLUDED: Assign Class isn't supported yet]") tree should include("No-Class Assignment Test Procedures") } - test("an exclusion must name a procedure of the revision, and give a reason") { - for (id <- List("9.9", "8.26", "all")) { - command31r1(withExclusion(id), describe = Some("8.1.2"))._1 should not equal 0 - command31r1(withExclusion(id), disabled = true)._1 should not equal 0 + test("an exclusion must name a revision and a procedure of it, and give a reason, whichever revision runs") { + // without a revision, an unknown revision, an id the revision doesn't have, a section, and all + val ids = List("8.26.2", "3.1r1.8.26.2", "3.0/8.26.2", "2.7/8.26.2", "3.1r1/9.9", "3.1r1/8.26", "3.1r1/all") + for (revision <- List("2.7", "3.1r1"); id <- ids) { + withClue(s"$revision $id: ") { + command31r1(withExclusion(id), revision, describe = Some("8.1.2"))._1 should not equal 0 + command31r1(withExclusion(id), revision, disabled = true)._1 should not equal 0 + } } - command31r1(withExclusion("8.26.2", reason = ""), disabled = true)._1 should not equal 0 + command31r1(withExclusion("3.1r1/8.26.2", reason = ""), disabled = true)._1 should not equal 0 command31r1(level2 + s"${TestOptions.exclude.id} = 8.26.2\n", disabled = true)._1 should not equal 0 + // the syntax and the revisions, or the revisions as for -rev + val (_, printed) = command31r1(withExclusion("3.1r1.8.26.2"), disabled = true) + printed should include("test.exclude./ = . The revisions are: 2.7, 3.1r1") + command31r1(withExclusion("3.0/8.26.2"), "2.7", disabled = true)._2 should include( + "Unknown revision: 3.0. The revisions are: 2.7, 3.1r1") + } + + test("one configuration serves every revision: a run excludes only its own revision's procedures") { + val (code, listed) = command31r1(withExclusion("3.1r1/8.26.2"), "2.7", disabled = true) + code should equal(0) + listed should not include "EXCLUDED" + command31r1(withExclusion("3.1r1/8.26.2"), "2.7", describe = Some("8.1.2"))._1 should equal(0) + + // 8.1.2 is in both, and 8.23.2.1 is a procedure of 2.7's but a section of 3.1r1's + val options = Options.from(Map( + OutstationOptions.subsetLevel.id -> "2", + s"${TestOptions.exclude.id}.3.1r1/8.26.2" -> "Assign Class isn't supported yet", + s"${TestOptions.exclude.id}.3.1r1/8.1.2" -> "3.1r1's", + s"${TestOptions.exclude.id}.2.7/8.23.2.1" -> "2.7's" + )) + def decision(revision: Revision, id: String) = + revision.decide(revision.elements(id).asInstanceOf[TestProcedure], options, 2) + for (revision <- Revisions.list) { + revision.checkExclusions(options) + revision.validate(options, Nil) should equal(2) + } + decision(Revisions.v3_1r1, "8.1.2") should equal(Decision.Exclude("3.1r1's")) + decision(Revisions.v2_7, "8.1.2") should equal(Decision.Run) + decision(Revisions.v2_7, "8.23.2.1") should equal(Decision.Exclude("2.7's")) + Main.disabledLines(options, Revisions.v2_7, 2).filter(_.contains("EXCLUDED")) should equal( + List(" NOT RUN — EXCLUDED: 2.7's")) + Main.disabledLines(options, Revisions.v3_1r1, 2).filter(_.contains("EXCLUDED")) should equal( + List(" NOT RUN — EXCLUDED: 3.1r1's", " NOT RUN — EXCLUDED: Assign Class isn't supported yet")) } // writes the results in a folder of their own @@ -213,7 +252,7 @@ class MainTest extends AnyFunSuite with Matchers { private val oneExcluded = "1 procedure was excluded and not run. Conformance has not been demonstrated for it." test("a run that excludes a procedure records it, says so, and succeeds") { - val (contents, output) = withOutput(withExclusion("8.26.2")) + val (contents, output) = withOutput(withExclusion("3.1r1/8.26.2")) val (code, printed) = command31r1(contents, run = Some("8.26")) code should equal(0) printed should include("NOT RUN — EXCLUDED 8.26.2: Assign Class isn't supported yet") @@ -229,7 +268,8 @@ class MainTest extends AnyFunSuite with Matchers { val plugin = new DefaultIntegrationPlugin { override def cyclePower(reporter: PluginReporter): Unit = throw new Exception("can't cycle power") } - val (contents, _) = withOutput(withExclusion("6.1.2", "no link layer access") + "tcp.device.port = 1\n") + val (contents, _) = + withOutput(withExclusion("3.1r1/6.1.2", "no link layer access") + "tcp.device.port = 1\n") val (code, printed) = command31r1(contents, run = Some("6"), plugin = plugin) code should not equal 0 printed should include("NOT RUN — EXCLUDED 6.1.2: no link layer access") @@ -240,7 +280,8 @@ class MainTest extends AnyFunSuite with Matchers { val failing = new DefaultIntegrationPlugin { override def startProcedure(reporter: PluginReporter): Unit = throw new Exception("can't start") } - val (aborted, told) = command31r1(withOutput(withExclusion("6.1.2"))._1, run = Some("6"), plugin = failing) + val (aborted, told) = + command31r1(withOutput(withExclusion("3.1r1/6.1.2"))._1, run = Some("6"), plugin = failing) aborted should not equal 0 told.indexOf(oneExcluded) should be > told.indexOf("can't start") } diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/api/OptionsTestSuite.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/api/OptionsTestSuite.scala index 1ef63cf..df2caf6 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/api/OptionsTestSuite.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/api/OptionsTestSuite.scala @@ -150,29 +150,34 @@ class OptionsTestSuite extends AnyFunSuite with Matchers { Files.readString(path) should include("# outstation.subset.level = ") Options.load(path).get(options.OutstationOptions.subsetLevel) should equal(None) // and so is an example of an exclusion - Files.readString(path) should include(s"# ${TestOptions.exclude.id}.8.26.2 = ") + Files.readString(path) should include(s"# ${TestOptions.exclude.id}.3.1r1/8.26.2 = ") Options.load(path).get(TestOptions.exclude) should equal(Map()) } - test("each exclusion is a property of its own, naming the procedure and giving the reason") { + test("each exclusion is a property of its own, naming the revision and procedure and giving the reason") { val exclusions = Options.load(file( - "test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480) ", - "test.exclude.8.6.5b = Relé: 50% done, see #12 — later", + "test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480) ", + "test.exclude.3.1r1/8.6.5b = Relé: 50% done, see #12 — later", s"${TestOptions.deviceUnderTest.id} = RTU" )).get(TestOptions.exclude) exclusions should equal(Map( - "8.26.2" -> "Assign Class isn't supported yet (stepfunc/dnp3#480)", - "8.6.5b" -> "Relé: 50% done, see #12 — later")) + "3.1r1/8.26.2" -> "Assign Class isn't supported yet (stepfunc/dnp3#480)", + "3.1r1/8.6.5b" -> "Relé: 50% done, see #12 — later")) Options.from(Map()).get(TestOptions.exclude) should equal(Map()) } test("an exclusion needs a procedure and a reason, and there's no list of them") { - problems("test.exclude.8.26.2 =", "test.exclude.8.1.2 = ", "test.exclude. = gap", "test.exclude = 8.26.2") - .map(_.key) should equal(List("test.exclude", "test.exclude.", "test.exclude.8.1.2", "test.exclude.8.26.2")) - problems("test.exclude = 8.26.2").map(_.message).head should include("test.exclude. = ") + problems( + "test.exclude.3.1r1/8.26.2 =", + "test.exclude.2.7/8.1.2 = ", + "test.exclude. = gap", + "test.exclude = 8.26.2") + .map(_.key) should + equal(List("test.exclude", "test.exclude.", "test.exclude.2.7/8.1.2", "test.exclude.3.1r1/8.26.2")) + problems("test.exclude = 8.26.2").map(_.message).head should include("test.exclude./") // keys like it are still unknown - problems("test.excluded.8.26.2 = gap", "test.exclude.8.26.2 = gap").map(_.key) should equal( - List("test.excluded.8.26.2")) + problems("test.excluded.3.1r1/8.26.2 = gap", "test.exclude.3.1r1/8.26.2 = gap").map(_.key) should equal( + List("test.excluded.3.1r1/8.26.2")) } test("writing the default configuration file fails when the write does") { diff --git a/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala b/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala index 9631a68..5887f33 100644 --- a/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala +++ b/src/test/scala/io/stepfunc/dnp3c/conformance/application/TestRunnerTest.scala @@ -174,10 +174,10 @@ class TestRunnerTest extends AnyFunSuite with Matchers { def logged(id: String): Boolean = Files.exists(folder.resolve(id).resolve("log.html")) - // at level 2, excluding procedures by id, each with its reason + // at level 2, excluding procedures of revision "t" by id, each with its reason def excluding(exclusions: (String, String)*): Options = Options.from( Map(TestOptions.outputDirectory.id -> directory.toString, OutstationOptions.subsetLevel.id -> "2") ++ - exclusions.map((id, reason) => s"${TestOptions.exclude.id}.$id" -> reason)) + exclusions.map((id, reason) => s"${TestOptions.exclude.id}.t/$id" -> reason)) // runs an element of a revision of its own, "t", whose results go in tFolder def in(revision: Revision, element: TestElement, options: Options): RunSummary = @@ -381,7 +381,7 @@ class TestRunnerTest extends AnyFunSuite with Matchers { val revision = revisionOf(procedure("a", run.fail), procedure("b", run.fail)) for (id <- List("9.9", "section", "all")) { val ex = the[ConfigurationException] thrownBy run.in(revision, revision.elements("a"), run.excluding(id -> "gap")) - ex.problems.map(_.key) should equal(List(s"${TestOptions.exclude.id}.$id")) + ex.problems.map(_.key) should equal(List(s"${TestOptions.exclude.id}.t/$id")) (the[ConfigurationException] thrownBy revision.checkExclusions(run.excluding(id -> "gap"))).problems should equal(ex.problems) }