Qualify exclusions by revision, so one configuration serves every revision

README.md

@@ -68,7 +68,7 @@ dnp3c -rev 2.7 -help                             # list the commands and test se
 
 Results, including `report.html`, are written to the revision's folder of the directory set by `test.output.directory`, e.g. `results/2.7/L2/`. The configuration must set `outstation.subset.level`, the DNP3 subset level the device is configured for: it has no default. See [Testing a device by hand](docs/manual-testing.md) for the whole workflow.
 
-A procedure the device can't pass yet, e.g. for a known gap, can be excluded in the configuration, with the reason: `test.exclude.8.26.2 = Assign Class isn't supported yet`. It isn't run, and the report shows it as not run, with the reason, apart from skipped procedures, which don't apply to the device. See [Exclude procedures](docs/manual-testing.md#exclude-procedures).
+A procedure the device can't pass yet, e.g. for a known gap, can be excluded in the configuration, with the reason: `test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet`. It isn't run, and the report shows it as not run, with the reason, apart from skipped procedures, which don't apply to the device. Each exclusion names its revision, so one configuration can carry exclusions for several revisions. See [Exclude procedures](docs/manual-testing.md#exclude-procedures).
 
 ## Library
 

docs/automation.md

@@ -57,10 +57,10 @@ A procedure that won't run is skipped before any of that: one that isn't impleme
 A device with a known gap would stop every run at the procedure that tests it. Exclude the procedure in the configuration, with the reason, and the run carries on past it:
 
 ```
-test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
+test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
 ```
 
-[Testing a device by hand](manual-testing.md#exclude-procedures) has the rules: one property per procedure of the revision, each with a reason, checked before anything runs, and a procedure that's skipped stays skipped. Exclude a procedure for a feature the device has; turn off a `supports.*` option only for one it lacks.
+[Testing a device by hand](manual-testing.md#exclude-procedures) has the rules: one property per procedure, naming its revision, each with a reason, checked before anything runs, and a procedure that's skipped stays skipped. A run excludes only its own revision's procedures, so one configuration can serve every revision. Exclude a procedure for a feature the device has; turn off a `supports.*` option only for one it lacks.
 
 An excluded procedure's outcome is `Excluded(reason)`, which is new: a match over `ProcedureOutcome` must handle it. Unlike `Skipped`, it means the procedure applies to the device but didn't run, so the device hasn't been shown to conform to it. The report shows it as not run, with the reason, and says that conformance hasn't been demonstrated.
 

docs/manual-testing.md

@@ -72,10 +72,11 @@ Procedures for features the DUT doesn't support are optional. If a `supports.*`
 A procedure that applies to the DUT, but that you can't run yet, e.g. for a known gap in the DUT that's being fixed, can be left out of runs by excluding it in the configuration, with the reason:
 
 ```
-test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
+test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
 ```
 
-- **One property per procedure,** `test.exclude.<procedure id>`, and each needs a reason. Only procedures can be excluded, not sections or `all`. Every exclusion must name a procedure of the revision given with `-rev`, even one outside what you run: `-run`, `-describe` and `-disabled` check them all, and fail on any that doesn't.
+- **One property per procedure,** `test.exclude.<revision id>/<procedure id>`, and each needs a reason. Only procedures can be excluded, not sections or `all`.
+- **Each names its revision,** as given to `-rev`, then a `/` and the procedure, since the revisions number their procedures differently: 8.23 is Device Attributes in 2.7 and Specific Point Ranges in 3.1r1. A run excludes only its own revision's procedures, so one configuration can exclude procedures of several revisions. Every exclusion must name a revision and a procedure of it, even one for another revision or outside what you run: `-run`, `-describe` and `-disabled` check them all, and fail on any that doesn't.
 - **An excluded procedure isn't run.** dnp3c doesn't connect to the DUT, capture or prompt for it, and the run carries on to the next procedure. It's recorded as excluded, with a log saying so, replacing the procedure's earlier result, log and capture.
 - **Excluded isn't skipped.** A skipped procedure doesn't apply to the DUT or its configuration, and the DUT can still be conformant. An excluded one does apply, so the DUT hasn't been shown to conform to it. The report shows it as not run, with the reason, and counts it apart from passed, failed and skipped procedures. A run that excludes procedures still succeeds, but it ends by saying how many it excluded.
 - **A procedure that's skipped stays skipped,** e.g. one that doesn't apply at the DUT's subset level, even if it's excluded too.

src/main/scala/io/stepfunc/dnp3c/conformance/Harness.scala

@@ -34,14 +34,15 @@ final class Harness(options: Options, plugin: IntegrationPlugin, revision: Revis
    * Run the procedures of a procedure or section, or "all" of them, in order, stopping at the first that fails.
    *
    * A procedure fails when one of its steps or validations does, including a plugin action or verification that
-   * throws. One the configuration excludes (`test.exclude.<id>`) isn't run: it's recorded as excluded, without
-   * touching the device or the plugin, and the run carries on.
+   * throws. One the configuration excludes for the revision (`test.exclude.<revision id>/<id>`) isn't run: it's
+   * recorded as excluded, without touching the device or the plugin, and the run carries on.
    *
    * @param id a procedure or section of the revision, e.g. "8.6.5.4" or "8.6", or "all"
    * @throws IllegalArgumentException if the revision has no procedure or section with the id, before anything runs
    * @throws api.ConfigurationException if the configuration has no subset level, or one the revision doesn't test, or
-   *                                    excludes something that isn't a procedure of the revision, or lacks what the
-   *                                    revision, or a procedure that will run, needs configured, before anything runs
+   *                                    excludes something that isn't a procedure of the revision it names, or lacks
+   *                                    what the revision, or a procedure that will run, needs configured, before
+   *                                    anything runs
    * @throws HarnessExecutionException if the harness or the plugin fails around a procedure, e.g. when
    *                                   `startProcedure` throws or the capture can't be opened; it holds the
    *                                   results so far. An interruption propagates as it is.

src/main/scala/io/stepfunc/dnp3c/conformance/Main.scala

@@ -52,7 +52,7 @@ object Main {
           Revisions.find(id) match {
             case Some(revision) => printHelp(Some(revision))
             case None =>
-              Colors.error.printLine(unknownRevision(id))
+              Colors.error.printLine(Revisions.unknown(id))
               return -1
           }
       }
@@ -109,7 +109,7 @@ object Main {
         Revisions.find(id) match {
           case Some(r) => r
           case None =>
-            Colors.error.printLine(unknownRevision(id))
+            Colors.error.printLine(Revisions.unknown(id))
             return -1
         }
     }
@@ -131,7 +131,7 @@ object Main {
     val level =
       try {
         val level = revision.subsetLevel(options)
-        // the exclusions -disabled and -describe show must name procedures of the revision
+        // the exclusions -disabled and -describe show, and every other revision's, must name procedures of them
         if (config.listDisabled || config.describe.nonEmpty) revision.checkExclusions(options)
         level
       } catch {
@@ -282,9 +282,6 @@ object Main {
     Colors.text.printLine("")
   }
 
-  private def unknownRevision(id: String): String =
-    s"Unknown revision: $id. The revisions are: ${Revisions.list.map(_.id).mkString(", ")}"
-
   def listDisabled(options: OptionSource, revision: Revision, level: Int): Unit = {
     Colors.text.printLine("Listing the test procedures the configuration skips or excludes: ")
     Colors.text.printLine("")

src/main/scala/io/stepfunc/dnp3c/conformance/Revision.scala

@@ -100,23 +100,36 @@ final class Revision private[conformance] (
   private[conformance] def decide(procedure: TestProcedure, options: OptionSource, level: Int): Decision =
     skip(procedure, options, level) match {
       case Some(s) => Decision.Skip(s)
-      case None => options.get(TestOptions.exclude).get(procedure.id).fold(Decision.Run)(Decision.Exclude(_))
+      case None =>
+        val exclude = TestOptions.exclude
+        options.get(exclude).get(exclude.key(id, procedure.id)).fold(Decision.Run)(Decision.Exclude(_))
     }
 
-  // every exclusion must name a procedure of this revision, whatever procedures a run selects
-  private def exclusionProblems(options: OptionSource): List[ConfigProblem] =
-    options.get(TestOptions.exclude).toList.sorted.flatMap { (procedure, reason) =>
-      val key = s"${TestOptions.exclude.id}.$procedure"
-      elements.get(procedure) match {
-        case Some(_: TestProcedure) => None
-        case Some(_) =>
-          Some(ConfigProblem(key, reason, "a section: exclude its procedures, each with its own property"))
-        case None => Some(ConfigProblem(key, reason, s"revision $id has no procedure $procedure"))
+  // every exclusion must name a revision and a procedure of it, whatever revision and procedures a run selects, so one
+  // configuration serves every revision; a test's revision isn't listed, but its own exclusions name it
+  private def exclusionProblems(options: OptionSource): List[ConfigProblem] = {
+    val exclude = TestOptions.exclude
+    val revisions = (Revisions.list :+ this).distinctBy(_.id)
+    options.get(exclude).toList.sorted.flatMap { (excluded, reason) =>
+      def problem(message: String) = Some(ConfigProblem(s"${exclude.id}.$excluded", reason, message))
+      exclude.split(excluded) match {
+        case None => problem(s"name the revision: ${exclude.syntax}. The revisions are: ${Revisions.ids}")
+        case Some((rev, procedure)) =>
+          revisions.find(_.id == rev) match {
+            case None => problem(Revisions.unknown(rev))
+            case Some(revision) =>
+              revision.elements.get(procedure) match {
+                case Some(_: TestProcedure) => None
+                case Some(_) => problem("a section: exclude its procedures, each with its own property")
+                case None => problem(s"revision $rev has no procedure $procedure")
+              }
+          }
       }
     }
+  }
 
   /**
-   * Check that every exclusion in the configuration names a procedure of this revision
+   * Check that every exclusion in the configuration names a revision and a procedure of it
    *
    * @throws ConfigurationException listing every one that doesn't
    */
@@ -127,8 +140,8 @@ final class Revision private[conformance] (
 
   /**
    * Check that the configuration is one this revision can test: it gives a subset level the revision tests, its
-   * exclusions name procedures of the revision, and it has what the revision needs configured, whatever procedures
-   * run, and what each procedure selected needs, if it will run (see [[decide]]).
+   * exclusions name revisions and procedures of them, and it has what the revision needs configured, whatever
+   * procedures run, and what each procedure selected needs, if it will run (see [[decide]]).
    *
    * @param selected the procedures a run will run; all of them unless given
    * @return the subset level
@@ -227,4 +240,9 @@ object Revisions {
   val list: List[Revision] = List(v2_7, v3_1r1)
 
   def find(id: String): Option[Revision] = list.find(_.id == id)
+
+  private[conformance] def ids: String = list.map(_.id).mkString(", ")
+
+  // for an id that isn't one of them, as -rev or an exclusion gives it
+  private[conformance] def unknown(id: String): String = s"Unknown revision: $id. The revisions are: $ids"
 }

src/main/scala/io/stepfunc/dnp3c/conformance/api/RunSummary.scala

@@ -14,8 +14,8 @@ object ProcedureOutcome {
   case class Skipped(reason: String) extends ProcedureOutcome
 
   /**
-   * The configuration left it out (`test.exclude.<id>`), for this reason. Unlike a skipped procedure, it applies to
-   * the device, so the device hasn't been shown to conform to it.
+   * The configuration left it out (`test.exclude.<revision id>/<id>`), for this reason. Unlike a skipped procedure,
+   * it applies to the device, so the device hasn't been shown to conform to it.
    */
   case class Excluded(reason: String) extends ProcedureOutcome
 }

src/main/scala/io/stepfunc/dnp3c/conformance/api/TestOption.scala

@@ -50,15 +50,32 @@ trait IndexListOption extends TypedTestOption[List[Index]]
 trait RequiredIntegerOption extends TypedTestOption[Option[Int]] {
   final override def required: Boolean = true
 }
-// procedures to leave out of a run, by id, each with the reason: a family of properties, one per procedure, under the
-// option's id, e.g. "test.exclude.8.26.2 = <reason>"
+// procedures to leave out of a run, by revision and id, each with the reason: a family of properties, one per
+// procedure, under the option's id, e.g. "test.exclude.3.1r1/8.26.2 = <reason>". Each is keyed by the rest of its key,
+// e.g. "3.1r1/8.26.2", which the revisions check: the options don't know them
 trait ExclusionsOption extends TypedTestOption[Map[String, String]] {
 
   // a property of the family, for the configurations written and listed, commented out
   def example: String
 
-  // the procedure id and reason of a property of the family, given the rest of its key after the option's id and "."
-  def entry(procedure: String, value: String): (String, String)
+  // the revision and procedure ids and reason of a property of the family, given the rest of its key after the
+  // option's id and "."
+  def entry(excluded: String, value: String): (String, String)
+
+  // between an exclusion's revision id and procedure id, which both have dots
+  final def separator: String = "/"
+
+  // an exclusion's key in the map, the rest of its property's key
+  final def key(revision: String, procedure: String): String = revision + separator + procedure
+
+  // the revision and procedure ids of an exclusion's key, split at the first separator, if it has one
+  final def split(excluded: String): Option[(String, String)] = excluded.indexOf(separator) match {
+    case -1 => None
+    case i => Some(excluded.take(i) -> excluded.drop(i + separator.length))
+  }
+
+  // how to write an exclusion, for messages
+  final def syntax: String = s"$id.${key("<revision id>", "<procedure id>")} = <reason>"
 }
 
 object OptionTypes {
@@ -122,13 +139,13 @@ object OptionTypes {
 
     // the family has no value of its own
     override def parse(s: String): Map[String, String] =
-      throw new Exception(s"exclude one procedure per property, with the reason: $id.<procedure id> = <reason>")
+      throw new Exception(s"exclude one procedure per property, with the reason: $syntax")
 
-    override def entry(procedure: String, value: String): (String, String) = {
-      if (procedure.isEmpty) throw new Exception(s"name the procedure: $id.<procedure id> = <reason>")
+    override def entry(excluded: String, value: String): (String, String) = {
+      if (excluded.isEmpty) throw new Exception(s"name the procedure: $syntax")
       val reason = value.trim
-      if (reason.isEmpty) throw new Exception(s"give the reason $procedure is excluded")
-      procedure -> reason
+      if (reason.isEmpty) throw new Exception(s"give the reason $excluded is excluded")
+      excluded -> reason
     }
   }
 

src/main/scala/io/stepfunc/dnp3c/conformance/api/options/TestOptions.scala

@@ -30,10 +30,12 @@ object TestOptions extends OptionGroup {
   val exclude: ExclusionsOption = OptionTypes.exclusions(
     this,
     "exclude",
-    "Procedures to leave out of a run, one property each, with the reason: test.exclude.<procedure id> = <reason>. " +
+    "Procedures to leave out of a run, one property each, with the reason: " +
+      "test.exclude.<revision id>/<procedure id> = <reason>. A run leaves out only its own revision's, so one " +
+      "configuration can exclude procedures of several revisions. " +
       "An excluded procedure applies to the DUT but isn't run, and is reported as not run, so the DUT isn't shown to " +
       "conform to it. A procedure for a feature the DUT lacks is skipped by its supports.* option instead",
-    "test.exclude.8.26.2 = Assign Class isn't supported yet"
+    "test.exclude.3.1r1/8.26.2 = Assign Class isn't supported yet"
   )
 
   def prefix: String = "test"