Exclude procedures through the configuration, reported as not run

README.md

@@ -68,6 +68,8 @@ dnp3c -rev 2.7 -help                             # list the commands and test se
 
 Results, including `report.html`, are written to the revision's folder of the directory set by `test.output.directory`, e.g. `results/2.7/L2/`. The configuration must set `outstation.subset.level`, the DNP3 subset level the device is configured for: it has no default. See [Testing a device by hand](docs/manual-testing.md) for the whole workflow.
 
+A procedure the device can't pass yet, e.g. for a known gap, can be excluded in the configuration, with the reason: `test.exclude.8.26.2 = Assign Class isn't supported yet`. It isn't run, and the report shows it as not run, with the reason, apart from skipped procedures, which don't apply to the device. See [Exclude procedures](docs/manual-testing.md#exclude-procedures).
+
 ## Library
 
 The library jar, `io.stepfunc:dnp3c`, is published to Maven Central. See [Automating the tests](docs/automation.md) for writing an integration plugin and running it in CI.

docs/automation.md

@@ -40,7 +40,7 @@ object Main {
 - `Options.load` parses and checks the whole file, and throws a `ConfigurationException` listing every problem.
 - The revision decides which procedures exist and what their ids mean. `Revisions.list` holds every revision dnp3c has, and `Revisions.find("2.7")` looks one up by id. There's no default.
 - The configuration must set `outstation.subset.level`, the subset level the device is configured for, to a level the revision tests: level 2 for 2.7. `run` and `writeReport` throw a `ConfigurationException` before doing anything if it doesn't. `run` also does if the configuration lacks a value the revision, or a procedure that will run, needs: e.g. a Device Profile value that a test checks the device's responses against.
-- `run` takes a procedure or section id of the revision, e.g. `"8.6.5.4"` or `"8.6"`, or `"all"`, and throws an `IllegalArgumentException` for an unknown one before anything runs. It runs the procedures in order and stops at the first that fails. It returns a `RunSummary` with each procedure's outcome (`Passed`, `Failed(message)` or `Skipped(reason)`) and the procedures it didn't reach.
+- `run` takes a procedure or section id of the revision, e.g. `"8.6.5.4"` or `"8.6"`, or `"all"`, and throws an `IllegalArgumentException` for an unknown one before anything runs. It runs the procedures in order and stops at the first that fails. It returns a `RunSummary` with each procedure's outcome (`Passed`, `Failed(message)`, `Skipped(reason)` or `Excluded(reason)`) and the procedures it didn't reach.
 - Results go in the folder for the revision and subset level in the output directory, e.g. `results/2.7/L2/`, so runs of different revisions or levels never mix.
 - `writeReport` writes `report.html` in that folder from every result there, from this run and earlier ones, and returns its path. `Harness.writeReport(options, revision)` does the same without a plugin.
 
@@ -50,7 +50,27 @@ For each procedure, the harness:
 2. connects to the DUT over TCP and runs the steps, calling the plugin for each step it can't perform itself;
 3. calls `endProcedure`, closes the connection, and writes the procedure's results.
 
-A procedure that won't run is skipped before any of that: one that isn't implemented, doesn't apply at the subset level, or needs a feature the device lacks. It gets a log and a result saying why, but the harness doesn't connect to the DUT or call the plugin, not even `startProcedure` or `endProcedure`.
+A procedure that won't run is skipped before any of that: one that isn't implemented, doesn't apply at the subset level, or needs a feature the device lacks. So is one the configuration excludes. It gets a log and a result saying why, but the harness doesn't connect to the DUT, capture or call the plugin, not even `startProcedure` or `endProcedure`.
+
+### Known gaps
+
+A device with a known gap would stop every run at the procedure that tests it. Exclude the procedure in the configuration, with the reason, and the run carries on past it:
+
+```
+test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
+```
+
+[Testing a device by hand](manual-testing.md#exclude-procedures) has the rules: one property per procedure of the revision, each with a reason, checked before anything runs, and a procedure that's skipped stays skipped. Exclude a procedure for a feature the device has; turn off a `supports.*` option only for one it lacks.
+
+An excluded procedure's outcome is `Excluded(reason)`, which is new: a match over `ProcedureOutcome` must handle it. Unlike `Skipped`, it means the procedure applies to the device but didn't run, so the device hasn't been shown to conform to it. The report shows it as not run, with the reason, and says that conformance hasn't been demonstrated.
+
+`RunSummary.succeeded` says whether the run completed without a failure, not whether the device conforms: a run with exclusions succeeds. `summary.exclusions` lists the procedures it excluded, in order. For a CI job that must fail until every gap is closed:
+
+```scala
+if (!summary.succeeded || summary.exclusions.nonEmpty || !reported) sys.exit(1)
+```
+
+An exclusion replaces a procedure's earlier result only when a run reaches it, as running it does: run the procedure, its section or `all`. `writeReport` never changes results.
 
 The plugin runs in the same JVM as dnp3c. It can control a device elsewhere, e.g. through a test fixture, an API or a simulator, or it can host the device itself. A software outstation can be started in `startProcedure` and stopped in `endProcedure`, so each procedure starts from a known state.
 
@@ -75,7 +95,7 @@ To get the whole command-line tool, with its commands and options, but with your
 io.stepfunc.dnp3c.conformance.Main.run(args, new MyDevicePlugin())
 ```
 
-`Main.run` exits the JVM with 0 if every procedure passed or was skipped, and nonzero otherwise.
+`Main.run` exits the JVM with 0 if every procedure passed, was skipped or was excluded, and nonzero otherwise.
 
 ## Dependency
 

docs/manual-testing.md

@@ -62,11 +62,28 @@ The procedures are organized in sections: 6 (link layer), 7 (transport layer) an
 ```
 dnp3c -rev 2.7 -config device.ini -describe 8.6       # the procedures in a section
 dnp3c -rev 2.7 -config device.ini -describe 8.6.5.4   # the steps of one procedure
-dnp3c -rev 2.7 -config device.ini -disabled           # optional procedures your configuration turns off
+dnp3c -rev 2.7 -config device.ini -disabled           # procedures your configuration skips or excludes
 ```
 
 Procedures for features the DUT doesn't support are optional. If a `supports.*` option turns one off, it's recorded as skipped instead of run.
 
+## Exclude procedures
+
+A procedure that applies to the DUT, but that you can't run yet, e.g. for a known gap in the DUT that's being fixed, can be left out of runs by excluding it in the configuration, with the reason:
+
+```
+test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
+```
+
+- **One property per procedure,** `test.exclude.<procedure id>`, and each needs a reason. Only procedures can be excluded, not sections or `all`. Every exclusion must name a procedure of the revision given with `-rev`, even one outside what you run: `-run`, `-describe` and `-disabled` check them all, and fail on any that doesn't.
+- **An excluded procedure isn't run.** dnp3c doesn't connect to the DUT, capture or prompt for it, and the run carries on to the next procedure. It's recorded as excluded, with a log saying so, replacing the procedure's earlier result, log and capture.
+- **Excluded isn't skipped.** A skipped procedure doesn't apply to the DUT or its configuration, and the DUT can still be conformant. An excluded one does apply, so the DUT hasn't been shown to conform to it. The report shows it as not run, with the reason, and counts it apart from passed, failed and skipped procedures. A run that excludes procedures still succeeds, but it ends by saying how many it excluded.
+- **A procedure that's skipped stays skipped,** e.g. one that doesn't apply at the DUT's subset level, even if it's excluded too.
+- **Options describe the DUT; exclusions record gaps in the testing.** Don't turn a `supports.*` option off to leave out a procedure for a feature the DUT has: that changes what other procedures expect of it.
+- **Only a run replaces results.** Removing an exclusion doesn't change the report until the procedure runs again, and adding one doesn't replace an earlier result until a run reaches the procedure: run it, its section or `all`. `-report` never changes results.
+
+`-describe` shows an excluded procedure's reason instead of its steps, and marks the excluded procedures in a section. `-disabled` lists them with their reasons.
+
 ## Run
 
 `-run` takes the id of any procedure or section, or `all`:
@@ -86,7 +103,7 @@ Some questions come after a procedure's steps, where stopping for your answer wo
 
 If you miss a time limit, the procedure fails with a message saying so; rerun it. A limit that's too short to work to by hand is usually a timeout on the DUT that you can lengthen, along with its option here.
 
-A run stops at the first procedure that fails, and the procedures after it don't run. The exit code is 0 if every procedure passed or was skipped, and nonzero otherwise.
+A run stops at the first procedure that fails, and the procedures after it don't run. The exit code is 0 if every procedure passed, was skipped or was excluded, and nonzero otherwise.
 
 ## Results
 
@@ -100,13 +117,13 @@ results/
       report.css
       8.6.5.4/
         log.html           every step, frame and check, with the outcome
-        results.properties the result (pass, fail or skip), the revision and subset level, and why it failed or was skipped
+        results.properties the result (pass, fail, skip or excluded), the revision and subset level, and why it failed, was skipped or was excluded
         dump.pcap          the packet capture, if capture is on
 ```
 
 A procedure that won't run, because the DUT lacks a feature it needs, it doesn't apply at the subset level, or it isn't implemented, is recorded as skipped without connecting to the DUT.
 
-The report is rebuilt after every run, including runs that fail. It names the revision and the subset level, and shows every procedure: the ones you've run with their result (and, for one that was skipped, why, or for one that passed, anything it noted, such as a device that complies with IEEE 1815-2012 but not 1815-202x), and the rest as not run. To rebuild it without running anything, use `dnp3c -rev 2.7 -config device.ini -report`.
+The report is rebuilt after every run, including runs that fail. It names the revision and the subset level, and shows every procedure: the ones you've run with their result (and, for one that was skipped or excluded, why, or for one that passed, anything it noted, such as a device that complies with IEEE 1815-2012 but not 1815-202x), and the rest as not run, with no saved result. Excluded procedures are shown as "NOT RUN — EXCLUDED", and if there are any, the report says at the top that conformance hasn't been demonstrated for them. To rebuild it without running anything, use `dnp3c -rev 2.7 -config device.ini -report`.
 
 The report counts only results saved under its revision and level. It lists anything else it finds at the top and leaves it out: a result saved under another revision or level, or a procedure whose last run didn't finish (it has a log but no result).
 
@@ -161,8 +178,8 @@ https://stepfunc.io
  -config <arg>       Path to configuration file
  -describe <arg>     Describe the steps to be executed in the specified
                       procedure
- -disabled           List all optional test procedures that are disabled by
-                      configuration
+ -disabled           List the test procedures the configuration skips or
+                      excludes
  -help               Print this help information
  -options            List all of the available configuration file options
  -report             Generate an HTML report containing all the results

src/main/resources/report.css

@@ -16,6 +16,8 @@
   --fail-bg: #fbe4e6;
   --skip: #8a6400;
   --skip-bg: #fff3cd;
+  --excluded: #6a1b9a;
+  --excluded-bg: #f3e5f5;
   --mono: SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", monospace;
 }
 
@@ -100,6 +102,7 @@ h1 {
 .total.pass { color: var(--pass); }
 .total.fail { color: var(--fail); }
 .total.skip { color: var(--skip); }
+.total.excluded { color: var(--excluded); }
 .total.no-result { color: var(--sf-grey); }
 
 main {
@@ -197,7 +200,7 @@ tr.incomplete > td:first-child {
 }
 
 /* the result column, shown as a badge */
-td.pass a, td.fail a, td.skip a {
+td.pass a, td.fail a, td.skip a, td.excluded a {
   display: inline-block;
   min-width: 52px;
   padding: 0 10px;
@@ -211,9 +214,10 @@ td.pass a, td.fail a, td.skip a {
 td.pass a { color: var(--pass); background-color: var(--pass-bg); }
 td.fail a { color: var(--fail); background-color: var(--fail-bg); }
 td.skip a { color: var(--skip); background-color: var(--skip-bg); }
+td.excluded a { color: var(--excluded); background-color: var(--excluded-bg); white-space: nowrap; }
 
 /* a procedure's id (underlined, as links are) and its result badge open its log */
-td.pass a:hover, td.fail a:hover, td.skip a:hover {
+td.pass a:hover, td.fail a:hover, td.skip a:hover, td.excluded a:hover {
   text-decoration: underline;
 }
 
@@ -229,6 +233,25 @@ p.notice {
   border-radius: 4px;
 }
 
+/* procedures the configuration excluded, which the device hasn't been shown to conform to */
+p.exclusions {
+  color: var(--excluded);
+  background-color: var(--excluded-bg);
+  padding: 6px 10px;
+  border-radius: 4px;
+}
+
+p.exclusions strong {
+  font-weight: 600;
+}
+
+/* why a procedure was excluded, under its title */
+span.exclusion {
+  color: var(--excluded);
+  font-size: 12px;
+  font-weight: 600;
+}
+
 /* what a procedure noted about a pass, under its title */
 span.note {
   color: var(--pass);

src/main/scala/io/stepfunc/dnp3c/conformance/Harness.scala

@@ -34,13 +34,14 @@ final class Harness(options: Options, plugin: IntegrationPlugin, revision: Revis
    * Run the procedures of a procedure or section, or "all" of them, in order, stopping at the first that fails.
    *
    * A procedure fails when one of its steps or validations does, including a plugin action or verification that
-   * throws.
+   * throws. One the configuration excludes (`test.exclude.<id>`) isn't run: it's recorded as excluded, without
+   * touching the device or the plugin, and the run carries on.
    *
    * @param id a procedure or section of the revision, e.g. "8.6.5.4" or "8.6", or "all"
    * @throws IllegalArgumentException if the revision has no procedure or section with the id, before anything runs
    * @throws api.ConfigurationException if the configuration has no subset level, or one the revision doesn't test, or
-   *                                    lacks what the revision, or a procedure that will run, needs configured,
-   *                                    before anything runs
+   *                                    excludes something that isn't a procedure of the revision, or lacks what the
+   *                                    revision, or a procedure that will run, needs configured, before anything runs
    * @throws HarnessExecutionException if the harness or the plugin fails around a procedure, e.g. when
    *                                   `startProcedure` throws or the capture can't be opened; it holds the
    *                                   results so far. An interruption propagates as it is.