Skip to content

Exclude procedures through the configuration, reported as not run - #86

Merged
jadamcrain merged 2 commits into
mainfrom
exclude-procedures
Oct 15, 2026
Merged

jadamcrain merged 2 commits into
mainfrom
exclude-procedures

Conversation

@jadamcrain

Copy link
Copy Markdown
Member

A run stops at the first procedure that fails, so a device with a known gap can't get past it. The dnp3 outstation, for example, stops at 3.1r1's 8.26.2 (Assign Class, stepfunc/dnp3#480), and the sections after it never run. The only way to leave a procedure out was a supports.* option. That's wrong for a gap: it changes what other procedures expect, and it reports the procedure as skipped, as if it didn't apply to the device.

This adds exclusions to the configuration, one property per procedure, each with a reason:

test.exclude.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
  • Not run. An excluded procedure isn't configured or run: no connection to the device, no capture, no plugin calls. The run carries on to the next procedure.
  • Recorded. It gets a log saying so and a result with its own value (result=excluded, with the reason). These replace the procedure's earlier result, log and capture when a run reaches it. -report never changes results.
  • Excluded is not skipped. Skipped means the procedure doesn't apply to the device or its configuration. Excluded means it applies but wasn't run, so conformance isn't shown.
    • The report labels it "NOT RUN — EXCLUDED", shows the reason under its title, counts it on its own, and gives no capture link.
    • With any exclusion, the report says at the top that conformance hasn't been demonstrated for those procedures.
    • Procedures with no saved result read "Not run — no saved result", and unimplemented ones "Not implemented".
  • Validated. Each exclusion must name a procedure of the revision; unknown ids, section ids and all are errors. It also needs a non-empty reason. -run, -describe and -disabled check every exclusion, not only those in what they cover. A procedure that's skipped anyway stays skipped.
  • CLI.
    • Each exclusion a run reaches is printed with its reason, and the run ends with the count and the conformance warning, even after a failure.
    • -disabled lists exclusions with their reasons.
    • -describe shows an excluded procedure's reason and marks excluded procedures in a section.
  • API. ProcedureOutcome.Excluded(reason) is new, so code that matches on outcomes must handle it. RunSummary.exclusions lists the excluded procedures in order. succeeded, and the CLI's exit code, mean the run completed without a failure, not that the device conforms. A CI job that must fail on exclusions can check summary.exclusions.nonEmpty. IntegrationPlugin and Harness are unchanged.

README, docs/manual-testing.md and docs/automation.md describe the syntax, the rules, and how exclusions differ from feature options and skips.

Closes #85

A property per procedure, test.exclude.<id> = <reason>, leaves a procedure
out of a run: it's recorded as excluded without touching the device or the
plugin, and the run carries on. The report shows it as not run, with the
reason, counts it apart from skips, and warns that conformance hasn't been
demonstrated for it.
@jadamcrain
jadamcrain added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit ba36362 Oct 15, 2026
5 checks passed
@jadamcrain
jadamcrain deleted the exclude-procedures branch October 15, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exclude procedures through the configuration, reported as not run

1 participant