Repository navigation
Exclude procedures through the configuration, reported as not run - #86
Merged
Merged
Conversation
A property per procedure, test.exclude.<id> = <reason>, leaves a procedure out of a run: it's recorded as excluded without touching the device or the plugin, and the run carries on. The report shows it as not run, with the reason, counts it apart from skips, and warns that conformance hasn't been demonstrated for it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A run stops at the first procedure that fails, so a device with a known gap can't get past it. The dnp3 outstation, for example, stops at 3.1r1's 8.26.2 (Assign Class, stepfunc/dnp3#480), and the sections after it never run. The only way to leave a procedure out was a
supports.*option. That's wrong for a gap: it changes what other procedures expect, and it reports the procedure as skipped, as if it didn't apply to the device.This adds exclusions to the configuration, one property per procedure, each with a reason:
result=excluded, with the reason). These replace the procedure's earlier result, log and capture when a run reaches it.-reportnever changes results.allare errors. It also needs a non-empty reason.-run,-describeand-disabledcheck every exclusion, not only those in what they cover. A procedure that's skipped anyway stays skipped.-disabledlists exclusions with their reasons.-describeshows an excluded procedure's reason and marks excluded procedures in a section.ProcedureOutcome.Excluded(reason)is new, so code that matches on outcomes must handle it.RunSummary.exclusionslists the excluded procedures in order.succeeded, and the CLI's exit code, mean the run completed without a failure, not that the device conforms. A CI job that must fail on exclusions can checksummary.exclusions.nonEmpty.IntegrationPluginandHarnessare unchanged.README, docs/manual-testing.md and docs/automation.md describe the syntax, the rules, and how exclusions differ from feature options and skips.
Closes #85