Skip to content

Qualify exclusions by revision, so one configuration serves every revision #87

Closed

Description

@jadamcrain

#86 checks each test.exclude.<id> against the revision being run, and an id the revision doesn't have is a configuration error. That breaks the one-configuration-for-every-revision design (#12): a configuration that excludes a 3.1r1 procedure can't run 2.7.

configuration has 1 invalid option(s):
  test.exclude.8.26.2 = Assign Class (function code 22) isn't supported yet (stepfunc/dnp3#480): revision 2.7 has no procedure 8.26.2

stepfunc/dnp3 runs both revisions in CI from one device.ini, and needs to exclude 3.1r1's 8.26.2.

Unqualified ids are also ambiguous across revisions, even where both have the id. 8.23 is Device Attributes in 2.7 and Specific Point Ranges in 3.1r1, so test.exclude.8.23.2.1 would exclude a different test depending on the revision.

Proposal

Qualify each exclusion with the revision it's for:

test.exclude.3.1r1.8.26.2 = Assign Class isn't supported yet (stepfunc/dnp3#480)
  • A run applies only the exclusions for its own revision.
  • Validation:
    • An unknown revision id is an error, as for -rev.
    • An id the named revision doesn't have is an error, whichever revision is running.
    • Exclusions for other revisions are otherwise ignored.
  • Everything else from Exclude procedures through the configuration, reported as not run #86 stays as it is: the not-run result, the report, and -disabled.

To decide

  • Whether to drop the unqualified form, or keep it for configurations used with a single revision. Dropping it avoids the ambiguity above.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions