Skip to content

Validate the scanner argument before initializing the benchmark. - #354

Merged
Pomidorry merged 6 commits into
Samsung:mainfrom
deep-sengupta:fix/validate-scanner-before-init
Oct 26, 2026
Merged

Pomidorry merged 6 commits into
Samsung:mainfrom
deep-sengupta:fix/validate-scanner-before-init

Conversation

@deep-sengupta

Copy link
Copy Markdown
Contributor

Problem

main() previously constructed Benchmark() before checking whether the supplied scanner name was supported. This could trigger unnecessary dataset preparation for invalid input. The manual error branch also returned normally, resulting in a successful exit status for invalid scanner names.

Additionally, nargs="?" allowed --scanner to be supplied without a value.

Changes

  • Add choices=SCANNER_LIST to the --scanner argument.
  • Remove nargs="?" so the scanner value is required.
  • Remove the redundant manual scanner validation.
  • Add regression tests for an unknown scanner name and a missing scanner value, verifying that Benchmark is not initialized.

Testing

  • New regression tests pass.
  • Existing unit tests pass.
  • python -m benchmark -h displays the required scanner argument.
  • Invalid scanner input exits with status 2 before benchmark initialization.

No full benchmark run was performed because this change only affects argument validation.

@deep-sengupta
deep-sengupta requested a review from a team as a code owner October 22, 2026 13:05

@babenek babenek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!
please update

def get_arguments() -> Namespace:

to

def get_arguments(argv:list[str]) -> Namespace:

and parse incoming arguments

also in main

def main(argv) -> None:
    args = get_arguments(argv[1:])

if __name__ == """__main__""":
    sys.exit(main(sys.argv))

then avoid extra patching in tests

@deep-sengupta

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback! I've made the requested changes and verified that all 9 tests pass.

@deep-sengupta
deep-sengupta requested a review from babenek October 23, 2026 20:15
Comment thread benchmark/__main__.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread test_benchmark_args.py

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mocking should be a last resort. Prefer real objects and test units directly by default, without invoking their parent functions just to reach them.
Function main() is easy byself.
Test only get_arguments() instead:

  1. test with unsupported scanner
  2. test with supported scanner (credsweeper ;)
    no auxiliary methods required!

@babenek babenek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code - OK
Test - rewrite

@deep-sengupta

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback! I've rewritten the tests to call get_arguments() directly, without mocking or auxiliary methods. The tests now cover both unsupported and supported scanner values.

@deep-sengupta
deep-sengupta requested a review from babenek October 24, 2026 14:14

@babenek babenek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @deep-sengupta

@Pomidorry Pomidorry left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Pomidorry
Pomidorry merged commit ff66381 into Samsung:main Oct 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants