Skip to content

A plain-script run always reports passed #309

Description

@vishnuv688

Part of #277

A plain-script run reports passed whatever happened. _send_default_suite is
called with a hardcoded state at
packages/selenium-devtools-py/src/selenium_devtools/instrumentation.py:356:

_send_default_suite(capturer, "passed")  # the whole script run is done

and the adapter has no failure path at all. The dashboard shows a green 1/1 passed for a run that raised, which is the one signal a user reads at a glance.

Reproduced on examples/selenium/python-test/login.py with a deliberately
broken assertion: the tree shows 1/1 passed while the Actions list stops dead
at the getElementText that preceded the failing line, with the logout click
and final getCurrentUrl absent. The command stream tells the truth; the tree
contradicts it.

pytest runs are unaffected. _external_suites suppresses the synthetic
suite entirely, and pytest_plugin derives each state from
report.passed/failed/skipped. This is the plain-script path only.

Why it is not a one-line change

The "passed" is sent from driver.quit(), which runs inside the user's
finally before the exception reaches top level — so at that moment the
adapter genuinely does not know the outcome yet. The final state has to move out
of the quit path and into teardown:

  • install a sys.excepthook wrapper in enable() that records the failure, and
    send the final suite state from disable() / the atexit handler, which run
    after the hook
  • sys.exc_info() inside the quit path is the tempting shortcut — it does
    return the in-flight exception during a finally — but it also returns one
    when the caller is merely inside an unrelated except block, so it would
    report false failures

Also worth deciding here: a plain script has no test boundaries, so "the run
failed" is the most the adapter can honestly say.

Acceptance criteria

A plain script that raises reports failed for its synthetic suite and test; a
clean run still reports passed; an unrelated except block in the user's code
does not make a passing run report failed. pytest runs are unchanged. Unit tests
cover all three.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingselenium-devtools-pyPython Selenium adapter for the DevTools dashboard

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions