Why
The CLI's failure flags are hard to reason about. Users need to know whether they're asking Vizzly to wait for results, fail on visual differences, or fail when visual testing couldn't finish. The current names and help text don't clearly separate those choices:
- Global
--strict says “Fail on any error,” but only changes a missing-build response in finalize. Even there, API 5xx errors still succeed in strict mode.
- TDD's
--fail-on-diff throws at the screenshot call, which can stop that test before later captures.
- Cloud
run --wait both waits for comparisons and can return a nonzero exit for visual differences. Its help only says “Wait for build completion.”
These distinctions affect whether CI stays green and whether the suite captures all its screenshots. They should be deliberate and visible in the docs.
What
Audit failure behavior across run, upload, finalize, and TDD, then make the flags, help, and documentation agree. Keep this separate from the current upload-error reporting work; this issue does not require expanding that change.
Start with a small behavior table covering ordinary test failures, visual differences, rejected uploads, API/network failures, missing builds, and comparison timeouts. Distinguish failing the wrapped test at capture time from returning a failing CLI exit after capture finishes.
Keep the default resilient to Vizzly upload/service issues when wrapping a passing test suite. Preserve the user's test failure code when their tests fail. Define an explicit opt-in policy for callers who require complete visual results. Reusing --strict is a candidate, not a settled design; don't add another overlapping flag without resolving the existing ones.
Update command help, CLI references, and the documentation site's overview, CI, scripting, SDK, and troubleshooting guidance together. Explain supported flag combinations and any intentional command-specific differences. Treat changes to existing flags as user-facing compatibility decisions, rather than silently changing CI behavior.
Implementation starting points: src/cli.js, src/commands/{run,upload,finalize,tdd}.js, src/client/index.js, and src/test-runner/operations.js. Nearby coverage includes tests/commands/finalize.test.js, the command subprocess suites, and tests/helpers/cli-runner.js.
Open questions
- Should
--strict make all incomplete visual runs fail, including API outages, or have a narrower meaning?
- Should
--wait continue to imply failure on visual differences, and how should that relationship be expressed alongside --fail-on-diff?
- Which differences between local capture-time failures and cloud end-of-run failures are useful to keep?
Testing
Run real CLI commands against a controlled external HTTP API and test process. Verify terminal output, JSON results, exit codes, whether later captures still run, and the resulting build status for the agreed behavior table. Cover combined flags and a nonzero test exit so Vizzly cannot overwrite the test's failure code. Mock only external boundaries; no internal-module mocks or arbitrary sleeps.
Done
Why
The CLI's failure flags are hard to reason about. Users need to know whether they're asking Vizzly to wait for results, fail on visual differences, or fail when visual testing couldn't finish. The current names and help text don't clearly separate those choices:
--strictsays “Fail on any error,” but only changes a missing-build response infinalize. Even there, API 5xx errors still succeed in strict mode.--fail-on-diffthrows at the screenshot call, which can stop that test before later captures.run --waitboth waits for comparisons and can return a nonzero exit for visual differences. Its help only says “Wait for build completion.”These distinctions affect whether CI stays green and whether the suite captures all its screenshots. They should be deliberate and visible in the docs.
What
Audit failure behavior across
run,upload,finalize, and TDD, then make the flags, help, and documentation agree. Keep this separate from the current upload-error reporting work; this issue does not require expanding that change.Start with a small behavior table covering ordinary test failures, visual differences, rejected uploads, API/network failures, missing builds, and comparison timeouts. Distinguish failing the wrapped test at capture time from returning a failing CLI exit after capture finishes.
Keep the default resilient to Vizzly upload/service issues when wrapping a passing test suite. Preserve the user's test failure code when their tests fail. Define an explicit opt-in policy for callers who require complete visual results. Reusing
--strictis a candidate, not a settled design; don't add another overlapping flag without resolving the existing ones.Update command help, CLI references, and the documentation site's overview, CI, scripting, SDK, and troubleshooting guidance together. Explain supported flag combinations and any intentional command-specific differences. Treat changes to existing flags as user-facing compatibility decisions, rather than silently changing CI behavior.
Implementation starting points:
src/cli.js,src/commands/{run,upload,finalize,tdd}.js,src/client/index.js, andsrc/test-runner/operations.js. Nearby coverage includestests/commands/finalize.test.js, the command subprocess suites, andtests/helpers/cli-runner.js.Open questions
--strictmake all incomplete visual runs fail, including API outages, or have a narrower meaning?--waitcontinue to imply failure on visual differences, and how should that relationship be expressed alongside--fail-on-diff?Testing
Run real CLI commands against a controlled external HTTP API and test process. Verify terminal output, JSON results, exit codes, whether later captures still run, and the resulting build status for the agreed behavior table. Cover combined flags and a nonzero test exit so Vizzly cannot overwrite the test's failure code. Mock only external boundaries; no internal-module mocks or arbitrary sleeps.
Done