Problem
The README smoke test silently exercises none of the README commands. All documented shell commands use a $ prompt, while the extractor only accepts lines starting directly with northstar. Pytest skips the empty parameterization and the suite stays green.
Verified evidence
Reviewed on 2026-09-17 at 2de6b83f21ccacae04f8050f5691d0dd8159fda1 (local checkout matched GitHub main). Reproduced in isolated macOS environments with Python 3.14.6. Framework default probe environment: Click 8.5.0 and Typer 0.27.2. Demo probes used released base-cli 0.4.3 unless noted.
From the repository root, evaluated readme_commands() in tests/test_readme_examples.py: returned [] (zero commands). All-extras suite: 35 passed, 1 skipped; minimal/dev suite: 34 passed, 2 skipped. One skip is this empty parameterization. The two hard-coded JSON tests still run, so their success does not prove the README walkthrough.
Once prompt stripping is fixed, examples using examples/northstar-commerce.json also need an explicit fixture policy because run_installed_command changes cwd to a temporary directory.
Sources:
Acceptance criteria
- Parse the documented prompt-prefixed command form, with well-defined fenced-block handling.
- Fail explicitly if zero runnable commands are discovered or expected README scenarios disappear.
- Execute the actual discovered commands against the installed application outside the source tree, copying required example configuration fixtures into the isolated cwd.
- Add extractor regressions for prompted/unprompted commands and prove a deliberately invalid README command fails the smoke test.
Related work
Distinct from #22: pytest itself currently passes without README coverage, even if the authoritative validation entry point is fixed. Related YAML scenario availability remains #21.
Project fields
- Status: Backlog
- Priority: P2
- Area: CI
- Initiative: Base-CLI Demo
- Size: S
- Assignee: @codeforester
- Milestone: v0.1.0
- Target date: unscheduled
Problem
The README smoke test silently exercises none of the README commands. All documented shell commands use a
$prompt, while the extractor only accepts lines starting directly with northstar. Pytest skips the empty parameterization and the suite stays green.Verified evidence
Reviewed on 2026-09-17 at
2de6b83f21ccacae04f8050f5691d0dd8159fda1(local checkout matched GitHub main). Reproduced in isolated macOS environments with Python 3.14.6. Framework default probe environment: Click 8.5.0 and Typer 0.27.2. Demo probes used released base-cli 0.4.3 unless noted.From the repository root, evaluated
readme_commands()in tests/test_readme_examples.py: returned[](zero commands). All-extras suite: 35 passed, 1 skipped; minimal/dev suite: 34 passed, 2 skipped. One skip is this empty parameterization. The two hard-coded JSON tests still run, so their success does not prove the README walkthrough.Once prompt stripping is fixed, examples using
examples/northstar-commerce.jsonalso need an explicit fixture policy because run_installed_command changes cwd to a temporary directory.Sources:
Acceptance criteria
Related work
Distinct from #22: pytest itself currently passes without README coverage, even if the authoritative validation entry point is fixed. Related YAML scenario availability remains #21.
Project fields