Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 19 additions & 16 deletions crates/vt_bin/tests/e2e_snapshots/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# E2E Snapshot Tests

End-to-end tests that execute the `vp` binary and verify its output.
End-to-end tests that execute the `vt` binary and verify its output.

## When to add tests here

Expand All @@ -11,48 +11,51 @@ End-to-end tests that execute the `vp` binary and verify its output.

## How it works

Each fixture in `fixtures/` is a self-contained workspace. Tests are defined in `snapshots.toml`:
Each fixture in `fixtures/` is a self-contained workspace. Tests are defined in `snapshots.toml`. A step is the argv of one command, as an array:

```toml
[[e2e]]
name = "descriptive test name"
name = "descriptive_test_name"
steps = [
"vp run build",
"vp run build", # second run to test caching
["vt", "run", "build"],
["vt", "run", "build"], # second run to test caching
]
```

Steps also support an object form with interactions:
Steps also support a table form with more options:

```toml
[[e2e]]
name = "interactive step"
name = "interactive_step"
steps = [
{ command = "vp interact", interactions = [{ expect-milestone = "ready" }, { write = "hello" }, { write-line = "world" }, { write-key = "up" }, { write-key = "down" }, { write-key = "enter" }] },
"echo -n | node check-stdin.js",
{ argv = ["vt", "run", "dev"], comment = "cache miss", envs = [["MY_ENV", "1"]], interactions = [{ expect-milestone = "ready" }, { write = "hello" }, { write-line = "world" }, { write-key = "up" }, { write-key = "down" }, { write-key = "enter" }] },
["vtt", "pipe-stdin", "hello", "--", "vt", "run", "read-stdin"],
]
```

Notes:

- String steps are shorthand for `{ command = "..." }`.
- `write-key` accepts `up`, `down`, and `enter`.
- An array step is shorthand for `{ argv = [...] }`. A string step such as `"vt run build"` doesn't parse, and the test binary panics while loading the fixtures, before any case runs.
- A table step takes `argv` (required), `cwd` (relative to the fixture root, defaulting to the case's `cwd`), `comment` (shown under the step's heading in the snapshot), `envs` (`[name, value]` pairs), `interactions`, and `formatted-snapshot` (keep colors and styles in the snapshot). Other keys are an error.
- A case takes `name` (ASCII letters, digits, and `_` only; it also names the snapshot file), `steps`, and optionally `comment`, `cwd`, `cfg` (a Rust cfg expression such as `'target_os = "linux"'`; the case runs only where it matches), and `ignore` (the case runs only with `-- --ignored`).
- `vt` and `vtt` are the binaries built for the test. Other programs are looked up in `PATH`, with `packages/tools/node_modules/.bin` (when installed) ahead of the inherited entries.
- `write-key` accepts `up`, `down`, `enter`, `escape`, `backspace`, and `ctrl-c`.
- Snapshots include every interaction line, and each `expect-milestone` records the screen at that point.
- For stdin pipe scenarios, write the step command with shell piping, for example: `echo -n | command`.
- Steps don't run in a shell, so pipes, `&&`, and redirections aren't available. To pipe data to a command's stdin, use `vtt pipe-stdin <data> -- <command> [args...]`.

The test runner:

1. Copies the fixture to a temp directory
2. Executes each step using `/bin/sh` (Unix) or `bash` (Windows)
3. Runs each step in PTY mode (`TestTerminal`)
2. Starts each step's `argv` directly, without a shell, in an environment holding only `PATH`, `TERM`, and the step's `envs` (plus `PLAYWRIGHT_HOST_PLATFORM_OVERRIDE` when set, and a few system variables on Windows)
3. Runs each step in PTY mode (`TestTerminal`), with a timeout of 20 seconds (60 seconds on Windows and x86_64 macOS)
4. Applies configured interactions in order for PTY steps
5. Captures output and exit codes
6. Compares against snapshot in `fixtures/<name>/snapshots/`
6. Compares against the snapshot `fixtures/<fixture>/snapshots/<name>.md`

## Adding a new test

1. Create a new fixture directory under `fixtures/`
2. Add `package.json` (and `pnpm-workspace.yaml` for monorepos)
3. Add `snapshots.toml` with test cases
4. Run `cargo test -p vt_bin --test e2e_snapshots`
5. Review and accept new snapshots
5. Review the new snapshots, written next to the expected ones as `<name>.md.new`, and accept them by re-running with `UPDATE_SNAPSHOTS=1`
2 changes: 1 addition & 1 deletion crates/vt_bin/tests/e2e_snapshots/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ struct StepConfig {
/// Defaults to the case-level `cwd`.
#[serde(default)]
cwd: Option<RelativePathBuf>,
/// Appended as `# comment` in the snapshot display line.
/// Shown under the step's heading in the snapshot.
#[serde(default)]
comment: Option<Str>,
/// Extra environment variables set for this step.
Expand Down
Loading