Skip to content

ci(examples): run the Python example gates on the code they depend on - #1702

Merged
dekobon merged 1 commit into
mainfrom
ci/examples-coverage
Oct 23, 2026
Merged

dekobon merged 1 commit into
mainfrom
ci/examples-coverage

Conversation

@dekobon

@dekobon dekobon commented Oct 23, 2026

Copy link
Copy Markdown
Owner

Summary

The Python example gates (test_book_examples.py, the CLI-parity rows in test_smoke.py / test_sarif.py, and the jupyter_quickstart.ipynb nbconvert job) were already in CI, but three things let them not run, or run against the wrong inputs. This closes those gaps without adding dependencies, action pins, or a second example inventory.

Changes

.github/workflows/ci.yml

  1. Path filter covers the code the examples depend on. The py filter listed only big-code-analysis-py/** and the root manifests. The bindings crate path-depends on the root crate, the parse layer and the five vendored grammars, and cli_parity.py / test_smoke.py byte-compare against the bca CLI — so a change to any of those left every python-* job skipped. Added src/**, big-code-analysis-ast/**, big-code-analysis-cli/** and tree-sitter-{ccomment,mozcpp,mozjs,preproc,tcl}/**. Docs/book/web-only PRs still skip; the ci aggregator's skipped tolerance is unchanged.
  2. Explicit, --locked CLI build for the parity examples. The bca_binary fixture built bca inside pytest without --locked. python-test now builds it in its own step and exports BCA_BINARY via $GITHUB_ENV (so it reaches the PowerShell pytest step on Windows, .exe suffix included). The fixture pytest.fails on a bad path, so this cannot degrade into a silent skip.
  3. BCA_NOTEBOOK_FIXTURES set on the nbconvert step. The notebook documents it as the contract CI uses; CI never set it.

big-code-analysis-py/tests/test_book_examples.py

  1. test_every_example_is_covered now reconciles *.ipynb against NON_PY_EXAMPLES, so a second notebook cannot land with nothing executing it. Single inventory kept.

Cost

Item 1 means python-lint, the 6-leg python-test matrix and python-examples-nbconvert now run on most code PRs rather than only bindings PRs. That is the intended trade — the alternative is parity tests that skip exactly when the CLI output changes. Item 2 adds no compile work: the fixture already ran the same cargo build on every leg; it is now visible and locked.

Validation

Check Result
make actionlint pass
cargo build -p big-code-analysis-cli --locked ok
venv from requirements/dev.txt --require-hashes + maturin develop + BCA_BINARY=… pytest tests/test_book_examples.py tests/test_smoke.py -v 77 passed, 0 skipped; all 4 test_cli_parity* pass
same with BCA_BINARY=/nonexistent/bca 4 ERRORs — fixture fails loudly
stray examples/zz_unclaimed.ipynb exactly test_every_example_is_covered fails
venv from requirements/examples.txt --require-hashes + exact nbconvert command with BCA_NOTEBOOK_FIXTURES exit 0, 6/6 code cells executed
same with a wrong fixtures path exit 1 via the notebook's own hello.rs assertion
make py-fmt-check py-lint py-typecheck clean
make pre-commit BCA_GATE: pass (gate=pre-commit)

Windows/macOS legs were not exercised locally; this PR's own matrix run is the test for the $GITHUB_ENV → pwsh handoff.

Out of scope

  • Hardening the notebook against a markdown-only rewrite (nbconvert passes on a notebook with no code cells). The current notebook is not vacuous — its first cell asserts hello.rs exists — but a positive-output check would be a separate change.
  • Reusing the test job's bca artifact in python-test instead of recompiling. Would save a cold-cache compile per leg but couples two jobs and changes the feature set the parity binary is built with.

The example tests under big-code-analysis-py (test_book_examples.py
and the parity rows in test_smoke.py / test_sarif.py) and the
jupyter_quickstart.ipynb nbconvert job were already wired into CI,
but three things let them not run, or run against the wrong inputs:

- The `py` path filter listed only the bindings crate and the root
  manifests, so a change to src/, the parse layer, the CLI or a
  vendored grammar left every python-* job `skipped` even though the
  extension links all of them and the parity examples byte-compare
  against the CLI. Add those directories to the filter.
- The `bca` the parity examples compare against was built implicitly
  inside the `bca_binary` fixture, without `--locked` and outside the
  job log. Build it in an explicit step and export `BCA_BINARY` via
  `$GITHUB_ENV` so it reaches the PowerShell pytest step on Windows.
- The notebook documents `BCA_NOTEBOOK_FIXTURES` as the contract the
  CI step uses, but the step never set it. Set it.

The runnable-example discovery gate also only enumerated `*.py`, so a
second notebook could land with nothing executing it; it now
reconciles `*.ipynb` against `NON_PY_EXAMPLES` too. Verified by
dropping a stray notebook into examples/ and watching exactly that
test fail.

Validated locally with the environments CI provisions: the dev and
examples requirement exports installed with --require-hashes, maturin
develop, pytest against the --locked CLI (77 passed, all cli_parity
rows passing not skipping), the exact nbconvert command with the env
var set (6/6 cells executed), and `make pre-commit` (BCA_GATE: pass).
@dekobon
dekobon merged commit e113773 into main Oct 23, 2026
58 checks passed
@dekobon
dekobon deleted the ci/examples-coverage branch October 23, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant