Repository navigation
ci(examples): run the Python example gates on the code they depend on - #1702
Merged
Merged
Conversation
The example tests under big-code-analysis-py (test_book_examples.py and the parity rows in test_smoke.py / test_sarif.py) and the jupyter_quickstart.ipynb nbconvert job were already wired into CI, but three things let them not run, or run against the wrong inputs: - The `py` path filter listed only the bindings crate and the root manifests, so a change to src/, the parse layer, the CLI or a vendored grammar left every python-* job `skipped` even though the extension links all of them and the parity examples byte-compare against the CLI. Add those directories to the filter. - The `bca` the parity examples compare against was built implicitly inside the `bca_binary` fixture, without `--locked` and outside the job log. Build it in an explicit step and export `BCA_BINARY` via `$GITHUB_ENV` so it reaches the PowerShell pytest step on Windows. - The notebook documents `BCA_NOTEBOOK_FIXTURES` as the contract the CI step uses, but the step never set it. Set it. The runnable-example discovery gate also only enumerated `*.py`, so a second notebook could land with nothing executing it; it now reconciles `*.ipynb` against `NON_PY_EXAMPLES` too. Verified by dropping a stray notebook into examples/ and watching exactly that test fail. Validated locally with the environments CI provisions: the dev and examples requirement exports installed with --require-hashes, maturin develop, pytest against the --locked CLI (77 passed, all cli_parity rows passing not skipping), the exact nbconvert command with the env var set (6/6 cells executed), and `make pre-commit` (BCA_GATE: pass).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Python example gates (
test_book_examples.py, the CLI-parity rows intest_smoke.py/test_sarif.py, and thejupyter_quickstart.ipynbnbconvert job) were already in CI, but three things let them not run, or run against the wrong inputs. This closes those gaps without adding dependencies, action pins, or a second example inventory.Changes
.github/workflows/ci.ymlpyfilter listed onlybig-code-analysis-py/**and the root manifests. The bindings crate path-depends on the root crate, the parse layer and the five vendored grammars, andcli_parity.py/test_smoke.pybyte-compare against thebcaCLI — so a change to any of those left everypython-*jobskipped. Addedsrc/**,big-code-analysis-ast/**,big-code-analysis-cli/**andtree-sitter-{ccomment,mozcpp,mozjs,preproc,tcl}/**. Docs/book/web-only PRs still skip; theciaggregator'sskippedtolerance is unchanged.--lockedCLI build for the parity examples. Thebca_binaryfixture builtbcainside pytest without--locked.python-testnow builds it in its own step and exportsBCA_BINARYvia$GITHUB_ENV(so it reaches the PowerShell pytest step on Windows,.exesuffix included). The fixturepytest.fails on a bad path, so this cannot degrade into a silent skip.BCA_NOTEBOOK_FIXTURESset on the nbconvert step. The notebook documents it as the contract CI uses; CI never set it.big-code-analysis-py/tests/test_book_examples.pytest_every_example_is_coverednow reconciles*.ipynbagainstNON_PY_EXAMPLES, so a second notebook cannot land with nothing executing it. Single inventory kept.Cost
Item 1 means
python-lint, the 6-legpython-testmatrix andpython-examples-nbconvertnow run on most code PRs rather than only bindings PRs. That is the intended trade — the alternative is parity tests that skip exactly when the CLI output changes. Item 2 adds no compile work: the fixture already ran the samecargo buildon every leg; it is now visible and locked.Validation
make actionlintcargo build -p big-code-analysis-cli --lockedrequirements/dev.txt --require-hashes+maturin develop+BCA_BINARY=… pytest tests/test_book_examples.py tests/test_smoke.py -vtest_cli_parity*passBCA_BINARY=/nonexistent/bcaexamples/zz_unclaimed.ipynbtest_every_example_is_coveredfailsrequirements/examples.txt --require-hashes+ exact nbconvert command withBCA_NOTEBOOK_FIXTUREShello.rsassertionmake py-fmt-check py-lint py-typecheckmake pre-commitBCA_GATE: pass (gate=pre-commit)Windows/macOS legs were not exercised locally; this PR's own matrix run is the test for the
$GITHUB_ENV→ pwsh handoff.Out of scope
hello.rsexists — but a positive-output check would be a separate change.testjob'sbcaartifact inpython-testinstead of recompiling. Would save a cold-cache compile per leg but couples two jobs and changes the feature set the parity binary is built with.