Pick your engine, and drive it from pytest in-process - #7
Merged
Conversation
One Sim, two in-process engines. backend="rust" hosts the simantic_rust extension (pyrite crates/simantic-rust) through a small backend adapter; the Renode path moves behind the same adapter unchanged. expect(), record paging and symbol lookup are shared Python, so a script runs on either engine. Rust-side pieces the .NET engine used to provide: .replx rendering (_replx.py, defaults + arithmetic), ELF .symtab lookup (_elf.py), and mcu= resolution via ~/.sim_cache / the backend (same as sim --mcu). What the Rust engine lacks raises NotSupported naming simantic-core#183, never a silent no-op. The engine wheel is fetched on first use from the pyrite product manifest (engine-rust-<rid>) into ~/.simantic/engine-rust/<version>; simantic install engine-rust fetches it up front. Bump to 0.3.0.
From docs/competitors/pyrenode3.md's checklist for our own Python path: - rule 3 (batch across the boundary): the Rust UART path crossed one Python object per byte. It now takes runs of bytes. Measured on the decode side alone, 1 MB of UART is 42.0 ms per-byte against 0.1 ms as runs -- 295x, before the pyo3 object construction it also removes. - their 4.7 (no version contract): check_engine_version refuses an engine older than the Session API this package calls, naming the fix. An unversioned development directory still passes. - their 4.8 (leaky error translation): engine exceptions are chained, so __cause__ keeps the original rather than only our summary.
705 ns per .NET->CPython crossing measured on this machine, so per-byte traffic is what turns a display frame or a flash write into seconds of pure overhead. Two tests pin what must stay batched: a 10,000-byte burst crosses as runs (<= 4 objects, not 10,000), and observation is pulled in bounded pages.
The plugin so far ran every fixture through the `sim` binary: one subprocess, and with it ~3 s of engine start-up, per fixture. That is the right shape for a manifest, which is one coarse pass/fail, but it is the wrong shape for hand-written tests -- many assertions against one running machine. Adds a `sim` fixture that drives the engine in-process, so start-up is paid once per worker rather than once per test, and every machine a test makes is closed when it ends. `--sim-backend=renode|rust|both` picks the engine; `both` runs each test on each and names the engine in the test id. A capability the chosen backend lacks becomes a skip with its reason rather than a failure, so one suite can target both engines and report honestly what each covers. On failure the UART transcript is attached to the report -- it is the useful evidence and it is gone once the session closes. The docstrings carry one measured discipline: on the Renode backend every hand-off costs ~400-800 us, because resuming rendezvouses with the time-source dispatcher threads (MasterTimeSource.cs:127,202, SlaveTimeSource.cs:278). Reading is free; the pause/resume is not. So `expect()` -- one crossing -- is the default verb, and a per-millisecond poll loop runs the same test ~10x slower. On the Rust backend the same hand-off is ~1 us. Converting the test.yaml collector to the in-process path is deliberately left out: manifests do not name a UART, and running the full timeout window rather than exiting at the last match would widen where expect_absent applies. Both need settling against the fixture suite before that path changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Product statement from simantic-dev/simantic-core#183:
pip install simanticgives one API and the user picks the engine. This branch delivers that, hardens the Python path against the pyrenode3 failure modes, and adds the pytest surface that makes it usable as a test harness.1.
Sim(..., backend="rust")simantic_rust(simantic-dev/pyrite#109) in-process through a small backend adapter; the Renode path moves behind the same adapter unchanged.expect(), record paging and symbols are shared Python — one script, either engine..replxrendering (_replx.py), ELF.symtablookup (_elf.py),mcu=via~/.sim_cache/backend likesim --mcu.simantic.NotSupportednaming core#183 — never a silent no-op.releases/pyrite/latest.json(engine-rust-<rid>) into~/.simantic/engine-rust/<version>;simantic install engine-rust.2. Review pass against the pyrenode3 analysis
docs/competitors/pyrenode3.md§7 sets four rules for any CPython-against-a-sim path. Two gaps found and closed:3. The in-process pytest surface
The plugin ran every fixture through a
simsubprocess — right for a manifest (one coarse pass/fail), wrong for hand-written tests that make many assertions against one running machine.--sim-backend=renode|rust|both;bothruns each test on each and names the engine in the test id.Both backends are genuinely in-process: no
subprocessanywhere in theSimpath, and a live renode-backend run spawns zero child processes (coreclr is hosted via pythonnet in the same process as pytest).The measured discipline it encodes
Same firmware, same 70 ms of simulated time, actively computing throughout; only the number of Python check-ins changes:
A chatty test is ~10x slower than the same test written coarsely on Renode, and the cost is the pause/resume, not the data — adding
read_uart(),interrupts()andread_u32()per slice changes nothing. It is architectural: resuming rendezvouses with Renode's time-source dispatcher threads (MasterTimeSource.cs:127,202,SlaveTimeSource.cs:278), whereasrn_engine::Machineis!Sendand single-threaded. Soexpect()— one crossing — is the documented default verb.Deliberately not in this PR
Converting the
test.yamlcollector to the in-process path: manifests do not name a UART, and running the full timeout window rather than exiting at the last match would widen whereexpect_absentapplies. Both need settling against the fixture suite first.Verification
dwt-cycleson both backends from one script:RESULT: PASS, t=0.0660 s (rust) vs 0.0658 s (renode), samemain, same memory word; 0.58 s vs 3.79 s wall.--sim-backend=both(the skip is the multi-machine test on rust, with its reason). A deliberate failure attaches the transcript.Bump to 0.3.0. Depends on pyrite#109 being released (
engine-rust-v0.3.0) before the first coldbackend="rust"run can fetch.