Skip to content

Pick your engine, and drive it from pytest in-process - #7

Merged
ShahriarAhnaf merged 5 commits into
mainfrom
agent/rust-backend-9c1e
Aug 24, 2026
Merged

Pick your engine, and drive it from pytest in-process#7
ShahriarAhnaf merged 5 commits into
mainfrom
agent/rust-backend-9c1e

Conversation

@ShahriarAhnaf

@ShahriarAhnaf ShahriarAhnaf commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Product statement from simantic-dev/simantic-core#183: pip install simantic gives one API and the user picks the engine. This branch delivers that, hardens the Python path against the pyrenode3 failure modes, and adds the pytest surface that makes it usable as a test harness.

1. Sim(..., backend="rust")

  • Hosts simantic_rust (simantic-dev/pyrite#109) in-process through a small backend adapter; the Renode path moves behind the same adapter unchanged. expect(), record paging and symbols are shared Python — one script, either engine.
  • Python takes over what the .NET engine did for the Rust path: .replx rendering (_replx.py), ELF .symtab lookup (_elf.py), mcu= via ~/.sim_cache/backend like sim --mcu.
  • Gaps raise simantic.NotSupported naming core#183 — never a silent no-op.
  • Wheel fetched on first use from releases/pyrite/latest.json (engine-rust-<rid>) into ~/.simantic/engine-rust/<version>; simantic install engine-rust.

2. Review pass against the pyrenode3 analysis

docs/competitors/pyrenode3.md §7 sets four rules for any CPython-against-a-sim path. Two gaps found and closed:

  • Batch across the boundary. UART bytes crossed one Python object per byte; they now cross as runs. 295x fewer objects on a 10 000-byte burst, pinned by a test that fails if the count regresses.
  • A version contract. The engine is now checked against a floor and refuses to load below it with a clear error, instead of failing later as a missing attribute.

3. The in-process pytest surface

The plugin ran every fixture through a sim subprocess — right for a manifest (one coarse pass/fail), wrong for hand-written tests that make many assertions against one running machine.

def test_timer_irq_fires(sim):
    s = sim(elf="fw.elf", mcu="STM32F401RE", uart="usart2")
    s.expect("fired=1", timeout=8)
  • Engine start-up paid once per worker instead of once per test; every machine a test makes is closed when it ends.
  • --sim-backend=renode|rust|both; both runs each test on each and names the engine in the test id.
  • A capability the chosen backend lacks becomes a skip with its reason, not a failure — so one suite targets both engines and reports honestly what each covers.
  • On failure the UART transcript is attached to the report. It is the useful evidence, and it is gone once the session closes.

Both backends are genuinely in-process: no subprocess anywhere in the Sim path, and a live renode-backend run spawns zero child processes (coreclr is hosted via pythonnet in the same process as pytest).

The measured discipline it encodes

Same firmware, same 70 ms of simulated time, actively computing throughout; only the number of Python check-ins changes:

check-ins Renode wall per check-in pyrite wall per check-in
1 0.42 s 0.34 s
1 000 0.86 s ~440 us
5 000 4.06 s ~730 us
10 000 0.36 s ~1 us

A chatty test is ~10x slower than the same test written coarsely on Renode, and the cost is the pause/resume, not the data — adding read_uart(), interrupts() and read_u32() per slice changes nothing. It is architectural: resuming rendezvouses with Renode's time-source dispatcher threads (MasterTimeSource.cs:127,202, SlaveTimeSource.cs:278), whereas rn_engine::Machine is !Send and single-threaded. So expect() — one crossing — is the documented default verb.

Deliberately not in this PR

Converting the test.yaml collector to the in-process path: manifests do not name a UART, and running the full timeout window rather than exiting at the last match would widen where expect_absent applies. Both need settling against the fixture suite first.

Verification

  • nucleo-f401re dwt-cycles on both backends from one script: RESULT: PASS, t=0.0660 s (rust) vs 0.0658 s (renode), same main, same memory word; 0.58 s vs 3.79 s wall.
  • New surface: 9 passed / 1 skipped on --sim-backend=both (the skip is the multi-machine test on rust, with its reason). A deliberate failure attaches the transcript.
  • Full suite: 116 passed, 6 skipped.

Bump to 0.3.0. Depends on pyrite#109 being released (engine-rust-v0.3.0) before the first cold backend="rust" run can fetch.

One Sim, two in-process engines. backend="rust" hosts the simantic_rust
extension (pyrite crates/simantic-rust) through a small backend adapter;
the Renode path moves behind the same adapter unchanged. expect(),
record paging and symbol lookup are shared Python, so a script runs on
either engine.

Rust-side pieces the .NET engine used to provide: .replx rendering
(_replx.py, defaults + arithmetic), ELF .symtab lookup (_elf.py), and
mcu= resolution via ~/.sim_cache / the backend (same as sim --mcu).
What the Rust engine lacks raises NotSupported naming simantic-core#183,
never a silent no-op.

The engine wheel is fetched on first use from the pyrite product
manifest (engine-rust-<rid>) into ~/.simantic/engine-rust/<version>;
simantic install engine-rust fetches it up front. Bump to 0.3.0.
From docs/competitors/pyrenode3.md's checklist for our own Python path:

- rule 3 (batch across the boundary): the Rust UART path crossed one
  Python object per byte. It now takes runs of bytes. Measured on the
  decode side alone, 1 MB of UART is 42.0 ms per-byte against 0.1 ms as
  runs -- 295x, before the pyo3 object construction it also removes.
- their 4.7 (no version contract): check_engine_version refuses an
  engine older than the Session API this package calls, naming the fix.
  An unversioned development directory still passes.
- their 4.8 (leaky error translation): engine exceptions are chained,
  so __cause__ keeps the original rather than only our summary.
705 ns per .NET->CPython crossing measured on this machine, so per-byte
traffic is what turns a display frame or a flash write into seconds of
pure overhead. Two tests pin what must stay batched: a 10,000-byte burst
crosses as runs (<= 4 objects, not 10,000), and observation is pulled in
bounded pages.
The plugin so far ran every fixture through the `sim` binary: one subprocess,
and with it ~3 s of engine start-up, per fixture. That is the right shape for a
manifest, which is one coarse pass/fail, but it is the wrong shape for
hand-written tests -- many assertions against one running machine.

Adds a `sim` fixture that drives the engine in-process, so start-up is paid once
per worker rather than once per test, and every machine a test makes is closed
when it ends. `--sim-backend=renode|rust|both` picks the engine; `both` runs
each test on each and names the engine in the test id. A capability the chosen
backend lacks becomes a skip with its reason rather than a failure, so one suite
can target both engines and report honestly what each covers. On failure the
UART transcript is attached to the report -- it is the useful evidence and it is
gone once the session closes.

The docstrings carry one measured discipline: on the Renode backend every
hand-off costs ~400-800 us, because resuming rendezvouses with the time-source
dispatcher threads (MasterTimeSource.cs:127,202, SlaveTimeSource.cs:278).
Reading is free; the pause/resume is not. So `expect()` -- one crossing -- is the
default verb, and a per-millisecond poll loop runs the same test ~10x slower. On
the Rust backend the same hand-off is ~1 us.

Converting the test.yaml collector to the in-process path is deliberately left
out: manifests do not name a UART, and running the full timeout window rather
than exiting at the last match would widen where expect_absent applies. Both
need settling against the fixture suite before that path changes.
@ShahriarAhnaf ShahriarAhnaf changed the title Sim(backend="rust"): the Rust engine behind the same API Pick your engine, and drive it from pytest in-process Aug 24, 2026
@ShahriarAhnaf
ShahriarAhnaf merged commit 167e7cc into main Aug 24, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant