Skip to content

rust backend: threads, interrupts and heap instead of NotSupported - #11

Open
ShahriarAhnaf wants to merge 3 commits into
mainfrom
agent/py-observability-5c31
Open

rust backend: threads, interrupts and heap instead of NotSupported#11
ShahriarAhnaf wants to merge 3 commits into
mainfrom
agent/py-observability-5c31

Conversation

@ShahriarAhnaf

Copy link
Copy Markdown
Contributor

backend="rust" could run firmware and read its UART, but every question about what the firmware was doing raised NotSupported. The engine had the answers all along — pyrite's #101 layer — they simply were not plumbed through.

Needs simantic-dev/pyrite#150, which makes Session serve them. Without it this PR's new paths do nothing.

Now supported

  • threads() — the kernel's tasks, in the same {"rtos", "threads", "truncated"} shape the Renode backend returns.
  • heap() — allocator accounting.
  • interrupts() — records fill during run_for, like uart.
  • trace_interrupts=True — no longer refused. The engine's exception hook is always on, so interrupts() is served either way; the argument stays accepted so one test runs unchanged on both backends.

trace_symbols= still raises: pyrite has no PC-hook binding yet, and pretending otherwise would hand back an empty list that reads as "nothing hit it".

Two deliberate gaps

  • largestFreeBlockBytes / fragmentationRatio are absent from heap() rather than guessed. This allocator view has no free-list walk, and a fabricated fragmentation number is worse than a missing key.
  • _vector_name returns "" outside the architecturally-defined Cortex-M range. On RISC-V vector is mcause, where the same integers mean something else; a wrong name costs more than none.

Where the new views live

switches(), task_usage() and isr_usage() have no Renode counterpart, so they sit on the backend (sim._b) rather than on Sim. That keeps Sim one interface across both backends instead of a union of them. Worth a second opinion — the alternative is putting them on Sim and having them raise on Renode.

Tests

7 new, against the fake engine so they run without a Rust build (this file's existing convention), covering the bare-metal None path, the RISC-V naming rule, and the cursor bug the cumulative ISR log invites — it is not drained by reading, unlike the UART queue, so a second run_for would otherwise duplicate every earlier record.

Full suite 130 passed / 2 skipped. Also exercised end-to-end against real FreeRTOS and Zephyr images through a built wheel; results in the pyrite PR.

`backend="rust"` could run firmware and read its UART, but every question
about what the firmware was *doing* raised NotSupported. The engine had the
answers all along -- pyrite's #101 layer -- they just were not plumbed
through. With `Session` now serving them, this is the Python half.

  threads()     the kernel's tasks, in the same {"rtos", "threads",
                "truncated"} shape the Renode backend returns
  heap()        allocator accounting
  interrupts()  records fill during run_for, like uart

`trace_interrupts=True` is no longer refused. The engine's exception hook is
always on, so interrupts() is served either way; the argument stays accepted
so one test runs unchanged on both backends. `trace_symbols=` still raises --
pyrite has no PC-hook binding yet, and pretending otherwise would return an
empty list that reads as "nothing hit it".

Two honest gaps, both deliberate:

- `largestFreeBlockBytes` / `fragmentationRatio` are absent from heap()
  rather than guessed. This allocator view has no free-list walk, and a
  fabricated fragmentation number is worse than a missing key.
- `_vector_name` returns "" outside the architecturally-defined Cortex-M
  range. On RISC-V `vector` is `mcause`, where the same integers mean
  something else entirely; a wrong name costs more than no name.

`switches()`, `task_usage()` and `isr_usage()` have no Renode counterpart, so
they are on the backend (`sim._b`) rather than on `Sim` -- keeping `Sim`
itself one interface across both backends rather than a union of them.

The ISR log is cumulative and is not drained by reading, unlike the UART
queue, so _advance re-slices from a cursor; a test covers the duplicate that
caused.

Tests: 7 new against the fake engine (so they run without a Rust build, per
this file's convention), including the bare-metal None path and the RISC-V
naming rule. Full suite 130 passed / 2 skipped. Also exercised end-to-end
against real FreeRTOS and Zephyr images through a built wheel.
It was hardcoded False. That was wrong, and wrong in the direction that hides
the problem: without the kernel's all-threads list the adapter sees only the
running thread, so a one-entry list marked complete reads as "this firmware
has one thread".

Caught on the irq-timer fixture. `threads()` returned a single unnamed thread;
`switches()` over the same run showed three distinct task ids, two of which
decode to `z_main_thread` and `z_idle_threads` in the ELF. The view was
one-third of the truth and said it was all of it.

The fixture is built without CONFIG_THREAD_MONITOR (no all-threads list),
CONFIG_THREAD_NAME (no names), CONFIG_DEBUG_THREAD_INFO (no published offsets,
so the adapter runs on its assumed layout) and CONFIG_INIT_STACKS (no paint,
hence peakUsedBytes=None). A stock Zephyr build has none of them, so this is
the ordinary case.

What the adapter did report on that assumed layout cross-checks exactly
against the ELF -- id 0x20000080 = z_idle_threads, stack size 320 =
CONFIG_IDLE_STACK_SIZE, priority 15 = NUM_PREEMPT_PRIORITIES, base
0x20000c40 = z_idle_stacks + guard -- so the numbers are right. They were
just incomplete, which is exactly what `truncated` exists to say.
@ShahriarAhnaf

Copy link
Copy Markdown
Contributor Author

Follow-up: truncated was hardcoded False, and that was a real bug.

Caught by testing against the Zephyr fixture rather than the fake. threads() returned a single unnamed thread; switches() over the same run showed three distinct task ids, two of which decode in the ELF to z_main_thread and z_idle_threads. The view was one-third of the truth and reported itself as complete — the failure mode that hides rather than announces itself.

truncated now comes from the adapter's own enumeration_available().

Why the Zephyr fixture looks sparse — it is the build, not the adapter. Read straight out of the ELF's Kconfig symbols:

config value consequence
CONFIG_THREAD_MONITOR absent no all-threads list → only the running thread
CONFIG_THREAD_NAME absent names are ""
CONFIG_DEBUG_THREAD_INFO absent no offsets table → adapter runs on its assumed layout
CONFIG_INIT_STACKS absent no paint → peakUsedBytes: None
CONFIG_THREAD_STACK_INFO 1 which is why base/size do come through

A stock Zephyr build has none of the first four, so this is the ordinary case and the API has to be honest about it rather than treat it as exotic.

The numbers it did report are correct, which is worth stating since they came from a guessed layout. Four independent cross-checks against the ELF:

id 0x20000080     = z_idle_threads
stack size 320    = CONFIG_IDLE_STACK_SIZE (0x140)
priority 15       = CONFIG_NUM_PREEMPT_PRIORITIES (lowest)
stack base 0xc40  = z_idle_stacks (0xc00) + guard

So the assumed layout reads this Zephyr version correctly. The data was right and incomplete — exactly what truncated exists to say.

Suite now 131 passed / 2 skipped.

pyrite #164 added a Zephyr sys_heap adapter and made `minimum_free`
optional: the chunk chain describes the heap as it is now and records no
history, so there is nothing to derive a peak from. ESP-IDF's multi_heap
keeps its own low-water mark, so it still has both.

`minimumFreeBytes` and `peakUsedBytes` now go None together rather than
`pool - None` raising. Both keys stay present so a caller can tell "not
tracked" from "nothing used".

Also document the debugging surface in docs/session-api.md: the shape of
threads() and heap(), which Kconfigs each field needs, why the Zephyr
chunk walk validates itself against the kernel's own sentinel, and the
switches()/task_usage()/isr_usage() observers the Rust backend adds.
@ShahriarAhnaf

Copy link
Copy Markdown
Contributor Author

Propagated the pyrite side, now that #162 and #164 are on pyrite main.

HeapStats::minimum_free is Option<u32> there: pyrite gained a Zephyr sys_heap adapter, and the chunk chain describes the heap as it is now and records no history, so there is nothing to derive a peak from. ESP-IDF's multi_heap keeps its own low-water mark and still has both.

minimumFreeBytes and peakUsedBytes now go None together instead of pool - None raising. Both keys stay present so a caller can tell "not tracked" from "nothing used".

Verified end-to-end against the real engine (built from pyrite main @ 49e52e8) on the nucleo-f401re/sys-heap fixture:

{"allocator": "Zephyr sys_heap", "arenaSizeBytes": 4180,
 "freeBytes": 3820, "usedBytes": 360,
 "minimumFreeBytes": 3532, "peakUsedBytes": 648, "regions": 1}

freeBytes matches what the firmware itself prints from Zephyr's incrementally-maintained counters, derived structurally instead.

Also documents the debugging surface in docs/session-api.md — the shape of threads() and heap(), which Kconfigs each field needs, why the chunk walk validates itself, and the switches()/task_usage()/isr_usage() observers this backend adds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant