V2 wire format

.github/workflows/release.yml

@@ -37,6 +37,12 @@ jobs:
       - uses: actions/setup-python@v6
         with:
           python-version: "3.12"
+      # The same gate the Python workflow applies on every push, repeated here:
+      # PyPI has no unpublish, so the upload below must not be the first time
+      # the tests are run against what is being shipped.
+      - run: pip install -e "python/.[diagram,dev]"
+      - run: pytest python/tests/ -v
+      - run: mypy python/src/execution_trace
       - run: pip install build
       - run: python -m build python/
       - uses: pypa/gh-action-pypi-publish@release/v1

.github/workflows/rust.yml

@@ -14,5 +14,5 @@ jobs:
       - uses: actions/checkout@v6
       - run: sudo apt-get install -y protobuf-compiler
       - run: cargo test --features std
-      - run: cargo clippy --features std -- -D warnings
+      - run: cargo clippy --all-targets --features std -- -D warnings
       - run: cargo fmt --check

CHANGELOG.md

@@ -0,0 +1,128 @@
+# Changelog
+
+All notable changes to the `execution-trace` crate are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+
+The Python decoder ships from this repository as
+[`embedded-etrace`](python/CHANGELOG.md) and is released from the same tag, so the two
+version numbers move together.
+
+## [0.2.0] - 2026-09-19
+
+**A breaking release.** The wire format is v2 and is not compatible with the v1 format
+0.1.x produced: a firmware built on 0.2.0 needs `embedded-etrace` 0.2.0 on the host, and
+a recording made with either version must be decoded by its own. A v2 stream is
+recognised by the `TRACE_START` frame at its head.
+
+### Wire format
+
+- Every frame class now travels in one flat `TraceFrame` message discriminated by
+  `event_type`, rather than a `oneof` envelope. proto3 omits unset fields, so an event
+  pays nothing for the dictionary fields it does not use, where an envelope taxes each
+  one with a tag and a length. Field numbers stay at or below 15 so every tag is a
+  single byte.
+- Names are interned. A `NAME_REGISTERED` frame carries the name once and assigns a
+  dictionary id, along with everything else the name fixes — source type, priority and
+  relative deadline — and per-occurrence events carry only the id.
+- Timestamps are deltas against the previous sequenced frame. `TRACE_START` and
+  `NAME_REGISTERED` carry an absolute timestamp and re-establish the origin.
+- `timestamp_ticks` is `sint64`, not unsigned. An event is stamped when recorded rather
+  than when queued, so an ISR preempting a task between those points legitimately
+  produces a negative delta; clamping it to zero drifted the host's reconstructed clock
+  permanently ahead of the device's, at milliseconds per second on a real trace. Zigzag
+  costs nothing at these magnitudes.
+- Sequence numbers wrap at 16 384 (`SEQUENCE_MODULUS`). The counter exists only to make
+  gaps visible, and a gap is read modulo the wrap, so a full 32-bit counter would spend
+  up to five varint bytes to buy nothing. Because a wrapped restart only looks backwards
+  from the low half of the range, `TRACE_START` — not the sequence — is now the reliable
+  reset signal.
+- A new `TRACE_START` header declares the timebase (nanoseconds, or core cycles with
+  `core_frequency_hz`) and the source-group mask the build was compiled with, so a host
+  can tell a masked group from one whose frames were lost without hard-coding the mask.
+
+Measured steady state is 11–12 bytes per event over the delta range the reference
+firmware produces, against 27 for v1.
+
+### Added
+
+- `TraceEncoder`, holding the three pieces of per-stream state the format needs: the
+  name dictionary, the delta base, and the sequence counter. Built for its clock with
+  `TraceEncoder::new()` (nanoseconds) or `TraceEncoder::with_cycle_counter(hz)`.
+- `TraceEncoder::encode_dictionary_entry`, for walking the dictionary and re-emitting it
+  periodically. Without this, a host attaching mid-run receives no dictionary at all and
+  can resolve nothing — names are registered in the first control cycles after boot, and
+  the device cannot observe an attach over a one-way transport.
+- `TraceEncoder::encode_trace_start`, `registered_names`, and the `Encoded` result, whose
+  `name_registry_full` flag reports that an event went out under `UNKNOWN_NAME_ID`
+  because the dictionary was full.
+- `RawTraceFrame`, `FrameKind` and `TimeBase`, the decoded form of a single frame.
+- `next_sequence()`, the process-global producer-side counter.
+- `MAX_TRACE_BURST_SIZE`, `NAME_REGISTRY_CAPACITY`, `SEQUENCE_MODULUS`, `UNKNOWN_NAME_ID`.
+- `TraceSink::ticks_per_us`, `tick_mask` and `ticks_since`, so that code measuring an
+  interval works in whichever timebase the sink reports. All three default to the
+  nanosecond case, so existing implementations need no change.
+
+### Changed
+
+- **`TraceSink::get_elapsed_nanoseconds` is now `now_ticks`.** The unit is whatever the
+  encoder for the stream declares, so the old name would be a lie on a cycle-counter
+  sink. A sink may return a free-running 32-bit counter widened to `u64`: the encoder
+  extends it, so reading the clock can be a single volatile load with no critical
+  section.
+- **`decode_trace_frame` returns `RawTraceFrame`, not `TraceEvent`.** In v2 an event is
+  not recoverable from one frame — its name is a dictionary id and its timestamp is a
+  delta — so name resolution and delta accumulation belong to the host decoder, which
+  holds the per-stream state.
+- **Frames are numbered at the producer**, by the recording layer immediately before it
+  hands the event to the transport, and by the encoder for the header and dictionary
+  frames it originates itself. Loss at the producer queue used to happen before any
+  number existed, so the host saw a contiguous stream with events simply absent; it now
+  leaves a hole. Nothing on the wire changed — only who fills the field.
+  - The cost is that arrival order is no longer numeric order: a dictionary frame is
+    written ahead of the event that triggered it but numbered after it, inverting a pair
+    on every first sight of a name. Hosts must tolerate a reorder window.
+- `NAME_REGISTRY_CAPACITY` is 40, down from 64. This array is the largest thing the
+  format puts in RAM, and under flip-link every byte of `.bss` is a byte the stack does
+  not get. 40 is half again as many entries as the reference firmware registers, and
+  overshooting degrades the trace rather than breaking it.
+- Interning is keyed on name content rather than on the `&'static str` pointer. By the
+  time an event reaches the encoder the literal's pointer is gone — `TraceEvent` carries
+  an inline copy. The scan is linear over at most `NAME_REGISTRY_CAPACITY` entries, with
+  a length check before any byte comparison, in the task that owns the wire.
+
+### Removed
+
+- `SequenceEncoder`, replaced by `TraceEncoder`. Sequencing moved to the producer and the
+  encoder now owns the dictionary and the delta base as well, so the old wrapper no
+  longer describes anything.
+
+### Fixed
+
+- A name first seen on a `SPAN_END` — its `SPAN_START` dropped upstream under load — is
+  backfilled when a `SPAN_START` for it arrives, so a dictionary refresh cannot re-send a
+  bare entry and lose that name's priority for the rest of the run.
+- A marker value of `0` is written to the wire instead of being dropped as falsy.
+
+## [0.1.6] - 2026-05-28
+
+### Added
+
+- The `enabled` feature flag. With it off, the recording API compiles to no-ops with
+  default implementations, needing neither a clock nor a transport, and the protobuf
+  codegen is skipped entirely.
+
+### Changed
+
+- README: dropped a Cargo.toml example that showed a less clean way to disable tracing.
+
+## [0.1.5] - 2026-05-25
+
+### Changed
+
+- Improved the crate and Python READMEs.
+
+## [0.1.4] - 2026-05-24
+
+Releases up to and including this one predate this file; see the git history for
+0.1.0–0.1.4.

Cargo.toml

@@ -9,7 +9,7 @@ license = "MIT OR Apache-2.0"
 repository = "https://github.com/erl987/execution-trace"
 keywords = ["embedded", "tracing", "no_std", "profiling", "rtos"]
 categories = ["development-tools::profiling", "embedded", "no-std"]
-include = ["/src/**", "/proto/**", "/build.rs", "/examples/simulate.rs", "/README.md", "/LICENSE-MIT", "/LICENSE-APACHE"]
+include = ["/src/**", "/proto/**", "/build.rs", "/examples/simulate.rs", "/README.md", "/CHANGELOG.md", "/LICENSE-MIT", "/LICENSE-APACHE"]
 
 [lints.clippy]
 unwrap_used = "deny"
@@ -23,7 +23,7 @@ enabled = ["dep:heapless", "dep:micropb"]
 std = []
 
 [dependencies]
-heapless = { version = "0.9.2", optional = true }
+heapless = { version = "0.9.3", optional = true }
 micropb = { version = "0.6.0", default-features = false, features = ["encode", "decode", "container-heapless-0-9", "enable-64bit"], optional = true }
 
 [build-dependencies]

README.md

@@ -36,7 +36,7 @@ impl TraceTransport for MyRttSink {
 }
 
 impl TraceSink for MyRttSink {
-    fn get_elapsed_nanoseconds(&self) -> u64 {
+    fn now_ticks(&self) -> u64 {
         0 // replace with your hardware timer
     }
 }
@@ -61,50 +61,97 @@ fn ukf_step(sink: &mut impl TraceSink) {
 }
 ```
 
-### 3. Encode with sequence tracking
+### 3. Encode with `TraceEncoder`
 
-Use [`SequenceEncoder`] when encoding events manually so the host can detect dropped frames:
+[`TraceEncoder`] holds the three pieces of per-stream state the wire format needs: the name
+dictionary, the timestamp base the deltas are taken against, and the sequence counter that
+lets the host detect dropped frames. Emit the stream header once, then encode events:
 
 ```rust
-use execution_trace::{SequenceEncoder, SourceType, TraceEvent, encode::MAX_TRACE_FRAME_SIZE};
+use execution_trace::{SourceType, TraceEncoder, TraceEvent};
+use execution_trace::encode::MAX_TRACE_BURST_SIZE;
+
+// `new()` reads timestamps as 64-bit nanoseconds. For a sink returning a raw
+// core cycle counter use `TraceEncoder::with_cycle_counter(72_000_000)`: the
+// encoder then extends the 32-bit counter itself, so reading it costs the
+// caller one volatile load.
+let mut enc = TraceEncoder::new();
+let mut buf = [0u8; MAX_TRACE_BURST_SIZE];
+
+// Once, at startup: declares the tick unit and the active source mask.
+if let Ok(n) = enc.encode_trace_start(0, 0x1F, &mut buf) {
+    let _ = &buf[..n];
+}
 
 let mut name = heapless::String::<32>::new();
 name.push_str("my_task").unwrap();
 let event = TraceEvent::SpanStart {
-    timestamp_ns: 0,
+    timestamp_ns: 1_000,
     name,
     source_type: SourceType::Task,
     sequence: 0,
     priority: 4,
     relative_deadline_ms: None,
 };
-let mut enc = SequenceEncoder::new();
-let mut buf = [0u8; MAX_TRACE_FRAME_SIZE];
-if let Ok(n) = enc.encode(&event, &mut buf) {
-    // forward buf[..n] over your transport (RTT, UART, USB, etc.)
-    let _ = &buf[..n];
+if let Ok(encoded) = enc.encode(&event, &mut buf) {
+    if encoded.name_registry_full {
+        // The dictionary is full: the event went out with the reserved "unknown"
+        // id. Count it — the stream stays decodable, but this name is lost.
+    }
+    // forward buf[..encoded.len] over your transport (RTT, UART, USB, etc.)
+    let _ = &buf[..encoded.len];
 }
 ```
 
+The first sight of a name writes **two** frames — the dictionary entry, then the event — which
+is why the buffer is [`encode::MAX_TRACE_BURST_SIZE`] rather than one frame.
+
 ### 4. Decode on the host
 
+A frame is not self-contained: its name is a dictionary id and its timestamp is a delta against
+the previous frame. [`encode::decode_trace_frame`] therefore returns a [`RawTraceFrame`], and the reader
+resolves both from state it carries across the stream:
+
 ```rust,ignore
-use execution_trace::encode::decode_trace_frame;
+use execution_trace::{FrameKind, encode::decode_trace_frame};
 
 // raw_bytes arrives from your transport (RTT, UART, file, etc.)
-let (event, consumed) = decode_trace_frame(raw_bytes).unwrap();
+let (frame, consumed) = decode_trace_frame(raw_bytes).unwrap();
+
+// A dictionary entry or the stream header carries an absolute timestamp and
+// re-establishes the origin; every other frame is a delta against it.
+clock = match frame.kind {
+    FrameKind::NameRegistered | FrameKind::TraceStart => frame.timestamp_ticks,
+    _ => clock + frame.timestamp_ticks,
+};
+if frame.kind == FrameKind::NameRegistered {
+    names.insert(frame.name_id, frame.name);
+}
 ```
 
+`examples/simulate.rs` carries a complete worked reader; the `execution-trace` Python package
+does the same job and renders a timing diagram from it.
+
 ## Wire format
 
 Each frame is a standard protobuf length-delimited record:
 
 ```text
-[ varint: payload byte count ][ protobuf-encoded TraceEvent ]
+[ varint: payload byte count ][ protobuf-encoded TraceFrame ]
 ```
 
-Maximum frame size is [`encode::MAX_TRACE_FRAME_SIZE`] (128 bytes). Name strings are capped at 32 bytes;
-longer names cause `record_*` to return [`TracingError::MessageDropped`] before sending.
+One flat message carries every frame class, discriminated by its `event_type`: the three
+per-occurrence events, the dictionary entry that assigns a name its id, and the stream header.
+proto3 omits unset fields, so an event pays nothing for the dictionary fields it does not use.
+
+A steady-state event costs **11-13 bytes** — the name, the priority and the deadline are sent
+once per name rather than on every occurrence, and the timestamp is a delta.
+
+Maximum frame size is [`encode::MAX_TRACE_FRAME_SIZE`] (128 bytes), and one `encode` call writes
+at most [`encode::MAX_TRACE_BURST_SIZE`]. Name strings are capped at 32 bytes; longer names cause
+`record_*` to return [`TracingError::MessageDropped`] before sending. The dictionary holds
+[`encode::NAME_REGISTRY_CAPACITY`] distinct names, after which events fall back to a reserved
+"unknown" id rather than to a wrong decode.
 
 ## Host-side tooling
 

build.rs

@@ -19,8 +19,9 @@ fn main() {
     generator.add_protoc_arg("-Iproto");
 
     // name carries the name of the source task/ISR (for spans) or the marker label.
+    // It appears only on NAME_REGISTERED frames, once per distinct name (§5.5).
     generator.configure(
-        ".tracing.TraceEvent.name",
+        ".tracing.TraceFrame.name",
         micropb_gen::Config::new()
             .string_type("heapless::String<$N>")
             .max_bytes(32),