diff --git a/README.md b/README.md index a3b565ee..3f85e489 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Drive large LED installations and DMX lighting from ESP32, Teensy, Raspberry Pi, ![Web UI](docs/assets/ui/ui_theme.gif) -๐Ÿ‘‰ **Try it now:** flash an ESP32 straight from your browser โ†’ โ€” step-by-step in the [Getting started guide](docs/gettingstarted.md). +๐Ÿ‘‰ **Try it now:** flash an ESP32 straight from your browser โ†’ . Step-by-step in the [Getting started guide](docs/gettingstarted.md). ๐Ÿ“ฆ **Release + downloads:** [latest release](https://github.com/MoonModules/projectMM/releases/latest) @@ -22,21 +22,21 @@ If you like projectMM, give it a โญ๏ธ, fork it, or open an issue or pull reque ๐ŸŽ›๏ธ **Pluggable pipeline**: Layouts โ†’ Layers (effects + modifiers) โ†’ Drivers. Build it visually in the browser, and every change applies live (settings also persist to flash across power cycles). -๐Ÿ”„ **No reboot to apply a configuration change**: edit a pin map, a strand length, an output protocol, or the mic on a running device and it takes effect on the very next frame โ€” no init-at-boot step, no restart. Where most LED-controller firmware needs a reboot for a pin or protocol change, projectMM applies it live. (Flashing new *firmware* over OTA still needs the usual power cycle โ€” that's a binary swap, not a config change.) +๐Ÿ”„ **No reboot to apply a configuration change**: edit a pin map, a strand length, an output protocol, or the mic on a running device and it takes effect on the very next frame, with no init-at-boot step, no restart. Where most LED-controller firmware needs a reboot for a pin or protocol change, projectMM applies it live. (Flashing new *firmware* over OTA still needs the usual power cycle, since that's a binary swap rather than a config change.) ๐Ÿ’ก **DMX *and* addressable LEDs in one setup**: RGB strips, RGBW pixels, par lights, moving heads, all through the same pipeline. -๐Ÿ”Œ **Parallel WS2812 output**: drive many strands at once over three ESP32 peripherals โ€” RMT (every chip), the S3's LCD_CAM i80 bus (8 lanes), and the P4's Parlio engine (up to 8 lanes) โ€” each with an on-device loopback self-test that bit-verifies the wire signal. +๐Ÿ”Œ **Parallel WS2812 output**: drive many strands at once over three ESP32 peripherals: RMT (every chip), the S3's LCD_CAM i80 bus (8 lanes), and the P4's Parlio engine (up to 8 lanes), each with an on-device loopback self-test that bit-verifies the wire signal. -๐ŸŒ **Industry protocols, both directions**: send *and* receive [Art-Net](https://art-net.org.uk/), [E1.31/sACN](https://tsp.esta.org/tsp/documents/docs/ANSI_E1-31-2018.pdf), and [DDP](http://www.3waylabs.com/ddp/) over the network โ€” interoperable with Falcon, Advatek, xLights, Resolume, LedFx and other industry gear. +๐ŸŒ **Industry protocols, both directions**: send *and* receive [Art-Net](https://art-net.org.uk/), [E1.31/sACN](https://tsp.esta.org/tsp/documents/docs/ANSI_E1-31-2018.pdf), and [DDP](http://www.3waylabs.com/ddp/) over the network, interoperable with Falcon, Advatek, xLights, Resolume, LedFx and other industry gear. -๐ŸŽต **Audio-reactive**: an IยฒS microphone drives a 16-band FFT spectrum + sound level, consumed by audio-reactive effects โ€” all built fresh from the mic datasheet and textbook DSP. +๐ŸŽต **Audio-reactive**: an IยฒS microphone drives a 16-band FFT spectrum + sound level, consumed by audio-reactive effects, all built fresh from the mic datasheet and textbook DSP. -๐Ÿ  **Home-automation control**: a device joins Homebridge (and any MQTT hub) over a dependency-free MQTT 3.1.1 client โ€” on/off, brightness, and a HomeKit color wheel that picks the nearest palette. See [the MQTT module docs](docs/moonmodules/core/services.md#mqtt). +๐Ÿ  **Home-automation control**: a device joins Homebridge (and any MQTT hub) over a dependency-free MQTT 3.1.1 client: on/off, brightness, and a HomeKit color wheel that picks the nearest palette. See [the MQTT module docs](docs/moonmodules/core/services.md#mqtt). -๐Ÿ“ **On-device File Manager**: browse and edit the device filesystem from the browser โ€” a lazy folder tree with an inline editor, drag-drop upload, and create/delete, plus [firmware upload OTA](docs/moonmodules/core/services.md#firmware-update) (flash a `.bin` over the LAN, no USB). See [the File Manager docs](docs/moonmodules/core/services.md#file-manager). +๐Ÿ“ **On-device File Manager**: browse and edit the device filesystem from the browser: a lazy folder tree with an inline editor, drag-drop upload, and create/delete, plus [firmware upload OTA](docs/moonmodules/core/services.md#firmware-update) (flash a `.bin` over the LAN, no USB). See [the File Manager docs](docs/moonmodules/core/services.md#file-manager). -๐Ÿ›ก๏ธ **Robust to any input**: add, delete, replace, or reconfigure any module in any order, at any grid size, and the device keeps running โ€” degraded or idle, never crashed. Every crash that's ever found becomes a regression test, so it stays fixed. +๐Ÿ›ก๏ธ **Robust to any input**: add, delete, replace, or reconfigure any module in any order, at any grid size, and the device keeps running, degraded or idle, but never crashed. Every crash that's ever found becomes a regression test, so it stays fixed. ๐Ÿ–ฅ๏ธ **One source tree, many targets**: the same code runs on ESP32, Teensy, Raspberry Pi, and macOS / Windows / Linux. @@ -50,7 +50,7 @@ If you like projectMM, give it a โญ๏ธ, fork it, or open an issue or pull reque ๐Ÿ“ฆ **No third-party libraries**: no FastLED, no ESPAsyncWebServer, no ArduinoJson. The color math, the HTTP/WebSocket server, and the control storage are all in-tree. A library, when genuinely needed, lives behind the platform boundary in `src/platform/`, never in core. The full rationale + replacements: [building.md ยง Third-party libraries](docs/building.md#third-party-libraries). -๐Ÿ”ฌ **Industry standards, our own code**: we study the prior art hard โ€” friend repos, peripheral datasheets, the Art-Net / E1.31 / WS2812 standards โ€” carry its *ideas* forward, and credit it by name; but we write our own code rather than copying theirs or tracing their structure. Each feature is spec'd from the primary source, its behaviour pinned with unit + scenario tests, then written fresh against our own architecture, so the result is independent by construction, not a renamed fork. Textbook algorithm, textbook name, our implementation. The method: [CLAUDE.md ยง Principles](CLAUDE.md#principles). +๐Ÿ”ฌ **Industry standards, our own code**: we study the prior art hard (friend repos, peripheral datasheets, the Art-Net / E1.31 / WS2812 standards), carry its *ideas* forward, and credit it by name; but we write our own code rather than copying theirs or tracing their structure. Each feature is spec'd from the primary source, its behaviour pinned with unit + scenario tests, then written fresh against our own architecture, so the result is independent by construction, not a renamed fork. Textbook algorithm, textbook name, our implementation. The method: [CLAUDE.md ยง Principles](CLAUDE.md#principles). ๐Ÿงฑ **One module model**: every effect, modifier, layout, and driver is a `MoonModule`: one base class, a uniform lifecycle, declared controls. That uniformity is why the UI renders any module with zero per-module code, and why a new capability is a new file, not a new framework. See [architecture.md ยง MoonModules](docs/architecture.md#moonmodules). @@ -162,15 +162,17 @@ We built, maintained, and contributed to these projects, so projectMM is grounde ## Credits -Specific people whose work directly shaped parts of projectMM. We study their thinking with respect and write our own code against our architecture rather than tracing theirs โ€” these credits name the prior art behind a feature: - -- **Frank ([softhack007](https://github.com/softhack007))** โ€” main author of the WLED-MM audio-reactive usermod, the most-used open-source audio-reactive LED implementation. The ideas behind [AudioService](docs/moonmodules/core/moxygen/AudioService.md) (including the adaptive noise-gate concept, analysed with his permission) descend from years of collaboration on WLED-SR / WLED-MM. -- **[troyhacks](https://github.com/troyhacks/WLED)** โ€” reworked the WLED-MM audio-reactive DSP to run on Espressif's [esp-dsp](https://github.com/espressif/esp-dsp) FFT (a low-latency, "stupid fast" alternative to ArduinoFFT); the same esp-dsp FFT choice [AudioService](docs/moonmodules/core/moxygen/AudioService.md) makes. See its Prior art notes. -- **[hpwit](https://github.com/hpwit) (Yves Bazin)** โ€” the clockless I2S / RMT / Parlio LED-driver techniques and the [ESPLiveScript](https://github.com/hpwit/ESPLiveScript) live-script engine behind the LED drivers and MoonLive. -- **Christophe Gagnier ([@Moustachauve](https://github.com/Moustachauve))** โ€” author of the native [WLED-Android](https://github.com/Moustachauve/WLED-Android) app. Its source let us reverse-engineer exactly what the WLED app reads, so projectMM devices appear in (and are controllable from) the native WLED apps. -- **The [Improv Wi-Fi](https://github.com/improv-wifi) project** โ€” the open Improv serial provisioning standard ([sdk-cpp](https://github.com/improv-wifi/sdk-cpp) / [sdk-js](https://github.com/improv-wifi/sdk-js)) that the projectMM web installer uses to provision a freshly-flashed device over USB. -- **[FastLED](https://github.com/FastLED/FastLED)** โ€” the canonical LED-effects library whose conventions the LED-effect world shares. projectMM links no part of FastLED, but it carries forward FastLED's recognisable *names and models* for the color/animation primitives โ€” `scale8`, `sin8`, the gradient-palette model (`CRGBPalette16` / `colorFromPalette`), the `beatsin8` / `inoise8` / `qadd8` family โ€” so a contributor recognises them on sight. The implementations are projectMM's own, integer-only and hot-path-tuned for our render loop; FastLED is the prior art behind the convention, credited here and in each primitive's notes. -- **wladi ([myhome-control](https://shop.myhome-control.de))** โ€” designer of the [MHC-WLED ESP32-P4 shield](https://shop.myhome-control.de/en/ABC-WLED-ESP32-P4-shield/HW10027), and the source of the hardware and the pinout details that got its **line-in audio** working in [AudioService](docs/moonmodules/core/moxygen/AudioService.md): the onboard PCM1808 I2S ADC (WS 26 / SD 33 / SCK 32 / MCLK 36), the PCM1808's stereo wiring, and its `FMT` format-select jumper (open = I2S/Philips, our default; tie to 3V3 for left-justified) โ€” which is what confirmed the standard-I2S path the ADC needs. +Specific people whose work directly shaped parts of projectMM. We study their thinking with respect and write our own code against our architecture rather than tracing theirs. These credits name the prior art behind a feature: + +- **[WLED](https://github.com/wled/WLED) and [WLED-MM](https://github.com/MoonModules/WLED)**: projectMM is born out of WLED, and takes the usermod idea to a new level. Here *everything* is a mod (a MoonModule): effects, drivers, networking, the file system, the system manager. It also integrates tightly with WLED: a projectMM device can act as a WLED device, and it talks to WLED devices (audio sync, discovery, and more). +- **Frank ([softhack007](https://github.com/softhack007))**: main author of the WLED-MM audio-reactive usermod, the most-used open-source audio-reactive LED implementation. The ideas behind [AudioService](docs/moonmodules/core/moxygen/AudioService.md) (including the adaptive noise-gate concept, analysed with his permission) descend from years of collaboration on WLED-SR / WLED-MM. +- **[troyhacks](https://github.com/troyhacks/WLED)**: reworked the WLED-MM audio-reactive DSP to run on Espressif's [esp-dsp](https://github.com/espressif/esp-dsp) FFT (a low-latency, "stupid fast" alternative to ArduinoFFT); the same esp-dsp FFT choice [AudioService](docs/moonmodules/core/moxygen/AudioService.md) makes. See its Prior art notes. +- **[hpwit](https://github.com/hpwit) (Yves Bazin)**: the clockless I2S / RMT / Parlio LED-driver techniques and the [ESPLiveScript](https://github.com/hpwit/ESPLiveScript) live-script engine behind the LED drivers and MoonLive. +- **Christophe Gagnier ([@Moustachauve](https://github.com/Moustachauve))**: author of the native [WLED-Android](https://github.com/Moustachauve/WLED-Android) and [WLED-iOS](https://github.com/Moustachauve/WLED-iOS) apps. Their source let us reverse-engineer exactly what those apps read, so projectMM devices appear in (and are controllable from) the native WLED apps. +- **The [Improv Wi-Fi](https://github.com/improv-wifi) project**: the open Improv serial provisioning standard ([sdk-cpp](https://github.com/improv-wifi/sdk-cpp) / [sdk-js](https://github.com/improv-wifi/sdk-js)) that the projectMM web installer uses to provision a freshly-flashed device over USB. +- **[FastLED](https://github.com/FastLED/FastLED)**: the canonical LED-effects library whose conventions the LED-effect world shares. projectMM links no part of FastLED, but it carries forward FastLED's recognisable *names and models* for the color/animation primitives (`scale8`, `sin8`, the gradient-palette model (`CRGBPalette16` / `colorFromPalette`), the `beatsin8` / `inoise8` / `qadd8` family), so a contributor recognises them on sight. The implementations are projectMM's own, integer-only and hot-path-tuned for our render loop; FastLED is the prior art behind the convention, credited here and in each primitive's notes. +- **[FPP](https://github.com/FalconChristmas/fpp) (Falcon Player)**: the show player that drives LED panel receiver cards from a Raspberry Pi. Seeing an FPP rig feed a wall of HUB75 panels is what prompted [PanelCardDriver](docs/moonmodules/light/drivers.md#panelcard): if a Linux host can send those frames, so can a board that is already rendering them, which removes the host from the installation entirely. FPP is the inspiration, and the reference point for what good looks like here: it sustains 50 fps. +- **wladi ([myhome-control](https://shop.myhome-control.de))**: designer of the [MHC-WLED ESP32-P4 shield](https://shop.myhome-control.de/en/ABC-WLED-ESP32-P4-shield/HW10027), and the source of the hardware and the pinout details that got its **line-in audio** working in [AudioService](docs/moonmodules/core/moxygen/AudioService.md): the onboard PCM1808 I2S ADC (WS 26 / SD 33 / SCK 32 / MCLK 36), the PCM1808's stereo wiring, and its `FMT` format-select jumper (open = I2S/Philips, our default; tie to 3V3 for left-justified), which is what confirmed the standard-I2S path the ADC needs. ## Contributing diff --git a/docs/assets/light/drivers/PanelCardDriver.png b/docs/assets/light/drivers/PanelCardDriver.png index 2b7dec4b..4ae69e68 100644 Binary files a/docs/assets/light/drivers/PanelCardDriver.png and b/docs/assets/light/drivers/PanelCardDriver.png differ diff --git a/docs/history/lessons.md b/docs/history/lessons.md index 870c5fcc..4326c870 100644 --- a/docs/history/lessons.md +++ b/docs/history/lessons.md @@ -399,3 +399,20 @@ The leading untested theory is that a short frame gives the DMA no runway to pre - **Read a control's applicability before drawing conclusions from its value.** `doubleBuffer`, `useRing` and `ringSnapshot` all appear in `/api/state` while being inapplicable to this path (`supportsDoubleBuffer()` is hard `false` here; the ring controls are hidden unless `pinExpanderMode()`, and `wantsRing()` returns false in direct mode outright). Two hypotheses were built on those values and both were dead ends โ€” the UI hides them for a reason, the API does not. - **A frame-time KPI that does not scale with the frame is measuring a timeout, not a wire.** `frameTime` read ~741 ยตs flat from 10 to 600 lights. Flat where it should scale is a signal in itself. - **A fix that works is not a cause that is understood.** The pool swap reliably removes the symptom, which is tempting to write up as a root cause; the arithmetic says the mechanism is still unknown. + +## A DMA buffer smaller than the frame multiplies descriptor use, and the symptom looked exactly like a failing cable + +Streaming panel-card frames from an S31 at ~5 300 packets/s degraded with uptime: clean for minutes, then refused frames, then a total transmit wedge every ~11 minutes that only a reboot cleared. It read as a hardware fault the whole way, and it was two config lines. + +**The arithmetic that caused it.** `CONFIG_ETH_DMA_BUFFER_SIZE` defaults to 512 B. A panel-card frame is 1512 B, so every frame consumed **three** descriptors. A 10-descriptor TX ring therefore held ~3.3 frames while the driver fires 132 back-to-back. Setting the buffer to 1536 B (64-byte aligned) gives one descriptor per frame and the failures stop. **Check DMA buffer size against your actual frame size; the default is sized for IP MTU traffic, not for a burst sender.** + +**The second half is a race, not a size.** `CONFIG_ETH_TRANSMIT_MUTEX` defaults to off, and `mac->transmit` advances a shared descriptor pointer with no locking. With the eth netif up, lwIP and the render task both reach it. Bench-isolated: buffer fixed but mutex off โ†’ 3 000 refused frames in 4 minutes; mutex on with only 10 descriptors โ†’ none in 11. **Ring depth was never the fix**: 30 descriptors ran no cleaner than 10, and a 30-deep TX ring totals ~46 KB of internal RAM against ~15 KB at 10. + +**Why it cost hours: two wrong turns worth naming.** +- *"It is not back-pressure."* Halving the send rate left the failure rate at ~13-15%, which reads as ruling out a full ring. It does not: at 3.3 frames of depth each 132-packet burst overruns the ring at any rate. The burst was the problem, never the rate. That misreading sent the investigation toward the cable and the PHY. +- *One counter for two faults.* `esp_eth_transmit` returns `ESP_ERR_INVALID_STATE` for a down link **before** touching the MAC, and `ESP_ERR_NO_MEM` for a full ring. Collapsing both into one bool made "5 million drops" unreadable. Splitting them settled the cause in one build: every failure was `ring`, none was `link`. **When a counter can be incremented by two different faults, split it before theorising.** + +**What the physical evidence did and did not prove.** Both link LEDs going dark said the wire genuinely dropped, which is true and correctly killed the "stale software flag" theory. But it does not identify a cause: a wedged MAC stops driving the wire, so the dark LED was a *consequence*. A longer cable was the visible difference from the working setup and looked compelling; it was untouched throughout and is exonerated. + +**Recovery still earns its place.** `esp_eth_stop()` + `esp_eth_start()` re-runs negotiation and resets the descriptor rings: the only way back from a wedge short of a reboot, since no ioctl writes the driver's link flag. Bench-verified twice, recovering in ~17 s. Attempted once per wedge, never repeatedly: a restart cannot fix an unplugged cable, and retrying would bounce the interface under the user. + diff --git a/docs/metrics/repo-health.json b/docs/metrics/repo-health.json index 5f76ce9e..2dd812ce 100644 --- a/docs/metrics/repo-health.json +++ b/docs/metrics/repo-health.json @@ -1,18 +1,18 @@ { - "commit": "1aff3df2", + "commit": "cca3fe8e", "flash": { "esp32": 1678960, - "esp32p4-eth": 1502192, + "esp32p4-eth": 1503232, "esp32p4-eth-wifi": 1793760, "esp32s3-n16r8": 1667216, "esp32s3-n8r8": 1666992, - "esp32s31": 1923856, - "desktop": 945336 + "esp32s31": 1924992, + "desktop": 945752 }, "perf": { "desktop": { - "tick_us": 125, - "fps": 8000 + "tick_us": 127, + "fps": 7874 }, "esp32": { "tick_us": 4164, @@ -21,11 +21,11 @@ }, "loc": { "core": 14889, - "light": 20150, - "platform": 12428, + "light": 20214, + "platform": 12526, "ui": 5811, - "test": 35207, - "moondeck": 19914 + "test": 35504, + "moondeck": 19968 }, "comments": { "core": { @@ -33,41 +33,41 @@ "ratio": 0.409 }, "light": { - "lines": 7783, - "ratio": 0.426 + "lines": 7815, + "ratio": 0.427 }, "platform": { - "lines": 4149, - "ratio": 0.369 + "lines": 4198, + "ratio": 0.371 }, "ui": { "lines": 1518, "ratio": 0.278 }, "test": { - "lines": 6035, + "lines": 6073, "ratio": 0.198 }, "moondeck": { - "lines": 3182, + "lines": 3187, "ratio": 0.183 } }, "tests": { - "cases": 1002, + "cases": 1008, "scenarios": 22 }, "docs": { "md_files": 170, - "md_lines": 22762, + "md_lines": 22818, "plans_files": 90, "backlog_lines": 3114, - "lessons_lines": 401, + "lessons_lines": 418, "claude_md_lines": 135 }, "complexity": { - "functions": 2179, - "over_threshold": 138, + "functions": 2188, + "over_threshold": 140, "worst_ccn": 93 } } diff --git a/docs/metrics/repo-health.md b/docs/metrics/repo-health.md index 8a83c97b..396c38d0 100644 --- a/docs/metrics/repo-health.md +++ b/docs/metrics/repo-health.md @@ -1,6 +1,6 @@ # Repo health -Measured at `1aff3df2`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run โ€” **do not edit by hand**. +Measured at `cca3fe8e`. Generated by [`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every KPI-gate run. **Do not edit by hand.** Current state only; the trend is this file's git history (`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make growth visible, the judgment stays human. @@ -8,45 +8,45 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Target | Flash | |---|---:| -| desktop | 923 KB (โˆ’0 KB) โœ“ | +| desktop | 924 KB (+0 KB) โš  | | esp32 | 1,640 KB | -| esp32p4-eth | 1,467 KB (+1 KB) โš  | +| esp32p4-eth | 1,468 KB | | esp32p4-eth-wifi | 1,752 KB | -| esp32s3-n16r8 | 1,628 KB (+0 KB) โš  | +| esp32s3-n16r8 | 1,628 KB | | esp32s3-n8r8 | 1,628 KB | -| esp32s31 | 1,879 KB (โˆ’0 KB) โœ“ | +| esp32s31 | 1,880 KB | ## Render performance | Target | Tick | FPS | |---|---:|---:| -| desktop | 125 ยตs (โˆ’3 ยตs) โœ“ | 8,000 (+188) โœ“ | +| desktop | 127 ยตs | 7,874 | | esp32 | 4,164 ยตs | 240 | ## Code | Area | Lines | Comments | Comment share | |---|---:|---:|---:| -| core | 14,889 (โˆ’1) โœ“ | 5,583 | 40.9 % | -| light | 20,150 (+3) โš  | 7,783 | 42.6 % | -| platform | 12,428 (โˆ’8) โœ“ | 4,149 | 36.9 % | +| core | 14,889 | 5,583 | 40.9 % | +| light | 20,214 (+16) โš  | 7,815 | 42.7 % | +| platform | 12,526 (+25) โš  | 4,198 | 37.1 % (+0.1 %) โš  | | ui | 5,811 | 1,518 | 27.8 % | -| test | 35,207 | 6,035 | 19.8 % | -| moondeck | 19,914 | 3,182 | 18.3 % | +| test | 35,504 (+75) โš  | 6,073 | 19.8 % | +| moondeck | 19,968 | 3,187 | 18.3 % | ## Tests | Kind | Count | |---|---:| -| unit cases | 1,002 | +| unit cases | 1,008 (+1) โœ“ | | scenarios | 22 | ## Complexity | Metric | Value | |---|---:| -| functions | 2,179 (โˆ’3) โš  | -| over threshold | 138 | +| functions | 2,188 (+1) โœ“ | +| over threshold | 140 (+1) โš  | | worst CCN | 93 | ## Documentation @@ -54,9 +54,9 @@ Current state only; the trend is this file's git history (`git log -p docs/metri | Metric | Value | |---|---:| | markdown files | 170 | -| markdown lines | 22,762 | +| markdown lines | 22,818 (+9) โš  | | plan files | 90 | | backlog lines | 3,114 | -| lessons lines | 401 | +| lessons lines | 418 | | CLAUDE.md lines | 135 | diff --git a/docs/performance.md b/docs/performance.md index 5efd30e2..42dd5ec1 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -259,28 +259,37 @@ The **acceptance floors** these establish for the parallel backends: RMT **8ร—25 ## Panel cards over raw Ethernet (`PanelCardDriver`, ESP32-S31) -Measured on an S31 driving two 128ร—64 HUB75 panels through a ColorLight 5A-75 receiver card, gigabit -RGMII link, 2026-07-30. +Measured on an S31 driving two 128x64 HUB75 panels through a ColorLight 5A-75 receiver card over a +gigabit RGMII link, 2026-07-31. -| | ยตs/tick | note | +| | us/tick | note | |---|---:|---| -| **PanelCardDriver** | **2 636** | 16 384 lights: correction + 130 frames handed to the MAC | +| **PanelCardDriver** | **~2 500** | 16 384 lights: correction + 132 frames handed to the MAC | | PreviewDriver | 5 687 | the browser preview, same buffer | -| GameOfLifeEffect | 17 592 | the render, and the largest single cost | +| a heavy effect (GameOfLife) | 17 592 | the render, and the largest single cost | -The wall runs at **~32 FPS**, which is `1 000 000 / (2 636 + 5 687 + 17 592 + overhead)` โ€” the render -dominates, and the panel driver is the cheapest of the three active modules despite pushing 130 -packets per frame (2 brightness + 128 rows + 2 sync, at 497 pixels per row packet). +The panel driver is the cheapest active module despite pushing 132 packets per frame (2 brightness + +128 rows + 2 sync, at 497 pixels per row packet). The wall's frame rate is set by the render, not by +the output: a heavy effect at 17.6 ms dominates a 26 ms tick, giving ~32 FPS, while a lighter effect +mix measures ~56 FPS on the same wall. **The ceiling is packets, not pixels.** Each frame is sent synchronously from `tick()`, so a taller wall costs proportionally more rows; a 256-row wall would double the packet count. The card format's -1 Gbit requirement is a wire-time constraint rather than a bandwidth one โ€” at 100 Mbit the same bytes +1 Gbit requirement is a wire-time constraint rather than a bandwidth one: at 100 Mbit the same bytes take ten times as long and overrun the inter-frame window the sync depends on ([drivers.md](moonmodules/light/drivers.md#panelcard)). +**The DMA ring is what makes it stable.** `CONFIG_ETH_DMA_BUFFER_SIZE` defaults to 512 B, so a +1512 B frame spanned three descriptors and a 10-descriptor ring held ~3.3 frames while the driver +fires 132 back-to-back. At that depth the S31 refused ~19 000 frames and wedged twice inside 20 +minutes; at 1536 B per buffer (one descriptor per frame) plus `CONFIG_ETH_TRANSMIT_MUTEX`, it runs +clean. Both are bench-isolated, and ring COUNT is not the lever: 30 descriptors ran no cleaner than +10. Cost: ~20 KB of internal DMA RAM, since the size applies to both rings +([lessons.md](history/lessons.md)). + **Static RAM: 0 B.** The driver's 1 512 B packet buffer is a class member, so it costs nothing on a board that never adds the driver; `check_footprint --module PanelCardDriver --firmware esp32s31` -reports 3 270 B of flash and no static RAM. +reports ~3 500 B of flash and no static RAM. No scenario contract yet: the driver needs a receiver card on the wire, so the numbers above are a bench record rather than an asserted ceiling. diff --git a/esp32/sdkconfig.defaults.esp32s31 b/esp32/sdkconfig.defaults.esp32s31 index 9f253300..76770ee2 100644 --- a/esp32/sdkconfig.defaults.esp32s31 +++ b/esp32/sdkconfig.defaults.esp32s31 @@ -45,9 +45,26 @@ CONFIG_ETH_USE_ESP32_EMAC=y # XTAL. The APLL is a *fractional* PLL built for exact frequencies (its reason to exist), so # it synthesises 125 MHz precisely. Bench-proven on the S31 CoreBoard. CONFIG_ETH_EMAC_RGMII_TX_CLK_SRC_APLL=y -CONFIG_ETH_DMA_BUFFER_SIZE=512 +# A panel-card frame is 1512 B. At the 512 B default every frame spanned THREE descriptors, so a +# 10-descriptor ring held only ~3 frames while PanelCardDriver fires 132 back-to-back, and the ring +# ran dry within minutes (bench: ~19 000 refused frames and two full transmit wedges in 20 minutes). +# 1536 is the 64-byte-aligned size that fits one frame per descriptor. +# +# The COUNT stays at 10: bench-isolated on 2026-07-31. Buffer size and the transmit mutex below are +# what fix it, and 30 descriptors ran no cleaner than 10 (a 30-deep TX ring totals ~46 KB against +# ~15 KB at 10). The size bump itself is not free: 512โ†’1536 applies to BOTH rings, so 20 buffers +# cost ~20 KB more internal RAM than the default. Cheaper than the alternative, and measured. +CONFIG_ETH_DMA_BUFFER_SIZE=1536 CONFIG_ETH_DMA_RX_BUFFER_NUM=10 CONFIG_ETH_DMA_TX_BUFFER_NUM=10 +# The eth netif is up alongside the raw-L2 sender, so lwIP and the render task both reach +# mac->transmit, which advances a shared descriptor pointer with no locking of its own. IDF's help +# text names exactly this case: "Ethernet interface is used as shared resource and multiple +# functionalities might try to access it at a time." Default is off. +# +# Load-bearing, not belt-and-braces: with it off (and 30 descriptors) the bench saw 3 000 refused +# frames in 4 minutes; with it on and only 10 descriptors, none in 11. +CONFIG_ETH_TRANSMIT_MUTEX=y # MoonLive native codegen needs an executable heap (allocExec โ†’ MALLOC_CAP_EXEC). Like the # S3 and P4, the S31 has hardware memory protection that IDF couples to disabling the exec diff --git a/moondeck/check/check_esp32_built.py b/moondeck/check/check_esp32_built.py index 7d287911..a6afe87a 100644 --- a/moondeck/check/check_esp32_built.py +++ b/moondeck/check/check_esp32_built.py @@ -20,6 +20,7 @@ """ import argparse +import subprocess import sys import time from pathlib import Path @@ -49,7 +50,7 @@ SKIP_PREFIXES = ("src/platform/desktop/",) -def newest_source(): +def newest_source(only=None): """The most recently modified source file that feeds a firmware image.""" newest_path, newest_mtime = None, 0.0 @@ -67,11 +68,15 @@ def newest_source(): continue if path.suffix not in SOURCE_SUFFIXES: continue + if only is not None and rel_posix not in only: + continue # this firmware does not compile it, so it cannot make it stale mtime = path.stat().st_mtime if mtime > newest_mtime: newest_path, newest_mtime = path, mtime for rel in SOURCE_FILES: + if only is not None and rel not in only: + continue path = ROOT / rel if path.exists() and path.stat().st_mtime > newest_mtime: newest_path, newest_mtime = path, path.stat().st_mtime @@ -79,6 +84,39 @@ def newest_source(): return newest_path, newest_mtime +def compiled_sources(firmware): + """The files this firmware's build actually reads, from ninja's own dependency database. + + A firmware compiles a SUBSET of src/: a driver gated behind a per-firmware flag (MM_PANEL_CARDS) + is not in the S3 build at all. Comparing every source against that binary flags it stale forever + over a file it cannot compile, and no rebuild can clear it โ€” the build correctly does nothing, so + the binary's timestamp never moves. Asking ninja what it depends on is the authoritative answer. + + Returns None when the database cannot be read, and the caller then falls back to scanning + everything: over-reporting staleness is the safe direction for a freshness gate. + """ + build_dir = ROOT / "build" / f"esp32-{firmware}" + if not (build_dir / ".ninja_deps").exists(): + return None + try: + out = subprocess.run(["ninja", "-t", "deps"], cwd=build_dir, + capture_output=True, text=True, timeout=60) + except (subprocess.SubprocessError, OSError): + return None + if out.returncode != 0: + return None + deps = set() + for line in out.stdout.splitlines(): + line = line.strip() + if not line.startswith(("output", "deps")) and line: + path = (build_dir / line).resolve() + try: + deps.add(path.relative_to(ROOT).as_posix()) + except ValueError: + pass # outside the repo (IDF, toolchain) โ€” not ours to watch + return deps or None + + def main(): parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--firmware", required=True, @@ -100,7 +138,7 @@ def main(): return 1 built = binary.stat().st_mtime - newest_path, newest_mtime = newest_source() + newest_path, newest_mtime = newest_source(compiled_sources(args.firmware)) age_h = (time.time() - built) / 3600 if newest_path is not None and newest_mtime > built: diff --git a/moondeck/check/repo_health.py b/moondeck/check/repo_health.py index ae1db237..eb8d5566 100644 --- a/moondeck/check/repo_health.py +++ b/moondeck/check/repo_health.py @@ -202,11 +202,69 @@ def snapshot(perf=None): } +def _valid_snapshot(data, source): + """A snapshot mapping, or {}, saying so out loud when it is neither. + + merge_carry_forward does `old.get(section, {})`, so a JSON list or scalar would raise, and a + section holding a list would corrupt the merge. A malformed baseline must not read as "no + previous numbers" either: that silently reports every metric as new, which looks like a clean + slate rather than a broken file. + """ + if not isinstance(data, dict): + print(f"repo-health: ignoring {source}: expected an object, got {type(data).__name__}", + file=sys.stderr) + return {} + for key in ("flash", "perf", "complexity"): + if key in data and not isinstance(data[key], dict): + print(f"repo-health: ignoring {source}: section '{key}' is " + f"{type(data[key]).__name__}, expected an object", file=sys.stderr) + return {} + return data + + def load_previous(): + """The COMMITTED snapshot, read from git rather than from the working tree. + + The delta is meant to answer "what did this commit change", so the baseline has to be the last + commit, not whatever a previous run left on disk. Reading the working-tree file made every run + after the first compare against the run before it, so a second run inside one commit showed a + delta of ~0 and the real change vanished. Running the check twice must give the same answer. + + Falls back to the working-tree file when git cannot answer (a fresh clone with no commit yet, or + the file untracked), which keeps a first-ever run working. + """ + try: + out = subprocess.run(["git", "show", f"HEAD:{HEALTH_FILE.relative_to(ROOT).as_posix()}"], + cwd=ROOT, capture_output=True, text=True, timeout=10) + if out.returncode == 0: + return _valid_snapshot(json.loads(out.stdout), "the committed snapshot") + except json.JSONDecodeError: + print("repo-health: the committed snapshot is not valid JSON, " + "falling back to the working tree", file=sys.stderr) + except (subprocess.SubprocessError, OSError): + pass if not HEALTH_FILE.exists(): return {} try: - return json.loads(_read(HEALTH_FILE)) + return _valid_snapshot(json.loads(_read(HEALTH_FILE)), str(HEALTH_FILE.name)) + except json.JSONDecodeError: + print(f"repo-health: {HEALTH_FILE.name} is not valid JSON, treating as empty", + file=sys.stderr) + return {} + + +def load_working_tree(): + """The snapshot on disk, whatever produced it. + + Carry-forward needs the NEWEST numbers, not the committed ones: a board-attached run writes fresh + perf figures here, and a later boardless run must preserve them rather than reverting to what the + last commit happened to hold. That is the opposite of what the delta baseline wants, which is why + the two read from different places. + """ + if not HEALTH_FILE.exists(): + return {} + try: + return _valid_snapshot(json.loads(_read(HEALTH_FILE)), str(HEALTH_FILE.name)) except json.JSONDecodeError: return {} @@ -288,7 +346,7 @@ def render_markdown(new, old): L = [f"# Repo health", "", f"Measured at `{new.get('commit', '?')}`. Generated by " f"[`moondeck/check/repo_health.py`](../../moondeck/check/repo_health.py) on every " - f"KPI-gate run โ€” **do not edit by hand**.", "", + f"KPI-gate run. **Do not edit by hand.**", "", "Current state only; the trend is this file's git history " "(`git log -p docs/metrics/repo-health.md`). Nothing here fails a build: the numbers make " "growth visible, the judgment stays human.", ""] @@ -349,8 +407,8 @@ def render_markdown(new, old): def write(perf=None, quiet=False): """Measure, print the delta, and rewrite both views. Called by the KPI gate.""" - old = load_previous() - new = merge_carry_forward(snapshot(perf), old) + old = load_previous() # committed โ†’ what the delta compares against + new = merge_carry_forward(snapshot(perf), load_working_tree()) # newest โ†’ what unmeasured metrics keep if not quiet: for line in format_delta(new, old): print(line) @@ -366,7 +424,7 @@ def main(): args = parser.parse_args() old = load_previous() - new = merge_carry_forward(snapshot(), old) + new = merge_carry_forward(snapshot(), load_working_tree()) for line in format_delta(new, old): print(line) if args.write: diff --git a/src/light/drivers/PanelCardDriver.h b/src/light/drivers/PanelCardDriver.h index ffbfd286..bba1dc5a 100644 --- a/src/light/drivers/PanelCardDriver.h +++ b/src/light/drivers/PanelCardDriver.h @@ -34,7 +34,8 @@ namespace mm { /// /// The deep dives are under *More info*, below the attribute/method lists: /// @xref{why-a-gigabit-link|why these cards need a gigabit link}, -/// @xref{running-this-on-a-host|running this on a desktop or a Pi}. +/// @xref{running-this-on-a-host|running this on a desktop or a Pi}, +/// @xref{other-card-vendors|other card vendors, and how to add one}. /// @card PanelCardDriver.png /// /// @moreinfo @@ -63,6 +64,35 @@ namespace mm { /// pin the wire format with no hardware and no privileges. /// /// On ESP32 `interface` is ignored: the chip has one MAC. +/// +/// ## Other card vendors +/// +/// ColorLight is one of several receiver-card makers, which is why this driver is named for the +/// category and carries a `format` selector rather than being a ColorLight driver: +/// +/// | vendor | position in the market | +/// |---|---| +/// | **[NovaStar](https://www.novastar.tech)** | the global leader; large-scale displays and stage events | +/// | **[ColorLight](https://www.colorlightinside.com)** | cost-effective, strong outdoor and fine-pitch support (the format implemented here) | +/// | **[Linsn](https://www.linsnled.com)** | affordable and stable, common on budget installations | +/// | **[Mooncell](https://www.mooncell.com.cn)** | full-color high-refresh niche | +/// | **[Huidu](https://www.huidu.cn)** | small and mid projects, storefront signage, mostly ASYNCHRONOUS | +/// | **DBstar**, **Xixun** | also in the field | +/// +/// **Contributions welcome.** Adding one is a `*Packet.h` beside ColorLight5A75Packet.h plus an +/// entry in `kFormatOptions`: the window, the correction, the chunking and the platform seam are +/// already shared, and the desktop capture path lets the byte layout be pinned by unit tests with no +/// hardware. What it actually costs is the research, not the code. +/// +/// Two things to know before starting. Each vendor speaks its **own proprietary L2 protocol**, so a +/// ColorLight frame will not drive a NovaStar card and the byte layout has to be obtained per +/// vendor. And whether another format fits this driver's row-plus-sync model is **unverified**: the +/// shape is an invitation, not a promise, and a format that addresses panels differently may need +/// the driver to grow rather than just gain a packet file. +/// +/// Huidu is the one to approach with care: its controllers are largely asynchronous, playing from +/// onboard storage rather than being fed live, which is a different product category from a +/// real-time sender. class PanelCardDriver : public DriverBase { public: /// Panel cards are RGB, so this references the "RGB" preset rather than the strips' "GRB" โ€” @@ -214,8 +244,13 @@ class PanelCardDriver : public DriverBase { // Sent TWICE, like the sync below. Card firmware v13+ acts on the second copy only; // older firmware ignores the duplicate, so sending both costs one frame and works on // every version rather than making the behaviour depend on a firmware probe we do not do. - if (platform::ethSendRaw(packet_, len)) framesSent_++; - if (platform::ethSendRaw(packet_, len)) framesSent_++; + // + // Counted like the rows: `dropped` and the platform's per-cause totals must describe the + // SAME set of frames, or comparing them tells you nothing. + for (int i = 0; i < 2; i++) { + if (platform::ethSendRaw(packet_, len)) framesSent_++; + else framesDroppedTotal_++; + } } // One card row per wall row, in order. The card's own row numbering runs across its outputs @@ -250,8 +285,10 @@ class PanelCardDriver : public DriverBase { if (anyRowSent) { const size_t len = buildColorLightSyncPacket(packet_, kCardGain); // Twice, for the same firmware reason as the brightness frame above. - if (platform::ethSendRaw(packet_, len)) framesSent_++; - if (platform::ethSendRaw(packet_, len)) framesSent_++; + for (int i = 0; i < 2; i++) { + if (platform::ethSendRaw(packet_, len)) framesSent_++; + else framesDroppedTotal_++; + } } } @@ -260,10 +297,19 @@ class PanelCardDriver : public DriverBase { /// class note says why a slow link is reported rather than refused. Runs on the 1 Hz path, so /// the snprintf here is the same accepted trade SystemModule makes. void writeLinkStatus() MM_NONBLOCKING { - if (!platform::ethLinkUp()) { - setStatus("no ethernet link", Severity::Warning); + // A failed restart outranks everything below: it is the one state a user cannot resolve by + // plugging a cable back in. Cleared only when frames flow again (see the re-arm below). + if (restartFailed_) { + setStatus("ethernet restart failed - restart the device", Severity::Error); + framesReported_ = framesSent_; return; } + + // The WEDGE is checked before the link, because a wedged transmit path is defined by sends + // failing, which is knowable whatever the link claims. Checking the link first hid this + // branch entirely on any platform reporting no link (the desktop stub among them), which is + // how the recovery path shipped unreachable by its own tests. + // // A short streak is ordinary back-pressure: the DMA ring fills while we push a whole frame // in one tick, the frame is dropped, the next one goes. Measured on a 128x128 wall at ~1900 // packets/s, streaks of 1-4 come and go and always clear themselves โ€” reporting those as an @@ -276,13 +322,37 @@ class PanelCardDriver : public DriverBase { static constexpr uint32_t kWedgedStreak = 500; const uint32_t failStreak = platform::ethSendFailStreak(); if (failStreak >= kWedgedStreak) { - std::snprintf(statusBuf_, sizeof(statusBuf_), "transmit wedged (%u refused)", - static_cast(failStreak)); + if (!restartTried_) { + restartTried_ = true; + // A failed restart leaves the driver STOPPED: no CONNECTED event can follow, so the + // link reads down forever and nothing else here can tell that apart from an unplugged + // cable. Report it as its own error rather than letting it masquerade as one. + if (platform::ethRestartTx()) { + setStatus("transmit wedged - restarting ethernet", Severity::Warning); + } else { + // LATCHED: after a failed restart the driver is stopped, so no frame is sent, the + // streak stops growing, and the next tick would otherwise fall through to the + // link branch and report a plain "no ethernet link", indistinguishable from an + // unplugged cable, for a board that needs a power cycle. + restartFailed_ = true; + setStatus("ethernet restart failed - restart the device", Severity::Error); + } + framesReported_ = framesSent_; + return; + } + std::snprintf(statusBuf_, sizeof(statusBuf_), "transmit wedged (%u refused, link %u Mbit)", + static_cast(failStreak), + static_cast(platform::ethLinkSpeedMbps())); setStatus(statusBuf_, Severity::Error); framesReported_ = framesSent_; return; } + if (!platform::ethLinkUp()) { + setStatus("no ethernet link", Severity::Warning); + framesReported_ = framesSent_; + return; + } const uint16_t mbps = platform::ethLinkSpeedMbps(); // Total frames the MAC refused since boot. A dropped frame is tolerated (the cards have no // acknowledgement, so a retry would cost the next frame instead), but it is NOT invisible: @@ -300,10 +370,22 @@ class PanelCardDriver : public DriverBase { setStatus(statusBuf_, Severity::Warning); return; } + // Re-arm the one-shot only on evidence of FLOW: frames sent this second and no failure + // streak at all. Re-arming merely because the link reads healthy would fire again while a + // wedge is still rebuilding its streak, bouncing the interface every ~20 s: the loop the + // one-shot exists to prevent. + if (sent > 0 && failStreak == 0) { restartTried_ = false; restartFailed_ = false; } + if (dropped) { - std::snprintf(statusBuf_, sizeof(statusBuf_), "%u Mbit - %u packets/s, %u dropped", + // Split by cause: a flapping link and a full TX ring are different faults with + // different fixes, and one total cannot tell them apart. + uint32_t linkDown = 0, ringFull = 0; + platform::ethSendFailCounts(linkDown, ringFull); + std::snprintf(statusBuf_, sizeof(statusBuf_), + "%u Mbit - %u pkt/s, %u lost (%u link, %u ring)", static_cast(mbps), static_cast(sent), - static_cast(dropped)); + static_cast(dropped), + static_cast(linkDown), static_cast(ringFull)); } else { std::snprintf(statusBuf_, sizeof(statusBuf_), "%u Mbit - %u packets/s", static_cast(mbps), static_cast(sent)); @@ -365,8 +447,17 @@ class PanelCardDriver : public DriverBase { /// Frames the MAC refused since boot. Cumulative on purpose: the per-second rate hides a slow /// trickle of drops, and a rising total is the signal that the sender is outrunning the wire. uint32_t framesDroppedTotal_ = 0; - /// Backing store for the status line (setStatus does not copy). - char statusBuf_[48] = {}; + /// Backing store for the status line (setStatus does not copy). Sized for the longest one: the + /// split-drop report, which carries three cumulative counters and reaches ~63 chars at millions + /// of drops. Truncation would cut the RING count, the number the line exists to show. + char statusBuf_[96] = {}; + /// Whether a restart has already been attempted for the CURRENT wedge. One attempt per wedge: + /// a restart cannot fix an unplugged cable, and retrying every second would bounce the interface + /// under the user. Cleared as soon as frames flow again. + bool restartTried_ = false; + /// Set when a recovery attempt itself failed, which leaves the interface stopped. Latched so the + /// error cannot be overwritten by the softer link warning on the next tick. + bool restartFailed_ = false; /// Whether this driver currently holds the raw-L2 claim, so prepare/release stay balanced /// however often the framework calls them. bool claimed_ = false; diff --git a/src/platform/desktop/platform_desktop.cpp b/src/platform/desktop/platform_desktop.cpp index 080869c9..5d3fa4ed 100644 --- a/src/platform/desktop/platform_desktop.cpp +++ b/src/platform/desktop/platform_desktop.cpp @@ -719,7 +719,10 @@ size_t ethTestCount_ = 0; bool ethTestSendFails_ = false; uint16_t ethTestLinkSpeed_ = 1000; // desktop reports gigabit unless a test says otherwise int ethRawClaims_ = 0; // drivers holding the link for direct L2 use -uint32_t ethSendFails_ = 0; // consecutive ethSendRaw failures +uint32_t ethSendFails_ = 0; // consecutive ethSendRaw failures (the streak) +uint32_t ethFailTotal_ = 0; // cumulative since boot; what ethSendFailCounts reports +uint32_t ethRestarts_ = 0; // ethRestartTx() calls, for the once-per-wedge test +bool ethRestartFails_ = false; // simulated recovery failure // The bound raw socket, or -1 for capture mode (the default, and all any test sees). int ethRawFd_ = -1; unsigned ethRawIfIndex_ = 0; // Linux AF_PACKET needs the index; BPF binds by name @@ -776,7 +779,7 @@ bool ethBindRawInterface(const char* ifName) { // what every unit test exercises; the raw branch is what makes a host a panel controller. bool ethSendRaw(const uint8_t* frame, size_t len) MM_NONBLOCKING { if (!frame || len == 0) return false; - if (ethTestSendFails_) { ethSendFails_++; return false; } // simulated link-down / full TX ring + if (ethTestSendFails_) { ethSendFails_++; ethFailTotal_++; return false; } // simulated link-down / full ring #ifndef _WIN32 if (ethRawFd_ >= 0) { @@ -793,7 +796,7 @@ bool ethSendRaw(const uint8_t* frame, size_t len) MM_NONBLOCKING { #endif // Track failures on the REAL send path too, not just the capture path: a bound host is // where frames actually reach a wire, so a streak here is the one that matters. - if (n != static_cast(len)) { ethSendFails_++; return false; } + if (n != static_cast(len)) { ethSendFails_++; ethFailTotal_++; return false; } ethSendFails_ = 0; return true; } @@ -819,6 +822,25 @@ bool ethSendRaw(const uint8_t* frame, size_t len) MM_NONBLOCKING { uint32_t ethSendFailStreak() MM_NONBLOCKING { return ethSendFails_; } +// A host socket has no driver link state to refuse against, so every failure is the +// ring-full analogue (a full socket buffer). +void ethSendFailCounts(uint32_t& linkDown, uint32_t& ringFull) MM_NONBLOCKING { + linkDown = 0; ringFull = ethFailTotal_; +} + +// A host raw socket has no driver-internal link state to desync, so there is nothing to +// restart, so clear the streak and let a test exercise the driver's recovery path. +bool ethRestartTx() { + ethRestarts_++; + if (ethRestartFails_) return false; + ethSendFails_ = 0; + return true; +} + +void setTestEthRestartFails(bool fail) { ethRestartFails_ = fail; } + +uint32_t ethRestartCountForTest() { return ethRestarts_; } + // See platform.h: a claim stated by the driver, reference-counted. void ethClaimRawL2(bool claim) { if (claim) ethRawClaims_++; @@ -837,7 +859,7 @@ size_t ethTestFrameLength(size_t i) { return i < kEthTestMaxFrames ? ethTestLens const uint8_t* ethTestFrameData(size_t i) { return (ethTestFrames_ && i < kEthTestMaxFrames) ? ethTestFrames_[i] : nullptr; } -void ethTestClearFrames() { ethTestCount_ = 0; ethSendFails_ = 0; } +void ethTestClearFrames() { ethTestCount_ = 0; ethSendFails_ = 0; ethFailTotal_ = 0; } void setTestEthSendFails(bool fail) { ethTestSendFails_ = fail; } void setTestEthLinkSpeed(uint16_t mbps) { ethTestLinkSpeed_ = mbps; } void ethGetIPv4(uint8_t out[4]) MM_NONBLOCKING { diff --git a/src/platform/esp32/platform_esp32.cpp b/src/platform/esp32/platform_esp32.cpp index dcfc6f7c..3572603f 100644 --- a/src/platform/esp32/platform_esp32.cpp +++ b/src/platform/esp32/platform_esp32.cpp @@ -937,18 +937,37 @@ bool ethRawL2Claimed() MM_NONBLOCKING { // Consecutive failures, so a caller can distinguish back-pressure from a wedged path (see // platform.h). Written on the render task, read by the driver's 1 Hz status tick. static std::atomic ethSendFails_{0}; +// Split by cause; see platform.h. esp_eth_transmit checks the link BEFORE the MAC, so the two +// errors are genuinely distinct conditions rather than degrees of the same one. +static std::atomic ethFailLinkDown_{0}; +static std::atomic ethFailRingFull_{0}; bool ethSendRaw(const uint8_t* frame, size_t len) MM_NONBLOCKING { if (!ethHandle_ || !frame || len == 0) return false; - if (!ethLinkUp_.load(std::memory_order_relaxed)) return false; - if (esp_eth_transmit(ethHandle_, const_cast(frame), len) != ESP_OK) { + if (!ethLinkUp_.load(std::memory_order_relaxed)) { + // Counted, not silent: the driver counts every false into its own total, so skipping this + // one would make `dropped` and the per-cause totals describe different sets of frames. + // Deliberately NOT part of the streak: the streak drives wedge detection and re-arming, + // and a link genuinely down is the case a restart cannot fix. + ethFailLinkDown_.fetch_add(1, std::memory_order_relaxed); + return false; + } + const esp_err_t err = esp_eth_transmit(ethHandle_, const_cast(frame), len); + if (err != ESP_OK) { ethSendFails_.fetch_add(1, std::memory_order_relaxed); + if (err == ESP_ERR_INVALID_STATE) ethFailLinkDown_.fetch_add(1, std::memory_order_relaxed); + else if (err == ESP_ERR_NO_MEM) ethFailRingFull_.fetch_add(1, std::memory_order_relaxed); return false; } ethSendFails_.store(0, std::memory_order_relaxed); return true; } +void ethSendFailCounts(uint32_t& linkDown, uint32_t& ringFull) MM_NONBLOCKING { + linkDown = ethFailLinkDown_.load(std::memory_order_relaxed); + ringFull = ethFailRingFull_.load(std::memory_order_relaxed); +} + uint32_t ethSendFailStreak() MM_NONBLOCKING { return ethSendFails_.load(std::memory_order_relaxed); } @@ -958,6 +977,22 @@ uint32_t ethSendFailStreak() MM_NONBLOCKING { // simply ignored here, which is what the driver's own comment tells the user. bool ethBindRawInterface(const char*) { return true; } +bool ethRestartTx() { + if (!ethHandle_) return false; + // Clear our own flag first: the restart re-runs negotiation and the CONNECTED event sets it + // again if the link really comes back. Leaving it true would keep ethSendRaw trying against a + // driver that is mid-restart. + // A failed stop leaves the driver in a state we did not establish; starting on top of that + // would compound it. Report instead: the caller turns this into a "restart the device" status. + // Nothing is cleared BEFORE this point: clearing ethLinkUp_ first and then failing would leave + // transmit permanently refused behind a "no ethernet link" warning, with no event able to set + // the flag again. + if (esp_eth_stop(ethHandle_) != ESP_OK) return false; + ethLinkUp_.store(false, std::memory_order_relaxed); + ethSendFails_.store(0, std::memory_order_relaxed); + return esp_eth_start(ethHandle_) == ESP_OK; +} + // Negotiated link speed, asked of the driver rather than assumed from the PHY type: a gigabit PHY // on a 100 Mbit switch (or a bad cable) negotiates down, and that is precisely the case worth // reporting. 0 when there is no link to describe. @@ -985,8 +1020,10 @@ void ethGetIPv4(uint8_t out[4]) MM_NONBLOCKING { out[0] = out[1] = out[2 bool ethSendRaw(const uint8_t*, size_t) MM_NONBLOCKING { return false; } // no MAC to hand a frame to void ethClaimRawL2(bool) {} // no link to claim bool ethRawL2Claimed() MM_NONBLOCKING { return false; } +bool ethRestartTx() { return false; } // no driver to restart uint16_t ethLinkSpeedMbps() MM_NONBLOCKING { return 0; } // no link to describe uint32_t ethSendFailStreak() MM_NONBLOCKING { return 0; } // nothing sends, nothing fails +void ethSendFailCounts(uint32_t& a, uint32_t& b) MM_NONBLOCKING { a = b = 0; } bool ethBindRawInterface(const char*) { return true; } // no MAC, nothing to bind #endif // MM_NO_ETH diff --git a/src/platform/platform.h b/src/platform/platform.h index acf365ed..a8d2914c 100644 --- a/src/platform/platform.h +++ b/src/platform/platform.h @@ -381,12 +381,41 @@ void ethClaimRawL2(bool claim); // apart from "Ethernet carries no IP because something is driving it directly". bool ethRawL2Claimed() MM_NONBLOCKING; +// Failures split by cause since boot, because the two are different faults with different fixes and +// a single counter cannot tell them apart: +// linkDown: esp_eth_transmit refused before touching the MAC because the driver's link reads +// down. A flapping PHY shows up here, and each flap costs seconds of refusals. +// ringFull: the MAC had no free TX descriptor. That is back-pressure: our sender outrunning the +// wire, or the DMA draining slower than it should. +void ethSendFailCounts(uint32_t& linkDown, uint32_t& ringFull) MM_NONBLOCKING; + // Consecutive ethSendRaw() failures since the last success. A raw sender polls this to tell a // dropped frame (normal back-pressure, count returns to 0) from a wedged transmit path: the IDF // driver refuses every frame once ITS link flag reads down, which can outlive our own event-driven // flag and would otherwise look like a healthy link sending nothing. uint32_t ethSendFailStreak() MM_NONBLOCKING; +// Restart the Ethernet driver after transmit has provably wedged: esp_eth_transmit refuses every +// frame once the driver's internal link state reads down, and that state can diverge from both the +// PHY and our own event-driven flag, observed on an S31 under sustained TX, with the link genuinely +// lost, no DISCONNECTED event delivered, and nothing recovering short of a reboot. +// +// stop/start re-runs the driver's link negotiation, which is the only supported way back (no ioctl +// writes the link flag). Heavier than a register poke, so a caller must gate it on a long failure +// streak rather than on ordinary back-pressure. Returns true when the restart succeeded; the link +// may still be down afterwards if the cable really is out, which is the honest outcome. +// +// BLOCKS FOR UP TO ~4 SECONDS, and the caller must treat that as the cost of the call. esp_eth_start +// restarts autonegotiation, which polls the PHY status register with vTaskDelay(100 ms) up to +// autonego_timeout_ms (4000 by default, and a wedged link is down so the loop runs to the timeout). +// NOT MM_NONBLOCKING for that reason. +// +// The trade is deliberate: the tick that calls this is the 1 Hz housekeeping tick, and it only fires +// in a wedged state where every frame is being refused anyway. A stalled render loop for one tick +// costs nothing a user can see when the wall is already dark, and the alternative is a wall that +// stays dark until someone power-cycles the board. Called at most once per wedge, never in a loop. +bool ethRestartTx(); + // Negotiated link speed in Mbit/s (10 / 100 / 1000); 0 when no link or no driver. Reported rather // than enforced: panel cards want a gigabit link, and a driver that knows the actual speed can say // "100 Mbit, expect tearing" instead of either failing silently or refusing to run. @@ -420,6 +449,12 @@ void ethTestClearFrames(); void setTestEthSendFails(bool fail); // Override the reported link speed so a test can exercise the too-slow-link status. void setTestEthLinkSpeed(uint16_t mbps); +// Make ethRestartTx() fail, so a test can exercise the recovery-failed path, the one case +// that strands transmit and must stay visible rather than reading as an unplugged cable. +void setTestEthRestartFails(bool fail); +// How many times ethRestartTx() has run, so a test can pin the once-per-wedge bound (the driver's +// own restartTried_ is private, and the count is what the bound is actually about). +uint32_t ethRestartCountForTest(); #endif bool wifiStaInit(const char* ssid, const char* password); diff --git a/test/python/test_repo_health_baseline.py b/test/python/test_repo_health_baseline.py new file mode 100644 index 00000000..d5ed9881 --- /dev/null +++ b/test/python/test_repo_health_baseline.py @@ -0,0 +1,135 @@ +"""repo-health's baseline must survive a malformed snapshot without corrupting a run. + +Two separate loaders read it: `load_previous` (the COMMITTED snapshot, what the delta compares +against) and `load_working_tree` (the newest numbers, what carry-forward preserves). Both feed +`merge_carry_forward`, which does `old.get(section, {})`, so a JSON list or scalar where an object +belongs would raise, and a section holding a list would corrupt the merge. + +A malformed file must also not read as "no previous numbers": that reports every metric as new, +which looks like a clean slate rather than a broken baseline. So the rule is reject-and-announce, +never silently accept. +""" + +import json +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(ROOT / "moondeck" / "check")) + +import repo_health # noqa: E402 + + +# ---- shape validation ---- + +def test_a_snapshot_that_is_not_an_object_is_rejected(): + """A list or scalar cannot be a snapshot; accepting one would raise inside the merge.""" + assert repo_health._valid_snapshot([1, 2, 3], "t") == {} + assert repo_health._valid_snapshot("nope", "t") == {} + assert repo_health._valid_snapshot(42, "t") == {} + + +def test_a_section_of_the_wrong_shape_is_rejected(): + """`flash`/`perf`/`complexity` are merged key-by-key, so each must be an object.""" + assert repo_health._valid_snapshot({"flash": []}, "t") == {} + assert repo_health._valid_snapshot({"perf": 7}, "t") == {} + assert repo_health._valid_snapshot({"complexity": "x"}, "t") == {} + + +def test_a_valid_snapshot_passes_through_unchanged(): + """The guard must not damage the normal case; this is the control for the rejections above.""" + good = {"flash": {"esp32": 1}, "perf": {}, "complexity": {}, "other": 5} + assert repo_health._valid_snapshot(good, "t") is good + + +def test_a_snapshot_with_no_sections_is_still_valid(): + """Sections are optional: a first-ever snapshot has none, and that is not malformed.""" + assert repo_health._valid_snapshot({"lines": {"src": 10}}, "t") == {"lines": {"src": 10}} + + +# ---- the merge the validation protects ---- + +def test_carry_forward_keeps_previous_values_for_unmeasured_sections(): + """The reason the shapes matter: a run that measured nothing must not drop the old numbers.""" + old = {"flash": {"esp32": 100, "esp32s3": 200}, "perf": {"tick": 5}, "complexity": {}} + new = {"flash": {"esp32": 150}} + merged = repo_health.merge_carry_forward(new, old) + assert merged["flash"]["esp32"] == 150 # this run measured it + assert merged["flash"]["esp32s3"] == 200 # this run did not, so the old value survives + assert merged["perf"]["tick"] == 5 + + +def test_carry_forward_survives_a_rejected_baseline(): + """A malformed baseline degrades to "nothing to carry", not to a crash.""" + merged = repo_health.merge_carry_forward({"flash": {"esp32": 1}}, + repo_health._valid_snapshot([1, 2], "t")) + assert merged["flash"] == {"esp32": 1} + + +# ---- the loaders ---- + +def test_malformed_json_on_disk_reads_as_empty(tmp_path, monkeypatch): + """Unparseable JSON is announced and treated as empty rather than raising mid-run.""" + bad = tmp_path / "repo-health.json" + bad.write_text("{ not json", encoding="utf-8") + monkeypatch.setattr(repo_health, "HEALTH_FILE", bad) + assert repo_health.load_working_tree() == {} + + +def test_a_wrong_shape_on_disk_reads_as_empty(tmp_path, monkeypatch): + """Valid JSON of the wrong shape is rejected by the same rule as unparseable JSON.""" + bad = tmp_path / "repo-health.json" + bad.write_text(json.dumps([1, 2, 3]), encoding="utf-8") + monkeypatch.setattr(repo_health, "HEALTH_FILE", bad) + assert repo_health.load_working_tree() == {} + + +def test_a_good_file_on_disk_loads(tmp_path, monkeypatch): + """The control: the loader must actually load a well-formed snapshot.""" + good = tmp_path / "repo-health.json" + good.write_text(json.dumps({"flash": {"esp32": 1}}), encoding="utf-8") + monkeypatch.setattr(repo_health, "HEALTH_FILE", good) + assert repo_health.load_working_tree() == {"flash": {"esp32": 1}} + + +# ---- the git baseline (the behaviour this branch adds) ---- + +def test_the_baseline_comes_from_the_commit_not_the_working_tree(tmp_path, monkeypatch): + """Running the check twice must give the same delta. + + Reading the working-tree file made every run after the first compare against the run BEFORE it, + so a second run inside one commit showed a delta of ~0 and the real change vanished. The + baseline has to be the committed snapshot. + """ + import subprocess + repo = tmp_path / "repo" + (repo / "docs" / "metrics").mkdir(parents=True) + health = repo / "docs" / "metrics" / "repo-health.json" + + run = lambda *a: subprocess.run(a, cwd=repo, check=True, capture_output=True) + run("git", "init", "-q") + run("git", "config", "user.email", "t@t") + run("git", "config", "user.name", "t") + health.write_text(json.dumps({"flash": {"esp32": 100}}), encoding="utf-8") + run("git", "add", "-A") + run("git", "commit", "-qm", "baseline") + + # A later run overwrites the working tree; the COMMITTED value must still be the baseline. + health.write_text(json.dumps({"flash": {"esp32": 999}}), encoding="utf-8") + + monkeypatch.setattr(repo_health, "ROOT", repo) + monkeypatch.setattr(repo_health, "HEALTH_FILE", health) + assert repo_health.load_previous() == {"flash": {"esp32": 100}} # from the commit + assert repo_health.load_working_tree() == {"flash": {"esp32": 999}} # from disk + + +def test_the_baseline_falls_back_to_disk_when_git_cannot_answer(tmp_path, monkeypatch): + """A fresh checkout with no commit yet must still produce a first snapshot rather than erroring.""" + repo = tmp_path / "norepo" + (repo / "docs" / "metrics").mkdir(parents=True) + health = repo / "docs" / "metrics" / "repo-health.json" + health.write_text(json.dumps({"flash": {"esp32": 7}}), encoding="utf-8") + + monkeypatch.setattr(repo_health, "ROOT", repo) + monkeypatch.setattr(repo_health, "HEALTH_FILE", health) + assert repo_health.load_previous() == {"flash": {"esp32": 7}} diff --git a/test/scenarios/light/scenario_perf_full.json b/test/scenarios/light/scenario_perf_full.json index b4db61b9..acd66ca4 100644 --- a/test/scenarios/light/scenario_perf_full.json +++ b/test/scenarios/light/scenario_perf_full.json @@ -1956,7 +1956,7 @@ "desktop-macos": { "tick_us": [ 1, - 8 + 15 ], "free_heap": [ 0, @@ -1968,7 +1968,7 @@ ], "at": [ "2026-06-17", - "2026-07-10" + "2026-07-31" ] }, "esp32": { diff --git a/test/scenarios/light/scenario_peripheral_grid_sweep.json b/test/scenarios/light/scenario_peripheral_grid_sweep.json index ec1be1eb..bc1bd77e 100644 --- a/test/scenarios/light/scenario_peripheral_grid_sweep.json +++ b/test/scenarios/light/scenario_peripheral_grid_sweep.json @@ -645,7 +645,7 @@ "desktop-macos": { "tick_us": [ 68, - 227 + 350 ], "free_heap": [ 0, @@ -657,7 +657,7 @@ ], "at": [ "2026-07-26", - "2026-07-28" + "2026-07-31" ] } } @@ -1311,7 +1311,7 @@ "desktop-macos": { "tick_us": [ 68, - 245 + 284 ], "free_heap": [ 0, @@ -1323,7 +1323,7 @@ ], "at": [ "2026-07-26", - "2026-07-28" + "2026-07-31" ] } } diff --git a/test/unit/light/unit_DistortionWavesEffect.cpp b/test/unit/light/unit_DistortionWavesEffect.cpp index 5ea0430e..5f14e3aa 100644 --- a/test/unit/light/unit_DistortionWavesEffect.cpp +++ b/test/unit/light/unit_DistortionWavesEffect.cpp @@ -4,6 +4,11 @@ #include "light/layouts/Layouts.h" #include "light/effects/DistortionWavesEffect.h" #include "light/layouts/GridLayout.h" +#include "platform/platform.h" // setTestNowMs: freeze millis() for a deterministic phase + +// Restore the real clock after any test that froze it, so a frozen value can't leak into +// order-dependent neighbours. +namespace { struct ClockGuard { ~ClockGuard() { mm::platform::setTestNowMs(0); } }; } static void buildLayer(mm::Layouts& layouts, mm::GridLayout& grid, mm::Layer& layer, mm::DistortionWavesEffect& fx, @@ -28,6 +33,14 @@ TEST_CASE("DistortionWavesEffect writes non-zero RGB data") { } TEST_CASE("DistortionWavesEffect produces spatial variation") { + // The clock is FROZEN because the pattern moves with time: a light's hue is the average of a + // horizontal and a vertical sine, both advanced by the elapsed-time phase. Layer::tick() reads + // platform::millis(), so a single-tick test starts at whatever the process uptime happens to + // be, and at ~2.3% of phases the two sampled lights land on the same hue and this assertion + // fails. It flaked in CI on exactly that. Freezing makes the phase (and so the property) fixed. + ClockGuard guard; + mm::platform::setTestNowMs(1); + mm::Layouts layouts; mm::GridLayout grid; mm::Layer layer; mm::DistortionWavesEffect fx; buildLayer(layouts, grid, layer, fx, 16, 16, 1); auto& buf = layer.buffer(); diff --git a/test/unit/light/unit_PanelCardDriver.cpp b/test/unit/light/unit_PanelCardDriver.cpp index 96c8781f..2b5ddaad 100644 --- a/test/unit/light/unit_PanelCardDriver.cpp +++ b/test/unit/light/unit_PanelCardDriver.cpp @@ -14,6 +14,7 @@ #include "light/layouts/Layouts.h" #include "light/layouts/GridLayout.h" #include "light/layouts/PanelsLayout.h" +#include // status() substring checks namespace { @@ -56,6 +57,12 @@ void setUp(mm::PanelCardDriver& driver, mm::Buffer& source, Wall& wall, // destroyed without release() being called, so the claim count would otherwise carry across cases. void clearClaims() { while (mm::platform::ethRawL2Claimed()) mm::platform::ethClaimRawL2(false); + // The platform's simulated-failure switches are process-global: a test that leaves one set + // poisons whichever test runs next, which is how a passing-alone/failing-in-sequence result + // appears. Reset them here so every case starts from a known platform state. + mm::platform::setTestEthSendFails(false); + mm::platform::setTestEthRestartFails(false); + mm::platform::ethTestClearFrames(); } // The packet-type byte of a captured frame (offset 12), which says row / sync / brightness. @@ -393,3 +400,111 @@ TEST_CASE("PanelCardDriver sends nothing when the buffer covers no row") { CHECK(mm::platform::ethTestFrameCount() == 0); } + +// A transmit path that refuses everything gets ONE recovery attempt, not one per second: a restart +// cannot fix an unplugged cable, and retrying would tear the interface down repeatedly under a user +// who is watching the card to find out what is wrong. +// Drive the failure streak past the wedge threshold. Split out because a test that does not REBUILD +// the streak between ticks proves nothing: the restart clears it, so the wedge branch is simply not +// re-entered and the guard under test never runs. +static void wedge(mm::PanelCardDriver& driver, uint32_t fromMs) { + mm::platform::setTestEthSendFails(true); + for (int i = 0; i < 200; i++) { + mm::platform::setTestNowMs(fromMs + i * 30); + driver.tick(); + } + REQUIRE(mm::platform::ethSendFailStreak() >= 500); +} + +// A wedge earns ONE recovery attempt. The second and third wedges in the same episode must NOT fire +// again: a restart cannot fix an unplugged cable, and retrying would tear the interface down every +// tick under a user reading the card to find out what is wrong. +TEST_CASE("PanelCardDriver attempts recovery once per wedge") { + clearClaims(); + mm::Buffer source; + mm::PanelCardDriver driver; + Wall wall(64, 1); + setUp(driver, source, wall, 64); + + wedge(driver, 1000); + const uint32_t before = mm::platform::ethRestartCountForTest(); + driver.tick1s(); + CHECK(mm::platform::ethRestartCountForTest() == before + 1); // fired + + // Rebuild the streak so the wedge branch is genuinely re-entered; without this the guard is + // never reached and the assertion below would pass even with the guard deleted. + wedge(driver, 20000); + driver.tick1s(); + wedge(driver, 40000); + driver.tick1s(); + CHECK(mm::platform::ethRestartCountForTest() == before + 1); // and still only once + + mm::platform::setTestEthSendFails(false); +} + +// A wedge that survives its restart is REPORTED, with the refusal count, rather than retried. +TEST_CASE("PanelCardDriver reports a wedge that survives its restart") { + clearClaims(); + mm::Buffer source; + mm::PanelCardDriver driver; + Wall wall(64, 1); + setUp(driver, source, wall, 64); + + wedge(driver, 5000); + driver.tick1s(); // fires the one restart + wedge(driver, 25000); // still wedged afterwards + driver.tick1s(); + + CHECK(std::string(driver.status()).find("transmit wedged") != std::string::npos); + mm::platform::setTestEthSendFails(false); +} + +// A recovery that FAILS leaves the interface stopped, which no user action short of a power cycle +// resolves. That must not be reported as "no ethernet link", the message for an unplugged cable. +TEST_CASE("PanelCardDriver keeps a failed restart visible") { + clearClaims(); + mm::Buffer source; + mm::PanelCardDriver driver; + Wall wall(64, 1); + setUp(driver, source, wall, 64); + + mm::platform::setTestEthRestartFails(true); + wedge(driver, 60000); + driver.tick1s(); + CHECK(std::string(driver.status()).find("restart failed") != std::string::npos); + + // Latched: the next tick must not soften it to the link warning. + driver.tick1s(); + CHECK(std::string(driver.status()).find("restart failed") != std::string::npos); + + mm::platform::setTestEthRestartFails(false); + mm::platform::setTestEthSendFails(false); +} + +// Failures are counted by CAUSE, because a down link and a full TX ring are different faults with +// different fixes; one total cannot tell them apart, which is what made a real bug unreadable. +TEST_CASE("PanelCardDriver counts send failures by cause") { + clearClaims(); + mm::platform::ethTestClearFrames(); + mm::Buffer source; + mm::PanelCardDriver driver; + Wall wall(64, 1); + setUp(driver, source, wall, 64); + + uint32_t linkDown = 0, ringFull = 0; + mm::platform::ethSendFailCounts(linkDown, ringFull); + CHECK(ringFull == 0); + + mm::platform::setTestEthSendFails(true); + mm::platform::setTestNowMs(30000); + driver.tick(); + mm::platform::setTestEthSendFails(false); + + mm::platform::ethSendFailCounts(linkDown, ringFull); + CHECK(ringFull > 0); // cumulative, not the streak; it survives a later success + mm::platform::setTestNowMs(30100); + driver.tick(); + uint32_t after = 0; + mm::platform::ethSendFailCounts(linkDown, after); + CHECK(after == ringFull); // a success does not erase the history +}