feat(rtps_embedded): embeddedRTPS integration + idiomatic espp refactor (facade, SocketReactor, protocol scheduler, Micro-CDR removal) - #706
Merged
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cade The example still used the raw engine API (rtps::Domain, ucdr, xTaskCreate, ESP_LOG). Rewrite it as the facade showcase: espp::RtpsParticipant with Config callbacks (publisher/subscriber matched), espp::CdrWriter/CdrReader payloads, espp::Timer for the periodic publish, and espp logging throughout. Topics/types are unchanged so it still pairs with the FastDDS host peer (example/pc/host_pubsub.cpp); ROS 2 naming is documented inline. Snippet markers preserved for the docs. Builds for esp32 on ESP-IDF 6.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eers Multi-interface hosts (e.g. a mac with WiFi + a USB-ethernet adapter direct- attached to the esp32) need to pin the RTPS interface; auto-detect stays the default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d-rtps # Conflicts: # .gitmodules # python/rtps_publisher.py # python/rtps_pubsub.py # python/rtps_subscriber.py
…ven cdr Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the engine's ~157 ucdr call sites onto the vendored reflection-cdr library's stream primitives, wire-format neutral: - New rtps/utils/CdrBuffer.hpp: thin aliases (cdr::basic_writer<xcdr1, span_sink> / cdr::reader<xcdr1>) plus small helpers reproducing ucdr's raw-bytes / clamp-to-end skip / align-to-4 semantics over the engine's fixed buffers. cdr::param_list_writer was deliberately NOT used inside the engine: it emits its own encapsulation header and requires a heap sink, neither of which matches the engine's mid-buffer fixed-array usage - byte identity outranks library purity. - TopicData / ParticipantProxyData / SPDPAgent / SEDPAgent / Locator migrated 1:1 (ucdrBuffer signatures -> span/writer parameters), preserving every wire quirk the goldens froze: lazily-zeroed alignment padding, length-4/write-2+pad PID_PROTOCOL_VERSION/PID_VENDORID, string null-in-length + pad-to-4, sentinel handling, and SEDP's historical parse of the encapsulation header as an unknown pid (now documented). - Micro-CDR submodule deleted (.gitmodules, thirdparty/, lib/espp.cmake, IDF CMakeLists); cdr added to the component's REQUIRES + manifest; golden test plumbing adapted to the new signatures with the golden bytes untouched. Gates: rtps_embedded_golden byte-identical (goldens never regenerated); engine + facade loopbacks PASS; docker interop 8/8 PASS vs ROS 2 - including SPDP/SEDP discovery announcements built entirely by the new codec; esp32 example builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pc/build-linux/ is created by components/rtps_embedded/interop when it builds the espp lib inside the container against the bind-mounted source tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR continues the components/rtps_embedded refactor toward an idiomatic ESPP component, including a new espp::RtpsParticipant facade, receive-path consolidation via SocketReactor, and Micro-CDR removal in favor of the reflection-driven cdr component, plus supporting interop/golden tests and related build/docs updates.
Changes:
- Add host-side golden/interop tests and a Docker-based FastDDS/ROS 2 interop harness for
rtps_embedded. - Replace Micro-CDR usage with
cdrstream primitives and reflection-driven serialization; remove CDR Python bindings accordingly. - Reduce RTPS engine concurrency primitives (ThreadPool removal; introduce
SocketReactor/shared worker pool) and add unicast-port reuse control plumbing.
Reviewed changes
Copilot reviewed 103 out of 104 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| suppressions.txt | Suppress static analysis warnings for new vendored detail/ trees. |
| python/rtps_subscriber.py | Update Python RTPS subscriber to string payloads + new facade API usage. |
| python/rtps_pubsub.py | Remove old in-process pub/sub Python test script. |
| python/rtps_publisher.py | Update Python RTPS publisher to string payloads + new facade API usage. |
| pc/tests/rtps_subscriber.cpp | Remove old standalone host subscriber test. |
| pc/tests/rtps_pubsub.cpp | Remove old standalone in-process host pub/sub test. |
| pc/tests/rtps_publisher.cpp | Remove old standalone host publisher test. |
| pc/tests/rtps_facade_pubsub.cpp | Add in-process facade loopback test (port probing / multi participant). |
| pc/tests/rtps_embedded_pubsub.cpp | Add host loopback pub/sub baseline test for embeddedRTPS engine. |
| pc/tests/rtps_embedded_interop_sub.cpp | Add host interop subscriber test for facade against DDS/ROS 2 peers. |
| pc/tests/rtps_embedded_interop_pub.cpp | Add host interop publisher test for facade against DDS/ROS 2 peers. |
| pc/tests/rtps_embedded_golden.inc | Add frozen golden wire-format byte arrays. |
| pc/tests/rtps_embedded_golden.cpp | Add golden wire-format tests for embeddedRTPS engine message encodings. |
| pc/tests/rtps_common.hpp | Migrate test helpers from espp::CdrWriter/CdrReader to cdr (expected-based). |
| pc/CMakeLists.txt | Bump host build standard to C++23. |
| lib/python_bindings/module.cpp | Remove registration of CDR Python bindings; keep RTPS + SocketReactor bindings. |
| lib/python_bindings/cdr_bindings.cpp | Remove hand-written pybind11 CDR bindings. |
| lib/include/espp.hpp | Switch public include from legacy rtps.hpp to rtps_participant.hpp. |
| lib/espp.cmake | Update PC build include/source lists for rtps_embedded, add reflect-cpp/cdr include dirs, drop old cdr/rtps sources, drop cdr bindings. |
| lib/CMakeLists.txt | Bump library build standard to C++23. |
| lib/autogenerate_bindings.py | Update notes re: excluding cdr.hpp from binding generation (no Python bindings). |
| doc/en/data/reflect_cpp.rst | Add docs page for reflect_cpp component. |
| doc/en/data/index.rst | Add reflect_cpp to data docs index. |
| doc/en/data/cdr.rst | Update CDR docs to reflection-driven cdr library and C++23 requirement. |
| doc/Doxyfile | Add jd9365.hpp to Doxygen inputs. |
| components/socket/src/socket.cpp | Add Socket::disable_reuse() to clear reuse flags before binding. |
| components/socket/include/socket.hpp | Document and expose Socket::disable_reuse() API. |
| components/rtps/src/rtps.cpp | Migrate RTPS parameter list CDR handling to reflection-driven cdr (and manual PL_CDR header). |
| components/rtps/example/main/rtps_example.cpp | Migrate example serialization to cdr::serialize/deserialize. |
| components/rtps_embedded/src/ThreadPool.cpp | Remove embeddedRTPS engine ThreadPool implementation. |
| components/rtps_embedded/src/rtps_participant.cpp | Add espp::RtpsParticipant facade implementation. |
| components/rtps_embedded/src/discovery/SEDPAgent.cpp | Replace Micro-CDR usage with rtps::CdrBuffer helpers for SEDP. |
| components/rtps_embedded/src/discovery/ParticipantProxyData.cpp | Replace Micro-CDR parsing with rtps::CdrReader-based parsing helpers. |
| components/rtps_embedded/src/communication/EsppTransport.cpp | Refactor transport receive path to SocketReactor + shared worker pool; add unicast reuse control; add stop/submit helpers. |
| components/rtps_embedded/README.md | Update README to reflect Micro-CDR removal and new discovery serialization approach. |
| components/rtps_embedded/interop/run.sh | Add host entrypoint script to build/run interop harness. |
| components/rtps_embedded/interop/run_interop.sh | Add container-side build + FastDDS/ROS 2 interop matrix script. |
| components/rtps_embedded/interop/README.md | Add documentation for the RTPS interop harness. |
| components/rtps_embedded/interop/Dockerfile | Add Docker image definition for interop harness (ROS jazzy base). |
| components/rtps_embedded/include/rtps/utils/CdrBuffer.hpp | Add Micro-CDR-compatible thin adapter over cdr stream primitives (discovery). |
| components/rtps_embedded/include/rtps/ThreadPool.hpp | Remove embeddedRTPS engine ThreadPool header. |
| components/rtps_embedded/include/rtps/entities/Writer.hpp | Remove ThreadPool pointer/member from Writer base. |
| components/rtps_embedded/include/rtps/entities/StatelessWriter.tpp | Route writer progress scheduling through transport worker pool instead of ThreadPool. |
| components/rtps_embedded/include/rtps/entities/StatelessWriter.hpp | Update StatelessWriter init signature (remove ThreadPool parameter). |
| components/rtps_embedded/include/rtps/entities/StatefulWriter.tpp | Remove per-writer heartbeat thread; schedule progress via transport pool; add heartbeat tick + protocol nudge hooks. |
| components/rtps_embedded/include/rtps/entities/StatefulWriter.hpp | Add heartbeat tick + protocol nudge API and state; update init signature. |
| components/rtps_embedded/include/rtps/entities/Domain.hpp | Replace ThreadPool usage with transport/SocketReactor receive + single protocol scheduler task. |
| components/rtps_embedded/include/rtps/discovery/TopicData.hpp | Replace Micro-CDR API with readFromBuffer / serializeInto using CdrBuffer helpers. |
| components/rtps_embedded/include/rtps/discovery/SPDPAgent.hpp | Replace Micro-CDR with CdrBuffer usage; add announce() and scheduler integration notes. |
| components/rtps_embedded/include/rtps/discovery/ParticipantProxyData.hpp | Replace Micro-CDR dependency with CdrBuffer reader. |
| components/rtps_embedded/include/rtps/config_desktop.hpp | Fix desktop GUID prefix config to avoid identity collisions (GUID_RANDOM). |
| components/rtps_embedded/include/rtps/communication/EsppTransport.hpp | Add reactor/worker-pool wiring, unicast reuse semantics, and stop/submit APIs. |
| components/rtps_embedded/include/rtps/common/Locator.hpp | Replace Micro-CDR locator read/write with CdrBuffer raw read helper. |
| components/rtps_embedded/include/rtps_participant.hpp | Add public espp::RtpsParticipant facade header/API. |
| components/rtps_embedded/idf_component.yml | Add dependency on espp/cdr. |
| components/rtps_embedded/example/sdkconfig.defaults | Adjust example defaults (perf opts, 1kHz FreeRTOS tick, CPU freq). |
| components/rtps_embedded/example/main/main.cpp | Port embedded example to espp::RtpsParticipant facade + cdr serialization. |
| components/rtps_embedded/example/main/CMakeLists.txt | Add required components for updated example (cdr/timer/etc.). |
| components/rtps_embedded/example/CMakeLists.txt | Add cdr/timer to example components list. |
| components/rtps_embedded/CMakeLists.txt | Drop Micro-CDR sources/includes; add facade source; add cdr dependency. |
| components/reflect_cpp/README.md | Add reflect_cpp component README. |
| components/reflect_cpp/idf_component.yml | Add new component manifest for reflect-cpp headers. |
| components/reflect_cpp/CMakeLists.txt | Register reflect-cpp header-only include dirs. |
| components/ethernet/idf_component.yml | Update dependency names for DM9051 / ENC28J60 components. |
| components/esp32-p4-nano/src/camera.cpp | Adjust SCCB/I2C frequency to 100kHz for camera reliability. |
| components/esp32-p4-nano/src/audio.cpp | Add clear_audio() and make I2S writes frame-aligned/constant cadence. |
| components/esp32-p4-nano/Kconfig.projbuild | Add JD9365 panel option and make it default. |
| components/esp32-p4-nano/include/esp32-p4-nano.hpp | Add JD9365 display support, panel params updates, touch inversion changes, backlight I2C notes, add clear_audio(). |
| components/esp32-p4-nano/example/main/gui.hpp | Add pending-point queue to make draw_circle non-blocking. |
| components/esp32-p4-nano/example/main/gui.cpp | Disable tab scroll flags; drain queued draw points in GUI task; implement queued draw path. |
| components/esp32-p4-nano/example/main/esp32_p4_nano_example.cpp | Improve touch feedback behavior; parse WAV data chunk robustly. |
| components/esp32-p4-function-ev-board/src/audio.cpp | Make I2S writes frame-aligned/constant cadence. |
| components/esp32-p4-function-ev-board/example/main/esp32_p4_function_ev_board_example.cpp | Migrate uint32 CDR helpers to cdr::serialize/deserialize. |
| components/esp32-p4-eth/src/camera.cpp | Adjust SCCB/I2C frequency to 100kHz for camera reliability. |
| components/esp32-p4-eth/src/audio.cpp | Add clear_audio() and make I2S writes frame-aligned/constant cadence. |
| components/esp32-p4-eth/Kconfig.projbuild | Add JD9365 panel option and make it default. |
| components/esp32-p4-eth/include/esp32-p4-eth.hpp | Add JD9365 display support, panel params updates, touch inversion changes, backlight I2C notes, add clear_audio(). |
| components/esp32-p4-eth/example/main/gui.hpp | Add pending-point queue to make draw_circle non-blocking. |
| components/esp32-p4-eth/example/main/gui.cpp | Disable tab scroll flags; drain queued draw points in GUI task; implement queued draw path. |
| components/esp32-p4-eth/example/main/esp32_p4_eth_example.cpp | Improve touch feedback behavior; parse WAV data chunk robustly. |
| components/cdr/src/cdr.cpp | Remove now-unneeded translation unit (cdr library is header-only). |
| components/cdr/README.md | Update README for reflection-driven cdr library and new feature set. |
| components/cdr/idf_component.yml | Update manifest for new cdr implementation and add reflect_cpp dependency + IDF>=5.2. |
| components/cdr/example/README.md | Update example README for new reflection-driven API and features. |
| components/cdr/example/main/cdr_example.cpp | Rewrite example to demonstrate cdr::serialize/deserialize, XCDR1/XCDR2, expected errors, serialize_into, and PL_CDR param lists. |
| components/cdr/example/CMakeLists.txt | Add reflect_cpp dependency and bump example to C++23. |
| components/cdr/CMakeLists.txt | Register header-only includes (detail+cdr) and require reflect_cpp. |
| .gitmodules | Remove Micro-CDR submodule; add reflect-cpp and cdr submodules. |
| .gitignore | Ignore docker interop harness build tree (pc/build-linux/). |
| .github/workflows/upload_components.yml | Add components/reflect_cpp to upload list. |
| .github/workflows/rtps_interop.yml | Add CI workflow to run dockerized RTPS interop matrix on relevant PR changes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
15 tasks
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Addressed all review comments in 5c1573c:
Verified: host lib + tests build, golden wire tests byte-identical. |
|
✅Static analysis result - no issues found! ✅ |
…auto-detect Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adding rtps_embedded to the host lib build compiles the engine with MSVC for the first time, exposing a latent portability bug: MemoryPoolIterator held a `MemoryPool&` reference member, which implicitly deletes copy-assignment. MSVC's std::find_if (used over m_proxies in the reliable writer/reader paths) requires Cpp17InputIterator, i.e. a copy-assignable iterator, so it failed with C2280; libstdc++/libc++ happened not to require the assignment and built fine. - Hold `MemoryPool*` instead of `MemoryPool&` (reseatable -> copy-assignable); deref sites updated to `->`. - Restore operator== (was commented out) as std::find_if also requires it; operator!= now defers to it. No behavior/wire change. Gates: POSIX lib + tests build; golden byte-identical; engine + facade reliable loopbacks PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
guo-max
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #672. This branch forked from #672's head, then integrated current
main(the new reflectioncdrcomponent,SocketReactor, C++23) via a clean merge and refactored on top — so it sidesteps the painful rebase of #672 onto today'smain. Targetingmaindirectly, it is a single self-contained PR: the complete embeddedRTPS integration and its refactor toward an idiomatic espp component.What this is
components/rtps_embeddedis an RTPS/DDS stack (the embeddedRTPS port) that interoperates with FastDDS and ROS 2 over the standard RTPS wire protocol — so an esp32 can pub/sub with a PC/Jetson running ROS 2 or any DDS participant on the same network. This PR contains the original integration plus an incremental, interop-gated refactor toward an idiomatic espp component, followingcomponents/rtps_embedded/REFACTOR_PLAN.md.The guiding invariant: never break FastDDS / ROS 2 wire compatibility. Every refactor phase is gated by a golden byte-for-byte wire test plus a dockerized ROS 2 interop matrix, and the esp32 example builds throughout. Validated on real hardware against a separate machine running FastDDS.
Refactor phases (each a commit series, individually interop-gated)
espp::RtpsParticipantfacadeSocket::disable_reuse()SocketReactorcdrcdrMeasured impact (per participant process)
publish()). Separately, lost-sample recovery latency dropped from up to ~2 s (a reader learned of a gap only on the next periodic HEARTBEAT,SF_WRITER_HB_PERIOD_MS) to prompt-on-next-publish via the piggyback (and ≤ theperiod/4≈ 500 ms speedup path while any sample is unacknowledged)stop(): ~4 s → near-instantHow it's verified
pc/tests/rtps_embedded_golden.cpp): byte-exact SPDP/SEDP/DATA/HEARTBEAT/ACKNACK/GAP + the SEDP parameter list. Never regenerated in this PR.components/rtps_embedded/interop/, wired into CI): builds the engine on Linux and runs espp↔espp loopback + cross-process, and espp↔ROS 2 (jazzy / rmw_fastrtps) reliable both directions + best-effort. 8/8 green.esp32p4on ESP-IDF 6.0 after every phase.Follow-ups (separate PRs)
Remaining plan phases — de-templating / hierarchy collapse, runtime
Limits, typedPublisher<T>/Subscriber<T>(now natural with reflect-cpp: publish any reflectable struct), final consolidation into a singlertpscomponent — seeREFACTOR_PLAN.md§5.🤖 Generated with Claude Code