From 430ca42a7935b16ab8f29d582a79d762ccfc3469 Mon Sep 17 00:00:00 2001 From: William Emfinger Date: Tue, 11 Aug 2026 10:40:50 -0500 Subject: [PATCH 1/3] feat(cdr)!: replace manual CdrWriter/CdrReader with reflection-driven library MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cdr component now vendors finger563/cdr (detail/ submodule): the compiler generates CDR serialization from plain struct definitions — no IDL, no hand-written write/read sequences. Adds XCDR2 (delimited/ appendable + DHEADER schema evolution), both endiannesses, std::expected errors with field names, bounded types, and PL_CDR parameter-list helpers for RTPS discovery. Wire format is byte-verified against pycdr2 (CycloneDDS). Depends on the new reflect_cpp component; requires C++23 (default on IDF 5.2+ toolchains). BREAKING CHANGE: espp::CdrWriter / espp::CdrReader are gone; use cdr::serialize / cdr::deserialize / cdr::param_list_writer instead. Example builds on ESP32-S3 / IDF 6 (389 KB total image incl. logger+fmt). Co-Authored-By: Claude Fable 5 --- .gitmodules | 3 + components/cdr/CMakeLists.txt | 7 +- components/cdr/README.md | 79 +-- components/cdr/detail/cdr | 1 + components/cdr/example/CMakeLists.txt | 4 +- components/cdr/example/main/cdr_example.cpp | 116 ++-- components/cdr/idf_component.yml | 8 +- components/cdr/include/cdr.hpp | 569 +------------------- components/cdr/src/cdr.cpp | 1 - 9 files changed, 159 insertions(+), 629 deletions(-) create mode 160000 components/cdr/detail/cdr delete mode 100644 components/cdr/src/cdr.cpp diff --git a/.gitmodules b/.gitmodules index d562b34f0..00c45af8d 100644 --- a/.gitmodules +++ b/.gitmodules @@ -40,3 +40,6 @@ [submodule "components/reflect_cpp/detail/reflect-cpp"] path = components/reflect_cpp/detail/reflect-cpp url = https://github.com/getml/reflect-cpp.git +[submodule "components/cdr/detail/cdr"] + path = components/cdr/detail/cdr + url = https://github.com/finger563/cdr.git diff --git a/components/cdr/CMakeLists.txt b/components/cdr/CMakeLists.txt index acbe2799e..91f3029e4 100644 --- a/components/cdr/CMakeLists.txt +++ b/components/cdr/CMakeLists.txt @@ -1,3 +1,6 @@ +# The cdr library (detail/cdr, github.com/finger563/cdr) is header-only; its +# reflection backend comes from the reflect_cpp component. Interface-only +# registration, like the serialization component. idf_component_register( - INCLUDE_DIRS "include" - SRC_DIRS "src") + INCLUDE_DIRS "detail/cdr/include" "include" + REQUIRES reflect_cpp) diff --git a/components/cdr/README.md b/components/cdr/README.md index 3a9e941e9..f086bc14a 100644 --- a/components/cdr/README.md +++ b/components/cdr/README.md @@ -1,34 +1,51 @@ -# CDR (Common Data Representation) Component - -[![Badge](https://components.espressif.com/components/espp/cdr/badge.svg)](https://components.espressif.com/components/espp/cdr) - -The `cdr` component provides a small, standalone Common Data Representation -(CDR) reader/writer utility aimed at standards-oriented protocols such as -DDS/RTPS. - -This initial slice is intentionally focused on the most immediately useful -pieces for building interoperable payloads: - -- encapsulation identifiers for `CDR_BE`, `CDR_LE`, `PL_CDR_BE`, and `PL_CDR_LE` -- endian-aware primitive read/write helpers -- CDR alignment and padding handling -- string serialization helpers using the standard CDR length-prefix + null terminator format -- headerless/body helpers for CDR fields embedded inside larger protocol elements -- fixed-array helpers and zero-copy payload/span views -- sequence helpers for homogeneous primitive collections -- standalone usage without depending on RTPS or DDS layers - -Current scope: - -- good fit for building RTPS payloads and parameter lists incrementally -- designed to stay reusable outside DDS/RTPS -- **not** yet a full DDS XTypes / XCDR2 implementation +# CDR (Common Data Representation) + +Reflection-driven CDR/XCDR serialization for plain C++ structs — no IDL +compiler, no hand-written read/write call sequences. The compiler generates +the serialization code from the struct definition itself (the same usability +pattern as the `serialization` component's alpaca, with a DDS/RTPS-compatible +wire format instead). + +The library is [finger563/cdr](https://github.com/finger563/cdr), vendored as +a git submodule under `detail/`; this component wires it into ESP-IDF and +depends on the `reflect_cpp` component for its reflection backend. See the +library's `README.md` for the supported type mapping and +`docs/DESIGN.md` for the architecture and wire-format rules. + +```cpp +struct ImuSample { + uint64_t stamp_us; + std::array accel; + std::array gyro; + float temperature; +}; + +auto bytes = cdr::serialize(sample); // XCDR2, appendable (default) +auto ros2 = cdr::serialize(sample); // ROS 2 / classic-CDR peers +auto back = cdr::deserialize(*bytes); // std::expected +``` + +Highlights: + +- XCDR1 (plain CDR — ROS 2, CycloneDDS defaults) and XCDR2 (plain + + delimited/appendable with DHEADER — FastDDS, OpenDDS defaults), both + endiannesses, with appendable schema evolution in both directions. +- Wire format byte-verified against pycdr2 (CycloneDDS's codec); the Python + side of a message is a plain `pycdr2` dataclass. +- `std::expected` error handling with error code, payload offset, and field + name; bounds-checked, fuzz-tested deserializers. +- `cdr::param_list_writer` / `param_list_reader` for the PL_CDR parameter + lists RTPS discovery (SPDP/SEDP) uses. +- Zero-allocation `cdr::serialize_into` and body-only variants for RTPS + submessage composition. + +Requires C++23 (`std::expected`), the default on ESP-IDF 5.2+ toolchains. + +This component replaces the previous manual `espp::CdrWriter`/`espp::CdrReader` +API (an imperative XCDR1-only reader/writer); the RTPS component and examples +have been migrated to the reflection-driven API. ## Example -The [example](./example) demonstrates a small round-trip using: - -- a little-endian CDR encapsulation header -- primitive values -- a CDR string -- a `uint16_t` sequence +The [example](./example) shows struct round-trips in both XCDR versions, the +zero-allocation path, error handling, and PL_CDR parameter lists. diff --git a/components/cdr/detail/cdr b/components/cdr/detail/cdr new file mode 160000 index 000000000..15d5aac85 --- /dev/null +++ b/components/cdr/detail/cdr @@ -0,0 +1 @@ +Subproject commit 15d5aac857997024bc743f5b785cf9c4567f5ebc diff --git a/components/cdr/example/CMakeLists.txt b/components/cdr/example/CMakeLists.txt index 491c40697..2fbb53b36 100644 --- a/components/cdr/example/CMakeLists.txt +++ b/components/cdr/example/CMakeLists.txt @@ -11,11 +11,11 @@ set(EXTRA_COMPONENT_DIRS set( COMPONENTS - "main esptool_py cdr logger" + "main esptool_py cdr reflect_cpp logger" CACHE STRING "List of components to include" ) project(cdr_example) -set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD 23) diff --git a/components/cdr/example/main/cdr_example.cpp b/components/cdr/example/main/cdr_example.cpp index 4942ce15e..2816d1ebc 100644 --- a/components/cdr/example/main/cdr_example.cpp +++ b/components/cdr/example/main/cdr_example.cpp @@ -1,4 +1,3 @@ -#include #include #include #include @@ -6,63 +5,84 @@ #include "cdr.hpp" #include "logger.hpp" -extern "C" void app_main(void) { - espp::Logger logger({.tag = "cdr_example", .level = espp::Logger::Verbosity::INFO}); - - std::array input_magic{'C', 'D', 'R', '!'}; - std::array input_values{10, 20, 30}; - - //! [cdr example] - espp::CdrWriter writer({ - .encapsulation = espp::CdrEncapsulation::CDR_LE, - .include_encapsulation = true, - }); - writer.write(42); - writer.write(3.25f); - writer.write_string("hello cdr"); - writer.write_sequence(input_values); +namespace { - auto payload = writer.take_buffer(); - logger.info("Serialized {} bytes of CDR data", payload.size()); +// A plain aggregate is all it takes — the compiler generates the +// serialization code from the struct definition itself. Types default to +// @appendable extensibility (XTypes default); opt into @final with +// `static constexpr auto cdr_extensibility = cdr::extensibility::final;`. +struct ImuSample { + uint64_t stamp_us{}; + std::array accel{}; + std::array gyro{}; + float temperature{}; + std::string frame_id{}; +}; - auto inline_writer = espp::CdrWriter::make_body_writer(espp::CdrEncapsulation::CDR_LE); - inline_writer.write_array(input_magic); - inline_writer.write_string("embedded field"); - auto inline_payload = - espp::CdrWriter::encapsulate(inline_writer.payload(), espp::CdrEncapsulation::PL_CDR_LE); +} // namespace - espp::CdrReader reader(payload); - espp::CdrReader inline_reader(inline_payload); - uint32_t decoded_count = 0; - float decoded_scale = 0.0f; - std::string decoded_text; - std::vector decoded_values; - std::array decoded_magic{}; - std::string decoded_inline_text; +extern "C" void app_main(void) { + espp::Logger logger({.tag = "cdr_example", .level = espp::Logger::Verbosity::INFO}); - bool ok = reader.read(decoded_count) && reader.read(decoded_scale) && - reader.read_string(decoded_text) && reader.read_sequence(decoded_values); - bool inline_ok = inline_reader.encapsulation() == espp::CdrEncapsulation::PL_CDR_LE && - inline_reader.read_array(decoded_magic) && - inline_reader.read_string(decoded_inline_text); //! [cdr example] + const ImuSample sample{ + .stamp_us = 123456789, + .accel = {0.0f, 0.0f, 9.81f}, + .gyro = {0.01f, -0.02f, 0.0f}, + .temperature = 25.5f, + .frame_id = "imu_link", + }; - if (!ok || !inline_ok) { - logger.error("Failed to decode CDR payload"); + // XCDR2 (appendable): what FastDDS / OpenDDS speak by default. + auto bytes = cdr::serialize(sample); + // XCDR1 (plain CDR): what ROS 2 and CycloneDDS speak by default. + auto ros2_bytes = cdr::serialize(sample); + if (!bytes) { + logger.error("XCDR2 serialize failed: {}", cdr::to_string(bytes.error().code)); return; } + if (!ros2_bytes) { + logger.error("XCDR1 serialize failed: {}", cdr::to_string(ros2_bytes.error().code)); + return; + } + logger.info("ImuSample: {} bytes as XCDR2, {} bytes as XCDR1 (ROS 2)", bytes->size(), + ros2_bytes->size()); - logger.info("Decoded count={}, scale={:.2f}, text='{}', sequence size={}, embedded='{}'", - decoded_count, decoded_scale, decoded_text, decoded_values.size(), - decoded_inline_text); - - if (decoded_count != 42 || decoded_scale != 3.25f || decoded_text != "hello cdr" || - decoded_values.size() != input_values.size() || - !std::equal(decoded_values.begin(), decoded_values.end(), input_values.begin()) || - decoded_magic != input_magic || decoded_inline_text != "embedded field") { - logger.error("CDR round-trip mismatch"); + // Deserialize picks version + endianness from the encapsulation header and + // returns std::expected — errors carry a code, payload offset, and the + // field that failed. + auto restored = cdr::deserialize(*bytes); + if (!restored) { + logger.error("deserialize failed at {} offset {} in field '{}'", + cdr::to_string(restored.error().code), restored.error().offset, + restored.error().field); return; } + logger.info("round-trip ok: stamp={} frame='{}' accel.z={}", restored->stamp_us, + restored->frame_id, restored->accel[2]); + + // Zero-allocation path for hot loops: serialize into a fixed buffer. + std::array fixed{}; + if (auto n = cdr::serialize_into(sample, fixed)) { + logger.info("serialize_into wrote {} bytes (exact size query: {})", *n, + cdr::serialized_size(sample)); + } + + // PL_CDR parameter lists — the encoding RTPS discovery (SPDP/SEDP) uses. + std::vector pl_buf; + cdr::param_list_writer pl(pl_buf); + pl.add(uint16_t{0x0050}, std::array{1, 2, 3, 4}); // PID_PARTICIPANT_GUID + pl.add(uint16_t{0x0062}, std::string("esp32_node")); // PID_ENTITY_NAME + if (pl.finish()) { + auto reader = cdr::param_list_reader::from_encapsulated(pl_buf); + while (reader) { + auto param = reader->next(); + if (!param || !*param) + break; + logger.info(" parameter pid=0x{:04x} ({} bytes)", (*param)->pid, (*param)->value.size()); + } + } + //! [cdr example] - logger.info("CDR round-trip succeeded"); + logger.info("example complete"); } diff --git a/components/cdr/idf_component.yml b/components/cdr/idf_component.yml index d467f553f..5affe040f 100644 --- a/components/cdr/idf_component.yml +++ b/components/cdr/idf_component.yml @@ -1,6 +1,6 @@ ## IDF Component Manager Manifest File license: "MIT" -description: "Common Data Representation (CDR) read/write helpers for ESP-IDF and cross-platform use." +description: "Reflection-driven CDR/XCDR serialization for plain C++ structs (no IDL compiler) — wire-compatible with DDS/RTPS, byte-verified against CycloneDDS." url: "https://github.com/esp-cpp/espp/tree/main/components/cdr" repository: "git://github.com/esp-cpp/espp.git" maintainers: @@ -14,8 +14,12 @@ tags: - CDR - DDS - RTPS + - ROS2 - Serialization - XCDR + - Reflection dependencies: idf: - version: '>=5.0' + version: '>=5.2' + espp/reflect_cpp: + version: '>=1.0' diff --git a/components/cdr/include/cdr.hpp b/components/cdr/include/cdr.hpp index 1dff9d53d..8b3839186 100644 --- a/components/cdr/include/cdr.hpp +++ b/components/cdr/include/cdr.hpp @@ -1,547 +1,30 @@ #pragma once -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -namespace espp { - -/// Supported CDR encapsulation identifiers. -enum class CdrEncapsulation : uint16_t { - CDR_BE = 0x0000, ///< Big-endian Common Data Representation. - CDR_LE = 0x0001, ///< Little-endian Common Data Representation. - PL_CDR_BE = 0x0002, ///< Big-endian parameter-list CDR. - PL_CDR_LE = 0x0003, ///< Little-endian parameter-list CDR. -}; - -namespace detail { -template inline T swap_endian(T value) { - auto bytes = std::bit_cast>(value); - std::reverse(bytes.begin(), bytes.end()); - return std::bit_cast(bytes); -} - -template inline T convert_endian(T value, bool target_little_endian) { - if constexpr (sizeof(T) == 1) { - return value; - } else { - if ((std::endian::native == std::endian::little) == target_little_endian) { - return value; - } - return swap_endian(value); - } -} - -constexpr size_t cdr_alignment_for_size(size_t size) { - return size >= 8 ? 8 : (size >= 4 ? 4 : (size >= 2 ? 2 : 1)); -} - -template constexpr size_t cdr_alignment() { return cdr_alignment_for_size(sizeof(T)); } -} // namespace detail - -/// Small helper for building CDR/XCDR1-style byte streams. +/// @file cdr.hpp +/// @brief espp umbrella header for the cdr serialization library. /// -/// \section cdr_ex1 CDR Example -/// \snippet cdr_example.cpp cdr example -class CdrWriter { -public: - /// @brief Configuration for a CDR writer instance. - struct Config { - CdrEncapsulation encapsulation{ - CdrEncapsulation::CDR_LE}; ///< Encapsulation kind to emit when writing. - bool include_encapsulation{ - true}; ///< If true, prepend the 4-byte encapsulation header to the buffer. - }; - - /// @brief Create a configuration for writing a CDR body without an encapsulation header. - /// @param encapsulation Endianness/encapsulation rules to use for the body payload. - /// @return A configuration with encapsulation emission disabled. - [[nodiscard]] static Config - body_config(CdrEncapsulation encapsulation = CdrEncapsulation::CDR_LE) { - return { - .encapsulation = encapsulation, - .include_encapsulation = false, - }; - } - - /// @brief Create a writer configured for a headerless/body-only CDR payload. - /// @param encapsulation Endianness/encapsulation rules to use for the body payload. - /// @return A ready-to-use writer with no encapsulation header in its output. - /// @note CDR alignment is measured from the start of the buffer. Because a body-only writer has - /// no 4-byte encapsulation header, 8-byte-aligned members (e.g. int64/double) land at different - /// offsets than in an encapsulated writer. A body produced here and later wrapped with - /// encapsulate() is therefore not byte-compatible with a directly-encapsulated buffer when it - /// contains 8-byte-aligned types. Current RTPS usage only emits <= 4-byte-aligned types. - [[nodiscard]] static CdrWriter - make_body_writer(CdrEncapsulation encapsulation = CdrEncapsulation::CDR_LE) { - return CdrWriter(body_config(encapsulation)); - } - - /// @brief Wrap an existing payload with a CDR encapsulation header. - /// @param payload Raw bytes to append after the generated encapsulation header. - /// @param encapsulation Encapsulation header to prepend. - /// @return A new byte buffer containing the encapsulation header followed by the payload. - [[nodiscard]] static std::vector - encapsulate(std::span payload, - CdrEncapsulation encapsulation = CdrEncapsulation::CDR_LE) { - CdrWriter writer({ - .encapsulation = encapsulation, - .include_encapsulation = true, - }); - writer.write_bytes(payload); - return writer.take_buffer(); - } - - /// @brief Construct a writer using the default little-endian encapsulated configuration. - CdrWriter() { reset(); } - - /// @brief Construct a writer using an explicit configuration. - /// @param config Writer configuration controlling encapsulation and endianness behavior. - explicit CdrWriter(const Config &config) - : config_(config) { - reset(); - } - - /// @brief Clear the current buffer and reinitialize the encapsulation header if configured. - void reset() { - data_.clear(); - if (config_.include_encapsulation) { - auto value = static_cast(config_.encapsulation); - data_.push_back(static_cast((value >> 8) & 0xff)); - data_.push_back(static_cast(value & 0xff)); - data_.push_back(0); - data_.push_back(0); - } - } - - /// @brief Get the configured encapsulation kind for this writer. - /// @return The encapsulation kind associated with this writer. - [[nodiscard]] CdrEncapsulation encapsulation() const { return config_.encapsulation; } - - /// @brief Determine whether values are encoded in little-endian order. - /// @return True for little-endian encapsulations, false for big-endian ones. - [[nodiscard]] bool uses_little_endian() const { - return config_.encapsulation == CdrEncapsulation::CDR_LE || - config_.encapsulation == CdrEncapsulation::PL_CDR_LE; - } - - /// @brief Get the total number of bytes currently written. - /// @return The size of the backing byte buffer, including any encapsulation header. - [[nodiscard]] size_t size() const { return data_.size(); } - - /// @brief Access the full serialized buffer built so far. - /// @return A const reference to the complete buffer, including any encapsulation header. - [[nodiscard]] const std::vector &buffer() const { return data_; } - - /// @brief Access only the payload portion of the buffer. - /// @return A view over the serialized bytes after any encapsulation header. - [[nodiscard]] std::span payload() const { - auto bytes = std::span{data_.data(), data_.size()}; - return bytes.subspan(std::min(payload_offset(), bytes.size())); - } - - /// @brief Move the complete serialized buffer out of the writer. - /// @return The current buffer contents, including any encapsulation header. - [[nodiscard]] std::vector take_buffer() { return std::move(data_); } - - /// @brief Pad the buffer with zeros until it satisfies the requested alignment. - /// @param alignment Required alignment in bytes. Values less than or equal to 1 are ignored. - /// @return Always returns true. This matches the reader API even though writer-side alignment - /// cannot currently fail. - bool align(size_t alignment) { - if (alignment <= 1) { - return true; - } - while (data_.size() % alignment != 0) { - data_.push_back(0); - } - return true; - } - - /// @brief Append a primitive scalar using CDR alignment and endianness rules. - /// @tparam T Primitive integral or floating-point type to encode. - /// @param value Value to append to the serialized buffer. - /// @return True after the value has been encoded and appended. - template - requires(std::is_integral_v || std::is_floating_point_v) bool write(T value) { - align(detail::cdr_alignment()); - auto encoded = detail::convert_endian(value, uses_little_endian()); - auto bytes = std::bit_cast>(encoded); - auto *raw = reinterpret_cast(bytes.data()); - data_.insert(data_.end(), raw, raw + bytes.size()); - return true; - } - - /// @brief Append a boolean value using the standard CDR 1-byte representation. - /// @param value Boolean value to encode. - /// @return True after the value has been appended. - bool write_bool(bool value) { return write(value ? 1 : 0); } - - /// @brief Append a CDR string. - /// @param text UTF-8 text to encode. A terminating null byte is written automatically. - /// @return True after the string length, contents, terminator, and alignment padding are written. - bool write_string(std::string_view text) { - align(4); - write(static_cast(text.size() + 1)); - data_.insert(data_.end(), text.begin(), text.end()); - data_.push_back(0); - align(4); - return true; - } - - /// @brief Append raw bytes with optional alignment. - /// @param bytes Bytes to copy into the serialized buffer. - /// @param alignment Alignment in bytes to satisfy before appending the data. - /// @return True after the bytes have been appended. - bool write_bytes(std::span bytes, size_t alignment = 1) { - align(alignment); - data_.insert(data_.end(), bytes.begin(), bytes.end()); - return true; - } - - /// @brief Append a fixed-size array of primitive values. - /// @tparam T Primitive integral or floating-point element type. - /// @tparam N Number of elements in the array. - /// @param values Array to encode element-by-element. - /// @return True after all elements have been encoded. - template - requires(std::is_integral_v || - std::is_floating_point_v) bool write_array(const std::array &values) { - return std::all_of(values.begin(), values.end(), - [this](const auto &value) { return write(value); }); - } - - /// @brief Append a variable-length CDR sequence of primitive values. - /// @tparam T Primitive integral or floating-point element type. - /// @param values Sequence elements to encode. - /// @return True after the sequence length and all elements have been encoded. - template - requires(std::is_integral_v || - std::is_floating_point_v) bool write_sequence(std::span values) { - align(4); - write(static_cast(values.size())); - for (const auto &value : values) { - write(value); - } - return true; - } - -private: - [[nodiscard]] size_t payload_offset() const { return config_.include_encapsulation ? 4 : 0; } - - Config config_; - std::vector data_{}; -}; - -/// Small helper for parsing CDR/XCDR1-style byte streams. -class CdrReader { -public: - /// @brief Configuration for a CDR reader instance. - struct Config { - bool expect_encapsulation{ - true}; ///< If true, consume and validate a 4-byte encapsulation header on reset. - CdrEncapsulation default_encapsulation{ - CdrEncapsulation::CDR_LE}; ///< Encapsulation to assume when no header is expected. - }; - - /// @brief Create a configuration for reading a headerless/body-only CDR payload. - /// @param encapsulation Encapsulation/endian rules to assume for the payload body. - /// @return A configuration with encapsulation consumption disabled. - [[nodiscard]] static Config - body_config(CdrEncapsulation encapsulation = CdrEncapsulation::CDR_LE) { - return { - .expect_encapsulation = false, - .default_encapsulation = encapsulation, - }; - } - - /// @brief Create a reader for a headerless/body-only CDR payload. - /// @param data Serialized payload bytes to read. - /// @param encapsulation Encapsulation/endian rules to assume for the payload body. - /// @return A reader initialized for body-only parsing. - [[nodiscard]] static CdrReader - make_body_reader(std::span data, - CdrEncapsulation encapsulation = CdrEncapsulation::CDR_LE) { - return CdrReader(data, body_config(encapsulation)); - } - - /// @brief Construct a reader that expects a standard encapsulated CDR payload. - /// @param data Serialized bytes to parse. - explicit CdrReader(std::span data) { reset(data); } - - /// @brief Construct a reader with an explicit configuration. - /// @param data Serialized bytes to parse. - /// @param config Reader configuration controlling encapsulation handling. - CdrReader(std::span data, const Config &config) - : config_(config) { - reset(data); - } - - /// @brief Reset the reader to the beginning of a new serialized buffer. - /// @param data Serialized bytes to parse. - void reset(std::span data) { - data_ = data; - offset_ = 0; - valid_ = true; - if (config_.expect_encapsulation) { - if (data_.size() < 4) { - valid_ = false; - return; - } - uint16_t raw = static_cast(data_[0] << 8) | static_cast(data_[1]); - switch (raw) { - case static_cast(CdrEncapsulation::CDR_BE): - case static_cast(CdrEncapsulation::CDR_LE): - case static_cast(CdrEncapsulation::PL_CDR_BE): - case static_cast(CdrEncapsulation::PL_CDR_LE): - encapsulation_ = static_cast(raw); - break; - default: - valid_ = false; - return; - } - offset_ = 4; - } else { - encapsulation_ = config_.default_encapsulation; - } - } - - /// @brief Check whether the reader is still in a valid state. - /// @return True if parsing can continue, false if a prior operation failed. - [[nodiscard]] bool valid() const { return valid_; } - - /// @brief Get the active encapsulation for the current buffer. - /// @return The parsed or assumed encapsulation value. - [[nodiscard]] CdrEncapsulation encapsulation() const { return encapsulation_; } - - /// @brief Determine whether values are decoded as little-endian. - /// @return True for little-endian encapsulations, false for big-endian ones. - [[nodiscard]] bool uses_little_endian() const { - return encapsulation_ == CdrEncapsulation::CDR_LE || - encapsulation_ == CdrEncapsulation::PL_CDR_LE; - } - - /// @brief Access only the payload bytes after any encapsulation header. - /// @return A view over the unread buffer excluding the encapsulation header. - [[nodiscard]] std::span payload() const { - return data_.subspan(std::min(payload_offset(), data_.size())); - } - - /// @brief Get the number of unread bytes remaining. - /// @return Remaining unread byte count, or 0 if the offset is beyond the buffer. - [[nodiscard]] size_t remaining() const { - return offset_ <= data_.size() ? data_.size() - offset_ : 0; - } - - /// @brief Access a view of the unread bytes without copying. - /// @return A span over the unread tail of the buffer, or an empty span if the reader is invalid. - [[nodiscard]] std::span remaining_view() const { - if (!valid_) { - return {}; - } - return data_.subspan(std::min(offset_, data_.size())); - } - - /// @brief Advance the read cursor by a fixed number of bytes. - /// @param length Number of bytes to skip. - /// @return True if the bytes were skipped, false if the reader became invalid. - bool skip(size_t length) { - if (!valid_ || remaining() < length) { - valid_ = false; - return false; - } - offset_ += length; - return true; - } - - /// @brief Advance the read cursor to satisfy an alignment requirement. - /// @param alignment Required alignment in bytes. Values less than or equal to 1 are ignored. - /// @return True if the padding bytes were skipped successfully, false if the reader became - /// invalid. - bool align(size_t alignment) { - if (alignment <= 1) { - return true; - } - size_t padding = (alignment - (offset_ % alignment)) % alignment; - return skip(padding); - } - - /// @brief Read a primitive scalar using CDR alignment and endianness rules. - /// @tparam T Primitive integral or floating-point type to decode. - /// @param value Output variable that receives the decoded value on success. - /// @return True if a complete value was decoded, false otherwise. - template - requires(std::is_integral_v || std::is_floating_point_v) bool read(T &value) { - constexpr size_t alignment = detail::cdr_alignment(); - if constexpr (alignment > 1) { - if (!align(alignment)) { // cppcheck-suppress knownConditionTrueFalse - valid_ = false; - return false; - } - } - if (remaining() < sizeof(T)) { - valid_ = false; - return false; - } - std::array bytes{}; - std::memcpy(bytes.data(), data_.data() + offset_, sizeof(T)); - offset_ += sizeof(T); - auto encoded = std::bit_cast(bytes); - value = detail::convert_endian(encoded, uses_little_endian()); - return true; - } - - /// @brief Read a boolean value encoded with the CDR 1-byte representation. - /// @param value Output variable that receives the decoded boolean on success. - /// @return True if the boolean was read successfully, false otherwise. - bool read_bool(bool &value) { - uint8_t raw = 0; - if (!read(raw)) { - return false; - } - value = raw != 0; - return true; - } - - /// @brief Read a CDR string. - /// @param text Output string receiving the decoded text without the trailing null terminator. - /// @return True if the string was decoded successfully, false otherwise. - bool read_string(std::string &text) { - if (!align(4)) { - return false; - } - uint32_t length = 0; - if (!read(length) || length == 0 || remaining() < length) { - valid_ = false; - return false; - } - auto span = data_.subspan(offset_, length); - offset_ += length; - // CDR strings are length-prefixed and null-terminated; a missing terminator is a malformed - // payload, so reject it rather than silently accepting the bytes. - if (span.back() != 0) { - valid_ = false; - return false; - } - span = span.first(span.size() - 1); - text.assign(reinterpret_cast(span.data()), span.size()); - // Re-align for any following element. CDR only inserts padding before an aligned element, so a - // trailing string at the end of the buffer may legitimately have no padding bytes; only align - // when there are bytes left to consume so a valid final string is not rejected. - if (remaining() == 0) { - return true; - } - return align(4); - } - - /// @brief Read a fixed number of bytes as a zero-copy span. - /// @param length Number of bytes to expose. - /// @param alignment Alignment in bytes to satisfy before reading. - /// @return A span over the requested bytes, or an empty span if the read failed. - std::span read_span(size_t length, size_t alignment = 1) { - if (!align(alignment) || remaining() < length) { - valid_ = false; - return {}; - } - auto span = data_.subspan(offset_, length); - offset_ += length; - return span; - } - - /// @brief Read bytes into a caller-provided mutable span. - /// @param bytes Destination span that receives the copied bytes. - /// @param alignment Alignment in bytes to satisfy before reading. - /// @return True if all requested bytes were read successfully, false otherwise. - bool read_bytes(std::span bytes, size_t alignment = 1) { - auto span = read_span(bytes.size(), alignment); - if (span.size() != bytes.size()) { - return false; - } - std::memcpy(bytes.data(), span.data(), bytes.size()); - return true; - } - - /// @brief Read bytes into a vector. - /// @param bytes Output vector replaced with the decoded bytes on success. - /// @param length Number of bytes to read. - /// @param alignment Alignment in bytes to satisfy before reading. - /// @return True if the requested bytes were read successfully, false otherwise. - bool read_bytes(std::vector &bytes, size_t length, size_t alignment = 1) { - auto span = read_span(length, alignment); - if (span.size() != length) { - return false; - } - bytes.assign(span.begin(), span.end()); - return true; - } - - /// @brief Read a fixed-size array of primitive values. - /// @tparam T Primitive integral or floating-point element type. - /// @tparam N Number of elements in the array. - /// @param values Output array receiving the decoded elements. - /// @return True if all array elements were read successfully, false otherwise. - template - requires(std::is_integral_v || - std::is_floating_point_v) bool read_array(std::array &values) { - for (auto &value : values) { - if (!read(value)) { - return false; - } - } - return true; - } - - /// @brief Read a variable-length CDR sequence of primitive values. - /// @tparam T Primitive integral or floating-point element type. - /// @param values Output vector replaced with the decoded sequence elements on success. - /// @return True if the sequence length and all elements were decoded successfully, false - /// otherwise. - template - requires(std::is_integral_v || - std::is_floating_point_v) bool read_sequence(std::vector &values) { - if (!align(4)) { - return false; - } - uint32_t length = 0; - if (!read(length)) { - return false; - } - // Bound the declared length against the bytes actually available before reserving so a - // malformed/malicious payload cannot request an enormous allocation (or OOM) on a - // memory-constrained target. Each element occupies at least sizeof(T) bytes, so a length larger - // than remaining() / sizeof(T) cannot possibly be satisfied; reject it up front. - if (length > remaining() / sizeof(T)) { - valid_ = false; - return false; - } - values.clear(); - values.reserve(length); - for (uint32_t i = 0; i < length; i++) { - T value{}; - if (!read(value)) { - return false; - } - values.push_back(value); - } - return true; - } - -private: - [[nodiscard]] size_t payload_offset() const { return config_.expect_encapsulation ? 4 : 0; } - - Config config_{}; - std::span data_{}; - size_t offset_{0}; - bool valid_{false}; - CdrEncapsulation encapsulation_{CdrEncapsulation::CDR_LE}; -}; +/// Reflection-driven CDR/XCDR serialization for plain C++ structs — the +/// compiler generates the serialization code from the struct definition +/// itself; there is no IDL compiler and no hand-written read/write sequence. +/// The full library lives at https://github.com/finger563/cdr (vendored under +/// detail/); see its docs/DESIGN.md for the architecture and wire-format +/// rules, and README.md for the type mapping and usage conventions. +/// +/// Everything is in the `cdr` namespace (not `espp`), matching the standalone +/// library: +/// +/// \code{.cpp} +/// struct ImuSample { +/// uint64_t stamp_us; +/// std::array accel; +/// }; +/// +/// auto bytes = cdr::serialize(sample); // XCDR2, appendable +/// auto ros2 = cdr::serialize(sample); // ROS 2 / classic CDR +/// auto back = cdr::deserialize(*bytes); // std::expected +/// \endcode +/// +/// Requires C++23 (std::expected) — the default C++ standard on ESP-IDF's +/// GCC 13+ toolchains (IDF 5.2 and newer). -} // namespace espp +#include diff --git a/components/cdr/src/cdr.cpp b/components/cdr/src/cdr.cpp deleted file mode 100644 index 584ad00cc..000000000 --- a/components/cdr/src/cdr.cpp +++ /dev/null @@ -1 +0,0 @@ -#include "cdr.hpp" From 5ea5de611445c46689ed1e21f3e152f2021a536e Mon Sep 17 00:00:00 2001 From: William Emfinger Date: Tue, 11 Aug 2026 10:51:22 -0500 Subject: [PATCH 2/3] refactor(rtps,lib,pc): migrate cdr consumers to the reflection-driven API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - rtps: discovery string/octet-sequence parameters serialize via cdr::serialize_body / deserialize_body; PL_CDR encapsulation handled inline (the old CdrWriter/CdrReader helpers are gone) - pc tests + rtps/p4 examples: uint32 payload helpers use cdr::serialize / cdr::deserialize - lib: espp builds C++23; cdr/reflect-cpp detail headers added to the include set; cdr python bindings removed — the python side of a CDR message is a plain pycdr2 dataclass (or struct.pack for primitives), so espp.CdrWriter/CdrReader no longer exist - python: rtps scripts use pure-python CDR helpers for uint32 payloads Verified: rtps example builds on ESP32-S3/IDF 6; host lib + _espp python module build; pc rtps_pubsub end-to-end test passes (16/16 samples through SPDP/SEDP discovery + user data on the new serializer). Co-Authored-By: Claude Fable 5 --- .../esp32_p4_function_ev_board_example.cpp | 18 +- components/rtps/example/main/rtps_example.cpp | 31 +-- components/rtps/src/rtps.cpp | 90 +++---- lib/CMakeLists.txt | 2 +- lib/autogenerate_bindings.py | 10 +- lib/espp.cmake | 6 +- lib/python_bindings/cdr_bindings.cpp | 246 ------------------ lib/python_bindings/module.cpp | 10 +- pc/CMakeLists.txt | 2 +- pc/tests/rtps_common.hpp | 18 +- python/rtps_publisher.py | 7 +- python/rtps_pubsub.py | 13 +- python/rtps_subscriber.py | 7 +- 13 files changed, 109 insertions(+), 351 deletions(-) delete mode 100644 lib/python_bindings/cdr_bindings.cpp diff --git a/components/esp32-p4-function-ev-board/example/main/esp32_p4_function_ev_board_example.cpp b/components/esp32-p4-function-ev-board/example/main/esp32_p4_function_ev_board_example.cpp index 1b50f2c36..4389339c9 100644 --- a/components/esp32-p4-function-ev-board/example/main/esp32_p4_function_ev_board_example.cpp +++ b/components/esp32-p4-function-ev-board/example/main/esp32_p4_function_ev_board_example.cpp @@ -90,19 +90,21 @@ static void ping_target(espp::Logger &logger, const char *name, const std::strin namespace { /// Serialize a uint32 as an encapsulated little-endian CDR payload (matches std_msgs/msg/UInt32). inline std::vector serialize_uint32(uint32_t value) { - espp::CdrWriter writer; // defaults: CDR_LE with a 4-byte encapsulation header - writer.write(value); - return writer.take_buffer(); + auto bytes = cdr::serialize(value); // CDR_LE with a 4-byte encapsulation header + if (!bytes) { + return {}; + } + const auto *data = reinterpret_cast(bytes->data()); + return std::vector(data, data + bytes->size()); } /// Parse a uint32 from an encapsulated CDR payload, or std::nullopt if invalid. -inline std::optional deserialize_uint32(std::span cdr) { - espp::CdrReader reader(cdr); - uint32_t value = 0; - if (!reader.valid() || !reader.read(value)) { +inline std::optional deserialize_uint32(std::span cdr_payload) { + auto value = cdr::deserialize(std::as_bytes(cdr_payload)); + if (!value) { return std::nullopt; } - return value; + return *value; } } // namespace diff --git a/components/rtps/example/main/rtps_example.cpp b/components/rtps/example/main/rtps_example.cpp index a18a56fd4..57859b6f2 100644 --- a/components/rtps/example/main/rtps_example.cpp +++ b/components/rtps/example/main/rtps_example.cpp @@ -21,28 +21,23 @@ constexpr std::string_view kTypeName = "std_msgs/msg/UInt32"; // The rtps `publish()` / `on_sample` API works with any CDR-encoded type, so serialization lives in // the application rather than the participant. std::vector serialize_uint32(uint32_t value) { - // The encapsulation options below are the CdrWriter defaults (little-endian CDR with a 4-byte - // encapsulation header); shown explicitly for clarity about the on-the-wire format. - espp::CdrWriter writer({ - .encapsulation = espp::CdrEncapsulation::CDR_LE, - .include_encapsulation = true, - }); - writer.write(value); - return writer.take_buffer(); + // XCDR1 = classic little-endian CDR with a 4-byte encapsulation header — the on-the-wire + // format ROS 2 uses for std_msgs/msg/UInt32. + auto bytes = cdr::serialize(value); + if (!bytes) { + return {}; + } + const auto *data = reinterpret_cast(bytes->data()); + return std::vector(data, data + bytes->size()); } -std::optional deserialize_uint32(std::span cdr) { - // The config below matches the CdrReader defaults (expects a little-endian CDR encapsulation - // header); shown explicitly to mirror serialize_uint32() above. - espp::CdrReader reader(cdr, { - .expect_encapsulation = true, - .default_encapsulation = espp::CdrEncapsulation::CDR_LE, - }); - uint32_t value = 0; - if (!reader.valid() || !reader.read(value)) { +std::optional deserialize_uint32(std::span cdr_payload) { + // deserialize() reads the encapsulation header itself (version + endianness). + auto value = cdr::deserialize(std::as_bytes(cdr_payload)); + if (!value) { return std::nullopt; } - return value; + return *value; } bool run_local_protocol_checks(espp::Logger &logger, const espp::RtpsParticipant &participant) { diff --git a/components/rtps/src/rtps.cpp b/components/rtps/src/rtps.cpp index 57696e539..9092d9976 100644 --- a/components/rtps/src/rtps.cpp +++ b/components/rtps/src/rtps.cpp @@ -431,28 +431,33 @@ void append_parameter_locator(ByteWriter &writer, ParameterId id, writer.append_bytes(locator.address); } +// Append a CDR-encoded parameter value, padding it to the 4-byte multiple the RTPS PL_CDR +// encoding requires (parameterLength declares the padded length so the next parameter starts +// 4-byte aligned). +void append_cdr_parameter(ByteWriter &writer, ParameterId id, std::span body) { + const size_t padded = (body.size() + 3) & ~size_t{3}; + if (padded > 0xffff) { + // parameterLength is a 16-bit field; dropping an oversized parameter keeps the list + // well-formed on the wire (discovery parameters are tiny in practice). + return; + } + append_parameter_header(writer, id, static_cast(padded)); + writer.append_bytes( + std::span(reinterpret_cast(body.data()), body.size())); + for (size_t i = body.size(); i < padded; ++i) { + writer.append_u8(0); + } +} + void append_parameter_string_cdr(ByteWriter &writer, ParameterId id, std::string_view text) { - auto cdr_writer = espp::CdrWriter::make_body_writer(espp::CdrEncapsulation::CDR_LE); - cdr_writer.write_string(text); - auto cdr_payload = cdr_writer.payload(); - // The RTPS PL_CDR encoding requires parameterLength to be a multiple of 4 so the next - // parameter starts 4-byte aligned. write_string() already trailing-aligns the body to 4, so the - // payload length is the padded length we must declare. - append_parameter_header(writer, id, static_cast(cdr_payload.size())); - writer.append_bytes(cdr_payload); + auto body = cdr::serialize_body(std::string(text)); + append_cdr_parameter(writer, id, *body); } void append_parameter_octet_sequence(ByteWriter &writer, ParameterId id, std::span bytes) { - auto cdr_writer = espp::CdrWriter::make_body_writer(espp::CdrEncapsulation::CDR_LE); - cdr_writer.write(static_cast(bytes.size())); - cdr_writer.write_bytes(bytes); - cdr_writer.align(4); - auto cdr_payload = cdr_writer.payload(); - // parameterLength must be a multiple of 4 (see append_parameter_string_cdr); the align(4) above - // padded the payload to that length, so declare the padded length. - append_parameter_header(writer, id, static_cast(cdr_payload.size())); - writer.append_bytes(cdr_payload); + auto body = cdr::serialize_body(std::vector(bytes.begin(), bytes.end())); + append_cdr_parameter(writer, id, *body); } void append_parameter_reliability(ByteWriter &writer, @@ -495,17 +500,16 @@ void append_parameter_sentinel(ByteWriter &writer) { std::vector parse_parameter_list(std::span payload) { std::vector parameters; - espp::CdrReader cdr_reader(payload); - // Limitation: only little-endian parameter lists (PL_CDR_LE) are decoded. The parameter value - // parsers below (parse_u32_le, parse_locator, parse_guid, ...) assume little-endian contents, so - // a big-endian (PL_CDR_BE) list is intentionally rejected rather than misparsed. In practice DDS - // and ROS 2 implementations emit PL_CDR_LE for SPDP/SEDP discovery, so this is a discovery-only - // gap. - if (!cdr_reader.valid() || cdr_reader.encapsulation() != espp::CdrEncapsulation::PL_CDR_LE) { + // Limitation: only little-endian parameter lists (PL_CDR_LE, encapsulation id 0x0003) are + // decoded. The parameter value parsers below (parse_u32_le, parse_locator, parse_guid, ...) + // assume little-endian contents, so a big-endian (PL_CDR_BE) list is intentionally rejected + // rather than misparsed. In practice DDS and ROS 2 implementations emit PL_CDR_LE for SPDP/SEDP + // discovery, so this is a discovery-only gap. + if (payload.size() < 4 || payload[0] != 0x00 || payload[1] != 0x03) { return parameters; } - ByteReader reader(cdr_reader.payload()); + ByteReader reader(payload.subspan(4)); while (reader.remaining() >= 4) { uint16_t pid = 0; uint16_t length = 0; @@ -574,35 +578,19 @@ std::optional parse_bool(std::span value) { } std::optional parse_cdr_string(std::span value) { - auto reader = espp::CdrReader::make_body_reader(value, espp::CdrEncapsulation::CDR_LE); - if (!reader.valid()) { - return std::nullopt; - } - uint32_t length = 0; - if (!reader.read(length) || length == 0) { + auto result = cdr::deserialize_body(std::as_bytes(value)); + if (!result) { return std::nullopt; } - auto text_bytes = reader.read_span(length); - if (text_bytes.size() != length || text_bytes.back() != 0) { - return std::nullopt; - } - return std::string(reinterpret_cast(text_bytes.data()), text_bytes.size() - 1); + return *std::move(result); } std::optional> parse_octet_sequence(std::span value) { - auto reader = espp::CdrReader::make_body_reader(value, espp::CdrEncapsulation::CDR_LE); - if (!reader.valid()) { - return std::nullopt; - } - uint32_t length = 0; - if (!reader.read(length)) { - return std::nullopt; - } - std::vector bytes; - if (!reader.read_bytes(bytes, length)) { + auto result = cdr::deserialize_body>(std::as_bytes(value)); + if (!result) { return std::nullopt; } - return bytes; + return *std::move(result); } std::optional parse_locator(std::span value) { @@ -750,7 +738,13 @@ DataSubmessageView parse_data_submessage(const espp::RtpsParticipant::Submessage std::vector build_parameter_list_payload(ByteWriter ¶meter_writer) { auto parameter_bytes = parameter_writer.take(); - return espp::CdrWriter::encapsulate(parameter_bytes, espp::CdrEncapsulation::PL_CDR_LE); + // PL_CDR_LE encapsulation header (representation id 0x0003, big-endian on the wire) followed by + // the parameter list. + std::vector payload; + payload.reserve(4 + parameter_bytes.size()); + payload.insert(payload.end(), {0x00, 0x03, 0x00, 0x00}); + payload.insert(payload.end(), parameter_bytes.begin(), parameter_bytes.end()); + return payload; } std::vector build_data_submessage_payload(const espp::RtpsParticipant::EntityId &reader_id, diff --git a/lib/CMakeLists.txt b/lib/CMakeLists.txt index 637515ac4..7bfd5ba42 100644 --- a/lib/CMakeLists.txt +++ b/lib/CMakeLists.txt @@ -67,7 +67,7 @@ else() if(WIN32) target_link_libraries(${TARGET_NAME} winmm) endif() - target_compile_features(${TARGET_NAME} PRIVATE cxx_std_20) + target_compile_features(${TARGET_NAME} PRIVATE cxx_std_23) # install build output and headers install(TARGETS ${TARGET_NAME} diff --git a/lib/autogenerate_bindings.py b/lib/autogenerate_bindings.py index e739ae8e8..09d2e8b07 100644 --- a/lib/autogenerate_bindings.py +++ b/lib/autogenerate_bindings.py @@ -3,10 +3,10 @@ import os import re -# NOTE: cdr.hpp is intentionally NOT generated here. srcmlcpp cannot parse it, and even when coerced -# the generated CdrReader API is unusable (output-reference reads can't return values, and the -# non-owning std::span would dangle). It is bound by hand instead in -# python_bindings/cdr_bindings.cpp (registered via py_init_cdr in module.cpp). +# NOTE: cdr.hpp is intentionally NOT generated here. The cdr component's API is template-based +# (compiler-generated serialization per struct type), which litgen cannot bind generically — and it +# needs no python bindings: the python side of a CDR message is a plain pycdr2 dataclass with the +# same wire format (see components/cdr/README.md). # //////////////////////////////////////////////////////////////////////////////////////////////// @@ -438,7 +438,7 @@ def autogenerate() -> None: include_dir = repository_dir + "/components/" header_files = [include_dir + "base_component/include/base_component.hpp", - # cdr.hpp is bound by hand in python_bindings/cdr_bindings.cpp (see note above). + # cdr.hpp is excluded: template-based API, python uses pycdr2 instead (see note above). include_dir + "cobs/include/cobs.hpp", include_dir + "cobs/include/cobs_stream.hpp", include_dir + "color/include/color.hpp", diff --git a/lib/espp.cmake b/lib/espp.cmake index bce856057..868a7c409 100644 --- a/lib/espp.cmake +++ b/lib/espp.cmake @@ -20,6 +20,8 @@ set(ESPP_INCLUDES ${ESPP_COMPONENTS}/base_component/include ${ESPP_COMPONENTS}/base_peripheral/include ${ESPP_COMPONENTS}/cdr/include + ${ESPP_COMPONENTS}/cdr/detail/cdr/include + ${ESPP_COMPONENTS}/reflect_cpp/detail/reflect-cpp/include ${ESPP_COMPONENTS}/cobs/include ${ESPP_COMPONENTS}/color/include ${ESPP_COMPONENTS}/csv/include @@ -48,7 +50,6 @@ set(ESPP_INCLUDES ) set(ESPP_SOURCES - ${ESPP_COMPONENTS}/cdr/src/cdr.cpp ${ESPP_COMPONENTS}/cobs/src/cobs.cpp ${ESPP_COMPONENTS}/cobs/src/cobs_stream.cpp ${ESPP_COMPONENTS}/color/src/color.cpp @@ -108,7 +109,6 @@ set(ESPP_PYTHON_BINDINGS_DIR ${CMAKE_CURRENT_LIST_DIR}/python_bindings) set(ESPP_PYTHON_SOURCES ${ESPP_PYTHON_BINDINGS_DIR}/module.cpp ${ESPP_PYTHON_BINDINGS_DIR}/pybind_espp.cpp - ${ESPP_PYTHON_BINDINGS_DIR}/cdr_bindings.cpp ${ESPP_PYTHON_BINDINGS_DIR}/rtps_bindings.cpp ${ESPP_PYTHON_BINDINGS_DIR}/socket_reactor_bindings.cpp ${ESPP_SOURCES} @@ -126,7 +126,7 @@ endfunction() # extension module target (the native part of the `espp` python package) function(espp_add_python_module) pybind11_add_module(_espp ${ESPP_PYTHON_SOURCES}) - target_compile_features(_espp PRIVATE cxx_std_20) + target_compile_features(_espp PRIVATE cxx_std_23) # disable certain compiler warnings for this module, but only if we're not on # Windows if(NOT MSVC) diff --git a/lib/python_bindings/cdr_bindings.cpp b/lib/python_bindings/cdr_bindings.cpp deleted file mode 100644 index ebdee5aac..000000000 --- a/lib/python_bindings/cdr_bindings.cpp +++ /dev/null @@ -1,246 +0,0 @@ -// Hand-written pybind11 bindings for the `cdr` component. -// -// Why hand-written instead of litgen-generated: -// - litgen/srcmlcpp cannot parse cdr.hpp (its class parser chokes on the method bodies / -// requires-clauses), and even when coerced it produces an unusable API: -// * CdrReader::read(T& value) is an *output* reference, which pybind cannot return to -// Python (the decoded value is lost). -// * CdrReader holds a NON-OWNING std::span, so a generated `CdrReader(bytes)` would dangle -// once the Python buffer is freed. -// - This shim exposes a clean, safe, Pythonic API: write_*(value), read_*() -> Optional[...], -// bytes in/out, and an owning reader wrapper that copies its input buffer. -// -// It is intentionally separate from the generated pybind_espp.cpp so regeneration never clobbers -// it, and it is easy to extend with new CDR helpers. - -#include -#include -#include -#include -#include - -#include -#include - -#include "cdr.hpp" - -namespace py = pybind11; - -namespace { - -std::vector bytes_to_vec(const py::bytes &data) { - // py::bytes -> std::string -> bytes copy (owning). - std::string s = data; - return std::vector(s.begin(), s.end()); -} - -py::bytes span_to_bytes(std::span s) { - return py::bytes(reinterpret_cast(s.data()), s.size()); -} - -py::bytes vec_to_bytes(const std::vector &v) { - return py::bytes(reinterpret_cast(v.data()), v.size()); -} - -// Owning wrapper around CdrReader: CdrReader stores a non-owning std::span, so we keep the decoded -// buffer alive for the lifetime of the reader. Declaration order matters: `storage_` must be -// initialized before `reader_` (which references it). -class PyCdrReader { -public: - explicit PyCdrReader(const py::bytes &data, - espp::CdrReader::Config config = espp::CdrReader::Config{}) - : storage_(bytes_to_vec(data)) - , reader_(std::span{storage_.data(), storage_.size()}, config) {} - - static PyCdrReader - make_body_reader(const py::bytes &data, - espp::CdrEncapsulation encapsulation = espp::CdrEncapsulation::CDR_LE) { - return PyCdrReader(data, espp::CdrReader::body_config(encapsulation)); - } - - bool valid() const { return reader_.valid(); } - espp::CdrEncapsulation encapsulation() const { return reader_.encapsulation(); } - bool uses_little_endian() const { return reader_.uses_little_endian(); } - size_t remaining() const { return reader_.remaining(); } - py::bytes payload() const { return span_to_bytes(reader_.payload()); } - py::bytes remaining_view() const { return span_to_bytes(reader_.remaining_view()); } - bool skip(size_t length) { return reader_.skip(length); } - bool align(size_t alignment) { return reader_.align(alignment); } - - template std::optional read() { - T value{}; - if (!reader_.read(value)) { - return std::nullopt; - } - return value; - } - - std::optional read_bool() { - bool value = false; - if (!reader_.read_bool(value)) { - return std::nullopt; - } - return value; - } - - std::optional read_string() { - std::string value; - if (!reader_.read_string(value)) { - return std::nullopt; - } - return value; - } - - std::optional read_bytes(size_t length, size_t alignment = 1) { - std::vector bytes; - if (!reader_.read_bytes(bytes, length, alignment)) { - return std::nullopt; - } - return vec_to_bytes(bytes); - } - - template std::optional> read_sequence() { - std::vector values; - if (!reader_.read_sequence(values)) { - return std::nullopt; - } - return values; - } - -private: - std::vector storage_; - espp::CdrReader reader_; -}; - -template void add_writer_scalar(py::class_ &c, const char *name) { - c.def( - name, [](espp::CdrWriter &w, T value) { return w.write(value); }, py::arg("value")); -} - -template void add_writer_sequence(py::class_ &c, const char *name) { - c.def( - name, - [](espp::CdrWriter &w, const std::vector &values) { - return w.write_sequence(std::span{values.data(), values.size()}); - }, - py::arg("values")); -} - -template void add_reader_scalar(py::class_ &c, const char *name) { - c.def(name, &PyCdrReader::read); -} - -template void add_reader_sequence(py::class_ &c, const char *name) { - c.def(name, &PyCdrReader::read_sequence); -} - -} // namespace - -void py_init_cdr(py::module &m) { - py::enum_(m, "CdrEncapsulation", - "Supported CDR encapsulation identifiers.") - .value("CDR_BE", espp::CdrEncapsulation::CDR_BE) - .value("CDR_LE", espp::CdrEncapsulation::CDR_LE) - .value("PL_CDR_BE", espp::CdrEncapsulation::PL_CDR_BE) - .value("PL_CDR_LE", espp::CdrEncapsulation::PL_CDR_LE); - - // ---- CdrWriter (owns its buffer, so it is safe to bind directly) ---- - auto writer = py::class_(m, "CdrWriter", py::dynamic_attr(), - "Helper for building CDR/XCDR1-style byte streams."); - - py::class_(writer, "Config") - .def(py::init<>()) - .def_readwrite("encapsulation", &espp::CdrWriter::Config::encapsulation) - .def_readwrite("include_encapsulation", &espp::CdrWriter::Config::include_encapsulation); - - writer.def(py::init<>()) - .def(py::init(), py::arg("config")) - .def_static("make_body_writer", &espp::CdrWriter::make_body_writer, - py::arg("encapsulation") = espp::CdrEncapsulation::CDR_LE) - .def_static( - "encapsulate", - [](const py::bytes &payload, espp::CdrEncapsulation encapsulation) { - auto vec = bytes_to_vec(payload); - return vec_to_bytes(espp::CdrWriter::encapsulate( - std::span{vec.data(), vec.size()}, encapsulation)); - }, - py::arg("payload"), py::arg("encapsulation") = espp::CdrEncapsulation::CDR_LE) - .def("reset", &espp::CdrWriter::reset) - .def("encapsulation", &espp::CdrWriter::encapsulation) - .def("uses_little_endian", &espp::CdrWriter::uses_little_endian) - .def("size", &espp::CdrWriter::size) - .def("align", &espp::CdrWriter::align, py::arg("alignment")) - .def("write_bool", &espp::CdrWriter::write_bool, py::arg("value")) - .def("write_string", &espp::CdrWriter::write_string, py::arg("text")) - .def( - "write_bytes", - [](espp::CdrWriter &w, const py::bytes &data, size_t alignment) { - auto vec = bytes_to_vec(data); - return w.write_bytes(std::span{vec.data(), vec.size()}, alignment); - }, - py::arg("data"), py::arg("alignment") = 1) - .def("buffer", [](const espp::CdrWriter &w) { return vec_to_bytes(w.buffer()); }) - .def("payload", [](const espp::CdrWriter &w) { return span_to_bytes(w.payload()); }) - .def("take_buffer", [](espp::CdrWriter &w) { return vec_to_bytes(w.take_buffer()); }); - - add_writer_scalar(writer, "write_uint8"); - add_writer_scalar(writer, "write_uint16"); - add_writer_scalar(writer, "write_uint32"); - add_writer_scalar(writer, "write_uint64"); - add_writer_scalar(writer, "write_int8"); - add_writer_scalar(writer, "write_int16"); - add_writer_scalar(writer, "write_int32"); - add_writer_scalar(writer, "write_int64"); - add_writer_scalar(writer, "write_float"); - add_writer_scalar(writer, "write_double"); - add_writer_sequence(writer, "write_sequence_uint8"); - add_writer_sequence(writer, "write_sequence_uint16"); - add_writer_sequence(writer, "write_sequence_uint32"); - add_writer_sequence(writer, "write_sequence_int32"); - add_writer_sequence(writer, "write_sequence_float"); - add_writer_sequence(writer, "write_sequence_double"); - - // ---- CdrReader (owning wrapper; read_* return Optional, None on failure) ---- - auto reader = py::class_(m, "CdrReader", py::dynamic_attr(), - "Helper for parsing CDR/XCDR1-style byte streams. Copies " - "the input buffer so it is safe to use independently."); - - py::class_(reader, "Config") - .def(py::init<>()) - .def_readwrite("expect_encapsulation", &espp::CdrReader::Config::expect_encapsulation) - .def_readwrite("default_encapsulation", &espp::CdrReader::Config::default_encapsulation); - - reader - .def(py::init(), py::arg("data"), - py::arg("config") = espp::CdrReader::Config{}) - .def_static("make_body_reader", &PyCdrReader::make_body_reader, py::arg("data"), - py::arg("encapsulation") = espp::CdrEncapsulation::CDR_LE) - .def("valid", &PyCdrReader::valid) - .def("encapsulation", &PyCdrReader::encapsulation) - .def("uses_little_endian", &PyCdrReader::uses_little_endian) - .def("remaining", &PyCdrReader::remaining) - .def("payload", &PyCdrReader::payload) - .def("remaining_view", &PyCdrReader::remaining_view) - .def("skip", &PyCdrReader::skip, py::arg("length")) - .def("align", &PyCdrReader::align, py::arg("alignment")) - .def("read_bool", &PyCdrReader::read_bool) - .def("read_string", &PyCdrReader::read_string) - .def("read_bytes", &PyCdrReader::read_bytes, py::arg("length"), py::arg("alignment") = 1); - - add_reader_scalar(reader, "read_uint8"); - add_reader_scalar(reader, "read_uint16"); - add_reader_scalar(reader, "read_uint32"); - add_reader_scalar(reader, "read_uint64"); - add_reader_scalar(reader, "read_int8"); - add_reader_scalar(reader, "read_int16"); - add_reader_scalar(reader, "read_int32"); - add_reader_scalar(reader, "read_int64"); - add_reader_scalar(reader, "read_float"); - add_reader_scalar(reader, "read_double"); - add_reader_sequence(reader, "read_sequence_uint8"); - add_reader_sequence(reader, "read_sequence_uint16"); - add_reader_sequence(reader, "read_sequence_uint32"); - add_reader_sequence(reader, "read_sequence_int32"); - add_reader_sequence(reader, "read_sequence_float"); - add_reader_sequence(reader, "read_sequence_double"); -} diff --git a/lib/python_bindings/module.cpp b/lib/python_bindings/module.cpp index 8dbfc84b7..365cbf152 100644 --- a/lib/python_bindings/module.cpp +++ b/lib/python_bindings/module.cpp @@ -6,10 +6,11 @@ namespace py = pybind11; void py_init_module_espp(py::module &m); -// Hand-written bindings for the `cdr` and `rtps` components (see *_bindings.cpp for why they are -// not generated). Both must run after py_init_module_espp so shared types (e.g. Logger::Verbosity) -// and the module's classes are already registered. -void py_init_cdr(py::module &m); +// Hand-written bindings for the `rtps` component (see rtps_bindings.cpp for why they are not +// generated). Must run after py_init_module_espp so shared types (e.g. Logger::Verbosity) and the +// module's classes are already registered. The cdr component has no python bindings: its C++ API +// is template-based, and the python side of a message is a plain pycdr2 dataclass instead (see +// the cdr component README). void py_init_rtps(py::module &m); // Hand-written bindings for espp::SocketReactor (litgen cannot parse it; see // socket_reactor_bindings.cpp). Runs after py_init_module_espp so UdpSocket / Socket::Info / @@ -26,7 +27,6 @@ PYBIND11_MODULE(_espp, m) { #endif py_init_module_espp(m); - py_init_cdr(m); py_init_rtps(m); py_init_socket_reactor(m); } diff --git a/pc/CMakeLists.txt b/pc/CMakeLists.txt index a53dea5d2..69e7eee40 100644 --- a/pc/CMakeLists.txt +++ b/pc/CMakeLists.txt @@ -1,6 +1,6 @@ cmake_minimum_required (VERSION 3.11) -set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD 23) include(${CMAKE_CURRENT_SOURCE_DIR}/../lib/espp.cmake) diff --git a/pc/tests/rtps_common.hpp b/pc/tests/rtps_common.hpp index 57583cdb6..508de5332 100644 --- a/pc/tests/rtps_common.hpp +++ b/pc/tests/rtps_common.hpp @@ -56,19 +56,21 @@ inline std::string guess_local_ipv4() { /// Serialize a uint32 as an encapsulated little-endian CDR payload (matches std_msgs/msg/UInt32). inline std::vector serialize_uint32(uint32_t value) { - espp::CdrWriter writer; // defaults: CDR_LE with a 4-byte encapsulation header - writer.write(value); - return writer.take_buffer(); + auto bytes = cdr::serialize(value); // CDR_LE with a 4-byte encapsulation header + if (!bytes) { + return {}; + } + const auto *data = reinterpret_cast(bytes->data()); + return std::vector(data, data + bytes->size()); } /// Parse a uint32 from an encapsulated CDR payload, or std::nullopt if invalid. -inline std::optional deserialize_uint32(std::span cdr) { - espp::CdrReader reader(cdr); - uint32_t value = 0; - if (!reader.valid() || !reader.read(value)) { +inline std::optional deserialize_uint32(std::span cdr_payload) { + auto value = cdr::deserialize(std::as_bytes(cdr_payload)); + if (!value) { return std::nullopt; } - return value; + return *value; } } // namespace rtps_test diff --git a/python/rtps_publisher.py b/python/rtps_publisher.py index f412899e5..1434cc72a 100644 --- a/python/rtps_publisher.py +++ b/python/rtps_publisher.py @@ -9,6 +9,7 @@ import datetime import socket +import struct import sys import time @@ -27,9 +28,9 @@ def guess_local_ipv4() -> str: def serialize_uint32(value: int) -> bytes: - writer = espp.CdrWriter() - writer.write_uint32(value) - return bytes(writer.take_buffer()) + # Little-endian CDR (XCDR1) with a 4-byte encapsulation header — the wire format of + # std_msgs/msg/UInt32. Plain struct.pack; no native bindings needed for CDR payloads. + return b"\x00\x01\x00\x00" + struct.pack(" int: diff --git a/python/rtps_pubsub.py b/python/rtps_pubsub.py index f12ab4d80..2ec8c197e 100644 --- a/python/rtps_pubsub.py +++ b/python/rtps_pubsub.py @@ -10,6 +10,7 @@ import datetime import socket +import struct import sys import time @@ -29,13 +30,17 @@ def guess_local_ipv4() -> str: def serialize_uint32(value: int) -> bytes: - writer = espp.CdrWriter() # default little-endian CDR with encapsulation header - writer.write_uint32(value) - return bytes(writer.take_buffer()) + # Little-endian CDR (XCDR1) with a 4-byte encapsulation header — the wire format of + # std_msgs/msg/UInt32. Plain struct.pack; no native bindings needed for CDR payloads. + return b"\x00\x01\x00\x00" + struct.pack("" + return struct.unpack_from(endian + "I", data, 4)[0] def main() -> int: diff --git a/python/rtps_subscriber.py b/python/rtps_subscriber.py index bbfc1b655..99dd384c3 100644 --- a/python/rtps_subscriber.py +++ b/python/rtps_subscriber.py @@ -9,6 +9,7 @@ import datetime import socket +import struct import sys import time @@ -27,7 +28,11 @@ def guess_local_ipv4() -> str: def deserialize_uint32(data: bytes): - return espp.CdrReader(data).read_uint32() + # Accepts either endianness; the encapsulation identifier's low bit selects little-endian. + if len(data) < 8 or data[0] != 0x00 or data[1] not in (0x00, 0x01): + return None # int, or None on failure + endian = "<" if data[1] & 1 else ">" + return struct.unpack_from(endian + "I", data, 4)[0] def main() -> int: From e1fc5c1c9bddf7f16c62ccfc602672f1336c492e Mon Sep 17 00:00:00 2001 From: William Emfinger Date: Tue, 11 Aug 2026 12:11:29 -0500 Subject: [PATCH 3/3] docs(cdr): rewrite component docs for the reflection-driven API Updates the sphinx page and example README for the new struct-based serialize/deserialize API, and excludes the vendored detail/ submodule from cppcheck (matching lib/). Co-Authored-By: Claude Fable 5 --- components/cdr/example/README.md | 24 ++++++----- doc/en/data/cdr.rst | 74 +++++++++++++++++++++++--------- suppressions.txt | 1 + 3 files changed, 69 insertions(+), 30 deletions(-) diff --git a/components/cdr/example/README.md b/components/cdr/example/README.md index 85b9fddb0..4167abc8c 100644 --- a/components/cdr/example/README.md +++ b/components/cdr/example/README.md @@ -1,16 +1,19 @@ # CDR Example -This example demonstrates a small CDR round-trip using the `cdr` component. +This example demonstrates the reflection-driven `cdr` component: the compiler +generates CDR serialization code directly from a plain struct definition. It exercises: -- a little-endian CDR encapsulation header -- primitive value serialization -- CDR string serialization -- `uint16_t` sequence serialization -- fixed-array serialization with `write_array` / `read_array` -- headerless/body CDR helpers for embedding fields inside a larger protocol value -- round-trip parsing with `CdrReader` +- struct serialization to XCDR2 (appendable, the FastDDS/OpenDDS default) and + XCDR1 (plain CDR, the ROS 2 / CycloneDDS default) +- deserialization driven by the received encapsulation header (version and + endianness) +- `std::expected` error handling with error code, payload offset, and field + name +- the zero-allocation `cdr::serialize_into` path and `cdr::serialized_size` +- PL_CDR parameter-list writing and reading (the encoding RTPS SPDP/SEDP + discovery uses) ## How to use example @@ -26,5 +29,6 @@ Replace `PORT` with the name of the serial port to use. ## Expected Output -The example logs the encoded byte count and the decoded values. It finishes by -printing `CDR round-trip succeeded`. +The example logs the serialized sizes for both XCDR versions, the round-tripped +field values, the zero-allocation write size, and the parameters found in the +PL_CDR parameter list, finishing with `example complete`. diff --git a/doc/en/data/cdr.rst b/doc/en/data/cdr.rst index 6e2868dd2..3f659fe09 100644 --- a/doc/en/data/cdr.rst +++ b/doc/en/data/cdr.rst @@ -1,26 +1,60 @@ CDR (Common Data Representation) ******************************** -The ``cdr`` component provides a small, standalone Common Data Representation -reader/writer utility aimed at standards-oriented protocols such as DDS/RTPS. - -This initial slice focuses on the pieces needed to start building interoperable -payloads without forcing applications to adopt DDS or RTPS as a whole: - -- CDR / PL_CDR encapsulation identifiers -- endian-aware primitive serialization helpers -- CDR alignment and padding handling -- string helpers using the standard CDR length-prefix + null terminator layout -- body helpers for CDR fields embedded inside larger protocol elements -- fixed-array helpers and zero-copy payload/span views -- primitive sequence helpers -- standalone usage outside RTPS - -Current scope: - -- useful as a reusable building block for future DDS/RTPS payload work -- suitable for direct use in other protocols that want CDR-style payloads -- not yet a full XTypes / XCDR2 implementation +The ``cdr`` component provides reflection-driven CDR/XCDR serialization for +plain C++ structs — the compiler generates the serialization code from the +struct definition itself. There is no IDL compiler and no hand-written +read/write call sequence; defining an aggregate struct is all it takes: + +.. code-block:: cpp + + struct ImuSample { + uint64_t stamp_us; + std::array accel; + std::array gyro; + float temperature; + }; + + auto bytes = cdr::serialize(sample); // XCDR2, appendable (default) + auto ros2 = cdr::serialize(sample); // ROS 2 / classic-CDR peers + auto back = cdr::deserialize(*bytes); // std::expected + +The implementation is the `finger563/cdr `_ +library, vendored as a git submodule under ``detail/``, with the +:doc:`reflect_cpp ` component as its reflection backend. See the +library's `README `_ for the +supported type mapping and its +`design document `_ +for the architecture and wire-format rules. + +Feature highlights: + +- XCDR1 (plain CDR — what ROS 2 and CycloneDDS speak by default) and XCDR2 + (plain + delimited/appendable with DHEADER — what Fast DDS and OpenDDS + speak by default), in both endiannesses +- appendable schema evolution in both directions: newer peers' extra members + are skipped, missing members keep their defaults +- wire format byte-verified against ``pycdr2`` (CycloneDDS's codec); the + Python side of a message is a plain ``pycdr2`` dataclass — no bindings +- ``std::expected`` error handling carrying an error code, payload offset, + and the field name that failed; bounds-checked, fuzz-tested deserializers +- ``cdr::bounded_string`` / ``cdr::bounded_vector`` for IDL bounded + types +- ``cdr::param_list_writer`` / ``cdr::param_list_reader`` for the PL_CDR + parameter lists used by RTPS discovery (SPDP/SEDP) +- zero-allocation ``cdr::serialize_into`` and body-only variants for + composing RTPS submessages + +Everything lives in the ``cdr`` namespace (not ``espp``), matching the +standalone library. Requires C++23 (``std::expected``), the default C++ +standard on ESP-IDF 5.2+ toolchains. + +.. note:: + + This component replaces the earlier manual ``espp::CdrWriter`` / + ``espp::CdrReader`` API. The ``rtps`` component and examples have been + migrated; payloads previously built with hand-written write/read call + sequences are now plain structs. .. toctree:: diff --git a/suppressions.txt b/suppressions.txt index aa6acf791..a445328fd 100644 --- a/suppressions.txt +++ b/suppressions.txt @@ -10,3 +10,4 @@ cstyleCast // [error id]:[filename]:[line] *:lib/* *:components/reflect_cpp/detail/* +*:components/cdr/detail/*