feat: upgrade libwebrtc to m150. - #1284
Conversation
Changeset ✓This PR includes a changeset covering all affected packages:
|
The webrtc::I420ToARGB etc. symbols from api/video/yuv_helper.h were not present in the webrtc.lib for aarch64-pc-windows-msvc, causing 22 linker errors. Replace the webrtc:: namespace wrappers with direct libyuv:: calls using third_party/libyuv/include/libyuv.h, which is already in the include path and whose symbols are always compiled into webrtc.lib. Also removes the erroneous duplicate i420_to_nv12 overload that incorrectly called NV12ToI420 instead of I420ToNV12.
|
|
…nker errors The pre-built libwebrtc.a at webrtc-a970b87 was compiled with use_custom_libcxx=true (Chromium's libc++ with ABI namespace std::__Cr::), making it ABI-incompatible with GCC/libstdc++ (std::__cxx11:: / std::). Every linker error in CI was caused by this mismatch, e.g.: SdpVideoFormat(std::__Cr::basic_string const&) [in library] SdpVideoFormat(std::__cxx11::basic_string const&) [called by our code] Revert to use_custom_libcxx=false so the library links with the system libstdc++, matching the GCC compilation used by our SDK build. The library must be rebuilt at a new webrtc-* tag for this fix to take effect in CI. After the rebuild, update WEBRTC_TAG in build/src/lib.rs.
024a137 to
bf52ad6
Compare
The m150 libwebrtc artifact ships libc++ 23 headers and is built with Chromium's clang 23. webrtc-sys compiles against those headers via -nostdinc++, so an older clang fails inside <limits> and <span> — __builtin_popcountg, __is_nothrow_convertible and __GCC_DESTRUCTIVE_SIZE are all missing, which cascades into ~20 lines about dynamic_extent not being a constant expression, in headers nobody wrote. build.rs now probes __clang_major__ before configuring the build and reports the real floor, which it reads from the artifact's own __configuration/compiler.h rather than hardcoding. Nothing in CI or the README provided a clang that new: Ubuntu 24 has 18 and AlmaLinux 8 has 17. Add .github/scripts/install-clang.sh, which fetches the official LLVM release tarball for the host arch, and wire it into the four Linux jobs that build webrtc-sys. tests.yml previously pointed CC/CXX at the host GCC through sccache, which build.rs now rejects outright — GCC ignores libc++'s trivial_abi annotations and miscompiles unique_ptr returns. uniffi-cdylib.yml is left alone; livekit-uniffi does not link libwebrtc. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
m150 replaced the old "release builds don't log" default with a LoggingConfig that defaults to log_to_stderr=true and debug_severity=LS_INFO, and installs itself on the first log call if nobody called InitializeLogging first. So libwebrtc started writing straight to stderr in parallel with our LogSink, and that copy is invisible to `log`/RUST_LOG — a bare PeerConnectionFactory::default() emits 29 unfilterable "(audio_device_impl.cc:146): ..." lines. Turn the debug stream off when the sink is installed so the sink is the only path out. While here, map LoggingSeverity onto log::Level instead of flattening everything to DEBUG. The sink is registered at LS_VERBOSE, so the old behavior left RUST_LOG choosing between all of libwebrtc's output and none of it. LS_INFO still maps to DEBUG, keeping libwebrtc's INFO chatter out of a RUST_LOG=info run as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| __attribute__((weak)) size_t __hash_memory(void const* p, size_t n) noexcept { | ||
| auto const* ptr = static_cast<unsigned char const*>(p); | ||
| #if defined(__LP64__) | ||
| // 64-bit FNV-1a | ||
| size_t hash = 14695981039346656037ULL; | ||
| for (size_t i = 0; i < n; ++i) { | ||
| hash ^= static_cast<size_t>(ptr[i]); | ||
| hash *= 1099511628211ULL; | ||
| } | ||
| #else | ||
| // 32-bit FNV-1a | ||
| size_t hash = 2166136261U; | ||
| for (size_t i = 0; i < n; ++i) { | ||
| hash ^= static_cast<size_t>(ptr[i]); | ||
| hash *= 16777619U; | ||
| } | ||
| #endif | ||
| return hash; | ||
| } |
There was a problem hiding this comment.
🟡 Android compatibility stub computes hashes with a different algorithm than the standard library
The Android fallback string-hash routine implements FNV-1a (__hash_memory at webrtc-sys/src/ndk_compat.cpp:37-55) whereas the C++ standard library it substitutes for uses a completely different algorithm, so the two halves of the build can compute different hashes for the same bytes.
Impact: On Android, hash-table lookups on data shared between the prebuilt WebRTC library and the wrapper code can silently miss, causing lost or duplicated entries.
Which callers get which implementation
LLVM libc++ has never used FNV-1a for __hash_memory; it implements __murmur2_or_cityhash (murmur2 on 32-bit, cityhash-derived on 64-bit). The comment at webrtc-sys/src/ndk_compat.cpp:29 ("the same FNV-1a algorithm that LLVM libc++ historically used") is therefore inaccurate.
With NDK r28+, __hash_memory is _LIBCPP_HIDE_FROM_ABI, so any translation unit compiled with the new headers inlines the real murmur2/cityhash implementation, while objects inside the prebuilt libwebrtc.a (compiled against older headers) call the out-of-line symbol and thus resolve to this weak FNV-1a stub. Any std::unordered_map/unordered_set with string or byte-array keys that is populated on one side and probed on the other will hash the same key to different buckets.
A faithful port of libc++'s __murmur2_or_cityhash would avoid the divergence.
Prompt for agents
webrtc-sys/src/ndk_compat.cpp provides a weak fallback for std::__ndk1::__hash_memory implemented as FNV-1a. libc++ actually implements this as __murmur2_or_cityhash (murmur2 on 32-bit, a cityhash variant on 64-bit). Because NDK r28+ inlines the real implementation into newly compiled translation units while the prebuilt libwebrtc.a calls the out-of-line symbol, the two halves would hash identical keys differently, which breaks any hash container shared across the boundary. Port libc++'s __murmur2_or_cityhash implementation (or otherwise guarantee bit-identical results with the headers being used) instead of FNV-1a, and correct the explanatory comment.
Was this helpful? React with 👍 or 👎 to provide feedback.
aa13ed7 to
db162a7
Compare
| { | ||
| "name": 'src', | ||
| "url": 'https://github.com/webrtc-sdk/webrtc.git@m144_release', | ||
| "url": 'https://github.com/webrtc-sdk/webrtc.git@duan/m150-patching', |
There was a problem hiding this comment.
🟨 Build container installs toolchains from an unpinned upstream branch and remote install scripts
The Linux build now sources WebRTC from a mutable personal branch (webrtc-sys/libwebrtc/.gclient:4, duan/m150-patching) instead of an immutable release tag, and the container steps continue to pipe remote scripts into shells (rustup, nodesource) with no verification. Combined, the native library that ships to users is built from sources that can change without any change to this repository.
Was this helpful? React with 👍 or 👎 to provide feedback.
| yum install clang clang-devel lld -y; \ | ||
| yum install protobuf-compiler -y; \ | ||
| yum groupinstall 'Development Tools' -y; \ | ||
| clang --version; \ | ||
| yum install openssl-devel libX11-devel mesa-libGL-devel libXext-devel libva-devel libdrm-devel libgbm-devel libXdamage-devel libXrandr-devel libXfixes-devel libXcomposite-devel -y; \ | ||
| clang --version; \ | ||
| clang++ --version; \ | ||
| export LIBCLANG_PATH=/usr/lib64; \ | ||
| export CC=clang; \ | ||
| export CXX=clang++; \ |
There was a problem hiding this comment.
🔴 Linux release binaries can no longer be built because the build container installs an outdated compiler
The Linux packaging container still installs and uses the distribution's own compiler (yum install clang / export CXX=clang++ at .github/workflows/ffi-builds.yml:218-226), which is far older than the minimum the build now insists on, so the Linux release artifacts stop being produced.
Impact: The published Linux x86_64 and arm64 FFI downloads fail to build, blocking releases for those platforms.
Compiler floor enforced by webrtc-sys/build.rs vs. the manylinux images
webrtc-sys/build.rs now calls configure_hermetic_libcxx() for every linux target, which in turn runs check_clang_version() (webrtc-sys/build.rs:574-605). That function reads the floor out of the artifact's __configuration/compiler.h (defaulting to 21) and panic!s when the active compiler reports a lower __clang_major__.
The FFI Linux job builds inside sameli/manylinux_2_28_x86_64_cuda_12.3 / quay.io/pypa/manylinux_2_28_aarch64 (AlmaLinux 8), whose clang package is version 17/18 — below the floor. builds.yml and tests.yml were updated to run .github/scripts/install-clang.sh (which even documents a docker: usage mode with LLVM_ROOT), but the FFI workflow was not, so the transformation is incomplete.
Prompt for agents
The Linux FFI build in .github/workflows/ffi-builds.yml runs inside manylinux_2_28 images and relies on the distro clang (17/18) via `yum install clang` plus `export CC=clang CXX=clang++`. webrtc-sys/build.rs now enforces a minimum clang major version (read from the libwebrtc artifact's hermetic libc++ `__configuration/compiler.h`, fallback 21) and panics otherwise, so this job will fail. The new .github/scripts/install-clang.sh already documents a docker usage mode (`export LLVM_ROOT=/opt/llvm; .github/scripts/install-clang.sh; export CC=$LLVM_ROOT/bin/clang CXX=$LLVM_ROOT/bin/clang++`). Update the docker command in the Linux build step to install and use that newer clang instead of the distro one (keeping LIBCLANG_PATH working for bindgen), for both the x86_64 and aarch64 matrix entries.
Was this helpful? React with 👍 or 👎 to provide feedback.
| source | ||
| .lines() | ||
| .find_map(|line| { | ||
| let (_, rest) = line.split_once("_LIBCPP_CLANG_VER < ")?; | ||
| let ver: u32 = rest.trim().parse().ok()?; | ||
| Some(ver / 100) | ||
| }) | ||
| .unwrap_or(FALLBACK) |
There was a problem hiding this comment.
🟡 Compiler version check accepts a compiler one point release too old
The required compiler version is rounded down to whole numbers when it is read from the bundled library (Some(ver / 100) at webrtc-sys/build.rs:623), so a compiler that is slightly older than required passes the check.
Impact: Users with a marginally old compiler get hundreds of confusing errors from system headers instead of the clear "your compiler is too old" message.
Minor version is discarded from _LIBCPP_CLANG_VER
_LIBCPP_CLANG_VER is __clang_major__ * 100 + __clang_minor__ (as the doc comment at webrtc-sys/build.rs:607-609 states). The artifact's __configuration/compiler.h currently says #if _LIBCPP_CLANG_VER < 2101, i.e. clang 21.1 is the true floor, but ver / 100 yields 21 and check_clang_version only compares major >= min (webrtc-sys/build.rs:588-589). clang 21.0.x therefore passes the guard and then fails deep inside <limits>/<span> — exactly the failure mode this guard was written to prevent.
Prompt for agents
libcxx_min_clang_major() in webrtc-sys/build.rs parses `_LIBCPP_CLANG_VER < 2101` and returns 2101/100 = 21, dropping the minor version, while check_clang_version() only compares __clang_major__. A clang 21.0 passes even though libc++ requires 21.1. Consider keeping the encoded major*100+minor value end-to-end: read __clang_major__ and __clang_minor__ from the compiler's preprocessor defines, compute the same major*100+minor encoding, and compare against the value parsed from __configuration/compiler.h, formatting the floor as major.minor in the panic message.
Was this helpful? React with 👍 or 👎 to provide feedback.
| diff --git a/abseil-cpp/absl/container/internal/raw_hash_set.h b/abseil-cpp/absl/container/internal/raw_hash_set.h | ||
| index 191b1c9bb8..ebf884e908 100644 | ||
| --- a/abseil-cpp/absl/container/internal/raw_hash_set.h | ||
| +++ b/abseil-cpp/absl/container/internal/raw_hash_set.h | ||
| @@ -226,7 +226,7 @@ | ||
| #endif | ||
|
|
||
| #ifdef __BMI2__ | ||
| -#include <bmi2intrin.h> | ||
| +#include <immintrin.h> | ||
| #endif // __BMI2__ | ||
|
|
||
| namespace absl { |
There was a problem hiding this comment.
🟡 Two newly added upstream fixes are never applied by any build script, so the problems they fix remain
Two new patch files are added to the build-patch directory but no build script ever applies them (webrtc-sys/libwebrtc/patches/abseil-cpp_include_fix.patch:1 and patches/fix_gcc_toolchain_ar_input.patch), so the compile failures they are meant to fix still occur.
Impact: Anyone rebuilding the native library from source hits the very build errors these files were added to fix.
Evidence
webrtc-sys/libwebrtc/build_linux.sh:74-95 applies add_licenses, fix_license_json_parsing, ssl_verify_callback_with_native_handle, add_deps, fix_desktop_capture_compile, external_audio_source, fix_pipewire_utils_compile, disable_crel (via git -C build apply), david_disable_gun_source_macro (from third_party) and disable_sme_for_libyuv. Neither abseil-cpp_include_fix.patch (paths relative to third_party/) nor fix_gcc_toolchain_ar_input.patch (paths relative to build/) appears in any of build_linux.sh, build_android.sh, build_macos.sh/build_ios.sh or build_windows.cmd.
Prompt for agents
Two new patch files were added under webrtc-sys/libwebrtc/patches/ (abseil-cpp_include_fix.patch, targeting third_party/abseil-cpp/absl/container/internal/raw_hash_set.h, and fix_gcc_toolchain_ar_input.patch, targeting build/toolchain/gcc_toolchain.gni) but no build script applies them. build_linux.sh applies its patches from src/, from src/third_party (david_disable_gun_source_macro.patch) and from src/build (disable_crel.patch via git -C build apply). Either wire these two patches into the appropriate build script(s) at the matching working directory, or remove them if they are obsolete.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # images (AlmaLinux 8: clang 17, Ubuntu 24: clang 18) fails deep inside <limits> | ||
| # and <span> instead of saying the compiler is too old. webrtc-sys/build.rs | ||
| # checks the version up front and reports the real floor, which it reads from | ||
| # the artifact's own __configuration/compiler.h. |
There was a problem hiding this comment.
do we need to provide information on the clang requirement in our webrtc repo's readme ?
There was a problem hiding this comment.
Clang 21 is only required for compilation on Ubuntu 24 or earlier Linux distributions, so I updated README.md.
https://github.com/livekit/rust-sdks/pull/1284/changes#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R185
ladvoc
left a comment
There was a problem hiding this comment.
Tested with frame metadata and zero playout delay mode, no regressions!
No description provided.