Repository navigation
Fix command-line collection and make native device-ID collection optional - #1546
Conversation
Collect only argv[0] without recursive regex and cap SDK-collected metadata, HTTP response buffers, session files, and diagnostic decoder allocations. Preserve caller event-size policies and add boundary and small-stack regressions. Files changed: - docs/CsProtocol-decoding.md - docs/PAL.md - lib/android_build/maesdk/src/main/java/com/microsoft/applications/events/HttpClient.java - lib/android_build/maesdk/src/test/java/com/microsoft/applications/events/HttpClientRequestTest.java - lib/bond/CompactBinaryProtocolReader.hpp - lib/decoder/PayloadDecoder.cpp - lib/http/HttpClient_Android.cpp - lib/http/HttpClient_Apple.mm - lib/http/HttpClient_Curl.hpp - lib/http/HttpClient_WinHttp.cpp - lib/http/HttpClient_WinInet.cpp - lib/http/HttpClient_WinRt.cpp - lib/include/public/IHttpClient.hpp - lib/offline/LogSessionDataProvider.cpp - lib/pal/PAL.cpp - lib/pal/desktop/WindowsDesktopDeviceInformationImpl.cpp - lib/pal/desktop/WindowsDesktopSystemInformationImpl.cpp - lib/pal/posix/DeviceInformationImpl_Android.cpp - lib/pal/posix/SystemInformationImpl_Android.cpp - lib/pal/posix/sysinfo_sources.cpp - lib/pal/posix/sysinfo_sources.hpp - lib/pal/posix/sysinfo_utils_apple.cpp - lib/pal/posix/sysinfo_utils_ios.mm - lib/pal/posix/sysinfo_utils_mac.mm - lib/pal/universal/WindowsRuntimeDeviceInformationImpl.cpp - lib/pal/universal/WindowsRuntimeSystemInformationImpl.cpp - lib/shared/PlatformHelpers.h - lib/utils/FileUtils.cpp - lib/utils/FileUtils.hpp - lib/utils/Utils.cpp - lib/utils/Utils.hpp - lib/utils/ZlibUtils.cpp - lib/utils/ZlibUtils.hpp - tests/unittests/CMakeLists.txt - tests/unittests/HttpClientCurlTests.cpp - tests/unittests/HttpClientTests.cpp - tests/unittests/LogSessionDataTests.cpp - tests/unittests/PayloadDecoderTests.cpp - tests/unittests/ZlibUtilsTests.cpp - tests/unittests/bounded-input-tests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Compile out native device-ID collection so downstream applications can provide their own IDs without redundant platform reads. Preserve default behavior, expose the setting through the vcpkg overlay, and cover disabled collectors and Android error paths. Files changed: - .github/workflows/build-posix-latest.yml - CMakeLists.txt - cmake/MatsdkOptions.cmake - docs/PAL.md - docs/building-with-vcpkg.md - docs/embedding-with-cmake.md - lib/CMakeLists.txt - lib/android_build/maesdk/src/main/java/com/microsoft/applications/events/HttpClient.java - lib/android_build/maesdk/src/test/java/com/microsoft/applications/events/EventsUnitTest.java - lib/android_build/maesdk/src/test/java/com/microsoft/applications/events/HttpClientRequestTest.java - lib/pal/PAL.cpp - lib/pal/desktop/WindowsDesktopDeviceInformationImpl.cpp - lib/pal/posix/DeviceInformationImpl.cpp - lib/pal/posix/DeviceInformationImpl_Android.cpp - lib/pal/posix/sysinfo_sources.cpp - lib/pal/posix/sysinfo_utils_ios.mm - lib/pal/posix/sysinfo_utils_mac.mm - lib/pal/universal/WindowsRuntimeDeviceInformationImpl.cpp - tests/unittests/CMakeLists.txt - tests/unittests/ContextFieldsProviderTests.cpp - tests/unittests/device-id-tests.cpp - tests/vcpkg/device-id-feature-tests.cmake - tools/ports/cpp-client-telemetry/portfile.cmake - tools/ports/cpp-client-telemetry/vcpkg.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
Install Linux system development libraries before configuring the opt-out build, and enable MSVC big-object support for shared test targets so template-heavy mocks compile with debug information. Files changed: - .github/workflows/build-posix-latest.yml - tests/CMakeLists.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Decoder processing remains quadratic for many records, and Windows text-mode reads can bypass the physical session-file limit.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Bounds telemetry-derived inputs and introduces optional native device-ID collection across supported platforms.
Changes:
- Adds metadata, HTTP, session-file, and decoder limits.
- Adds the default-on
MATSDK_ENABLE_DEVICE_IDoption and vcpkg feature. - Expands cross-platform regression tests, CI, and documentation.
| File | Description |
|---|---|
tools/ports/cpp-client-telemetry/vcpkg.json |
Adds default device-ID feature. |
tools/ports/cpp-client-telemetry/portfile.cmake |
Maps feature to CMake. |
tests/vcpkg/device-id-feature-tests.cmake |
Tests vcpkg mapping. |
tests/unittests/ZlibUtilsTests.cpp |
Tests inflation limits. |
tests/unittests/PayloadDecoderTests.cpp |
Tests decoder rejection. |
tests/unittests/LogSessionDataTests.cpp |
Tests session limit. |
tests/unittests/HttpClientTests.cpp |
Tests oversized headers. |
tests/unittests/HttpClientCurlTests.cpp |
Tests Curl limits/parsing. |
tests/unittests/device-id-tests.cpp |
Tests device-ID opt-out. |
tests/unittests/ContextFieldsProviderTests.cpp |
Adapts disabled-ID expectations. |
tests/unittests/CMakeLists.txt |
Registers new tests. |
tests/unittests/bounded-input-tests.cpp |
Tests bounded inputs. |
lib/utils/ZlibUtils.hpp |
Defines inflation limit. |
lib/utils/ZlibUtils.cpp |
Bounds decompression output. |
lib/utils/Utils.hpp |
Declares metadata limits. |
lib/utils/Utils.cpp |
Implements bounded conversion. |
lib/utils/FileUtils.hpp |
Defines file limit. |
lib/utils/FileUtils.cpp |
Bounds file reads. |
lib/shared/PlatformHelpers.h |
Adds bounded WinRT conversion. |
lib/pal/universal/WindowsRuntimeSystemInformationImpl.cpp |
Bounds WinRT metadata. |
lib/pal/universal/WindowsRuntimeDeviceInformationImpl.cpp |
Makes WinRT ID optional. |
lib/pal/posix/SystemInformationImpl_Android.cpp |
Bounds Android metadata. |
lib/pal/posix/sysinfo_utils_mac.mm |
Bounds macOS data and ID. |
lib/pal/posix/sysinfo_utils_ios.mm |
Bounds iOS data and ID. |
lib/pal/posix/sysinfo_utils_apple.cpp |
Hardens sysctl buffer. |
lib/pal/posix/sysinfo_sources.hpp |
Replaces regex selectors. |
lib/pal/posix/sysinfo_sources.cpp |
Adds bounded linear selection. |
lib/pal/posix/DeviceInformationImpl.cpp |
Disables POSIX ID assignment. |
lib/pal/posix/DeviceInformationImpl_Android.cpp |
Gates Android ID collection. |
lib/pal/PAL.cpp |
Bounds registered metadata. |
lib/pal/desktop/WindowsDesktopSystemInformationImpl.cpp |
Bounds Windows metadata buffers. |
lib/pal/desktop/WindowsDesktopDeviceInformationImpl.cpp |
Gates adapter-based IDs. |
lib/offline/LogSessionDataProvider.cpp |
Rejects oversized sessions. |
lib/include/public/IHttpClient.hpp |
Defines header limit. |
lib/http/HttpClient_WinRt.cpp |
Bounds WinRT headers. |
lib/http/HttpClient_WinInet.cpp |
Bounds WinInet headers. |
lib/http/HttpClient_WinHttp.cpp |
Bounds WinHTTP headers. |
lib/http/HttpClient_Curl.hpp |
Bounds and linearly parses Curl responses. |
lib/http/HttpClient_Apple.mm |
Bounds Apple headers. |
lib/http/HttpClient_Android.cpp |
Bounds JNI responses. |
lib/decoder/PayloadDecoder.cpp |
Bounds diagnostic decoding. |
lib/CMakeLists.txt |
Makes IOKit conditional. |
lib/bond/CompactBinaryProtocolReader.hpp |
Bounds container counts. |
lib/android_build/maesdk/src/test/java/com/microsoft/applications/events/HttpClientRequestTest.java |
Tests Java response limits. |
lib/android_build/maesdk/src/test/java/com/microsoft/applications/events/EventsUnitTest.java |
Tests Java failures/logging. |
lib/android_build/maesdk/src/main/java/com/microsoft/applications/events/HttpClient.java |
Bounds Java responses and gates IDs. |
docs/PAL.md |
Documents limits and opt-out. |
docs/embedding-with-cmake.md |
Documents CMake option. |
docs/CsProtocol-decoding.md |
Documents decoder limits. |
docs/building-with-vcpkg.md |
Documents vcpkg feature. |
CMakeLists.txt |
Defines disabled-ID macro. |
cmake/MatsdkOptions.cmake |
Adds device-ID option. |
.github/workflows/build-posix-latest.yml |
Adds opt-out CI matrix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Exercise disabled collection with the existing POSIX and Windows build/test scripts instead of maintaining separate prerequisite installation and compiler settings. Pass the Windows disable definition through the existing custom-properties hook and include the regression source in its native project. Files changed: - .github/workflows/build-posix-latest.yml - tests/CMakeLists.txt - tests/device-id-optout.props - tests/unittests/UnitTests.vcxproj Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
Copilot comment 4186871826: eliminate repeated copies of the remaining diagnostic payload by scanning request bytes in place and copying only each record. Verified in lib/decoder/PayloadDecoder.cpp:81-109. Copilot comment 4186871913: count physical session-file bytes using binary reads, preserve CRLF parsing, and reject trailing content. Verify exact/oversize CRLF and Ctrl+Z cases plus session regeneration, including registration in the native Windows test project. Verified in lib/utils/FileUtils.cpp:102 and lib/offline/LogSessionDataProvider.cpp:166-177. Files changed: - docs/CsProtocol-decoding.md - docs/PAL.md - lib/decoder/PayloadDecoder.cpp - lib/offline/LogSessionDataProvider.cpp - lib/utils/FileUtils.cpp - tests/unittests/LogSessionDataTests.cpp - tests/unittests/PayloadDecoderTests.cpp - tests/unittests/UnitTests.vcxproj - tests/unittests/bounded-input-tests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
Normalize the CRLF terminator in the legacy functional-test reader now that FileGetContents preserves physical bytes. Retain exact UID comparisons instead of relaxing the persistence assertion. Files changed: - tests/functests/LogSessionDataFuncTests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
Address Copilot comment 4187949583: UTF-16 character counts undercount non-ASCII headers. Match JNI modified UTF-8 sizes without allocating encoded strings, retaining the exact ASCII limit and aggregate framing. Verified JNI encoding at lib/http/HttpClient_Android.cpp:573-589. Files changed: - lib/android_build/maesdk/src/main/java/com/microsoft/applications/events/HttpClient.java - lib/android_build/maesdk/src/test/java/com/microsoft/applications/events/HttpClientRequestTest.java - docs/PAL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Broad cross-platform security-sensitive changes, including uncompiled Apple and WinRT paths, require final human and platform review.
Review effort: Balanced
Findings: 2
Open (2)
Resolved since last review (1)
Address Copilot comments 4187752458 and 4187752558. Forward the Gradle property through the shared Android configuration so SDK and app native builds cannot silently retain collection. Exercise OFF through the same CI Gradle path and compile the native device-ID regressions in the app. Mark the WinRT constructor parameter unused only in the disabled branch so /W4 /WX builds retain the advertised opt-out without suppressions. Verified both Android CMake caches and compile commands with OFF, built SDK/app arm64 targets with ON and OFF, and compiled the WinRT collector with VS 2026 in enabled Release and disabled Release/Debug configurations. Files changed: - .github/workflows/build-android.yml - docs/PAL.md - lib/android_build/app/src/main/cpp/CMakeLists.txt - lib/android_build/tools.gradle - lib/pal/universal/WindowsRuntimeDeviceInformationImpl.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The broad security-sensitive changes span multiple native transports and platforms, including Apple and WinRT paths requiring dedicated platform validation.
Review effort: Balanced
Findings: None
Resolved since last review (2)
Remove blanket metadata, OS-buffer and session-file limits to preserve existing values and persisted identities rather than truncate or reject unlikely outliers. Keep NUL-aware bounded command-line reads, HTTP buffering limits and decompression safeguards. Reject impossible binary counts based on the actual input, not an arbitrary element/record cap. Replace cap-enforcement tests with exact preservation regressions for large metadata, session identifiers and valid containers. Retain the device-ID opt-out and existing platform text-read behavior. Files changed: - docs/CsProtocol-decoding.md - docs/PAL.md - lib/bond/CompactBinaryProtocolReader.hpp - lib/decoder/PayloadDecoder.cpp - lib/offline/LogSessionDataProvider.cpp - lib/pal/PAL.cpp - lib/pal/desktop/WindowsDesktopDeviceInformationImpl.cpp - lib/pal/desktop/WindowsDesktopSystemInformationImpl.cpp - lib/pal/posix/DeviceInformationImpl_Android.cpp - lib/pal/posix/SystemInformationImpl_Android.cpp - lib/pal/posix/sysinfo_sources.cpp - lib/pal/posix/sysinfo_sources.hpp - lib/pal/posix/sysinfo_utils_ios.mm - lib/pal/posix/sysinfo_utils_mac.mm - lib/pal/universal/WindowsRuntimeDeviceInformationImpl.cpp - lib/pal/universal/WindowsRuntimeSystemInformationImpl.cpp - lib/shared/PlatformHelpers.h - lib/utils/FileUtils.cpp - lib/utils/FileUtils.hpp - lib/utils/Utils.cpp - lib/utils/Utils.hpp - tests/unittests/LogSessionDataTests.cpp - tests/unittests/bounded-input-tests.cpp - tests/unittests/device-id-tests.cpp Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 95c41974-5533-4eec-89fb-6885edb2c3a5
…story Preserve both header-bound regressions and upstream Curl connection tests. Files changed: - .github/scripts/prepare-vcpkg-release.py - .github/workflows/memory-leak-analysis.yml - .github/workflows/test-vcpkg.yml - .github/workflows/vcpkg-release-bump.yml - README.md - docs/building-custom-SKU.md - docs/building-with-vcpkg.md - docs/maintainer-onboarding.md - lib/callbacks/DebugSource.cpp - lib/callbacks/DebugSourceInternal.hpp - lib/http/HttpClient_Curl.cpp - lib/http/HttpClient_Curl.hpp - lib/include/public/IHttpClient.hpp - lib/pal/desktop/NetworkDetector.cpp - lib/pal/desktop/NetworkDetector.hpp - tests/CMakeLists.txt - tests/common/network-detector-test-access.hpp - tests/dll-unload/CMakeLists.txt - tests/dll-unload/README.md - tests/dll-unload/debug-listener-unload-module.cpp - tests/dll-unload/debug-listener-unload-test.cpp - tests/dll-unload/network-detector-reload-test.cpp - tests/unittests/DebugEventSourceTests.cpp - tests/unittests/HttpClientCurlTests.cpp - tests/unittests/NetworkDetectorTests.cpp - tests/vcpkg/test-release-port.py - tests/vcpkg/test-vcpkg-android.sh - tests/vcpkg/test-vcpkg-linux.sh - tests/vcpkg/vcpkg.json Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bb70eac5-2067-42ac-8ca9-e87c32427301


Summary
Address the telemetry initialization crash reported in microsoft/onnxruntime#32771. Reading all of
/proc/self/cmdlineand applying libstdc++'s recursive regex engine to long arguments can exhaust a small host thread stack.Collect only the executable identifier (
argv[0]), stopping at the first NUL and bounding the collected bytes. Replace POSIX metadata regex selectors with linear selectors and exact OS-release key matching. Also replace Curl header regex parsing with first-colon parsing, preserving colons in values.Keep targeted safeguards for command-line reads, HTTP buffering, and decompression rather than blanket metadata caps. Normal metadata and OS-reported buffers retain their existing sizes. Session files retain their platform text-read behavior and are not regenerated because of a new size limit. Explicit application event properties and semantic context retain the existing configured serialized-event, upload, and offline-cache policies.
Add a default-on
MATSDK_ENABLE_DEVICE_IDbuild option. Setting it toOFFcompiles out native device-ID collection on Linux/MinGW, Windows desktop/WinRT, macOS/iOS, and Android. Matching Android Java sources also skipANDROID_IDreads. Caller-supplied semantic-context IDs remain supported, and an empty disabled collector does not overwrite them. Other metadata and SDK/session IDs are unchanged.The repository vcpkg overlay adds a default-enabled
device-idfeature mapped explicitly to the CMake option. Disable defaults and select the remaining transport/storage features to opt out. Feature requests are additive, and opting out against an older source revision fails explicitly. This is not yet available in the official vcpkg registry; it needs a released SDK and a port update.Limits and behavior
/proc/self/cmdlineexecutable identifierWinHTTP/WinInet use queried raw-header buffer bytes; WinRT conservatively budgets UTF-8 expansion before conversion. Android counts exact JNI modified UTF-8 bytes without allocating encoded strings, then JNI rechecks the byte budget. Non-ASCII, NUL, and surrogate boundaries retain the same aggregate limit. OS networking frameworks can have their own internal buffering; these limits govern SDK copies and streaming reads.
Regression coverage
Exercise initialization with a real 100 KiB command line and metadata collection on a 64 KiB pthread stack, exact command-line limits and all two-/three-/four-byte UTF-8 truncation boundaries, long Curl header values, aggregate/raw-response header rejection, Android exact/oversize readers, highly compressible inflation, and short malformed decoder input.
Compatibility regressions preserve 100 KiB raw metadata and OS-release values, 100 KiB text files, 16 KiB persisted session identifiers, and valid containers exceeding 65,536 elements. Review follow-up removes quadratic copying from record-boundary scanning and checks the order/content of 4,096 decoded records. CRLF session parsing remains supported without changing the persisted identifier.
No new test skips, negative test filters, or warning suppressions are added. The local runs select the affected and adjacent public-source suites; they are not claims of a full optional-module test run.
/W4 /WXDisabled objects were checked for native collection imports/sources on Windows, Linux, and Android. An opt-out CI matrix covers Linux, desktop macOS, and Windows. The existing Android JVM error-path test mocks and verifies production logging rather than suppressing it.
The opt-out matrix reuses
build-tests.sh debugon Linux/macOS andbuild-tests.cmd x64 Debugon Windows, retaining the established dependency setup, compiler settings, and unit/functional runners. Windows uses the existing custom-properties import to disable collection in SDK and test compilation; the device-ID regression source is also registered in the checked-in Visual Studio project. The separate dependency installation and CMake-only MSVC/bigobjworkaround are removed.Android's shared Gradle configuration forwards
-PMATSDK_ENABLE_DEVICE_ID=OFFto both the SDK and test app, with an environment fallback and default ON. Linux checks cover property/default/environment precedence. Android CI additionally assembles SDK/app Debug with collection disabled, and the test app compiles the native device-ID regressions.The existing POSIX script's
APITest.C_API_Testexclusion is unchanged, and that test passed when checked separately. An earlier local full Windows functional run reached the existing timeout inAPITest.LogManager_Initialize_DebugEventListener; no timeout or exclusion was added or relaxed. All three reused-script opt-out CI jobs subsequently passed on Linux, macOS, and Windows at commit43a5f3d5. The current local validation above is targeted, not a claim of a full platform test run.The Linux and Windows selected-suite logs contain no skipped or failed tests.
Integration
This fixes the SDK source. ONNX Runtime still needs a released SDK dependency update; do not close microsoft/onnxruntime#32771 solely on this PR.
Apple changes and the full WinRT runtime need their platform CI/review; the affected WinRT device collector has been compiled locally in enabled and disabled modes. Optional private modules are not present in the public-source build and are outside the local coverage claim.