Skip to content

feat(native): replace GStreamer with cross-platform Rust streamer v2 - #776

Merged
zortos293 merged 145 commits into
devfrom
capy/native-streamer-v2
Sep 6, 2026
Merged

feat(native): replace GStreamer with cross-platform Rust streamer v2#776
zortos293 merged 145 commits into
devfrom
capy/native-streamer-v2

Conversation

@zortos293

@zortos293 zortos293 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • replaces the former GStreamer implementation and checked-in runtime payload with a self-contained Rust 2024 native streamer
  • implements explicit NVST transport with authenticated SRTP, replay protection, ordered packet/frame assembly, bounded queues, H.264 Annex-B delivery, and fail-closed selection without WebRTC media fallback
  • completes the negotiated ICE/DTLS/SCTP bundle path with routable peer handoff, standard data-channel ordering, RFC 3550 receiver metrics, RFC 4585 generic NACK/PLI, retry-safe feedback, channel-close handling, and typed input-handshake timeout
  • adds robust RTSP-over-WebSocket lifecycle handling: CSeq validation, timeout invalidation, health-aware reuse, ping/pong, and idempotent native UDP reservation release
  • ships Linux x64/ARM64 V4L2 and VA-API decode with Vulkan X11/XWayland presentation, PipeWire/ALSA output, and OpenH264/SDL fallback; production and CI now compile the VA-API path
  • adds Windows x64/ARM64 Media Foundation/D3D11/WASAPI support with honest bundled capability probing, proactive default audio-endpoint switching, and software fallback
  • adds macOS VideoToolbox/Metal/CoreAudio failure propagation, keyframe recovery, one-shot OpenH264/SDL fallback, and frontmost-owned overlay ordering
  • removes unsupported runtime bitrate/shortcut/stat contracts, disables external native rendering until OS-level input capture exists, and keeps renderer input pending until native readiness or an explicit unavailable event
  • preserves session-stable state and structurally redacted diagnostics; generated media, authentication state, session secrets, and packaged binaries are excluded

Validation

A fresh authorized explicit-NVST Linux session on the final code established:

  • ICE/DTLS and SCTP completion with stats_channel, reliable input, partially-reliable input, and rtcp_on_sctp_private open
  • authenticated H.264 receive and sustained assembly (611 frames in the observed burst)
  • OpenH264/SDL first-frame presentation
  • recurring receiver reports plus an RFC 4585 PLI on the accepted feedback channel
  • explicit input downgrade after the server sent no protocol handshake; no packets were silently accepted

The live environment did not expose an Opus SSRC and used a dummy SDL audio driver, so it does not prove audio-device output. It also did not produce a live packet-loss event, so NACK interoperability remains covered by frozen wire-layout/reordering tests rather than live loss injection.

The final source passes:

  • cargo fmt --all -- --check
  • cargo test --workspace (147 unit/integration tests plus doc-test targets)
  • Linux backend all-feature tests, including the production VA-API feature chain
  • cargo clippy --workspace --all-targets -- -D warnings
  • release workspace and packaged native-streamer builds
  • Windows platform checks for x86_64-pc-windows-msvc and aarch64-pc-windows-msvc
  • macOS backend host tests plus aarch64-apple-darwin backend type-check/Clippy validation
  • npm run lint (zero warnings)
  • npm run typecheck
  • npm test (726 passing)
  • npm run build
  • npm run native:build
  • git diff --check

Remaining hardware acceptance

Real-device acceptance is still required for Media Foundation/D3D11/WASAPI and endpoint/device-loss behavior on Windows x64 and ARM64, V4L2/VA-API/Vulkan/PipeWire/ALSA device-loss paths on representative Linux systems, and VideoToolbox/Metal/CoreAudio fallback behavior on macOS hardware. Live server delivery of Opus and the proprietary input handshake is also still required before those session capabilities can be called operational; the implementation now reports them unavailable instead of silently claiming success.

@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: f9fea874-af86-451d-9eb7-5273d993adf2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@capy-ai capy-ai Bot changed the title refactor(native): replace GStreamer streamer with Rust v2 foundation feat(native): replace GStreamer with Rust streamer v2 Aug 18, 2026
@blacksmith-sh

This comment has been minimized.

zortos293 and others added 21 commits September 4, 2026 23:46
Retain decoder samples through GPU presentation, recover reference chains after missing data and queue overflow, and request startup keyframes until received. Bound audio and video pipelines, improve backend readiness, and expand redacted transport and FFI diagnostics.
Index cached games for bounded pages and ranked search, poll resumed sessions until ready, validate embedded decoder capabilities, and improve updates and diagnostics. Set version 1.0.0 and persist opt-in console switching with legacy preference migration.
Polish settings, sidebar motion, Store paging, release notes and desktop updates. Fix persistent console-mode opt-out, controller diagnostics, session recovery, input capture, statistics and GPU texture presentation. Add regression coverage and deploy required native runtime libraries in portable builds.
Prioritize codex/redesign-qt-settings-and-overlays during conflict resolution. Keep asynchronous diagnostics and discovery/claim/poll recovery; integrate compatible UDP routing, first-frame gating, and MinGW linking fixes. Adapt recovery fixtures and retain socket ownership in the parallel ping test.
…gs-and-overlays

Integrate Desktop Renew and native streaming performance fixes
…-framework

refactor: OpenNOW Rebuild V1.0.0
@zortos293
zortos293 marked this pull request as ready for review September 6, 2026 14:49
@zortos293

Copy link
Copy Markdown
Collaborator Author

Integration update: PR #781 is now merged into this branch, including Desktop Renew and the native/Qt fixes from #792. Merge using a merge commit only, preserving all original commits. The latest local Qt/native integration checks passed as documented in #792; GitHub CI remains non-green (Qt6GuiPrivate configuration, non-Windows warning-as-error issues, and headless Windows D3D11 video tests). The original validation section above describes the earlier branch snapshot, not all current hosted CI results. The dev-only changes merge cleanly and are documentation-only.

@zortos293

Copy link
Copy Markdown
Collaborator Author

Pushed 2e1d64b and e1433c6: Qt GuiPrivate discovery supports Qt 6.8 and newer layouts; Windows-only Rust state is cfg-scoped; GPU tests skip only an actual E_NOINTERFACE capability probe and still fail on subsequent initialization/rendering errors. CodeQL alert 55 was independently reviewed and dismissed as a protocol-specific false positive (RFC 8489 section 14.5); no crypto or validation changes. Local Qt build, focused native tests, and strict Windows workspace Clippy pass. Full local tests and required CI are being checked before merge. Evidence: docs/dev-merge-checks.md.

@zortos293

Copy link
Copy Markdown
Collaborator Author

Verification at b741678: local Qt Release build and 137/137 CTest tests pass; Rust core and native streamer workspace tests pass; strict Clippy passes for both Windows workspaces and Linux platform cross-targets. Real-GPU conversion and presentation tests passed locally. CI Windows x64, Windows ARM64, Linux x64 and both macOS Qt builds/tests/packages have passed; Linux ARM64 jobs and one duplicate required Windows native job are still pending. Four focused commits fix package discovery, platform cfg/test setup, and ARM64 host shader tools without changing runtime streaming or cryptographic behavior. Alert 55 was dismissed only after two independent reviews confirmed the standard-required STUN HMAC-SHA1 field; CodeQL remains enabled and passes. Normal merge remains gated on required checks.

@zortos293
zortos293 merged commit 2e0e063 into dev Sep 6, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants