GPU-direct hardware encode (#521) slice 1 completion — realtime GPU stream (T3–T6) - #523
Merged
Merged
Conversation
…64 round-trip (#521 slice1 T3 complete) Completes Task 3: the compositor exports a DEDICATED keyed-mutex BGRA shared texture for hardware encode (exportEncoderSharedTexture / ensureEncoderSharedTexture), mirroring the vcam tap's producer keying (AcquireSync(0,0)/CopyResource/ReleaseSync(1)) so encode never contends with WinUI's ProgramFrame::sharedTexture consumer. Carried on ProgramFrame::encoderSharedTexture. Uses legacy GetSharedHandle to match the encoder's legacy OpenSharedResource. Encoder consumer keying is AcquireSync(1)/ReleaseSync(0). Proven end to end on the dev rig (RTX 4090): compositor renders solid mid-gray -> dedicated keyed-mutex texture -> MF hardware H.264 MFT (VideoProcessor BGRA->NV12) -> Annex-B bitstream -> ffmpeg decode. Test asserts a keyframe, non-empty bitstream, and the decoded coded Y plane within 16 of the encoded gray (measured 128.0 exactly over 12 frames). Reads the yuv420p Y plane, not -pix_fmt gray, to avoid limited->full range expansion. Self-skips without a hardware MFT or ffmpeg. Full native suite: 939 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yuH7EMvWCWdkhvevrtMyJ
…m to muxer; env fallback (#521 slice1 T4) The RTMP/SRT sender now decides its encode path once at process start via the pure chooseStreamEncodePath policy (folded into GpuVideoEncoder.h): GPU-direct only when the platform has an MF encoder, a hardware session is (probably) available, the resolved codec is H.264, the compositor is exporting the dedicated encoder texture on the starting frame, and COREVIDEO_GPU_ENCODE is not "0". Any base blocker keeps precedence in the logged reason. On the GPU path the sender starts the hardware encoder BEFORE FFmpeg (so encoder->start() is the real capability gate; failure downgrades to raw and FFmpeg is never launched in bitstream mode with nothing feeding it), builds FFmpeg args with videoBitstreamInput=true (-c:v copy muxer, T2), submits each frame's compositor encoder texture to the encoder, and the encoder's sink writes the ~6 Mbps bitstream to FFmpeg stdin. stopFfmpegProcess stops and joins the encoder before closing the pipe (no write-after-close). The raw NV12/BGRA pipe stays the fallback for every non-capable machine, non-H.264 codec, and the env toggle. Every path selection is logged ([gpu-encode] path=...). Availability never REFUSES on a pending probe (the TESTER rule); start() is the gate. Tests: chooseStreamEncodePath (5 cases: gpu-direct, base-blocker precedence, non-H.264, missing encoder texture, env toggle). The runtime submit-vs-raw routing is proven by the Task 6 localhost acceptance gate (real core + real FFmpeg), not a mocked-process unit test. Full native suite: 944 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yuH7EMvWCWdkhvevrtMyJ
…estarts (#521 slice1 T5) The encode loop now distinguishes a genuine device loss from a transient encode miss: on a convert/ProcessInput failure it reads ID3D11Device::GetDeviceRemovedReason() and, when the device is removed/reset/hung, sets healthy_=false and breaks with a [gpu-encode] device lost line. submit() already returns false once unhealthy, so the sender reports a video-write failure and the existing OutputDestinationSupervisor restarts it, which re-decides the encode path (Task 4) against the recreated device. Capacity is already priority-1 for slice 1: the stream is the only GPU encoder, and the sender gates GPU-direct on the EncoderCapacityProbe session ceiling (sessionAvailable -> "no-free-encoder-session" fallback, covered by the existing policy tests) without ever refusing on a pending probe. Test: the submit-fails-when-not-running contract (the supervisor's restart trigger) on the real impl. A real TDR is not injected (matching the shell device-loss recovery's documented posture); the classification is verified by inspection and the round-trip test proves healthy() stays true on the happy path. Full native suite: 945 passed, 0 failed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yuH7EMvWCWdkhvevrtMyJ
…-direct stream at realtime (#521 slice1 T6) Adds scripts/validate-gpu-encode.mjs (streams the fake-engine program to a localhost SRT sink; FAILS unless the GPU path is taken and the received stream is >=58fps and the sink's -stats speed >=0.97x; --force-raw confirms the fallback) and the CLAUDE.md section. Gate result on the dev rig: path=gpu-direct, received 60.0fps of 60, realtime. Writing the gate surfaced four real defects, each fixed here (the raw-pipe path capped at ~0.76x; this is what makes GPU-direct actually keep up): - Compositor exports the dedicated encoder texture whenever streaming, BUFFERED or not (it was gated !buffered, so with the default 3-frame program buffer the sender saw no encoder texture and fell back to raw). The stable handle rides the program buffer to the sender; the copy is the latest composed frame, so the stream taps live pixels. - Encoder consumer keyed-mutex timeout 4ms -> 34ms (~2 frames): 4ms missed the 16ms production cadence on almost every frame, wasted the MFT input slot and starved the encoder to ~2fps. - Encode loop no longer waits for a fresh submit per NeedInput (that serialized with the mutex wait to ~30fps); it reads the latest handle and lets the keyed mutex pace it to the producer's 60fps. - FFmpeg bitstream input: -use_wallclock_as_timestamps 1 -r <fps>. A live Annex-B stream on a pipe has no container timestamps; -r alone left stream 0's PTS unset once audio was mapped, and -c:v copy then muxed a stream the endpoint read at 0x/stalled. Wallclock stamps each arriving access unit at realtime (monotonic for a 60fps feed). Verified: validate-gpu-encode.mjs PASS (gpu-direct, 60.0fps realtime) and --force-raw PASS (fallback still streams); mac-show-drill --load 8 PASS (p50 27.8ms, 100% delivered, no regression); full native suite 945 passed, 0 failed. Live RTMP to real YouTube remains the final manual acceptance step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yuH7EMvWCWdkhvevrtMyJ
…dable (#521) Adds -stats -stats_period 1 to both the bitstream and raw ffmpeg arg branches. This is what makes the sender's realtime keep-up (speed=x, media-time/wall-time) readable from ffmpeg's own stderr — used to confirm GPU-direct streams to real YouTube at ~1.02x vs the raw path's ~0.76x. Matches the deploy branch's intent; the raw path lacked it on this branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014yuH7EMvWCWdkhvevrtMyJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GPU-direct streaming could submit nearly 60 frames/sec while FFmpeg received only 9–35 encoded frames/sec. Synchronous pipe writes blocked the Media Foundation event thread, and missing textures could consume input credits permanently.
This change exports a dedicated encoder texture, selects hardware H.264 with raw fallback, and adds a bounded Windows bitstream writer. It retains asynchronous input credits across texture waits, consumes one output per output event, and corrects sample ownership. RTMP/RTMPS disables TCP delay; SRT excludes that option.
Validation: 946 native tests pass, including delayed-first-frame GPU decoding. The localhost GPU gate received 60.1 fps; forced raw received 59.9 fps. Real eight-source 1080p60 YouTube streaming now runs around 60 encoded fps with excellent receiver health. Broken-pipe recovery was exercised.
Release gate remains open: an earlier live sample had two Program-buffer underruns and another had a deadline miss. Investigation continues; clean average throughput does not waive these failures. Evidence and exact test limits: docs/gpu-encode-live-validation-2026-09-13.md.
Part of #521. Recording/ISO hardware encoding and macOS encoding remain outside this slice.
Follow-up: local candidate archive and real installer validation pass; all 2,224 MediaCore tests pass after the dedicated-worker test correction in 8e92536. The unresolved intermittent buffer failure is tracked in #526 and remains a release blocker.