Skip to content

perf(producer): the audio task finds sound with a probe and reads URL clips in parallel - #6174

Queued
miguel-heygen wants to merge 6 commits into
mainfrom
perf/audio-task-probe-only
Queued

miguel-heygen wants to merge 6 commits into
mainfrom
perf/audio-task-probe-only

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

Two changes to the distributed audio task, both aimed at cut-heavy edits (one <video> per cut over a few large sources):

  1. No frame extraction. The task ran the whole video extract stage (frame decoding, including a tail decode per clip window) only to learn which videos carry sound. It now probes each source for an audio stream instead and adds the same sound entries, so the mix is unchanged. With the probe cache already on main, each source is probed once however many clips use it.
  2. Clips held at a URL are read many at a time. The mix prepares each clip in a slot sized for CPU (half the cores). A clip read from a URL spends its time waiting on the network, so those clips now use their own 16-wide slot; clips read from disk keep the CPU-sized one.

What I measured

  • videoAudioProbe.test.ts: three clips over a source with sound and one silent source give one audio entry for the first, and no frames are written.
  • audioMixer.urlRead.test.ts: 24 one-second clips of a slow URL peak above the CPU-sized limit of open requests. With the new slot narrowed back to 2 the test fails (2 open requests, expected more than 4).
  • Existing buildPlanAudio tests (a video's sound, a script-set src, deferred audio) pass unchanged.
  • Before this change, on 17 and 93 clips of small sources (Lambda, arm64): the audio task's extract stage took 6.2 s and 25.7 s, and its audio stage 5.0 s and 23.2 s. On the same 93 clips with local files, the audio stage takes 5.6 s and the extract stage 2 ms with this change.

Lambda before/after (a cut-heavy 4K edit)

A 840 s, 93-cut, 22 GB, 6-source 4K edit on a site, default flags, 3 cold runs each, same stack. Before is the release without #6172 and this change; after has both.

Before (3 runs) After (3 runs)
Wall, submit to assemble 388.9 / 386.7 / 302.7 s 172.8 / 178.6 / 169.8 s
Lambda cost $0.965 / $0.921 / $0.915 $0.899 / $0.884 / $0.878
Audio task 182 / 179 s 49.0 / 52.8 / 45.4 s
Audio task extract stage ~112 s 0.8 s
Max memory, plan Lambda 3,026 MB of 10,240 3,034 / 3,031 / 3,035 MB
Max memory, audio task 2,896 MB 1,096 / 1,175 / 1,151 MB

The audio task now finishes inside the chunk window. The plan (118 to 120 s here, all in its faster mode; the runs before took 198, 196 and 118 s) is now the largest part of the wall. These runs are on the first commit of this PR; the later commits only fix a start-before-timeline case and formatting.

Probe dedupe on a site

The probes in appendVideoAudioFromProbe start together, one per <video>. On a site the sources' URLs are content-addressed (declared at compile in both the plan and the audio task), so the video-metadata cache returns the first call's in-flight promise to the rest and keeps its result: one probe per source however many clips use it. This is from the cache code, and the Lambda run's request count is the check. A plain https source that was not declared is probed per clip, as before.

What I did NOT exercise

  • Output parity (frames and audio against the baseline output) is being checked on the same runs and is not in this body yet.
  • The mix itself is untouched: no change to summing order, limiter or loudness.

Base automatically changed from perf/remote-source-probe-cache to main October 24, 2026 03:10
@miguel-heygen
miguel-heygen force-pushed the perf/audio-task-probe-only branch from 661ee22 to 9dd3154 Compare October 24, 2026 03:13
@miguel-heygen miguel-heygen changed the title perf(producer): the audio task finds a video's sound with a probe, not a frame extraction perf(producer): the audio task finds sound with a probe and reads URL clips in parallel Oct 24, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 2 blocker · 0 important · 0 nits · head e6e639ac

  • blocker: Preflight (lint + format) fails with Type 'ExtractedFrames' is imported but never used at extractVideosStage.ts:36. The narrowed appendAutoDetectedVideoAudio signature left that import with no use. The red regression and Windows render verification runs on this head are downstream of that failure.
  • blocker (inline): the probe path skips the timing rebase that extraction used to apply, so "the mix is unchanged" doesn't hold for every clip.
Scope

I read all 5 files. I ran the new probe test, and ran the buildPlanAudio script-src test from plan.test.ts plus two variants of it (open-ended slot, negative data-start) at this head and at the merge base da3e9b63. The open-ended variant passes on both. The negative-start variant passes at the base and fails at this head. I didn't re-time anything on Lambda. The 16-wide remote slot looks right: it's chosen per element, and on-disk clips keep the CPU-sized limit.

Comment thread packages/producer/src/services/render/stages/extractVideosStage.ts Outdated
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 150 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (2)

  • seqrotatenudge-none-center-r0-nested-z100: tracking 0.02, pressJump 0, drop 0, reload 3.03, render 0.04, renderKey -, undo false, teleport true / tracking 0.02, pressJump 0, drop 0, reload 0, render 0.05, renderKey -, undo true, teleport true / tracking 0.02, pressJump 0, drop 0, reload 0, render 0.05, renderKey -, undo true, teleport true
  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 2 resolved, 0 open, 0 new · e6e639ac → c37fdd37

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 24, 2026
Any commits made after this event will not be merged.
@miguel-heygen miguel-heygen changed the title perf(producer): the audio task finds sound with a probe and reads URL clips in parallel perf(producer): the audio task finds sound with a probe, and remote clips are read in parallel Oct 24, 2026
@miguel-heygen miguel-heygen changed the title perf(producer): the audio task finds sound with a probe, and remote clips are read in parallel perf(producer): the audio task finds sound with a probe and reads URL clips in parallel Oct 24, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant