[AksharP5]

What

Keep the clip speed of browser-discovered native video and audio in exports, including runtime-inserted media and media reconciled after a source update. Runtime-inserted clips also keep their data-automation speed ramps in picture and audio export.

Why

Browser discovery drops playback rate before the producer creates or reconciles media entries. Automatically detected video audio also drops the video's rate. A clip authored at 1.5× exports at 1×, and a runtime-inserted speed ramp plays flat.

Related work

The duplicate search found no open issue or PR for this discovery path. #5300 audits authored audio slot length; #4736 carries existing video metadata into distributed rendering; #3542 scales extraction coverage; #3536 handles snapshot rate; #5440 handles preview transport speed. These paths do not populate the missing browser-discovered rate.

How

Use the existing playback-rate reader and rate-lane resolver at the browser discovery boundary. Carry the resulting RateSpec through discovery and reconciliation, and inherit the video's rate when creating its audio entry. Already-collected lanes retain their scene-trimmed phase. When a browser lane replaces a scalar rate, shift the lane by the collected scene trim and recompute the source in-point using that lane. Picture and linked audio share the adjustment. The parser, probe and extraction stage share the engine's video-audio ID builder.

For a scene trimmed by 2 seconds, a clip in-point of 1 second and a collected 2× rate, a browser lane that runs at 1× through the trimmed portion must start at source 3 seconds. Before this fix, it started at source 5 seconds and restarted the ramp.

Test plan

• Discovery/reconciliation, detected-audio rate inheritance and serialized video/audio lane regressions cover the original missing-rate paths.
• Four new regressions fail before the scene-trim correction. They exercise the real collector and probe stage for video, linked video audio, native audio and runtime in-point updates. Collected trimmed lanes stay unchanged.
• 323 tests pass across compiler, collector, probe, extraction and engine audio suites. Engine and producer builds, producer/core typechecks, lint, formatting, comment checks, package-cycle checks and the Fallow gate pass.
• Speed-ramp documentation describes trim preservation. No dependency installation was needed.

The preceding revision was checked with 17 strict browser checks, 17 exports and ten byte-identical picture/PCM control comparisons. Those exports were not rerun for the scene-trim correction; its fresh verification is through the collector and probe regressions. Full Studio/scripts typechecks are limited by missing installed checkout dependencies, and Mintlify validation awaits CI because mint is unavailable locally.

Minimal reproduction: generate a two-second video with a chirp, save the HTML below as index.html, and run hyperframes check <dir> followed by hyperframes render <dir> --fps 30 --strict-all. The clip is inserted by JavaScript rather than appearing in the authored HTML.

ffmpeg -f lavfi -i "testsrc2=size=256x64:rate=30:duration=2" -f lavfi -i "aevalsrc=0.2*sin(2*PI*(300*t+100*t*t)):s=48000:d=2" -c:v libx264 -pix_fmt yuv420p -c:a aac source.mp4

<!doctype html>
<html><head><style>html,body{margin:0}#root{width:256px;height:64px}video{width:100%;height:100%;object-fit:fill}</style></head>
<body>
<div id="root" data-composition-id="main" data-width="256" data-height="64" data-fps="30" data-duration="1" data-no-timeline="true"></div>
<script>
const video = document.createElement("video");
video.id = "clip";
video.className = "clip";
video.src = "source.mp4";
for (const [name, value] of Object.entries({
  "data-start": "0", "data-duration": "1", "data-media-start": "0.5",
  "data-playback-rate": "1.5", "data-has-audio": "true"
})) video.setAttribute(name, value);
document.querySelector("#root").append(video);
</script>
</body></html>

At timeline 0.2/0.5/0.8 seconds, expected chirp frequencies are 460/550/640 Hz. At output frame n, source frame must be floor(15 + n × 1.5). Dropping the rate leaves the 1× timing of source frame 15 + n and chirp frequencies 440/500/560 Hz.

For the runtime-inserted ramp case raised in review, change the scalar rate to 0.5 and also set:

video.setAttribute("data-automation", JSON.stringify({
  version: 1,
  lanes: [{ target: "rate", points: [{ t: 0, v: 1 }, { t: 1, v: 2 }] }]
}));

Its exported picture and PCM audio should match the same clip placed directly in static HTML. The scalar-only revision flattened this lane to the 0.5× fallback.

[miguel-heygen]

Carries a runtime-inserted or runtime-updated clip's speed into the export, and lets a video's detected audio inherit it.

Strengths

• htmlCompiler.ts:2318 reads defaultPlaybackRate rather than playbackRate, so the transport speed never leaks into the export rate.
• probeStage.ts:603 leaves an existing rate lane alone, so a ramp the collector already shifted for scene trimming keeps its phase.

Important (fine as a follow-up)

• htmlCompiler.ts:2376: discovery reads only the scalar rate. A runtime-inserted clip with a rate lane in data-automation is pushed with its default rate, so its ramp plays flat. Not a regression from main, but the description's "existing speed ramps stay intact" holds only for clips already in the static HTML. Returning data-automation raw and resolving it with the same resolveRateSpec(automation, readPlaybackRate(...)) that readElementRateSpec uses (core/src/runtime/playbackRate.ts:34) would give the browser path and the static path one owner for clip rate.

Nit

• probeStage.ts:604-613 rebuilds the ${id}-audio naming that appendAutoDetectedVideoAudio owns (extractVideosStage.ts) and scans every audio entry per video. A small helper exported from there keeps the id format in one place.

Verdict: APPROVE
Reasoning: The fix is correct for scalar rates on both the new-element and reconcile paths, with regressions that fail before the change; the lane gap is pre-existing and can follow.

[AksharP5]

Addressed the runtime-inserted speed-ramp follow-up in 692ec34 with the shared resolveRateSpec parser, and reused the video/audio identity predicate. Already-collected lanes keep their scene-trimmed phase; probe-trigger behavior is unchanged.

A fresh Astra xhigh review approved this exact revision after 17 strict browser checks, 17 actual exports, and ten byte-identical picture/PCM control comparisons. The 238 focused tests pass. Ready for re-review while the remaining CI jobs finish.

[AksharP5]

@miguel-heygen Both review follow-ups are addressed at 692ec34, and a fresh run of the 238 focused tests plus producer typecheck passes. All required checks pass or skip appropriately; the cancelled capture has a successful replacement on the same SHA. GitHub still requires approval of the latest push. Ready for another look.

[miguel-heygen]

This is a clean follow through. Discovery now resolves the rate lane with the same resolveRateSpec and readPlaybackRate pair the static path uses, so a runtime inserted clip and an authored one get their speed from one place, and reading defaultPlaybackRate rather than playbackRate keeps the preview transport out of the export. The lane and scalar cases are both covered through the real probe stage.

Two small follow ups, neither blocking. The detected audio id format now lives in two files, so exporting the builder from the stage that creates those entries would keep it to one owner. And when an existing scalar rate is replaced by a lane the browser reported, I could not tell whether the scene trim phase is applied to the new lane the way a collected one carries it. A one line answer is enough.

Approving.

[AksharP5]

@miguel-heygen On the scene-trim question: scalar-to-lane reconciliation adopts the browser lane unchanged; only an already-collected lane carries the trim phase applied by startAtPlayed and is retained. A lane first introduced at runtime inside a trimmed scene therefore does not get that adjustment in this path. Agreed that this and moving the detected-audio ID builder to its creating stage are nonblocking follow-ups.

[mintlify]

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 5cc2d9a.

Approve preview

[AksharP5]

@miguel-heygen Both follow-ups are addressed in 5cc2d9a. A focused reproduction confirmed the scalar-to-lane trim gap: with a 2s scene trim, the old path started at source 5s instead of 3s and restarted the ramp. Reconciliation now adjusts both the lane phase and source in-point for picture and linked audio, while retaining collected lanes. Video-audio IDs share the engine parser's builder. All 323 focused tests pass; engine/producer builds, producer/core typechecks, lint and comment checks pass. The description distinguishes these regressions from the prior browser/export comparisons. All 81 current checks now pass at 5cc2d9a. Ready for re-review; approval of the latest push is the only remaining merge gate.

[miguel-heygen]

Thanks for closing both of these out. The detected audio id now has one owner: videoAudioId lives next to the parser that creates those entries and the three places that used to spell out the suffix all call it, so the format cannot drift.

On the scene trim question, your answer was the right one and the follow-up fix is right too. I lifted the arithmetic out of adoptRuntimeMediaTiming and replayed it on your worked example. With a 2 second scene trim, an authored in-point of 1 and a collected 2x rate, a browser lane running at 1x now lands the source in-point at 3 seconds and rebases the lane to the start of the kept portion. The old path left it at 5 and restarted the ramp. Replaying the untrimmed case gives a no-op, and a lane the collector already carried stays untouched, which is what the comment claims.

I also checked the degenerate shapes. A head past the lane's last point collapses to a single point, and because the table keeps the last rate as both ends, that still integrates at the right speed rather than silently falling back to 1x. The clamp on consumed keeps the shift non negative, so shiftRateLane never sees a negative offset.

Two small things for later, neither blocking. In the audio reconcile branch the entry picked by id has no type filter while the timing helper filters out video sourced audio, so a project with a video and an authored audio whose ids collide on the -audio suffix could update different entries. That predates this change. And nothing covers a runtime in-point change and a scalar to lane replacement happening in the same reconcile; the new tests exercise them separately.

Checks at this head look good: every non skipped job is green, the one cancelled capture run has a successful run at the same commit, and the Mintlify preview is waiting on a maintainer rather than on you.

[AksharP5]

@miguel-heygen The combined in-point change and scalar-to-lane replacement is covered by the pageStart: 3 video/audio cases in the regression matrix. Each starts with authored in-point 1s at 2×, then adopts browser in-point 3s and the lane in one probe call, asserting source in-point 5s and the shifted lane, including linked video audio. Noted the pre-existing ID-collision case as a nonblocking follow-up.

[jrusso1020]

Approving. Media the producer only finds through the browser probe (clips inserted by script, or reconciled after a runtime source change) lost their speed on main: discoverMediaFromBrowser never read data-playback-rate or data-automation, and the auto-detected audio for a video never copied the video's rate. So a 1.5x clip exported at 1x and a runtime ramp played flat.

What I checked:

• Discovery now resolves the rate with the same resolveRateSpec(automation, readPlaybackRate(...)) pair the static parser uses, and falls back to defaultPlaybackRate, not playbackRate, so the preview transport speed cannot leak into the export. Static and runtime clips now get their speed from one owner.
• appendAutoDetectedVideoAudio copies the video's rate, so picture and sound stay in sync.
• In adoptRuntimeMediaTiming, lanes the collector already shifted for a scene trim are left alone. A scalar replaced by a browser lane gets the trim applied to both the lane phase and the source in-point. The collector does not compose parent rates into a clip's scalar, so replacing a scalar with the element's own rate loses nothing.
• I merged this head onto current main (no conflicts), and the audio mixer and probe/extract stage suites pass. Producer tsc --noEmit is clean.

Miguel's three rounds of follow-ups (shared rate parser, single owner for the -audio id via videoAudioId, the scene-trim phase) are all addressed at this head. His remaining note, that the audio reconcile branch picks an entry by id without a type filter, predates this PR and is fine as a follow-up.

Need: yes, runtime-discovered clips export at the wrong speed · Size: ok for the surface it covers · Reuse: reuses resolveRateSpec, readPlaybackRate, shiftRateLane and consolidates the video-audio id into videoAudioId · Simpler: already minimal

— Rames

[jrusso1020]

Thanks, @AksharP5. This was approved and in the merge queue, but another PR that merged ahead of it touched the same lines, so GitHub took it out of the queue with a merge conflict. Could you rebase onto the latest main? If it's a clean rebase with no code changes, we'll requeue it right away.

— Rames