From 9ab17fe350e1536a74a87b0add40aaa726e9716e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 13:38:23 -0800 Subject: [PATCH 1/7] chore(quest): claim quest/m0/media-file.md Quest-Claim: e9b9dea144e3610e From 68562ef741dcd3965b9345f97f16c4c828e11d87 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 14:09:08 -0800 Subject: [PATCH 2/7] feat(scene): a media file source that loops or plays on demand A `media` source plays a local video or audio file on the program clock, demuxed by moq-mux and decoded by moq-video and moq-audio. It loops, or plays once and holds its last picture; going on program can restart it, and `Feed::play` restarts it on demand. A file that ends or fails reads as ended or failed in Sources and Health. Co-Authored-By: Claude Opus 5.5 --- README.md | 44 +- quest/m0/README.md | 1 - quest/m0/media-file.md | 29 -- quest/m0/stinger.md | 9 +- quest/m0/web.md | 4 + rs/moq-scene/src/collection.rs | 46 ++- rs/moq-scene/src/lib.rs | 8 +- rs/moq-scene/src/media.rs | 187 +++++++++ rs/moq-scene/src/publish.rs | 68 ++++ rs/moq-scene/src/source.rs | 574 ++++++++++++++++++++++++--- rs/moq-scene/src/test_data/clip.mkv | Bin 0 -> 13211 bytes rs/moq-scene/src/test_data/clip.mp4 | Bin 0 -> 14210 bytes rs/moq-scene/src/test_data/tone.webm | Bin 0 -> 1771 bytes rs/moq-studio/src/engine.rs | 70 ++-- rs/moq-studio/src/health.rs | 22 +- rs/moq-studio/src/ui.rs | 58 ++- 16 files changed, 983 insertions(+), 137 deletions(-) delete mode 100644 quest/m0/media-file.md create mode 100644 rs/moq-scene/src/media.rs create mode 100644 rs/moq-scene/src/test_data/clip.mkv create mode 100644 rs/moq-scene/src/test_data/clip.mp4 create mode 100644 rs/moq-scene/src/test_data/tone.webm diff --git a/README.md b/README.md index 0353a07..23d6a86 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Every panel is a tab you can drag, split, or stack. | Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still leaving it in a transition, cannot be removed, even by an undo, until program cuts away from it. | | History | See every edit, and undo or redo back to any of them. | | Items | Place, show, hide, and order what the preview scene or the top layer draws, back to front. Select an item to filter or copy it. | -| Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. | +| Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. A media file reads as ended once it plays to its end without looping, **Play** plays it from its start, and **Reload** reads the file again. | | Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. | | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. | | Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | @@ -223,6 +223,38 @@ costs nothing until it is shown again, where its animation has carried on. Overlays publish no track of their own, so viewers see them only in the program. +### Media files + +A `media` source plays a video or audio file, such as an intro clip or a music +bed, on the program clock, and its sound joins the mixes like any other +source's. It reads fragmented MP4, MKV or WebM, MPEG-TS, and FLV, as moq's +importers do, and fails to start on a codec this machine cannot decode. A +regular MP4 fails to start with a remux to fragmented MP4 that makes it readable: +`ffmpeg -i in.mp4 -c copy -movflags frag_keyframe+empty_moov out.mp4`. The +whole file is read into memory, compressed, when the source starts, so it +plays from its start at once and loops without reading the disk again; keep it +to clips and music, not hours of video. + +```toml +[sources.intro] +kind = "media" +path = "intro.mkv" # relative to the scene file, like an image's +restart = true # play from the start each time it goes on program + +[sources.music] +kind = "media" +path = "music.webm" +loop = true # start over at the end +``` + +Without `loop`, it plays once, then holds its last picture, silent, and reads +as ended in Sources and, while the program shows or plays it, in Health. +**Play** in Sources plays it from its start again, as does going on program +with `restart`, in a scene on air or the top layer, when it was not on program +before. A file that cannot be read or decoded fails, with the reason. Like a +camera's, its pictures have a track of their own (see [Source +tracks](#source-tracks)), on one timeline across plays; its sound has none yet. + ## Guests Guests and phones join a room the studio owns on its relay, from a web page, @@ -404,7 +436,8 @@ rect = { x = 1600, y = 40, width = 280, height = 120 } Source kinds are `camera`, `display`, and `microphone` (each with an optional `id`, the default device without one), `window` (with an `id`), `system_audio` -(macOS), `image` (PNG or JPEG), `color` (`#rrggbb` or `#rrggbbaa`), +(macOS), `image` (PNG or JPEG), `media` (a video or audio file; see +[Media files](#media-files)), `color` (`#rrggbb` or `#rrggbbaa`), `broadcast`, `browser` (a `url`, and a `width` and `height` up to 4096), `overlay` (its `html`, and a `width` and `height` up to 4096), `clone` (with the `source` it shows), `lut` (a 3D Adobe or Resolve `.cube` file, which no item places, but which grades what a `lut` filter names it in), @@ -514,7 +547,7 @@ stream. ### Source tracks -Each camera, display, window, image, and color also has a track of its own, so +Each camera, display, window, image, media file, and color also has a track of its own, so the studio can replay or record it on its own. These tracks stay in the studio: the program is the only video it publishes to a relay. They are listed in a local catalog under the `dev.moq.studio` section's `sources`, by source name, each @@ -542,7 +575,7 @@ encode engine for them all. The Output panel counts the sessions open: | The program's canvas size, live | 1, or none without `--connect` | | Its half size | 1 while someone watches it | | The recording's own canvas size | 1 while recording | -| Each captured source | 1 while on program or recorded | +| Each captured source or media file | 1 while on program or recorded | So a live scene showing a camera, a display, and a window holds 4 sessions, 5 with the half size watched, and 6 while recording too. When the program @@ -639,7 +672,8 @@ with its cause in plain words: a relay is unreachable, the uplink is congested or losing packets, the relay is far away, the encoder is behind or stuck, a source stalled (a camera or broadcast the program draws, or a microphone or system audio the mix on program plays: unmuted and not left -out of it), or viewers are late, stalling, or cannot +out of it), a source the program shows or plays failed or, as a media file +that does not loop, played to its end, or viewers are late, stalling, or cannot decode a track. A relay lost while the other serves is degraded, since viewers have nothing left to fall back to; with none serving it is unhealthy. Below that it shows: diff --git a/quest/m0/README.md b/quest/m0/README.md index 48c63b8..3e00632 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -94,7 +94,6 @@ repository. - [Broadcast sources play in sync](/quest/m0/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker - [Keyframes](/quest/m0/animation.md) - keyframed item properties with easing, starting when their scene goes on program -- [Media file source](/quest/m0/media-file.md) - a local video or audio file plays as a source, looping or from the start on demand - [Stinger transitions](/quest/m0/stinger.md) - a clip played over the program that switches scenes at its cut point - [ML video effects](/quest/m0/effects.md) - background removal and other ML effects, baked into their source's track - [Destinations](/quest/m0/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse diff --git a/quest/m0/media-file.md b/quest/m0/media-file.md deleted file mode 100644 index 8f5008d..0000000 --- a/quest/m0/media-file.md +++ /dev/null @@ -1,29 +0,0 @@ -# [M] Media file source - -## Goal - -A source in `moq-studio` plays a local video or audio file, such as an intro -clip or background music: looping, or from the start each time something asks, -with its sound joining the mixes like any other source. - -## Plan - -Decided 2026-11-04 (maintainer): planned so stinger transitions have a clip -that starts on demand. Demux and decode through the moq crates the studio -already uses (moq-mux import, moq-video and moq-audio decode) rather than a -new media stack; anything they lack lands in moq first, with a moq-side test, -then a pin. Start with the container and codecs those crates already read, and -refuse anything else at setup with the reason. The file path is a local path -in the scene file, like an image source's. The web studio has no local paths: -there a file comes from a browser file picker for the session, and resolving -scene-file paths in the browser belongs to the [web studio](/quest/m0/web.md), -as it does for image sources. - -A file is played on the program clock. Settings: loop, or play once and hold -the last frame (an audio-only file goes silent after its last sample); and whether going on program restarts it. A restart on demand -(the stinger's need) is an API on the source, not only a setting. A file that -ends or fails reads as ended or failed in the Sources panel and Health. - -## Related - -- [Stinger transitions](/quest/m0/stinger.md) - the first consumer of a clip that starts on demand diff --git a/quest/m0/stinger.md b/quest/m0/stinger.md index 9c6847e..f6279a5 100644 --- a/quest/m0/stinger.md +++ b/quest/m0/stinger.md @@ -11,9 +11,6 @@ Split from [Keyframes](/quest/m0/animation.md) on 2026-11-04 (maintainer). A stinger is a transition kind beside fade, slide, wipe, and move ([moq-dev/studio#33](https://github.com/moq-dev/studio/pull/33)), drawn on the program clock, whose clip is a media file source. It must start its clip as -the transition starts, or its cut point lands at random. Its settings (clip -and cut point) change through `Edit`, so they undo. - -## Required - -- [Media file source](/quest/m0/media-file.md) - a clip that starts on demand +the transition starts, or its cut point lands at random: `Feed::play` in +`rs/moq-scene/src/source.rs` starts a media source from its start at once. +Its settings (clip and cut point) change through `Edit`, so they undo. diff --git a/quest/m0/web.md b/quest/m0/web.md index 6fe0701..d490116 100644 --- a/quest/m0/web.md +++ b/quest/m0/web.md @@ -31,6 +31,10 @@ same PR as this page, and the page follows each pin bump. The page was planned for moq's demo/web; it now ships from this repository, and where it is hosted is settled in this quest. +Decided 2026-11-04 (maintainer): a page has no local paths, so an image or +media source's file comes from a browser file picker for the session, and +resolving the scene file's paths in the browser is settled here. + ## Required - [moq-video's browser backends are pinned](/quest/m0/pin-browser-video.md) - the video slice's capture, codecs, and GPU import diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 9c74226..44b0fbf 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -1014,6 +1014,20 @@ pub enum Source { /// The file, relative to the scene file's directory unless absolute. path: PathBuf, }, + /// A video or audio file, such as an intro clip or a music bed, played on + /// the program clock from its start, and again whenever + /// [`Feed::play`](crate::source::Feed::play) asks. + Media { + /// The file, relative to the scene file's directory unless absolute. + path: PathBuf, + /// Whether it starts over at its end, rather than holding its last + /// picture and going silent. + #[serde(default, rename = "loop", skip_serializing_if = "std::ops::Not::not")] + looping: bool, + /// Whether it plays from its start each time it goes on program. + #[serde(default, skip_serializing_if = "std::ops::Not::not")] + restart: bool, + }, /// A solid color. Color { /// The color, as `#rrggbb` or `#rrggbbaa`. @@ -1074,10 +1088,10 @@ impl Source { !matches!(self, Self::Microphone { .. } | Self::SystemAudio | Self::Lut { .. }) } - /// The file an image or LUT source reads. + /// The file an image, media, or LUT source reads. pub(crate) fn path(&self) -> Option<&std::path::Path> { match self { - Self::Image { path } | Self::Lut { path } => Some(path), + Self::Image { path } | Self::Media { path, .. } | Self::Lut { path } => Some(path), _ => None, } } @@ -1085,7 +1099,7 @@ impl Source { /// The file the source reads, to resolve or relativize it. pub(crate) fn path_mut(&mut self) -> Option<&mut PathBuf> { match self { - Self::Image { path } | Self::Lut { path } => Some(path), + Self::Image { path } | Self::Media { path, .. } | Self::Lut { path } => Some(path), _ => None, } } @@ -1100,7 +1114,12 @@ impl Source { pub fn has_audio(&self) -> bool { matches!( self, - Self::Microphone { .. } | Self::SystemAudio | Self::Broadcast { .. } | Self::Browser { .. } | Self::Replay + Self::Microphone { .. } + | Self::SystemAudio + | Self::Media { .. } + | Self::Broadcast { .. } + | Self::Browser { .. } + | Self::Replay ) } @@ -1268,6 +1287,14 @@ mod tests { path: "studio/guest".into(), }, ); + collection.sources.insert( + "intro".into(), + Source::Media { + path: "intro.mkv".into(), + looping: false, + restart: true, + }, + ); collection.sources.insert( "chat".into(), Source::Browser { @@ -1561,7 +1588,16 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } collection.validate().unwrap(); collection.top.push(Item::new("grade", Rect::fill(&collection.canvas))); assert!(matches!(collection.validate(), Err(Error::Invalid(_)))); - for source in [Source::Lut { path: "".into() }, Source::Image { path: "".into() }] { + let media = Source::Media { + path: "".into(), + looping: true, + restart: false, + }; + for source in [ + Source::Lut { path: "".into() }, + Source::Image { path: "".into() }, + media, + ] { let mut collection = example(); collection.sources.insert("blank".into(), source.clone()); assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "{source:?}"); diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index f5d0d77..46d7ec9 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -12,7 +12,7 @@ //! scenes and collections as text. //! - [`source::Sources`] runs each source: a camera, display, window, //! microphone, or system-audio capture (with the `capture` feature), a still -//! image, a solid color, a broadcast subscribed over MoQ, or a web page +//! image, a video or audio file, a solid color, a broadcast subscribed over MoQ, or a web page //! rendered by a browser engine (the [`browser`] module). Pictures go to the //! compositor and sound goes to a `moq_audio::bus::Bus`, routed into its //! program mix. @@ -36,6 +36,7 @@ mod compositor; mod edit; mod effect; mod lut; +mod media; mod overlay; pub mod publish; pub mod replay; @@ -86,6 +87,11 @@ pub enum Error { #[error("image: {0}")] Image(String), + /// A media source's file could not be read, or holds nothing the studio + /// plays. + #[error("media: {0}")] + Media(String), + /// A LUT source could not be read or parsed. #[error("LUT: {0}")] Lut(String), diff --git a/rs/moq-scene/src/media.rs b/rs/moq-scene/src/media.rs new file mode 100644 index 0000000..d77bbc0 --- /dev/null +++ b/rs/moq-scene/src/media.rs @@ -0,0 +1,187 @@ +//! Media files, such as an intro clip or a music bed, demuxed by moq-mux into a +//! broadcast of their own for a media source to decode. +//! +//! The whole file is imported when the source opens, still compressed, so each +//! play decodes from its first frame at once, as a stinger needs, and a loop +//! never reads the disk again. The source holds the file's size in memory for +//! as long as it runs. + +use std::path::Path; + +use hang::catalog::{AudioConfig, VideoConfig}; +use moq_mux::import::ContainerFormat; + +use crate::Error; + +/// A media file's video and audio, held in a broadcast of their own. +pub(crate) struct Clip { + broadcast: moq_net::broadcast::Producer, + /// Keeps the imported tracks readable. + _import: moq_mux::import::Container, + _catalog: moq_mux::catalog::Producer, + /// The video rendition, by catalog name. + pub video: Option<(String, VideoConfig)>, + /// The audio rendition, by catalog name. + pub audio: Option<(String, AudioConfig)>, +} + +impl Clip { + /// Read and demux the file at `path`. Refuses a container moq-mux does not + /// read, saying which it does, and a file with neither video nor audio in + /// it that moq-mux reads. + pub fn open(path: &Path) -> Result { + let failed = |err: String| Error::Media(format!("{}: {err}", path.display())); + let bytes = std::fs::read(path).map_err(|err| failed(err.to_string()))?; + let format = container(&bytes).map_err(failed)?; + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + // The file is written whole before anything reads it, so nothing in it + // may go stale behind the newest frame. + let config = moq_mux::catalog::Config::default().with_max_age(None); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, config)?; + let init = moq_mux::import::ContainerInit::new(format, bytes); + let mut import = moq_mux::import::Container::new(broadcast.clone(), catalog.reserve(), &init) + .map_err(|err| failed(err.to_string()))?; + import.finish().map_err(|err| failed(err.to_string()))?; + + let listed = catalog.snapshot(); + let video = crate::source::pick(listed.video.enabled())?; + let audio = crate::source::pick(listed.audio.enabled())?; + if video.is_none() && audio.is_none() { + return Err(failed("it holds no video or audio the studio plays".into())); + } + Ok(Self { + broadcast, + _import: import, + _catalog: catalog, + video, + audio, + }) + } + + /// The clip's broadcast, to decode from its start. + pub fn consume(&self) -> moq_net::broadcast::Consumer { + self.broadcast.consume() + } +} + +/// The containers a media file may be in, as moq-mux reads them. +const READ: &str = "fragmented MP4, MKV or WebM, MPEG-TS, or FLV"; + +/// Which container `bytes` are in, by their first bytes. +/// +/// A regular MP4, with its samples indexed in a `moov` rather than in +/// fragments, is refused: moq-mux demuxes only fragmented MP4. +fn container(bytes: &[u8]) -> Result { + const TS_PACKET: usize = 188; + if bytes.starts_with(&[0x1a, 0x45, 0xdf, 0xa3]) { + return Ok(ContainerFormat::Mkv); + } + if bytes.starts_with(b"FLV") { + return Ok(ContainerFormat::Flv); + } + if bytes.first() == Some(&0x47) && bytes.get(TS_PACKET).is_none_or(|sync| *sync == 0x47) { + return Ok(ContainerFormat::Ts); + } + if bytes.get(4..8) == Some(b"ftyp") { + return match fragmented(bytes) { + true => Ok(ContainerFormat::Fmp4), + false => Err(format!( + "a regular MP4 is not read yet; remux it into one of {READ}, such as with \ + `ffmpeg -i in.mp4 -c copy -movflags frag_keyframe+empty_moov out.mp4`" + )), + }; + } + Err(format!("not a container the studio reads; it reads {READ}")) +} + +/// Whether the MP4 in `bytes` holds a fragment (`moof`) among its top-level boxes. +fn fragmented(mut bytes: &[u8]) -> bool { + while let Some(header) = bytes.get(..8) { + if &header[4..] == b"moof" { + return true; + } + let size = u32::from_be_bytes(header[..4].try_into().expect("four bytes")); + let size = match size { + // To the end of the file. + 0 => return false, + // A 64-bit size follows the type. + 1 => match bytes.get(8..16) { + Some(large) => u64::from_be_bytes(large.try_into().expect("eight bytes")), + None => return false, + }, + size => u64::from(size), + }; + match usize::try_from(size).ok().filter(|size| *size >= 8) { + Some(size) if size <= bytes.len() => bytes = &bytes[size..], + _ => return false, + } + } + false +} + +#[cfg(test)] +pub(crate) mod tests { + use super::*; + + /// One second of 64x64 H.264 at 10 fps with a keyframe every half second, + /// and stereo Opus, in MKV: + /// + /// ```sh + /// ffmpeg -f lavfi -i testsrc2=size=64x64:rate=10:duration=1 \ + /// -f lavfi -i sine=frequency=440:sample_rate=48000:duration=1 \ + /// -c:v libx264 -profile:v baseline -g 5 -bf 0 -c:a libopus -b:a 24k -ac 2 clip.mkv + /// ``` + pub(crate) const CLIP: &[u8] = include_bytes!("test_data/clip.mkv"); + + /// The same second, remuxed with `-c copy -movflags frag_keyframe+empty_moov`. + const FRAGMENTED: &[u8] = include_bytes!("test_data/clip.mp4"); + + /// Half a second of stereo Opus alone, in WebM. + pub(crate) const TONE: &[u8] = include_bytes!("test_data/tone.webm"); + + #[test] + fn reads_by_the_first_bytes() { + assert_eq!(container(CLIP), Ok(ContainerFormat::Mkv)); + assert_eq!(container(FRAGMENTED), Ok(ContainerFormat::Fmp4)); + assert_eq!(container(b"FLV\x01"), Ok(ContainerFormat::Flv)); + let mut ts = vec![0; 2 * 188]; + (ts[0], ts[188]) = (0x47, 0x47); + assert_eq!(container(&ts), Ok(ContainerFormat::Ts)); + ts[188] = 0; + assert!(container(&ts).is_err(), "one sync byte is not a stream"); + assert!(container(b"\x89PNG\r\n\x1a\n").unwrap_err().contains(READ)); + } + + /// A regular MP4 says how to make one moq-mux reads. + #[test] + fn a_regular_mp4_is_refused() { + let mut mp4 = Vec::new(); + for (kind, body) in [(b"ftyp", 8), (b"moov", 32), (b"mdat", 64)] { + mp4.extend_from_slice(&(8 + body as u32).to_be_bytes()); + mp4.extend_from_slice(kind); + mp4.resize(mp4.len() + body, 0); + } + let err = container(&mp4).unwrap_err(); + assert!(err.contains("regular MP4") && err.contains("frag_keyframe"), "{err}"); + // Even when a box says it is larger than the file. + mp4.truncate(mp4.len() - 1); + assert!(container(&mp4).is_err()); + } + + #[test] + fn opens_both_renditions() { + let dir = tempfile::tempdir().unwrap(); + for (name, bytes) in [("clip.mkv", CLIP), ("clip.mp4", FRAGMENTED)] { + let path = dir.path().join(name); + std::fs::write(&path, bytes).unwrap(); + let clip = Clip::open(&path).unwrap(); + assert!(clip.video.is_some(), "{name} has no video"); + assert!(clip.audio.is_some(), "{name} has no audio"); + } + + let missing = dir.path().join("missing.mkv"); + let err = Clip::open(&missing).err().unwrap().to_string(); + assert!(err.contains("missing.mkv"), "{err}"); + } +} diff --git a/rs/moq-scene/src/publish.rs b/rs/moq-scene/src/publish.rs index 8ee7ee2..4ab0f01 100644 --- a/rs/moq-scene/src/publish.rs +++ b/rs/moq-scene/src/publish.rs @@ -1188,4 +1188,72 @@ mod tests { }) .await; } + + /// A media source on program encodes its pictures onto its own track as + /// they show, on one timeline when it plays again from its start. + #[tokio::test(start_paused = true)] + async fn a_media_file_is_published_on_one_timeline() { + local(async { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join("clip.mkv"), crate::media::tests::CLIP).unwrap(); + let (publish, catalog, consumer) = studio(); + publish.show(["clip".to_string()].into()); + let (mut context, _origin) = context(dir.path()); + context.publish = Some(publish); + let mut collection = Collection::new(); + let clip = Source::Media { + path: "clip.mkv".into(), + looping: false, + restart: false, + }; + collection.sources.insert("clip".into(), clip); + let mut sources = Sources::new(context); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + + // The first picture is due at once, so the clock stays put while + // the encoder opens. + let listed = until(|| { + let listed = catalog.snapshot().ext.studio.sources.remove("clip"); + listed.filter(|listed| !listed.video.renditions.is_empty()) + }) + .await; + let feed = sources.get("clip").unwrap(); + assert_eq!(feed.track(), Some(Status::Live)); + let (name, config) = listed.video.renditions.iter().next().unwrap(); + let track = consumer + .track(hang::catalog::Reference::track(config, name)) + .unwrap() + .subscribe( + moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(None)), + ) + .await + .unwrap(); + let container = moq_mux::catalog::hang::Container::try_from(config).unwrap(); + let mut track = moq_mux::container::Consumer::new(track, container); + + // Half way, it starts over. + let begin = kio::clock::now(); + until_time(begin + std::time::Duration::from_millis(450)).await; + feed.play().unwrap(); + until_time(begin + std::time::Duration::from_millis(900)).await; + let now = kio::Waiter::noop(); + let mut stamps = Vec::new(); + until(|| { + while let std::task::Poll::Ready(read) = track.poll_read(&now) { + if let moq_mux::container::Read::Frame { frame, .. } = read.unwrap().expect("open") { + stamps.push(frame.timestamp); + } + } + // Five pictures of each play. + (stamps.len() >= 10).then_some(()) + }) + .await; + assert!( + stamps.is_sorted() && stamps.windows(2).all(|pair| pair[0] < pair[1]), + "{stamps:?}" + ); + }) + .await; + } } diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 96ad01c..d68f1e1 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -9,7 +9,7 @@ //! publishes it as its own track in the studio's local broadcast, web pages //! aside; see [`crate::publish`]. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::future::Future; use std::path::{Path, PathBuf}; use std::pin::Pin; @@ -26,6 +26,7 @@ use crate::Error; use crate::browser::Page; use crate::collection::{Collection, Source}; use crate::lut::Cube; +use crate::media::Clip; use crate::publish::{Claim, Listing, Publish, Still}; use crate::replay::{MIN_SPEED, Take, Target}; @@ -129,6 +130,9 @@ pub enum Status { /// broadcast to come back; every other source stays stopped until edited /// or [restarted](Sources::restart). Failed(String), + /// Played to its end: a media file that does not loop, holding its last + /// picture, silent, until [played](Feed::play) again. + Ended, } /// What every source shares. @@ -155,6 +159,8 @@ pub struct Sources { context: Context, feeds: BTreeMap, mixes: BTreeMap, + /// The sources on program, as last set. + on_program: BTreeSet, } impl Sources { @@ -164,9 +170,28 @@ impl Sources { context, feeds: BTreeMap::new(), mixes: BTreeMap::new(), + on_program: BTreeSet::new(), } } + /// Note the sources on program now, by name, such as those a scene on air + /// shows. A media source set to restart plays from its start as it goes on + /// program. + pub fn set_on_program(&mut self, on_program: &BTreeSet) { + if *on_program == self.on_program { + return; + } + for name in on_program.difference(&self.on_program) { + let Some(feed) = self.feeds.get(name) else { continue }; + if matches!(feed.source, Source::Media { restart: true, .. }) + && let Err(err) = feed.play() + { + tracing::warn!(source = %name, %err, "cannot restart the source"); + } + } + self.on_program.clone_from(on_program); + } + /// What every source shares. pub fn context(&self) -> &Context { &self.context @@ -288,6 +313,8 @@ pub struct Feed { played: Playback, /// A replay source's next replay. plays: Option>, + /// Marked changed to play a media source from its start. + starts: Option>, } impl Feed { @@ -322,11 +349,25 @@ impl Feed { } let play = Play { take, speed }; // Fails once the driver is gone, which leaves the reason in the status. - plays.send(Arc::new(Mutex::new(Some(play)))).map_err(|_| { - Error::Stopped(match self.status() { - Status::Failed(reason) => reason, - status => format!("{status:?}"), - }) + plays.send(Arc::new(Mutex::new(Some(play)))).map_err(|_| self.stopped()) + } + + /// Play a media source from its start, now, cutting off wherever it is. + /// One that ended plays again, and one that loops loops from there. + /// + /// Refused by any other kind of source, and by a media source that stopped. + pub fn play(&self) -> Result<(), Error> { + let Some(starts) = &self.starts else { + return Err(Error::Unsupported("only a media source plays from its start".into())); + }; + starts.send(()).map_err(|_| self.stopped()) + } + + /// Why the driver is gone, from the reason it left in the status. + fn stopped(&self) -> Error { + Error::Stopped(match self.status() { + Status::Failed(reason) => reason, + status => format!("{status:?}"), }) } @@ -405,6 +446,7 @@ pub struct Driver { track: Option, played: Playback, plays: Option>, + starts: Option>, } /// Where a source's own track goes, its claim on the local catalog's listing @@ -517,6 +559,11 @@ impl Driver { self.show(Picture::Overlay); std::future::pending().await } + Source::Media { path, looping, .. } => { + let starts = self.starts.clone().expect("a media source has starts"); + self.media(&self.context.dir.join(path), *looping, starts, listing) + .await + } Source::Broadcast { path } => self.broadcast(path).await, Source::Browser { url, width, height } => self.browser(url, *width, *height).await, Source::Replay => { @@ -564,6 +611,7 @@ impl Driver { Source::SystemAudio => return self.listen(Audio::System).await, Source::Color { .. } | Source::Image { .. } + | Source::Media { .. } | Source::Broadcast { .. } | Source::Browser { .. } | Source::Overlay { .. } @@ -646,7 +694,6 @@ impl Driver { /// Fail the source's own track with `err`, ending it, while the source /// keeps drawing. - #[cfg(feature = "capture")] fn fail_encoding(&self, track: &mut Option>, err: Error) { tracing::warn!(source = %self.name, %err, "cannot encode the source"); self.fail_track(Status::Failed(err.to_string())); @@ -840,7 +887,7 @@ impl Driver { let Some(strip) = self.strip.clone() else { return Ok(()); }; - let bus = self.context.bus.config().clone(); + let bus = self.context.bus.config(); let mut options = moq_audio::decode::Options::new(); options.start = moq_audio::decode::Start::Latest; options.output.format = moq_audio::Format::F32; @@ -848,9 +895,7 @@ impl Driver { options.output.layout = Some(bus.layout); let mut consumer = moq_audio::decode::Consumer::new(broadcast, &config, name, options).await?; - let mut input = moq_audio::encode::Input::new(bus.sample_rate, bus.layout); - input.format = moq_audio::Format::F32; - let mut channel = self.context.bus.channel(input, strip)?; + let mut channel = self.channel(strip)?; // Gaps and encoder boundaries need nothing. Like a picture, which shows // on arrival, the bus plays a source at its live edge on the program // clock: a stall plays as silence while it lasts, and a burst after it @@ -921,13 +966,8 @@ impl Driver { let Some((name, config)) = &take.video else { return Ok(()); }; - let mut options = moq_video::decode::Options::new(); - options.decoder.output = self.context.output; - options.delay = REPLAY; - let mut consumer = moq_video::decode::Consumer::new(&broadcast, config, name, options).await?; - // Opened at the first picture, which says its size. - let mut track: Option = None; - let mut opened = false; + let mut consumer = moq_video::decode::Consumer::new(&broadcast, config, name, self.pictures()).await?; + let mut own = Own::new(listing, begin); while let Some(read) = consumer.read().await? { match read { // The group in progress at the start decodes from its @@ -937,26 +977,7 @@ impl Driver { moq_video::decode::Read::Frame(frame) if frame.timestamp >= take.end => break, moq_video::decode::Read::Frame(mut frame) => { kio::clock::sleep_until(at(frame.timestamp)).await; - if let Some(listing) = listing.filter(|_| !std::mem::replace(&mut opened, true)) { - let framerate = config - .framerate - .filter(|rate| rate.is_finite() && *rate >= 1.0) - .and_then(|rate| moq_video::FrameRate::new(rate.round() as u32, 1).ok()) - .unwrap_or_else(|| moq_video::FrameRate::new(30, 1).expect("a valid frame rate")); - let video = - crate::publish::Video::new(listing, frame.size(), framerate, frame.surface.color()); - track = self.publish(video.await); - } - if let Some(video) = &mut track { - // Stamped when it shows, on a timeline of the replay's own. - frame.timestamp = moq_net::Timestamp::from_micros(begin.elapsed().as_micros() as u64) - .map_err(moq_video::Error::from)?; - if let Err(err) = video.encode(&mut frame) { - tracing::warn!(source = %self.name, %err, "cannot encode the replay"); - self.fail_track(Status::Failed(err.to_string())); - track.take().expect("encoding").abort(&err); - } - } + self.encode(&mut own, &mut frame, config).await?; self.show(Picture::Frame(frame)); } // A stretch the broadcast was offline, paused, or lost @@ -966,12 +987,10 @@ impl Driver { | moq_video::decode::Read::End { end: timestamp } => { kio::clock::sleep_until(at(timestamp.clamp(take.start, take.end))).await; *self.picture.lock().unwrap() = Some(clear()); - if let Some(video) = &mut track + if let Some(video) = &mut own.track && let Err(err) = video.discontinuity() { - tracing::warn!(source = %self.name, %err, "cannot mark a gap in the replay"); - self.fail_track(Status::Failed(err.to_string())); - track.take().expect("encoding").abort(&err); + self.fail_encoding(&mut own.track, err); } } } @@ -984,16 +1003,9 @@ impl Driver { let (Some((name, config)), Some(strip), true) = (&take.audio, &self.strip, play.speed == 1.0) else { return Ok(()); }; - let bus = self.context.bus.config().clone(); - let mut options = moq_audio::decode::Options::new(); - options.delay = REPLAY; - options.output.format = moq_audio::Format::F32; - options.output.sample_rate = Some(bus.sample_rate); - options.output.layout = Some(bus.layout); - let mut consumer = moq_audio::decode::Consumer::new(&broadcast, config, name.clone(), options).await?; - let mut input = moq_audio::encode::Input::new(bus.sample_rate, bus.layout); - input.format = moq_audio::Format::F32; - let mut channel = self.context.bus.channel(input, strip.clone())?; + let mut consumer = + moq_audio::decode::Consumer::new(&broadcast, config, name.clone(), self.sounds()).await?; + let mut channel = self.channel(strip.clone())?; while let Some(read) = consumer.read().await? { match read { moq_audio::decode::Read::Frame(frame) if frame.timestamp < take.start => {} @@ -1023,10 +1035,250 @@ impl Driver { () = kio::clock::sleep_until(at(take.end)) => Ok(()), } } + + /// Play the media file at `path` from its start, and again each time + /// `starts` changes, cutting off the play under way. At its end it starts + /// over when `looping`, or holds its last picture, silent, until the next. + /// + /// With `listing`, the pictures are also encoded onto the source's own + /// track as they show, while it is used, on one timeline across plays. + async fn media( + &self, + path: &Path, + looping: bool, + mut starts: watch::Receiver<()>, + listing: Option<&Listing<'_>>, + ) -> Result<(), Error> { + let clip = Clip::open(path)?; + // One channel for every play, since a closed one drops what it has not + // played yet. + let mut sound = match (&clip.audio, &self.strip) { + (Some(_), Some(strip)) => Some(self.channel(strip.clone())?), + _ => None, + }; + let mut own = Own::new(listing, kio::clock::now()); + loop { + let restarted = tokio::select! { + result = self.play_clip(&clip, sound.as_mut(), &mut own) => { + result?; + false + } + changed = starts.changed() => { + // The feed is gone, which stops this driver anyway. + if changed.is_err() { + return std::future::pending().await; + } + true + } + }; + if restarted || looping { + continue; + } + set(&self.status, Status::Ended); + if starts.changed().await.is_err() { + return std::future::pending().await; + } + } + } + + /// Play `clip` once, each picture and sound when its moment comes, the + /// first at once. Ends once the last has played for as long as it lasts: + /// a sound for its samples, and a picture for as long as the one before it. + async fn play_clip( + &self, + clip: &Clip, + mut sound: Option<&mut moq_audio::bus::Channel>, + own: &mut Own<'_, '_>, + ) -> Result<(), Error> { + set(&self.status, Status::Live); + let broadcast = clip.consume(); + let mut video = match &clip.video { + Some((name, config)) => { + let consumer = + moq_video::decode::Consumer::new(&broadcast, config, name.clone(), self.pictures()).await?; + Some((consumer, config)) + } + None => None, + }; + let mut audio = match (&clip.audio, &sound) { + (Some((name, config)), Some(_)) => { + Some(moq_audio::decode::Consumer::new(&broadcast, config, name.clone(), self.sounds()).await?) + } + _ => None, + }; + + // The first of each is decoded before the clock starts, so the earlier + // of them is due at once. + let mut picture = match &mut video { + Some((consumer, _)) => next_picture(consumer).await?, + None => None, + }; + let mut samples = match &mut audio { + Some(consumer) => next_samples(consumer).await?, + None => None, + }; + let first = [ + picture.as_ref().map(|frame| Duration::from(frame.timestamp)), + samples.as_ref().map(|frame| Duration::from(frame.timestamp)), + ]; + let Some(start) = first.into_iter().flatten().min() else { + return Ok(()); + }; + let begin = kio::clock::now(); + let at = |timestamp: moq_net::Timestamp| begin + Duration::from(timestamp).saturating_sub(start); + let end = std::cell::Cell::new(begin); + let last = |until: Instant| end.set(end.get().max(until)); + + let pictures = async { + let Some((consumer, config)) = &mut video else { + return Ok(()); + }; + let mut previous = None; + while let Some(mut frame) = picture.take() { + let due = at(frame.timestamp); + kio::clock::sleep_until(due).await; + last(due + previous.map_or(Duration::ZERO, |previous| due - previous)); + previous = Some(due); + self.encode(own, &mut frame, config).await?; + self.show(Picture::Frame(frame)); + picture = next_picture(consumer).await?; + } + Ok::<_, Error>(()) + }; + let sounds = async { + let (Some(consumer), Some(channel)) = (&mut audio, &mut sound) else { + return Ok(()); + }; + let bus = self.context.bus.config(); + let bytes = std::mem::size_of::() * bus.layout.channels() as usize; + while let Some(frame) = samples.take() { + let due = at(frame.timestamp); + kio::clock::sleep_until(due).await; + let length = (frame.data.len() / bytes) as u64 * 1_000_000_000 / u64::from(bus.sample_rate); + last(due + Duration::from_nanos(length)); + channel.write(&frame.data)?; + samples = next_samples(consumer).await?; + } + Ok::<_, Error>(()) + }; + tokio::try_join!(pictures, sounds)?; + kio::clock::sleep_until(end.get()).await; + Ok(()) + } + + /// Encode a played picture onto the source's own track, opened at the + /// first picture, which says its size, of the rendition `config` + /// describes. Stamped when it shows, on the timeline `own` keeps. + async fn encode( + &self, + own: &mut Own<'_, '_>, + frame: &mut moq_video::Frame, + config: &VideoConfig, + ) -> Result<(), Error> { + if let Some(listing) = own.listing.filter(|_| !own.opened) { + let video = crate::publish::Video::new(listing, frame.size(), framerate(config), frame.surface.color()); + own.track = self.publish(video.await); + // Only once it has, so a play cut off while it opens leaves the + // next picture to open it. + own.opened = true; + } + if let Some(video) = &mut own.track { + frame.timestamp = moq_net::Timestamp::from_micros(own.origin.elapsed().as_micros() as u64) + .map_err(moq_video::Error::from)?; + if let Err(err) = video.encode(frame) { + self.fail_encoding(&mut own.track, err); + } + } + Ok(()) + } + + /// How a replay or media source decodes pictures: from the first group, + /// for the compositor's device. + fn pictures(&self) -> moq_video::decode::Options { + let mut options = moq_video::decode::Options::new(); + options.decoder.output = self.context.output; + options.delay = REPLAY; + options + } + + /// How a replay or media source decodes sound: from the first group, as + /// the bus mixes it. + fn sounds(&self) -> moq_audio::decode::Options { + let bus = self.context.bus.config(); + let mut options = moq_audio::decode::Options::new(); + options.delay = REPLAY; + options.output.format = moq_audio::Format::F32; + options.output.sample_rate = Some(bus.sample_rate); + options.output.layout = Some(bus.layout); + options + } + + /// A channel into the bus for decoded sound, as [`sounds`](Self::sounds) + /// and broadcasts decode it, through `strip`. + fn channel(&self, strip: Strip) -> Result { + let bus = self.context.bus.config(); + let mut input = moq_audio::encode::Input::new(bus.sample_rate, bus.layout); + input.format = moq_audio::Format::F32; + Ok(self.context.bus.channel(input, strip)?) + } +} + +/// A played source's own track, opened at its first picture, and the timeline +/// its pictures are stamped on: from `origin`, when they show. +struct Own<'a, 'b> { + listing: Option<&'a Listing<'b>>, + track: Option>, + /// Whether the track was opened, or failed to, so it is tried once. + opened: bool, + origin: Instant, +} + +impl<'a, 'b> Own<'a, 'b> { + fn new(listing: Option<&'a Listing<'b>>, origin: Instant) -> Self { + Self { + listing, + track: None, + opened: false, + origin, + } + } } -/// How a replay reads its take: from the first group, where the take's tracks -/// declare they start, skipping nothing as late, since a take is history. +/// The next picture `consumer` decodes, passing over gaps, or `None` at the +/// end of its track. +async fn next_picture(consumer: &mut moq_video::decode::Consumer) -> Result, Error> { + while let Some(read) = consumer.read().await? { + if let moq_video::decode::Read::Frame(frame) = read { + return Ok(Some(frame)); + } + } + Ok(None) +} + +/// The next sound `consumer` decodes, passing over gaps, or `None` at the end +/// of its track. +async fn next_samples(consumer: &mut moq_audio::decode::Consumer) -> Result, Error> { + while let Some(read) = consumer.read().await? { + if let moq_audio::decode::Read::Frame(frame) = read { + return Ok(Some(frame)); + } + } + Ok(None) +} + +/// The frame rate a rendition's own track is encoded at: its catalog's, or 30 +/// fps when it names none. +fn framerate(config: &VideoConfig) -> moq_video::FrameRate { + config + .framerate + .filter(|rate| rate.is_finite() && *rate >= 1.0) + .and_then(|rate| moq_video::FrameRate::new(rate.round() as u32, 1).ok()) + .unwrap_or_else(|| moq_video::FrameRate::new(30, 1).expect("a valid frame rate")) +} + +/// How a replay or media source reads its broadcast: from the first group, +/// where its tracks declare they start, skipping nothing as late, since all of +/// it is history. const REPLAY: moq_net::track::Delay = moq_net::track::Delay { min: Some(Duration::ZERO), max: None, @@ -1038,7 +1290,7 @@ const REPLAY: moq_net::track::Delay = moq_net::track::Delay { /// One served from another broadcast would need that broadcast's own /// subscription, which a source does not open, so a catalog offering only /// those is refused rather than shown as black or played as silence. -fn pick<'a, C: Reference + Clone + 'a>( +pub(crate) fn pick<'a, C: Reference + Clone + 'a>( renditions: impl Iterator, ) -> Result, Error> { let mut foreign = None; @@ -1083,6 +1335,13 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { } _ => (None, None), }; + let (starts, started) = match source { + Source::Media { .. } => { + let (starts, started) = watch::channel(()); + (Some(starts), Some(started)) + } + _ => (None, None), + }; let feed = Feed { source: source.clone(), @@ -1094,6 +1353,7 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { track: track.as_ref().map(|track| track.status.consume()), played: played.clone(), plays, + starts, }; let driver = Driver { name: name.to_string(), @@ -1107,6 +1367,7 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { track, played, plays: replays, + starts: started, }; (feed, driver) } @@ -1876,3 +2137,206 @@ mod tests { .await; } } + +#[cfg(test)] +mod media_tests { + use super::test::{context, local, start, until, until_time}; + use super::*; + use crate::media::tests::{CLIP, TONE}; + + /// A collection of media sources, each named for its file in `dir`, which + /// is written with its bytes, and whether it loops and restarts. + fn media(dir: &Path, files: &[(&str, &[u8], bool, bool)]) -> Collection { + let mut collection = Collection::new(); + for (name, bytes, looping, restart) in files { + std::fs::write(dir.join(name), bytes).unwrap(); + let source = Source::Media { + path: name.into(), + looping: *looping, + restart: *restart, + }; + collection.sources.insert(name.to_string(), source); + } + collection + } + + /// The next picture `feed` shows, and when it showed. + async fn shown(feed: &Feed) -> (moq_net::Timestamp, Instant) { + let timestamp = until(|| match feed.take() { + Some(Picture::Frame(frame)) => Some(frame.timestamp), + Some(_) => panic!("not a decoded picture"), + None => None, + }) + .await; + (timestamp, kio::clock::now()) + } + + /// Wait until `name` reads as `status`, without moving the clock. + async fn until_status(sources: &Sources, name: &str, status: Status) { + until(|| (sources.get(name).unwrap().status() == status).then_some(())).await; + } + + /// A file shows each picture at its moment, the first at once. Past its + /// end it holds the last picture, reads as ended, and plays again from + /// its start when asked. + #[tokio::test(start_paused = true)] + async fn a_media_file_plays_once_and_holds_its_end() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let collection = media(dir.path(), &[("clip.mkv", CLIP, false, false)]); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let feed = sources.get("clip.mkv").unwrap(); + + let (first, begin) = shown(feed).await; + assert_eq!(feed.status(), Status::Live); + for index in 1..10u64 { + let into = Duration::from_millis(index * 100); + // Within the timer wheel's millisecond of when the test saw the first. + until_time(begin + into - Duration::from_millis(2)).await; + assert!(feed.take().is_none(), "picture {index} before its moment"); + until_time(begin + into).await; + let (timestamp, _) = shown(feed).await; + assert_eq!(Duration::from(timestamp) - Duration::from(first), into); + } + + // The last picture lasts as long as the one before it, and the + // sound runs a little past it. + until_time(begin + Duration::from_millis(990)).await; + assert_eq!(feed.status(), Status::Live); + until_time(begin + Duration::from_millis(1_100)).await; + until_status(&sources, "clip.mkv", Status::Ended).await; + assert!(feed.take().is_none(), "the last picture holds"); + + feed.play().unwrap(); + let (again, _) = shown(feed).await; + assert_eq!(again, first, "from the start"); + assert_eq!(feed.status(), Status::Live); + }) + .await; + } + + /// A looping file starts over at its end, never reading as ended. + #[tokio::test(start_paused = true)] + async fn a_looping_media_file_starts_over() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let collection = media(dir.path(), &[("clip.mkv", CLIP, true, false)]); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let feed = sources.get("clip.mkv").unwrap(); + + let (first, begin) = shown(feed).await; + until_time(begin + Duration::from_millis(950)).await; + let (last, _) = shown(feed).await; + assert_eq!(Duration::from(last) - Duration::from(first), Duration::from_millis(900)); + until_time(begin + Duration::from_millis(998)).await; + assert!(feed.take().is_none(), "the start again before the end"); + until_time(begin + Duration::from_millis(1_100)).await; + let (again, _) = shown(feed).await; + assert_eq!(again, first); + assert_eq!(feed.status(), Status::Live); + }) + .await; + } + + /// A file set to restart plays from its start as it goes on program, not + /// again while it stays on, and one not set to restart never does. + #[tokio::test(start_paused = true)] + async fn a_media_file_restarts_going_on_program() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let collection = media( + dir.path(), + &[("intro.mkv", CLIP, false, true), ("still.mkv", CLIP, false, false)], + ); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let (first, begin) = shown(sources.get("intro.mkv").unwrap()).await; + until_time(begin + Duration::from_millis(1_100)).await; + until_status(&sources, "intro.mkv", Status::Ended).await; + until_status(&sources, "still.mkv", Status::Ended).await; + for name in ["intro.mkv", "still.mkv"] { + sources.get(name).unwrap().take(); + } + + let both: BTreeSet = ["intro.mkv".into(), "still.mkv".into()].into(); + sources.set_on_program(&both); + let (again, begin) = shown(sources.get("intro.mkv").unwrap()).await; + assert_eq!(again, first); + assert_eq!(sources.get("still.mkv").unwrap().status(), Status::Ended); + + until_time(begin + Duration::from_millis(1_100)).await; + until_status(&sources, "intro.mkv", Status::Ended).await; + sources.set_on_program(&both); + assert_eq!( + sources.get("intro.mkv").unwrap().status(), + Status::Ended, + "still on program" + ); + + sources.set_on_program(&BTreeSet::new()); + sources.set_on_program(&both); + until_status(&sources, "intro.mkv", Status::Live).await; + assert!(sources.get("still.mkv").unwrap().take().is_none()); + }) + .await; + } + + /// A file of sound alone shows nothing, and goes silent at its end. + #[tokio::test(start_paused = true)] + async fn an_audio_file_ends_silent() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let collection = media(dir.path(), &[("tone.webm", TONE, false, false)]); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + assert_eq!(sources.get("tone.webm").unwrap().status(), Status::Live); + until_time(kio::clock::now() + Duration::from_millis(600)).await; + until_status(&sources, "tone.webm", Status::Ended).await; + assert!(sources.get("tone.webm").unwrap().take().is_none()); + }) + .await; + } + + /// A file the studio cannot read fails its source, saying why, and only a + /// media source plays from its start. + #[tokio::test] + async fn a_media_file_it_cannot_read_fails() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let mut collection = media(dir.path(), &[("notes.txt", b"not a video", false, false)]); + let red = Source::Color { + color: crate::Rgba([255, 0, 0, 255]), + }; + collection.sources.insert("red".into(), red); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + + let feed = sources.get("notes.txt").unwrap(); + let reason = until(|| match feed.status() { + Status::Failed(reason) => Some(reason), + _ => None, + }) + .await; + assert!( + reason.contains("notes.txt") && reason.contains("not a container"), + "{reason}" + ); + assert!(matches!(feed.play(), Err(Error::Stopped(_)))); + let red = sources.get("red").unwrap(); + assert!(matches!(red.play(), Err(Error::Unsupported(_)))); + }) + .await; + } +} diff --git a/rs/moq-scene/src/test_data/clip.mkv b/rs/moq-scene/src/test_data/clip.mkv new file mode 100644 index 0000000000000000000000000000000000000000..3e30bd429c968abcd231406b2c6c883b7bd7b101 GIT binary patch literal 13211 zcmch8by!sUxA&f5fT4$yh9Ly$4#}ZGQA$7s=?3W*m|&z+P(maPQbI!Mlu|(yC8R__ zloq7f@8CJ!^SrjmaB@EucF38zJ{r62PE*6!7`OWiGSkyr~KGA4v{psH#;XYy^{=Kn(%LK{A0zCD( zQGs5Lu~;BF#oXqSk+!akxW1{OxVpJ!6c)m}5QQh+nFr=ltFfpiEGh>JHLJ#(y}&}u z*08`IMWF!TQ1k?%(LoN*157T2`vi=?Lkj?i0NClpL|HEz2?2;*hoXj%e?+nUh=u+m z3iSULC0gCa(8$cxye!H=tiZe(59_G|qOj2Uw0tbIJ{Gm)^IHRZM=au04u_(2L!Ur@ zJtsRy2vh-rY5qNXdOWC-c6`{IGp7FjZyY)j9RAVVr9WG{I7a)jrlFR$iB?P^!GsP- zPdEDzDLEOJ{zC2ISS>RX^(*>@m$hQjB_tG2|0KmE{=9(yBnbVV2znC)`+pHs_=_aK zrP&Z@j%mk#;5(LH!UDykH~_$e#~eMsh>tp!as>oSo|IK@@3}sb!ro%r$bGwM%sMTf zs|qx2Kgslvav`dy_h8+l4HbG6UA|S_LCQZvhr4n1W^>8$>>b7$=gc=jZ&1a0x>y2H z0DxMP5QE_LPuk0^ixYZ{_e92#6TEBdr@A9Wkq+K|PDn{PS)`w|l#~?GURqX8Le`Ej zp(jRIAb#p1qB2N=e-KtWIyn#~G`)R7J)N8bkWvy7(xOrlQnG|W*MI;YMRD=q;9#*J zH%BLLPdhI$Z$B6DQ!Zkz0XIDfYrK5|+`PT~6_E~h_I3^`5=cKMXBBCrqm#X-w}Xd@ zq$0uUNINe(&rp9S6^Rf+KSWYe5_!`}#nmYU=^toMXvrge{6keFPp>w9jw+{zq$okS zAaA;bI62y!Qas&Z<7emP;-n%ehjeiD^S)_kL)a>b4DfUE^mOwlv=l-V9327(A~^Wo zBwQT{z7*!|<)k7dDMH}rZ08?fkfpIxYpgZE7z zyWbKKOcdZ}=jKHqL0IT#ce=*e&+ev^KcN-uBe*xYpUB9QiX zaPo3;2nkvL`JXLo86KPP`z!VW(N6$d|OwZ6}j_-0sjCe9~Bv-n-9ULgaahloIu;omvCOE)<5+y;%TQ?yg(0+FbRzN z34~nE7V^XB1CV){oiOYN=eM0FYHhB{rRLY^crJw_0CoU@(4q!N zv>*Va&6%O1YyP$fc(0!{;F*3S^1LayZ7ZBLnpBEmONeKPl@YU)p3HQOuS>OB6|DGVLhcZCk0%RvEy@!7ROtdkVRpCOuymoP;_C^z)w9D zYm!QPb76Leu^%MQX!$q?^4D{DrP2ob6>%5O$xd%=5LN7=NuXF#^t7N=Kwd+1;g6LIbHYmSjaTBmH9wEmAr zbBW*HxTue*KkC)vzpFN7<<9&mUp~d+=@ZIG-K_MnWxzw4&j*c0uM#D{x!c(cwvoWQ z<^f0t;2%A@kt2@k7njm7)TAmM8KAN*2-5}InSrzF@KQF@&zw$t>PB04wWX1)#Tegu zRpK@0=S2d%q8LE0II~Xf2>`s;pk46|faxS0T@4wAP_0$oJo`D1p~DvFEPw&$630oh zGJuJVkpVy{5fj*h8g%T`wx;rs3-7q?gV9!;yTMLn++9ge+T8B{3Tz2(;bdcGtk2*G zhGDQ;Fa%BXT`5K)?mRh|-|7H9@N;Va^;FokI#b1tx;I}OsqRinu+YpGewe4v+9}=2 z;uCdWIShZ9JXRtFLOVmR1$R;TKyU>R45f5?e_(3E0A*~IM&CMOdbWtV7Vr7M{?`5m zFiaX7ldKqjR3AJam9w5_u+%;ap#rbJ-s1gXgyQw@LP$``{G6=Lc*d9R+2|XdM$*oV z;c2oq`NVm?c=d;d$$IJ{u|=~cCD8(jf;rUj^6TsE*nUcwlL!r*+Gx%eKu{n7W)efs zVxRY~q`fxc$mbo2YyrMH*~=B&N__p2b9^)RsWjT8{5WCtuH5*yQDp!K=7UWsr* zJ^%uMQ%3_N82`r?_u+clB6wTai(i8Vuev;oQUSc`H30Brd+LXvB1>PxXM&C+U;rs<4pVK*D;W)N z!v+CjG4=-E1iTHIxVeCG4+VE6<_6-VEbg7Scqhwk7rr|&>ppPe9+*(Op2C$hZ2D0H z_uxALfs>=M!1JcZH%Xr?BK-S%eL@bG`cu@o4*0?PzP(3< z3adQ?XlMO8(%8}6p|T%$b`NZNP>Ls_ce&e_g(H3*g>}z46 zy|Z+;9D@?=%f_Y>&Waeqq)^<00~iU}2`JS)SS^<$t@l3q>7CCSX3|ruwF8u_TVfx$ z8pIa3kb)+dw>kRnwx0Gg%+2gp`cG({i+_x_k&eii-k91b?#2OvnDAJrBgEmkdumND z_Lr{nE7{PuhdN(23W((L7`+2#@Nco#encv>nr|g7Kfd*)YBjDGn34H%RkHVMBLu%O zSlosKRQ@4i;!@>(Z^qQW&&1A?E~iUZfq6ReQDZ;kI6dIf#e|)N3>yQh+lo~(@w;Cs zuO2{PG{s#w!1(X|Po{Fu6O)LwtnuNut-Llr9&2<+#wBQ6Ouk>E`03L1PfFGIhBUvW zA=@8c@5aRpcPiOevg~T$2a7v#fYU!R)NNEqU1l}ydAm>(;TNvZUL5^aK{66uKEWC$ zyvJbYy?h1o_M6VYajA|+R+YmjyM&p8Qp;Ikv_xY2d!z(rNTIxsva9DhSf(4MZf=N2D=V$0%FPjhecx$=sXm1+ zXPz)L@SG?OG781Cp0RLG#{Y~BXDHvUDJ&GqoK?V(2-mI370#LdwE8kUG*7M1{TW(z z`D}ks`fT^iqutBghmw9?qRK;+_8N*^T+zMi3F_zSAQU`sodGiR2s`j#EnIez0I`_T z7h8MZW^_2ChYiCaN_+-i=OA8H;yx+6!D3tc>!EWAj253f3`&|7x=uO^vGj`C`gX`yb zf(eEtY^US^T*$sSo{E_R#xe_vQHt-3y8?PPf>F zT&7-)={ZDD({>pS@JAojL-MGeWdL67D-A!6GgdeV9zRY~+>Zm2{y0i}tF9L_Gkpr1 zG6%1Gmy}UeYg^JtQrOpROg3rV*vq7?{~Nbw>;YQ*!k- zhhEVSrS|5*n9%*mA4O6o)&PA&|Ao44PA@&h3%(nDF>K~3*FVR6;Z=4SEm8cK*SA47 zdAuq7ePi?Tj#4b-ot|6qxbhmeU)B6XThrk3XsR zf0fT8bO+Xi&mdqNymRW&x4DQpp{bP$-03xQ*@}IoFrKJhNWukknXXiN?A-ghzG@;* zw}7_x*=@443BB{ZE_1XZ!DA+Q75YwLPump)#!-q47lQ5ydF@6*rbGq3PdfN$>Mcjq zFp}x<_IGTMD0UbzK3c*i>QRE7jm4r3roQ~{7&x1KD0!kPp06#Szr{-g)agePIVU)i zjcr|N`mAq-VI^1L&mOsM=RePo{<>o)viv|qc0>kmk-Uo2XhjK|Wyh>UcBC%cE8-T) zUp+n}l$;Af0y^$ve|nKFiSF*sO*oGn1cD%XS^`CX+n3r)K`lgj!}@UD^KQ1SK+M4E zzI-`-rjD*z-|)({s1NSf+nWtq zFi$OcMP}{XRGzXM+6xR8Kf^&-{@Hq_z4rW75W)cujC|2fX`9Y8>NCX9B+T_>?ZZi_ zZNWf~F<-9$rnnvl5&gYMd6gjr7XkZzX-I#Dof#<_--fI=`gu$Fki;aG?inW2lt{@T z$@I;Y!;!@Y8pY~4Scu4}-``QsvDMu}M_bRTajPYEl{;4lW9xt(rH5^Yk>z8vfjK+I zp6lCr%?BZLDnG|s`O~dSb?2YwADKtORX-f;X(2L1)jw}(Ty9-ZaGmSqx28%*u8e5Z zV!mDI^q5m-We~Nf!fSrwAT|_Go27PS(7T=G_9gRm88&fUr7(QieA+w?2)GLZQ7OA;V(F<0#>l-5H>Obv};o2gtn ztc3AV&(HdfzG7wXr+pJS!R|HUt8!0Yyi&YyxQGutk!V|4Z#q+upiBL%z&d@5Wzxr% z`<4wgC;lPD4Nm?PlO7JAxgT9B==Ugh*u%FI-6p1ophloO>Jo38TQ<|Kv3S{+70i8j zHapcp5nnypCjOy)HX`pRuM~NquZ!6^N+2an@DQnLX?@ha8Z5f4s2CS!{&^Yv-d%b% z$^EeEsJH|NQ9nIXyF=0trxo0I3xmFvvY1zwu86)~teRe1=Yng^zHIt}VsRl3V)pmS zpx+M4xAP`EOMgCYdLv8xMMINwQSnP0#OsZ#z9X4gTVc{+FoVL5b%Q_o!mToA}HvGJd|T3)*8@L-APU>u4L4i92*Lb6@_0vjvO zQ{_?I3wY{w^1+)%i^us2aYK;q1leIfrVRcZYyvSHxC5I8*EQa?V)a8TB=s~7eyZnS zbocB@Er(oS`?(&F^qZF&COZNOIjsU^_C4MuqF>NCmhfyIriJ+5dTi72a4M3J%D|K1 zF@wXeO1TxNA3vX!?qg_9F{ER4a+!^zX4eIzr=odp`M(9Kiy6k8#OpsM@4|tUhcEyc zcU8C0C+5?51F^WFcratDdUG}v|=0|2)I|h#`loY>Ipg?Yi@qvI#iZpYsEZ4}^H&Jnu6$uz+lL17GRqHl zG*yKJ>U0mkla2?kUMgIK_f|Slt#Cwa?f?{{IiAT(OuyuXN`lAf5ly*gT|->Y#o1ol z=WEj5c|6^8^v>YnOj)b`yI*463z1QffDz~O!WMg+a~O(Pe#n*B$PmHCuinjNi!UB_ zP1BtLyit%?=m8u7o-{MEHLl4j%7&mR5uv$CN(TfIlyIZ^i)q!TL~ zbfhj#WGHm!OeXlZHax0M21zs2s%+|nMldN?mK?t8T2bX@zjb)HXv=0&)OdR^shs|d zSPY75@cG0Ys@A$;Y*NnflNfn`6vU2N$_K5RtraigATJ1<#O_}*A+d7wOjxHr&3XBz*49YcEpQSbK3Y<4-gz z?vK5fY~&hy!x?5=Z~pbIn8Y5q2gxp#1G5nXe_p~tUjM}>jp4KGd9Dnu3fby7bnsS7 zOtwiOa(wEfH-X9h#-I>YK%PwWV8FA9g++-0MRMPE)==opA`XrivCCZ;PEH>WI;d8< z^Q7kT8y&F=Ymsby^$SS=%}Hbkon=#b^Y0Ly|6{sM_HuF-_jhLKD_wWo=tWY@xFRS$ zg=8om&@HtePM#gwu2o(FykPG}ZZ#a8D^tR}=@cGORvs0|;W?+$arY|pnQb1ebnWLt zL-hE(C7$UjaTB<~fw^^yKIoOp+G!Q~x^z1}zs&wFLAxD-iiC z2EiQtVX0IP8957So_qY#h};zagALdcSYz^2Dos}(R9XKQy&XMg`OtIIqaZxTJPP<%ljF4^&ZvHMB)5WRO6 zDp^L7&+(M(vs&A#=6FH8yY{!8uw?~1qpI@)<)Z-2X(^0ReEqnZQ;=_(s#m?UVrHR7m_6Ha%prav#3Y z!_{2O($vvn4n6@@JC0SDxQKh-=dE=Gh7Nktp`oLG<9f>&Rod$zGk+3JfJU6fpH7vl zAKd9S>f(+Jqp`p~tqBrlOtE+fgu(~np@)Z{Rbp9Kq6hZVYcE{a4x2dn^x6R-L zRE@bFKRVjjq-bj)9_~kg}OVCe(W?iOS+^ zfot0Qb2kl17IVc1c@6F*h>cqf+l%jaQhw9O9tpfEvsY?+i3i>G+?~1K=6+xKNBwRN z@}G2Jtww^bM&_o;J# zriByh_d!e}^4awX?%FwXZU+vKmxxM9R;=w_U9$;%=fzkB9(Jy{a!W zhz$zBSe78zo z2xzksbu4%oD{6mp1R}5{`AinR_G%GZLzexiqflD`Q3`)Gk+xf&)owFGt1gEH7@(gW z!;Anhde95N6rTT9FGdPR|KhgNbLHUH=54x6a8iLwvL|)9P@dPjguTJTNQM*5+ROUk zRtO2ONP!^TDC&Ah6bf5tst>=RFk`T{((ZPguCH<|EY9!jqIGNS*8}jD%i1272@$TL zo|Ik+Lb*>>PudRGqs0+U#+E3_r$i|$EDsS|yZ&`-@@EbEk4Ihe9DC3&nnGJ-3D#%? zpc$L_*xy!n(7S{jwC#X0o3G<})N9_ebydRv2A}#|Q>M835EUDj#1U zYikd3iQDv_#M?GJLhK^}!ttmwmcmFvCAR z6qIFgZ4W}Xry~4QeW;b_YPIMOHYeyI$sOlD9d0N9a7YO=5jGFE{V5WJ~M%R6$0H6RcBC!Il1C+jD4W`&LF^F|k@iGoz`15@IpHdT5 zH30;;#4PeQtxJUkdR6S`hQ)_p^xKtR@7hsR`lHPnPt>*-)%)M7Im$*Vto&rQnOy3) zTc+ibC!GjzosjB#uK&(WzTtp#f8`ASQ*NTh9u({)BCQ`a+%u97gDX!7nmihIgdCA{ z8iyoM9q||sRk@D_>TBynEgpV|f3(>>Z2HPi0(r=VQ4$8{NSq%|{gs^%lv4hS(|^lO z?j+x)5MxWGHsg%Hl)=E4qg%5nTk-09qejmH=W4wBR`>S&XJ)EvfiV(XmHR5~0suST z{^GZj@Gth5yA#@@{)g?G4*|9P{W*QJ-}%XR9ANave*c-Dod0$1qU@z#7u^~!6W{Ei zd@CHRNrw@ovbX3wYm|Nc_-de@(SD;Kr7wE`yZZL`!Oe1Sv6r6;u3kBp0y;!H%N5t%$XAX|q+xJk4LL`%G zVp`X@h&msTa8iDgXy}3nvZ`tmJyy!4S5hcYfJr3eYr*YaF;gJ%J<&?#CWzF`4&aM} z)vI6+P#F-c)6b4wiI-_Z#lX{`RA!zQQy2;zSDK;ppCC$<2R-C{0(Nvtc`PNDv^o^s zOJ{UBY;m?d0wizWYRxS!l#w8jsQTp_@d&&?OVw?r`jB6TIGZ_YHtX{Cm1)N>`b=H? zm}xxhwixa>?nf5#znqR=(lnQL`C|vVOdLIY6psv&igm!|vCmam!nE#@JDmYMJN?;HhJZaNp<{;HazN8C1dwDq7Q zuNpw_ahz>>{X4V?o|*Vp!2M@ve<((@m|JD1=t@oPY;H}g#0v~Rb=*sIZ0)7Zh#Dgs z4}lEc)76IOJ(=1X814f0?leQdg=mX#Dw6W>GBE0`pwSOhTHHfi5-)V#&6Kc|I)AMc6F#f>u{d*kGU?TUe zJM`DKTR+i;i*Xd%jXk8UP<=N{^wL}#CTFlUjPx7HW`Cbm%1P4n>F}-7cyx2hdU<>1 z2`wB)9~a!$d>n#8vc^P!bCjG_-jyIoWq1JMnWsKd!0e2nej>vri4^UuMJpxOQm@#S z!gI*HX%bn8cgm@c=$~8z>MQV7PmXm6QO}B^BQpU~VRUzK0Ms^`X`=v;S0lm@0d!yT zo#C)2a6AM#E^#USn0^e}IE}3kQ@*02?jVZMS&(^6rZ%>vumbjl&LJEOrG>V)&?^Uf zH_r5}qhydqdFKtFjO)YJzI2$3JCZvUoEFVo&ObuY6F3N>(cl(4$@HrFCkBh`IP-X6 z9iwpYV$~+PLbjICyMEoYc`;>uRZn3v*&PY4mw#-ga>4+Zd*=W+8R0Wa%wtx%oMZqA zbI275)SbTy;k%h!7~gzlspKbDf>}>CaSgF<`*g_18}-`u;7LL5to^kMZIK!1thX`S zv9Va9zO8Sn`h46UywFEL$H@s8w4r*556`saJ-Zdbcah;7zr5ke+t%S%H?MmYPpvGQ z$W+x$x303S|LjbR8kB>Hf{Km5qW52c3JB~kX8#IQK$!p7@!x<7 zh{&mG{|;0LVj;o=sDN@4%J{SM!!#i<-%ioOr{Q$UEz&>V`W-<7kF9_H_^guwS-}c^ zZ)_orp%9?P2We}pB=0Moter(FaAB^5=F1}ZLo|-nI(#o@N@v7=m=>>!qs*l_eVo3D zf~RKjXvIP$EmX1O@bFoh_nAvixHglw(7pf$rF&Zot|i0r9V9BGO>Ii%wmk)oI)uV= zqT@0D!wsS*$7pgsse4?P6=`p8gt{;g#Y3=Kw)-6yOl4;RBB7S|IBaUHX(h&4{nKm2 zJ0?|oof6H@b--V$s&p}`UBCYz%{)V0@s=eQ`cQRQZS7ry0Lar-6(u3fbeA1y`j(ZD z6FG_@2Fq=-$ZL2e&L^kH?wyO4(=t+$+49Dq-&_dvP*_eV+ePQsXvIAb(gE`~h()9A zIqR7<_#GYRi*?hUOwZ9=s?%m3qu(ody}X}Nav7+ePJAOAp(VK5v@>K*3!g7xhqK#L zP{XJtNf0n|5#o+HAcSRCnZx`0yp;-dZEwo?Th!NfO=w0#p5%!?yuK>|@AgAWZA!JW z=&$y3Znk3|k+h0&Wo4{T%}tu`JAVX!L<$r~xtSKNmZPfbeWfqh@QB=r)>)9Ja8Yu7 ze|>L3*0_9?zI#Y)Gri&Y)-KqyhXz<%Md%n~RIKAbz+0W+vw4AZH3rBw{?xGKy%fV` zoB_PL^J-h9+~`8E)tA0q$b6hqe6NOzHwakiEd|kL)tYx92~SK2>Q5N_8*l=-@>hia zJ8a{S$>ni`7|!&Q-2p7>WmXDNDMkMILQ^kUtv5P$6&K+P;gn{5}Pquy*&g zU4=*Vae?4Q#cubycNob~`VkRR(Q|{Ads7GVNKKkp7bDv&SV4ys)|MIKDU^w+)j=`a z4_4aH&`U#L$oMf3?oWvfiwC5oZ-?ceI}R<}UfsX3b9hT`oJcArr#Z=HL-nl(Z&uS2 zm1myb@5_E%hZDg+DXrl*RUE6{kVb1EHu`b39X}W-nFVeG$*Wc9*Q9OdpGJ7Z2P^ef zQ%vq;av>npJPXBsEj&>th~n|t;r-(h32o|%^Lh}hJ@xx}1 z>>95JykvzTkEiL5AV$V4WS3P7x_&GPt#1IosAc%@TX~~kF&+wuWv(onxO`xEAu8}E9w9Jn_svmHLF9N$QDXKM^;bhcu> z5%FtdZ(&rEEie67SA%Uo=B&4fZg`fAX+bN0ez`A54LXa_{_b)o9SxLsCu!!<9UKmxL0022!`74V8U zg{_Bw!tGyyt>=Fs$-e_zul{28@4(jFA3Of9!&U(bTeyRRYy+R_1@R2WE!vWoSZRw` zn$fjckFAX*z0z(!N~m^}s;Xs@{m{Z^?0W&aJvfw?##>KA);HQ`b_d@C^(Uo~GzMIo zp7IHMAOkaIce9t@zJ;$b&GIm0oFgHBPUlq)m2zaWKdLs1O``W+;P70 z000Mw6uK0={p6$p0O9HF>l+LJfR}f$n-fI;)<{18WMBXQ{ruJbje*2}@R+|e|EULu z488;gU_GHhn?TQ>F-iWW_=mUuF!F~chNw!s5ibCi;O*p&g*s?&r$5)FP9SQScN6b` zxSNw#z|S6Z3wC#M{@o{13yAkM!1_3OITLkY(%$YqE>MRl*!!o`zfVEH>DL&iQ-Cv3 zhuDbY)IkAW=!5R-nN;(l*7QRgBy*bz#`&-Qnpza$uFNd{1tco7?n+6dA| zd$~J=$|%VFlKuOg6$BxHB&4Mc)9RoEX8;ygh>juzJ9Gem1V9jPrm6;oXc$21HV`*} zhSX7hSeze3i4!1RC|qD54x+ys#N!-)GZ1Bh`0&5+#4e=c==+xpmh>Mo4OPPb)?hzZ z|K}F`!DIf-3?}eDXS$g{RQ;z+<9vgB{_e&M;+_7=G%$FOX%y~Prhn`H-()%GAj|rH zWLfc)U(oLj{cA~Z07Z2EYpMXF(B+B6cw02@(vN`}jYX7PTfGN<$De#ZkU;+b(NhBeK^q~vZ@T;PJ z*+KRgNS1Rz#}8=U0o{PcIR7Co99k~S^aGT(GEz4IiMN` zRQZ6aA5gUes&_z%k0jCV_qfyn4?WsH{D4Bo_)ody0fkPEpS-{U((@(9j{PCX04*cJT6b^i-2pmQt2NV|}n*VK`?ssZfXym6n!9dpoPSIftTgK@Lz) z5$%TyQHiilVcz>1xJ5x=0 z36eW_1qEQk>>Pc){jfi01Z@#i)b2iz3rH~lOVn`*zw+a!vtW7TN2=mbAv1b9Mv2HT+rSQ&^|yLjD{FB8CeM_ zv_CPpnv{eB)b%6!`AM;%Y6?do0WQ$lPfZT(?gwoult5_DA>UYkC||_g2MExEJK>xZ zFVxK^N`~OQ38Pf7hkZ4EL*oZ`2zAw>FzkBi3Kqt6Bj7UoCn&7jzQ%^z%FGEwnQO7U z7dVr(bCd=@6e{d*Msr=GtvOwrBj@AFm-W{(nSHAb>)|nPTwZkUYqYtkR$828;5!kC z0yqH_N{1F8(}4l7HW#Mqu9@>tQTJ(z`g?H(0P;}%u2Y#qz!}d6MuVtUrIrNK<=2Ja zXHE=SR#Kk#u2SBEsoSKD?3%H{l`Q_tl{02bo*sI(n%xJ%9eYpgwQ!r9HH;j z#y~=O`jhVlYPMuG4i=)Ej>BKc?$QfzKPg=*=9kIt?^BK?05OLF7#!#TQXV?kGl~M} zu>x%*%VYsT4;%sD6Ozt3{9Ae45d@Dmkg-AjmSMX*ULWNtOM>TZIF0`DkNmS zjwsMyy295bvcx30jLNRNU1KaL$cE!krp*i4BxKR*d_B(*p{HQ2<5fP|eIsjb4%jF_TOrNBT)+ue8%AqGTwq@b^U^^>tSQA;vVfLQeSwPcx z^`fpUn*A!lzfqm^o6G$xLj2+cz~E|rgThS!c&$ah;2((4OWm~!9fZ+*tMNYkzJ#g6 z9_TDX07ueB$O>|S(dD5hfJz!6s2kJo)TwJri7Dotj}^X<%|=FJs{8^f3liaMW9c*;eNiN0ReIJ9u*8O-GK;= zj!En=(s^Isr~+lk5BiLRq)h~`aNG_+ zQIqsp6?~?=6IVzLGBG&s1FzC-gFFB#{ql?Im$HO*=1-}V0a_ZL5i7HXUl{C6f+mCa zq7VQ%W|~lE&o7+-@FMyFQVGr`|I~Op3P}qg)ov=@8p1i${;lL!l9HWl=UoLZBrf?O zNxKmu?FLGxvtx30n|S+Q2?_1*-U>Quw&zWLa}I^;>+uWSneSyaaeCrR(Li_Q=^*nW zsc<}iNP}tRl9}*WQ+(@DZ52!LGBK;aY|kQy*~LbO+9S1H(oMQV6N2Og9nTP06NOse z=QDs4?cf5710fUxq-pX3Fsa8J)6-~vZd!mK%d@=Sc^VC*t!e>q&2-7Q7}&_NAClzG zZdMC9wh!gRnSb(DGq_f~`o`d72*vV+kZVIR8dVL;HJPo`t_1RUfHY>{2pUi&DRDaV z%7WV<`5O930HXKvYfPA^#x;O`Dxf2q6aOeo{_Xn4wp}+yd0+e@Z~MGx)c4)cPL3OX z@$fY?0ze^P_dZ1Twy@D(_;j&~fD!L4jLCv>MG9kqKFYx|HH(z^H-n8zV`E*;e$@<=7L1X*jKd4M9gNsjpi`Xp$Vxf4#77x`S<^Y215RGC@^kF-u{Z1nm7nA42mk zY(9UVsfll2rJq?OvGtIpM@Ibjq)4W!wL9hIBKcEF1TxWvMTPQdv+p(^1_x$n4SC!7n^2KWOVRXNs>39TNw1l47Z3$pN8=Qd$5(dSY;3Z_|VA zcIx5q+^HYEJi&yjssh~|OCy=V8TkpvwARlXc0DBY-X&l6?Hta$uX`5!yyU1kC+OzM z^L-@Ml)x6gUa<|%79JnTEdue=P)gov7M(_}hep>GU$qv*h`WRGoBI>PwxYjYkukLe7@PWzH9X?>F;G6{zucS1VUc7#;o8=AomD(AGK@ zpOy-2>U0~&xOyRCSM3$F^Y?+`$}B`h0GPAP`y`W&k>pXZzS53D_jg{BpxieR&d-(h zO(xUdey82~Q8k0ofAUTA5DLM?KdqT?o`;kho>im7n{#KnP`S4f!57y9OFd>G*OkQ> zGyS@uw~oZiJ+Q5PYKomPs$gpWiwV`wP?RF)h5sWg^vEV!rpN(kPxVW(I zeun^EqxFymK{_YJ;es6+!-*h`Pmr>UyPk@*vz)UdG*(?42B!*lr1#Y&OLc_|SNTbR z2EzmrmsA&u;nmX*-y7Nx*eTTn3y02PaWhOg&pOtltG31DhvecdGZtgDS}~&Lg^3H% z9a*z^S9nEA7xxZ{WE6w2z>Z5X-+jnGiEnI7kGc$P2Z11dLJGrp-k;V-Nh4Ht*>-2i z>tdn3P~wxtEyXIvd_8^h-ob@4ac?~GqJ4Lv)Lzgmve)0lC)iGD@M@%WRk_rK#54fi zD%aX}qN|3df{NBnyq4BVnzut4)V>e53g*~W>d)LS-L;5Ds=wLZ)IsHnYrbFAI@vm_ z+&LnPG8?XJ2i}aL`#uTj+lfn4{_Ye6O#H#?Av*D8^ z8)Licn|^OOZljPv>9MS*Ps?o_ed?WcY+BY&g~L{U`s0mu$qXbb+AsMe&n<$RYl1T6 z*TLfNbAal#NOO7?o+a4(#bWxapQih(!&J2eOa+L7>>KwOVrfD~q-}KimHT-Lg2jbd`(MP6cJD^sO2j&ES zn|mS4hKb~@4-14#$U;tialXD-Y1&IIs(hNZGz;sJYS{Z zA22vo@U84*hvFh`*mYbRkA{vqWCk*^fADD~uM+Kz`%|*LOwF067}%X%r;=$o^#R%O z1pWxz3!v^Q)3CE-F^_vUY>LNl7`a569M?w})l+Pi1YMf~nxT03p0jifp zUKuPbKNLkOLPi)-4~q}Gg}NO{wm-8a@KAUC#>B(jS4P(+Z?zh}`XTXXHaZR#IOK9v z)N+%1nn0B#2s@n=9V&e9qi^%Axd+!eXnewx$Kc}BiBba}#FhszR+lXcE`8(YWzOx@ zO3S--OL7?Rs##%}Az!<*^uT@i-EwUd*m2V7>0K2^#*ObQCr)m+F`FuDikeS#%JRux zr`p{uZHjeFe23&MVGiEfv71|+ByABA-PQ*0XC&r6YoRy^NWc4>RhuNjGlVK5O}FbZF%cZg#V=M&*e4W=)q#g=H}gGZTuG`MJ=r(u z!$htpRIS!^kPA*S#GX6s-CVIZb%FVgOaAlV=a|SuQ07s8okEeU!Rv(E7v5``%Z@MB zKcQw{m3YI`Br(f_7B(fkC^CGtdb^KldUB%%H>!Ok5VJ~*{J}j4!D3mb!22RGmh&l5$T9w^mHGTd@#E06&$1^V7QGR#g>^qgZyaDac+35BgXCK8T z4tAsC9hF{3^|8@E~-v?~M&I8s$P8)1q6~eD^mVAFb9urtIn*!71W8{&d*DF^ZPH z>(mp$gxyA13C-PHz^8qo>FZwZ0vEJ%BL@q(pVF%D!^+B-$)U={uh=DHT-(}~K9nB* zag9(&-f;NGScXt4&U30IQ0_!S?z&O=CRh{I~79+3f3a+;C=6Yo$@(Fnk(` z0HS*j*U)#mj=g6=V9G4TAIO>4F&f?63$O~GP$cK@Gl8XYdoR-&CBYreqMO24*^J4H z?kV;DXdXwttVzYECx=NaTPURL-?6?(Lnf z{iH2xLVcp^dwPDqa@KfrV?-wP^wOISqt06|x?D}B4`KviYYY`t%FF=%s9vuCAJg_uLo7voGyN=q>Q;n znLaIY)A0)}t9BAijS5&MHU8H!qPd2oHNQS9xqU0&%w1SeE2n`K=wJt=Y-qV z{9;pBceUG`ryPX(|7CQjz5b{b2;)KqML%e#w$EXC={LaiP?YE9w>SHh_GM4HO$2&` zeojbrkts7a3i0K{fZ-%3D_T6I4c%z`JwkPd%ruGNE+OBHM8z@P?D^@Pp}EUikbmqB zd8qS3>_`ifp^mDAPnW)!p+S;{L0jjn+np`>oLS{xI>A3zDX{)LvcPaAd1ARh@ zmn;7oJoKhTNjFCK%+O3;npxa1>jmlj`f3Lj!o|fUGx20~6_cu$TAELojVzc$IH*KW z<4+WaJx@v9yo)Nnb1n1B7m_X;8WZ2N8RiisyJy=g_wmBC_Pm*GeK~cp%A|DlVJM?9 z>oN^4S4Ah!C&7=$G}<4Gu;n~`O<}-(>C;6v{yFZQET8lIYsh0uCx1xG>I+<{{ zMAxD3n|9mNs&B=0JWpg|dOqS#d`(+lx^&g6RzKHz-~zXShZ;C5RAg$0o3yH`Tyebo ziDY{Ojt_|pW^zVz9RGkAt}0RIQ{D``?Y{rUmrjSzL5;P9NbG@;>+wByh!U z+cmj;&=eQGaJ)8W&tZDy-ueFYD#k++i5Q;#`=b|VS{nvq(u)RfCMp8tV9ZNv#o#4# zD2BwP6Xv`EnGFn6V6&@j*bJv;Cyug`vt||?VRiYg%PWd4FtjUho zpPWph%dSg8OOqKK=2B4jhY zmp{sr%Tq01my8ctZAmONEk}=x@Ass#c%17Op$ROJOXv^0J32cj^+cJ{znwh{?tO)e zD^cQP7lE7G&yxYJljAa`J^x%!;@G!nj^4)EbbxL@I+VfsVO8_*jauQXqGa=UWsX1H zb-|rcEYqAPirPy=j_NYQr}mw(!vkyes-FNK#H*o*rrjgAR0z*IMTb;XUkVlR9Z~DJ zXa&D(UqUZi|GxYbeq_cvp2dpvA-K#%h=^bee(FkGM$Tb@Yo0M*ayobx1Zp4nDg2=7 zBKJ!YNj_h9tm1ZVFY=t{`{LzVL*qy7Me|Iu^8Wkafuw4q5U#%PEShV~ z+~sulJ%4CLuSovRcT5H1UdV*(H|Ud1)<1yya0twH`Uh~G2u^%24_&YVqBEim)V~h_(vT1MVdzSSc&2}d0jd84vz>q0{rTbmON{aEZvZ|U1mK7L zw)q1HcRt{M1L&>?yxRf&&Htu<%Kzm5Kh%R6L+kPf05?O}$WYaRO*KA`KQ_SDq@0YN=>QRS;54aj%006i?bHDCcxcK84g++CYH$2K zPd5t*YcnUyY2+weV=qZ*^my#+USFNl2z>C(4lO;+b7xzIX|kUUnB&N}0Xj)G9D_Pf zAGrR=xQjPBoX#@l_MKo+=1j|LKp65#3Vde=v`H(@FhzfXZBH6pRe6&VR@QRkF8(Hc zK7V0WkX6K|UIZ~}6)FcIpjDCy;-mo#&%FCx zL68W!-Zb4@{|ULGs0W@I$jBypULKpaHvXZ+S9GUd&6c>;_74Ue+V?O#(rF{pluck{ z869V@j~~Je-)19E&BroV9Nt3dQgBQ=YfjC!%<~RArpaz&JYH6qGN9}WB>*RV4IrpV z0Q#UiQgCN`5}L|58-iUlUxMO(2f(00MsEnAuGV*(FxiJrEfqd~K+a(X8`bQoA-R=o z51i2zoc2CNHp~B%Zlts=DGpiHc77jtp^ZIvw#>*#ww&fNxpng{Cv2EC+i(D%+-=MD zBAqgBr+t@ny+pvx^clFBDluX+=peb(N&Q)?a45)1ZnM(<1RuWbz6Wcc-KE~Dw}y|n zD8Dm=x1vc5^bCwO+S`8+W(?+!kuB|SQUfRmfr0}VQk1A?r`*60J>0=QI}|-uLRL7j zLtc3?dW!&DBoh_$J{7O3D~GuyFlUk`08CHOg+n2(wSG;5x_yFHbd zJ-m{o`tTvO$9G7L*JrRYKLA>VpgZNI6YX?7j|Tx5Ok+`&U*t92Qb!>_14qI7^-h4E z2|B!BXaG%xv2$CA4=L7Fb~LcES~=Tj@7kID#%KgZP>`a3w4VTWUqBnVqO)foq(vAic%I!e+P}I)dB0ite0?mA#p04hFDvKSDpLhz@qC;83E|y zB(S0Yf&u&(Fdi=$(L<0yFg`f1a$hy1wRw#p9~@KSk?zi#FIVLEt>A3(G?wGWv-hyS zi3lYF=BQBQ%U8Otk%S>;A2vpwQ<^l|TxfUS%P~~j6O|Nnan*_V_Tw@b;reZp$CM{C`MbbNP$N|h-pE58Hy;ArzS#1m8ngc}SClNM6s+AxX8 zY&ea%m*qHtO3#gMAm=-X5$D4X`0j%poid)EGAddfujJ)0yY95O*zW?;=ObE+ua?V6 zkxAA5@Q=C<9;2suWUhWqP>-~bHEycl$*uCVh1?X*$g+L9V zx0^1+nJ&9(#_u30keK<5Pmc)N$Oj;e6N2djYKHVEfH|f`cmSZ@ON5!>UXOT?U0T}{ z4#slhr{3q7KgL9&*dOjjuY)@_4bN+s#gZ=p!Y#%CW=OXsgv<4U#;z z&_yU1tq!6Ch6*`f7gTbSJ$$$G!g(a2IdiG1J%67b8OxX)(%ZZjib1m{MuF4R+_k(x9zB10nZ< z^m;Y7WiyY<*D(BOEDUweD1wu0Vo~!QlVxGDMT)4NaU^)Wb_HK8U(f8@xMbEmm$|fP zptO?Vfd-eV-Zs-X697Wq5dcX6y@ny&U}q@G0MH1>VzD6o87t`PNg3rS&8I(AeCJ6u z?=B?0L#p3C5&HIAqpkzEUsgQjaOPNBbS}Q&MdDggQVdD&>SuLB0p2%0_+6l5e;*89 zRzD#S&$8+}wHhUGoau<5;;H=?t%Fa!&-z>)Uzj(Qt8JKQUF2B$-kBEHFN64M-|-@t zy`J+H_?%hFzK^68L0Nw`x$>~+lvGkwEA+rkjbduoGmg+H-$jB+W z*ZH4r@~mX6;{5>vM*qAHQb&&M3rJE7_>4aXEJ+JI2&|oYO3hbRZwAt93DJ zoV|28+agy}Il`I;zoR~{@$FTU5Xje68z&{pa*-2w__-jpDEcLV6s)o%dE(>vKcP+aj&$;TqkRl;3>#m zpqU=C*m8LbevKBo8s~0yWw8oV+vqQQ%#Kg&LW16`B9*I(+v~HNv+^cYi;Ry3Bvx{o z?yqiu-J5uTy;Y2XIZ@3v83cSanciEJ$<`5oLX&q*pL{l7Gm?E zcLO$)tdi2BrREC)HU^)9=?m&Dy3kOOBts~6aS2DtlHH?+8EuRl9m3QNfik?v>av$c z?I#)8a;jx_4Ca+37rSVD4rS3a7xYm-p6PS9 zVVS8o-%(k%*c>+MT;vHsVEAHB6v#eoQN-JBm%fZ+s#dgZZQsCKe;Eh&*&p?`Ss|l( zc|dTvdgIZOZ#daN&MpbdmHS35dE?tNXl=SAS7ZAEL|KPTj6EyNOC+CAuZLlE=&!M( zW0Zv<&?&}+npNmLkg-^ce@5PR^mdeIN{C8N9o5)+ zJ`V~;%Qt&9poK4PA9Z!)?CZ3`mue$59q?J0kc;vy_`9f0wJ0|8m*p0tGLk#Zo`rXO zF7s2Ahu)ZA*o7IJuu+^;FYEgHNn~jm_(3Zt5WiaT61*A@hb6Jr+!{T(ed<`am)XA1 zA-lfyblJhBWqWy;&eix+lx}yHj&Dd=m?SCac&V@VwAUgyONTI#eDs|3oXjuORxr~f z36)3+!BqcdP14aBG_Z~C|BOBAR70H@!!KHu`DEL;;>u>YXz+CB?JH_AVk(EHCicK} zA(nB^p1CoWQkT<*qW8;hs^Y}(QEv#m?DXWzuex;{d6#Pn9p9?$oy+#%xEFfQ#fJS{ z)Q{!O*_YZJB{@I3n(X@shxIl(!xUX=cdvw*SAz7%ImF6)pCliri`hxUjqN;IAsO$7 z_rW^guBvs*1`|o|-MSb|HXMU~BI4e&h!S!o^YD zm$;qwm1}^;*4DJ4`F}v0j{Z1uT>iw5uPl6rSjzYq=4W*~^uI9PTbHZDB7XBA{% zyme2Q+MhF!Q*-T0|4Nmw#N&5mR;Q0-f{qC;3Rlh-ezXQVf?XX4P zVDmyV44ggVT&EOCZov|V3?<|HX8grm5+tw{<%Il)0!VxF9n{Y*L#B7TUVhu{6#Mq$ z;A90^_H_Ca${+FjdOYv@MjajtFjQwe%!_+3W!LvUzZJ6-dj9ynD@`zG zGSwx9q}iVEeDaf5KFXm{xlitRQ=X^qJO9pnMrkj%TpwHlVI$=C9{&HU2nzmN1ph69 z{}#c2PlCh$ErS2&MewdsIG$;*MOXR+JN*^5W_*3Y4O^$(O}^Ec8_^uFvhzYzZrjw7d&HUZoo-WCw`L|9S{g@{z}Wc{$>sRi%Y=o|AWf?Uozrc3jhEB literal 0 HcmV?d00001 diff --git a/rs/moq-scene/src/test_data/tone.webm b/rs/moq-scene/src/test_data/tone.webm new file mode 100644 index 0000000000000000000000000000000000000000..779704a25238b724b9f7c4ae6635de4281899848 GIT binary patch literal 1771 zcmcIli#t?l7(ZvCY$Z}-$ux!76jClplFLktFz%BgF@q!#jZ`l4jcLXjiO~hQpT<2> zDC4rTHliyl)Kju5G|0HNZME#2>Gl`w_ngb~KJV{#-gD0LtsxE|dDd(ITmLONr-6+1 z43M*CvQ7s0hgx$aZRyBhIrv2dQmmvm z!mo*z@j|I58g{7iYZmW507teaehmX>14yGbJArh?BV8FmO$0d_iSLv z+^K;Q0A?^P4`6B&El0woG=CaUq!y-=!|9QB0e&B)XOd=4}UO+?=2_CLi?j*7kfs<}vU?x5EjSWlM+e?Z^a$A8?;et&DA+j?IlZ z8nMkPtX3TYro8;WbXdj{Mo%w(d4jx`LNYj_e8Yr{v#7rTT0McPZXWihr5TQ~B zt_u@%FWdEMz-Ntk&xPKWin|A|->!=K9FpCf!Tuz3iHF<-&@L38R)hP^bt7l!kPr+OVPtWNE$mgTkP-c0CDA-ix?7tO0>7%6EX0sx)*=0_sQdiB4VihtgAK#lRD zWn}9pZ>;L@hrLERVHXq3ys3hOCqGyEJhl)4ET#z0L}~zZ4&|Z)>FmDhBfr&f{OGl& zh`ufzt|p(mj+IqHW;3x0ZqR^APJwz=hFd2WI{h5QlyAG7%>Q-iFzn1nm{z%n@ zn9u2H{w2$U|8zZk-Xrv^tNMP6M{?COoA)XIg;W8k|GS0#=xF(n?TXn2ovTwbxx};S zwVma1mYH~6kHC9WP~@HAx*e$n&=g8=k-Y4F_xpyO7I!X@d+I*6cEqpUZLS!iPU(Gq zx@U5Is&SK+A2|Jbqv}hN90REUuTXn?`GrhuN1Btbt#$($mPqU7QtG#LdQ&;kx~Hbp>WAV z69mJbLNwx`vZfh0htQf5&C|}g(_G>+J#y^K&Ljp>2w67#z(wRhW& z&y~mbWGWk5BG~};Krs}__oaXDZsr8>t>&XEo2H$2&Nh`IR{$LRLtzVx@a}+*4KAk$ z8;ui6#tFxMYZ5zK&ph6fnPprCu;xJvoPN=bt_OZNyO~C&XUpm_XLWpwKT-tXG)Bg? zij(!{jvW#_OiHbD-#aginNq{a&+5q}`Pyzmo{Jbz-<}PISKEwgK9QF$%u4YD1FxaXnKZ>ozaS zO>0Uj_Yj4}2CYjr!HMp@s@t;Cogo->u<8Do7Z+?HSP*xpsT+mW zc}G4~5exPu{w`1wzTSuH776FJ_KxZ|<8^EC@?ytSA1&Sdo$TFAakXgm^fl9pD_{w| dkOe)QKZI^KX$|KkKm4sE$+kAdgxQl|(SMA(4yynF literal 0 HcmV?d00001 diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index da4dc9e..35fe2c2 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -21,7 +21,7 @@ //! edits to them settle, and takes or holds an outside edit to the scene file. //! - The UI runs on the main thread and edits [`State`] under its lock. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, Mutex, mpsc}; @@ -32,7 +32,7 @@ use bytes::Bytes; use kio::clock::Instant; use moq_net::bandwidth; use moq_scene::publish::{Encoding, Sessions}; -use moq_scene::source::{self, Driver, Sources}; +use moq_scene::source::{self, Driver, Sources, Status}; use moq_scene::{Canvas, Collection, Compositor, Draft, Edit, History, Scaler, wgpu}; use moq_video::render::export::{Exporter, Pending, Route}; @@ -201,6 +201,20 @@ impl State { Ok(()) } + /// The sources the program shows: in a scene on air, including one + /// leaving in a transition, or the top layer over them, a clone by the + /// source it shows. + fn shown(&self) -> BTreeSet { + let collection = &self.collection; + let scenes = self.compositor.on_air().filter_map(|scene| collection.scene(scene)); + scenes + .flat_map(|scene| &scene.items) + .chain(&collection.top) + .filter(|item| item.visible) + .map(|item| collection.capture(&item.source).to_string()) + .collect() + } + /// Restart the source called `name`, as an edit would, such as to reload a /// web page. pub fn restart(&mut self, name: &str) { @@ -506,17 +520,9 @@ impl Program { } } - /// Encode the local sources the program shows, from `state`, whether or - /// not anything reads their tracks, so a replay of one has its past. - fn show(&self, state: &State) { - let collection = &state.collection; - let scenes = state.compositor.on_air().filter_map(|scene| collection.scene(scene)); - let shown = scenes - .flat_map(|scene| &scene.items) - .chain(&collection.top) - .filter(|item| item.visible) - .map(|item| collection.capture(&item.source).to_string()) - .collect(); + /// Encode the local sources the program shows, `shown`, whether or not + /// anything reads their tracks, so a replay of one has its past. + fn show(&self, shown: BTreeSet) { self.sources.show(shown); } } @@ -844,30 +850,33 @@ impl Engine { // A stalled picture matters where the program draws it: a scene on air, // including one leaving in a transition, or the top layer over them. // Stalled sound matters only if the mix on program plays it, not one - // only listened to. - let stalled = { + // only listened to. A source that stopped matters either way. + let (stalled, stopped) = { let state = self.state.lock().unwrap(); - let shown: Vec<&str> = state - .compositor - .on_air() - .filter_map(|name| state.collection.scene(name)) - .flat_map(|scene| &scene.items) - .chain(&state.collection.top) - .filter(|item| item.visible) - .map(|item| state.collection.capture(&item.source)) - .collect(); - state + let shown = state.shown(); + let heard = |name: &str| state.sources.plays(&state.collection.mixes.program, name); + let stalled = state .sources .iter() .filter_map(|(name, feed)| { let stall = feed.stalled(now)?; let matters = match stall { - source::Stall::Picture(_) => shown.contains(&name), - source::Stall::Sound(_) => state.sources.plays(&state.collection.mixes.program, name), + source::Stall::Picture(_) => shown.contains(name), + source::Stall::Sound(_) => heard(name), }; matters.then(|| (name.to_string(), stall)) }) - .collect() + .collect(); + let stopped = state + .sources + .iter() + .filter(|(name, _)| shown.contains(*name) || heard(name)) + .filter_map(|(name, feed)| match feed.status() { + status @ (Status::Ended | Status::Failed(_)) => Some((name.to_string(), status)), + _ => None, + }) + .collect(); + (stalled, stopped) }; let canvas = &stats.renditions[0]; health::Local { @@ -880,6 +889,7 @@ impl Engine { encoder: stats.encoder(), failed: stats.error(), stalled, + stopped, } } @@ -942,7 +952,9 @@ impl Engine { tracing::error!(%err, "cannot draw the canvas"); continue; } - self.program.show(state); + let shown = state.shown(); + state.sources.set_on_program(&shown); + self.program.show(shown); (state.compositor.program().clone(), state.compositor.canvas()) }; self.stats.rendered.fetch_add(1, Ordering::Relaxed); diff --git a/rs/moq-studio/src/health.rs b/rs/moq-studio/src/health.rs index 5cb30a5..410d283 100644 --- a/rs/moq-studio/src/health.rs +++ b/rs/moq-studio/src/health.rs @@ -21,7 +21,7 @@ pub use hang::health::Verdict; use hang::health::{self, Health, Input, Observer, Thresholds}; use kio::clock::Instant; use moq_net::bandwidth::Rate; -use moq_scene::source::Stall; +use moq_scene::source::{Stall, Status}; /// How often the health is graded: as often as reporters write. pub const INTERVAL: Duration = Duration::from_secs(1); @@ -214,6 +214,8 @@ pub struct Local { pub failed: Option, /// Sources the program shows or plays that stopped delivering, and what. pub stalled: Vec<(String, Stall)>, + /// Sources the program shows or plays that ended or failed. + pub stopped: Vec<(String, Status)>, } /// What the studio publishes, for grading its delivery. @@ -629,6 +631,18 @@ fn problems(facts: &Facts<'_>) -> Vec { ), ); } + for (name, status) in &local.stopped { + match status { + Status::Ended => add( + Verdict::Degraded, + format!( + "Source `{name}` played to its end, so the program holds its last picture and plays no sound for it." + ), + ), + Status::Failed(reason) => add(Verdict::Unhealthy, format!("Source `{name}` failed: {reason}.")), + _ => {} + } + } if let Some(drops) = facts.drops && local.encoding @@ -1295,6 +1309,10 @@ mod tests { ("camera".into(), Stall::Picture(Duration::from_secs(3))), ("mic".into(), Stall::Sound(Duration::from_secs(2))), ], + stopped: vec![ + ("intro".into(), Status::Ended), + ("music".into(), Status::Failed("the file is gone".into())), + ], ..Default::default() }; let uplink = Uplink { @@ -1311,6 +1329,8 @@ mod tests { "The relay https://a is unreachable (timed out), so nothing reaches viewers until a relay reconnects.", "Source `camera` stalled: no new picture for 3 s, so the program repeats its last one.", "Source `mic` stalled: no new sound for 2 s, so the program plays silence for it.", + "Source `music` failed: the file is gone.", + "Source `intro` played to its end, so the program holds its last picture and plays no sound for it.", "The relay is far away: a round trip to it takes 400 ms.", ] ); diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index f31be18..680a9e2 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -1282,6 +1282,7 @@ impl Panels { let mut remove = None; let mut clone = None; let mut reload = None; + let mut play = None; let now = kio::clock::now(); egui::ScrollArea::vertical().show(ui, |ui| { @@ -1358,11 +1359,23 @@ impl Panels { { clone = Some(name.clone()); } - if matches!(source, Source::Browser { .. }) + if matches!(source, Source::Media { .. }) && ui - .small_button("Reload") - .on_hover_text("Restart the page, keeping the cookies and logins every page shares") + .small_button("Play") + .on_hover_text("Play the file from its start") .clicked() + { + play = Some(name.clone()); + } + let reloads = match source { + Source::Browser { .. } => { + Some("Restart the page, keeping the cookies and logins every page shares") + } + Source::Media { .. } => Some("Read the file again, such as after it changed"), + _ => None, + }; + if let Some(hover) = reloads + && ui.small_button("Reload").on_hover_text(hover).clicked() { reload = Some(name.clone()); } @@ -1389,6 +1402,12 @@ impl Panels { if let Some(name) = reload { state.restart(&name); } + if let Some(name) = play + && let Some(feed) = state.sources.get(&name) + { + let played = feed.play().with_context(|| format!("cannot play `{name}`")); + self.report(played.map(|()| format!("playing `{name}` from its start"))); + } let result = if let Some(name) = remove { // Removing the items that show it moves the ones after them. self.deselect_item(); @@ -3000,6 +3019,10 @@ fn describe(source: &Source) -> String { Source::Microphone { id } => format!("microphone: {}", or_default(id)), Source::SystemAudio => "system audio".into(), Source::Image { path } => format!("image: {}", path.display()), + Source::Media { path, looping, .. } => { + let ending = if *looping { "looping" } else { "once" }; + format!("media: {} ({ending})", path.display()) + } Source::Color { color } => format!("color: {color}"), Source::Broadcast { path } => format!("broadcast: {path}"), Source::Browser { url, width, height } => format!("browser: {url} ({width}x{height})"), @@ -3015,6 +3038,7 @@ fn status(ui: &mut Ui, status: &Status) { Status::Starting => ui.weak("starting"), Status::Live => ui.colored_label(HEALTHY, "live"), Status::Offline => ui.colored_label(DEGRADED, "offline"), + Status::Ended => ui.colored_label(DEGRADED, "ended"), Status::Failed(reason) => ui .colored_label(ui.visuals().error_fg_color, "failed") .on_hover_text(reason), @@ -3062,6 +3086,10 @@ struct AddSource { /// A browser source's page size, or an overlay's. width: u32, height: u32, + /// Whether a media file loops, and plays from its start each time it goes + /// on program. + looping: bool, + restart: bool, } impl Default for AddSource { @@ -3074,6 +3102,8 @@ impl Default for AddSource { color: Color32::WHITE, width: canvas.width, height: canvas.height, + looping: false, + restart: true, } } } @@ -3083,6 +3113,7 @@ enum Kind { #[default] Color, Image, + Media, Broadcast, Camera, Display, @@ -3096,9 +3127,10 @@ enum Kind { } impl Kind { - const ALL: [Self; 12] = [ + const ALL: [Self; 13] = [ Self::Color, Self::Image, + Self::Media, Self::Broadcast, Self::Camera, Self::Display, @@ -3115,6 +3147,7 @@ impl Kind { match self { Self::Color => "Color", Self::Image => "Image", + Self::Media => "Media file", Self::Broadcast => "Broadcast", Self::Camera => "Camera", Self::Display => "Display", @@ -3132,7 +3165,7 @@ impl Kind { fn value(self) -> Option<&'static str> { match self { Self::Color | Self::SystemAudio | Self::Replay => None, - Self::Image => Some("file"), + Self::Image | Self::Media => Some("file"), Self::Lut => Some(".cube file"), Self::Broadcast => Some("path"), Self::Window => Some("window id"), @@ -3183,6 +3216,16 @@ impl AddSource { egui::color_picker::color_edit_button_srgba(ui, &mut self.color, egui::color_picker::Alpha::OnlyBlend); ui.end_row(); } + if self.kind == Kind::Media { + ui.label("Playback"); + ui.horizontal(|ui| { + ui.checkbox(&mut self.looping, "Loop") + .on_hover_text("Start over at the end, rather than hold the last picture"); + ui.checkbox(&mut self.restart, "Restart on program") + .on_hover_text("Play from the start each time it goes on program"); + }); + ui.end_row(); + } add = ui.button("Add source").clicked(); ui.end_row(); }); @@ -3197,6 +3240,11 @@ impl AddSource { color: moq_scene::Rgba(self.color.to_srgba_unmultiplied()), }, Kind::Image => Source::Image { path: value.into() }, + Kind::Media => Source::Media { + path: value.into(), + looping: self.looping, + restart: self.restart, + }, Kind::Lut => Source::Lut { path: value.into() }, Kind::Broadcast => Source::Broadcast { path: value }, Kind::Camera => Source::Camera { id }, From 56c7fc8626b0750b1bdfa8e8b550c70201e5ba79 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 14:31:06 -0800 Subject: [PATCH 3/7] fix(scene): release an ended media file's encoder, and cut off its sound on restart An ended clip delivers no more pictures, so its own track's encoder was held until the next play even off program. It now settles and releases like a still capture. A restart takes a new bus channel, so the sound the cut-off play queued does not play before the new start. Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/publish.rs | 31 ++++++++++++----------- rs/moq-scene/src/source.rs | 50 +++++++++++++++++++++++++++++++------ 2 files changed, 59 insertions(+), 22 deletions(-) diff --git a/rs/moq-scene/src/publish.rs b/rs/moq-scene/src/publish.rs index 4ab0f01..28161e6 100644 --- a/rs/moq-scene/src/publish.rs +++ b/rs/moq-scene/src/publish.rs @@ -549,16 +549,13 @@ impl<'a> Video<'a> { } /// Wait until whether the source is used disagrees with whether its - /// encoder is open, so a capture that delivers nothing new, such as a - /// still screen, can act without waiting on a picture: [`release`] the - /// encoder when it is open and nothing uses the source, or bring a picture - /// when it is closed and the source goes into use. + /// encoder is open, so a source that delivers nothing new, such as a + /// still screen or a media file that ended, can act without waiting on a + /// picture: [`release`] the encoder when it is open and nothing uses the + /// source, or bring a picture when it is closed and the source goes into + /// use. /// /// [`release`]: Self::release - #[cfg_attr( - not(any(feature = "capture", test)), - expect(dead_code, reason = "only a capture waits on it") - )] pub(crate) async fn settle(&self) { let mut demand = self.track.demand(); let mut shown = self.shown.clone(); @@ -573,7 +570,6 @@ impl<'a> Video<'a> { } /// Whether the encoder is open. - #[cfg_attr(not(feature = "capture"), expect(dead_code, reason = "only a capture asks"))] pub(crate) fn is_open(&self) -> bool { self.track.is_open() } @@ -1198,8 +1194,9 @@ mod tests { std::fs::write(dir.path().join("clip.mkv"), crate::media::tests::CLIP).unwrap(); let (publish, catalog, consumer) = studio(); publish.show(["clip".to_string()].into()); + let sessions = publish.sessions.clone(); let (mut context, _origin) = context(dir.path()); - context.publish = Some(publish); + context.publish = Some(publish.clone()); let mut collection = Collection::new(); let clip = Source::Media { path: "clip.mkv".into(), @@ -1249,10 +1246,16 @@ mod tests { (stamps.len() >= 10).then_some(()) }) .await; - assert!( - stamps.is_sorted() && stamps.windows(2).all(|pair| pair[0] < pair[1]), - "{stamps:?}" - ); + assert!(stamps.windows(2).all(|pair| pair[0] < pair[1]), "{stamps:?}"); + + // Ended, it shows nothing new, yet gives its session back once off + // program and unread. + until_time(begin + std::time::Duration::from_millis(1_600)).await; + until(|| (feed.status() == Status::Ended).then_some(())).await; + assert_eq!(sessions.open(), 1, "on program"); + drop(track); + publish.show(Default::default()); + until(|| (sessions.open() == 0).then_some(())).await; }) .await; } diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index d68f1e1..91f7512 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -1050,12 +1050,14 @@ impl Driver { listing: Option<&Listing<'_>>, ) -> Result<(), Error> { let clip = Clip::open(path)?; - // One channel for every play, since a closed one drops what it has not - // played yet. - let mut sound = match (&clip.audio, &self.strip) { - (Some(_), Some(strip)) => Some(self.channel(strip.clone())?), - _ => None, + // A closed channel drops what it has not played yet, so a loop keeps + // its channel, sounding on, while a restart takes a new one, cutting + // off what the play it cut off still had queued. + let channel = || match (&clip.audio, &self.strip) { + (Some(_), Some(strip)) => self.channel(strip.clone()).map(Some), + _ => Ok(None), }; + let mut sound = channel()?; let mut own = Own::new(listing, kio::clock::now()); loop { let restarted = tokio::select! { @@ -1071,12 +1073,44 @@ impl Driver { true } }; - if restarted || looping { + if restarted { + sound = channel()?; + continue; + } + if looping { continue; } set(&self.status, Status::Ended); - if starts.changed().await.is_err() { - return std::future::pending().await; + self.ended(&mut starts, &mut own).await; + } + } + + /// Wait out a media source that ended until `starts` changes. Its own + /// track gets no more pictures, so its encoder is released here once + /// nothing uses the source, giving the encode session back. + async fn ended(&self, starts: &mut watch::Receiver<()>, own: &mut Own<'_, '_>) { + loop { + let settled = async { + match &own.track { + Some(video) if video.is_open() => video.settle().await, + _ => std::future::pending().await, + } + }; + tokio::select! { + changed = starts.changed() => { + // The feed is gone, which stops this driver anyway. + if changed.is_err() { + std::future::pending::<()>().await; + } + return; + } + () = settled => { + if let Some(video) = &mut own.track + && let Err(err) = video.release() + { + self.fail_encoding(&mut own.track, err); + } + } } } } From 606377379ebae870485333000ef6effda7c47121 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 14:51:07 -0800 Subject: [PATCH 4/7] fix(scene): drop a restarted media file's old channel before opening the next A channel left on the strip kept the strip's queue and delay for the new one, so the cut-off play's sound still played first. Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/source.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 91f7512..751a17f 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -1050,9 +1050,9 @@ impl Driver { listing: Option<&Listing<'_>>, ) -> Result<(), Error> { let clip = Clip::open(path)?; - // A closed channel drops what it has not played yet, so a loop keeps - // its channel, sounding on, while a restart takes a new one, cutting - // off what the play it cut off still had queued. + // The last channel on a strip to close drops what it has not played + // yet, so a loop keeps its channel, sounding on, while a restart takes + // a new one, cutting off what the play it cut off still had queued. let channel = || match (&clip.audio, &self.strip) { (Some(_), Some(strip)) => self.channel(strip.clone()).map(Some), _ => Ok(None), @@ -1074,6 +1074,9 @@ impl Driver { } }; if restarted { + // Dropped first: a channel left on the strip keeps the strip's + // queue, delay included, for the new one. + drop(sound.take()); sound = channel()?; continue; } From 56672bd6de98670f9c498e2cc192e127d3739d83 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:08:03 -0800 Subject: [PATCH 5/7] feat(scene): a media file's sound and pictures follow program use apart Publish::set_on_program takes OnProgram, the sources shown and those heard: a picture track encodes while a scene on air shows its source, a sound track while the program mix plays it. A music bed heard but not shown encodes its sound alone, as a microphone does. Co-Authored-By: Claude Opus 5.5 --- README.md | 15 +++-- rs/moq-scene/src/publish.rs | 119 +++++++++++++++++++++++++++++------- rs/moq-studio/src/engine.rs | 24 +++----- 3 files changed, 117 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 1adaab0..8a6ef17 100644 --- a/README.md +++ b/README.md @@ -271,8 +271,11 @@ with `restart`, in a scene on air or the top layer, when it was not on program before. Either clears its picture at once, so the end of the last play never shows on program. A file that cannot be read or decoded fails, with the reason. Its pictures and its sound each have a track of their own (see [Source -tracks](#source-tracks)), on one timeline across plays, used while it is shown -on program or read. A file with no sound has no strip in the Audio panel. +tracks](#source-tracks)), its pictures on one timeline across plays. Each is +used apart, as a camera's and a microphone's are: the pictures while a scene on +air shows it, the sound while the mix on program plays it, and either while +read. So a music bed that no scene shows encodes its sound alone. A file with +no sound has no strip in the Audio panel. ## Guests @@ -585,8 +588,8 @@ plain capture, before any filter, and a clone has none, since it shows its source's capture. A browser source or an overlay has no track of its own, and a broadcast source is already its publisher's. -A microphone or system audio encodes its track as Opus, from the same device -open the mixes hear, only while it is used: while the mix on program plays it +A microphone, system audio, or media file's sound encodes its track as Opus, +from the same sound the mixes hear, only while it is used: while the mix on program plays it (unmuted and not left out of it), so a replay of it has its recent past, or while a recording reads its track. The track carries the sound as captured, before its gain, mute, and delay, which stay the mixes'. A buffer the capture @@ -685,8 +688,8 @@ it is written, shortly after the click. Nothing records in the background for this. The program's tracks keep their last 30 seconds for late viewers anyway, and each source's track while it is on -program (shown in a scene on air, or for a microphone or system audio, played by -the mix on program) or recorded, and a replay copies from those, so a replay +program (a picture shown in a scene on air, or a sound played by the mix on +program) or recorded, and a replay copies from those, so a replay goes back at most that far. A replay of the program needs `--connect`; a local source keeps its track without a relay. With `--on-demand`, the program has no past from before someone watched it, and a source has none from before it diff --git a/rs/moq-scene/src/publish.rs b/rs/moq-scene/src/publish.rs index bbb19b6..39cbbcd 100644 --- a/rs/moq-scene/src/publish.rs +++ b/rs/moq-scene/src/publish.rs @@ -6,10 +6,10 @@ //! sessions a GPU allows. Its renditions are listed in a hang catalog of its //! own, nested by source name under the local catalog's [`SECTION`] section. //! -//! A source encodes only while it is used: while it is on program -//! ([`Publish::set_on_program`]), shown in a scene on air or heard in the -//! program mix, so a replay of it has its recent past, or while something in -//! the studio subscribes to its track, such as a recording. +//! A track encodes only while it is used: while it is on program +//! ([`Publish::set_on_program`]), a picture shown in a scene on air or a sound +//! heard in the program mix, so a replay of it has its recent past, or while +//! something in the studio subscribes to it, such as a recording. use std::collections::{BTreeMap, BTreeSet}; use std::sync::atomic::{AtomicUsize, Ordering}; @@ -88,7 +88,18 @@ pub struct Publish { sessions: Sessions, owners: Arc>, /// The sources on program, which each source's track follows. - on_program: Arc>>, + on_program: Arc>, +} + +/// The sources on program, by name, apart for pictures and sound: a media +/// file heard in the program mix but shown in no scene on air encodes its +/// sound and not its pictures. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct OnProgram { + /// Those whose picture a scene on air shows. + pub shown: BTreeSet, + /// Those whose sound the program mix plays. + pub heard: BTreeSet, } /// Which [`Claim`] owns each source name's entry in the local catalog, and @@ -134,13 +145,14 @@ impl Publish { self.catalog.snapshot() } - /// Encode the sources named in `names`, those on program, whether or not - /// anything reads their tracks; any other encodes only while read. - pub fn set_on_program(&self, names: BTreeSet) { + /// Encode the picture of each source `on_program` shows and the sound of + /// each it hears, whether or not anything reads their tracks; any other + /// track encodes only while read. + pub fn set_on_program(&self, on_program: OnProgram) { // Only a change wakes the sources, since this runs every frame. self.on_program.send_if_modified(|current| { - let changed = *current != names; - *current = names; + let changed = *current != on_program; + *current = on_program; changed }); } @@ -205,7 +217,7 @@ pub(crate) struct Listing<'a> { claim: &'a Claim, kind: moq_video::encode::Kind, sessions: Sessions, - on_program: tokio::sync::watch::Receiver>, + on_program: tokio::sync::watch::Receiver, broadcast: moq_net::broadcast::Producer, studio: Catalog, catalog: moq_mux::catalog::Producer, @@ -471,7 +483,7 @@ pub(crate) struct Video<'a> { track: Encoding, claim: &'a Claim, /// The sources on program. - on_program: tokio::sync::watch::Receiver>, + on_program: tokio::sync::watch::Receiver, config: moq_video::encode::Config, clock: moq_mux::Clock, /// Where the capture's private timeline starts on the studio clock, in @@ -546,7 +558,7 @@ impl<'a> Video<'a> { /// Whether the source is used: shown on program, or its track read. fn used(&self) -> bool { - self.on_program.borrow().contains(&self.claim.name) || self.track.watched() + self.on_program.borrow().shown.contains(&self.claim.name) || self.track.watched() } /// Wait until whether the source is used disagrees with whether its @@ -621,7 +633,7 @@ pub(crate) struct Audio<'a> { producer: Option, claim: &'a Claim, /// The sources on program. - on_program: tokio::sync::watch::Receiver>, + on_program: tokio::sync::watch::Receiver, clock: moq_mux::Clock, /// Whether sound went to the encoder since the source last went out of /// use, so leaving use marks one gap. @@ -651,7 +663,7 @@ impl<'a> Audio<'a> { /// sound after one is stamped afresh, so the track's timeline holds it. pub(crate) fn write(&mut self, samples: &[f32], captured: kio::clock::Instant, gap: bool) -> Result<(), Error> { let producer = self.producer.as_mut().expect("open until it ends"); - let used = self.on_program.borrow().contains(&self.claim.name) || producer.demand().wanted().is_some(); + let used = self.on_program.borrow().heard.contains(&self.claim.name) || producer.demand().wanted().is_some(); if !used { if std::mem::take(&mut self.encoding) { producer.discontinuity()?; @@ -751,6 +763,22 @@ mod tests { (publish, catalog, consumer) } + /// On program with only `name`'s picture shown. + fn shown(name: &str) -> OnProgram { + OnProgram { + shown: [name.to_string()].into(), + ..Default::default() + } + } + + /// On program with only `name`'s sound heard. + fn heard(name: &str) -> OnProgram { + OnProgram { + heard: [name.to_string()].into(), + ..Default::default() + } + } + /// Yield until `f` holds of the local catalog's source listings. async fn listed(catalog: &Catalog, f: impl Fn(&BTreeMap) -> bool) -> hang::Catalog { for _ in 0..1000 { @@ -888,7 +916,7 @@ mod tests { let snapshot = listed(&catalog, |sources| width(sources) == Some(64)).await; assert!(snapshot.video.renditions.is_empty(), "no source is a program rendition"); - publish.set_on_program(["camera".to_string()].into()); + publish.set_on_program(shown("camera")); for millis in [66, 100, 133] { video.encode(&mut gray(small, millis)).unwrap(); } @@ -950,7 +978,7 @@ mod tests { kio::clock::timeout(wait, video.settle()).await.is_err(), "unused and closed" ); - publish.set_on_program(["display".to_string()].into()); + publish.set_on_program(shown("display")); kio::clock::timeout(wait, video.settle()) .await .expect("on program without a picture, so the capture brings one"); @@ -968,7 +996,7 @@ mod tests { // lands on the program's clock now rather than a minute back. kio::clock::sleep(std::time::Duration::from_secs(60)).await; video.reopened(); - publish.set_on_program(["display".to_string()].into()); + publish.set_on_program(shown("display")); let now = publish.catalog.clock().now(); let mut picture = gray(size, 0); video.encode(&mut picture).unwrap(); @@ -1116,7 +1144,7 @@ mod tests { let refused = crate::replay::Take::new(&target(), std::time::Duration::from_secs(1)); assert!(refused.is_err(), "nothing to replay"); - publish.set_on_program(["mic".to_string()].into()); + publish.set_on_program(heard("mic")); let heard = clock.now(); capture(&mut audio, false).await; assert!(written() > 0, "on program"); @@ -1341,7 +1369,7 @@ mod tests { let dir = tempfile::tempdir().unwrap(); let (publish, catalog, consumer) = studio(); // On program, so its track encodes. - publish.set_on_program(["replay".to_string()].into()); + publish.set_on_program(shown("replay")); let (mut context, _origin) = context(dir.path()); context.publish = Some(publish); let mut collection = Collection::new(); @@ -1405,7 +1433,7 @@ mod tests { let dir = tempfile::tempdir().unwrap(); std::fs::write(dir.path().join("clip.mkv"), crate::media::tests::CLIP).unwrap(); let (publish, catalog, consumer) = studio(); - publish.set_on_program(["clip".to_string()].into()); + publish.set_on_program(shown("clip")); let sessions = publish.sessions.clone(); let (mut context, _origin) = context(dir.path()); context.publish = Some(publish.clone()); @@ -1478,6 +1506,55 @@ mod tests { .await; } + /// A music bed the program mix plays encodes its sound, while its + /// pictures, which no scene on air shows, encode nothing until one does. + #[tokio::test(start_paused = true)] + async fn a_heard_media_file_encodes_only_its_sound() { + local(async { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join("bed.mkv"), crate::media::tests::CLIP).unwrap(); + let (publish, catalog, _consumer) = studio(); + publish.set_on_program(heard("bed")); + let sessions = publish.sessions.clone(); + let (mut context, _origin) = context(dir.path()); + context.publish = Some(publish.clone()); + let mut collection = Collection::new(); + let bed = Source::Media { + path: "bed.mkv".into(), + looping: true, + restart: false, + }; + collection.sources.insert("bed".into(), bed); + let mut sources = Sources::new(context); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + + // Both listed, the pictures' at the first picture. + let listed = until(|| { + let listed = catalog.snapshot().ext.studio.sources.remove("bed"); + listed.filter(|listed| !listed.video.renditions.is_empty() && !listed.audio.renditions.is_empty()) + }) + .await; + let audio = listed.audio.renditions.keys().next().unwrap().clone(); + let video = listed.video.renditions.keys().next().unwrap().clone(); + let written = |id: &str| publish.stats().get(id).map_or(0, |rendition| rendition.frames); + + until_time(kio::clock::now() + std::time::Duration::from_millis(300)).await; + until(|| (written(&audio) > 0).then_some(())).await; + assert_eq!(written(&video), 0, "heard, not shown"); + assert_eq!(sessions.open(), 0, "no encoder for the pictures"); + + // Shown too, its pictures encode as they play. + publish.set_on_program(OnProgram { + shown: ["bed".to_string()].into(), + heard: ["bed".to_string()].into(), + }); + until_time(kio::clock::now() + std::time::Duration::from_millis(300)).await; + until(|| (written(&video) > 0).then_some(())).await; + }) + .await; + } + /// A file of sound alone lists its sound's track, live, and no pictures. #[tokio::test(start_paused = true)] async fn an_audio_file_lists_its_sound() { diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 4c7cf29..b7584de 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -534,24 +534,20 @@ impl Program { } } - /// Encode the local sources on program, whether or not anything reads - /// their tracks, so a replay of one has its past: those `shown` in a scene - /// on air, and the microphones and system audio the program mix plays. A - /// replay source's track is its pictures, so being heard alone does not - /// put it on program. + /// Encode the local sources' tracks on program, whether or not anything + /// reads them, so a replay of one has its past: the pictures `shown` in a + /// scene on air, and the sound the program mix plays. A media file heard + /// and not shown encodes only its sound. fn on_program(&self, state: &State, shown: BTreeSet) { - let collection = &state.collection; + let program = &state.collection.mixes.program; let heard = state .sources .iter() - .filter(|(name, feed)| { - matches!( - feed.source(), - moq_scene::Source::Microphone { .. } | moq_scene::Source::SystemAudio - ) && state.sources.plays(&collection.mixes.program, name) - }) - .map(|(name, _)| name.to_string()); - self.sources.set_on_program(shown.into_iter().chain(heard).collect()); + .filter(|(name, _)| state.sources.plays(program, name)) + .map(|(name, _)| name.to_string()) + .collect(); + self.sources + .set_on_program(moq_scene::publish::OnProgram { shown, heard }); } } From 079b9af013f19ae5f3a7bf3363df5b79592078fc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:13:52 -0800 Subject: [PATCH 6/7] fix(scene): a media file's sound track runs on through loops; a play after its end takes a new channel A natural loop no longer marks a gap on the sound's own track, which dropped the codec's partial frame and lookahead each loop. A play after the clip ended now resets the bus channel and marks the gap, as a restart does. Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/source.rs | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index a7c008e..69874fd 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -1249,11 +1249,29 @@ impl Driver { self.publish(crate::publish::Audio::new(listing, input)) }); let channel = Some(self.channel(strip.clone())?); - Some(Sound { channel, track }) + Some(Sound { + channel, + track, + gap: false, + }) } _ => None, }; let mut own = Own::new(listing, kio::clock::now()); + // The last channel on a strip to close drops what it has not played + // yet, so a loop keeps its channel and its track's timeline, sounding + // on, while a play after a restart or an end takes a new channel, + // cutting off what the play before still had queued, and marks a gap on + // its track. The old channel goes first: one left on the strip would + // keep the strip's queue, delay included, for the new. + let restart = |sound: &mut Option>| -> Result<(), Error> { + if let (Some(sound), Some(strip)) = (sound, &self.strip) { + drop(sound.channel.take()); + sound.channel = Some(self.channel(strip.clone())?); + sound.gap = true; + } + Ok(()) + }; loop { let restarted = tokio::select! { result = self.play_clip(&clip, sound.as_mut(), &mut own) => { @@ -1268,16 +1286,8 @@ impl Driver { true } }; - // The last channel on a strip to close drops what it has not - // played yet, so a loop keeps its channel, sounding on, while a - // restart takes a new one, cutting off what the play it cut off - // still had queued. The old one goes first: a channel left on the - // strip would keep the strip's queue, delay included, for the new. if restarted { - if let (Some(sound), Some(strip)) = (&mut sound, &self.strip) { - drop(sound.channel.take()); - sound.channel = Some(self.channel(strip.clone())?); - } + restart(&mut sound)?; continue; } if looping { @@ -1285,6 +1295,7 @@ impl Driver { } set(&self.status, Status::Ended); self.ended(&mut starts, &mut own).await; + restart(&mut sound)?; } } @@ -1393,8 +1404,6 @@ impl Driver { }; let bus = self.context.bus.config(); let bytes = std::mem::size_of::() * bus.layout.channels() as usize; - // Each play starts the track's timeline afresh. - let mut gap = true; while let Some(frame) = samples.take() { let due = at(frame.timestamp); kio::clock::sleep_until(due).await; @@ -1407,7 +1416,7 @@ impl Driver { .write(&frame.data)?; if let Some(track) = &mut sound.track { let samples: Vec = bytemuck::pod_collect_to_vec(&frame.data); - if let Err(err) = track.write(&samples, kio::clock::now(), std::mem::take(&mut gap)) { + if let Err(err) = track.write(&samples, kio::clock::now(), std::mem::take(&mut sound.gap)) { tracing::warn!(source = %self.name, %err, "cannot encode the source"); self.fail_track(Status::Failed(err.to_string())); sound.track.take().expect("encoding").abort(&err); @@ -1490,6 +1499,9 @@ impl Driver { struct Sound<'a> { channel: Option, track: Option>, + /// Whether the next sound follows a break, a restart or a pause after the + /// end, rather than a loop, which the track's timeline runs on through. + gap: bool, } /// A played source's own track, opened at its first picture, and the timeline From 05ce7572c97ef0b5c65041745533c66a473a0d92 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 23:14:13 -0800 Subject: [PATCH 7/7] fix(scene): keep a looping media file's sound on time, and fail one that decodes to nothing - A loop's next play starts where the last one ended, so loops keep exact time, and its sound track marks a gap unless its sound starts right where the last one ended: a loop whose sound is shorter than its pictures keeps the silence between, rather than drifting earlier each loop. - A restart, or a play after the end, starts a new group on the pictures' track, a cut to the clip's start. - A clip that decodes to no picture or sound fails, rather than a loop replaying it at once forever. - The media publish tests walk the clip picture by picture, holding the paused clock on what the driver shows, so they count on neither a decoder's thread nor an encoder that skips pictures to hold its rate, as openh264 on Linux does at 64x64. Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/media.rs | 3 + rs/moq-scene/src/publish.rs | 128 +++++++++++++++++++++----- rs/moq-scene/src/source.rs | 65 ++++++++----- rs/moq-scene/src/test_data/uneven.mkv | Bin 0 -> 9854 bytes 4 files changed, 153 insertions(+), 43 deletions(-) create mode 100644 rs/moq-scene/src/test_data/uneven.mkv diff --git a/rs/moq-scene/src/media.rs b/rs/moq-scene/src/media.rs index 0c50bc2..2de44ce 100644 --- a/rs/moq-scene/src/media.rs +++ b/rs/moq-scene/src/media.rs @@ -143,6 +143,9 @@ pub(crate) mod tests { /// The first half second of [`CLIP`]'s pictures alone. pub(crate) const SILENT: &[u8] = include_bytes!("test_data/silent.mkv"); + /// [`CLIP`]'s second of pictures with [`TONE`]'s half second of sound. + pub(crate) const UNEVEN: &[u8] = include_bytes!("test_data/uneven.mkv"); + #[test] fn reads_by_the_first_bytes() { assert_eq!(container(CLIP), Ok(ContainerFormat::Mkv)); diff --git a/rs/moq-scene/src/publish.rs b/rs/moq-scene/src/publish.rs index 39cbbcd..2eb6a92 100644 --- a/rs/moq-scene/src/publish.rs +++ b/rs/moq-scene/src/publish.rs @@ -606,6 +606,16 @@ impl<'a> Video<'a> { Ok(()) } + /// Start a group with the next picture, as a keyframe, such as where a + /// media file plays again from its start. A closed encoder starts with one + /// anyway. + pub(crate) fn cut(&mut self) -> Result<(), Error> { + if self.track.is_open() { + self.track.cut()?; + } + Ok(()) + } + /// Mark a break, such as a gap in a replay: what follows does not continue /// what came before, and a viewer arriving during it waits for what follows /// rather than being served the last picture as if it were live. @@ -779,6 +789,19 @@ mod tests { } } + /// Walk a media source playing a 10 fps clip through `count` pictures, the + /// first due at `first`. The clock moves to each picture's moment and is + /// held while it decodes, encodes, and shows, so no wait counts on work in + /// threads the runtime cannot see, such as a decoder's or an encoder's, + /// finishing within any span of the clock. + async fn pictures(feed: &crate::source::Feed, first: kio::clock::Instant, count: u32) { + let shown = || matches!(feed.take(), Some(crate::source::Picture::Frame(_))).then_some(()); + for index in 0..count { + until_time(first + std::time::Duration::from_millis(100) * index).await; + until(shown).await; + } + } + /// Yield until `f` holds of the local catalog's source listings. async fn listed(catalog: &Catalog, f: impl Fn(&BTreeMap) -> bool) -> hang::Catalog { for _ in 0..1000 { @@ -1469,35 +1492,36 @@ mod tests { let container = moq_mux::catalog::hang::Container::try_from(config).unwrap(); let mut track = moq_mux::container::Consumer::new(track, container); - // Each wait holds the clock: a decoder working on its own thread - // leaves the runtime idle, which would otherwise jump the clock past - // pictures still to come. + // Half way, five pictures in, it starts over, and plays to its end. + let first = kio::clock::now(); + pictures(feed, first, 5).await; + until_time(first + std::time::Duration::from_millis(450)).await; + let cut = publish.catalog.clock().now(); + feed.play().unwrap(); + let restarted = kio::clock::now(); + pictures(feed, restarted, 10).await; + until_time(restarted + std::time::Duration::from_millis(1_100)).await; + until(|| (feed.status() == Status::Ended).then_some(())).await; + + // How many pictures land depends on the encoder, which may skip + // some to hold its rate, as openh264 does at this size; the + // keyframe each play opens with always does. let now = kio::Waiter::noop(); let mut stamps = Vec::new(); - let mut read = |count: usize| { + until(|| { while let std::task::Poll::Ready(read) = track.poll_read(&now) { if let moq_mux::container::Read::Frame { frame, .. } = read.unwrap().expect("open") { stamps.push(frame.timestamp); } } - (stamps.len() >= count).then_some(()) - }; - - // Half way, five pictures in, it starts over. - let begin = kio::clock::now(); - until_time(begin + std::time::Duration::from_millis(450)).await; - until(|| read(5)).await; - feed.play().unwrap(); - until(|| read(6)).await; - let restarted = kio::clock::now(); - until_time(restarted + std::time::Duration::from_millis(350)).await; - until(|| read(9)).await; + stamps.iter().any(|stamp| *stamp >= cut).then_some(()) + }) + .await; + assert!(stamps[0] < cut, "the first play's keyframe: {stamps:?}"); assert!(stamps.windows(2).all(|pair| pair[0] < pair[1]), "{stamps:?}"); // Ended, it shows nothing new, yet gives its session back once off // program and unread. - until_time(restarted + std::time::Duration::from_millis(1_200)).await; - until(|| (feed.status() == Status::Ended).then_some(())).await; assert_eq!(sessions.open(), 1, "on program"); drop(track); publish.set_on_program(Default::default()); @@ -1539,22 +1563,84 @@ mod tests { let video = listed.video.renditions.keys().next().unwrap().clone(); let written = |id: &str| publish.stats().get(id).map_or(0, |rendition| rendition.frames); - until_time(kio::clock::now() + std::time::Duration::from_millis(300)).await; + let feed = sources.get("bed").unwrap(); + let first = kio::clock::now(); + pictures(feed, first, 4).await; until(|| (written(&audio) > 0).then_some(())).await; assert_eq!(written(&video), 0, "heard, not shown"); assert_eq!(sessions.open(), 0, "no encoder for the pictures"); - // Shown too, its pictures encode as they play. + // Shown too, its pictures encode as they play, past what a + // pipelined encoder holds back. publish.set_on_program(OnProgram { shown: ["bed".to_string()].into(), heard: ["bed".to_string()].into(), }); - until_time(kio::clock::now() + std::time::Duration::from_millis(300)).await; + pictures(feed, first + std::time::Duration::from_millis(400), 6).await; until(|| (written(&video) > 0).then_some(())).await; }) .await; } + /// A looping file whose sound is shorter than its pictures keeps its sound + /// track on time: each loop's sound starts where the loop does, the + /// silence between kept, rather than right after the last loop's sound. + #[tokio::test(start_paused = true)] + async fn a_looping_media_file_keeps_its_sound_on_time() { + local(async { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join("bed.mkv"), crate::media::tests::UNEVEN).unwrap(); + let (publish, catalog, consumer) = studio(); + publish.set_on_program(heard("bed")); + let (mut context, _origin) = context(dir.path()); + context.publish = Some(publish.clone()); + let mut collection = Collection::new(); + let bed = Source::Media { + path: "bed.mkv".into(), + looping: true, + restart: false, + }; + collection.sources.insert("bed".into(), bed); + let mut sources = Sources::new(context); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + + let listed = until(|| { + let listed = catalog.snapshot().ext.studio.sources.remove("bed"); + listed.filter(|listed| !listed.audio.renditions.is_empty()) + }) + .await; + let (name, config) = listed.audio.renditions.iter().next().unwrap(); + let track = consumer + .track(hang::catalog::Reference::track(config, name)) + .unwrap() + .subscribe( + moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(None)), + ) + .await + .unwrap(); + let container = moq_mux::catalog::hang::Container::try_from(config).unwrap(); + let mut track = moq_mux::container::Consumer::new(track, container); + // Two and a half loops of a second, each with half a second of sound. + pictures(sources.get("bed").unwrap(), kio::clock::now(), 25).await; + let now = kio::Waiter::noop(); + let mut stamps = Vec::new(); + let silences = until(|| { + while let std::task::Poll::Ready(read) = track.poll_read(&now) { + if let moq_mux::container::Read::Frame { frame, .. } = read.unwrap().expect("open") { + stamps.push(std::time::Duration::from(frame.timestamp)); + } + } + let gaps = stamps.windows(2).map(|pair| pair[1].saturating_sub(pair[0])); + let silences = gaps.filter(|gap| *gap > std::time::Duration::from_millis(300)).count(); + (silences >= 2).then_some(silences) + }) + .await; + assert_eq!(silences, 2, "each loop's silence is kept: {stamps:?}"); + }) + .await; + } + /// A file of sound alone lists its sound's track, live, and no pictures. #[tokio::test(start_paused = true)] async fn an_audio_file_lists_its_sound() { diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 69874fd..dea576d 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -1252,30 +1252,38 @@ impl Driver { Some(Sound { channel, track, - gap: false, + ended: None, }) } _ => None, }; let mut own = Own::new(listing, kio::clock::now()); // The last channel on a strip to close drops what it has not played - // yet, so a loop keeps its channel and its track's timeline, sounding - // on, while a play after a restart or an end takes a new channel, - // cutting off what the play before still had queued, and marks a gap on - // its track. The old channel goes first: one left on the strip would - // keep the strip's queue, delay included, for the new. - let restart = |sound: &mut Option>| -> Result<(), Error> { + // yet, so a loop keeps its channel, sounding on, while a play after a + // restart or an end takes a new channel, cutting off what the play + // before still had queued. The old channel goes first: one left on the + // strip would keep the strip's queue, delay included, for the new. + // Its pictures start a group there too, a cut to the clip's start. + let restart = |sound: &mut Option>, own: &mut Own<'_, '_>| -> Result<(), Error> { if let (Some(sound), Some(strip)) = (sound, &self.strip) { drop(sound.channel.take()); sound.channel = Some(self.channel(strip.clone())?); - sound.gap = true; + sound.ended = None; + } + if let Some(video) = &mut own.track + && let Err(err) = video.cut() + { + self.fail_encoding(&mut own.track, err); } Ok(()) }; + // Where a loop's next play starts: where the last one ended, so loops + // keep exact time whatever opening the decoders takes. + let mut next = None; loop { let restarted = tokio::select! { - result = self.play_clip(&clip, sound.as_mut(), &mut own) => { - result?; + result = self.play_clip(&clip, sound.as_mut(), &mut own, next) => { + next = Some(result?); false } changed = starts.changed() => { @@ -1287,7 +1295,8 @@ impl Driver { } }; if restarted { - restart(&mut sound)?; + restart(&mut sound, &mut own)?; + next = None; continue; } if looping { @@ -1295,7 +1304,8 @@ impl Driver { } set(&self.status, Status::Ended); self.ended(&mut starts, &mut own).await; - restart(&mut sound)?; + restart(&mut sound, &mut own)?; + next = None; } } @@ -1330,14 +1340,16 @@ impl Driver { } /// Play `clip` once, each picture and sound when its moment comes, the - /// first at once. Ends once the last has played for as long as it lasts: - /// a sound for its samples, and a picture for as long as the one before it. + /// first at `begin`, or at once. Returns where it ends, once the last has + /// played for as long as it lasts: a sound for its samples, and a picture + /// for as long as the one before it. A clip that decodes to nothing fails. async fn play_clip( &self, clip: &Clip, mut sound: Option<&mut Sound<'_>>, own: &mut Own<'_, '_>, - ) -> Result<(), Error> { + begin: Option, + ) -> Result { set(&self.status, Status::Live); let broadcast = clip.consume(); let mut video = match &clip.video { @@ -1370,14 +1382,16 @@ impl Driver { samples.as_ref().map(|frame| Duration::from(frame.timestamp)), ]; let Some(start) = first.into_iter().flatten().min() else { - return Ok(()); + // Its tracks hold no frame that decodes, which a loop would only + // play again at once, forever. + return Err(Error::Media("the file decodes to no picture or sound".into())); }; // The pictures' track opens before the clock starts, so the encoder's // probe holds back neither the first picture nor the sound beside it. if let (Some(frame), Some((_, config))) = (&picture, &video) { self.open(own, frame, config).await; } - let begin = kio::clock::now(); + let begin = begin.unwrap_or_else(kio::clock::now); let at = |timestamp: moq_net::Timestamp| begin + Duration::from(timestamp).saturating_sub(start); let end = std::cell::Cell::new(begin); let last = |until: Instant| end.set(end.get().max(until)); @@ -1404,11 +1418,18 @@ impl Driver { }; let bus = self.context.bus.config(); let bytes = std::mem::size_of::() * bus.layout.channels() as usize; + // The track's timeline runs on by its samples only from a sound + // that starts right where the last one ended, as a loop's does + // when its sound is its longest track: any other start, such as + // after a loop's silence while its pictures play on, is a gap. + let mut first = true; while let Some(frame) = samples.take() { let due = at(frame.timestamp); kio::clock::sleep_until(due).await; let length = (frame.data.len() / bytes) as u64 * 1_000_000_000 / u64::from(bus.sample_rate); last(due + Duration::from_nanos(length)); + let gap = std::mem::take(&mut first) && sound.ended != Some(due); + sound.ended = Some(due + Duration::from_nanos(length)); sound .channel .as_mut() @@ -1416,7 +1437,7 @@ impl Driver { .write(&frame.data)?; if let Some(track) = &mut sound.track { let samples: Vec = bytemuck::pod_collect_to_vec(&frame.data); - if let Err(err) = track.write(&samples, kio::clock::now(), std::mem::take(&mut sound.gap)) { + if let Err(err) = track.write(&samples, due, gap) { tracing::warn!(source = %self.name, %err, "cannot encode the source"); self.fail_track(Status::Failed(err.to_string())); sound.track.take().expect("encoding").abort(&err); @@ -1428,7 +1449,7 @@ impl Driver { }; tokio::try_join!(pictures, sounds)?; kio::clock::sleep_until(end.get()).await; - Ok(()) + Ok(end.get()) } /// Open the source's own track for pictures like `frame`, which says their @@ -1499,9 +1520,9 @@ impl Driver { struct Sound<'a> { channel: Option, track: Option>, - /// Whether the next sound follows a break, a restart or a pause after the - /// end, rather than a loop, which the track's timeline runs on through. - gap: bool, + /// When the last sound written ends, or `None` after a break, such as a + /// restart, so the next sound's track starts its timeline afresh. + ended: Option, } /// A played source's own track, opened at its first picture, and the timeline diff --git a/rs/moq-scene/src/test_data/uneven.mkv b/rs/moq-scene/src/test_data/uneven.mkv new file mode 100644 index 0000000000000000000000000000000000000000..471ec3881406c1a413d589b625da4906e6f602e6 GIT binary patch literal 9854 zcmb_?cT^Ky*Y8OIg0#>S1R)ebdPk%Nq>Iu76p`M0ZvrVqx)f0XMXDlIKmnB+5Cla9 z3yLTL7K(s`CY`y7@AG``x_7PbzdI{AbIRVovuE$$oSE4(hYcpm%k`27Fue`xe}#b5 zTP2|Mh$#Us;h{lcelB{+)UL|}IFB{n3kGPLm?`_|TjAs`QY<}s^>kahbq$Hzq{`E} zmQebi*1@y){iv;%r`s~9V;pCndRf;JVekLb0wXfNoFpQo_4x_clxO**evYdIcdh~!n;UGzt2tH|Hq`Xj|5m)U&=o= z!3qgLxVbPfBG5gB0Ftt7olaXA7%9t{Sewh~+MY}#z(l?$lIYf_z+`>{0ozKz-Xy?n z8ZvC25MVZssQ<^*=5GS{_kiI5xRu?5>5&gWQ{a#nb^i$|M%D)a0g$x9O4GOi0|Rv4 zgNcKfe@$}!NPz#>B>4YrQj)HdxrL3jZFQoXY>90f3EA5O5()6BoMHmJIR!ft{P##) z-3b_)9d2d0=D`tR#vU&2Ft{2F-}ZNu7A9l3BKLuI&e>d>|0>b|{ zz#yt&{~MsZtctwae;|S2$(D$-$GS)^T%T5E2%!8m5deHN;oe361ne{b2Xx%c_GX&S zD~XxgE??{zDF1DD*JCik?S6dcN_5+SLo{;Mbt zDPi0_T>XRG{InF*sUF9;1iJXggn4MmM^oF;3JMCC01qv1k7!I-ge$dn91|QCqoqJA zokHEUXaQ1}rxut1pJ)$vCmJG6!zt7y(91(hK^5cX9U2ti;zU(dz=Vf-`1|{WQEO_^ zYVK~~R1@4n0;r`sl}E8bfgW0l3ewaWJzc`Wor1&se1fT>{{VypI|X@qhIxc*$tYsN zy+f%QG$8+=AU_vxYQyQjePRASZd6E4e7wBFgQ+z?j~ME(mdZaB zrvRToDs-5eN1%sWM7Wl+JZ+j#7n(~#J;J=H8li4lZlRu-09UFXs1C+Z8(NA=vhtV^ z+T>dDvZ~Z-Fb(JL-X&T~^#rveEZielOBv%6Om!+XfK;DTv0Xx_@uGR3#xS~Zj}(%4 zudoz7Qs^G+fT}ZW-r@sGFx)lTOOMf{=dBl1K4zn{t)SZP|1C4BIF)4i|L>8)91YtJEkIt~r^ z0UE8(3h4D=0M_ZrRogRlSsJ=Az!3h>LCPR7&8@lA`y$+bZr4<{WL zAN)Qt8PymWF6Fz9sC}8!y^rfuGxQSKOswSmENWR8q;mHJb~Kq2SB3P_v%4Z;;j?;& z!$#*bGE^Rq?yauSRs?;7wR6mcT#p6iZ=5#fx(>xxlpoIy?ekikO&#PioJ-h`j^T`t zWFCGKFco+msuLNom$+XWwP4nJmhr)(L}P$D_+Zz)#5MH1-xISTbcb4dk`?8B0W`~x z#VE=z6u{ts4ANR%ojIidW-R|C z>d@DGydg9S;0(@59#^U+b2(bD*ExzOlx`U_2beMV@~@Z7-fIAl)HV>l(W*&V?=R&W zEtLuPNG@^7QP4RJ_v;e01lZ!_%7_kJ|d>j%v)eZ!g|p1%*v}{4c8uu=K4Ph ziOZqy;=v2II$C`ZI8nYhuV=NCJxyoVc9I$KonFlr?tbQ#eHUSX8R;R-hGMmtbOvZ9 z4B(+R_b(5A|57QCl6qju9pMZ_n^ex!3TcQAC^S9JJ8q0eo_NFrgDa4c(FufPvYGz3 z2hAGP2n7Q)K(QtPdVE;LsV$VTfi%e(`Q*@~A=`?>!>j$@;4~b!ST@s?5Q71jGSoTOIMW~3moyYSei+6m>JR)AL$r0jQ}#ej?zJpk9eB1gRy8>IxJ5RJRp?F=phXm|d+ zGu3U-TH(qE<1(0}0R(29IHYCZfpw;5%CZi0x|^>em;_(LAORW)d-Nlr zznz!;>df^TJXWT^fRIg%6&>u5TKN+q;KoN6d%&W-K0ec&Q|_|Jn-kSW!|>(u`$QmtkG}cfs-*#_GE28465cUseVe#= zc**eEho0Jg$A-uCk-L6{o%u;S$QxJ79}lRS)vn12)%Uv-_`^b%T z{jZuT+QL|Ftsg|c5?wY;w=*#T4MXq~BJXMQdC*kzES5*Ao57o4>bCA+e<6f>F5ZTH1nJK@St3ATKDqPq@|kbL%F zVq6|PjM%6-Cf2;0)A7|wUJqSH(oHzBflBiKY#9G0YukMBg~WB83vE`<_J?(tDaK23 z5+65`fc#GLEId1|LF|{H#vj&(tf|7XVN)5PpF(6(VgTZoidg}Y?)1>$-li|_{q&=; z*Czh-3r6B=YVr+s?93F0rc}n=GCF2Aoca*#0Y&rsPOj&_HN1*^U39{RAM*ASTpdlf z!t=(iS8hY!3r>w*y9wbY%Ug*c=3ge5n9!Xc|FMkw{OkfB{!7n{!mq3I_1>GOlqG_$ zC#pNRwI;qlbpPUz_8fs5ULim%Zz6&#G+i21hu__~$fBUM;9u7f`tglKy2#yh!Sw7h zwu;mI&wkgN{HnYFRra8-#*)g(L{Li8`b$_#B;CZkJ}-{(z$J>Y^Akz0iVx(Ej;ihs zTQ7rVtNiBva7T3TJzE`VV=8mvM#j0M_x`+F-be(^c;lPu#Z>m1us+Lv#?9$;uSPoI zIe&&*xX9J19S5$tfT*j621m~jD7dVo-DKKX^_-?+WnFD-dc=JQ=Uaa*ZZ|i_wUXKA zt(A9{O~7R|3d=8uVP$+65;)@vFF8^0 zNSj$*cRUPVdG7KNK}{xDU4%#%Tm86%V*X#Qt*FN3`WC+6{R=R!9=2_(>)%R{JgOZvlBC&zWN5p zlX|A4jC8Q6-)lke@<$R^oK;We?SsOqY-D99luHqMp2fzAcp7P>wxim+Dujr*_TiGp zYqfpLuU9^V(1aBk4#<@=l3&(_NhGrk(K3HP=U=iGm%4eQmj7196iT z5d>&5NkVw0doqr!oNfJP;($MVKufG(*xn^t&zuzbmk}O`_J1Ci12*JY1w+O3&3^Ott85!uiAxzBdwrcFXIDVElhbWcuq@%}*oNHAv#l@@kR6 z<=p1s8WgrbK-$Lhg&U#5A>ZnFc{33#(-bVJ3+?#XL8W2s-W%p4E!%z1aYuAC(SAa9 zA3n#9P%zml9Y}Sn@R+S3C0FJ+4JKA!=~xSrUhgSyPoL zxk12=^(%`=ZFwCL#`{0g!+q(kDbu`PA{N&Y1irD|QGSC~Yc;Q7jzqwYJ_?vuwJ5J5 z!enUTR9fq|PVe{$<3M%CD(tp?UuN`GvD>|!l^0iL0xxN#;mdVz5@6CaMqkyv>1?!# zOLCmh5z@)%sqt)xA~b8?+P0%Pv{pwt#Bv?0jHVtJ@7gAy zv_EWb>Z7m8=zd$#JJa!9&3m$2%#kG*voNgp06%Nm?Ki1;m`ld4o^V>CKUBCn4K(k>*{~lHT!MXDEMzaAnS9}5H(v}bPhKq)HtE1Y zYn%(mm0@X1r{y7Vs!-<~oZZh*RtJaY`&u`%Q->5~otg-F`|>dV0Q>8NPyB(yAzC*~ z8X|V4x|_1i#;vU=~HlM!&0vL#{&Kqs(_zmyG*nh=ahEGwE$hTTU3QH?eF37 zOcc+GmDs^(UvS)gx)>$1rmmhEYx`{u`t`Kx+!f!Q`rYy}B24$+u$!0)%PjFIGidgc z_!bpI)@yJh1qW5I<%L9;4Nb<5df}CQ(O>Y}U{>VyEy)!NjkS)d@>@ih&wml*NcxKx zE}rDSUBMmAPo{Dm7XR;{!EoQDA;W^3D%AvTO)n^UeD1-UmoEIqtL;SF92cKV88};N$pmXc1wCt9%r}oKl%Y6JSYcj|*CnL%7Pv)h+%XN1MuxuKsR&{UU zjW%5wE)04hw+j7WxdEqJ#>nBN0uB+%TYm2{a8EegD}|SLa-zd7RXCCFd`_TVzEPQ| zejlnR6WW@R~?vYf+&DK@mrtMB?&P*o*OjuF=vER|y34e?Tiil)Oa6iqD&mgZ+Hsm*`AXL= zF139UdYqYj?NvMD8KCfGHoHDWQgArTwC$&NUjiG2#*(}E{Kn9mR&$h2-O}r3OOH8F z%nwzj9xVL|6&gM77Iu1dyZ(3iT_WrWKEJ^r*ywh2iIU=k`{ulzKvlhHnox+08kzL? zYNYki<=-^8S2VGBFTo5|{<#<5TIASG?cCuXruPY>_AX%ju6j^J&|kSJe182L#)vKQ z+k^{yS^L(X+v=Uh13(|@kwQGwD=Z1fB%ObfI&awT zVvs#+-&S}}J++F08r1N3PebA-nOzA ztljM&Cdnsz4P7&Yt2kD2=WiS)FjZt6!HCFrL&aXsqZA-z5 zyStQSrp`8mgYBg+ObW%@-I^&m`1Z?tf5g5?B0rsrR;K1(YUGk+=t4PO1-$TrAX>ub zp4I01--HjEdGM97qEf2g?Xk%V7Jt-)Z#F*Aa-_N1`N^Pb*B(|-A!Bs%KnoOC!p7ep z7>u;Wbvmdp3m=*;jBP*QnQwYBds@q}#INx%w#8}VGf7F}KxoOY0{~9<9w30}c&(b`xsO=fcAlY|cURrGjTq82GGVpLF}`5Y?&9;Jks@ zWPmyScafKDqs5&miKv>+%lqKXNtcWEC1z$yr7X7@F0}3P!$x>=Oa^hOy^g$Zt{h0* z>Dr}RFB0{(dIfE!%Z@q>xyo&IGtcT33`d+(-mG#yEsX1Y?8`IYbgRGSqsh}F2Ua;^ zJ1_`iLt_h_uC71$DdRa)`lbC%W`L$X|H1*54lU)^tvoo)4tI6ViN=f-(HD&GFjQSn z*usPB^it9R<|Iu6Wo)(Rv}J}U7(W0kn>86PDS5GkFzu}Zp#Pd{ue4bWM3dOQpIW}E zoHX;pPn0Oxa!Y4YU|4>dw61(yo|Fx+=YRl}o7%sgl)PwT8*36>DNRcK(!KSyodE-q zj)aCQRf=F0$k@EYypAAFSW|_k#Mxrqp%`vAo8P`Ig#v`1x_AuF4Idw8CTVAQ)uGy> z2eqCx(pjuexhVbttf~=>rN!f2Y=X~*01T$Hs3ao!o^6R-Ai}|yzkatHuyX-842A(1 zCajb9GAXLiK;6yw+_`1@W@oSNoDXKBXuPUC`_ug-sP`(y%nOqGu)IRYMZ zAW?u77%i@S)*K+aurzqO&FcJA_Eynjc`aRY5Vhv*z?2$*W7tIiNVmQ}FOm-qki0W6 zLgw4Ggv|kl5>iFX{!?RdrA<~02Zp0-oP3cHKEk8{yBn6|$VK1V_r2#1^avOnMv%Yz zS*jb%x`TK^W$_`q5Lcb}mB9-G*>B;_;xbHOI?|n&{2P3}4la7_l;FVtKb{8-@Wyac z@R=lbsSmG+3FgNQ+XZ*7jdsP@%tbD_&;FV{qCPfAYA(@X!-??1PpOJ2V>1+5UomSzS{MHurI zOEW_k%9x!yA5*Y+M)EKoV9fw4k|j6<i&+#Yu6)^xf#<{H!LC--sk_kog(kD+l9W`Lw*(QD+ zx3tL{pOie(dR$T@PaN{pn)3&@MwQ5#7$8q)qDy@lgn?fkkZE~B=@sNSm^EC&tiIlhDZ!{q0NgzFd7>V3Rv$w za;@&t%B@YvZ)SIMHV};JnBZD7q4lkqpAWb0AWKxi_kfz*;O-ALwS#136 zKI54Tz1aqHS0Sv7Fm5ntA0l@v`pslkcDR+@$mRChY4>#A1qzohbrhDDD$CQ$*Z&EL zzYCeNvpluYz9VKxSHP1vk$-0G?6~_6lS4gX_;C{QvMg~gbv_@n?Yj`JBmQyIlV&N! zOP92RVnku*GH0IRPokbcEbd6IALuE%698+fhkpXl-(+K~!`_ej(%)L!lZbQ?z)gJ1 zwRwh(LmzJ4O<0F^9GYI&aedy3lS)vOtXbWlh>78x(`|fq++E~@-FWQ)y=y`FX*3t&pOvBqr`_%>s3i z%k}68pr01LJ!+nTZ+xl~9xE16vvuRhL!|UAN3zN9wT>_Bak58BT}JM()@r{SLOioI zK&qOp3}Hft3;5sXR|(L!e%X2BF`CqtwN%rUx6h6ua;8T0x9vq^F^7}mp-EhJ9t zQbT$e%fV;`vXLCui2&}Lbcj*{j_V-s2*B~9*b{}kha$l+qw=S7_c%x3kH!hLvYKbL zblqg|hToN6G3t!0s4YPKF_UOFb0x{O72Lx1=EZ~kOIT%$MbQZ}IQPK@LC^UZ+$SOo{mASRKmGWk?iVh*f>hfyDMO1m=v4hOu2$s%cTn?^b=!2-(xS22 za;7f^TB`Zj#^Qko_#4LnijkTU##bEXxS0tsNVh`i2&1WUFwua_(zLd-GnK1?={CIu zbd7XIUE|RoFE$&vLi;6!6Rzh?I}@(q^4}z{rKAuL{VTKDCZa+g0&%;5yuS}cQnXKt zk`Ao|O{~O=p5i(tcHDgbO~=s70Q%_jO=vDzdAMGDbr4mVWU?lv63xW$#B7nfIQv@^l?M_8m=u)< za3Si``NY%^TQoAHTc-4W9EWDR((0SgU3kQvfCMn`t}(ZgZ6jdJ8M?LLnT+hb(ZGi|LN zZXNquw{vW->8fA4Ac))1p40jLu0HHXobnGe3V%A0 znWeIMEJ;=0LPL2a2#{B6;tSBl3WFvdGy5G-KWZpa_)^! zvYl=+;2GiEEO|Azl~s8LG>m7wmWtPxSZrM%v}H$4Rq~_wU71*stP1pKq^&d^c@jkP zZfG6}`ZeXC#rmKx>%=A2^BY!dZ=>%O$=$KvkVicY#VIZ;cJP`k4hSrF5$@7=$O`6P zTVRzhNAOr3=%_8U1LD}Wp zmd7g_Q12!V9PW_j;7-&WtoP1;n=B98( zAyjS2TeGe+oV>ZUN;}4L>T-)cEP)5JS=#akn4dFp)_>m1KmHlYz+Wqj*iDok^M7>6 z`-7HYQQm|LHxvICCcAcrqh{TUf>B7UF!6N0QfvEhoa1)!+eEI~VtRFlJ0_8cldFdwFQPaN<6?eANDZM!{RH(K<5xJ51bDy&C={ep4%+*XC`h?I%UK zoi@LM#=zSmjHS^P;~cv%3rk+cGukCR^D~l56!3>tS(LO=^cE^7!C@&pb=9BFY@3_L z`djadA9Na6zoIm>L~&Mu>6eqt4|q2&o!XGMwM#9fc7Y}3OgxUGo`MKX{ zEn_DUNmVFDvGkBOUAj+G7_g0bG3#=|y^cAVAR<+h^?ci+vTQR}Dsr;>ewh|QTI1-% z_#U(_e(1Ab-}Km_V$ZV&6ZT8*X@*JT;y>Vp4zn{*-u3Fc2~p|_+&*gUUCi<2dldc1 z)8X*N_&=1*?{81?73KcvX>lIFA2rXKj3ay}50$iu#yA!Q@2 w#RKDM-*;7Yk)Dkx3bbWCB!EY0)bGlciEtu?aK0`3ArZFrPtU;y3c>aN05P&pjsO4v literal 0 HcmV?d00001