diff --git a/README.md b/README.md index 037ab46..97cc2bf 100644 --- a/README.md +++ b/README.md @@ -94,11 +94,11 @@ Every panel is a tab you can drag, split, or stack. | Panel | Use it to | | --- | --- | -| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. With more than one canvas, pick the canvas they show, and that the Items panel edits, beside the preview. **Cut** puts the preview on program at once; **Transition** takes it there by the transition picked beside it, over the time set there (see [Transitions](#transitions)). **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | +| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. With more than one canvas, pick the canvas they show, and that the Items panel edits, beside the preview. **Cut** puts the preview on program at once; **Transition** takes it there by the transition picked beside it, over the time set there, or for a stinger by the media source and cut point set there (see [Transitions](#transitions)). **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | | Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still leaving it in a transition, cannot be removed, even by an undo, until program cuts away from it. | | History | See every edit, and undo or redo back to any of them. | | Items | Place, show, hide, and order what the preview scene or the top layer draws on the canvas shown, back to front. Select an item to filter or copy it. | -| Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. A media file reads as ended once it plays to its end without looping, **Play** plays it from its start, and **Reload** reads the file again. | +| Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. A media file reads as ended once it plays to its end without looping, **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again. | | Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. | | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. | | Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | @@ -140,12 +140,28 @@ layer and every scene's layout on it. | Slide | The incoming scene pushes the outgoing one off the canvas, both moving left, right, up, or down. | | Wipe | An edge sweeps across the canvas left, right, up, or down, uncovering the incoming scene behind it. | | Move | Each item showing a source both scenes place glides from its place in the outgoing scene to its place in the incoming one, resizing, recropping, and mixing its filters' settings on the way. If its blend mode or kinds of filter differ, it changes look halfway. The rest of the outgoing scene fades out and the rest of the incoming scene fades in. Items stack as the outgoing scene has them until halfway, then as the incoming one does. A source placed more than once pairs up in order, back to front. | +| Stinger | A media source's clip plays over the program from its start, and the program switches scenes under it at the cut point, the time into the clip when it covers the canvas. It lasts as long as the clip. | -All but a fade ease in and out. A transition is drawn on the program clock, so -the stream and a recording show the same frames of it, and a second +A slide, wipe, or move eases in and out. A transition is drawn on the program +clock, so the stream and a recording show the same frames of it, and a second **Transition** while one runs starts from the scene it was heading to. A transition set to 0 ms is a cut. +A stinger's clip fills the canvas, cropped evenly to the canvas's shape rather +than stretched, through its source's crop and filters, so a clip without an +alpha channel can be keyed. The cut point counts from the clip's first picture +or sound, however long the file takes to start decoding. Its sound joins the +mixes as its source's does. Its source waits blank and silent for the first +stinger rather than playing once when it opens, and rests ended between +stingers, which the Health panel does not count unless a scene on air places +it. A clip played to its end holds its last picture, as any media file does, +and a looping one stops, blank and silent, after one pass. A +clip that cannot play, or that is played again from the Sources panel or +reloaded while a stinger plays it, cuts at once; one stopped there ends the +stinger at once. A **Cut**, or another kind of transition, while a stinger +runs stops its clip, sound included; another stinger plays the clip again from +its start. + ### Keyframes An item's keyframes animate it: an item can fly in, fade in, or blur into @@ -317,8 +333,11 @@ as ended in Sources and, while the program shows or plays it, in Health. **Play** in Sources plays it from its start again, as does going on program with `restart`, in a scene on air or the top layer, when it was not on program before. Either clears its picture at once, so the end of the last play never -shows on program. A file that cannot be read or decoded fails, with the reason. -Its pictures and its sound each have a track of their own (see [Source +shows on program. **Stop** in Sources stops it at once: blank, silent, sound +already queued included, and reading as ended until played again. A file that +cannot be read or decoded fails, with the reason. +The source a [stinger](#transitions) plays waits, blank and silent, for the +first stinger instead of playing as it opens. Its pictures and its sound each have a track of their own (see [Source tracks](#source-tracks)), its pictures on one timeline across plays. Each is used apart, as a camera's and a microphone's are: the pictures while a scene on air shows it, the sound while the mix on program plays it, and either while @@ -445,9 +464,13 @@ height = 1920 top = [{ source = "logo", rect = { x = 780, y = 40, width = 280, height = 120 } }] [transition] -kind = "slide" # fade, slide, wipe, or move -duration = "300ms" +kind = "slide" # fade, slide, wipe, move, or stinger +duration = "300ms" # all but a stinger direction = "left" # slide and wipe: left, right, up, or down +# A stinger names a media source and its cut point instead: +# kind = "stinger" +# source = "sting" +# cut = "500ms" [sources.camera] kind = "camera" diff --git a/quest/m0/README.md b/quest/m0/README.md index 838bf5b..872b7dc 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -91,7 +91,6 @@ repository. - [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux - [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker -- [Stinger transitions](/quest/m0/stinger.md) - a clip played over the program that switches scenes at its cut point - [ML video effects](/quest/m0/effects.md) - background removal and other ML effects, baked into their source's track - [Destinations](/quest/m0/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse - [Multiple canvases](/quest/m0/multi-canvas.md) - landscape and portrait canvases composite at once from shared sources diff --git a/quest/m0/stinger.md b/quest/m0/stinger.md deleted file mode 100644 index 4c4fdf3..0000000 --- a/quest/m0/stinger.md +++ /dev/null @@ -1,16 +0,0 @@ -# [S] Stinger transitions - -## Goal - -A scene change in `moq-studio` can run a stinger: a video clip played over the -program that switches scenes at its cut point, as broadcast graphics do. - -## Plan - -Split from the keyframes quest on 2026-11-04 (maintainer). A -stinger is a transition kind beside fade, slide, wipe, and move -([moq-dev/studio#33](https://github.com/moq-dev/studio/pull/33)), drawn on the -program clock, whose clip is a media file source. It must start its clip as -the transition starts, or its cut point lands at random: `Feed::play` in -`rs/moq-scene/src/source.rs` starts a media source from its start at once. -Its settings (clip and cut point) change through `Edit`, so they undo. diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 4f9beda..6d5e23f 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -204,9 +204,9 @@ impl Collection { } /// Check what the types cannot: the canvas can be encoded, names are unique, - /// every item, clone, audio, and video entry names a source that exists, - /// and every crop and filter's settings are in range, each filter naming a - /// source of the kind it reads. + /// every item, clone, audio, and video entry names a source that exists, a + /// stinger names a media source, and every crop and filter's settings are + /// in range, each filter naming a source of the kind it reads. pub fn validate(&self) -> Result<(), Error> { self.canvas.validate()?; for (name, extra) in &self.canvases { @@ -323,6 +323,14 @@ impl Collection { self.validate_chain(&video.filters, &on)?; } + if let Some(name) = self.transition.stinger() + && !matches!(self.sources.get(name), Some(Source::Media { .. })) + { + return Err(Error::Invalid(format!( + "the stinger plays `{name}`, which is not a media source" + ))); + } + self.mixes.validate()?; for (name, audio) in &self.audio { if !self.sources.contains_key(name) { @@ -487,9 +495,9 @@ impl Extra { /// How the program changes when the preview is taken to it, short of a cut. /// -/// A fade runs at an even pace; the others ease in and out, so motion starts -/// and lands gently. One that takes no time is a cut. -#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +/// A fade runs at an even pace; the slide, wipe, and move ease in and out, so +/// motion starts and lands gently. One that takes no time is a cut. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)] pub enum Transition { /// The incoming scene fades in over the outgoing one. @@ -524,26 +532,49 @@ pub enum Transition { #[serde(with = "humantime_serde")] duration: Duration, }, + /// A media source's clip plays over the program from its start, and the + /// program switches scenes under it at `cut` into the clip, while it + /// covers the canvas. It lasts as long as the clip's first play, filling + /// each canvas it is drawn on, cropped to the canvas's shape, through the + /// source's own crop and filters. + Stinger { + /// The [`Source::Media`] whose clip plays. + source: String, + /// How far into the clip the scenes switch. + #[serde(with = "humantime_serde")] + cut: Duration, + }, } impl Transition { - /// How long it takes. - pub fn duration(&self) -> Duration { + /// How long it takes, or `None` for a stinger, which lasts as long as its + /// clip. + pub fn duration(&self) -> Option { match *self { Self::Fade { duration } | Self::Slide { duration, .. } | Self::Wipe { duration, .. } - | Self::Move { duration } => duration, + | Self::Move { duration } => Some(duration), + Self::Stinger { .. } => None, } } - /// How long it takes, to change. - pub fn duration_mut(&mut self) -> &mut Duration { + /// How long it takes, to change, or `None` for a stinger. + pub fn duration_mut(&mut self) -> Option<&mut Duration> { match self { Self::Fade { duration } | Self::Slide { duration, .. } | Self::Wipe { duration, .. } - | Self::Move { duration } => duration, + | Self::Move { duration } => Some(duration), + Self::Stinger { .. } => None, + } + } + + /// The media source a stinger plays. + pub fn stinger(&self) -> Option<&str> { + match self { + Self::Stinger { source, .. } => Some(source), + _ => None, } } @@ -1902,6 +1933,44 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } assert_eq!(collection.top[0].source, "bg"); } + /// A stinger names its media source and its cut point, and round-trips. + #[test] + fn reads_a_stinger() { + let text = r#" +version = 1 + +[canvas] +width = 1280 +height = 720 +framerate = 30 + +[transition] +kind = "stinger" +source = "sting" +cut = "800ms" + +[sources.sting] +kind = "media" +path = "sting.webm" +"#; + let collection = Collection::from_toml(text).unwrap(); + let stinger = Transition::Stinger { + source: "sting".into(), + cut: Duration::from_millis(800), + }; + assert_eq!(collection.transition, stinger); + assert_eq!(collection.transition.stinger(), Some("sting")); + assert_eq!(collection.transition.duration(), None); + assert_eq!( + Collection::from_toml(&collection.to_toml().unwrap()).unwrap(), + collection + ); + + let missing = text.replace("source = \"sting\"", "source = \"nothing\""); + let err = Collection::from_toml(&missing).unwrap_err().to_string(); + assert!(err.contains("`nothing`, which is not a media source"), "{err}"); + } + #[test] fn refuses_another_version() { let text = example().to_toml().unwrap().replacen("version = 1", "version = 2", 1); diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index e45e017..5ab516b 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -14,7 +14,7 @@ use crate::collection::{ }; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; use crate::overlay::{Overlay, Overlays}; -use crate::source::{self, Sources}; +use crate::source::{self, Feed, Sources, Status}; /// The format of every canvas texture: gamma-encoded RGBA, what /// `moq_video::render` produces and what its exporter hands an encoder. @@ -116,6 +116,24 @@ struct Switch { from_start: Instant, start: Instant, transition: Transition, + /// The play of its clip a stinger asked for, which the first render does. + played: Option, +} + +impl Switch { + /// Stop the clip a stinger played, unless another play took it over + /// since, so its sound ends with a stinger cut short, or with a looping + /// clip's first pass. + fn stop(&self, sources: &Sources) { + let (Transition::Stinger { source, .. }, Some(play)) = (&self.transition, self.played) else { + return; + }; + if let Some(feed) = sources.get(source).filter(|feed| feed.run().play == play) + && let Err(err) = feed.stop() + { + tracing::warn!(%source, %err, "cannot stop the stinger"); + } + } } /// A canvas-sized texture drawn into, and bound to sample. @@ -277,6 +295,11 @@ impl Compositor { self.switch.is_some() } + /// The media source a stinger under way plays over the program. + pub fn stinger(&self) -> Option<&str> { + self.switch.as_ref()?.transition.stinger() + } + /// The scenes program draws: the one on program, and the one a transition /// under way leaves. pub fn on_air(&self) -> impl Iterator { @@ -332,9 +355,12 @@ impl Compositor { .chain(leaving) } - /// Put the preview on program at once, ending any transition. - pub fn cut(&mut self) { - self.switch = None; + /// Put the preview on program at once, ending any transition, and + /// stopping a stinger's clip in `sources`. + pub fn cut(&mut self, sources: &Sources) { + if let Some(switch) = self.switch.take() { + switch.stop(sources); + } self.program_scene = self.preview_scene.clone(); self.program_start = None; } @@ -342,18 +368,66 @@ impl Compositor { /// Take the preview to program by `transition`, starting at `now`. /// /// A transition already under way lands on its scene first, so this one - /// starts from what that one was heading to. - pub fn transition(&mut self, transition: Transition, now: Instant) { + /// starts from what that one was heading to, and a stinger's clip in + /// `sources` stops. A stinger plays its clip from the start at the next + /// [`render`](Self::render), which switches the scenes `cut` into the clip + /// as it plays, and ends with the clip's play, leaving the clip holding its + /// last picture, or stopping a looping one; a clip that cannot play, or + /// fails, cuts at once instead. + pub fn transition(&mut self, transition: Transition, sources: &Sources, now: Instant) { + if let Some(switch) = self.switch.take() { + switch.stop(sources); + } let from = std::mem::replace(&mut self.program_scene, self.preview_scene.clone()); let from_start = self.program_start.replace(now).unwrap_or(now); - self.switch = (!transition.duration().is_zero()).then_some(Switch { + self.switch = (transition.duration() != Some(Duration::ZERO)).then_some(Switch { from, from_start, start: now, transition, + played: None, }); } + /// How far the transition under way has switched the scenes at `now`, + /// from 0 to 1 and eased, ending one that is over. A stinger switches + /// them at once at its cut point, its first call plays its clip, and its + /// end stops a looping one. + fn progress(&mut self, sources: &Sources, now: Instant) -> Option { + let switch = self.switch.as_mut()?; + let progress = match &switch.transition { + Transition::Stinger { source, cut } => { + let feed = sources.get(source); + match stinger(feed, *cut, &mut switch.played, now) { + Ok(Some(past)) => Some(if past { 1.0 } else { 0.0 }), + // Played to its end, a clip holds its last picture, as any + // media file does, but a looping one would play on, sound + // and all. + Ok(None) => { + if feed.is_some_and(|feed| matches!(feed.source(), Source::Media { looping: true, .. })) { + switch.stop(sources); + } + None + } + Err(err) => { + tracing::warn!(%source, %err, "cannot play the stinger; cutting instead"); + None + } + } + } + transition => { + let duration = transition.duration().expect("only a stinger is untimed"); + let elapsed = now.saturating_duration_since(switch.start); + let linear = (elapsed.as_secs_f32() / duration.as_secs_f32()).min(1.0); + (linear < 1.0).then(|| transition.progress(linear)) + } + }; + if progress.is_none() { + self.switch = None; + } + progress + } + /// Draw preview and program on every canvas as of `now`, from the newest /// picture each of `sources` has. /// @@ -361,6 +435,9 @@ impl Compositor { /// through [`error`](Self::error); this fails only when a canvas itself /// cannot be drawn. pub fn render(&mut self, collection: &Collection, sources: &Sources, now: Instant) -> Result<(), Error> { + // Before the pictures are pulled, so a stinger's clip starting now is + // cleared in this frame rather than showing where its last play ended. + let progress = self.progress(sources, now); self.update(collection, sources, now); self.paint(collection, now); @@ -369,14 +446,6 @@ impl Compositor { self.preview_shown = preview_since; let program_since = since(&mut self.program_start); - let linear = self.switch.as_ref().map(|switch| { - let elapsed = now.saturating_duration_since(switch.start); - (elapsed.as_secs_f32() / switch.transition.duration().as_secs_f32()).min(1.0) - }); - if linear.is_some_and(|linear| linear >= 1.0) { - self.switch = None; - } - let mut frame = self.effects.frame(); let mut draw = Draw { collection, @@ -389,9 +458,9 @@ impl Compositor { }; let preview = On(self.preview_scene.as_deref(), preview_since); let program = On(self.program_scene.as_deref(), program_since); - let switch = self.switch.as_ref().zip(linear).map(|(switch, linear)| { + let switch = self.switch.as_ref().zip(progress).map(|(switch, progress)| { let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); - (switch, from, linear) + (switch, from, progress) }); let extras = self.extras.iter().map(|(name, output)| (Some(name.as_str()), output)); for (canvas, output) in std::iter::once((None, &self.main)).chain(extras) { @@ -528,7 +597,8 @@ impl Compositor { } /// The sources whose pictures preview and program draw on any canvas: - /// each visible item's, and each mask's, through clones. + /// each visible item's, a stinger's clip under way, and each mask's, + /// through clones. fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { let leaving = self.switch.as_ref().and_then(|switch| switch.from.as_deref()); let scenes: Vec> = [self.preview_scene.as_deref(), self.program_scene.as_deref(), leaving] @@ -539,6 +609,13 @@ impl Compositor { .collect(); let canvases = std::iter::once(None).chain(self.extras.keys().map(Some)); let mut shown = HashSet::new(); + let masks = |shown: &mut HashSet<&'a str>, filters: &mut dyn Iterator| { + for filter in filters { + if let Filter::Mask { source, .. } = filter { + shown.insert(collection.capture(source)); + } + } + }; for canvas in canvases { for scene in &scenes { let place = Place { @@ -548,18 +625,75 @@ impl Compositor { let items = collection.items(&place).unwrap_or_default(); for item in items.iter().filter(|item| item.visible) { shown.insert(collection.capture(&item.source)); - for filter in collection.filters(item) { - if let Filter::Mask { source, .. } = filter { - shown.insert(collection.capture(source)); - } - } + masks(&mut shown, &mut collection.filters(item)); } } } + // Drawn through its source's filters alone. + if let Some((name, _)) = self.stinger().and_then(|name| collection.sources.get_key_value(name)) { + shown.insert(name); + let video = collection.video.get(name); + masks(&mut shown, &mut video.into_iter().flat_map(|video| &video.filters)); + } shown } } +/// Whether a stinger playing `feed`'s clip is past `cut` into it at `now`, or +/// `None` once the clip's play has ended. The first call plays the clip from +/// its start, noting which play it is in `played`. Fails when the clip cannot +/// play, or another play or a reload of the source cut it off. +fn stinger(feed: Option<&Feed>, cut: Duration, played: &mut Option, now: Instant) -> Result, Error> { + let feed = feed.ok_or_else(|| Error::Stopped("the source is not running".into()))?; + let play = match *played { + Some(play) => play, + None => { + feed.play()?; + *played.insert(feed.run().play) + } + }; + if let Status::Failed(reason) = feed.status() { + return Err(Error::Stopped(reason)); + } + let run = feed.run(); + if run.play != play { + return Err(Error::Media("the clip was played again or reloaded".into())); + } + if run.ended.is_some_and(|ended| ended <= now) { + return Ok(None); + } + Ok(Some( + run.begun + .is_some_and(|begun| now.saturating_duration_since(begun) >= cut), + )) +} + +/// The crop that trims what `crop` leaves of a `size` picture evenly from two +/// opposite sides to `canvas`'s shape, so it covers the canvas unstretched. +fn cover(crop: Crop, size: (u32, u32), canvas: &Canvas) -> Crop { + let width = size.0 as f32 * (1.0 - crop.left - crop.right); + let height = size.1 as f32 * (1.0 - crop.top - crop.bottom); + let (shown, wanted) = (width / height, canvas.width as f32 / canvas.height as f32); + if !shown.is_finite() || shown <= 0.0 { + return Crop::default(); + } + if shown > wanted { + let trim = (1.0 - wanted / shown) / 2.0; + Crop { + left: trim, + right: trim, + ..Crop::default() + } + } else { + let trim = (1.0 - shown / wanted) / 2.0; + Crop { + top: trim, + bottom: trim, + ..Crop::default() + } + } +} + /// How far moving `amount` of `canvas`'s width or height in `direction` goes, /// in whole pixels so a moving scene stays sharp and two meet without a seam. fn offset(direction: Direction, amount: f32, canvas: &Rect) -> (f32, f32) { @@ -699,8 +833,8 @@ struct Draw<'a> { impl Draw<'_> { /// Draw `output`'s preview, showing `preview`, and its program, showing - /// `program`, or `switch` to it from the scene it leaves, `linear` of its - /// time along. + /// `program`, or `switch` to it from the scene it leaves, `progress` of + /// the way there. fn output(&mut self, output: &Output, preview: On, program: On, switch: Option<(&Switch, On, f32)>) { let top = self.place(None); // The top layer goes over both outputs, and over a transition rather @@ -711,10 +845,9 @@ impl Draw<'_> { let to = program; match switch { - Some((switch, from, linear)) => { - let progress = switch.transition.progress(linear); + Some((switch, from, progress)) => { let canvas = Rect::fill(&self.size); - match switch.transition { + match &switch.transition { Transition::Fade { .. } => { self.whole(&output.incoming, to); self.target(&output.program); @@ -726,8 +859,8 @@ impl Draw<'_> { self.whole(&output.incoming, to); self.target(&output.program); self.scene(&output.program, from); - let (x, y) = offset(direction, progress, &canvas); - let (dx, dy) = offset(direction, 1.0, &canvas); + let (x, y) = offset(*direction, progress, &canvas); + let (dx, dy) = offset(*direction, 1.0, &canvas); let uncovered = shifted(canvas, x - dx, y - dy); self.picture(&output.program, &output.incoming, canvas, uncovered, 1.0); } @@ -735,8 +868,8 @@ impl Draw<'_> { self.whole(&output.incoming, to); self.whole(&output.outgoing, from); self.target(&output.program); - let (x, y) = offset(direction, progress, &canvas); - let (dx, dy) = offset(direction, 1.0, &canvas); + let (x, y) = offset(*direction, progress, &canvas); + let (dx, dy) = offset(*direction, 1.0, &canvas); self.picture(&output.program, &output.outgoing, shifted(canvas, x, y), canvas, 1.0); let entering = shifted(canvas, x - dx, y - dy); self.picture(&output.program, &output.incoming, entering, canvas, 1.0); @@ -745,6 +878,11 @@ impl Draw<'_> { self.target(&output.program); self.moving(&output.program, from, to, progress); } + Transition::Stinger { source, .. } => { + self.target(&output.program); + self.scene(&output.program, if progress < 1.0 { from } else { to }); + self.stinger(&output.program, source); + } } } None => { @@ -800,6 +938,23 @@ impl Draw<'_> { } } + /// Draw the picture of `source`, a stinger's clip, over the whole of + /// `target`, cropped to the shape of the canvas drawn now, through the + /// source's crop and filters. Nothing before its first picture. + fn stinger(&mut self, target: &Target, source: &str) { + let canvas = self.size; + let Some(picture) = self.library.0.get(source).and_then(|layer| layer.picture.as_ref()) else { + return; + }; + let crop = self.collection.video.get(source).map(|video| video.crop); + let item = Item { + crop: cover(crop.unwrap_or_default(), picture.size(), &canvas), + ..Item::new(source, Rect::fill(&canvas)) + }; + let pose = Pose::of(self.collection, &item); + self.posed(target, &item, pose); + } + /// Draw `item`, posed, where it is placed onto `target`, shared with every /// other draw of it under `key`. fn item(&mut self, target: &Target, key: (Place, usize, Settled), item: &Item) { @@ -1019,7 +1174,7 @@ mod tests { use super::*; use crate::collection::{Blend, Extra, MaskMode, Rgba, Scene}; - use crate::source::test::{context, local, start}; + use crate::source::test::{context, listen, local, start}; /// A fade over a second. const FADE: Transition = Transition::Fade { @@ -1083,10 +1238,12 @@ mod tests { /// A compositor over running sources, read back to check pixels. pub(super) struct Studio { pub(super) collection: Collection, - sources: Sources, + pub(super) sources: Sources, pub(super) compositor: Compositor, pub(super) device: wgpu::Device, queue: wgpu::Queue, + /// Mixes the sources' sound, only as a test advances it. + mixer: moq_audio::bus::Driver, _origin: moq_net::origin::Producer, _dir: tempfile::TempDir, } @@ -1101,6 +1258,8 @@ mod tests { let (device, queue) = gpu().await; let (mut context, origin) = context(dir.path()); context.output = moq_video::render::output(&device); + let (bus, mixer) = moq_audio::bus::Bus::new(Default::default()).unwrap(); + context.bus = bus; let mut sources = Sources::new(context); let drivers = sources.sync(&collection); start(&sources, drivers).await; @@ -1110,6 +1269,7 @@ mod tests { queue, collection, sources, + mixer, _origin: origin, _dir: dir, } @@ -1182,7 +1342,7 @@ mod tests { local(async { let mut studio = Studio::new(collection()).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(Instant::now()); let program = studio.program(); assert_eq!(pixel(&program, 1, 1), RED); @@ -1212,7 +1372,7 @@ mod tests { collection.scenes[0].items.truncate(1); let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(Instant::now()); let actual = pixel(&studio.program(), 1, 1); assert!(near(actual, [128, 0, 0, 255]), "{actual:?}"); @@ -1229,9 +1389,9 @@ mod tests { let mut studio = Studio::new(collection()).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, start); + studio.compositor.transition(FADE, &studio.sources, start); assert_eq!(studio.compositor.program_scene(), Some("blue")); studio.render(start + Duration::from_millis(500)); @@ -1282,7 +1442,7 @@ mod tests { let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(Instant::now()); let program = studio.program(); let actual = pixel(&program, 1, 6); @@ -1303,12 +1463,12 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(start); assert_eq!(pixel(&studio.program(), 1, 6), BLUE); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, start); + studio.compositor.transition(FADE, &studio.sources, start); studio.render(start + Duration::from_millis(500)); let program = studio.program(); let actual = pixel(&program, 1, 1); @@ -1356,14 +1516,14 @@ mod tests { let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(start); assert_eq!(pixel(&studio.program(), 1, 1), RED); assert_eq!(tall(&mut studio), (BLUE, RED)); assert_eq!(studio.compositor.layers.len(), 2, "each source is uploaded once"); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, start); + studio.compositor.transition(FADE, &studio.sources, start); studio.render(start + Duration::from_millis(500)); let (fading, kept) = tall(&mut studio); assert!(near(fading, [0, 0, 128, 255]), "{fading:?}"); @@ -1393,10 +1553,10 @@ mod tests { assert_eq!(pixel(&studio.program(), 1, 1), BLACK); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, start); + studio.compositor.transition(FADE, &studio.sources, start); studio.render(start + Duration::from_millis(500)); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); assert!(!studio.compositor.transitioning()); studio.render(start + Duration::from_millis(600)); assert_eq!(pixel(&studio.program(), 1, 1), RED); @@ -1455,9 +1615,9 @@ mod tests { ] { let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(transition, start); + studio.compositor.transition(transition.clone(), &studio.sources, start); studio.render(start + Duration::from_millis(500)); let program = studio.program(); let actual = [(1, 1), (1, 6), (6, 1), (6, 6)].map(|(x, y)| pixel(&program, x, y)); @@ -1503,10 +1663,12 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); - studio.compositor.transition(Transition::Move { duration }, start); + studio + .compositor + .transition(Transition::Move { duration }, &studio.sources, start); // Eased halfway, blue is at (2, 2) and both fades are at half. studio.render(start + Duration::from_millis(500)); @@ -1586,7 +1748,7 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); studio.render(at(0)); @@ -1594,7 +1756,7 @@ mod tests { assert_eq!(tall(&mut studio, true), (BLUE, BLACK), "halfway in, in preview"); assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the main canvas too"); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(at(500)); assert_eq!(tall(&mut studio, false), (BLACK, BLACK), "on program from the start"); assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "the main canvas too"); @@ -1620,7 +1782,7 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); studio.render(at(0)); @@ -1635,7 +1797,7 @@ mod tests { assert_eq!(pixel(&preview, 1, 6), BLUE, "halfway in"); assert_eq!(pixel(&preview, 6, 6), BLACK); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(at(500)); assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "on program from the start"); assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the preview carries on"); @@ -1674,7 +1836,7 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(start); // Program has reached the jump; preview starts over before it. studio.compositor.set_preview(Some("blue".into())); @@ -1698,11 +1860,11 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("red".into())); studio.render(at(0)); - studio.compositor.transition(FADE, at(9000)); + studio.compositor.transition(FADE, &studio.sources, at(9000)); studio.render(at(9500)); let program = studio.program(); // Red has slid halfway in over the left half, blue is in place, and @@ -1733,10 +1895,12 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); - studio.compositor.transition(Transition::Move { duration }, start); + studio + .compositor + .transition(Transition::Move { duration }, &studio.sources, start); studio.render(start); let program = studio.program(); @@ -1763,7 +1927,7 @@ mod tests { local(async { let mut studio = Studio::new(collection()).await; studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(Instant::now()); assert!(studio.sources.get("blue").unwrap().take().is_none(), "drawn once"); studio.render(Instant::now()); @@ -1789,6 +1953,267 @@ mod tests { .await; } + /// Scenes "red" on program and "blue" in preview, with a stinger playing + /// `file` as the media source "sting", cutting 300 ms into it. + async fn stinging(file: &[u8]) -> Studio { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join("sting.mkv"), file).unwrap(); + let mut collection = collection(); + let media = Source::Media { + path: "sting.mkv".into(), + looping: false, + restart: false, + }; + collection.sources.insert("sting".into(), media); + collection.transition = Transition::Stinger { + source: "sting".into(), + cut: Duration::from_millis(300), + }; + collection.validate().unwrap(); + let mut studio = Studio::with_files(collection, dir).await; + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(&studio.sources); + studio.compositor.set_preview(Some("blue".into())); + studio + } + + /// Darken or hide the stinger's clip, through its source's filters. + fn sting(studio: &mut Studio, filter: Filter) { + let video = crate::collection::Video { + filters: vec![filter], + ..Default::default() + }; + studio.collection.video.insert("sting".into(), video); + } + + /// A stinger plays its clip from its start over the program as it starts, + /// switches the scenes under it at its cut point into the clip, and ends + /// with the clip, which holds its last picture. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_stinger_cuts_under_its_clip() { + local(async { + let mut studio = stinging(crate::media::tests::CLIP).await; + let feed = studio.sources.get("sting").unwrap(); + assert_eq!(feed.status(), Status::Ended, "waiting for the stinger"); + assert_eq!(feed.run(), source::Run::default()); + + let transition = studio.collection.transition.clone(); + studio + .compositor + .transition(transition, &studio.sources, Instant::now()); + assert_eq!(studio.compositor.program_scene(), Some("blue")); + + // What the clip's filters read is drawn too, such as an overlay to paint. + let mask = Filter::Mask { + source: "red".into(), + mode: MaskMode::Alpha, + }; + let mut collection = studio.collection.clone(); + collection.scenes.retain(|scene| scene.name == "blue"); + collection.video.insert( + "sting".into(), + crate::collection::Video { + filters: vec![mask], + ..Default::default() + }, + ); + let shown = studio.compositor.shown(&collection); + assert!(shown.contains("sting") && shown.contains("red"), "{shown:?}"); + + sting(&mut studio, Filter::Brightness { amount: 0.0 }); + // Black over the whole canvas once its first picture arrives. + loop { + studio.render(Instant::now()); + assert!(studio.compositor.transitioning()); + assert_eq!(studio.compositor.stinger(), Some("sting")); + if studio.program().chunks(4).all(|pixel| pixel == BLACK) { + break; + } + kio::clock::sleep(Duration::from_millis(5)).await; + } + let begun = studio.sources.get("sting").unwrap().run().begun.unwrap(); + + // Seen through, the outgoing scene shows until the cut point. + sting(&mut studio, Filter::Opacity { amount: 0.0 }); + studio.render(begun + Duration::from_millis(299)); + assert_eq!(pixel(&studio.program(), 1, 1), RED); + studio.render(begun + Duration::from_millis(300)); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + assert!(studio.compositor.transitioning(), "until the clip ends"); + + let ended = loop { + if let Some(ended) = studio.sources.get("sting").unwrap().run().ended { + break ended; + } + kio::clock::sleep(Duration::from_millis(5)).await; + }; + assert!(ended >= begun + Duration::from_secs(1), "{:?}", ended - begun); + studio.render(ended); + assert!(!studio.compositor.transitioning()); + assert_eq!(studio.compositor.stinger(), None); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + + // Played to its end, the clip rests ended on its last picture, not + // blank, as any media file that played once does. + while studio.sources.get("sting").unwrap().status() != Status::Ended { + kio::clock::sleep(Duration::from_millis(5)).await; + } + studio.render(Instant::now()); + let held = studio.compositor.layers["sting"].picture.as_ref().unwrap().size(); + assert_eq!(held, (64, 64), "holds its last picture"); + + // The next stinger starts blank, never on the last one's end. + sting(&mut studio, Filter::Brightness { amount: 0.0 }); + studio.compositor.set_preview(Some("red".into())); + let transition = studio.collection.transition.clone(); + studio + .compositor + .transition(transition, &studio.sources, Instant::now()); + studio.render(Instant::now()); + assert!(studio.compositor.transitioning()); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + }) + .await; + } + + /// A stinger whose clip cannot play cuts at once, as does one whose clip + /// is played again from elsewhere, such as the Sources panel. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_stinger_that_cannot_play_cuts() { + local(async { + let mut studio = stinging(b"not a video").await; + let transition = studio.collection.transition.clone(); + studio + .compositor + .transition(transition, &studio.sources, Instant::now()); + studio.render(Instant::now()); + assert!(!studio.compositor.transitioning()); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + + let mut studio = stinging(crate::media::tests::CLIP).await; + let transition = studio.collection.transition.clone(); + studio + .compositor + .transition(transition, &studio.sources, Instant::now()); + studio.render(Instant::now()); + assert!(studio.compositor.transitioning()); + studio.sources.get("sting").unwrap().play().unwrap(); + studio.render(Instant::now()); + assert!(!studio.compositor.transitioning()); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + }) + .await; + } + + /// A **Cut** mid-stinger stops its clip, sound it queued included, so + /// nothing of it plays on the program mix. Stopped from elsewhere, the + /// clip ends its stinger at once, and a looping clip stops with its + /// stinger after one pass. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_stinger_stops_its_clip() { + local(async { + let mut studio = stinging(crate::media::tests::CLIP).await; + let program = studio.sources.mix(crate::Mixes::DEFAULT).unwrap().clone(); + let mut at = Instant::now(); + studio.mixer.advance(480); + let transition = studio.collection.transition.clone(); + studio + .compositor + .transition(transition, &studio.sources, Instant::now()); + loop { + studio.render(Instant::now()); + let until = at + Duration::from_millis(10); + if listen(&mut studio.mixer, &program, &mut at, until).await > 0.0 { + break; + } + } + studio.compositor.cut(&studio.sources); + assert!(!studio.compositor.transitioning()); + let feed = studio.sources.get("sting").unwrap(); + while feed.status() != Status::Ended { + kio::clock::sleep(Duration::from_millis(5)).await; + } + let until = at + Duration::from_millis(500); + let heard = listen(&mut studio.mixer, &program, &mut at, until).await; + assert_eq!(heard, 0.0, "the clip's sound stops with the cut"); + + // Stopped from the Sources panel, the stinger ends with it. + studio.compositor.set_preview(Some("red".into())); + let transition = studio.collection.transition.clone(); + studio + .compositor + .transition(transition, &studio.sources, Instant::now()); + studio.render(Instant::now()); + assert!(studio.compositor.transitioning()); + studio.sources.get("sting").unwrap().stop().unwrap(); + studio.render(Instant::now()); + assert!(!studio.compositor.transitioning()); + assert_eq!(pixel(&studio.program(), 1, 1), RED); + + // A looping clip plays one pass, then stops with the stinger. + let looping = Source::Media { + path: "sting.mkv".into(), + looping: true, + restart: false, + }; + studio.collection.sources.insert("sting".into(), looping); + let drivers = studio.sources.sync(&studio.collection); + start(&studio.sources, drivers).await; + studio.compositor.set_preview(Some("blue".into())); + let transition = studio.collection.transition.clone(); + studio + .compositor + .transition(transition, &studio.sources, Instant::now()); + while studio.compositor.transitioning() { + studio.render(Instant::now()); + let until = at + Duration::from_millis(10); + listen(&mut studio.mixer, &program, &mut at, until).await; + } + let feed = studio.sources.get("sting").unwrap(); + assert!(feed.run().ended.is_some(), "after its first pass"); + while feed.status() != Status::Ended { + kio::clock::sleep(Duration::from_millis(5)).await; + } + let until = at + Duration::from_millis(500); + let heard = listen(&mut studio.mixer, &program, &mut at, until).await; + assert_eq!(heard, 0.0, "the loop stops with the stinger"); + }) + .await; + } + + /// A stinger's clip covers a canvas of another shape unstretched, trimmed + /// evenly from the sides that overflow, after its source's own crop. + #[test] + fn a_stinger_covers_the_canvas() { + let portrait = Canvas { + width: 1080, + height: 1920, + framerate: 30, + }; + let crop = cover(Crop::default(), (1920, 1080), &portrait); + let kept = 1.0 - crop.left - crop.right; + assert!((kept - (1080.0 / 1920.0) / (1920.0 / 1080.0)).abs() < 1e-6, "{crop:?}"); + assert_eq!((crop.left, crop.top, crop.bottom), (crop.right, 0.0, 0.0)); + + let landscape = Canvas::default(); + let crop = cover(Crop::default(), (1080, 1920), &landscape); + assert_eq!((crop.left, crop.right), (0.0, 0.0)); + assert!(crop.top > 0.0 && crop.top == crop.bottom, "{crop:?}"); + assert_eq!(cover(Crop::default(), (1920, 1080), &landscape), Crop::default()); + + // A source cropped to a square is trimmed from what it leaves. + let square = Crop { + left: 0.21875, + right: 0.21875, + ..Crop::default() + }; + let crop = cover(square, (1920, 1080), &landscape); + assert!((crop.top - 0.21875).abs() < 1e-6, "{crop:?}"); + } + /// A picture the GPU cannot hold says why, rather than leaving its source /// live and missing from the canvas. #[tokio::test] @@ -1822,7 +2247,7 @@ mod tests { async fn drawn(collection: Collection, dir: tempfile::TempDir) -> Vec { let mut studio = Studio::with_files(collection, dir).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(Instant::now()); studio.program() } @@ -2280,7 +2705,7 @@ mod tests { ); let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(Instant::now()); let old = studio.compositor.program().clone(); @@ -2352,9 +2777,9 @@ mod tests { let mut studio = Studio::new(collection()).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, start); + studio.compositor.transition(FADE, &studio.sources, start); // `blue` takes the name `red` gives up, which is followed once. let mut draft = studio.collection.draft("Rename"); @@ -2440,12 +2865,12 @@ mod tests { collection.scenes[1].items = vec![Item::new("overlay", Rect::fill(&collection.canvas))]; let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); // Opened here, but drawn nowhere until the cut. let start = Instant::now(); studio.render(start); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); for (at, expected) in [(500, [128, 0, 128, 255]), (750, [64, 0, 191, 255]), (2000, BLUE)] { studio.render(start + Duration::from_millis(at)); @@ -2487,10 +2912,12 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); - studio.compositor.transition(Transition::Move { duration }, start); + studio + .compositor + .transition(Transition::Move { duration }, &studio.sources, start); studio.render(start + Duration::from_millis(500)); let program = studio.program(); @@ -2526,11 +2953,11 @@ mod tests { // Paint the aside overlay, then leave it off air. studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(start); assert_eq!(pixel(&studio.program(), 3, 3), [255, 255, 0, 255]); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(start); studio.compositor.overlays.as_ref().unwrap().panic(); @@ -2541,7 +2968,7 @@ mod tests { } studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); studio.render(start); assert_eq!(pixel(&studio.program(), 3, 3), RED, "no frozen overlay"); }) @@ -2608,7 +3035,7 @@ mod bench { studio.compositor.set_canvases(&studio.collection).unwrap(); let scene = studio.collection.scenes[0].name.clone(); studio.compositor.set_preview(Some(scene)); - studio.compositor.cut(); + studio.compositor.cut(&studio.sources); let mut total = Duration::ZERO; for frame in 0..FRAMES * 2 { let start = Instant::now(); diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index fb00dc9..91e1812 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -244,7 +244,10 @@ impl Change { after: Some(a.clone()), }, (Self::Transition { before, after }, Self::Transition { before: b, after: a }) if after == *b => { - Self::Transition { before, after: *a } + Self::Transition { + before, + after: a.clone(), + } } (Self::Canvas { before, after }, Self::Canvas { before: b, after: a }) if after == *b => { Self::Canvas { before, after: *a } @@ -294,7 +297,7 @@ impl Change { if collection.transition != *before { return Err(conflict("the transition changed".into())); } - collection.transition = *after; + collection.transition = after.clone(); Ok(()) } Self::Canvas { before, after } => { @@ -501,8 +504,9 @@ impl Draft { } /// Remove the source called `name` and everything that refers to it: its - /// clones, the items showing either, their audio and video settings, and - /// the filters reading it, such as a mask. + /// clones, the items showing either, their audio and video settings, the + /// filters reading it, such as a mask, and a stinger playing it, which + /// becomes the default fade. pub fn remove_source(&mut self, name: &str) -> Result<(), Error> { let clones: Vec = self .collection @@ -563,6 +567,9 @@ impl Draft { if self.collection.video.contains_key(name) { self.set_video(name, None)?; } + if self.collection.transition.stinger() == Some(name) { + self.set_transition(Transition::default())?; + } self.set_source(name, None) } @@ -662,7 +669,7 @@ impl Draft { /// Set how the preview is taken to program, short of a cut. pub fn set_transition(&mut self, transition: Transition) -> Result<(), Error> { - let before = self.collection.transition; + let before = self.collection.transition.clone(); self.change(Change::Transition { before, after: transition, @@ -1344,6 +1351,52 @@ mod tests { assert!(matches!(draft.finish(), Err(Error::Invalid(_)))); } + /// A stinger's clip and cut point are edits, it plays only a media source, + /// and removing that source sets the default fade back, undoing whole. + #[test] + fn a_stinger_follows_its_source() { + let mut collection = collection(); + let media = Source::Media { + path: "sting.webm".into(), + looping: false, + restart: false, + }; + collection.sources.insert("sting".into(), media); + let start = collection.clone(); + let mut history = History::new(); + let stinger = |millis| Transition::Stinger { + source: "sting".into(), + cut: std::time::Duration::from_millis(millis), + }; + for millis in [400, 500] { + let mut draft = collection.draft("Set the transition"); + draft.set_transition(stinger(millis)).unwrap(); + commit(&mut collection, &mut history, draft, None); + } + assert_eq!(collection.transition, stinger(500)); + + let mut draft = collection.draft("Set the transition"); + draft + .set_transition(Transition::Stinger { + source: "red".into(), + cut: std::time::Duration::ZERO, + }) + .unwrap(); + let err = draft.finish().unwrap_err().to_string(); + assert!(err.contains("not a media source"), "{err}"); + + let removed = collection.clone(); + let mut draft = collection.draft("Remove sting"); + draft.remove_source("sting").unwrap(); + commit(&mut collection, &mut history, draft, None); + assert_eq!(collection.transition, Transition::default()); + history.undo(&mut collection).unwrap(); + assert_eq!(collection, removed); + history.undo(&mut collection).unwrap(); + history.undo(&mut collection).unwrap(); + assert_eq!(collection, start); + } + /// A further canvas is added, laid out, and resized as edits; removing a /// source takes its items off every canvas; and removing the canvas takes /// its top layer and every scene's layout on it, undoing whole. A layout diff --git a/rs/moq-scene/src/effect.rs b/rs/moq-scene/src/effect.rs index 94518ee..b2f010e 100644 --- a/rs/moq-scene/src/effect.rs +++ b/rs/moq-scene/src/effect.rs @@ -323,6 +323,15 @@ fn reach(sigma: f32) -> f32 { pub(crate) struct Picture { view: wgpu::TextureView, bind: wgpu::BindGroup, + /// Its width and height in pixels. + size: (u32, u32), +} + +impl Picture { + /// Its width and height in pixels. + pub fn size(&self) -> (u32, u32) { + self.size + } } /// A LUT source's table on the GPU. @@ -677,7 +686,11 @@ impl Effects { }, ], }); - Picture { view, bind } + Picture { + view, + bind, + size: (texture.width(), texture.height()), + } } /// Upload a LUT's table. diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 8c1a0ab..98748c6 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -86,6 +86,42 @@ struct Play { /// ended holds its take no longer. type Next = Arc>>; +/// A media source's latest play: which it is, and when it began and ended on +/// the program clock. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Run { + /// Which play it is: 0 for the one the source opens with, and one more + /// for each [`Feed::play`] since. + pub play: u64, + /// When its first picture or sound was due, or `None` until it is decoded. + pub begun: Option, + /// When its last picture or sound finished, a looping one's first pass + /// did, or [`Feed::stop`] stopped it, or `None` until then. A play cut off + /// by the next never ends. + pub ended: Option, +} + +/// What a media source's driver is asked to do, which the feed sends. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +struct Cue { + /// The play, as [`Run::play`] counts them. + play: u64, + /// Whether it is stopped, blank and silent until the next play. + stopped: bool, +} + +/// A media source's latest [`Run`], which the feed starts and only the driver +/// playing that one writes, and whether the feed stopped it. +#[derive(Default)] +struct Latest { + run: Run, + stopped: bool, +} + +/// The feed changes it, and blanks the picture, under its lock, which the +/// driver takes to show a picture, so a play cut off never shows over that. +type Runs = Arc>; + /// A picture that shows nothing, for a replay source between replays. fn clear() -> Picture { Picture::Rgba { @@ -218,7 +254,9 @@ impl Sources { /// Run exactly the sources and mixes `collection` has: start new and edited /// sources and stop removed and edited ones. A clone shares the feed of the /// source it clones rather than opening one. Returns a driver for each - /// source started, which does nothing until run. + /// source started, which does nothing until run. The media source a + /// stinger plays opens silent and blank, waiting for its first + /// [play](Feed::play), rather than playing once at the start. /// /// Also applies the collection's gain, mute, delay, and routing to every /// audio strip, and its frame rate to every web page, which never restarts @@ -245,7 +283,8 @@ impl Sources { let mut drivers = Vec::new(); for (name, source) in collection.sources.iter().filter(|(_, source)| source.opens()) { if !self.feeds.contains_key(name) { - let (feed, driver) = open(name, source, &self.context); + let held = collection.transition.stinger() == Some(name.as_str()); + let (feed, driver) = open(name, source, &self.context, held); self.feeds.insert(name.clone(), feed); drivers.push(driver); } @@ -340,8 +379,10 @@ pub struct Feed { played: Playback, /// A replay source's next replay. plays: Option>, - /// Marked changed to play a media source from its start. - starts: Option>, + /// Sent each play or stop a media source is asked for. + cues: Option>, + /// A media source's latest play. + runs: Runs, /// Set once a media source's file turns out to hold no sound. silent: Arc, /// A broadcast source's playout clock. @@ -390,21 +431,64 @@ impl Feed { } /// Play a media source from its start, now, cutting off wherever it is. - /// One that ended plays again, and one that loops loops from there. + /// One that ended or stopped plays again, and one that loops loops from + /// there. /// - /// Refused by any other kind of source, and by a media source that stopped. + /// Refused by any other kind of source, and by a media source whose driver + /// is gone, such as after it failed. pub fn play(&self) -> Result<(), Error> { - let Some(starts) = &self.starts else { + let Some(cues) = &self.cues else { return Err(Error::Unsupported("only a media source plays from its start".into())); }; - if starts.is_closed() { + if cues.is_closed() { return Err(self.stopped()); } + // Counted before the driver hears of it, so whatever the play before + // still reports is not taken for this one's. + let mut latest = self.runs.lock().unwrap(); + *latest = Latest { + run: Run { + play: latest.run.play + 1, + ..Run::default() + }, + stopped: false, + }; // Cleared before the driver hears of it, so the next frame drawn, such // as the first of a scene going on program, shows nothing rather than // where the last play was, and the new first picture replaces it. *self.picture.lock().unwrap() = Some(clear()); - starts.send(()).map_err(|_| self.stopped()) + let cue = Cue { + play: latest.run.play, + stopped: false, + }; + cues.send(cue).map_err(|_| self.stopped()) + } + + /// Stop a media source now: blank and silent, dropping any sound it has + /// queued, until the next [`play`](Self::play). The play under way ends + /// now. + /// + /// Refused by any other kind of source. + pub fn stop(&self) -> Result<(), Error> { + let Some(cues) = &self.cues else { + return Err(Error::Unsupported("only a media source stops".into())); + }; + let mut latest = self.runs.lock().unwrap(); + latest.run.ended.get_or_insert_with(kio::clock::now); + latest.stopped = true; + *self.picture.lock().unwrap() = Some(clear()); + // Kept even with the driver gone, which plays nothing anyway. + cues.send_replace(Cue { + play: latest.run.play, + stopped: true, + }); + Ok(()) + } + + /// A media source's latest play, from its start or the latest + /// [`play`](Self::play). Never begins for any other source. + pub fn run(&self) -> Run { + self.runs.lock().unwrap().run } /// Why the driver is gone, from the reason it left in the status. @@ -500,7 +584,8 @@ pub struct Driver { track: Option, played: Playback, plays: Option>, - starts: Option>, + cues: Option>, + runs: Runs, silent: Arc, clock: Option, } @@ -616,9 +701,8 @@ impl Driver { std::future::pending().await } Source::Media { path, looping, .. } => { - let starts = self.starts.clone().expect("a media source has starts"); - self.media(&self.context.dir.join(path), *looping, starts, listing) - .await + let cues = self.cues.clone().expect("a media source has cues"); + self.media(&self.context.dir.join(path), *looping, cues, listing).await } Source::Broadcast { path } => self.broadcast(path).await, Source::Browser { url, width, height } => self.browser(url, *width, *height).await, @@ -1232,9 +1316,10 @@ impl Driver { } } - /// Play the media file at `path` from its start, and again each time - /// `starts` changes, cutting off the play under way. At its end it starts - /// over when `looping`, or holds its last picture, silent, until the next. + /// Play the media file at `path` as `cues` asks: from its start for each + /// play, cutting off the play under way, and blank and silent while + /// stopped. At its end it starts over when `looping`, or holds its last + /// picture, silent, until the next. /// /// With `listing`, the pictures are also encoded onto the source's own /// track as they show, while it is used, on one timeline across plays. @@ -1242,14 +1327,14 @@ impl Driver { &self, path: &Path, looping: bool, - mut starts: watch::Receiver<()>, + mut cues: watch::Receiver, listing: Option<&Listing<'_>>, ) -> Result<(), Error> { let clip = Clip::open(path)?; // Without sound it has no strip to mix or meter. self.silent.store(clip.audio.is_none(), Ordering::Relaxed); let mut sound = match (&clip.audio, &self.strip) { - (Some(_), Some(strip)) => { + (Some(_), Some(_)) => { // Its track opens now, the pictures' at the first picture. let track = listing.and_then(|listing| { let bus = self.context.bus.config(); @@ -1257,9 +1342,8 @@ impl Driver { input.format = moq_audio::Format::F32; self.publish(crate::publish::Audio::new(listing, input)) }); - let channel = Some(self.channel(strip.clone())?); Some(Sound { - channel, + channel: None, track, ended: None, }) @@ -1268,15 +1352,13 @@ impl Driver { }; let mut own = Own::new(listing, kio::clock::now()); // The last channel on a strip to close drops what it has not played - // yet, so a loop keeps its channel, sounding on, while a play after a - // restart or an end takes a new channel, cutting off what the play - // before still had queued. The old channel goes first: one left on the - // strip would keep the strip's queue, delay included, for the new. - // Its pictures start a group there too, a cut to the clip's start. - let restart = |sound: &mut Option>, own: &mut Own<'_, '_>| -> Result<(), Error> { - if let (Some(sound), Some(strip)) = (sound, &self.strip) { - drop(sound.channel.take()); - sound.channel = Some(self.channel(strip.clone())?); + // yet, so a loop keeps its channel, sounding on, while a new cue drops + // it, cutting off what the play before still had queued; the next play + // takes a new one. Its pictures start a group there too, a cut to the + // clip's start. + let cut = |sound: &mut Option>, own: &mut Own<'_, '_>| { + if let Some(sound) = sound { + sound.channel = None; sound.ended = None; } if let Some(video) = &mut own.track @@ -1284,44 +1366,74 @@ impl Driver { { self.fail_encoding(&mut own.track, err); } - Ok(()) }; + let mut cue = *cues.borrow_and_update(); // Where a loop's next play starts: where the last one ended, so loops // keep exact time whatever opening the decoders takes. let mut next = None; loop { - let restarted = tokio::select! { - result = self.play_clip(&clip, sound.as_mut(), &mut own, next) => { - next = Some(result?); - false + // Whether to rest until the next cue, rather than take it at once. + let rest = if cue.stopped { + true + } else { + if let (Some(sound), Some(strip)) = (&mut sound, &self.strip) + && sound.channel.is_none() + { + sound.channel = Some(self.channel(strip.clone())?); } - changed = starts.changed() => { - // The feed is gone, which stops this driver anyway. - if changed.is_err() { - return std::future::pending().await; + let played = tokio::select! { + result = self.play_clip(&clip, sound.as_mut(), &mut own, next, cue.play) => Some(result?), + changed = cues.changed() => { + // The feed is gone, which stops this driver anyway. + if changed.is_err() { + return std::future::pending().await; + } + None + } + }; + if let Some(end) = played { + self.ran(cue.play, |run| { + run.ended.get_or_insert(end); + }); + next = Some(end); + if looping { + continue; } - true } + played.is_some() }; - if restarted { - restart(&mut sound, &mut own)?; - next = None; - continue; + if rest { + set(&self.status, Status::Ended); + self.ended(&mut cues, &mut own).await; } - if looping { - continue; - } - set(&self.status, Status::Ended); - self.ended(&mut starts, &mut own).await; - restart(&mut sound, &mut own)?; + cut(&mut sound, &mut own); + cue = *cues.borrow_and_update(); next = None; } } - /// Wait out a media source that ended until `starts` changes. Its own - /// track gets no more pictures, so its encoder is released here once + /// Note how a media source's `play` runs, unless the feed has asked for + /// another since or stopped it. + fn ran(&self, play: u64, note: impl FnOnce(&mut Run)) { + let mut latest = self.runs.lock().unwrap(); + if latest.run.play == play && latest.run.ended.is_none() { + note(&mut latest.run); + } + } + + /// Show a picture of a media source's `play`, unless the feed has asked + /// for another or stopped it since. + fn show_play(&self, play: u64, picture: Picture) { + let latest = self.runs.lock().unwrap(); + if latest.run.play == play && !latest.stopped { + self.show(picture); + } + } + + /// Wait out a media source that ended or stopped until `cues` changes. Its + /// own track gets no more pictures, so its encoder is released here once /// nothing uses the source, giving the encode session back. - async fn ended(&self, starts: &mut watch::Receiver<()>, own: &mut Own<'_, '_>) { + async fn ended(&self, cues: &mut watch::Receiver, own: &mut Own<'_, '_>) { loop { let settled = async { match &own.track { @@ -1330,7 +1442,7 @@ impl Driver { } }; tokio::select! { - changed = starts.changed() => { + changed = cues.changed() => { // The feed is gone, which stops this driver anyway. if changed.is_err() { std::future::pending::<()>().await; @@ -1349,15 +1461,17 @@ impl Driver { } /// Play `clip` once, each picture and sound when its moment comes, the - /// first at `begin`, or at once. Returns where it ends, once the last has - /// played for as long as it lasts: a sound for its samples, and a picture - /// for as long as the one before it. A clip that decodes to nothing fails. + /// first at `begin`, or at once, as part of the feed's `play`. Returns + /// where it ends, once the last has played for as long as it lasts: a + /// sound for its samples, and a picture for as long as the one before it. + /// A clip that decodes to nothing fails. async fn play_clip( &self, clip: &Clip, mut sound: Option<&mut Sound<'_>>, own: &mut Own<'_, '_>, begin: Option, + play: u64, ) -> Result { set(&self.status, Status::Live); let broadcast = clip.consume(); @@ -1401,6 +1515,9 @@ impl Driver { self.open(own, frame, config).await; } let begin = begin.unwrap_or_else(kio::clock::now); + self.ran(play, |run| { + run.begun.get_or_insert(begin); + }); let at = |timestamp: moq_net::Timestamp| begin + Duration::from(timestamp).saturating_sub(start); let end = std::cell::Cell::new(begin); let last = |until: Instant| end.set(end.get().max(until)); @@ -1416,7 +1533,7 @@ impl Driver { last(due + previous.map_or(Duration::ZERO, |previous| due - previous)); previous = Some(due); self.encode(own, &mut frame, config).await?; - self.show(Picture::Frame(frame)); + self.show_play(play, Picture::Frame(frame)); picture = next_picture(consumer).await?; } Ok::<_, Error>(()) @@ -1619,8 +1736,9 @@ pub(crate) fn pick<'a, C: Reference + Clone + 'a>( } } -/// Start the feed for `source`, and the driver that fills it. -fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { +/// Start the feed for `source`, and the driver that fills it, `held` from +/// playing a media file until the first [`Feed::play`]. +fn open(name: &str, source: &Source, context: &Context, held: bool) -> (Feed, Driver) { let picture = Mailbox::default(); let delivered = Delivered::default(); let strip = source.has_audio().then(Strip::default); @@ -1647,14 +1765,18 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { } _ => (None, None), }; - let (starts, started) = match source { + let (cues, cued) = match source { Source::Media { .. } => { - let (starts, started) = watch::channel(()); - (Some(starts), Some(started)) + let (cues, cued) = watch::channel(Cue { play: 0, stopped: held }); + (Some(cues), Some(cued)) } _ => (None, None), }; let silent = Arc::::default(); + let runs = Runs::new(Mutex::new(Latest { + run: Run::default(), + stopped: held, + })); let feed = Feed { source: source.clone(), @@ -1666,7 +1788,8 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { track: track.as_ref().map(|track| track.status.consume()), played: played.clone(), plays, - starts, + cues, + runs: runs.clone(), silent: silent.clone(), clock: clock.as_ref().map(timeline::Producer::consume), }; @@ -1682,7 +1805,8 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { track, played, plays: replays, - starts: started, + cues: cued, + runs, silent, clock, }; @@ -1873,6 +1997,23 @@ pub(crate) mod test { kio::clock::sleep_until(at).await; } + /// Mix `mixer` along with the clock from `at` to `until`, 10 ms at a time, + /// returning the loudest `mix` got. + pub(crate) async fn listen(mixer: &mut moq_audio::bus::Driver, mix: &Mix, at: &mut Instant, until: Instant) -> f32 { + const STEP: Duration = Duration::from_millis(10); + let frames = + (STEP.as_nanos() * u128::from(moq_audio::bus::Config::default().sample_rate) / 1_000_000_000) as usize; + mix.level(); + let mut loudest = 0.0f32; + while *at < until { + *at += STEP; + until_time(*at).await; + mixer.advance(frames); + loudest = loudest.max(mix.level().peak()); + } + loudest + } + /// Run `f` on a local task set, as drivers need. pub(crate) async fn local(f: F) -> F::Output { tokio::task::LocalSet::new().run_until(f).await @@ -1939,7 +2080,7 @@ mod tests { let (context, _origin) = context(dir.path()); let start = kio::clock::now(); let opened = |source: Source| { - let (feed, driver) = open("source", &source, &context); + let (feed, driver) = open("source", &source, &context, false); set(&driver.status, Status::Live); (feed, driver) }; @@ -2649,7 +2790,7 @@ mod tests { #[cfg(test)] mod media_tests { - use super::test::{context, local, start, until, until_time}; + use super::test::{context, listen, local, start, until, until_time}; use super::*; use crate::media::tests::{CLIP, SILENT, TONE}; @@ -2734,6 +2875,137 @@ mod media_tests { .await; } + /// The file a stinger plays opens blank and silent, waiting for its first + /// play, and each play reports when it began and ended, never a play it + /// cut off. + #[tokio::test(start_paused = true)] + async fn a_stinger_clip_waits_for_its_play() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let mut collection = media(dir.path(), &[("clip.mkv", CLIP, false, false)]); + collection.transition = crate::Transition::Stinger { + source: "clip.mkv".into(), + cut: Duration::from_millis(500), + }; + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let feed = sources.get("clip.mkv").unwrap(); + assert_eq!(feed.status(), Status::Ended); + until_time(kio::clock::now() + Duration::from_millis(1_100)).await; + assert!(feed.take().is_none(), "nothing plays until asked"); + assert_eq!(feed.run(), Run::default()); + + feed.play().unwrap(); + assert_eq!(feed.run().play, 1); + let (first, shown_at) = shown(feed).await; + let begun = feed.run().begun.expect("begun once its first picture is due"); + assert!(begun <= shown_at, "{begun:?} after {shown_at:?}"); + assert_eq!(feed.run().ended, None); + + // Cut off halfway, it never ends; the new play starts over. + until_time(begun + Duration::from_millis(500)).await; + feed.play().unwrap(); + let forgotten = Run { + play: 2, + ..Run::default() + }; + assert_eq!(feed.run(), forgotten, "the play before is forgotten"); + let (again, _) = shown(feed).await; + assert_eq!(again, first); + let begun = feed.run().begun.unwrap(); + + until_time(begun + Duration::from_millis(1_100)).await; + until_status(&sources, "clip.mkv", Status::Ended).await; + let ended = feed.run().ended.expect("ended at its end"); + assert!( + ended >= begun + Duration::from_secs(1) && ended <= kio::clock::now(), + "{:?} after it began", + ended - begun + ); + }) + .await; + } + + /// A stinger's file asked to play while it still opens plays once open. + #[tokio::test(start_paused = true)] + async fn a_stinger_clip_asked_while_opening_plays() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let mut collection = media(dir.path(), &[("clip.mkv", CLIP, false, false)]); + collection.transition = crate::Transition::Stinger { + source: "clip.mkv".into(), + cut: Duration::ZERO, + }; + let drivers = sources.sync(&collection); + let feed = sources.get("clip.mkv").unwrap(); + assert_eq!(feed.status(), Status::Starting); + feed.play().unwrap(); + start(&sources, drivers).await; + shown(feed).await; + assert_eq!(feed.run().play, 1); + assert!(feed.run().begun.is_some()); + }) + .await; + } + + /// A playing file stopped goes blank and silent at once, sound it had + /// queued included, ending its play there, and plays from its start + /// again when asked. Only a media source stops. + #[tokio::test(start_paused = true)] + async fn a_stopped_media_file_goes_blank_and_silent() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (mut context, _origin) = context(dir.path()); + let (bus, mut mixer) = Bus::new(Default::default()).unwrap(); + context.bus = bus; + let mut at = kio::clock::now(); + mixer.advance(480); + let mut sources = Sources::new(context); + let mut collection = media(dir.path(), &[("clip.mkv", CLIP, false, false)]); + let red = Source::Color { + color: crate::Rgba([255, 0, 0, 255]), + }; + collection.sources.insert("red".into(), red); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let feed = sources.get("clip.mkv").unwrap(); + let program = sources.mix(crate::Mixes::DEFAULT).unwrap().clone(); + + let (first, begun) = shown(feed).await; + let heard = listen(&mut mixer, &program, &mut at, begun + Duration::from_millis(300)).await; + assert!(heard > 0.0, "the clip sounds before it stops"); + + feed.stop().unwrap(); + let stopped = kio::clock::now(); + let blank = feed.take(); + assert!( + matches!(blank, Some(Picture::Rgba { pixels, .. }) if pixels == [0; 4]), + "blank at once" + ); + assert_eq!(feed.run().ended, Some(stopped)); + until_status(&sources, "clip.mkv", Status::Ended).await; + let heard = listen(&mut mixer, &program, &mut at, stopped + Duration::from_millis(500)).await; + assert_eq!(heard, 0.0, "nothing it queued plays"); + assert!( + !matches!(feed.take(), Some(Picture::Frame(_))), + "no picture after the stop" + ); + assert_eq!(feed.run().ended, Some(stopped), "the play stays ended there"); + + feed.play().unwrap(); + let (again, _) = shown(feed).await; + assert_eq!(again, first, "from the start"); + assert_eq!(feed.status(), Status::Live); + let red = sources.get("red").unwrap(); + assert!(matches!(red.stop(), Err(Error::Unsupported(_)))); + }) + .await; + } + /// A looping file starts over at its end, never reading as ended. #[tokio::test(start_paused = true)] async fn a_looping_media_file_starts_over() { diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index c26b25e..fc42fb7 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -205,9 +205,24 @@ impl State { } /// The sources whose pictures any canvas's program draws, through clones: - /// each visible item's of the scenes on air, including one leaving in a - /// transition, and of the top layers over them, and each mask's it reads. + /// those [`placed`](Self::placed) there, and the clip a stinger under way + /// plays with each mask its source's filters read. fn shown(&self) -> BTreeSet { + let collection = &self.collection; + let stinger = self.compositor.stinger().into_iter().flat_map(|clip| { + let filters = collection.video.get(clip).into_iter().flat_map(|video| &video.filters); + std::iter::once(clip).chain(masks(filters).map(String::as_str)) + }); + let mut shown = self.placed(); + shown.extend(stinger.map(|source| collection.capture(source).to_string())); + shown + } + + /// The sources whose pictures the scenes on air place on any canvas's + /// program, through clones: each visible item's of the scenes on air, + /// including one leaving in a transition, and of the top layers over them, + /// and each mask's it reads. + fn placed(&self) -> BTreeSet { let collection = &self.collection; let scenes: Vec> = self.compositor.on_air().map(Some).chain([None]).collect(); let canvases = std::iter::once(None).chain(collection.canvases.keys().map(Some)); @@ -220,15 +235,8 @@ impl State { }) .flat_map(|layer| collection.items(&layer).unwrap_or_default()) .filter(|item| item.visible); - // A mask's picture shapes what the program draws too. - let masks = |item| { - collection.filters(item).filter_map(|filter| match filter { - moq_scene::Filter::Mask { source, .. } => Some(source), - _ => None, - }) - }; items - .flat_map(|item| std::iter::once(&item.source).chain(masks(item))) + .flat_map(|item| std::iter::once(&item.source).chain(masks(collection.filters(item)))) .map(|source| collection.capture(source).to_string()) .collect() } @@ -248,6 +256,15 @@ impl State { } } +/// The sources the masks among `filters` read: a mask's picture shapes what +/// the program draws too. +fn masks<'a>(filters: impl Iterator) -> impl Iterator { + filters.filter_map(|filter| match filter { + moq_scene::Filter::Mask { source, .. } => Some(source), + _ => None, + }) +} + impl autosave::Document for State { /// Both under one lock, so the history describes the collection beside it. fn text(&self) -> anyhow::Result { @@ -757,7 +774,7 @@ impl Engine { let mut compositor = Compositor::new(&gpu.device, &gpu.queue, collection.canvas)?; compositor.set_canvases(&collection)?; compositor.set_preview(collection.scenes.first().map(|scene| scene.name.clone())); - compositor.cut(); + compositor.cut(&sources); let engine = Arc::new(Self { state: Mutex::new(State { @@ -1054,6 +1071,10 @@ impl Engine { matters.then(|| (name.to_string(), stall)) }) .collect(); + // A stinger's clip rests ended between stingers, unless a scene on + // air places it. + let stinger = state.collection.transition.stinger(); + let placed = state.placed(); let stopped = state .sources .iter() @@ -1064,7 +1085,12 @@ impl Engine { Some(err) => Status::Failed(err.to_string()), None => feed.status(), }; - matches!(status, Status::Ended | Status::Failed(_)).then(|| (name.to_string(), status)) + let stopped = match status { + Status::Ended => stinger != Some(name) || placed.contains(name), + Status::Failed(_) => true, + _ => false, + }; + stopped.then(|| (name.to_string(), status)) }) .collect(); (stalled, stopped) @@ -2761,6 +2787,97 @@ mod tests { assert!(engine.stats.error().is_none()); } + /// The clip a stinger plays rests ended between stingers without Health + /// counting it, while another media file that ended and is heard counts, + /// as does the clip once a scene on air places it. The program shows the + /// clip, and what its source's mask reads, while a stinger plays it. + #[tokio::test(flavor = "multi_thread")] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn the_program_rests_a_stinger_clip() { + let dir = tempfile::tempdir().unwrap(); + let clip = include_bytes!("../../moq-scene/src/test_data/clip.mkv"); + let mut loaded = red_canvas(); + for name in ["sting", "bed"] { + let path = format!("{name}.mkv"); + std::fs::write(dir.path().join(&path), clip).unwrap(); + let media = moq_scene::Source::Media { + path: path.into(), + looping: false, + restart: false, + }; + loaded.collection.sources.insert(name.into(), media); + } + loaded.collection.transition = moq_scene::Transition::Stinger { + source: "sting".into(), + cut: Duration::from_millis(300), + }; + let matte = moq_scene::Source::Color { + color: moq_scene::Rgba([255, 255, 255, 255]), + }; + loaded.collection.sources.insert("matte".into(), matte); + let mask = moq_scene::Filter::Mask { + source: "matte".into(), + mode: moq_scene::MaskMode::Alpha, + }; + let video = moq_scene::Video { + filters: vec![mask], + ..Default::default() + }; + loaded.collection.video.insert("sting".into(), video); + let gpu = Gpu::new().await.expect("a GPU"); + let (autosave, _) = Autosave::open(dir.path().join("show.toml")).unwrap(); + let engine = Engine::start(launch(gpu, Vec::new(), dir.path()), autosave, loaded) + .await + .expect("the studio starts"); + let status = |name: &str| engine.state.lock().unwrap().sources.get(name).map(source::Feed::status); + let ended = || async { + while status("bed") != Some(Status::Ended) || status("sting") != Some(Status::Ended) { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }; + kio::clock::timeout(TIMEOUT, ended()).await.expect("the bed ends"); + let stopped = engine.local(kio::clock::now()).stopped; + assert_eq!(stopped, [("bed".to_string(), Status::Ended)]); + + { + let mut state = engine.state.lock().unwrap(); + let transition = state.collection().transition.clone(); + let state = &mut *state; + state + .compositor + .transition(transition, &state.sources, kio::clock::now()); + let shown = state.shown(); + assert!(shown.contains("sting") && shown.contains("matte"), "{shown:?}"); + } + let played = || async { + while engine.state.lock().unwrap().compositor.transitioning() { + kio::clock::sleep(Duration::from_millis(10)).await; + } + }; + kio::clock::timeout(TIMEOUT, played()).await.expect("the stinger ends"); + let shown = engine.state.lock().unwrap().shown(); + assert!(!shown.contains("sting") && !shown.contains("matte"), "{shown:?}"); + kio::clock::timeout(TIMEOUT, ended()).await.expect("the clip rests"); + let stopped = engine.local(kio::clock::now()).stopped; + assert_eq!(stopped, [("bed".to_string(), Status::Ended)]); + + { + let mut state = engine.state.lock().unwrap(); + let scene = state.compositor.program_scene().unwrap().to_string(); + let fill = moq_scene::Rect::fill(&state.collection().canvas); + let mut draft = state.collection().draft("Place sting"); + let item = moq_scene::Item::new("sting", fill); + draft.insert_item(&moq_scene::Layer::scene(scene), 1, item).unwrap(); + state.commit(draft, None).unwrap(); + } + let stopped = engine.local(kio::clock::now()).stopped; + let both = [("bed".to_string(), Status::Ended), ("sting".to_string(), Status::Ended)]; + assert_eq!(stopped, both, "placed, it counts"); + + let stopping = engine.clone(); + tokio::task::spawn_blocking(move || stopping.stop()).await.unwrap(); + } + /// A rendition a resize retired while its encoder was still opening must /// not report that encoder over the next rendition's. #[test] diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 923a95b..6a498e1 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -117,6 +117,8 @@ impl Studio { gestures: Gestures::default(), snapping: Snapping::default(), direction: Direction::Left, + duration: Transition::default().duration().unwrap_or_default(), + cut: STINGER_CUT, drag: None, new_scene: String::new(), renaming: None, @@ -278,6 +280,12 @@ struct Panels { /// The direction a slide or wipe last had, which the transition picker /// offers again after a fade or move. direction: Direction, + /// How long a timed transition last took, which the transition picker + /// offers again after a stinger. + duration: Duration, + /// The cut point a stinger last had, which the transition picker offers + /// again after another kind. + cut: Duration, /// The item a drag in the preview holds. drag: Option, new_scene: String, @@ -988,25 +996,48 @@ impl Panels { let engine = self.engine.clone(); let mut state = engine.state.lock().unwrap(); let canvas = self.drawn; - let transition = state.collection().transition; - let mut next = transition; + let transition = state.collection().transition.clone(); + let mut next = transition.clone(); + let media: Vec = state + .collection() + .sources + .iter() + .filter(|(_, source)| matches!(source, Source::Media { .. })) + .map(|(name, _)| name.clone()) + .collect(); let mut gesture = None; ui.horizontal(|ui| { if ui.button("Cut").clicked() { - state.compositor.cut(); + let state = &mut *state; + state.compositor.cut(&state.sources); } if ui .button("Transition") .on_hover_text("Take the preview to program by the transition set beside it") .clicked() { - state.compositor.transition(transition, kio::clock::Instant::now()); - } - let duration = next.duration(); - if let Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } = next { - self.direction = direction; - } - let direction = self.direction; + let state = &mut *state; + let now = kio::clock::Instant::now(); + state.compositor.transition(transition.clone(), &state.sources, now); + } + match &next { + Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } => self.direction = *direction, + Transition::Stinger { cut, .. } => self.cut = *cut, + _ => {} + } + if let Some(duration) = next.duration() { + self.duration = duration; + } + let (duration, direction) = (self.duration, self.direction); + // A stinger plays a media source's clip, so it is offered only + // once there is one. + let stinger = match &next { + Transition::Stinger { .. } => Some(next.clone()), + _ => media.first().map(|source| Transition::Stinger { + source: source.clone(), + cut: self.cut, + }), + }; egui::ComboBox::from_id_salt("transition") .selected_text(transition_name(&next)) .show_ui(ui, |ui| { @@ -1016,27 +1047,55 @@ impl Panels { Transition::Wipe { duration, direction }, Transition::Move { duration }, ]; - for kind in kinds { - ui.selectable_value(&mut next, kind, transition_name(&kind)); + for kind in kinds.into_iter().chain(stinger) { + let name = transition_name(&kind); + ui.selectable_value(&mut next, kind, name); } - }); - if let Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } = &mut next { - egui::ComboBox::from_id_salt("direction") - .selected_text(direction_name(*direction)) - .show_ui(ui, |ui| { - for to in Direction::ALL { - ui.selectable_value(direction, to, direction_name(to)); - } - }); + }) + .response + .on_hover_text("How the preview goes to program; a stinger plays a media source's clip over it"); + match &mut next { + Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } => { + egui::ComboBox::from_id_salt("direction") + .selected_text(direction_name(*direction)) + .show_ui(ui, |ui| { + for to in Direction::ALL { + ui.selectable_value(direction, to, direction_name(to)); + } + }); + } + Transition::Stinger { source, cut } => { + egui::ComboBox::from_id_salt("stinger") + .selected_text(source.as_str()) + .show_ui(ui, |ui| { + for name in &media { + ui.selectable_value(source, name.clone(), name); + } + }) + .response + .on_hover_text("The media source whose clip plays"); + let mut millis = cut.as_millis() as u64; + let response = ui + .add(egui::DragValue::new(&mut millis).prefix("cut at ").suffix(" ms")) + .on_hover_text("How far into the clip the scenes switch, while it covers the canvas"); + let dragging = self.gestures.of(&response); + if response.changed() { + *cut = Duration::from_millis(millis); + gesture = dragging; + } + } + _ => {} } - let mut millis = duration.as_millis() as u64; - let response = ui - .add(egui::DragValue::new(&mut millis).suffix(" ms")) - .on_hover_text("How long the transition takes"); - let dragging = self.gestures.of(&response); - if response.changed() { - *next.duration_mut() = Duration::from_millis(millis); - gesture = dragging; + if let Some(duration) = next.duration_mut() { + let mut millis = duration.as_millis() as u64; + let response = ui + .add(egui::DragValue::new(&mut millis).suffix(" ms")) + .on_hover_text("How long the transition takes"); + let dragging = self.gestures.of(&response); + if response.changed() { + *duration = Duration::from_millis(millis); + gesture = dragging; + } } ui.label(format!( "Program: {}", @@ -1396,6 +1455,7 @@ impl Panels { let mut clone = None; let mut reload = None; let mut play = None; + let mut stop = None; let now = kio::clock::now(); egui::ScrollArea::vertical().show(ui, |ui| { @@ -1479,13 +1539,21 @@ impl Panels { { clone = Some(name.clone()); } - if matches!(source, Source::Media { .. }) - && ui + if matches!(source, Source::Media { .. }) { + if ui .small_button("Play") .on_hover_text("Play the file from its start") .clicked() - { - play = Some(name.clone()); + { + play = Some(name.clone()); + } + if ui + .small_button("Stop") + .on_hover_text("Stop the file now, blank and silent until played again") + .clicked() + { + stop = Some(name.clone()); + } } let reloads = match source { Source::Browser { .. } => { @@ -1528,6 +1596,12 @@ impl Panels { let played = feed.play().with_context(|| format!("cannot play `{name}`")); self.report(played.map(|()| format!("playing `{name}` from its start"))); } + if let Some(name) = stop + && let Some(feed) = state.sources.get(&name) + { + let stopped = feed.stop().with_context(|| format!("cannot stop `{name}`")); + self.report(stopped.map(|()| format!("stopped `{name}`"))); + } let result = if let Some(name) = remove { // Removing the items that show it moves the ones after them. self.deselect_item(); @@ -2963,9 +3037,13 @@ fn transition_name(transition: &Transition) -> &'static str { Transition::Slide { .. } => "slide", Transition::Wipe { .. } => "wipe", Transition::Move { .. } => "move", + Transition::Stinger { .. } => "stinger", } } +/// Where the transition picker puts a new stinger's cut point. +const STINGER_CUT: Duration = Duration::from_millis(500); + fn direction_name(direction: Direction) -> &'static str { match direction { Direction::Left => "left", diff --git a/rs/moq-studio/tests/browser.rs b/rs/moq-studio/tests/browser.rs index 124bcd4..29effe4 100644 --- a/rs/moq-studio/tests/browser.rs +++ b/rs/moq-studio/tests/browser.rs @@ -87,7 +87,7 @@ impl Studio { } let mut compositor = Compositor::new(&device, &queue, collection.canvas).unwrap(); compositor.set_preview(Some("Main".into())); - compositor.cut(); + compositor.cut(&sources); Self { collection, sources,