diff --git a/README.md b/README.md index c3cc1be..b3ad1ef 100644 --- a/README.md +++ b/README.md @@ -90,8 +90,8 @@ Every panel is a tab you can drag, split, or stack. | Panel | Use it to | | --- | --- | -| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. **Cut** puts the preview on program at once; **Fade** blends over the fade time set beside it. **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | -| Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still fading out, cannot be removed, even by an undo, until program cuts away from it. | +| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. **Cut** puts the preview on program at once; **Transition** takes it there by the transition picked beside it, over the time set there (see [Transitions](#transitions)). **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | +| Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still leaving it in a transition, cannot be removed, even by an undo, until program cuts away from it. | | History | See every edit, and undo or redo back to any of them. | | Items | Place, show, hide, and order what the preview scene or the top layer draws, back to front. Select an item to filter or copy it. | | Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. | @@ -105,8 +105,22 @@ Every panel is a tab you can drag, split, or stack. A new source, from the Sources or Devices panel, is placed over whatever the Items panel shows. The **top layer** is drawn over every scene, in preview and -on program, and stays on through a cut or fade: a downstream keyer for a logo -or a lower third. +on program, and stays on through a cut or transition: a downstream keyer for a +logo or a lower third. + +### Transitions + +| Transition | What it does | +| --- | --- | +| Fade | The incoming scene fades in over the outgoing one, at an even pace. | +| Slide | The incoming scene pushes the outgoing one off the canvas, both moving left, right, up, or down. | +| Wipe | An edge sweeps across the canvas left, right, up, or down, uncovering the incoming scene behind it. | +| Move | Each item showing a source both scenes place glides from its place in the outgoing scene to its place in the incoming one, resizing, recropping, and mixing its filters' settings on the way. If its blend mode or kinds of filter differ, it changes look halfway. The rest of the outgoing scene fades out and the rest of the incoming scene fades in. Items stack as the outgoing scene has them until halfway, then as the incoming one does. A source placed more than once pairs up in order, back to front. | + +All but a fade ease in and out. A transition is drawn on the program clock, so +the stream and a recording show the same frames of it, and a second +**Transition** while one runs starts from the scene it was heading to. A +transition set to 0 ms is a cut. In the preview, click an item of whatever the Items panel shows to select it, drag it to move it, and drag its edges or corners to resize it. A drag snaps @@ -284,7 +298,9 @@ height = 1080 framerate = 30 [transition] -fade = "300ms" +kind = "slide" # fade, slide, wipe, or move +duration = "300ms" +direction = "left" # slide and wipe: left, right, up, or down [sources.camera] kind = "camera" diff --git a/quest/m0/README.md b/quest/m0/README.md index 6d3413d..0c85959 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -94,7 +94,7 @@ repository. - [Broadcast sources play in sync](/quest/m0/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker - [Live encode control](/quest/m0/encode-control.md) - simulcast rungs that follow the bandwidth grant, live bitrate and resolution, keyframes on request -- [Animation and transitions](/quest/m0/animation.md) - move, slide, wipe, and stinger transitions, and keyframed item properties +- [Keyframes and stingers](/quest/m0/animation.md) - keyframed item properties with easing, and stinger transitions - [ML video effects](/quest/m0/effects.md) - background removal and other ML effects, baked into their source's track - [Destinations](/quest/m0/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse - [Multiple canvases](/quest/m0/multi-canvas.md) - landscape and portrait canvases composite at once from shared sources diff --git a/quest/m0/animation.md b/quest/m0/animation.md index 991968c..64a8c92 100644 --- a/quest/m0/animation.md +++ b/quest/m0/animation.md @@ -1,24 +1,38 @@ -# [M] Animation and transitions +# [M] Keyframes and stingers ## Goal -Scene changes in `moq-studio` use transitions beyond a cut and fade: move -(items glide between their positions in the two scenes), slide, wipe, and -stinger (a video clip with a cut point). Any item property can be keyframed -with easing, so an item can fly in or a camera can push in. +Any item property in `moq-studio` can be keyframed with easing, so an item +can fly in or a camera can push in, and a scene change can run a stinger: a +video clip played over the program that switches scenes at its cut point. ## Plan -Move is OBS's most-installed plugin (Move Transition), and stingers are -standard in broadcast. Animate in moq-scene on the media clock, so the program, a recording, and a -replay match. Carrying transitions and keyframes in the layout track, for -viewers to compose, waits for the m1 -[composition epic](/quest/m1/compose/README.md) (changed 2026-11-04, -maintainer: the program comes first). -Overlays animate through their own CSS. No Lottie runtime for now (decided -2026-10-26): the open ones either lack text (velato) or add a C++ build -(ThorVG). A stinger clip is a media source like any other. Transitions and -keyframes change through moq-scene's `Edit`, so they undo like any edit. +Fade, slide, wipe, and move transitions are drawn by moq-scene's compositor +on the program clock +([moq-dev/studio#33](https://github.com/moq-dev/studio/pull/33)). Animate in +moq-scene on the media clock, so the program, a recording, and a replay match. +Carrying transitions and keyframes in the layout track, for viewers to +compose, waits for the m1 [composition epic](/quest/m1/compose/README.md) +(changed 2026-11-04, maintainer: the program comes first). Overlays animate +through their own CSS. No Lottie runtime for now (decided 2026-10-26): the +open ones either lack text (velato) or add a C++ build (ThorVG). Keyframes and +stinger settings change through moq-scene's `Edit`, so they undo like any edit. + +A stinger clip is a media source like any other, but no source plays a media +file yet, and a stinger must start its clip as the transition starts or its +cut point lands at random. It waits on a media-file source that plays from +the start on demand, which no quest plans yet. + +Open questions for the maintainer: + +- What starts an item's keyframes: its scene going on program (and into + preview, for the preview), a loop on the program clock, or a trigger such + as a hotkey. Recommended: the time since its scene went on program, playing + once and holding the last keyframe. +- How a keyframed item edits: a drag in the preview sets the keyframe at the + preview's time, or the item's resting place, from which keyframes are + offsets. ## Related diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 3ef19cc..b197919 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -22,7 +22,7 @@ use crate::Error; pub struct Collection { /// The output every scene is drawn onto. pub canvas: Canvas, - /// How a fade from preview to program behaves. + /// How the preview is taken to program, short of a cut. #[serde(default)] pub transition: Transition, /// Every source, by its unique name. Scenes refer to sources by that name, @@ -329,23 +329,108 @@ impl Default for Canvas { } } -/// How the program changes when the preview is faded in. +/// How the program changes when the preview is taken to it, short of a cut. +/// +/// A fade runs at an even pace; the others ease in and out, so motion starts +/// and lands gently. One that takes no time is a cut. #[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] -#[serde(deny_unknown_fields)] -pub struct Transition { - /// How long a fade takes. - #[serde(with = "humantime_serde")] - pub fade: Duration, +#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)] +pub enum Transition { + /// The incoming scene fades in over the outgoing one. + Fade { + /// How long it takes. + #[serde(with = "humantime_serde")] + duration: Duration, + }, + /// The incoming scene pushes the outgoing one off the canvas, both moving + /// in `direction`. + Slide { + /// How long it takes. + #[serde(with = "humantime_serde")] + duration: Duration, + /// Which way the scenes move. + direction: Direction, + }, + /// An edge sweeps across the canvas in `direction`, uncovering the + /// incoming scene behind it. + Wipe { + /// How long it takes. + #[serde(with = "humantime_serde")] + duration: Duration, + /// Which way the edge moves. + direction: Direction, + }, + /// Each item showing a source both scenes place glides from where the + /// outgoing scene has it to where the incoming one does, resizing and + /// recropping on the way; the other items fade out and in. + Move { + /// How long it takes. + #[serde(with = "humantime_serde")] + duration: Duration, + }, +} + +impl Transition { + /// How long it takes. + pub fn duration(&self) -> Duration { + match *self { + Self::Fade { duration } + | Self::Slide { duration, .. } + | Self::Wipe { duration, .. } + | Self::Move { duration } => duration, + } + } + + /// How long it takes, to change. + pub fn duration_mut(&mut self) -> &mut Duration { + match self { + Self::Fade { duration } + | Self::Slide { duration, .. } + | Self::Wipe { duration, .. } + | Self::Move { duration } => duration, + } + } + + /// How far along it looks once `linear` of its time has passed, both from + /// 0 to 1: as is for a fade, and eased in and out otherwise. + pub(crate) fn progress(&self, linear: f32) -> f32 { + let linear = linear.clamp(0.0, 1.0); + match self { + Self::Fade { .. } => linear, + // The cubic ease in and out. + _ if linear < 0.5 => 4.0 * linear.powi(3), + _ => 1.0 - (2.0 - 2.0 * linear).powi(3) / 2.0, + } + } } impl Default for Transition { fn default() -> Self { - Self { - fade: Duration::from_millis(300), + Self::Fade { + duration: Duration::from_millis(300), } } } +/// Which way a [`Transition`] moves across the canvas. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Direction { + /// Toward the left edge. + Left, + /// Toward the right edge. + Right, + /// Toward the top edge. + Up, + /// Toward the bottom edge. + Down, +} + +impl Direction { + /// Every direction. + pub const ALL: [Self; 4] = [Self::Left, Self::Right, Self::Up, Self::Down]; +} + /// One arrangement of sources on the canvas. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -566,6 +651,84 @@ impl Filter { } } + /// `progress` of the way from this filter's settings to `other`'s, or + /// `None` if they are of different kinds or read different sources or + /// mask modes. A mix of valid settings is valid. + pub(crate) fn toward(&self, other: &Self, progress: f32) -> Option { + let mix = |from: f32, to: f32| mix(from, to, progress); + let color = |from: Rgba, to: Rgba| { + Rgba(std::array::from_fn(|channel| { + mix(f32::from(from.0[channel]), f32::from(to.0[channel])).round() as u8 + })) + }; + Some(match (self, other) { + (Self::Opacity { amount: a }, Self::Opacity { amount: b }) => Self::Opacity { amount: mix(*a, *b) }, + (Self::Brightness { amount: a }, Self::Brightness { amount: b }) => { + Self::Brightness { amount: mix(*a, *b) } + } + (Self::Contrast { amount: a }, Self::Contrast { amount: b }) => Self::Contrast { amount: mix(*a, *b) }, + (Self::Saturate { amount: a }, Self::Saturate { amount: b }) => Self::Saturate { amount: mix(*a, *b) }, + (Self::Grayscale { amount: a }, Self::Grayscale { amount: b }) => Self::Grayscale { amount: mix(*a, *b) }, + (Self::Sepia { amount: a }, Self::Sepia { amount: b }) => Self::Sepia { amount: mix(*a, *b) }, + (Self::Invert { amount: a }, Self::Invert { amount: b }) => Self::Invert { amount: mix(*a, *b) }, + (Self::HueRotate { degrees: a }, Self::HueRotate { degrees: b }) => { + Self::HueRotate { degrees: mix(*a, *b) } + } + (Self::Blur { radius: a }, Self::Blur { radius: b }) => Self::Blur { radius: mix(*a, *b) }, + ( + Self::DropShadow { x, y, blur, color: c }, + Self::DropShadow { + x: x2, + y: y2, + blur: blur2, + color: c2, + }, + ) => Self::DropShadow { + x: mix(*x, *x2), + y: mix(*y, *y2), + blur: mix(*blur, *blur2), + color: color(*c, *c2), + }, + ( + Self::ChromaKey { + color: c, + similarity, + smoothness, + spill, + }, + Self::ChromaKey { + color: c2, + similarity: similarity2, + smoothness: smoothness2, + spill: spill2, + }, + ) => Self::ChromaKey { + color: color(*c, *c2), + similarity: mix(*similarity, *similarity2), + smoothness: mix(*smoothness, *smoothness2), + spill: mix(*spill, *spill2), + }, + ( + Self::LumaKey { min, max, smoothness }, + Self::LumaKey { + min: min2, + max: max2, + smoothness: smoothness2, + }, + ) => Self::LumaKey { + min: mix(*min, *min2), + max: mix(*max, *max2), + smoothness: mix(*smoothness, *smoothness2), + }, + (Self::Lut { source, amount: a }, Self::Lut { source: to, amount: b }) if source == to => Self::Lut { + source: source.clone(), + amount: mix(*a, *b), + }, + (Self::Mask { .. }, Self::Mask { .. }) if self == other => self.clone(), + _ => return None, + }) + } + /// The source this filter reads, to rename it. pub(crate) fn source_mut(&mut self) -> Option<&mut String> { match self { @@ -731,6 +894,15 @@ impl Crop { } } +/// `progress` of the way from `from` to `to`, finite whenever both are, since +/// it never leaves the range between them. +pub(crate) fn mix(from: f32, to: f32, progress: f32) -> f32 { + // In f64, where neither product can overflow, and each term's weight + // rounds no larger than it is. + let progress = f64::from(progress); + (f64::from(from) * (1.0 - progress) + f64::from(to) * progress) as f32 +} + pub(crate) fn validate_filters(filters: &[Filter], on: &str) -> Result<(), Error> { for filter in filters { filter @@ -1085,7 +1257,9 @@ height = 720 framerate = 60 [transition] -fade = "500ms" +kind = "slide" +duration = "500ms" +direction = "up" [sources.cam] kind = "camera" @@ -1132,7 +1306,13 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } "##; let collection = Collection::from_toml(text).unwrap(); assert_eq!(collection.canvas.framerate, 60); - assert_eq!(collection.transition.fade, Duration::from_millis(500)); + assert_eq!( + collection.transition, + Transition::Slide { + duration: Duration::from_millis(500), + direction: Direction::Up + } + ); assert_eq!(collection.sources["cam"], Source::Camera { id: None }); assert_eq!( collection.sources["bg"], @@ -1342,6 +1522,29 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } } } + /// Mixing two valid settings is valid, even from one end of the floats to + /// the other, and a mix of different kinds is refused. + #[test] + fn filters_mix_within_their_settings() { + let shadow = |x: f32| Filter::DropShadow { + x, + y: -x, + blur: 0.0, + color: Rgba([0, 0, 0, 255]), + }; + for progress in [0.0, 0.25, 0.5, 1.0] { + let mixed = shadow(-f32::MAX).toward(&shadow(f32::MAX), progress).unwrap(); + validate_filters(&[mixed], "mixed").unwrap(); + let hue = |degrees| Filter::HueRotate { degrees }; + let mixed = hue(f32::MAX).toward(&hue(-f32::MAX), progress).unwrap(); + validate_filters(&[mixed], "mixed").unwrap(); + } + assert_eq!(mix(-f32::MAX, f32::MAX, 0.5), 0.0); + assert_eq!(mix(f32::MAX, f32::MAX, 0.3), f32::MAX); + let invert = Filter::Invert { amount: 1.0 }; + assert!(invert.toward(&Filter::Sepia { amount: 1.0 }, 0.5).is_none()); + } + #[test] fn refuses_a_crop_that_leaves_nothing() { let crops = [ diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 9fa3bf0..492c612 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -1,14 +1,13 @@ -//! Draw scenes onto the canvas on the GPU: a preview, a program, and the fade -//! between them. +//! Draw scenes onto the canvas on the GPU: a preview, a program, and the +//! transition between them. use std::collections::{BTreeMap, HashMap, HashSet}; -use std::time::Duration; use kio::clock::Instant; use moq_video::render::{Renderer, wgpu}; use crate::Error; -use crate::collection::{Canvas, Collection, Filter, Item, Layer as Place, Source}; +use crate::collection::{Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Rect, Source, Transition}; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; use crate::overlay::{Overlay, Overlays}; use crate::source::{self, Sources}; @@ -18,7 +17,8 @@ use crate::source::{self, Sources}; pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; /// Draws a [`Collection`]'s scenes onto its canvas: one texture for the scene -/// in preview, one for the scene on program, and the fade from one to the other. +/// in preview, one for the scene on program, and the [`Transition`] from one +/// to the other. /// /// Both textures are plain [`wgpu::Texture`]s on the caller's device, so a UI /// shows them without a copy and an encoder reads the program from the same @@ -29,6 +29,10 @@ pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; /// Pixels are blended in the gamma-encoded space sources arrive in, as a /// broadcast switcher and CSS do, so a 50% fade is a 50% mix of the stored /// values. +/// +/// A transition runs on the clock [`render`](Self::render) is given, so every +/// output drawn from the program, such as the encoder and a recording, sees the +/// same frame of it. pub struct Compositor { device: wgpu::Device, queue: wgpu::Queue, @@ -37,23 +41,25 @@ pub struct Compositor { preview: Target, program: Target, - /// The scene fading in, drawn whole so it blends over the program as one - /// picture rather than layer by layer. + /// The scene coming in, drawn whole so it fades, slides, or is uncovered + /// as one picture rather than item by item. incoming: Target, + /// The scene going out, drawn whole to slide it off the canvas. + outgoing: Target, layers: HashMap, /// Lays out and paints overlays, started with the first one. overlays: Option, preview_scene: Option, program_scene: Option, - fade: Option, + switch: Option, } -/// A fade from `from` to the program scene. -struct Fade { +/// A transition under way from `from` to the program scene. +struct Switch { from: Option, start: Instant, - duration: Duration, + transition: Transition, } /// A canvas-sized texture drawn into, and bound to sample. @@ -106,6 +112,7 @@ impl Compositor { preview: target("moq-scene preview"), program: target("moq-scene program"), incoming: target("moq-scene incoming"), + outgoing: target("moq-scene outgoing"), device: device.clone(), queue: queue.clone(), canvas, @@ -114,7 +121,7 @@ impl Compositor { overlays: None, preview_scene: None, program_scene: None, - fade: None, + switch: None, }) } @@ -126,7 +133,7 @@ impl Compositor { /// Draw `canvas` from the next [`render`](Self::render) on. A new size /// replaces the [`preview`](Self::preview) and [`program`](Self::program) /// textures, and every other texture sized from the canvas, with ones of - /// that size, blank until drawn; a fade under way carries on. + /// that size, blank until drawn; a transition under way carries on. /// /// Fails, changing nothing, if the canvas is invalid or the device cannot /// hold a texture that large. @@ -138,6 +145,7 @@ impl Compositor { self.preview = target("moq-scene preview"); self.program = target("moq-scene program"); self.incoming = target("moq-scene incoming"); + self.outgoing = target("moq-scene outgoing"); } self.canvas = canvas; Ok(()) @@ -160,7 +168,7 @@ impl Compositor { self.preview_scene.as_deref() } - /// The scene on program, or the one fading in. + /// The scene on program, or the one a transition brings in. pub fn program_scene(&self) -> Option<&str> { self.program_scene.as_deref() } @@ -170,15 +178,15 @@ impl Compositor { self.layers.get(source)?.error.as_deref() } - /// Whether a fade is under way. - pub fn fading(&self) -> bool { - self.fade.is_some() + /// Whether a transition is under way. + pub fn transitioning(&self) -> bool { + self.switch.is_some() } - /// The scenes program draws: the one on program, and the one a fade under - /// way leaves. + /// The scenes program draws: the one on program, and the one a transition + /// under way leaves. pub fn on_air(&self) -> impl Iterator { - let leaving = self.fade.as_ref().and_then(|fade| fade.from.as_deref()); + let leaving = self.switch.as_ref().and_then(|switch| switch.from.as_deref()); self.program_scene.as_deref().into_iter().chain(leaving) } @@ -188,9 +196,9 @@ impl Compositor { } /// Follow each renamed scene to its new name, wherever it is: in preview, - /// on program, or fading out, and drop each removed one, so a scene that - /// later takes its name does not go on program without a cut. `renames` - /// maps each old name to its new one, or `None`, as + /// on program, or leaving in a transition, and drop each removed one, so a + /// scene that later takes its name does not go on program without a cut. + /// `renames` maps each old name to its new one, or `None`, as /// [`Collection::renames`] reads them, and each scene is looked up once, so /// a name one scene gave up and another took is not followed twice. pub fn rename(&mut self, renames: &BTreeMap>) { @@ -202,8 +210,8 @@ impl Compositor { } /// Drop each scene `collection` does not have from preview, program, and a - /// fade under way, such as one a reloaded file left out, so a scene that - /// later takes its name does not go on program without a cut. + /// transition under way, such as one a reloaded file left out, so a scene + /// that later takes its name does not go on program without a cut. pub fn retain(&mut self, collection: &Collection) { for scene in self.scenes() { if scene.as_deref().is_some_and(|name| collection.scene(name).is_none()) { @@ -212,30 +220,30 @@ impl Compositor { } } - /// The scenes followed by name: in preview, on program, and fading out. + /// The scenes followed by name: in preview, on program, and leaving. fn scenes(&mut self) -> impl Iterator> { - let fading = self.fade.as_mut().map(|fade| &mut fade.from); + let leaving = self.switch.as_mut().map(|switch| &mut switch.from); [&mut self.preview_scene, &mut self.program_scene] .into_iter() - .chain(fading) + .chain(leaving) } - /// Put the preview on program at once, ending any fade. + /// Put the preview on program at once, ending any transition. pub fn cut(&mut self) { - self.fade = None; + self.switch = None; self.program_scene = self.preview_scene.clone(); } - /// Fade the preview onto program over `duration`, starting at `now`. + /// Take the preview to program by `transition`, starting at `now`. /// - /// A fade already under way lands on its scene first, so this one starts - /// from what that fade was heading to. - pub fn fade(&mut self, duration: Duration, now: Instant) { + /// A transition already under way lands on its scene first, so this one + /// starts from what that one was heading to. + pub fn transition(&mut self, transition: Transition, now: Instant) { let from = std::mem::replace(&mut self.program_scene, self.preview_scene.clone()); - self.fade = (!duration.is_zero()).then_some(Fade { + self.switch = (!transition.duration().is_zero()).then_some(Switch { from, start: now, - duration, + transition, }); } @@ -249,12 +257,12 @@ impl Compositor { self.update(collection, sources, now); self.paint(collection, now); - let progress = self.fade.as_ref().map(|fade| { - let elapsed = now.saturating_duration_since(fade.start); - (elapsed.as_secs_f32() / fade.duration.as_secs_f32()).min(1.0) + let linear = self.switch.as_ref().map(|switch| { + let elapsed = now.saturating_duration_since(switch.start); + (elapsed.as_secs_f32() / switch.transition.duration().as_secs_f32()).min(1.0) }); - if progress.is_some_and(|progress| progress >= 1.0) { - self.fade = None; + if linear.is_some_and(|linear| linear >= 1.0) { + self.switch = None; } let mut frame = self.effects.frame(); @@ -266,24 +274,54 @@ impl Compositor { prepared: HashMap::new(), }; - // The top layer goes over both outputs, and over a fade rather than - // through it, so it stays on across the switch. + // The top layer goes over both outputs, and over a transition rather + // than through it, so it stays on across the switch. draw.target(&self.preview); draw.scene(&self.preview, self.preview_scene.as_deref()); draw.layer(&self.preview, Place::Top); - match (&self.fade, progress) { - (Some(fade), Some(progress)) => { - draw.target(&self.incoming); - draw.scene(&self.incoming, self.program_scene.as_deref()); - draw.target(&self.program); - draw.scene(&self.program, fade.from.as_deref()); - let incoming = draw.effects.fade(&self.incoming.picture, progress); - draw.effects.draw(draw.frame, &self.program.texture, &incoming); + let to = self.program_scene.as_deref(); + match (&self.switch, linear) { + (Some(switch), Some(linear)) => { + let from = switch.from.as_deref(); + let progress = switch.transition.progress(linear); + let canvas = Rect::fill(&self.canvas); + match switch.transition { + Transition::Fade { .. } => { + draw.whole(&self.incoming, to); + draw.target(&self.program); + draw.scene(&self.program, from); + draw.place(&self.program, &self.incoming, canvas, canvas, progress); + } + // Uncovered where a slide would have moved it in. + Transition::Wipe { direction, .. } => { + draw.whole(&self.incoming, to); + draw.target(&self.program); + draw.scene(&self.program, from); + let (x, y) = offset(direction, progress, &canvas); + let (dx, dy) = offset(direction, 1.0, &canvas); + let uncovered = shifted(canvas, x - dx, y - dy); + draw.place(&self.program, &self.incoming, canvas, uncovered, 1.0); + } + Transition::Slide { direction, .. } => { + draw.whole(&self.incoming, to); + draw.whole(&self.outgoing, from); + draw.target(&self.program); + let (x, y) = offset(direction, progress, &canvas); + let (dx, dy) = offset(direction, 1.0, &canvas); + draw.place(&self.program, &self.outgoing, shifted(canvas, x, y), canvas, 1.0); + let entering = shifted(canvas, x - dx, y - dy); + draw.place(&self.program, &self.incoming, entering, canvas, 1.0); + } + Transition::Move { .. } => { + draw.target(&self.program); + draw.moving(&self.program, from, to, progress); + } + } } _ => { draw.target(&self.program); - draw.scene(&self.program, self.program_scene.as_deref()); + draw.scene(&self.program, to); } } draw.layer(&self.program, Place::Top); @@ -413,7 +451,7 @@ impl Compositor { /// The sources whose pictures preview and program draw: each visible /// item's, and each mask's, through clones. fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { - let leaving = self.fade.as_ref().and_then(|fade| fade.from.as_deref()); + let leaving = self.switch.as_ref().and_then(|switch| switch.from.as_deref()); let scenes = [self.preview_scene.as_deref(), self.program_scene.as_deref(), leaving] .into_iter() .flatten() @@ -438,6 +476,113 @@ impl Compositor { } } +/// How far moving `amount` of `canvas`'s width or height in `direction` goes, +/// in whole pixels so a moving scene stays sharp and two meet without a seam. +fn offset(direction: Direction, amount: f32, canvas: &Rect) -> (f32, f32) { + let (x, y) = match direction { + Direction::Left => (-canvas.width, 0.0), + Direction::Right => (canvas.width, 0.0), + Direction::Up => (0.0, -canvas.height), + Direction::Down => (0.0, canvas.height), + }; + ((x * amount).round(), (y * amount).round()) +} + +/// `rect` moved `x` right and `y` down. +fn shifted(rect: Rect, x: f32, y: f32) -> Rect { + Rect { + x: rect.x + x, + y: rect.y + y, + ..rect + } +} + +/// Where an item is drawn and how much of it shows: what a move transition +/// changes as it goes. +#[derive(Clone, Copy, PartialEq)] +struct Pose { + rect: Rect, + crop: Crop, + opacity: f32, +} + +impl Pose { + /// `item` where `collection` places it. + fn of(collection: &Collection, item: &Item) -> Self { + Self { + rect: item.rect, + crop: collection.crop(item), + opacity: 1.0, + } + } + + /// `progress` of the way from `self` to `other`. + fn toward(&self, other: &Self, progress: f32) -> Self { + let mix = |from: f32, to: f32| crate::collection::mix(from, to, progress); + Self { + rect: Rect { + x: mix(self.rect.x, other.rect.x), + y: mix(self.rect.y, other.rect.y), + width: mix(self.rect.width, other.rect.width), + height: mix(self.rect.height, other.rect.height), + }, + crop: Crop { + left: mix(self.crop.left, other.crop.left), + top: mix(self.crop.top, other.crop.top), + right: mix(self.crop.right, other.crop.right), + bottom: mix(self.crop.bottom, other.crop.bottom), + }, + opacity: mix(self.opacity, other.opacity), + } + } +} + +/// `old`'s filters `progress` of the way to `new`'s, or `None` if their blend +/// modes differ or their filters do not pair up by kind, in order, as +/// [`Filter::toward`] mixes them. +fn mixed(old: &Item, new: &Item, progress: f32) -> Option> { + if old.blend != new.blend || old.filters.len() != new.filters.len() { + return None; + } + let pairs = old.filters.iter().zip(&new.filters); + pairs.map(|(old, new)| old.toward(new, progress)).collect() +} + +/// The order to draw a move's visible items in, back to front, as `lead` +/// stacks them: each slot is an item of `lead`, of `other`, or a pair of +/// both, as `lead_pairs` and `other_pairs` pair them. An unpaired item of +/// `other` goes just over the paired item it was over in `other`, or under +/// every slot if it was under every paired item. +fn stack( + lead: &[Item], + lead_pairs: &[Option], + other: &[Item], + other_pairs: &[Option], +) -> Vec<(Option, Option)> { + let mut over: HashMap, Vec> = HashMap::new(); + let mut under = None; + for index in (0..other.len()).filter(|&index| other[index].visible) { + match other_pairs[index] { + Some(_) => under = Some(index), + None => over.entry(under).or_default().push(index), + } + } + let mut unpaired = |under: Option| { + let items = over.remove(&under).unwrap_or_default(); + items.into_iter().map(|index| (None, Some(index))) + }; + + let mut slots: Vec<_> = unpaired(None).collect(); + for index in (0..lead.len()).filter(|&index| lead[index].visible) { + let pair = lead_pairs[index]; + slots.push((Some(index), pair)); + if pair.is_some() { + slots.extend(unpaired(pair)); + } + } + slots +} + /// Refuse an invalid canvas, or one larger than `device` can hold as a /// texture. fn fits(device: &wgpu::Device, canvas: Canvas) -> Result<(), Error> { @@ -475,35 +620,152 @@ impl Draw<'_> { } } + /// Start drawing `target`, and draw `scene` onto it whole. + fn whole(&mut self, target: &Target, scene: Option<&str>) { + self.target(target); + self.scene(target, scene); + } + + /// Draw `picture`, a canvas-sized target drawn this render, onto `target` + /// stretched over `at`, showing only within `within`, at `opacity`. + fn place(&mut self, target: &Target, picture: &Target, at: Rect, within: Rect, opacity: f32) { + if let Some(placed) = self.effects.place(&picture.picture, at, within, opacity) { + self.effects.draw(self.frame, &target.texture, &placed); + } + } + /// Draw `place`'s items onto `target`, back to front, each through its crop, /// filters, and blend mode. A hidden item, a source with no picture yet, or /// a filter whose source has nothing yet draws nothing. fn layer(&mut self, target: &Target, place: Place) { let items = self.collection.items(&place).map_or(&[][..], Vec::as_slice); for (index, item) in items.iter().enumerate().filter(|(_, item)| item.visible) { - let key = (place.clone(), index); - let prepared = match self.prepared.get(&key) { - Some(prepared) => prepared.clone(), - None => { - let prepared = self.prepare(item); - self.prepared.insert(key, prepared.clone()); - prepared + self.item(target, &place, index, item); + } + } + + /// Draw `item`, at `index` of `place`, onto `target` where it is placed. + fn item(&mut self, target: &Target, place: &Place, index: usize, item: &Item) { + let key = (place.clone(), index); + let prepared = match self.prepared.get(&key) { + Some(prepared) => prepared.clone(), + None => { + let prepared = self.prepare(item, Pose::of(self.collection, item)); + self.prepared.insert(key, prepared.clone()); + prepared + } + }; + if let Some(prepared) = prepared { + self.effects.draw(self.frame, &target.texture, &prepared); + } + } + + /// Draw `item` onto `target` in `pose` rather than where it is placed. + fn posed(&mut self, target: &Target, item: &Item, pose: Pose) { + if pose.opacity <= 0.0 { + return; + } + if let Some(prepared) = self.prepare(item, pose) { + self.effects.draw(self.frame, &target.texture, &prepared); + } + } + + /// Draw `progress` of a move from scene `from` to scene `to` onto + /// `target`. + /// + /// Each visible item of `to` is paired with the first visible item of + /// `from` showing the same source that no earlier one took, and glides + /// from its pose to its own, its filters' settings mixing from its look in + /// `from` to its look in `to`. The rest of `from` + /// fades out and the rest of `to` fades in, where each was placed. Items + /// stack as `from` stacks them until halfway, then as `to` does, each + /// unpaired item just over the paired item it was over, so the move + /// starts from `from` and lands on `to` as each is drawn. A paired item + /// whose blend mode or kinds of filter differ changes look halfway too. + fn moving(&mut self, target: &Target, from: Option<&str>, to: Option<&str>, progress: f32) { + let collection = self.collection; + let items = |scene: Option<&str>| { + let scene = scene.and_then(|name| collection.scene(name)); + scene.map_or(&[][..], |scene| scene.items.as_slice()) + }; + let (leaving, entering) = (items(from), items(to)); + + // Which item of the other scene each visible item pairs with. + let mut left = vec![None; leaving.len()]; + let mut entered = vec![None; entering.len()]; + for (index, item) in entering.iter().enumerate().filter(|(_, item)| item.visible) { + let pair = (0..leaving.len()) + .find(|&at| left[at].is_none() && leaving[at].visible && leaving[at].source == item.source); + if let Some(at) = pair { + left[at] = Some(index); + entered[index] = Some(at); + } + } + + // Each slot is an index into `leaving`, `entering`, or both. + let slots: Vec<(Option, Option)> = if progress < 0.5 { + stack(leaving, &left, entering, &entered) + } else { + let slots = stack(entering, &entered, leaving, &left); + slots.into_iter().map(|(to, from)| (from, to)).collect() + }; + + let place = Place::Scene(to.unwrap_or_default().to_string()); + for slot in slots { + match slot { + (Some(at), None) => { + let pose = Pose::of(collection, &leaving[at]); + let opacity = 1.0 - progress; + self.posed(target, &leaving[at], Pose { opacity, ..pose }); } - }; - if let Some(prepared) = prepared { - self.effects.draw(self.frame, &target.texture, &prepared); + (None, Some(index)) => { + let pose = Pose::of(collection, &entering[index]); + self.posed( + target, + &entering[index], + Pose { + opacity: progress, + ..pose + }, + ); + } + (Some(at), Some(index)) => { + let (old, new) = (&leaving[at], &entering[index]); + let own = Pose::of(collection, new); + let pose = Pose::of(collection, old).toward(&own, progress); + if old.filters == new.filters && old.blend == new.blend { + if pose == own { + // Shared with the preview, which draws it the same. + self.item(target, &place, index, new); + } else { + self.posed(target, new, pose); + } + } else if let Some(filters) = mixed(old, new, progress) { + let item = Item { filters, ..new.clone() }; + self.posed(target, &item, pose); + } else { + // A look that cannot be mixed changes as the stacking does. + self.posed(target, if progress < 0.5 { old } else { new }, pose); + } + } + (None, None) => unreachable!("a slot holds an item"), } } } - fn prepare(&mut self, item: &Item) -> Option { + fn prepare(&mut self, item: &Item, pose: Pose) -> Option { let collection = self.collection; let picture = self.library.0.get(collection.capture(&item.source))?.picture.as_ref()?; + let fade = Filter::Opacity { amount: pose.opacity }; + let mut filters: Vec<&Filter> = collection.filters(item).collect(); + if pose.opacity < 1.0 { + filters.push(&fade); + } let item = effect::Item { picture, - rect: item.rect, - crop: collection.crop(item), - filters: collection.filters(item).collect(), + rect: pose.rect, + crop: pose.crop, + filters, blend: item.blend, }; self.effects.prepare(self.frame, item, &self.library) @@ -583,12 +845,20 @@ impl Target { #[cfg(test)] mod tests { + use std::time::Duration; + use super::*; - use crate::collection::{Blend, Filter, MaskMode, Rect, Rgba, Scene}; + use crate::collection::{Blend, MaskMode, Rgba, Scene}; use crate::source::test::{context, local, start}; + /// A fade over a second. + const FADE: Transition = Transition::Fade { + duration: Duration::from_secs(1), + }; + const RED: [u8; 4] = [255, 0, 0, 255]; const BLUE: [u8; 4] = [0, 0, 255, 255]; + const GREEN: [u8; 4] = [0, 255, 0, 255]; const BLACK: [u8; 4] = [0, 0, 0, 255]; async fn gpu() -> (wgpu::Device, wgpu::Queue) { @@ -791,11 +1061,11 @@ mod tests { studio.compositor.set_preview(Some("red".into())); studio.compositor.cut(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.fade(Duration::from_secs(1), start); + studio.compositor.transition(FADE, start); assert_eq!(studio.compositor.program_scene(), Some("blue")); studio.render(start + Duration::from_millis(500)); - assert!(studio.compositor.fading()); + assert!(studio.compositor.transitioning()); let program = studio.program(); let actual = pixel(&program, 1, 1); assert!(near(actual, [128, 0, 127, 255]), "{actual:?}"); @@ -803,7 +1073,7 @@ mod tests { assert_eq!(pixel(&studio.preview(), 1, 1), BLUE); studio.render(start + Duration::from_secs(1)); - assert!(!studio.compositor.fading()); + assert!(!studio.compositor.transitioning()); assert_eq!(pixel(&studio.program(), 1, 1), BLUE); }) .await; @@ -868,7 +1138,7 @@ mod tests { assert_eq!(pixel(&studio.program(), 1, 6), BLUE); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.fade(Duration::from_secs(1), start); + studio.compositor.transition(FADE, start); studio.render(start + Duration::from_millis(500)); let program = studio.program(); let actual = pixel(&program, 1, 1); @@ -893,17 +1163,183 @@ mod tests { assert_eq!(pixel(&studio.program(), 1, 1), BLACK); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.fade(Duration::from_secs(1), start); + studio.compositor.transition(FADE, start); studio.render(start + Duration::from_millis(500)); studio.compositor.set_preview(Some("red".into())); studio.compositor.cut(); - assert!(!studio.compositor.fading()); + assert!(!studio.compositor.transitioning()); studio.render(start + Duration::from_millis(600)); assert_eq!(pixel(&studio.program(), 1, 1), RED); }) .await; } + /// Halfway through a slide to the left, eased to half the canvas, the + /// outgoing scene's right half shows on the left and the incoming scene's + /// left half on the right; a wipe leaves the outgoing scene in place and + /// uncovers the incoming one where the slide brought it in. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_slide_pushes_and_a_wipe_uncovers() { + local(async { + let mut collection = collection(); + collection.sources.insert( + "green".into(), + Source::Color { + color: Rgba([0, 255, 0, 255]), + }, + ); + collection.scenes[1].items = vec![Item::new( + "green", + Rect { + x: 0.0, + y: 0.0, + width: 4.0, + height: 8.0, + }, + )]; + let mut studio = Studio::new(collection).await; + let duration = Duration::from_secs(1); + for (transition, expected) in [ + ( + Transition::Slide { + duration, + direction: Direction::Left, + }, + [BLUE, RED, GREEN, GREEN], + ), + ( + Transition::Wipe { + duration, + direction: Direction::Left, + }, + [RED, RED, BLACK, BLACK], + ), + ( + Transition::Wipe { + duration, + direction: Direction::Down, + }, + [GREEN, RED, BLACK, RED], + ), + ] { + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.transition(transition, start); + studio.render(start + Duration::from_millis(500)); + let program = studio.program(); + let actual = [(1, 1), (1, 6), (6, 1), (6, 6)].map(|(x, y)| pixel(&program, x, y)); + assert_eq!(actual, expected, "{transition:?}"); + assert_eq!(pixel(&studio.preview(), 1, 1), GREEN, "preview stays put"); + } + }) + .await; + } + + /// A move glides an item both scenes place from one place to the other, + /// fades out what only the outgoing scene places, still over what it was + /// over, and fades in what only the incoming one does, landing on the + /// incoming scene as drawn. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_move_glides_what_both_scenes_place() { + local(async { + let mut collection = collection(); + collection.sources.insert( + "green".into(), + Source::Color { + color: Rgba([0, 255, 0, 255]), + }, + ); + let tile = Rect { + x: 6.0, + y: 0.0, + width: 2.0, + height: 2.0, + }; + collection + .sources + .insert("white".into(), Source::Color { color: Rgba([255; 4]) }); + let badge = Rect { + x: 4.0, + y: 2.0, + width: 2.0, + height: 2.0, + }; + collection.scenes[0].items.push(Item::new("white", badge)); + collection.scenes[1].items = vec![Item::new("blue", CORNER), Item::new("green", tile)]; + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + let duration = Duration::from_secs(1); + studio.compositor.transition(Transition::Move { duration }, start); + + // Eased halfway, blue is at (2, 2) and both fades are at half. + studio.render(start + Duration::from_millis(500)); + let program = studio.program(); + assert_eq!(pixel(&program, 3, 3), BLUE, "blue in between"); + assert_eq!(pixel(&program, 5, 4), BLUE); + let actual = pixel(&program, 5, 3); + assert!(near(actual, [128, 128, 255, 255]), "white half over blue: {actual:?}"); + let actual = pixel(&program, 1, 6); + assert!(near(actual, [128, 0, 0, 255]), "red half gone: {actual:?}"); + let actual = pixel(&program, 7, 0); + assert!(near(actual, [64, 128, 0, 255]), "green half over half red: {actual:?}"); + + studio.render(start + duration); + assert!(!studio.compositor.transitioning()); + let program = studio.program(); + assert_eq!(pixel(&program, 1, 6), BLUE); + assert_eq!(pixel(&program, 6, 6), BLACK); + assert_eq!(pixel(&program, 7, 0), GREEN); + }) + .await; + } + + /// A move starts from the outgoing scene's stacking and switches to the + /// incoming scene's halfway, and an item's filter settings mix from one + /// scene's to the other's as it goes. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_move_restacks_halfway_and_mixes_filters() { + local(async { + let mut collection = collection(); + let fill = Rect::fill(&collection.canvas); + let invert = |amount| Item { + filters: vec![Filter::Invert { amount }], + ..Item::new("blue", fill) + }; + collection.scenes[0].items = vec![Item::new("red", CORNER), invert(0.0)]; + collection.scenes[1].items = vec![invert(1.0), Item::new("red", CORNER)]; + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + let duration = Duration::from_secs(1); + studio.compositor.transition(Transition::Move { duration }, start); + + studio.render(start); + let program = studio.program(); + assert_eq!(pixel(&program, 1, 6), BLUE, "blue over red, as the outgoing scene"); + assert_eq!(pixel(&program, 6, 1), BLUE); + + studio.render(start + Duration::from_millis(500)); + let program = studio.program(); + assert_eq!(pixel(&program, 1, 6), RED, "red over blue from halfway"); + let actual = pixel(&program, 6, 1); + assert!(near(actual, [128, 128, 128, 255]), "half inverted: {actual:?}"); + + studio.render(start + duration); + assert_eq!(pixel(&studio.program(), 6, 1), [255, 255, 0, 255]); + }) + .await; + } + /// A source with no new picture repeats its last one; an edited source never /// shows the old one's picture, and a removed source disappears. #[tokio::test] @@ -1403,7 +1839,7 @@ mod tests { studio.compositor.set_preview(Some("red".into())); studio.compositor.cut(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.fade(Duration::from_secs(1), start); + studio.compositor.transition(FADE, start); // `blue` takes the name `red` gives up, which is followed once. let mut draft = studio.collection.draft("Rename"); @@ -1510,6 +1946,49 @@ mod tests { .await; } + /// An overlay a move glides is painted on the program clock while both + /// scenes are on air, and drawn where the move has it. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_move_glides_an_overlay() { + local(async { + let mut collection = collection(); + collection.sources.insert( + "overlay".into(), + Source::Overlay { + html: "".into(), + width: 4, + height: 4, + }, + ); + let tile = |x, y| Rect { + x, + y, + width: 2.0, + height: 2.0, + }; + collection.scenes[0].items = vec![Item::new("overlay", tile(0.0, 0.0))]; + collection.scenes[1].items = vec![Item::new("overlay", tile(4.0, 4.0))]; + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + let duration = Duration::from_secs(1); + studio.compositor.transition(Transition::Move { duration }, start); + + studio.render(start + Duration::from_millis(500)); + let program = studio.program(); + assert_eq!(pixel(&program, 2, 2), GREEN, "halfway, at (2, 2)"); + assert_eq!(pixel(&program, 0, 0), BLACK); + assert_eq!(pixel(&program, 5, 5), BLACK); + + studio.render(start + duration); + assert_eq!(pixel(&studio.program(), 5, 5), GREEN); + }) + .await; + } + /// When the overlay thread stops, every overlay it held fails, including /// one off air, rather than showing its last paint when cut to. #[tokio::test] diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index 44d6d40..e7db3a4 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -608,7 +608,7 @@ impl Draft { item.ok_or_else(|| conflict(format!("there is no item {index} in {layer}"))) } - /// Set how a fade behaves. + /// Set how the preview is taken to program, short of a cut. pub fn set_transition(&mut self, transition: Transition) -> Result<(), Error> { let before = self.collection.transition; self.change(Change::Transition { @@ -1158,10 +1158,10 @@ mod tests { let start = collection.clone(); let mut history = History::new(); for millis in [400, 500, 600] { - let mut draft = collection.draft("Set the fade time"); + let mut draft = collection.draft("Set the transition"); draft - .set_transition(Transition { - fade: std::time::Duration::from_millis(millis), + .set_transition(Transition::Fade { + duration: std::time::Duration::from_millis(millis), }) .unwrap(); commit(&mut collection, &mut history, draft, Some(1)); diff --git a/rs/moq-scene/src/effect.rs b/rs/moq-scene/src/effect.rs index 6b8be00..b1a0ca1 100644 --- a/rs/moq-scene/src/effect.rs +++ b/rs/moq-scene/src/effect.rs @@ -933,25 +933,32 @@ impl Effects { }) } - /// A whole canvas-sized `picture` faded to `opacity`, to draw over another. - pub fn fade(&self, picture: &Picture, opacity: f32) -> Prepared { - let canvas = Area::canvas(self.canvas); - Prepared { + /// A whole canvas-sized `picture` stretched over `at`, showing only within + /// `within`, at `opacity`, to draw over another: `None` if none of it + /// shows on the canvas. + pub fn place(&self, picture: &Picture, at: Rect, within: Rect, opacity: f32) -> Option { + let area = Area::of(at); + let bounds = area.intersect(&Area::of(within)).intersect(&Area::canvas(self.canvas)); + if bounds.is_empty() || opacity <= 0.0 { + return None; + } + let ops = (opacity < 1.0).then(|| Op::of(&Filter::Opacity { amount: opacity }).expect("per pixel")); + Some(Prepared { input: Input { picture: picture.clone(), - area: canvas, + area, uv: UNIT, valid: UNIT, premultiplied: false, }, segment: Segment { - ops: vec![Op::of(&Filter::Opacity { amount: opacity }).expect("per pixel")], + ops: ops.into_iter().collect(), ..Default::default() }, - item: canvas, - bounds: canvas, + item: area, + bounds, blend: Blend::Normal, - } + }) } /// Draw `prepared` onto the target `frame` last started, which is `texture`. diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index b08e144..5155270 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -17,9 +17,9 @@ //! compositor and sound goes to a `moq_audio::bus::Bus`, routed into its //! program mix. //! - [`Compositor`] draws the scene in preview and the scene on program into -//! textures on the caller's `wgpu` device, cutting or fading between them. -//! `moq_video::render::export` hands the program to an encoder, on the GPU -//! where the platform allows. +//! textures on the caller's `wgpu` device, cutting or running a +//! [`Transition`] between them. `moq_video::render::export` hands the +//! program to an encoder, on the GPU where the platform allows. //! - [`publish`] gives each local source its own track in the studio's //! broadcast, so viewers can compose the scene themselves. //! @@ -42,7 +42,8 @@ pub mod source; pub use clip::Clip; pub use collection::{ - Audio, Blend, Canvas, Collection, Crop, Filter, Item, Layer, MaskMode, Rect, Rgba, Scene, Source, Transition, Video, + Audio, Blend, Canvas, Collection, Crop, Direction, Filter, Item, Layer, MaskMode, Rect, Rgba, Scene, Source, + Transition, Video, }; pub use compositor::{Compositor, FORMAT}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index d8a131e..ccc7582 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -664,7 +664,7 @@ impl Engine { fn local(&self, now: Instant) -> health::Local { let stats = &self.stats; // A stalled picture matters where the program draws it: a scene on air, - // including one fading out, or the top layer over them. Stalled sound + // including one leaving in a transition, or the top layer over them. Stalled sound // matters wherever the program plays it. let stalled = { let state = self.state.lock().unwrap(); @@ -1626,7 +1626,7 @@ mod tests { }; let removed = [remove(&collection, 0)]; assert!(keep_on_air([live.as_str()], &removed).is_err()); - assert!(keep_on_air(["Renamed", live.as_str()], &removed).is_err(), "fading out"); + assert!(keep_on_air(["Renamed", live.as_str()], &removed).is_err(), "leaving"); keep_on_air([live.as_str()], &[remove(&collection, 1)]).unwrap(); keep_on_air([], &removed).unwrap(); diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 5f80a5e..4a9b53a 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -12,8 +12,8 @@ use moq_scene::browser::Paint; use moq_scene::replay::Take; use moq_scene::source::{Feed, Status}; use moq_scene::{ - Blend, Canvas, Clip, Collection, Crop, Draft, Filter, Item, Layer, MaskMode, Rect, Scene, Source, Transition, - Video, wgpu, + Blend, Canvas, Clip, Collection, Crop, Direction, Draft, Filter, Item, Layer, MaskMode, Rect, Scene, Source, + Transition, Video, wgpu, }; use crate::autosave::Conflict; @@ -105,6 +105,7 @@ impl Studio { selection: None, gestures: Gestures::default(), snapping: Snapping::default(), + direction: Direction::Left, drag: None, new_scene: String::new(), renaming: None, @@ -207,6 +208,9 @@ struct Panels { selection: Option, gestures: Gestures, snapping: Snapping, + /// The direction a slide or wipe last had, which the transition picker + /// offers again after a fade or move. + direction: Direction, /// The item a drag in the preview holds. drag: Option, new_scene: String, @@ -848,34 +852,71 @@ impl Panels { let engine = self.engine.clone(); let mut state = engine.state.lock().unwrap(); let canvas = state.collection().canvas; - let fade = state.collection().transition.fade; - let mut result = Ok(()); + let transition = state.collection().transition; + let mut next = transition; + let mut gesture = None; ui.horizontal(|ui| { if ui.button("Cut").clicked() { state.compositor.cut(); } - if ui.button("Fade").clicked() { - state.compositor.fade(fade, kio::clock::Instant::now()); + if ui + .button("Transition") + .on_hover_text("Take the preview to program by the transition set beside it") + .clicked() + { + state.compositor.transition(transition, kio::clock::Instant::now()); + } + let duration = next.duration(); + if let Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } = next { + self.direction = direction; + } + let direction = self.direction; + egui::ComboBox::from_id_salt("transition") + .selected_text(transition_name(&next)) + .show_ui(ui, |ui| { + let kinds = [ + Transition::Fade { duration }, + Transition::Slide { duration, direction }, + Transition::Wipe { duration, direction }, + Transition::Move { duration }, + ]; + for kind in kinds { + ui.selectable_value(&mut next, kind, transition_name(&kind)); + } + }); + if let Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } = &mut next { + egui::ComboBox::from_id_salt("direction") + .selected_text(direction_name(*direction)) + .show_ui(ui, |ui| { + for to in Direction::ALL { + ui.selectable_value(direction, to, direction_name(to)); + } + }); } - let mut millis = fade.as_millis() as u64; + let mut millis = duration.as_millis() as u64; let response = ui .add(egui::DragValue::new(&mut millis).suffix(" ms")) - .on_hover_text("How long a fade takes"); - let gesture = self.gestures.of(&response); + .on_hover_text("How long the transition takes"); + let dragging = self.gestures.of(&response); if response.changed() { - let fade = Duration::from_millis(millis); - result = edit(&mut state, "Set the fade time", gesture, |draft| { - draft.set_transition(Transition { fade }) - }); + *next.duration_mut() = Duration::from_millis(millis); + gesture = dragging; } ui.label(format!( "Program: {}", state.compositor.program_scene().unwrap_or("nothing") )); - if state.compositor.fading() { - ui.weak("fading"); + if state.compositor.transitioning() { + ui.weak("in transition"); } }); + let result = if next == transition { + Ok(()) + } else { + edit(&mut state, "Set the transition", gesture, |draft| { + draft.set_transition(next) + }) + }; drop(state); self.check(result); picture(ui, self.program.id, canvas, egui::Sense::hover()); @@ -2264,6 +2305,24 @@ fn filter_name(filter: &Filter) -> &'static str { } } +fn transition_name(transition: &Transition) -> &'static str { + match transition { + Transition::Fade { .. } => "fade", + Transition::Slide { .. } => "slide", + Transition::Wipe { .. } => "wipe", + Transition::Move { .. } => "move", + } +} + +fn direction_name(direction: Direction) -> &'static str { + match direction { + Direction::Left => "left", + Direction::Right => "right", + Direction::Up => "up", + Direction::Down => "down", + } +} + fn blend_name(blend: Blend) -> &'static str { match blend { Blend::Normal => "normal",