diff --git a/README.md b/README.md index cf507872..8c67c082 100644 --- a/README.md +++ b/README.md @@ -132,6 +132,30 @@ Each further canvas publishes a broadcast of its own beneath the main one, at encoding it again. Removing a canvas ends its broadcast, and removes its top layer and every scene's layout on it. +### Scene sources + +A **scene source** draws another scene as one picture, which an item places, +scales, crops, filters, and blends like any other: a lower third built once +and shown in every scene, or a whole layout shrunk into a corner. Add one in +the Sources panel as kind **Scene**, picking the scene it draws, or +**Program**, which shows whatever is on air, a transition under way included, +on the same frame, such as for a portrait canvas that follows the program, +cropped. + +A nested scene is drawn at the main canvas's size, transparent where it draws +nothing, once per frame however many items show it, and only while something +drawn shows it. Its items' keyframes play from when the scene showing it goes +on, as that scene's own items' do, so the top layer, which never goes on, +cannot show a scene with keyframes. A media file or overlay in a nested scene +set to restart starts over when the scene showing it goes on program. + +A scene cannot draw itself, even through other scenes, so an edit that would +is refused. A scene showing the program cannot go on program, nor can the main +canvas's top layer show it, since either would draw the program into itself: +**Cut** and **Transition** refuse such a scene, and an edit making the scene on +air show the program is refused. Renaming a scene renames what its scene +sources draw, and removing one removes them, with the items showing them. + ### Transitions | Transition | What it does | @@ -543,6 +567,13 @@ delay = "80ms" kind = "clone" source = "camera" +[sources.main-view] +kind = "scene" +scene = "Main" # draws the Main scene as one picture + +[sources.on-air] +kind = "program" # whatever the program shows + [video.camera-faded] filters = [{ kind = "opacity", amount = 0.5 }] @@ -579,6 +610,7 @@ items = [ { at = "0s", x = 700, filters = [{ kind = "blur", radius = 12 }] }, { at = "500ms", ease = "out", filters = [{ kind = "blur", radius = 0 }] }, ] }, + { source = "main-view", rect = { x = 80, y = 80, width = 640, height = 360 } }, ] [[top]] @@ -593,7 +625,8 @@ Source kinds are `camera`, `display`, and `microphone` (each with an optional `broadcast`, `browser` (a `url`, and a `width` and `height` up to 4096), `overlay` (its `html`, a `width` and `height` up to 4096, and whether it `restart`s, true by default), `clone` (with the `source` it shows), `lut` (a 3D Adobe or Resolve `.cube` file, which no item places, but which grades what a `lut` filter names it in), -and `replay`, where the Replay panel plays replays. The Devices panel fills in +`replay`, where the Replay panel plays replays, `scene` (with the `scene` it +draws), and `program` (see [Scene sources](#scene-sources)). The Devices panel fills in ids. `[[top]]` items make up the top layer. `[canvases.]` adds a further canvas, with its `width` and `height` and diff --git a/quest/m0/scenes/README.md b/quest/m0/scenes/README.md index db94bf4c..b41a9ab7 100644 --- a/quest/m0/scenes/README.md +++ b/quest/m0/scenes/README.md @@ -51,6 +51,5 @@ reads skips drawing. Per-scene cost grows only with what that scene draws; ## Required -- [Scene source](/quest/m0/scenes/scene-source.md) - a scene draws another scene, or the live program, as a source - [Published scenes](/quest/m0/scenes/published.md) - any scene publishes as its own broadcast at its own size, replacing canvases - [Audio scenes](/quest/m0/scenes/audio.md) - mixes become audio-only scenes that nest diff --git a/quest/m0/scenes/audio.md b/quest/m0/scenes/audio.md index 119f86e9..4120aac4 100644 --- a/quest/m0/scenes/audio.md +++ b/quest/m0/scenes/audio.md @@ -23,7 +23,3 @@ Decided 2026-11-05 with the maintainer (/quest-plan); see the - Switching or editing an audio scene never drops or repeats audio, as with mixes today. - `README.md` documents audio scenes. - -## Required - -- [Scene source](/quest/m0/scenes/scene-source.md) - shares the scene graph and its cycle check diff --git a/quest/m0/scenes/published.md b/quest/m0/scenes/published.md index ef6b2817..3c5514e8 100644 --- a/quest/m0/scenes/published.md +++ b/quest/m0/scenes/published.md @@ -7,7 +7,7 @@ broadcast at `<--broadcast>/`, at its own size, with the audio scene it picks (the one on program by default). This replaces canvases: further canvases, their per-scene layouts, and the canvas picker are removed, and a portrait broadcast is a published portrait scene that draws the program -through a [scene source](/quest/m0/scenes/scene-source.md). +through a program source (`kind = "program"`). ## Plan @@ -15,7 +15,8 @@ Decided 2026-11-05 with the maintainer (/quest-plan); see the [epic](/quest/m0/scenes/README.md). - Every scene has a size, defaulting to the program's; the switcher offers - only scenes of the program's size. + only scenes of the program's size. A scene source then draws its scene at + that size, rather than the main canvas's as it does now. - The program keeps one switcher at `<--broadcast>`. A published scene is always live and never switched. - Each published scene gets the program's two sizes (full and half) and the @@ -26,7 +27,3 @@ Decided 2026-11-05 with the maintainer (/quest-plan); see the and no output reads skips drawing. - Remove `[canvases]` from the scene file with no migration; nothing is released. `README.md` documents published scenes. - -## Required - -- [Scene source](/quest/m0/scenes/scene-source.md) - a portrait scene follows the program through it diff --git a/quest/m0/scenes/scene-source.md b/quest/m0/scenes/scene-source.md deleted file mode 100644 index ab40477f..00000000 --- a/quest/m0/scenes/scene-source.md +++ /dev/null @@ -1,23 +0,0 @@ -# [M] Scene source - -## Goal - -A new source kind, `scene`, draws another scene, or `program` (whatever is on -air, including a transition in progress), as a picture that an item places, -scales, crops, and filters like any other. A scene cannot contain itself, -directly or through other scenes; such an edit is refused with an error. - -## Plan - -Decided 2026-11-05 with the maintainer (/quest-plan); see the -[epic](/quest/m0/scenes/README.md) for the model. - -- A nested scene renders at its own size into a texture the compositor - samples, once per frame however many items draw it, and only while - something that is drawn reads it. -- `program` reads the frame the program output draws, so a scene drawing it - shows the same cut on the same frame, with no extra composite. -- Keyframes, stingers, and restarts in a nested scene start when the scene - that draws it goes on program, as its items would. -- Measure frame time and GPU copies with and without a nested scene. -- `README.md` documents the source kind. diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index 67878827..9e3f96cd 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -102,11 +102,26 @@ impl Clip { match &clip { // The rules a collection holding just the clip would break, such as // an item showing a source the clip lacks. It has whatever mixes the - // clip's sources are left out of, which pasting checks. + // clip's sources are left out of, and an empty scene for each its + // scene sources draw, which pasting checks. Self::Items(items) => { let excluded = items.audio.values().flat_map(|audio| &audio.exclude); let mut names: BTreeSet = excluded.cloned().collect(); names.insert(Mixes::DEFAULT.into()); + let drawn: BTreeSet<&str> = items + .sources + .values() + .filter_map(|source| match source { + Source::Scene { scene } => Some(scene.as_str()), + _ => None, + }) + .collect(); + // Named apart from them, so it draws none of them by name. + let clip = std::iter::once("clip".to_string()) + .chain((2..).map(|n| format!("clip {n}"))) + .find(|name| !drawn.contains(name.as_str())) + .expect("an unused name"); + let drawn = drawn.into_iter().map(Scene::new); Collection { sources: items.sources.clone(), mixes: Mixes { @@ -118,10 +133,12 @@ impl Clip { video: items.video.clone(), // In a scene, where keyframes play; pasting into the top // layer checks it holds none. - scenes: vec![Scene { + scenes: std::iter::once(Scene { items: items.items.clone(), - ..Scene::new("clip") - }], + ..Scene::new(clip) + }) + .chain(drawn) + .collect(), ..Collection::new() } .validate()? @@ -290,6 +307,39 @@ mod tests { pasted } + /// An item showing a scene pastes linked within its collection, but not + /// into one without that scene, nor into the scene it shows. + #[test] + fn a_scene_source_pastes_where_its_scene_is() { + let mut collection = collection(); + let fill = Rect::fill(&collection.canvas); + collection.scenes.push(Scene::new("clip")); + collection.scenes.push(Scene::new("Other")); + let shown = Source::Scene { scene: "clip".into() }; + collection.sources.insert("nested".into(), shown); + collection.scenes[1].items.push(Item::new("cam", fill)); + collection.scenes[2].items.push(Item::new("nested", fill)); + collection.validate().unwrap(); + let clip = Clip::items(&collection, collection.scenes[2].items.clone(), Path::new(DIR)); + let text = clip.to_toml().unwrap(); + + let pasted = paste(&collection, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.sources, collection.sources, "linked"); + + let Clip::Items(items) = Clip::from_toml(&text).unwrap() else { + panic!("not items") + }; + let refused = |collection: &Collection, layer: &Layer| { + let mut draft = collection.draft("Paste"); + draft.paste(layer, &items, Path::new(DIR)).unwrap(); + draft.finish().unwrap_err().to_string() + }; + let err = refused(&collection, &Layer::scene("clip")); + assert!(err.contains("scene `clip` draws itself"), "{err}"); + let err = refused(&Collection::new(), &Layer::scene("Scene")); + assert!(err.contains("draws scene `clip`, which is not a scene"), "{err}"); + } + /// Within a collection, a pasted item shows the very source copied. #[test] fn pastes_linked_within_a_collection() { diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 2bbe1222..b5515420 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -190,6 +190,122 @@ impl Collection { } } + /// What the source called `name` draws if it is a scene or program + /// source, through a clone. + pub fn nested(&self, name: &str) -> Option> { + match self.sources.get(self.capture(name))? { + Source::Scene { scene } => Some(Nested::Scene(scene)), + Source::Program => Some(Nested::Program), + _ => None, + } + } + + /// The visible items `layers` draw, then those of each scene they draw + /// through a scene source, and so on, each scene once. + pub fn drawn(&self, layers: impl IntoIterator) -> Vec<&Item> { + let mut items: Vec<&Item> = layers + .into_iter() + .flat_map(|layer| self.items(&layer).unwrap_or_default()) + .filter(|item| item.visible) + .collect(); + let mut nested = BTreeSet::new(); + let mut next = 0; + while let Some(item) = items.get(next) { + if let Some(Nested::Scene(scene)) = self.nested(&item.source) + && nested.insert(scene) + && let Some(scene) = self.scene(scene) + { + items.extend(scene.items.iter().filter(|item| item.visible)); + } + next += 1; + } + items + } + + /// Whether the scene called `scene` places a program source, itself or in + /// a scene it draws, shown or hidden. It then cannot go on program, which + /// it would draw into itself. + pub fn draws_program(&self, scene: &str) -> bool { + let items = self.reached(scene); + items + .iter() + .any(|item| self.nested(&item.source) == Some(Nested::Program)) + } + + /// Every item of the scene called `scene` and of the scenes it draws, + /// shown or hidden, each scene once. + fn reached(&self, scene: &str) -> Vec<&Item> { + let mut scenes = vec![scene]; + let mut seen = BTreeSet::new(); + let mut items = Vec::new(); + while let Some(scene) = scenes.pop() { + let Some(scene) = self.scene(scene).filter(|scene| seen.insert(&scene.name)) else { + continue; + }; + for item in &scene.items { + if let Some(Nested::Scene(scene)) = self.nested(&item.source) { + scenes.push(scene); + } + items.push(item); + } + } + items + } + + /// Refuse a scene that draws itself, directly or through other scenes, + /// shown or hidden, since it would never finish drawing. + fn validate_nesting(&self) -> Result<(), Error> { + // Walked with a stack of its own rather than the call stack, which a + // long enough chain of scenes would overflow. + let mut done = BTreeSet::new(); + for root in &self.scenes { + if done.contains(root.name.as_str()) { + continue; + } + // The scenes drawing the one walked, each with how many of its + // items are walked. + let mut path = vec![(root, 0)]; + while let Some(&mut (scene, ref mut walked)) = path.last_mut() { + let Some(item) = scene.items.get(*walked) else { + done.insert(scene.name.as_str()); + path.pop(); + continue; + }; + *walked += 1; + let Some(Nested::Scene(nested)) = self.nested(&item.source) else { + continue; + }; + let Some(nested) = self.scene(nested).filter(|nested| !done.contains(nested.name.as_str())) else { + continue; + }; + if let Some(at) = path.iter().position(|(on, _)| on.name == nested.name) { + let through: Vec = path[at + 1..].iter().map(|(on, _)| format!("`{}`", on.name)).collect(); + let mut err = format!("scene `{}` draws itself", nested.name); + if !through.is_empty() { + err = format!("{err} through {}", through.join(", ")); + } + return Err(Error::Invalid(err)); + } + path.push((nested, 0)); + } + } + + for item in &self.top { + let draws = match self.nested(&item.source) { + Some(Nested::Program) => true, + Some(Nested::Scene(scene)) => self.draws_program(scene), + None => false, + }; + if draws { + return Err(Error::Invalid(format!( + "the top layer places `{}`, which draws the program the top layer is drawn over", + item.source + ))); + } + } + Ok(()) + } + /// The filters `item` draws through: its source's, then its own. pub fn filters<'a>(&'a self, item: &'a Item) -> impl Iterator { let source = self.video.get(&item.source).map(|video| video.filters.as_slice()); @@ -268,10 +384,20 @@ impl Collection { .validate() .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; self.validate_chain(&item.filters, &on)?; - if layer.scene.is_none() && !item.keyframes.is_empty() { - return Err(Error::Invalid(format!( - "{on} has keyframes, but the top layer never goes on to start them" - ))); + if layer.scene.is_none() { + if !item.keyframes.is_empty() { + return Err(Error::Invalid(format!( + "{on} has keyframes, but the top layer never goes on to start them" + ))); + } + if let Some(Nested::Scene(scene)) = self.nested(&item.source) + && let Some(keyed) = self.reached(scene).into_iter().find(|item| !item.keyframes.is_empty()) + { + return Err(Error::Invalid(format!( + "{on} draws `{}` with keyframes, but the top layer never goes on to start them", + keyed.source + ))); + } } if !item.keyframes.is_sorted_by_key(|keyframe| keyframe.at) { return Err(Error::Invalid(format!("{on} has keyframes out of order"))); @@ -293,6 +419,13 @@ impl Collection { } for (name, source) in &self.sources { source.validate(name)?; + if let Source::Scene { scene } = source + && self.scene(scene).is_none() + { + return Err(Error::Invalid(format!( + "`{name}` draws scene `{scene}`, which is not a scene" + ))); + } let Source::Clone { source } = source else { continue }; match self.sources.get(source) { Some(Source::Clone { .. }) => { @@ -329,6 +462,8 @@ impl Collection { self.validate_chain(&video.filters, &on)?; } + self.validate_nesting()?; + if let Some(name) = self.transition.stinger() && !matches!(self.sources.get(name), Some(Source::Media { .. })) { @@ -730,6 +865,15 @@ impl fmt::Display for Layer { } } +/// What a [`Source::Scene`] or [`Source::Program`] draws. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Nested<'a> { + /// The scene by this name. + Scene(&'a str), + /// The main canvas's program. + Program, +} + /// A source placed in a scene or the top layer. /// /// Every item naming the same source shows the same capture, with the @@ -757,7 +901,8 @@ pub struct Item { /// How the item moves away from and back to where it rests, from when its /// scene goes on, in order of time. It plays once: before the first /// keyframe the item holds that one, and after the last it holds the last. - /// A top-layer item has none, since the top layer never goes on. + /// A top-layer item has none, nor does a scene it draws, since the top + /// layer never goes on. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub keyframes: Vec, } @@ -1524,6 +1669,18 @@ pub enum Source { /// source, on demand through [`Feed::replay`](crate::source::Feed::replay). /// Transparent while nothing plays. Replay, + /// Another scene, drawn whole at the main canvas's size, as a picture. Its + /// items' keyframes play from when the scene drawing it goes on, as that + /// scene's own items' do, so the top layer cannot draw one with keyframes. + /// A scene cannot draw itself, even through other scenes. + Scene { + /// The scene drawn. + scene: String, + }, + /// What the main canvas's program shows, transitions included, on the + /// same frame. A scene drawing it cannot go on program, and the top layer + /// cannot draw it, since either would draw the program into itself. + Program, } impl Source { @@ -1549,9 +1706,9 @@ impl Source { } /// Whether the source opens a capture, file, or subscription of its own, - /// rather than showing another source's. + /// rather than showing another source's or a scene. pub fn opens(&self) -> bool { - !matches!(self, Self::Clone { .. }) + !matches!(self, Self::Clone { .. } | Self::Scene { .. } | Self::Program) } /// Whether the source's picture can have its background removed: a @@ -2528,6 +2685,149 @@ background = "remove" assert!(matches!(collection.validate(), Err(Error::Invalid(_)))); } + /// `example()` with scenes `A`, `B`, and `C`, where `A` shows `B` through + /// the scene source `b`, and `on air` a program source placed nowhere. + fn nesting() -> Collection { + let mut collection = example(); + let fill = Rect::fill(&collection.canvas); + for (name, source) in [("a", "A"), ("b", "B"), ("c", "C")] { + let scene = Source::Scene { scene: source.into() }; + collection.sources.insert(name.into(), scene); + collection.scenes.push(Scene::new(source)); + } + collection.sources.insert("on air".into(), Source::Program); + collection.scenes[1].items.push(Item::new("b", fill)); + collection.validate().unwrap(); + collection + } + + /// The scene and program sources read as written. + #[test] + fn reads_a_scene_source() { + let text = "version = 1\n\n\ + [canvas]\nwidth = 1920\nheight = 1080\nframerate = 30\n\n\ + [sources.wide]\nkind = \"scene\"\nscene = \"Wide\"\n\n\ + [sources.\"on air\"]\nkind = \"program\"\n\n\ + [[scenes]]\nname = \"Wide\"\n"; + let collection = Collection::from_toml(text).unwrap(); + assert_eq!(collection.sources["wide"], Source::Scene { scene: "Wide".into() }); + assert_eq!(collection.sources["on air"], Source::Program); + assert_eq!( + Collection::from_toml(&collection.to_toml().unwrap()).unwrap(), + collection + ); + assert_eq!(collection.nested("wide"), Some(Nested::Scene("Wide"))); + } + + /// A scene source names a scene, and no scene draws itself, even through + /// others or a clone, shown or hidden. The top layer, drawn over the + /// program, cannot show it, nor a scene with keyframes, which it never + /// starts, and a mask cannot read a scene. + #[test] + fn refuses_a_scene_drawing_itself() { + let fill = Rect::fill(&Canvas::default()); + let refused = |edit: &dyn Fn(&mut Collection), wanted: &str| { + let mut collection = nesting(); + edit(&mut collection); + let err = collection.validate().unwrap_err().to_string(); + assert!(err.contains(wanted), "{err}"); + }; + refused( + &|collection| { + collection + .sources + .insert("gone".into(), Source::Scene { scene: "Gone".into() }); + }, + "`gone` draws scene `Gone`, which is not a scene", + ); + refused( + &|collection| { + collection.scenes[1].items.push(Item { + visible: false, + ..Item::new("a", fill) + }); + }, + "scene `A` draws itself", + ); + refused( + &|collection| { + collection + .sources + .insert("a copy".into(), Source::Clone { source: "a".into() }); + collection.scenes[2].items.push(Item::new("c", fill)); + collection.scenes[3].items.push(Item::new("a copy", fill)); + }, + "scene `A` draws itself through `B`, `C`", + ); + refused( + &|collection| collection.top.push(Item::new("on air", fill)), + "the top layer places `on air`, which draws the program", + ); + refused( + &|collection| { + collection.scenes[2].items.push(Item::new("on air", fill)); + collection.top.push(Item::new("a", fill)); + }, + "the top layer places `a`, which draws the program", + ); + refused( + &|collection| { + let source = collection.scenes[0].items[0].source.clone(); + collection.scenes[2].items.push(Item { + keyframes: vec![Keyframe::new(Duration::ZERO)], + ..Item::new(source, fill) + }); + collection.top.push(Item::new("a", fill)); + }, + "with keyframes, but the top layer never goes on to start them", + ); + refused( + &|collection| { + let mask = Filter::Mask { + source: "b".into(), + mode: MaskMode::Alpha, + }; + collection.scenes[0].items[0].filters.push(mask); + }, + "which is not a source with a picture of its own", + ); + + // Showing a scene is not drawing it: `B` and `C` both show `C`. + let mut collection = nesting(); + collection.scenes[2].items.push(Item::new("c", fill)); + collection.scenes[1].items.push(Item::new("c", fill)); + collection.top.push(Item::new("c", fill)); + collection.validate().unwrap(); + } + + /// What layers draw reaches through the scenes they show, each once, and + /// leaves hidden items out; whether a scene draws the program counts hidden + /// ones too, since showing one is an edit away. + #[test] + fn drawn_reaches_through_scenes() { + let fill = Rect::fill(&Canvas::default()); + let mut collection = nesting(); + collection.scenes[1].items.push(Item::new("b", fill)); + collection.scenes[2].items = vec![Item::new("logo", fill), Item::new("c", fill)]; + collection.scenes[3].items = vec![ + Item::new("background", fill), + Item { + visible: false, + ..Item::new("on air", fill) + }, + ]; + collection.validate().unwrap(); + let drawn: Vec<&str> = collection + .drawn([Layer::scene("A")]) + .into_iter() + .map(|item| item.source.as_str()) + .collect(); + assert_eq!(drawn, ["b", "b", "logo", "c", "background"]); + assert!(collection.draws_program("A")); + assert!(collection.draws_program("C")); + assert!(!collection.draws_program("Scene")); + } + /// A further canvas is named as a path segment, since it names its /// broadcast, is a size the encoder can take, and is the only canvas a /// scene lays items out on. diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 64e1d0a6..db0df021 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -10,7 +10,8 @@ use moq_video::render::{Renderer, wgpu}; use crate::Error; use crate::collection::{ - Background, Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Rect, Settled, Source, Transition, + Background, Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Nested, Rect, Settled, Source, + Transition, }; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; use crate::matte::Matte; @@ -43,6 +44,14 @@ pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; /// them. An item's keyframes start when its scene goes on program, and /// separately when it goes into preview, for the preview, on every canvas /// alike. +/// +/// A scene a [`Source::Scene`] draws is drawn first, at the main canvas's +/// size, into a texture of its own that is transparent where it draws +/// nothing, once per render however many items show it, and only while one +/// that is drawn does. Its keyframes play from when the scene drawing it went +/// on, so where preview and program show it at different moments of them, it +/// is drawn once for each. A [`Source::Program`] shows the main canvas's +/// program texture itself, drawn before anything showing it. pub struct Compositor { device: wgpu::Device, queue: wgpu::Queue, @@ -52,6 +61,10 @@ pub struct Compositor { extras: BTreeMap, layers: HashMap, + /// The textures scenes drawn through scene sources are drawn into, by + /// scene: as many as the last render drew it at different moments, + /// reused by the next. + nested: HashMap>, /// Lays out and paints overlays, started with the first one. overlays: Option, preview_scene: Option, @@ -138,6 +151,7 @@ impl Switch { } /// A canvas-sized texture drawn into, and bound to sample. +#[derive(Clone)] struct Target { texture: wgpu::Texture, picture: Picture, @@ -269,6 +283,7 @@ impl Compositor { queue: queue.clone(), effects, layers: HashMap::new(), + nested: HashMap::new(), overlays: None, preview_scene: None, program_scene: None, @@ -442,12 +457,27 @@ impl Compositor { /// Put the preview on program at once, ending any transition, and /// stopping a stinger's clip in `sources`. - pub fn cut(&mut self, sources: &Sources) { + /// + /// Refused, changing nothing, if the preview scene draws the program in + /// `collection`, which it would draw into itself. + pub fn cut(&mut self, collection: &Collection, sources: &Sources) -> Result<(), Error> { + self.can_take(collection)?; if let Some(switch) = self.switch.take() { switch.stop(sources); } self.program_scene = self.preview_scene.clone(); self.program_start = None; + Ok(()) + } + + /// Refuse to take a preview scene that draws the program to program. + fn can_take(&self, collection: &Collection) -> Result<(), Error> { + match self.preview_scene.as_deref() { + Some(scene) if collection.draws_program(scene) => Err(Error::Invalid(format!( + "`{scene}` draws the program, so it cannot go on program" + ))), + _ => Ok(()), + } } /// Take the preview to program by `transition`, starting at `now`. @@ -459,7 +489,16 @@ impl Compositor { /// as it plays, and ends with the clip's play, leaving the clip holding its /// last picture, or stopping a looping one; a clip that cannot play, or /// fails, cuts at once instead. - pub fn transition(&mut self, transition: Transition, sources: &Sources, now: Instant) { + /// + /// Refused, changing nothing, as [`cut`](Self::cut) is. + pub fn transition( + &mut self, + collection: &Collection, + transition: Transition, + sources: &Sources, + now: Instant, + ) -> Result<(), Error> { + self.can_take(collection)?; if let Some(switch) = self.switch.take() { switch.stop(sources); } @@ -472,6 +511,7 @@ impl Compositor { transition, played: None, }); + Ok(()) } /// How far the transition under way has switched the scenes at `now`, @@ -533,6 +573,34 @@ impl Compositor { self.preview_shown = preview_since; let program_since = now.saturating_duration_since(program_start); + let preview = On(self.preview_scene.as_deref(), preview_since); + let program = On(self.program_scene.as_deref(), program_since); + let switch = self.switch.as_ref().zip(progress).map(|(switch, progress)| { + let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); + (switch, from, progress) + }); + + // Each canvas's scenes and top layer, and the scenes they draw. + let shown: Vec = std::iter::once(preview).chain(on_program(program, switch)).collect(); + let mut roots = Vec::new(); + for canvas in std::iter::once(None).chain(self.extras.keys().cloned().map(Some)) { + for On(scene, since) in &shown { + if let Some(scene) = scene { + let place = Place::scene(*scene).on(canvas.clone()); + roots.push((place, *since)); + } + } + roots.push((Place::top().on(canvas), Duration::ZERO)); + } + let nesting = Nesting::of(collection, roots); + let targets = nested_targets( + &mut self.nested, + &self.device, + &self.effects, + self.main.canvas, + &nesting, + ); + let mut frame = self.effects.frame(); let mut draw = Draw { collection, @@ -542,18 +610,28 @@ impl Compositor { canvas: None, size: self.main.canvas, prepared: HashMap::new(), + nesting: &nesting, + scenes: HashMap::new(), + program: None, }; - let preview = On(self.preview_scene.as_deref(), preview_since); - let program = On(self.program_scene.as_deref(), program_since); - let switch = self.switch.as_ref().zip(progress).map(|(switch, progress)| { - let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); - (switch, from, progress) - }); - let extras = self.extras.iter().map(|(name, output)| (Some(name.as_str()), output)); - for (canvas, output) in std::iter::once((None, &self.main)).chain(extras) { - draw.canvas = canvas; + // What the program shows is drawn first, then the program, then what + // shows it, then everything else. + let nests = nesting.order.iter().zip(&targets).enumerate(); + let (late, early): (Vec<_>, Vec<_>) = nests.partition(|(_, (nest, _))| nest.program); + for (index, (nest, target)) in early { + draw.nest(index, nest, target); + } + draw.draw_program(&self.main, program, switch); + draw.program = Some(self.main.program.picture.clone()); + for (index, (nest, target)) in late { + draw.nest(index, nest, target); + } + draw.draw_preview(&self.main, preview); + for (name, output) in &self.extras { + draw.canvas = Some(name); draw.size = output.canvas; - draw.output(output, preview, program, switch); + draw.draw_preview(output, preview); + draw.draw_program(output, program, switch); } self.effects.submit(frame); @@ -749,8 +827,8 @@ impl Compositor { } /// The sources whose pictures preview and program draw on any canvas: - /// each visible item's, a stinger's clip under way, and each mask's, - /// through clones. + /// each visible item's, through the scenes they draw, a stinger's clip + /// under way, and each mask's, through clones. fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { let mut shown = self.on_air_sources(collection); shown.extend(self.drawn(collection, self.preview_scene.as_deref().map(Some))); @@ -759,7 +837,8 @@ impl Compositor { /// The sources whose pictures program draws on any canvas: each visible /// item's in the scene on program, the scene a transition leaves, and the - /// top layer, a stinger's clip under way, and each mask's, through clones. + /// top layer, through the scenes they draw, a stinger's clip under way, + /// and each mask's, through clones. fn on_air_sources<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { let mut shown = self.drawn(collection, self.on_air().map(Some).chain([None])); // Drawn through its source's filters alone. @@ -776,39 +855,53 @@ impl Compositor { } /// The sources whose pictures `scenes` draw on any canvas, `None` being - /// the top layer: each visible item's and each mask's, through clones. + /// the top layer: each visible item's, through the scenes they draw, and + /// each mask's, through clones. fn drawn<'a, 'b>( &self, collection: &'a Collection, scenes: impl IntoIterator>, ) -> HashSet<&'a str> { - let scenes: Vec> = scenes.into_iter().map(|scene| scene.map(str::to_string)).collect(); + let scenes: Vec> = scenes.into_iter().collect(); + let canvases = std::iter::once(None).chain(self.extras.keys().map(Some)); + let places = canvases.flat_map(|canvas| { + scenes.iter().map(move |scene| Place { + canvas: canvas.cloned(), + scene: scene.map(str::to_string), + }) + }); let mut shown = HashSet::new(); - for canvas in std::iter::once(None).chain(self.extras.keys().map(Some)) { - for scene in &scenes { - let place = Place { - canvas: canvas.cloned(), - scene: scene.clone(), - }; - drawn(collection, &mut shown, &place); - } + for item in collection.drawn(places) { + shown.insert(collection.capture(&item.source)); + masks(collection, &mut shown, collection.filters(item)); } shown } } -/// Add the sources whose pictures `place` draws to `shown`: each visible -/// item's and each mask's, through clones. -fn drawn<'a>(collection: &'a Collection, shown: &mut HashSet<&'a str>, place: &Place) { - for item in collection - .items(place) - .unwrap_or_default() - .iter() - .filter(|item| item.visible) - { - shown.insert(collection.capture(&item.source)); - masks(collection, shown, collection.filters(item)); +/// A texture of `canvas` for each scene `nesting` draws, reusing those in +/// `pool`, the last render's, and leaving there only those used now. +fn nested_targets( + pool: &mut HashMap>, + device: &wgpu::Device, + effects: &Effects, + canvas: Canvas, + nesting: &Nesting, +) -> Vec { + let mut free = std::mem::take(pool); + let mut targets = Vec::with_capacity(nesting.order.len()); + for nest in &nesting.order { + let reused = free.get_mut(nest.scene).and_then(Vec::pop); + let target = match reused { + Some(target) if (target.texture.width(), target.texture.height()) == (canvas.width, canvas.height) => { + target + } + _ => Target::new(device, effects, canvas, "moq-scene nested"), + }; + pool.entry(nest.scene.to_string()).or_default().push(target.clone()); + targets.push(target); } + targets } /// Add the sources the masks among `filters` read to `shown`, through clones. @@ -996,6 +1089,159 @@ fn fits(device: &wgpu::Device, canvas: Canvas) -> Result<(), Error> { #[derive(Clone, Copy)] struct On<'a>(Option<&'a str>, Duration); +/// The scenes program draws: `program`, or those `switch` draws as it goes +/// to it. +fn on_program<'a>(program: On<'a>, switch: Option<(&Switch, On<'a>, f32)>) -> Vec> { + match switch { + // Under its clip, the program switches scenes at once. + Some((switch, from, progress)) if switch.transition.stinger().is_some() => { + vec![if progress < 1.0 { from } else { program }] + } + Some((_, from, _)) => vec![from, program], + None => vec![program], + } +} + +/// The scenes a render draws through scene sources, each at the moment of its +/// keyframes it is shown at. +struct Nesting<'a> { + /// In an order that draws each after the scenes it shows. + order: Vec>, + /// Which of `order` draws a scene shown in a scene that went on a while + /// ago, by the scene's name and that while. + drawn: HashMap<(&'a str, Duration), usize>, +} + +/// A scene drawn through a scene source. +struct Nest<'a> { + scene: &'a str, + /// How long ago the scene drawing it went on, which its keyframes play + /// from. + since: Duration, + /// `since`, or `None` once every item it draws, through the scenes it + /// draws, holds its last keyframe, which every later moment draws alike. + moment: Option, + /// Whether it shows the program, itself or through the scenes it draws. + program: bool, +} + +impl<'a> Nesting<'a> { + /// The scenes the items of each of `roots` draw, from when its scene went + /// on, and those they draw in turn. + fn of(collection: &'a Collection, roots: impl IntoIterator) -> Self { + let mut nesting = Self { + order: Vec::new(), + drawn: HashMap::new(), + }; + let mut walked = HashMap::new(); + for (place, since) in roots { + let items = collection.items(&place).unwrap_or_default(); + nesting.walk(collection, items, since, &mut walked); + } + nesting + } + + /// Add the scenes `items` show, from when the scene they are in went on + /// `since` ago. `walked` holds each scene and moment added, or being + /// walked. + fn walk( + &mut self, + collection: &'a Collection, + items: &'a [Item], + since: Duration, + walked: &mut HashMap<(&'a str, Option), Option>, + ) { + /// A run of items being walked, with the scene and moment they draw + /// unless they are the root's. + struct Walk<'a> { + nest: Option<(&'a str, Option)>, + items: &'a [Item], + next: usize, + /// Whether any item walked shows the program. + program: bool, + } + + // A stack of its own rather than the call stack, which a long enough + // chain of scenes would overflow. + let mut stack = vec![Walk { + nest: None, + items, + next: 0, + program: false, + }]; + while let Some(walk) = stack.last_mut() { + let items = walk.items; + let Some(item) = items.get(walk.next) else { + let Some(Walk { + nest: Some((scene, moment)), + program, + .. + }) = stack.pop() + else { + break; + }; + self.order.push(Nest { + scene, + since, + moment, + program, + }); + let index = self.order.len() - 1; + walked.insert((scene, moment), Some(index)); + self.drawn.insert((scene, since), index); + stack.last_mut().expect("a nest is walked from items").program |= program; + continue; + }; + walk.next += 1; + if !item.visible { + continue; + } + let scene = match collection.nested(&item.source) { + Some(Nested::Scene(scene)) => scene, + Some(Nested::Program) => { + walk.program = true; + continue; + } + None => continue, + }; + if let Some(&index) = self.drawn.get(&(scene, since)) { + walk.program |= self.order[index].program; + continue; + } + let moment = moment(collection, scene, since); + match walked.get(&(scene, moment)) { + Some(Some(index)) => { + let index = *index; + self.drawn.insert((scene, since), index); + walk.program |= self.order[index].program; + } + // A scene drawing itself, which validation refuses: it shows + // nothing there rather than walking forever. + Some(None) => {} + None => { + walked.insert((scene, moment), None); + let items = collection.scene(scene).map_or(&[][..], |scene| scene.items.as_slice()); + stack.push(Walk { + nest: Some((scene, moment)), + items, + next: 0, + program: false, + }); + } + } + } + } +} + +/// The moment of its keyframes the scene called `scene` is drawn at, `since` +/// the scene drawing it went on, or `None` once every item it draws, through +/// the scenes it draws, holds its last keyframe. +fn moment(collection: &Collection, scene: &str, since: Duration) -> Option { + let items = collection.drawn([Place::scene(scene)]); + let moving = items.iter().any(|item| item.settled(since) != Settled::After); + moving.then_some(since) +} + /// One render's draws onto its targets. struct Draw<'a> { collection: &'a Collection, @@ -1010,20 +1256,37 @@ struct Draw<'a> { /// where it is, on which canvas, and its [`Item::settled`] time, as two /// outputs showing its scene can be at different points of its keyframes. prepared: HashMap<(Place, usize, Settled), Option>, + /// The scenes drawn through scene sources this render. + nesting: &'a Nesting<'a>, + /// The picture of each of `nesting`'s scenes drawn so far, by its place + /// in the order. + scenes: HashMap, + /// The main canvas's program, once drawn. + program: Option, } impl Draw<'_> { - /// Draw `output`'s preview, showing `preview`, and its program, showing - /// `program`, or `switch` to it from the scene it leaves, `progress` of - /// the way there. - fn output(&mut self, output: &Output, preview: On, program: On, switch: Option<(&Switch, On, f32)>) { - let top = self.place(None); + /// Draw the scene `nest` names, the `index`th of the nesting, into + /// `target`, transparent where it draws nothing, for items to show. + fn nest(&mut self, index: usize, nest: &Nest, target: &Target) { + self.frame.target(&target.picture, self.size, wgpu::Color::TRANSPARENT); + self.layer(target, Place::scene(nest.scene), nest.since); + self.scenes.insert(index, target.picture.clone()); + } + + /// Draw `output`'s preview, showing `preview` under the top layer. + fn draw_preview(&mut self, output: &Output, preview: On) { // The top layer goes over both outputs, and over a transition rather // than through it, so it stays on across the switch. self.target(&output.preview); self.scene(&output.preview, preview); - self.layer(&output.preview, top.clone(), Duration::ZERO); + self.layer(&output.preview, self.place(None), Duration::ZERO); + } + /// Draw `output`'s program, showing `program`, or `switch` to it from the + /// scene it leaves, `progress` of the way there, under the top layer. + fn draw_program(&mut self, output: &Output, program: On, switch: Option<(&Switch, On, f32)>) { + let top = self.place(None); let to = program; match switch { Some((switch, from, progress)) => { @@ -1082,9 +1345,9 @@ impl Draw<'_> { } } - /// Start drawing `target`. + /// Start drawing `target`, a canvas, opaque black where nothing is drawn. fn target(&mut self, target: &Target) { - self.frame.target(&target.picture, self.size); + self.frame.target(&target.picture, self.size, wgpu::Color::BLACK); } /// Draw `on`'s items onto `target`. A missing scene draws nothing. @@ -1115,7 +1378,27 @@ impl Draw<'_> { fn layer(&mut self, target: &Target, place: Place, since: Duration) { let items = self.collection.items(&place).unwrap_or_default(); for (index, item) in items.iter().enumerate().filter(|(_, item)| item.visible) { - self.item(target, (place.clone(), index, item.settled(since)), &item.posed(since)); + let key = (place.clone(), index, self.settled(item, since)); + self.item(target, key, &item.posed(since), since); + } + } + + /// Where `since` falls among `item`'s keyframes, as [`Item::settled`], but + /// at that moment itself while a scene it shows moves, so a draw of it is + /// shared only with draws showing the scene alike. + fn settled(&self, item: &Item, since: Duration) -> Settled { + let moving = match self.collection.nested(&item.source) { + Some(Nested::Scene(scene)) => self + .nesting + .drawn + .get(&(scene, since)) + .is_some_and(|&index| self.nesting.order[index].moment.is_some()), + _ => false, + }; + if moving { + Settled::At(since) + } else { + item.settled(since) } } @@ -1133,16 +1416,16 @@ impl Draw<'_> { ..Item::new(source, Rect::fill(&canvas)) }; let pose = Pose::of(self.collection, &item); - self.posed(target, &item, pose); + self.posed(target, &item, pose, Duration::ZERO); } - /// Draw `item`, posed, where it is placed onto `target`, shared with every - /// other draw of it under `key`. - fn item(&mut self, target: &Target, key: (Place, usize, Settled), item: &Item) { + /// Draw `item`, posed `since` its scene went on, where it is placed onto + /// `target`, shared with every other draw of it under `key`. + fn item(&mut self, target: &Target, key: (Place, usize, Settled), item: &Item, since: Duration) { let prepared = match self.prepared.get(&key) { Some(prepared) => prepared.clone(), None => { - let prepared = self.prepare(item, Pose::of(self.collection, item)); + let prepared = self.prepare(item, Pose::of(self.collection, item), since); self.prepared.insert(key, prepared.clone()); prepared } @@ -1152,12 +1435,13 @@ impl Draw<'_> { } } - /// Draw `item` onto `target` in `pose` rather than where it is placed. - fn posed(&mut self, target: &Target, item: &Item, pose: Pose) { + /// Draw `item` onto `target` in `pose` rather than where it is placed, + /// `since` its scene went on. + fn posed(&mut self, target: &Target, item: &Item, pose: Pose, since: Duration) { if pose.opacity <= 0.0 { return; } - if let Some(prepared) = self.prepare(item, pose) { + if let Some(prepared) = self.prepare(item, pose, since) { self.effects.draw(self.frame, &target.texture, &prepared); } } @@ -1210,7 +1494,7 @@ impl Draw<'_> { (Some(at), None) => { let pose = Pose::of(collection, &leaving[at]); let opacity = 1.0 - progress; - self.posed(target, &leaving[at], Pose { opacity, ..pose }); + self.posed(target, &leaving[at], Pose { opacity, ..pose }, from.1); } (None, Some(index)) => { let pose = Pose::of(collection, &entering[index]); @@ -1221,26 +1505,37 @@ impl Draw<'_> { opacity: progress, ..pose }, + to.1, ); } (Some(at), Some(index)) => { let (old, new) = (&*leaving[at], &*entering[index]); let own = Pose::of(collection, new); let pose = Pose::of(collection, old).toward(&own, progress); + // A nested scene's picture cannot be mixed between the + // moments the two scenes are at, so it changes as the + // stacking does. + let since = match collection.nested(&new.source) { + Some(Nested::Scene(_)) if progress < 0.5 => from.1, + _ => to.1, + }; if old.filters == new.filters && old.blend == new.blend { - if pose == own { + // The preview never draws the incoming scene's item + // with the outgoing scene's picture. + if pose == own && since == to.1 { // Shared with the preview when it draws the same. - let key = (place.clone(), index, resting[index].settled(to.1)); - self.item(target, key, new); + let key = (place.clone(), index, self.settled(&resting[index], since)); + self.item(target, key, new, since); } else { - self.posed(target, new, pose); + self.posed(target, new, pose, since); } } else if let Some(filters) = mixed(old, new, progress) { let item = Item { filters, ..new.clone() }; - self.posed(target, &item, pose); + self.posed(target, &item, pose, since); } else { // A look that cannot be mixed changes as the stacking does. - self.posed(target, if progress < 0.5 { old } else { new }, pose); + let item = if progress < 0.5 { old } else { new }; + self.posed(target, item, pose, since); } } (None, None) => unreachable!("a slot holds an item"), @@ -1248,9 +1543,11 @@ impl Draw<'_> { } } - fn prepare(&mut self, item: &Item, pose: Pose) -> Option { + /// `item` in `pose`, `since` its scene went on, ready to draw: `None` + /// while it shows nothing yet. + fn prepare(&mut self, item: &Item, pose: Pose, since: Duration) -> Option { let collection = self.collection; - let picture = self.library.0.get(collection.capture(&item.source))?.picture.as_ref()?; + let picture = &self.shows(item, since)?; let fade = Filter::Opacity { amount: pose.opacity }; let mut filters: Vec<&Filter> = collection.filters(item).collect(); if pose.opacity < 1.0 { @@ -1267,6 +1564,25 @@ impl Draw<'_> { } } +impl Draw<'_> { + /// The picture `item` shows, `since` its scene went on: its source's + /// newest, the scene it draws as drawn this render, or the main canvas's + /// program once drawn. + fn shows(&self, item: &Item, since: Duration) -> Option { + match self.collection.nested(&item.source) { + Some(Nested::Scene(scene)) => { + let index = self.nesting.drawn.get(&(scene, since))?; + self.scenes.get(index).cloned() + } + Some(Nested::Program) => self.program.clone(), + None => { + let layer = self.library.0.get(self.collection.capture(&item.source))?; + layer.picture.clone() + } + } + } +} + /// Premultiply tightly packed RGBA in place. fn premultiply(pixels: &mut [u8]) { for pixel in pixels.as_chunks_mut::<4>().0 { @@ -1523,7 +1839,7 @@ mod tests { local(async { let mut studio = Studio::new(collection()).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let program = studio.program(); assert_eq!(pixel(&program, 1, 1), RED); @@ -1553,7 +1869,7 @@ mod tests { collection.scenes[0].items.truncate(1); let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let actual = pixel(&studio.program(), 1, 1); assert!(near(actual, [128, 0, 0, 255]), "{actual:?}"); @@ -1570,9 +1886,12 @@ mod tests { let mut studio = Studio::new(collection()).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); assert_eq!(studio.compositor.program_scene(), Some("blue")); studio.render(start + Duration::from_millis(500)); @@ -1623,7 +1942,7 @@ mod tests { let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let program = studio.program(); let actual = pixel(&program, 1, 6); @@ -1644,12 +1963,15 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 1, 6), BLUE); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); let program = studio.program(); let actual = pixel(&program, 1, 1); @@ -1697,14 +2019,17 @@ mod tests { let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 1, 1), RED); assert_eq!(tall(&mut studio), (BLUE, RED)); assert_eq!(studio.compositor.layers.len(), 2, "each source is uploaded once"); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); let (fading, kept) = tall(&mut studio); assert!(near(fading, [0, 0, 128, 255]), "{fading:?}"); @@ -1734,10 +2059,13 @@ mod tests { assert_eq!(pixel(&studio.program(), 1, 1), BLACK); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); assert!(!studio.compositor.transitioning()); studio.render(start + Duration::from_millis(600)); assert_eq!(pixel(&studio.program(), 1, 1), RED); @@ -1796,9 +2124,12 @@ mod tests { ] { let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(transition.clone(), &studio.sources, start); + studio + .compositor + .transition(&studio.collection, transition.clone(), &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); let program = studio.program(); let actual = [(1, 1), (1, 6), (6, 1), (6, 6)].map(|(x, y)| pixel(&program, x, y)); @@ -1844,12 +2175,18 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); studio .compositor - .transition(Transition::Move { duration }, &studio.sources, start); + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + start, + ) + .unwrap(); // Eased halfway, blue is at (2, 2) and both fades are at half. studio.render(start + Duration::from_millis(500)); @@ -1929,7 +2266,7 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); studio.render(at(0)); @@ -1937,7 +2274,7 @@ mod tests { assert_eq!(tall(&mut studio, true), (BLUE, BLACK), "halfway in, in preview"); assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the main canvas too"); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(at(500)); assert_eq!(tall(&mut studio, false), (BLACK, BLACK), "on program from the start"); assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "the main canvas too"); @@ -1963,7 +2300,7 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); studio.render(at(0)); @@ -1978,7 +2315,7 @@ mod tests { assert_eq!(pixel(&preview, 1, 6), BLUE, "halfway in"); assert_eq!(pixel(&preview, 6, 6), BLACK); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(at(500)); assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "on program from the start"); assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the preview carries on"); @@ -2017,7 +2354,7 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); // Program has reached the jump; preview starts over before it. studio.compositor.set_preview(Some("blue".into())); @@ -2041,11 +2378,14 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("red".into())); studio.render(at(0)); - studio.compositor.transition(FADE, &studio.sources, at(9000)); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, at(9000)) + .unwrap(); studio.render(at(9500)); let program = studio.program(); // Red has slid halfway in over the left half, blue is in place, and @@ -2076,12 +2416,18 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); studio .compositor - .transition(Transition::Move { duration }, &studio.sources, start); + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + start, + ) + .unwrap(); studio.render(start); let program = studio.program(); @@ -2108,7 +2454,7 @@ mod tests { local(async { let mut studio = Studio::new(collection()).await; studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); assert!(studio.sources.get("blue").unwrap().take().is_none(), "drawn once"); studio.render(Instant::now()); @@ -2153,7 +2499,7 @@ mod tests { collection.validate().unwrap(); let mut studio = Studio::with_files(collection, dir).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); studio } @@ -2182,7 +2528,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); assert_eq!(studio.compositor.program_scene(), Some("blue")); // What the clip's filters read is drawn too, such as an overlay to paint. @@ -2250,7 +2597,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(studio.compositor.transitioning()); assert_eq!(pixel(&studio.program(), 1, 1), BLUE); @@ -2268,7 +2616,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(!studio.compositor.transitioning()); assert_eq!(pixel(&studio.program(), 1, 1), BLUE); @@ -2277,7 +2626,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(studio.compositor.transitioning()); studio.sources.get("sting").unwrap().play().unwrap(); @@ -2303,7 +2653,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); loop { studio.render(Instant::now()); let until = at + Duration::from_millis(10); @@ -2311,7 +2662,7 @@ mod tests { break; } } - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); assert!(!studio.compositor.transitioning()); let feed = studio.sources.get("sting").unwrap(); while feed.status() != Status::Ended { @@ -2326,7 +2677,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(studio.compositor.transitioning()); studio.sources.get("sting").unwrap().stop().unwrap(); @@ -2347,7 +2699,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); while studio.compositor.transitioning() { studio.render(Instant::now()); let until = at + Duration::from_millis(10); @@ -2425,7 +2778,8 @@ mod tests { } /// A source with its background removed draws only the person, each - /// picture once the model has it, and its whole picture again once kept. + /// picture once the model has it, and its whole picture again once kept, + /// in its scene or a scene that scene shows. #[tokio::test] #[ignore = "needs a Vulkan device; run by just rs render-sw"] async fn removes_the_background() { @@ -2452,7 +2806,7 @@ mod tests { let mut studio = Studio::with_files(collection, dir).await; studio.compositor.set_preview(Some(scene)); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); let at = |rgba: &[u8], x: usize, y: usize| -> [u8; 4] { let at = (y * 256 + x) * 4; rgba[at..at + 4].try_into().unwrap() @@ -2484,6 +2838,21 @@ mod tests { studio.collection.video.insert("person".into(), removed); cut(&mut studio); + + // Shown through a nested scene, the cut-out in hand draws at once. + let items = studio.collection.scenes[0].items.clone(); + studio.collection.scenes.push(Scene { + items, + ..Scene::new("person") + }); + let nested = Source::Scene { scene: "person".into() }; + studio.collection.sources.insert("person scene".into(), nested); + studio.collection.scenes[0].items = vec![Item::new("person scene", Rect::fill(&studio.collection.canvas))]; + studio.collection.validate().unwrap(); + studio.render(Instant::now()); + let program = studio.program(); + assert_ne!(at(&program, face.0, face.1), BLACK, "through the nested scene"); + assert_eq!(at(&program, flag.0, flag.1), BLACK); }) .await; } @@ -2546,7 +2915,7 @@ mod tests { async fn drawn(collection: Collection, dir: tempfile::TempDir) -> Vec { let mut studio = Studio::with_files(collection, dir).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); studio.program() } @@ -3005,7 +3374,7 @@ mod tests { ); let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let old = studio.compositor.program().clone(); @@ -3077,9 +3446,12 @@ mod tests { let mut studio = Studio::new(collection()).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); // `blue` takes the name `red` gives up, which is followed once. let mut draft = studio.collection.draft("Rename"); @@ -3199,12 +3571,12 @@ mod tests { collection.scenes[1].items = vec![Item::new("overlay", Rect::fill(&collection.canvas))]; let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); // Opened here, but drawn nowhere until the cut. let start = Instant::now(); studio.render(start); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); for (at, expected) in [(500, [128, 0, 128, 255]), (750, [64, 0, 191, 255]), (2000, BLUE)] { studio.render(start + Duration::from_millis(at)); @@ -3256,7 +3628,7 @@ mod tests { // Both open here, drawn nowhere. let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); let previewed = start + Duration::from_secs(2); @@ -3272,7 +3644,7 @@ mod tests { assert_eq!(pixel(&preview, 3, 6), BLUE, "the clock ran on"); let cut = previewed + Duration::from_secs(3); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(cut); assert_eq!(pixel(&studio.program(), 3, 1), RED, "on program, from the start"); studio.render(cut + Duration::from_millis(500)); @@ -3296,7 +3668,7 @@ mod tests { // Nor does it restart one the top layer shows on program. studio.collection.top = vec![Item::new("third", fill)]; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(cut + Duration::from_secs(1)); studio.compositor.set_preview(Some("blue".into())); studio.render(cut + Duration::from_millis(1100)); @@ -3305,6 +3677,68 @@ mod tests { .await; } + /// An overlay set to restart in a nested scene restarts when the scene + /// showing it goes on program, and a preview showing it directly never + /// restarts it while program shows it through that scene. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn an_overlay_in_a_nested_scene_restarts_going_on() { + local(async { + let mut collection = collection(); + let overlay = Source::Overlay { + html: "
" + .into(), + width: 8, + height: 8, + restart: true, + }; + collection.sources.insert("third".into(), overlay); + let fill = Rect::fill(&collection.canvas); + collection.scenes.push(Scene { + items: vec![Item::new("third", fill)], + ..Scene::new("lower") + }); + let lower = Source::Scene { scene: "lower".into() }; + collection.sources.insert("lower scene".into(), lower); + collection.scenes[1].items = vec![Item::new("lower scene", fill)]; + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + let near = + |actual: [u8; 4], expected: [u8; 4]| actual.iter().zip(expected).all(|(a, e)| a.abs_diff(e) <= 2); + + // Opened here, drawn nowhere. + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(start); + + let cut = start + Duration::from_secs(2); + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(cut); + assert_eq!(pixel(&studio.program(), 3, 1), RED, "on program, from the start"); + studio.render(cut + Duration::from_millis(500)); + let program = studio.program(); + assert!( + near(pixel(&program, 3, 1), [128, 0, 128, 255]), + "{:?}", + pixel(&program, 3, 1) + ); + + studio.compositor.set_preview(Some("lower".into())); + studio.render(cut + Duration::from_millis(750)); + for (output, picture) in [("program", studio.program()), ("preview", studio.preview())] { + let actual = pixel(&picture, 3, 1); + assert!(near(actual, [64, 0, 191, 255]), "{output} ran on: {actual:?}"); + } + }) + .await; + } + /// A preview never restarts an overlay a stinger under way masks its clip /// with on program, though no scene on air shows it. #[tokio::test] @@ -3334,7 +3768,10 @@ mod tests { let now = Instant::now(); let transition = studio.collection.transition.clone(); - studio.compositor.transition(transition, &studio.sources, now); + studio + .compositor + .transition(&studio.collection, transition, &studio.sources, now) + .unwrap(); studio.render(now); let opened = studio.compositor.layers["third"].overlay.as_ref().unwrap().start(); @@ -3374,12 +3811,18 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); studio .compositor - .transition(Transition::Move { duration }, &studio.sources, start); + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + start, + ) + .unwrap(); studio.render(start + Duration::from_millis(500)); let program = studio.program(); @@ -3416,11 +3859,11 @@ mod tests { // Paint the aside overlay, then leave it off air. studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 3, 3), [255, 255, 0, 255]); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); studio.compositor.overlays.as_ref().unwrap().panic(); @@ -3431,12 +3874,310 @@ mod tests { } studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 3, 3), RED, "no frozen overlay"); }) .await; } + + /// `collection()` with a scene source drawing each of its scenes, named + /// after it with ` scene`, and a program source, `on air`, and `scene` + /// added showing `items`. + fn nesting(items: Vec) -> Collection { + let mut collection = collection(); + for scene in ["red", "blue"] { + let source = Source::Scene { scene: scene.into() }; + collection.sources.insert(format!("{scene} scene"), source); + } + collection.sources.insert("on air".into(), Source::Program); + collection.scenes.push(Scene { + items, + ..Scene::new("scene") + }); + collection.validate().unwrap(); + collection + } + + /// A scene source draws its scene's items, scaled and cropped like any + /// picture, and transparent where the scene draws nothing. It is drawn once + /// however many items show it, and not at all once nothing drawn does. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_scene_source_draws_its_scene() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![ + Item::new("blue", fill), + // The red scene, red with blue over its top-right quarter. + Item::new("red scene", CORNER), + Item { + visible: false, + ..Item::new("red scene", fill) + }, + // Again, over the top-left quarter. + Item::new("red scene", Rect { y: 0.0, ..CORNER }), + ]); + // Blue over the top-right quarter alone, transparent elsewhere. + collection.scenes.push(Scene { + items: vec![collection.scenes[0].items[1].clone()], + ..Scene::new("corner") + }); + let corner = Source::Scene { scene: "corner".into() }; + collection.sources.insert("corner scene".into(), corner); + collection.scenes[1].items = vec![Item::new("red", fill), Item::new("corner scene", fill)]; + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let program = studio.program(); + assert_eq!(pixel(&program, 0, 7), RED, "the red scene's red"); + assert_eq!(pixel(&program, 3, 4), BLUE, "the red scene's corner"); + assert_eq!(pixel(&program, 6, 6), BLUE, "under it"); + assert_eq!(pixel(&program, 0, 0), RED, "shown twice"); + assert_eq!(studio.compositor.nested.get("red").map(Vec::len), Some(1)); + + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let program = studio.program(); + assert_eq!(pixel(&program, 1, 1), RED, "through the nested scene"); + assert_eq!(pixel(&program, 6, 1), BLUE); + let drawn: Vec<&String> = studio.compositor.nested.keys().collect(); + assert_eq!(drawn, ["corner"], "only what is drawn"); + }) + .await; + } + + /// A program source shows what the program draws on the same frame, + /// a transition under way included, and a scene showing it cannot go on + /// program. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_program_source_shows_the_program_on_the_same_frame() { + local(async { + let mut studio = Studio::new(nesting(vec![Item::new("on air", CORNER)])).await; + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.compositor.set_preview(Some("blue".into())); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); + studio.compositor.set_preview(Some("scene".into())); + + studio.render(start + Duration::from_millis(500)); + let faded = pixel(&studio.program(), 1, 1); + assert!(near(faded, [128, 0, 127, 255]), "{faded:?}"); + let preview = studio.preview(); + assert_eq!(pixel(&preview, 1, 5), faded, "the program as it fades"); + assert_eq!(pixel(&preview, 3, 4), BLUE, "blue under blue"); + assert_eq!(pixel(&preview, 6, 1), BLACK); + + let err = studio.compositor.cut(&studio.collection, &studio.sources).unwrap_err(); + assert!(err.to_string().contains("draws the program"), "{err}"); + let err = studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap_err(); + assert!(err.to_string().contains("draws the program"), "{err}"); + assert_eq!(studio.compositor.program_scene(), Some("blue")); + assert!(studio.compositor.transitioning(), "the fade carries on"); + }) + .await; + } + + /// A scene source showing a scene that shows the program, even through + /// another, is drawn after the program, on the same frame. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_scene_showing_the_program_is_drawn_after_it() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(Vec::new()); + collection.scenes[2].items = vec![Item::new("outer scene", fill)]; + for (name, shows) in [("inner", "on air"), ("outer", "inner scene")] { + collection.scenes.push(Scene { + items: vec![Item::new(shows, fill)], + ..Scene::new(name) + }); + let source = Source::Scene { scene: name.into() }; + collection.sources.insert(format!("{name} scene"), source); + } + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.compositor.set_preview(Some("scene".into())); + studio.render(Instant::now()); + assert_eq!(pixel(&studio.preview(), 1, 1), BLUE); + }) + .await; + } + + /// An item showing a scene source in both scenes of a move shows the + /// nested scene at the moment the scene it leaves is at until halfway, + /// then at the moment the scene it lands on is at, as the stacking + /// changes. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_move_shows_a_nested_scene_as_the_scene_it_leaves_until_halfway() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("blue scene", fill)]); + collection.scenes[1].items[0].keyframes = slide_in(); + collection.scenes.push(Scene { + items: vec![Item::new("blue scene", fill)], + ..Scene::new("again") + }); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(at(0)); + studio.render(at(2000)); + assert_eq!(pixel(&studio.program(), 7, 1), BLUE, "all the way in"); + + studio.compositor.set_preview(Some("again".into())); + let duration = Duration::from_secs(1); + studio + .compositor + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + at(2000), + ) + .unwrap(); + studio.render(at(2250)); + assert_eq!(pixel(&studio.program(), 7, 1), BLUE, "as it was"); + studio.render(at(2750)); + let program = studio.program(); + assert_eq!( + (pixel(&program, 5, 1), pixel(&program, 6, 1)), + (BLUE, BLACK), + "sliding in again" + ); + }) + .await; + } + + /// A non-normal blend over a backdrop that is partly transparent, in a + /// nested scene, mixes with the backdrop as much as the backdrop covers, + /// as CSS does. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_blend_mode_over_a_partly_transparent_backdrop_mixes_partly() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("blue scene", fill)]); + let half = Item { + filters: vec![Filter::Opacity { amount: 0.5 }], + ..Item::new("red", fill) + }; + collection.scenes[1].items[0].blend = Blend::Multiply; + collection.scenes[1].items.insert(0, half); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + // Half blue, half blue multiplied by red, which is black. + let actual = pixel(&studio.program(), 1, 1); + assert!(near(actual, [0, 0, 128, 255]), "{actual:?}"); + }) + .await; + } + + /// A nested scene's keyframes play from when the scene showing it went + /// on, so preview and program can show it at two moments, each drawn once, + /// and once both settle, drawn once for both. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_nested_scene_plays_its_keyframes_from_when_its_scene_goes_on() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("blue scene", fill)]); + collection.scenes[1].items[0].keyframes = slide_in(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(at(0)); + assert_eq!(pixel(&studio.program(), 1, 1), BLACK, "still off the canvas"); + studio.render(at(500)); + let program = studio.program(); + assert_eq!( + (pixel(&program, 3, 1), pixel(&program, 4, 1)), + (BLUE, BLACK), + "halfway in" + ); + + // Into preview again, so its keyframes start over there. + studio.compositor.set_preview(Some("scene".into())); + studio.render(at(500)); + studio.render(at(750)); + let program = studio.program(); + assert_eq!((pixel(&program, 5, 1), pixel(&program, 6, 1)), (BLUE, BLACK)); + let preview = studio.preview(); + assert_eq!((pixel(&preview, 1, 1), pixel(&preview, 2, 1)), (BLUE, BLACK)); + assert_eq!(studio.compositor.nested["blue"].len(), 2, "one per moment"); + + studio.render(at(2000)); + assert_eq!(pixel(&studio.program(), 7, 1), BLUE); + assert_eq!(pixel(&studio.preview(), 7, 1), BLUE); + assert_eq!(studio.compositor.nested["blue"].len(), 1, "settled alike"); + }) + .await; + } + + /// A filter reading around each pixel, such as a blur, reads the nested + /// scene drawn this render, not the last one. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_blur_reads_the_nested_scene_drawn_now() { + local(async { + let fill = Rect::fill(&collection().canvas); + let blurred = Item { + filters: vec![Filter::Blur { radius: 0.5 }], + ..Item::new("blue scene", fill) + }; + let mut studio = Studio::new(nesting(vec![blurred])).await; + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let actual = pixel(&studio.program(), 4, 4); + assert!(near(actual, BLUE), "{actual:?}"); + }) + .await; + } + + /// A blend mode in a nested scene mixes with what the scene draws under it, + /// and where it draws nothing, leaves the picture as it is, as CSS does. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_blend_mode_over_nothing_in_a_nested_scene_draws_the_picture() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("red", fill), Item::new("blue scene", fill)]); + collection.scenes[1].items[0].blend = Blend::Multiply; + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + }) + .await; + } } /// What drawing more canvases costs, against how many sources the collection @@ -3494,11 +4235,20 @@ mod bench { /// The mean time to draw and finish every canvas once. async fn frame(collection: Collection) -> Duration { + let scene = collection.scenes[0].name.clone(); + cost(collection, &scene).await.0 + } + + /// The mean time to draw and finish every canvas once with the first scene + /// on program and `preview` in preview, and the render passes and texture + /// copies each frame runs. + async fn cost(collection: Collection, preview: &str) -> (Duration, (usize, usize)) { let mut studio = Studio::new(collection).await; studio.compositor.set_canvases(&studio.collection).unwrap(); let scene = studio.collection.scenes[0].name.clone(); studio.compositor.set_preview(Some(scene)); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.compositor.set_preview(Some(preview.to_string())); let mut total = Duration::ZERO; for frame in 0..FRAMES * 2 { let start = Instant::now(); @@ -3508,7 +4258,53 @@ mod bench { total += Instant::now().saturating_duration_since(start); } } - total / FRAMES + (total / FRAMES, studio.compositor.effects.ran) + } + + /// What a scene source costs: every source placed in the scene on program + /// and preview (direct), the same items in a scene it shows through a + /// scene source (nested), and the program shown again in preview through a + /// program source (program). + #[tokio::test] + #[ignore = "a benchmark; run by hand on a GPU"] + async fn nested_scenes() { + local(async { + println!("sources case frame passes copies"); + for sources in [4, 16, 64] { + let direct = collection(sources, 1, sources); + let scene = direct.scenes[0].name.clone(); + + let mut nested = direct.clone(); + let fill = Rect::fill(&nested.canvas); + let inner = std::mem::take(&mut nested.scenes[0].items); + nested.scenes.push(crate::Scene { + items: inner, + ..crate::Scene::new("inner") + }); + let inner = Source::Scene { scene: "inner".into() }; + nested.sources.insert("inner".into(), inner); + nested.scenes[0].items = vec![Item::new("inner", fill)]; + nested.validate().unwrap(); + + let mut program = direct.clone(); + program.sources.insert("on air".into(), Source::Program); + program.scenes.push(crate::Scene { + items: vec![Item::new("on air", fill)], + ..crate::Scene::new("multiview") + }); + program.validate().unwrap(); + + for (case, collection, preview) in [ + ("direct", direct, scene.as_str()), + ("nested", nested, scene.as_str()), + ("program", program, "multiview"), + ] { + let (frame, (passes, copies)) = cost(collection, preview).await; + println!("{sources:>7} {case:<8} {frame:>7.2?} {passes:>6} {copies:>6}"); + } + } + }) + .await; } /// Sources against canvases, each further canvas placing every source diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index 91e18123..04ca97a0 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -402,50 +402,105 @@ impl Collection { /// is the same one moved. Whatever follows a scene by name, such as the one /// in preview, looks its name up here once. pub fn renames(&self, edit: &Edit) -> BTreeMap> { - // Each place's scene: its name here, or `None` for one the edit added. - let mut scenes: Vec<(Option<&str>, &str)> = self - .scenes - .iter() - .map(|scene| (Some(scene.name.as_str()), scene.name.as_str())) - .collect(); - // Scenes the edit removed, with what each was, until one is put back. - let mut removed: Vec<(Option<&str>, &Scene)> = Vec::new(); + let mut places = Places::new(self); for change in &edit.changes { - let Change::Scene { index, before, after } = change else { - continue; - }; - match (before, after) { - (None, Some(after)) if *index <= scenes.len() => { - let back = removed.iter().position(|(_, scene)| *scene == after); - let was = back.and_then(|back| removed.remove(back).0); - scenes.insert(*index, (was, &after.name)); - } - (Some(before), None) if *index < scenes.len() => { - let (was, _) = scenes.remove(*index); - removed.push((was, before)); - } - (Some(_), Some(after)) if *index < scenes.len() => scenes[*index].1 = &after.name, - _ => {} - } + places.change(change); } - let kept = scenes - .into_iter() - .filter_map(|(was, now)| Some((was?, Some(now)))) - .filter(|(was, now)| Some(*was) != *now); - let gone = removed.into_iter().filter_map(|(was, _)| Some((was?, None))); - kept.chain(gone) - .map(|(was, now)| (was.to_string(), now.map(str::to_string))) + let now = self + .scenes + .iter() + .enumerate() + .map(|(id, scene)| (scene, places.name(id))); + now.filter(|(scene, now)| *now != Some(scene.name.as_str())) + .map(|(scene, now)| (scene.name.clone(), now.map(str::to_string))) .collect() } /// Start an edit labelled `label` against this collection. pub fn draft(&self, label: impl Into) -> Draft { + let places = Places::new(self); + let drawing = self.sources.iter().filter_map(|(name, source)| match source { + Source::Scene { scene } => Some((name.clone(), places.id(scene)?)), + _ => None, + }); Draft { collection: self.clone(), edit: Edit { label: label.into(), changes: Vec::new(), }, + drawing: drawing.collect(), + places, + } + } +} + +/// Which scene each place of a collection's scene list holds through a run of +/// changes, by an id that follows it through renames and moves: its index in +/// the list the run started from, or one past those for a scene the run adds. +struct Places { + /// The id and name of the scene at each place. + scenes: Vec<(usize, String)>, + /// Scenes taken out, until one is put back unchanged, which is then the + /// same scene moved. + removed: Vec<(usize, Scene)>, + /// The id the next scene added takes. + next: usize, +} + +impl Places { + fn new(collection: &Collection) -> Self { + let scenes: Vec<_> = collection + .scenes + .iter() + .map(|scene| scene.name.clone()) + .enumerate() + .collect(); + Self { + next: scenes.len(), + scenes, + removed: Vec::new(), + } + } + + /// Follow `change`, which leaves the places alone unless it is a scene's. + fn change(&mut self, change: &Change) { + let Change::Scene { index, before, after } = change else { + return; + }; + match (before, after) { + (None, Some(after)) if *index <= self.scenes.len() => { + let back = self.removed.iter().position(|(_, scene)| scene == after); + let id = match back { + Some(back) => self.removed.remove(back).0, + None => { + self.next += 1; + self.next - 1 + } + }; + self.scenes.insert(*index, (id, after.name.clone())); + } + (Some(before), None) if *index < self.scenes.len() => { + let (id, _) = self.scenes.remove(*index); + self.removed.push((id, before.clone())); + } + (Some(_), Some(after)) if *index < self.scenes.len() => self.scenes[*index].1 = after.name.clone(), + _ => {} + } + } + + /// The name of scene `id`, or `None` once it is removed. + fn name(&self, id: usize) -> Option<&str> { + let scene = self.scenes.iter().find(|(at, _)| *at == id); + scene.map(|(_, name)| name.as_str()) + } + + /// The id of the scene called `name`, unless no scene or more than one is. + fn id(&self, name: &str) -> Option { + let mut called = self.scenes.iter().filter(|(_, at)| at == name); + match (called.next(), called.next()) { + (Some((id, _)), None) => Some(*id), + _ => None, } } } @@ -455,6 +510,11 @@ impl Collection { pub struct Draft { collection: Collection, edit: Edit, + places: Places, + /// The scene each scene source draws, by its id in `places`, so renaming + /// or removing one reaches its own sources even while another scene has + /// the name it had or takes. + drawing: BTreeMap, } impl Draft { @@ -466,6 +526,23 @@ impl Draft { /// Make `change` and record it. pub fn change(&mut self, change: Change) -> Result<(), Error> { change.apply(&mut self.collection)?; + self.places.change(&change); + if let Change::Source { name, after, .. } = &change { + // A source still naming the scene it follows keeps following it; + // otherwise it follows the one scene by the name it gives. + let kept = + |id: &usize| matches!(after, Some(Source::Scene { scene }) if self.places.name(*id) == Some(scene)); + if !self.drawing.get(name).is_some_and(kept) { + let id = match after { + Some(Source::Scene { scene }) => self.places.id(scene), + _ => None, + }; + match id { + Some(id) => self.drawing.insert(name.clone(), id), + None => self.drawing.remove(name), + }; + } + } self.edit.changes.push(change); Ok(()) } @@ -582,14 +659,36 @@ impl Draft { }) } - /// Remove the scene at `index`, with its items. + /// Remove the scene at `index`, with its items, and every scene source + /// drawing it, as [`remove_source`](Self::remove_source) removes one. pub fn remove_scene(&mut self, index: usize) -> Result<(), Error> { + self.scene_at(index)?; + let drawing = self.drawing(index); + // Taken first, so the items its sources' removal finds by scene name + // are another scene's, even one that took its name in this draft. + self.take_scene(index)?; + for source in drawing { + self.remove_source(&source)?; + } + Ok(()) + } + + /// Take the scene at `index` out, leaving what refers to it. + fn take_scene(&mut self, index: usize) -> Result { let before = self.scene_at(index)?; self.change(Change::Scene { index, - before: Some(before), + before: Some(before.clone()), after: None, - }) + })?; + Ok(before) + } + + /// The scene sources drawing the scene at `index`. + fn drawing(&self, index: usize) -> Vec { + let id = self.places.scenes[index].0; + let drawing = self.drawing.iter().filter(|(_, drawn)| **drawn == id); + drawing.map(|(name, _)| name.clone()).collect() } /// Replace the scene at `index`. @@ -602,19 +701,28 @@ impl Draft { }) } - /// Rename the scene at `index`, keeping what it shows. + /// Rename the scene at `index`, keeping what it shows, and every scene + /// source drawing it. pub fn rename_scene(&mut self, index: usize, name: impl Into) -> Result<(), Error> { - let scene = Scene { - name: name.into(), - ..self.scene_at(index)? - }; - self.set_scene(index, scene) + let before = self.scene_at(index)?; + let name = name.into(); + self.set_scene( + index, + Scene { + name: name.clone(), + ..before + }, + )?; + for source in self.drawing(index) { + let scene = name.clone(); + self.set_source(&source, Some(Source::Scene { scene }))?; + } + Ok(()) } /// Move the scene at `from` to `to`, counted after its removal. pub fn move_scene(&mut self, from: usize, to: usize) -> Result<(), Error> { - let scene = self.scene_at(from)?; - self.remove_scene(from)?; + let scene = self.take_scene(from)?; self.insert_scene(to, scene) } @@ -1146,6 +1254,40 @@ mod tests { assert_eq!(collection, before); } + /// A scene source follows its scene through a rename and a move, and goes + /// with it, with the items showing it, when it is removed, all undone whole. + #[test] + fn a_scene_source_follows_its_scene() { + let mut collection = collection(); + let fill = Rect::fill(&collection.canvas); + collection.scenes.push(Scene { + items: vec![Item::new("red", fill)], + ..Scene::new("Inner") + }); + let inner = Source::Scene { scene: "Inner".into() }; + collection.sources.insert("inner".into(), inner); + collection.scenes[0].items.push(Item::new("inner", fill)); + collection.validate().unwrap(); + let mut history = History::new(); + + let mut draft = collection.draft("Rename `Inner`"); + draft.rename_scene(1, "Nested").unwrap(); + draft.move_scene(1, 0).unwrap(); + commit(&mut collection, &mut history, draft, None); + assert_eq!(collection.sources["inner"], Source::Scene { scene: "Nested".into() }); + assert_eq!(collection.scenes[0].name, "Nested"); + let renamed = collection.clone(); + + let mut draft = collection.draft("Remove `Nested`"); + draft.remove_scene(0).unwrap(); + commit(&mut collection, &mut history, draft, None); + assert!(!collection.sources.contains_key("inner")); + assert_eq!(collection.scenes[0].items.len(), 1, "the item showing it went too"); + + history.undo(&mut collection).unwrap(); + assert_eq!(collection, renamed); + } + /// A removed source's filters go with it, from items and other sources /// alike, leaving their other filters. #[test] @@ -1239,11 +1381,19 @@ mod tests { /// A rename keeps the scene's items and place, a move keeps its name, and /// each scene maps to its own new name, even through names the edit - /// frees and reuses. + /// frees and reuses, as the scene sources drawing it follow it. #[test] fn renames_and_reorders_scenes() { let mut collection = collection(); collection.scenes.push(Scene::new("Other")); + for (source, scene) in [("first", "Scene"), ("second", "Other")] { + let scene = scene.to_string(); + collection.sources.insert(source.into(), Source::Scene { scene }); + } + let draws = |collection: &Collection, source: &str| match collection.sources.get(source) { + Some(Source::Scene { scene }) => Some(scene.clone()), + _ => None, + }; let start = collection.clone(); let mut history = History::new(); @@ -1258,6 +1408,8 @@ mod tests { history.record(edit, None); assert_eq!(collection.scenes[0].name, "Other"); assert_eq!(collection.scenes[0].items, start.scenes[0].items); + assert_eq!(draws(&collection, "first").as_deref(), Some("Other")); + assert_eq!(draws(&collection, "second").as_deref(), Some("Scene")); let mut draft = collection.draft("Move `Other` down"); draft.move_scene(0, 1).unwrap(); @@ -1285,6 +1437,8 @@ mod tests { "an added scene renames nothing" ); collection.apply(&edit).unwrap(); + assert_eq!(draws(&collection, "first").as_deref(), Some("Other")); + assert_eq!(draws(&collection, "second").as_deref(), Some("Next")); assert_eq!( collection.renames(&edit.inverse()), renamed(&[("Other", Some("Scene")), ("Next", Some("Other")), ("New", None)]), @@ -1300,6 +1454,9 @@ mod tests { collection.renames(&edit), renamed(&[("Other", None), ("Next", Some("Other"))]) ); + collection.apply(&edit).unwrap(); + assert_eq!(draws(&collection, "first"), None, "it went with its scene"); + assert_eq!(draws(&collection, "second").as_deref(), Some("Other")); // Names stay unique and non-empty. for name in ["Other", ""] { @@ -1307,6 +1464,20 @@ mod tests { draft.rename_scene(0, name).unwrap(); assert!(matches!(draft.finish(), Err(Error::Invalid(_))), "{name:?}"); } + + // Removing a scene whose name another just took removes only its own + // scene sources, and the items showing them from the other. + let scene = "New".to_string(); + collection.sources.insert("third".into(), Source::Scene { scene }); + let fill = Rect::fill(&collection.canvas); + collection.scenes[1].items.push(Item::new("third", fill)); + let mut draft = collection.draft("Take over"); + draft.rename_scene(1, "New").unwrap(); + draft.remove_scene(0).unwrap(); + collection.apply(&draft.finish().unwrap()).unwrap(); + assert_eq!(draws(&collection, "third"), None); + assert_eq!(draws(&collection, "second").as_deref(), Some("New")); + assert!(collection.scenes[0].items.is_empty()); } /// The canvas and the transition are edits like any other: a drag of one diff --git a/rs/moq-scene/src/effect.rs b/rs/moq-scene/src/effect.rs index b2f010eb..d345da9c 100644 --- a/rs/moq-scene/src/effect.rs +++ b/rs/moq-scene/src/effect.rs @@ -405,23 +405,30 @@ enum Command { Backdrop { texture: wgpu::Texture, area: Area }, } -/// One render's passes: the rasters items are filtered through, then the -/// targets, in the order they are drawn. +/// One render's passes: each target's after the rasters its items are +/// filtered through, and the targets in the order they are drawn, so an item +/// can show a target drawn before its own. #[derive(Default)] pub(crate) struct Frame { + /// The passes of the targets finished, in order. + done: Vec, + /// The rasters of the target drawn now. rasters: Vec, + /// The target drawn now: its passes, and the copies a blend mode makes. targets: Vec, /// The canvas of the target drawn now, which items are prepared for. canvas: Canvas, } impl Frame { - /// Start drawing a target the size of `canvas`, cleared to opaque black. - pub fn target(&mut self, target: &Picture, canvas: Canvas) { + /// Start drawing a target the size of `canvas`, cleared to `clear`. + pub fn target(&mut self, target: &Picture, canvas: Canvas, clear: wgpu::Color) { + self.done.append(&mut self.rasters); + self.done.append(&mut self.targets); self.canvas = canvas; self.targets.push(Command::Pass { view: target.view.clone(), - clear: Some(wgpu::Color::BLACK), + clear: Some(clear), draws: Vec::new(), }); } @@ -492,6 +499,11 @@ pub(crate) struct Effects { free: Vec, /// Rasters used by this render. used: Vec, + + /// The render passes and texture copies the last submit ran, for the + /// benchmarks. + #[cfg(test)] + pub(crate) ran: (usize, usize), } /// What a blend mode mixes with, `width` by `height`. @@ -656,6 +668,8 @@ impl Effects { backdrop, free: Vec::new(), used: Vec::new(), + #[cfg(test)] + ran: (0, 0), } } @@ -738,7 +752,22 @@ impl Effects { /// Run `frame`'s passes. pub fn submit(&mut self, frame: Frame) { - let commands: Vec = frame.rasters.into_iter().chain(frame.targets).collect(); + let Frame { + mut done, + rasters, + targets, + .. + } = frame; + done.extend(rasters.into_iter().chain(targets)); + let commands = done; + #[cfg(test)] + { + let copies = commands + .iter() + .filter(|command| matches!(command, Command::Backdrop { .. })) + .count(); + self.ran = (commands.len() - copies, copies); + } let draws = commands.iter().flat_map(|command| match command { Command::Pass { draws, .. } => draws.as_slice(), Command::Backdrop { .. } => &[], diff --git a/rs/moq-scene/src/effect.wgsl b/rs/moq-scene/src/effect.wgsl index 3afcab6a..0a7f58f4 100644 --- a/rs/moq-scene/src/effect.wgsl +++ b/rs/moq-scene/src/effect.wgsl @@ -306,13 +306,24 @@ fn mix_blend(backdrop: vec3, source: vec3, mode: u32) -> vec3 { } } -// The input through the ops, mixed with the opaque backdrop in `aux` by the -// draw's blend mode: (1 - αs) Cb + αs B(Cb, Cs). +// The input through the ops, mixed with the premultiplied backdrop in `aux` +// by the draw's blend mode, as the Compositing and Blending spec composites +// over a backdrop that may be transparent, as a nested scene's is: +// Cs' = (1 - αb) Cs + αb B(Cb, Cs), then Cs' over the backdrop. Over an opaque +// backdrop that is (1 - αs) Cb + αs B(Cb, Cs). @fragment fn fs_blend(in: Varyings) -> @location(0) vec4 { let source = apply(straight(in.input), in.item); - let backdrop = textureLoad(aux, vec2(in.position.xy), 0).rgb; - let mixed = mix_blend(backdrop, source.rgb, draw.mode.z); - return vec4(mix(backdrop, mixed, source.a), 1.0); + let backdrop = textureLoad(aux, vec2(in.position.xy), 0); + var straight_backdrop = vec3(0.0); + if backdrop.a > 0.0 { + straight_backdrop = backdrop.rgb / backdrop.a; + } + let mixed = mix_blend(straight_backdrop, source.rgb, draw.mode.z); + let blended = mix(source.rgb, mixed, backdrop.a); + return vec4( + source.a * blended + (1.0 - source.a) * backdrop.rgb, + source.a + (1.0 - source.a) * backdrop.a, + ); } // One direction of a separable Gaussian blur of a premultiplied raster. With diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index c9c9b458..913ba9a4 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -20,9 +20,10 @@ //! textures on the caller's `wgpu` device, one pair per canvas from one //! upload of each source's picture, cutting or running a [`Transition`] //! between them on every canvas together, and plays each item's -//! [`Keyframe`]s from when its scene went on. A source whose [`Background`] -//! is removed is cut out first, by a segmentation model on the CPU. -//! `moq_video::render::export` hands the +//! [`Keyframe`]s from when its scene went on. A scene source shows another +//! scene, drawn first into a texture of its own, or the program itself. A +//! source whose [`Background`] is removed is cut out first, by a +//! segmentation model on the CPU. `moq_video::render::export` hands the //! program to an encoder, on the GPU where the platform allows, and a //! [`Scaler`] draws it smaller for a smaller rendition. //! - [`publish`] gives each local source its own track, in a broadcast the @@ -52,7 +53,7 @@ pub mod source; pub use clip::Clip; pub use collection::{ Audio, Background, Blend, Canvas, Collection, Crop, Direction, Ease, Extra, Filter, Item, Keyframe, Layer, - MaskMode, Mixes, Rect, Rgba, Scene, Source, Transition, Video, + MaskMode, Mixes, Nested, Rect, Rgba, Scene, Source, Transition, Video, }; pub use compositor::{Compositor, FORMAT}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index fa20256b..ac9fd00b 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -755,7 +755,9 @@ impl Driver { let plays = self.plays.clone().expect("a replay source has plays"); self.replays(plays, listing).await } - Source::Clone { .. } => unreachable!("a clone opens no feed"), + Source::Clone { .. } | Source::Scene { .. } | Source::Program => { + unreachable!("a clone or scene opens no feed") + } device => self.device(device, listing).await, } } @@ -802,7 +804,9 @@ impl Driver { | Source::Overlay { .. } | Source::Clone { .. } | Source::Lut { .. } - | Source::Replay => unreachable!("not a device source"), + | Source::Replay + | Source::Scene { .. } + | Source::Program => unreachable!("not a device source"), }; let mut config = moq_video::capture::Config::default(); @@ -1866,8 +1870,8 @@ fn open(name: &str, source: &Source, context: &Context, held: bool) -> (Feed, Dr /// A broadcast is already published by its own publisher, and a clone shows /// its source's track, its filters carried by the layout rather than encoded /// again. Web pages and overlays are seen only in the program, since H.264 -/// would encode their transparency over black, and a LUT is looked up, never -/// shown. +/// would encode their transparency over black, a LUT is looked up, never +/// shown, and a scene is drawn from other sources. fn publishes(source: &Source) -> bool { match source { Source::Camera { .. } @@ -1883,7 +1887,9 @@ fn publishes(source: &Source) -> bool { | Source::Browser { .. } | Source::Overlay { .. } | Source::Clone { .. } - | Source::Lut { .. } => false, + | Source::Lut { .. } + | Source::Scene { .. } + | Source::Program => false, } } diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index f4ea6ce1..2b608ba2 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -186,8 +186,8 @@ impl State { /// the one drawn, once the compositor takes its canvas. The scenes in /// preview and on program follow the renames and removals `made` holds, /// each edit read against the collection it was made to, and any - /// `collection` lacks is dropped. Edits that remove a scene on air are - /// refused, changing nothing. + /// `collection` lacks is dropped. Edits that remove a scene on air, or + /// leave one drawing the program, are refused, changing nothing. fn adopt(&mut self, collection: Collection, made: &[Edit]) -> anyhow::Result<()> { let mut before = self.collection.clone(); let mut renames = Vec::with_capacity(made.len()); @@ -195,7 +195,7 @@ impl State { renames.push(before.renames(edit)); before.apply(edit)?; } - keep_on_air(self.compositor.on_air(), &renames)?; + keep_on_air(self.compositor.on_air(), &renames, &collection)?; self.compositor.set_canvases(&collection)?; for renames in &renames { self.compositor.rename(renames); @@ -222,26 +222,38 @@ impl State { /// The sources whose pictures the scenes on air place on any canvas's /// program, through clones: each visible item's of the scenes on air, /// including one leaving in a transition, and of the top layers over them, - /// and each mask's it reads. + /// and of the scenes they draw, and each mask's it reads. fn placed(&self) -> BTreeSet { let collection = &self.collection; let scenes: Vec> = self.compositor.on_air().map(Some).chain([None]).collect(); let canvases = std::iter::once(None).chain(collection.canvases.keys().map(Some)); - let items = canvases - .flat_map(|canvas| { - scenes.iter().map(move |scene| moq_scene::Layer { - canvas: canvas.cloned(), - scene: scene.map(str::to_string), - }) + let layers = canvases.flat_map(|canvas| { + scenes.iter().map(move |scene| moq_scene::Layer { + canvas: canvas.cloned(), + scene: scene.map(str::to_string), }) - .flat_map(|layer| collection.items(&layer).unwrap_or_default()) - .filter(|item| item.visible); + }); + let items = collection.drawn(layers); items + .into_iter() .flat_map(|item| std::iter::once(&item.source).chain(masks(collection.filters(item)))) .map(|source| collection.capture(source).to_string()) .collect() } + /// Put the preview on program at once; see [`Compositor::cut`]. + pub fn cut(&mut self) -> anyhow::Result<()> { + Ok(self.compositor.cut(&self.collection, &self.sources)?) + } + + /// Take the preview to program by `transition`, starting at `now`; see + /// [`Compositor::transition`]. + pub fn transition(&mut self, transition: moq_scene::Transition, now: Instant) -> anyhow::Result<()> { + Ok(self + .compositor + .transition(&self.collection, transition, &self.sources, now)?) + } + /// Restart the source called `name`, as an edit would, such as to reload a /// web page. pub fn restart(&mut self, name: &str) { @@ -291,11 +303,14 @@ impl autosave::Document for State { } /// Refuse edits, each read as its [`Collection::renames`], that remove a -/// scene `on_air` program draws, so a slip cannot put black on air. A reload -/// that leaves one out still drops it, having nothing to show instead. +/// scene `on_air` program draws, so a slip cannot put black on air, or that +/// leave one drawing the program in `collection`, which it would draw into +/// itself. A reload that leaves one out still drops it, having nothing to +/// show instead. fn keep_on_air<'a>( on_air: impl IntoIterator, renames: &'a [BTreeMap>], + collection: &Collection, ) -> anyhow::Result<()> { for mut scene in on_air { for renames in renames { @@ -305,6 +320,10 @@ fn keep_on_air<'a>( None => {} } } + anyhow::ensure!( + !collection.draws_program(scene), + "`{scene}` is on air, so it cannot draw the program, which it would draw into itself" + ); } Ok(()) } @@ -774,8 +793,13 @@ impl Engine { }); let mut compositor = Compositor::new(&gpu.device, &gpu.queue, collection.canvas)?; compositor.set_canvases(&collection)?; - compositor.set_preview(collection.scenes.first().map(|scene| scene.name.clone())); - compositor.cut(&sources); + // The first scene that can go on program starts there, and in preview. + let first = collection + .scenes + .iter() + .find(|scene| !collection.draws_program(&scene.name)); + compositor.set_preview(first.map(|scene| scene.name.clone())); + compositor.cut(&collection, &sources)?; let engine = Arc::new(Self { state: Mutex::new(State { @@ -2845,9 +2869,7 @@ mod tests { let mut state = engine.state.lock().unwrap(); let transition = state.collection().transition.clone(); let state = &mut *state; - state - .compositor - .transition(transition, &state.sources, kio::clock::now()); + state.transition(transition, kio::clock::now()).unwrap(); let shown = state.shown(); assert!(shown.contains("sting") && shown.contains("matte"), "{shown:?}"); } @@ -2916,16 +2938,36 @@ mod tests { collection.renames(&draft.finish().unwrap()) }; let removed = [remove(&collection, 0)]; - assert!(keep_on_air([live.as_str()], &removed).is_err()); - assert!(keep_on_air(["Renamed", live.as_str()], &removed).is_err(), "leaving"); - keep_on_air([live.as_str()], &[remove(&collection, 1)]).unwrap(); - keep_on_air([], &removed).unwrap(); + assert!(keep_on_air([live.as_str()], &removed, &collection).is_err()); + assert!( + keep_on_air(["Renamed", live.as_str()], &removed, &collection).is_err(), + "leaving" + ); + keep_on_air([live.as_str()], &[remove(&collection, 1)], &collection).unwrap(); + keep_on_air([], &removed, &collection).unwrap(); // Undoing both edits, newest first, removes `Other` once renamed. let undo = [collection.renames(&rename.inverse()), added.renames(&add.inverse())]; - let err = keep_on_air(["Renamed"], &undo).unwrap_err(); + let err = keep_on_air(["Renamed"], &undo, &collection).unwrap_err(); assert!(err.to_string().contains("`Other` is on air"), "{err}"); - keep_on_air([live.as_str()], &undo).unwrap(); + keep_on_air([live.as_str()], &undo, &collection).unwrap(); + + // Nor does one leave a scene on air drawing the program, which only a + // scene off air may. + let mut draft = collection.draft("Show the program"); + draft.set_source("on air", Some(moq_scene::Source::Program)).unwrap(); + let fill = moq_scene::Rect::fill(&collection.canvas); + let item = moq_scene::Item::new("on air", fill); + draft + .insert_item(&moq_scene::Layer::scene(live.clone()), 0, item) + .unwrap(); + let edit = draft.finish().unwrap(); + let mut drawing = collection.clone(); + drawing.apply(&edit).unwrap(); + let renames = [collection.renames(&edit)]; + let err = keep_on_air([live.as_str()], &renames, &drawing).unwrap_err(); + assert!(err.to_string().contains("cannot draw the program"), "{err}"); + keep_on_air(["Renamed"], &renames, &drawing).unwrap(); } async fn subscribe( diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 64e88af7..75b0b98a 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -1036,17 +1036,16 @@ impl Panels { let mut gesture = None; ui.horizontal(|ui| { if ui.button("Cut").clicked() { - let state = &mut *state; - state.compositor.cut(&state.sources); + let cut = state.cut(); + self.check(cut); } if ui .button("Transition") .on_hover_text("Take the preview to program by the transition set beside it") .clicked() { - let state = &mut *state; - let now = kio::clock::Instant::now(); - state.compositor.transition(transition.clone(), &state.sources, now); + let taken = state.transition(transition.clone(), kio::clock::Instant::now()); + self.check(taken); } match &next { Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } => self.direction = *direction, @@ -1243,9 +1242,17 @@ impl Panels { SceneAction::Move(from, to) => edit(&mut state, format!("Reorder scene `{name}`"), None, |draft| { draft.move_scene(from, to) }), - SceneAction::Remove(index) => edit(&mut state, format!("Remove scene `{name}`"), None, |draft| { - draft.remove_scene(index) - }), + SceneAction::Remove(index) => { + let result = edit(&mut state, format!("Remove scene `{name}`"), None, |draft| { + draft.remove_scene(index) + }); + if result.is_ok() { + // Removing its scene sources removes the items showing + // them, which moves the ones after them. + self.deselect_item(); + } + result + } }; drop(state); self.check(result); @@ -1510,6 +1517,8 @@ impl Panels { .on_hover_text(health::without(stall)); } (None, Some(feed), None) => status(ui, &feed.status()), + // A scene is drawn from other sources, so it runs nothing. + (None, None, None) if collection.nested(name).is_some() => status(ui, &Status::Live), (None, None, None) => status(ui, &Status::Starting), } // A track can fail while its source still draws, such as on a GPU @@ -1604,7 +1613,13 @@ impl Panels { }); ui.separator(); - if let Some(source) = self.add.ui(ui) { + let scenes: Vec = state + .collection() + .scenes + .iter() + .map(|scene| scene.name.clone()) + .collect(); + if let Some(source) = self.add.ui(ui, &scenes) { let name = self.add.name.trim().to_string(); let layer = self.layer(&state).ok(); let result = add(&mut state, layer, name, source, None); @@ -2507,7 +2522,7 @@ impl Panels { }; // What a source shows or plays, a clone its source's, has a past // only if something encodes it: not an image, a color, a web page, - // an overlay, or a LUT. + // an overlay, a LUT, or a scene or the program drawn into another. let collection = state.collection(); let replayable = collection .sources @@ -2520,6 +2535,7 @@ impl Panels { Source::Image { .. } | Source::Color { .. } | Source::Browser { .. } | Source::Overlay { .. } | Source::Lut { .. } + | Source::Scene { .. } | Source::Program ) }) }) @@ -3651,6 +3667,8 @@ fn describe(source: &Source) -> String { Source::Clone { source } => format!("clone of {source}"), Source::Replay => "replay".into(), Source::Lut { path } => format!("LUT: {}", path.display()), + Source::Scene { scene } => format!("scene: {scene}"), + Source::Program => "program".into(), } } @@ -3711,6 +3729,8 @@ struct AddSource { /// each time it goes on program. looping: bool, restart: bool, + /// The scene a scene source draws, or `None` for the program. + scene: Option, } impl Default for AddSource { @@ -3725,6 +3745,7 @@ impl Default for AddSource { height: canvas.height, looping: false, restart: true, + scene: None, } } } @@ -3745,10 +3766,11 @@ enum Kind { Overlay, Replay, Lut, + Scene, } impl Kind { - const ALL: [Self; 13] = [ + const ALL: [Self; 14] = [ Self::Color, Self::Image, Self::Media, @@ -3762,6 +3784,7 @@ impl Kind { Self::Overlay, Self::Replay, Self::Lut, + Self::Scene, ]; fn label(self) -> &'static str { @@ -3779,13 +3802,14 @@ impl Kind { Self::Overlay => "Overlay", Self::Replay => "Replay", Self::Lut => "LUT", + Self::Scene => "Scene", } } /// What the value field holds, if this kind takes one. fn value(self) -> Option<&'static str> { match self { - Self::Color | Self::SystemAudio | Self::Replay => None, + Self::Color | Self::SystemAudio | Self::Replay | Self::Scene => None, Self::Image | Self::Media => Some("file"), Self::Lut => Some(".cube file"), Self::Broadcast => Some("path"), @@ -3798,8 +3822,9 @@ impl Kind { } impl AddSource { - /// Draw the form; the source to add once the user asks for it. - fn ui(&mut self, ui: &mut Ui) -> Option { + /// Draw the form, offering `scenes` to a scene source; the source to add + /// once the user asks for it. + fn ui(&mut self, ui: &mut Ui, scenes: &[String]) -> Option { let mut add = false; egui::Grid::new("add source").show(ui, |ui| { ui.label("Name"); @@ -3837,6 +3862,23 @@ impl AddSource { egui::color_picker::color_edit_button_srgba(ui, &mut self.color, egui::color_picker::Alpha::OnlyBlend); ui.end_row(); } + if self.kind == Kind::Scene { + // A pick whose scene was removed or renamed away names no scene. + if self.scene.as_ref().is_some_and(|scene| !scenes.contains(scene)) { + self.scene = None; + } + ui.label("Draws"); + egui::ComboBox::from_id_salt("scene") + .selected_text(self.scene.as_deref().unwrap_or("Program")) + .show_ui(ui, |ui| { + ui.selectable_value(&mut self.scene, None, "Program") + .on_hover_text("Whatever is on air, transitions included"); + for scene in scenes { + ui.selectable_value(&mut self.scene, Some(scene.clone()), scene); + } + }); + ui.end_row(); + } if self.kind == Kind::Overlay { ui.label("Animation"); ui.checkbox(&mut self.restart, "Restart on program").on_hover_text( @@ -3892,6 +3934,10 @@ impl AddSource { restart: self.restart, }, Kind::Replay => Source::Replay, + Kind::Scene => match &self.scene { + Some(scene) => Source::Scene { scene: scene.clone() }, + None => Source::Program, + }, }) } } diff --git a/rs/moq-studio/tests/browser.rs b/rs/moq-studio/tests/browser.rs index bfe3baa2..e35729cc 100644 --- a/rs/moq-studio/tests/browser.rs +++ b/rs/moq-studio/tests/browser.rs @@ -87,7 +87,7 @@ impl Studio { } let mut compositor = Compositor::new(&device, &queue, collection.canvas).unwrap(); compositor.set_preview(Some("Main".into())); - compositor.cut(&sources); + compositor.cut(&collection, &sources).unwrap(); Self { collection, sources,