From 1fc79a7416ba3d8676d9fbab8e863d0e2e749499 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 10:29:16 -0800 Subject: [PATCH 1/8] chore(quest): claim quest/m0/scenes/scene-source.md Quest-Claim: 88e2392b2973a100 From 832dba28e27302ec63194135c22ca8a3c3ae5b2f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 11:03:44 -0800 Subject: [PATCH 2/8] feat(scene)!: a scene source draws another scene, or the program A new source kind, `scene`, draws another scene as one picture, and `program` draws whatever the program output shows, transitions included, on the same frame. A nested scene renders once per frame at the main canvas's size into a transparent texture of its own, only while something drawn shows it; its keyframes play from when the scene showing it went on. A scene cannot draw itself, even through others; a scene drawing the program cannot go on program, so Compositor::cut and transition now take the collection and refuse it, and the studio refuses an edit making the scene on air draw it. Renaming or removing a scene carries its scene sources along. Frames now run each target's rasters just before it, so a blur reads a nested scene drawn this frame, and a blend mode composites over a transparent backdrop as the CSS spec does. Co-Authored-By: Claude Opus 5.5 --- README.md | 34 +- quest/m0/scenes/README.md | 1 - quest/m0/scenes/audio.md | 4 - quest/m0/scenes/published.md | 9 +- quest/m0/scenes/scene-source.md | 23 - rs/moq-scene/src/collection.rs | 274 +++++++++++- rs/moq-scene/src/compositor.rs | 765 +++++++++++++++++++++++++++----- rs/moq-scene/src/edit.rs | 71 ++- rs/moq-scene/src/effect.rs | 41 +- rs/moq-scene/src/effect.wgsl | 21 +- rs/moq-scene/src/lib.rs | 5 +- rs/moq-scene/src/source.rs | 16 +- rs/moq-studio/src/engine.rs | 92 ++-- rs/moq-studio/src/ui.rs | 57 ++- rs/moq-studio/tests/browser.rs | 2 +- 15 files changed, 1217 insertions(+), 198 deletions(-) delete mode 100644 quest/m0/scenes/scene-source.md diff --git a/README.md b/README.md index 97cc2bfd..ded988f2 100644 --- a/README.md +++ b/README.md @@ -132,6 +132,29 @@ Each further canvas publishes a broadcast of its own beneath the main one, at encoding it again. Removing a canvas ends its broadcast, and removes its top layer and every scene's layout on it. +### Scene sources + +A **scene source** draws another scene as one picture, which an item places, +scales, crops, filters, and blends like any other: a lower third built once +and shown in every scene, or a whole layout shrunk into a corner. Add one in +the Sources panel as kind **Scene**, picking the scene it draws, or +**Program**, which shows whatever is on air, a transition under way included, +on the same frame, such as for a portrait canvas that follows the program, +cropped. + +A nested scene is drawn at the main canvas's size, transparent where it draws +nothing, once per frame however many items show it, and only while something +drawn shows it. Its items' keyframes play from when the scene showing it goes +on, as that scene's own items' do, and a media file in it set to restart plays +from its start when the scene showing it goes on program. + +A scene cannot draw itself, even through other scenes, so an edit that would +is refused. A scene showing the program cannot go on program, nor can the main +canvas's top layer show it, since either would draw the program into itself: +**Cut** and **Transition** refuse such a scene, and an edit making the scene on +air show the program is refused. Renaming a scene renames what its scene +sources draw, and removing one removes them, with the items showing them. + ### Transitions | Transition | What it does | @@ -516,6 +539,13 @@ delay = "80ms" kind = "clone" source = "camera" +[sources.main-view] +kind = "scene" +scene = "Main" # draws the Main scene as one picture + +[sources.on-air] +kind = "program" # whatever the program shows + [video.camera-faded] filters = [{ kind = "opacity", amount = 0.5 }] @@ -551,6 +581,7 @@ items = [ { at = "0s", x = 700, filters = [{ kind = "blur", radius = 12 }] }, { at = "500ms", ease = "out", filters = [{ kind = "blur", radius = 0 }] }, ] }, + { source = "main-view", rect = { x = 80, y = 80, width = 640, height = 360 } }, ] [[top]] @@ -565,7 +596,8 @@ Source kinds are `camera`, `display`, and `microphone` (each with an optional `broadcast`, `browser` (a `url`, and a `width` and `height` up to 4096), `overlay` (its `html`, and a `width` and `height` up to 4096), `clone` (with the `source` it shows), `lut` (a 3D Adobe or Resolve `.cube` file, which no item places, but which grades what a `lut` filter names it in), -and `replay`, where the Replay panel plays replays. The Devices panel fills in +`replay`, where the Replay panel plays replays, `scene` (with the `scene` it +draws), and `program` (see [Scene sources](#scene-sources)). The Devices panel fills in ids. `[[top]]` items make up the top layer. `[canvases.]` adds a further canvas, with its `width` and `height` and diff --git a/quest/m0/scenes/README.md b/quest/m0/scenes/README.md index db94bf4c..b41a9ab7 100644 --- a/quest/m0/scenes/README.md +++ b/quest/m0/scenes/README.md @@ -51,6 +51,5 @@ reads skips drawing. Per-scene cost grows only with what that scene draws; ## Required -- [Scene source](/quest/m0/scenes/scene-source.md) - a scene draws another scene, or the live program, as a source - [Published scenes](/quest/m0/scenes/published.md) - any scene publishes as its own broadcast at its own size, replacing canvases - [Audio scenes](/quest/m0/scenes/audio.md) - mixes become audio-only scenes that nest diff --git a/quest/m0/scenes/audio.md b/quest/m0/scenes/audio.md index 119f86e9..4120aac4 100644 --- a/quest/m0/scenes/audio.md +++ b/quest/m0/scenes/audio.md @@ -23,7 +23,3 @@ Decided 2026-11-05 with the maintainer (/quest-plan); see the - Switching or editing an audio scene never drops or repeats audio, as with mixes today. - `README.md` documents audio scenes. - -## Required - -- [Scene source](/quest/m0/scenes/scene-source.md) - shares the scene graph and its cycle check diff --git a/quest/m0/scenes/published.md b/quest/m0/scenes/published.md index ef6b2817..3c5514e8 100644 --- a/quest/m0/scenes/published.md +++ b/quest/m0/scenes/published.md @@ -7,7 +7,7 @@ broadcast at `<--broadcast>/`, at its own size, with the audio scene it picks (the one on program by default). This replaces canvases: further canvases, their per-scene layouts, and the canvas picker are removed, and a portrait broadcast is a published portrait scene that draws the program -through a [scene source](/quest/m0/scenes/scene-source.md). +through a program source (`kind = "program"`). ## Plan @@ -15,7 +15,8 @@ Decided 2026-11-05 with the maintainer (/quest-plan); see the [epic](/quest/m0/scenes/README.md). - Every scene has a size, defaulting to the program's; the switcher offers - only scenes of the program's size. + only scenes of the program's size. A scene source then draws its scene at + that size, rather than the main canvas's as it does now. - The program keeps one switcher at `<--broadcast>`. A published scene is always live and never switched. - Each published scene gets the program's two sizes (full and half) and the @@ -26,7 +27,3 @@ Decided 2026-11-05 with the maintainer (/quest-plan); see the and no output reads skips drawing. - Remove `[canvases]` from the scene file with no migration; nothing is released. `README.md` documents published scenes. - -## Required - -- [Scene source](/quest/m0/scenes/scene-source.md) - a portrait scene follows the program through it diff --git a/quest/m0/scenes/scene-source.md b/quest/m0/scenes/scene-source.md deleted file mode 100644 index ab40477f..00000000 --- a/quest/m0/scenes/scene-source.md +++ /dev/null @@ -1,23 +0,0 @@ -# [M] Scene source - -## Goal - -A new source kind, `scene`, draws another scene, or `program` (whatever is on -air, including a transition in progress), as a picture that an item places, -scales, crops, and filters like any other. A scene cannot contain itself, -directly or through other scenes; such an edit is refused with an error. - -## Plan - -Decided 2026-11-05 with the maintainer (/quest-plan); see the -[epic](/quest/m0/scenes/README.md) for the model. - -- A nested scene renders at its own size into a texture the compositor - samples, once per frame however many items draw it, and only while - something that is drawn reads it. -- `program` reads the frame the program output draws, so a scene drawing it - shows the same cut on the same frame, with no extra composite. -- Keyframes, stingers, and restarts in a nested scene start when the scene - that draws it goes on program, as its items would. -- Measure frame time and GPU copies with and without a nested scene. -- `README.md` documents the source kind. diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 6d5e23f4..27d6f793 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -190,6 +190,115 @@ impl Collection { } } + /// What the source called `name` draws if it is a scene or program + /// source, through a clone. + pub fn nested(&self, name: &str) -> Option> { + match self.sources.get(self.capture(name))? { + Source::Scene { scene } => Some(Nested::Scene(scene)), + Source::Program => Some(Nested::Program), + _ => None, + } + } + + /// The visible items `layers` draw, then those of each scene they draw + /// through a scene source, and so on, each scene once. + pub fn drawn(&self, layers: impl IntoIterator) -> Vec<&Item> { + let mut items: Vec<&Item> = layers + .into_iter() + .flat_map(|layer| self.items(&layer).unwrap_or_default()) + .filter(|item| item.visible) + .collect(); + let mut nested = BTreeSet::new(); + let mut next = 0; + while let Some(item) = items.get(next) { + if let Some(Nested::Scene(scene)) = self.nested(&item.source) + && nested.insert(scene) + && let Some(scene) = self.scene(scene) + { + items.extend(scene.items.iter().filter(|item| item.visible)); + } + next += 1; + } + items + } + + /// Whether the scene called `scene` places a program source, itself or in + /// a scene it draws, shown or hidden. It then cannot go on program, which + /// it would draw into itself. + pub fn draws_program(&self, scene: &str) -> bool { + let mut scenes = vec![scene]; + let mut seen = BTreeSet::new(); + while let Some(scene) = scenes.pop() { + let Some(scene) = self.scene(scene).filter(|scene| seen.insert(&scene.name)) else { + continue; + }; + for item in &scene.items { + match self.nested(&item.source) { + Some(Nested::Program) => return true, + Some(Nested::Scene(scene)) => scenes.push(scene), + None => {} + } + } + } + false + } + + /// Refuse a scene that draws itself, directly or through other scenes, + /// shown or hidden, since it would never finish drawing. + fn validate_nesting(&self) -> Result<(), Error> { + /// Walk what `scene` draws, refusing a scene already on `path`, the + /// scenes drawing it. `done` holds the scenes walked whole. + fn visit<'a>( + collection: &'a Collection, + scene: &'a Scene, + done: &mut BTreeSet<&'a str>, + path: &mut Vec<&'a str>, + ) -> Result<(), Error> { + if done.contains(scene.name.as_str()) { + return Ok(()); + } + if let Some(at) = path.iter().position(|name| *name == scene.name) { + let through: Vec = path[at + 1..].iter().map(|name| format!("`{name}`")).collect(); + let mut err = format!("scene `{}` draws itself", scene.name); + if !through.is_empty() { + err = format!("{err} through {}", through.join(", ")); + } + return Err(Error::Invalid(err)); + } + path.push(&scene.name); + for item in &scene.items { + if let Some(Nested::Scene(nested)) = collection.nested(&item.source) + && let Some(nested) = collection.scene(nested) + { + visit(collection, nested, done, path)?; + } + } + path.pop(); + done.insert(&scene.name); + Ok(()) + } + + let mut done = BTreeSet::new(); + for scene in &self.scenes { + visit(self, scene, &mut done, &mut Vec::new())?; + } + + for item in &self.top { + let draws = match self.nested(&item.source) { + Some(Nested::Program) => true, + Some(Nested::Scene(scene)) => self.draws_program(scene), + None => false, + }; + if draws { + return Err(Error::Invalid(format!( + "the top layer places `{}`, which draws the program the top layer is drawn over", + item.source + ))); + } + } + Ok(()) + } + /// The filters `item` draws through: its source's, then its own. pub fn filters<'a>(&'a self, item: &'a Item) -> impl Iterator { let source = self.video.get(&item.source).map(|video| video.filters.as_slice()); @@ -293,6 +402,13 @@ impl Collection { } for (name, source) in &self.sources { source.validate(name)?; + if let Source::Scene { scene } = source + && self.scene(scene).is_none() + { + return Err(Error::Invalid(format!( + "`{name}` draws scene `{scene}`, which is not a scene" + ))); + } let Source::Clone { source } = source else { continue }; match self.sources.get(source) { Some(Source::Clone { .. }) => { @@ -323,6 +439,8 @@ impl Collection { self.validate_chain(&video.filters, &on)?; } + self.validate_nesting()?; + if let Some(name) = self.transition.stinger() && !matches!(self.sources.get(name), Some(Source::Media { .. })) { @@ -724,6 +842,15 @@ impl fmt::Display for Layer { } } +/// What a [`Source::Scene`] or [`Source::Program`] draws. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Nested<'a> { + /// The scene by this name. + Scene(&'a str), + /// The main canvas's program. + Program, +} + /// A source placed in a scene or the top layer. /// /// Every item naming the same source shows the same capture, with the @@ -1482,6 +1609,18 @@ pub enum Source { /// source, on demand through [`Feed::replay`](crate::source::Feed::replay). /// Transparent while nothing plays. Replay, + /// Another scene, drawn whole at the main canvas's size, as a picture. Its + /// items' keyframes play from when the scene drawing it goes on, as that + /// scene's own items' do. A scene cannot draw itself, even through other + /// scenes. + Scene { + /// The scene drawn. + scene: String, + }, + /// What the main canvas's program shows, transitions included, on the + /// same frame. A scene drawing it cannot go on program, and the top layer + /// cannot draw it, since either would draw the program into itself. + Program, } impl Source { @@ -1507,9 +1646,9 @@ impl Source { } /// Whether the source opens a capture, file, or subscription of its own, - /// rather than showing another source's. + /// rather than showing another source's or a scene. pub fn opens(&self) -> bool { - !matches!(self, Self::Clone { .. }) + !matches!(self, Self::Clone { .. } | Self::Scene { .. } | Self::Program) } /// Whether the source feeds the program audio. @@ -2422,6 +2561,137 @@ path = "sting.webm" assert!(matches!(collection.validate(), Err(Error::Invalid(_)))); } + /// `example()` with scenes `A`, `B`, and `C`, where `A` shows `B` through + /// the scene source `b`, and `on air` a program source placed nowhere. + fn nesting() -> Collection { + let mut collection = example(); + let fill = Rect::fill(&collection.canvas); + for (name, source) in [("a", "A"), ("b", "B"), ("c", "C")] { + let scene = Source::Scene { scene: source.into() }; + collection.sources.insert(name.into(), scene); + collection.scenes.push(Scene::new(source)); + } + collection.sources.insert("on air".into(), Source::Program); + collection.scenes[1].items.push(Item::new("b", fill)); + collection.validate().unwrap(); + collection + } + + /// The scene and program sources read as written. + #[test] + fn reads_a_scene_source() { + let text = "version = 1\n\n\ + [canvas]\nwidth = 1920\nheight = 1080\nframerate = 30\n\n\ + [sources.wide]\nkind = \"scene\"\nscene = \"Wide\"\n\n\ + [sources.\"on air\"]\nkind = \"program\"\n\n\ + [[scenes]]\nname = \"Wide\"\n"; + let collection = Collection::from_toml(text).unwrap(); + assert_eq!(collection.sources["wide"], Source::Scene { scene: "Wide".into() }); + assert_eq!(collection.sources["on air"], Source::Program); + assert_eq!( + Collection::from_toml(&collection.to_toml().unwrap()).unwrap(), + collection + ); + assert_eq!(collection.nested("wide"), Some(Nested::Scene("Wide"))); + } + + /// A scene source names a scene, and no scene draws itself, even through + /// others or a clone, shown or hidden. The top layer, drawn over the + /// program, cannot show it, and a mask cannot read a scene. + #[test] + fn refuses_a_scene_drawing_itself() { + let fill = Rect::fill(&Canvas::default()); + let refused = |edit: &dyn Fn(&mut Collection), wanted: &str| { + let mut collection = nesting(); + edit(&mut collection); + let err = collection.validate().unwrap_err().to_string(); + assert!(err.contains(wanted), "{err}"); + }; + refused( + &|collection| { + collection + .sources + .insert("gone".into(), Source::Scene { scene: "Gone".into() }); + }, + "`gone` draws scene `Gone`, which is not a scene", + ); + refused( + &|collection| { + collection.scenes[1].items.push(Item { + visible: false, + ..Item::new("a", fill) + }); + }, + "scene `A` draws itself", + ); + refused( + &|collection| { + collection + .sources + .insert("a copy".into(), Source::Clone { source: "a".into() }); + collection.scenes[2].items.push(Item::new("c", fill)); + collection.scenes[3].items.push(Item::new("a copy", fill)); + }, + "scene `A` draws itself through `B`, `C`", + ); + refused( + &|collection| collection.top.push(Item::new("on air", fill)), + "the top layer places `on air`, which draws the program", + ); + refused( + &|collection| { + collection.scenes[2].items.push(Item::new("on air", fill)); + collection.top.push(Item::new("a", fill)); + }, + "the top layer places `a`, which draws the program", + ); + refused( + &|collection| { + let mask = Filter::Mask { + source: "b".into(), + mode: MaskMode::Alpha, + }; + collection.scenes[0].items[0].filters.push(mask); + }, + "which is not a source with a picture of its own", + ); + + // Showing a scene is not drawing it: `B` and `C` both show `C`. + let mut collection = nesting(); + collection.scenes[2].items.push(Item::new("c", fill)); + collection.scenes[1].items.push(Item::new("c", fill)); + collection.top.push(Item::new("c", fill)); + collection.validate().unwrap(); + } + + /// What layers draw reaches through the scenes they show, each once, and + /// leaves hidden items out; whether a scene draws the program counts hidden + /// ones too, since showing one is an edit away. + #[test] + fn drawn_reaches_through_scenes() { + let fill = Rect::fill(&Canvas::default()); + let mut collection = nesting(); + collection.scenes[1].items.push(Item::new("b", fill)); + collection.scenes[2].items = vec![Item::new("logo", fill), Item::new("c", fill)]; + collection.scenes[3].items = vec![ + Item::new("background", fill), + Item { + visible: false, + ..Item::new("on air", fill) + }, + ]; + collection.validate().unwrap(); + let drawn: Vec<&str> = collection + .drawn([Layer::scene("A")]) + .into_iter() + .map(|item| item.source.as_str()) + .collect(); + assert_eq!(drawn, ["b", "b", "logo", "c", "background"]); + assert!(collection.draws_program("A")); + assert!(collection.draws_program("C")); + assert!(!collection.draws_program("Scene")); + } + /// A further canvas is named as a path segment, since it names its /// broadcast, is a size the encoder can take, and is the only canvas a /// scene lays items out on. diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 5ab516b4..a810ad1b 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -10,7 +10,7 @@ use moq_video::render::{Renderer, wgpu}; use crate::Error; use crate::collection::{ - Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Rect, Settled, Source, Transition, + Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Nested, Rect, Settled, Source, Transition, }; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; use crate::overlay::{Overlay, Overlays}; @@ -42,6 +42,14 @@ pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; /// them. An item's keyframes start when its scene goes on program, and /// separately when it goes into preview, for the preview, on every canvas /// alike. +/// +/// A scene a [`Source::Scene`] draws is drawn first, at the main canvas's +/// size, into a texture of its own that is transparent where it draws +/// nothing, once per render however many items show it, and only while one +/// that is drawn does. Its keyframes play from when the scene drawing it went +/// on, so where preview and program show it at different moments of them, it +/// is drawn once for each. A [`Source::Program`] shows the main canvas's +/// program texture itself, drawn before anything showing it. pub struct Compositor { device: wgpu::Device, queue: wgpu::Queue, @@ -51,6 +59,10 @@ pub struct Compositor { extras: BTreeMap, layers: HashMap, + /// The textures scenes drawn through scene sources are drawn into, by + /// scene: as many as the last render drew it at different moments, + /// reused by the next. + nested: HashMap>, /// Lays out and paints overlays, started with the first one. overlays: Option, preview_scene: Option, @@ -137,6 +149,7 @@ impl Switch { } /// A canvas-sized texture drawn into, and bound to sample. +#[derive(Clone)] struct Target { texture: wgpu::Texture, picture: Picture, @@ -189,6 +202,7 @@ impl Compositor { queue: queue.clone(), effects, layers: HashMap::new(), + nested: HashMap::new(), overlays: None, preview_scene: None, program_scene: None, @@ -357,12 +371,27 @@ impl Compositor { /// Put the preview on program at once, ending any transition, and /// stopping a stinger's clip in `sources`. - pub fn cut(&mut self, sources: &Sources) { + /// + /// Refused, changing nothing, if the preview scene draws the program in + /// `collection`, which it would draw into itself. + pub fn cut(&mut self, collection: &Collection, sources: &Sources) -> Result<(), Error> { + self.can_take(collection)?; if let Some(switch) = self.switch.take() { switch.stop(sources); } self.program_scene = self.preview_scene.clone(); self.program_start = None; + Ok(()) + } + + /// Refuse to take a preview scene that draws the program to program. + fn can_take(&self, collection: &Collection) -> Result<(), Error> { + match self.preview_scene.as_deref() { + Some(scene) if collection.draws_program(scene) => Err(Error::Invalid(format!( + "`{scene}` draws the program, so it cannot go on program" + ))), + _ => Ok(()), + } } /// Take the preview to program by `transition`, starting at `now`. @@ -374,7 +403,16 @@ impl Compositor { /// as it plays, and ends with the clip's play, leaving the clip holding its /// last picture, or stopping a looping one; a clip that cannot play, or /// fails, cuts at once instead. - pub fn transition(&mut self, transition: Transition, sources: &Sources, now: Instant) { + /// + /// Refused, changing nothing, as [`cut`](Self::cut) is. + pub fn transition( + &mut self, + collection: &Collection, + transition: Transition, + sources: &Sources, + now: Instant, + ) -> Result<(), Error> { + self.can_take(collection)?; if let Some(switch) = self.switch.take() { switch.stop(sources); } @@ -387,6 +425,7 @@ impl Compositor { transition, played: None, }); + Ok(()) } /// How far the transition under way has switched the scenes at `now`, @@ -446,6 +485,34 @@ impl Compositor { self.preview_shown = preview_since; let program_since = since(&mut self.program_start); + let preview = On(self.preview_scene.as_deref(), preview_since); + let program = On(self.program_scene.as_deref(), program_since); + let switch = self.switch.as_ref().zip(progress).map(|(switch, progress)| { + let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); + (switch, from, progress) + }); + + // Each canvas's scenes and top layer, and the scenes they draw. + let shown: Vec = std::iter::once(preview).chain(on_program(program, switch)).collect(); + let mut roots = Vec::new(); + for canvas in std::iter::once(None).chain(self.extras.keys().cloned().map(Some)) { + for On(scene, since) in &shown { + if let Some(scene) = scene { + let place = Place::scene(*scene).on(canvas.clone()); + roots.push((place, *since)); + } + } + roots.push((Place::top().on(canvas), Duration::ZERO)); + } + let nesting = Nesting::of(collection, roots); + let targets = nested_targets( + &mut self.nested, + &self.device, + &self.effects, + self.main.canvas, + &nesting, + ); + let mut frame = self.effects.frame(); let mut draw = Draw { collection, @@ -455,18 +522,28 @@ impl Compositor { canvas: None, size: self.main.canvas, prepared: HashMap::new(), + nesting: &nesting, + scenes: HashMap::new(), + program: None, }; - let preview = On(self.preview_scene.as_deref(), preview_since); - let program = On(self.program_scene.as_deref(), program_since); - let switch = self.switch.as_ref().zip(progress).map(|(switch, progress)| { - let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); - (switch, from, progress) - }); - let extras = self.extras.iter().map(|(name, output)| (Some(name.as_str()), output)); - for (canvas, output) in std::iter::once((None, &self.main)).chain(extras) { - draw.canvas = canvas; + // What the program shows is drawn first, then the program, then what + // shows it, then everything else. + let nests = nesting.order.iter().zip(&targets).enumerate(); + let (late, early): (Vec<_>, Vec<_>) = nests.partition(|(_, (nest, _))| nest.program); + for (index, (nest, target)) in early { + draw.nest(index, nest, target); + } + draw.draw_program(&self.main, program, switch); + draw.program = Some(self.main.program.picture.clone()); + for (index, (nest, target)) in late { + draw.nest(index, nest, target); + } + draw.draw_preview(&self.main, preview); + for (name, output) in &self.extras { + draw.canvas = Some(name); draw.size = output.canvas; - draw.output(output, preview, program, switch); + draw.draw_preview(output, preview); + draw.draw_program(output, program, switch); } self.effects.submit(frame); @@ -597,8 +674,8 @@ impl Compositor { } /// The sources whose pictures preview and program draw on any canvas: - /// each visible item's, a stinger's clip under way, and each mask's, - /// through clones. + /// each visible item's, through the scenes they draw, a stinger's clip + /// under way, and each mask's, through clones. fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { let leaving = self.switch.as_ref().and_then(|switch| switch.from.as_deref()); let scenes: Vec> = [self.preview_scene.as_deref(), self.program_scene.as_deref(), leaving] @@ -607,7 +684,14 @@ impl Compositor { .map(|name| Some(name.to_string())) .chain([None]) .collect(); - let canvases = std::iter::once(None).chain(self.extras.keys().map(Some)); + let canvases = std::iter::once(None).chain(self.extras.keys().cloned().map(Some)); + let layers = canvases.flat_map(|canvas| { + let scenes = scenes.iter().cloned(); + scenes.map(move |scene| Place { + canvas: canvas.clone(), + scene, + }) + }); let mut shown = HashSet::new(); let masks = |shown: &mut HashSet<&'a str>, filters: &mut dyn Iterator| { for filter in filters { @@ -616,18 +700,9 @@ impl Compositor { } } }; - for canvas in canvases { - for scene in &scenes { - let place = Place { - canvas: canvas.cloned(), - scene: scene.clone(), - }; - let items = collection.items(&place).unwrap_or_default(); - for item in items.iter().filter(|item| item.visible) { - shown.insert(collection.capture(&item.source)); - masks(&mut shown, &mut collection.filters(item)); - } - } + for item in collection.drawn(layers) { + shown.insert(collection.capture(&item.source)); + masks(&mut shown, &mut collection.filters(item)); } // Drawn through its source's filters alone. if let Some((name, _)) = self.stinger().and_then(|name| collection.sources.get_key_value(name)) { @@ -639,6 +714,31 @@ impl Compositor { } } +/// A texture of `canvas` for each scene `nesting` draws, reusing those in +/// `pool`, the last render's, and leaving there only those used now. +fn nested_targets( + pool: &mut HashMap>, + device: &wgpu::Device, + effects: &Effects, + canvas: Canvas, + nesting: &Nesting, +) -> Vec { + let mut free = std::mem::take(pool); + let mut targets = Vec::with_capacity(nesting.order.len()); + for nest in &nesting.order { + let reused = free.get_mut(nest.scene).and_then(Vec::pop); + let target = match reused { + Some(target) if (target.texture.width(), target.texture.height()) == (canvas.width, canvas.height) => { + target + } + _ => Target::new(device, effects, canvas, "moq-scene nested"), + }; + pool.entry(nest.scene.to_string()).or_default().push(target.clone()); + targets.push(target); + } + targets +} + /// Whether a stinger playing `feed`'s clip is past `cut` into it at `now`, or /// `None` once the clip's play has ended. The first call plays the clip from /// its start, noting which play it is in `played`. Fails when the clip cannot @@ -815,6 +915,119 @@ fn fits(device: &wgpu::Device, canvas: Canvas) -> Result<(), Error> { #[derive(Clone, Copy)] struct On<'a>(Option<&'a str>, Duration); +/// The scenes program draws: `program`, or those `switch` draws as it goes +/// to it. +fn on_program<'a>(program: On<'a>, switch: Option<(&Switch, On<'a>, f32)>) -> Vec> { + match switch { + // Under its clip, the program switches scenes at once. + Some((switch, from, progress)) if switch.transition.stinger().is_some() => { + vec![if progress < 1.0 { from } else { program }] + } + Some((_, from, _)) => vec![from, program], + None => vec![program], + } +} + +/// The scenes a render draws through scene sources, each at the moment of its +/// keyframes it is shown at. +struct Nesting<'a> { + /// In an order that draws each after the scenes it shows. + order: Vec>, + /// Which of `order` draws a scene shown in a scene that went on a while + /// ago, by the scene's name and that while. + drawn: HashMap<(&'a str, Duration), usize>, +} + +/// A scene drawn through a scene source. +struct Nest<'a> { + scene: &'a str, + /// How long ago the scene drawing it went on, which its keyframes play + /// from. + since: Duration, + /// `since`, or `None` once every item it draws, through the scenes it + /// draws, holds its last keyframe, which every later moment draws alike. + moment: Option, + /// Whether it shows the program, itself or through the scenes it draws. + program: bool, +} + +impl<'a> Nesting<'a> { + /// The scenes the items of each of `roots` draw, from when its scene went + /// on, and those they draw in turn. + fn of(collection: &'a Collection, roots: impl IntoIterator) -> Self { + let mut nesting = Self { + order: Vec::new(), + drawn: HashMap::new(), + }; + let mut walked = HashMap::new(); + for (place, since) in roots { + let items = collection.items(&place).unwrap_or_default(); + nesting.walk(collection, items, since, &mut walked); + } + nesting + } + + /// Add the scenes `items` show, from when the scene they are in went on + /// `since` ago. Returns whether any shows the program. `walked` holds each + /// scene and moment added, or being walked. + fn walk( + &mut self, + collection: &'a Collection, + items: &'a [Item], + since: Duration, + walked: &mut HashMap<(&'a str, Option), Option>, + ) -> bool { + let mut program = false; + for item in items.iter().filter(|item| item.visible) { + let scene = match collection.nested(&item.source) { + Some(Nested::Scene(scene)) => scene, + Some(Nested::Program) => { + program = true; + continue; + } + None => continue, + }; + if let Some(&index) = self.drawn.get(&(scene, since)) { + program |= self.order[index].program; + continue; + } + let moment = moment(collection, scene, since); + let index = match walked.get(&(scene, moment)) { + Some(Some(index)) => *index, + // A scene drawing itself, which validation refuses: it shows + // nothing there rather than recursing forever. + Some(None) => continue, + None => { + walked.insert((scene, moment), None); + let items = collection.scene(scene).map_or(&[][..], |scene| scene.items.as_slice()); + let shows = self.walk(collection, items, since, walked); + self.order.push(Nest { + scene, + since, + moment, + program: shows, + }); + let index = self.order.len() - 1; + walked.insert((scene, moment), Some(index)); + index + } + }; + self.drawn.insert((scene, since), index); + program |= self.order[index].program; + } + program + } +} + +/// The moment of its keyframes the scene called `scene` is drawn at, `since` +/// the scene drawing it went on, or `None` once every item it draws, through +/// the scenes it draws, holds its last keyframe. +fn moment(collection: &Collection, scene: &str, since: Duration) -> Option { + let items = collection.drawn([Place::scene(scene)]); + let moving = items.iter().any(|item| item.settled(since) != Settled::After); + moving.then_some(since) +} + /// One render's draws onto its targets. struct Draw<'a> { collection: &'a Collection, @@ -829,20 +1042,37 @@ struct Draw<'a> { /// where it is, on which canvas, and its [`Item::settled`] time, as two /// outputs showing its scene can be at different points of its keyframes. prepared: HashMap<(Place, usize, Settled), Option>, + /// The scenes drawn through scene sources this render. + nesting: &'a Nesting<'a>, + /// The picture of each of `nesting`'s scenes drawn so far, by its place + /// in the order. + scenes: HashMap, + /// The main canvas's program, once drawn. + program: Option, } impl Draw<'_> { - /// Draw `output`'s preview, showing `preview`, and its program, showing - /// `program`, or `switch` to it from the scene it leaves, `progress` of - /// the way there. - fn output(&mut self, output: &Output, preview: On, program: On, switch: Option<(&Switch, On, f32)>) { - let top = self.place(None); + /// Draw the scene `nest` names, the `index`th of the nesting, into + /// `target`, transparent where it draws nothing, for items to show. + fn nest(&mut self, index: usize, nest: &Nest, target: &Target) { + self.frame.target(&target.picture, self.size, wgpu::Color::TRANSPARENT); + self.layer(target, Place::scene(nest.scene), nest.since); + self.scenes.insert(index, target.picture.clone()); + } + + /// Draw `output`'s preview, showing `preview` under the top layer. + fn draw_preview(&mut self, output: &Output, preview: On) { // The top layer goes over both outputs, and over a transition rather // than through it, so it stays on across the switch. self.target(&output.preview); self.scene(&output.preview, preview); - self.layer(&output.preview, top.clone(), Duration::ZERO); + self.layer(&output.preview, self.place(None), Duration::ZERO); + } + /// Draw `output`'s program, showing `program`, or `switch` to it from the + /// scene it leaves, `progress` of the way there, under the top layer. + fn draw_program(&mut self, output: &Output, program: On, switch: Option<(&Switch, On, f32)>) { + let top = self.place(None); let to = program; match switch { Some((switch, from, progress)) => { @@ -901,9 +1131,9 @@ impl Draw<'_> { } } - /// Start drawing `target`. + /// Start drawing `target`, a canvas, opaque black where nothing is drawn. fn target(&mut self, target: &Target) { - self.frame.target(&target.picture, self.size); + self.frame.target(&target.picture, self.size, wgpu::Color::BLACK); } /// Draw `on`'s items onto `target`. A missing scene draws nothing. @@ -934,7 +1164,27 @@ impl Draw<'_> { fn layer(&mut self, target: &Target, place: Place, since: Duration) { let items = self.collection.items(&place).unwrap_or_default(); for (index, item) in items.iter().enumerate().filter(|(_, item)| item.visible) { - self.item(target, (place.clone(), index, item.settled(since)), &item.posed(since)); + let key = (place.clone(), index, self.settled(item, since)); + self.item(target, key, &item.posed(since), since); + } + } + + /// Where `since` falls among `item`'s keyframes, as [`Item::settled`], but + /// at that moment itself while a scene it shows moves, so a draw of it is + /// shared only with draws showing the scene alike. + fn settled(&self, item: &Item, since: Duration) -> Settled { + let moving = match self.collection.nested(&item.source) { + Some(Nested::Scene(scene)) => self + .nesting + .drawn + .get(&(scene, since)) + .is_some_and(|&index| self.nesting.order[index].moment.is_some()), + _ => false, + }; + if moving { + Settled::At(since) + } else { + item.settled(since) } } @@ -952,16 +1202,16 @@ impl Draw<'_> { ..Item::new(source, Rect::fill(&canvas)) }; let pose = Pose::of(self.collection, &item); - self.posed(target, &item, pose); + self.posed(target, &item, pose, Duration::ZERO); } - /// Draw `item`, posed, where it is placed onto `target`, shared with every - /// other draw of it under `key`. - fn item(&mut self, target: &Target, key: (Place, usize, Settled), item: &Item) { + /// Draw `item`, posed `since` its scene went on, where it is placed onto + /// `target`, shared with every other draw of it under `key`. + fn item(&mut self, target: &Target, key: (Place, usize, Settled), item: &Item, since: Duration) { let prepared = match self.prepared.get(&key) { Some(prepared) => prepared.clone(), None => { - let prepared = self.prepare(item, Pose::of(self.collection, item)); + let prepared = self.prepare(item, Pose::of(self.collection, item), since); self.prepared.insert(key, prepared.clone()); prepared } @@ -971,12 +1221,13 @@ impl Draw<'_> { } } - /// Draw `item` onto `target` in `pose` rather than where it is placed. - fn posed(&mut self, target: &Target, item: &Item, pose: Pose) { + /// Draw `item` onto `target` in `pose` rather than where it is placed, + /// `since` its scene went on. + fn posed(&mut self, target: &Target, item: &Item, pose: Pose, since: Duration) { if pose.opacity <= 0.0 { return; } - if let Some(prepared) = self.prepare(item, pose) { + if let Some(prepared) = self.prepare(item, pose, since) { self.effects.draw(self.frame, &target.texture, &prepared); } } @@ -1029,7 +1280,7 @@ impl Draw<'_> { (Some(at), None) => { let pose = Pose::of(collection, &leaving[at]); let opacity = 1.0 - progress; - self.posed(target, &leaving[at], Pose { opacity, ..pose }); + self.posed(target, &leaving[at], Pose { opacity, ..pose }, from.1); } (None, Some(index)) => { let pose = Pose::of(collection, &entering[index]); @@ -1040,6 +1291,7 @@ impl Draw<'_> { opacity: progress, ..pose }, + to.1, ); } (Some(at), Some(index)) => { @@ -1049,17 +1301,18 @@ impl Draw<'_> { if old.filters == new.filters && old.blend == new.blend { if pose == own { // Shared with the preview when it draws the same. - let key = (place.clone(), index, resting[index].settled(to.1)); - self.item(target, key, new); + let key = (place.clone(), index, self.settled(&resting[index], to.1)); + self.item(target, key, new, to.1); } else { - self.posed(target, new, pose); + self.posed(target, new, pose, to.1); } } else if let Some(filters) = mixed(old, new, progress) { let item = Item { filters, ..new.clone() }; - self.posed(target, &item, pose); + self.posed(target, &item, pose, to.1); } else { // A look that cannot be mixed changes as the stacking does. - self.posed(target, if progress < 0.5 { old } else { new }, pose); + let (item, since) = if progress < 0.5 { (old, from.1) } else { (new, to.1) }; + self.posed(target, item, pose, since); } } (None, None) => unreachable!("a slot holds an item"), @@ -1067,9 +1320,11 @@ impl Draw<'_> { } } - fn prepare(&mut self, item: &Item, pose: Pose) -> Option { + /// `item` in `pose`, `since` its scene went on, ready to draw: `None` + /// while it shows nothing yet. + fn prepare(&mut self, item: &Item, pose: Pose, since: Duration) -> Option { let collection = self.collection; - let picture = self.library.0.get(collection.capture(&item.source))?.picture.as_ref()?; + let picture = &self.shows(item, since)?; let fade = Filter::Opacity { amount: pose.opacity }; let mut filters: Vec<&Filter> = collection.filters(item).collect(); if pose.opacity < 1.0 { @@ -1086,6 +1341,25 @@ impl Draw<'_> { } } +impl Draw<'_> { + /// The picture `item` shows, `since` its scene went on: its source's + /// newest, the scene it draws as drawn this render, or the main canvas's + /// program once drawn. + fn shows(&self, item: &Item, since: Duration) -> Option { + match self.collection.nested(&item.source) { + Some(Nested::Scene(scene)) => { + let index = self.nesting.drawn.get(&(scene, since))?; + self.scenes.get(index).cloned() + } + Some(Nested::Program) => self.program.clone(), + None => { + let layer = self.library.0.get(self.collection.capture(&item.source))?; + layer.picture.clone() + } + } + } +} + /// Premultiply tightly packed RGBA in place. fn premultiply(pixels: &mut [u8]) { for pixel in pixels.as_chunks_mut::<4>().0 { @@ -1342,7 +1616,7 @@ mod tests { local(async { let mut studio = Studio::new(collection()).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let program = studio.program(); assert_eq!(pixel(&program, 1, 1), RED); @@ -1372,7 +1646,7 @@ mod tests { collection.scenes[0].items.truncate(1); let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let actual = pixel(&studio.program(), 1, 1); assert!(near(actual, [128, 0, 0, 255]), "{actual:?}"); @@ -1389,9 +1663,12 @@ mod tests { let mut studio = Studio::new(collection()).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); assert_eq!(studio.compositor.program_scene(), Some("blue")); studio.render(start + Duration::from_millis(500)); @@ -1442,7 +1719,7 @@ mod tests { let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let program = studio.program(); let actual = pixel(&program, 1, 6); @@ -1463,12 +1740,15 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 1, 6), BLUE); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); let program = studio.program(); let actual = pixel(&program, 1, 1); @@ -1516,14 +1796,17 @@ mod tests { let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 1, 1), RED); assert_eq!(tall(&mut studio), (BLUE, RED)); assert_eq!(studio.compositor.layers.len(), 2, "each source is uploaded once"); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); let (fading, kept) = tall(&mut studio); assert!(near(fading, [0, 0, 128, 255]), "{fading:?}"); @@ -1553,10 +1836,13 @@ mod tests { assert_eq!(pixel(&studio.program(), 1, 1), BLACK); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); assert!(!studio.compositor.transitioning()); studio.render(start + Duration::from_millis(600)); assert_eq!(pixel(&studio.program(), 1, 1), RED); @@ -1615,9 +1901,12 @@ mod tests { ] { let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(transition.clone(), &studio.sources, start); + studio + .compositor + .transition(&studio.collection, transition.clone(), &studio.sources, start) + .unwrap(); studio.render(start + Duration::from_millis(500)); let program = studio.program(); let actual = [(1, 1), (1, 6), (6, 1), (6, 6)].map(|(x, y)| pixel(&program, x, y)); @@ -1663,12 +1952,18 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); studio .compositor - .transition(Transition::Move { duration }, &studio.sources, start); + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + start, + ) + .unwrap(); // Eased halfway, blue is at (2, 2) and both fades are at half. studio.render(start + Duration::from_millis(500)); @@ -1748,7 +2043,7 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); studio.render(at(0)); @@ -1756,7 +2051,7 @@ mod tests { assert_eq!(tall(&mut studio, true), (BLUE, BLACK), "halfway in, in preview"); assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the main canvas too"); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(at(500)); assert_eq!(tall(&mut studio, false), (BLACK, BLACK), "on program from the start"); assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "the main canvas too"); @@ -1782,7 +2077,7 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); studio.render(at(0)); @@ -1797,7 +2092,7 @@ mod tests { assert_eq!(pixel(&preview, 1, 6), BLUE, "halfway in"); assert_eq!(pixel(&preview, 6, 6), BLACK); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(at(500)); assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "on program from the start"); assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the preview carries on"); @@ -1836,7 +2131,7 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); // Program has reached the jump; preview starts over before it. studio.compositor.set_preview(Some("blue".into())); @@ -1860,11 +2155,14 @@ mod tests { let start = Instant::now(); let at = |millis| start + Duration::from_millis(millis); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("red".into())); studio.render(at(0)); - studio.compositor.transition(FADE, &studio.sources, at(9000)); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, at(9000)) + .unwrap(); studio.render(at(9500)); let program = studio.program(); // Red has slid halfway in over the left half, blue is in place, and @@ -1895,12 +2193,18 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); studio .compositor - .transition(Transition::Move { duration }, &studio.sources, start); + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + start, + ) + .unwrap(); studio.render(start); let program = studio.program(); @@ -1927,7 +2231,7 @@ mod tests { local(async { let mut studio = Studio::new(collection()).await; studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); assert!(studio.sources.get("blue").unwrap().take().is_none(), "drawn once"); studio.render(Instant::now()); @@ -1972,7 +2276,7 @@ mod tests { collection.validate().unwrap(); let mut studio = Studio::with_files(collection, dir).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); studio } @@ -2001,7 +2305,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); assert_eq!(studio.compositor.program_scene(), Some("blue")); // What the clip's filters read is drawn too, such as an overlay to paint. @@ -2069,7 +2374,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(studio.compositor.transitioning()); assert_eq!(pixel(&studio.program(), 1, 1), BLUE); @@ -2087,7 +2393,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(!studio.compositor.transitioning()); assert_eq!(pixel(&studio.program(), 1, 1), BLUE); @@ -2096,7 +2403,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(studio.compositor.transitioning()); studio.sources.get("sting").unwrap().play().unwrap(); @@ -2122,7 +2430,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); loop { studio.render(Instant::now()); let until = at + Duration::from_millis(10); @@ -2130,7 +2439,7 @@ mod tests { break; } } - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); assert!(!studio.compositor.transitioning()); let feed = studio.sources.get("sting").unwrap(); while feed.status() != Status::Ended { @@ -2145,7 +2454,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); studio.render(Instant::now()); assert!(studio.compositor.transitioning()); studio.sources.get("sting").unwrap().stop().unwrap(); @@ -2166,7 +2476,8 @@ mod tests { let transition = studio.collection.transition.clone(); studio .compositor - .transition(transition, &studio.sources, Instant::now()); + .transition(&studio.collection, transition, &studio.sources, Instant::now()) + .unwrap(); while studio.compositor.transitioning() { studio.render(Instant::now()); let until = at + Duration::from_millis(10); @@ -2247,7 +2558,7 @@ mod tests { async fn drawn(collection: Collection, dir: tempfile::TempDir) -> Vec { let mut studio = Studio::with_files(collection, dir).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); studio.program() } @@ -2705,7 +3016,7 @@ mod tests { ); let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); let old = studio.compositor.program().clone(); @@ -2777,9 +3088,12 @@ mod tests { let mut studio = Studio::new(collection()).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.transition(FADE, &studio.sources, start); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); // `blue` takes the name `red` gives up, which is followed once. let mut draft = studio.collection.draft("Rename"); @@ -2865,12 +3179,12 @@ mod tests { collection.scenes[1].items = vec![Item::new("overlay", Rect::fill(&collection.canvas))]; let mut studio = Studio::new(collection).await; studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); // Opened here, but drawn nowhere until the cut. let start = Instant::now(); studio.render(start); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); for (at, expected) in [(500, [128, 0, 128, 255]), (750, [64, 0, 191, 255]), (2000, BLUE)] { studio.render(start + Duration::from_millis(at)); @@ -2912,12 +3226,18 @@ mod tests { let mut studio = Studio::new(collection).await; let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.compositor.set_preview(Some("blue".into())); let duration = Duration::from_secs(1); studio .compositor - .transition(Transition::Move { duration }, &studio.sources, start); + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + start, + ) + .unwrap(); studio.render(start + Duration::from_millis(500)); let program = studio.program(); @@ -2953,11 +3273,11 @@ mod tests { // Paint the aside overlay, then leave it off air. studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 3, 3), [255, 255, 0, 255]); studio.compositor.set_preview(Some("blue".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); studio.compositor.overlays.as_ref().unwrap().panic(); @@ -2968,12 +3288,202 @@ mod tests { } studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(start); assert_eq!(pixel(&studio.program(), 3, 3), RED, "no frozen overlay"); }) .await; } + + /// `collection()` with a scene source drawing each of its scenes, named + /// after it with ` scene`, and a program source, `on air`, and `scene` + /// added showing `items`. + fn nesting(items: Vec) -> Collection { + let mut collection = collection(); + for scene in ["red", "blue"] { + let source = Source::Scene { scene: scene.into() }; + collection.sources.insert(format!("{scene} scene"), source); + } + collection.sources.insert("on air".into(), Source::Program); + collection.scenes.push(Scene { + items, + ..Scene::new("scene") + }); + collection.validate().unwrap(); + collection + } + + /// A scene source draws its scene's items, scaled and cropped like any + /// picture, and transparent where the scene draws nothing. It is drawn once + /// however many items show it, and not at all once nothing drawn does. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_scene_source_draws_its_scene() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![ + Item::new("blue", fill), + // The red scene, red with blue over its top-right quarter. + Item::new("red scene", CORNER), + Item { + visible: false, + ..Item::new("red scene", fill) + }, + ]); + // Blue over the top-right quarter alone, transparent elsewhere. + collection.scenes.push(Scene { + items: vec![collection.scenes[0].items[1].clone()], + ..Scene::new("corner") + }); + let corner = Source::Scene { scene: "corner".into() }; + collection.sources.insert("corner scene".into(), corner); + collection.scenes[1].items = vec![Item::new("red", fill), Item::new("corner scene", fill)]; + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let program = studio.program(); + assert_eq!(pixel(&program, 0, 7), RED, "the red scene's red"); + assert_eq!(pixel(&program, 3, 4), BLUE, "the red scene's corner"); + assert_eq!(pixel(&program, 6, 6), BLUE, "under it"); + assert_eq!(studio.compositor.nested.get("red").map(Vec::len), Some(1)); + + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let program = studio.program(); + assert_eq!(pixel(&program, 1, 1), RED, "through the nested scene"); + assert_eq!(pixel(&program, 6, 1), BLUE); + let drawn: Vec<&String> = studio.compositor.nested.keys().collect(); + assert_eq!(drawn, ["corner"], "only what is drawn"); + }) + .await; + } + + /// A program source shows what the program draws on the same frame, + /// a transition under way included, and a scene showing it cannot go on + /// program. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_program_source_shows_the_program_on_the_same_frame() { + local(async { + let mut studio = Studio::new(nesting(vec![Item::new("on air", CORNER)])).await; + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.compositor.set_preview(Some("blue".into())); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); + studio.compositor.set_preview(Some("scene".into())); + + studio.render(start + Duration::from_millis(500)); + let faded = pixel(&studio.program(), 1, 1); + assert!(near(faded, [128, 0, 127, 255]), "{faded:?}"); + let preview = studio.preview(); + assert_eq!(pixel(&preview, 1, 5), faded, "the program as it fades"); + assert_eq!(pixel(&preview, 3, 4), BLUE, "blue under blue"); + assert_eq!(pixel(&preview, 6, 1), BLACK); + + let err = studio.compositor.cut(&studio.collection, &studio.sources).unwrap_err(); + assert!(err.to_string().contains("draws the program"), "{err}"); + let err = studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap_err(); + assert!(err.to_string().contains("draws the program"), "{err}"); + assert_eq!(studio.compositor.program_scene(), Some("blue")); + assert!(studio.compositor.transitioning(), "the fade carries on"); + }) + .await; + } + + /// A nested scene's keyframes play from when the scene showing it went + /// on, so preview and program can show it at two moments, each drawn once, + /// and once both settle, drawn once for both. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_nested_scene_plays_its_keyframes_from_when_its_scene_goes_on() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("blue scene", fill)]); + collection.scenes[1].items[0].keyframes = slide_in(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(at(0)); + assert_eq!(pixel(&studio.program(), 1, 1), BLACK, "still off the canvas"); + studio.render(at(500)); + let program = studio.program(); + assert_eq!( + (pixel(&program, 3, 1), pixel(&program, 4, 1)), + (BLUE, BLACK), + "halfway in" + ); + + // Into preview again, so its keyframes start over there. + studio.compositor.set_preview(Some("scene".into())); + studio.render(at(500)); + studio.render(at(750)); + let program = studio.program(); + assert_eq!((pixel(&program, 5, 1), pixel(&program, 6, 1)), (BLUE, BLACK)); + let preview = studio.preview(); + assert_eq!((pixel(&preview, 1, 1), pixel(&preview, 2, 1)), (BLUE, BLACK)); + assert_eq!(studio.compositor.nested["blue"].len(), 2, "one per moment"); + + studio.render(at(2000)); + assert_eq!(pixel(&studio.program(), 7, 1), BLUE); + assert_eq!(pixel(&studio.preview(), 7, 1), BLUE); + assert_eq!(studio.compositor.nested["blue"].len(), 1, "settled alike"); + }) + .await; + } + + /// A filter reading around each pixel, such as a blur, reads the nested + /// scene drawn this render, not the last one. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_blur_reads_the_nested_scene_drawn_now() { + local(async { + let fill = Rect::fill(&collection().canvas); + let blurred = Item { + filters: vec![Filter::Blur { radius: 0.5 }], + ..Item::new("blue scene", fill) + }; + let mut studio = Studio::new(nesting(vec![blurred])).await; + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let actual = pixel(&studio.program(), 4, 4); + assert!(near(actual, BLUE), "{actual:?}"); + }) + .await; + } + + /// A blend mode in a nested scene mixes with what the scene draws under it, + /// and where it draws nothing, leaves the picture as it is, as CSS does. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_blend_mode_over_nothing_in_a_nested_scene_draws_the_picture() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("red", fill), Item::new("blue scene", fill)]); + collection.scenes[1].items[0].blend = Blend::Multiply; + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + }) + .await; + } } /// What drawing more canvases costs, against how many sources the collection @@ -3031,11 +3541,20 @@ mod bench { /// The mean time to draw and finish every canvas once. async fn frame(collection: Collection) -> Duration { + let scene = collection.scenes[0].name.clone(); + cost(collection, &scene).await.0 + } + + /// The mean time to draw and finish every canvas once with the first scene + /// on program and `preview` in preview, and the render passes and texture + /// copies each frame runs. + async fn cost(collection: Collection, preview: &str) -> (Duration, (usize, usize)) { let mut studio = Studio::new(collection).await; studio.compositor.set_canvases(&studio.collection).unwrap(); let scene = studio.collection.scenes[0].name.clone(); studio.compositor.set_preview(Some(scene)); - studio.compositor.cut(&studio.sources); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.compositor.set_preview(Some(preview.to_string())); let mut total = Duration::ZERO; for frame in 0..FRAMES * 2 { let start = Instant::now(); @@ -3045,7 +3564,53 @@ mod bench { total += Instant::now().saturating_duration_since(start); } } - total / FRAMES + (total / FRAMES, studio.compositor.effects.ran) + } + + /// What a scene source costs: every source placed in the scene on program + /// and preview (direct), the same items in a scene it shows through a + /// scene source (nested), and the program shown again in preview through a + /// program source (program). + #[tokio::test] + #[ignore = "a benchmark; run by hand on a GPU"] + async fn nested_scenes() { + local(async { + println!("sources case frame passes copies"); + for sources in [4, 16, 64] { + let direct = collection(sources, 1, sources); + let scene = direct.scenes[0].name.clone(); + + let mut nested = direct.clone(); + let fill = Rect::fill(&nested.canvas); + let inner = std::mem::take(&mut nested.scenes[0].items); + nested.scenes.push(crate::Scene { + items: inner, + ..crate::Scene::new("inner") + }); + let inner = Source::Scene { scene: "inner".into() }; + nested.sources.insert("inner".into(), inner); + nested.scenes[0].items = vec![Item::new("inner", fill)]; + nested.validate().unwrap(); + + let mut program = direct.clone(); + program.sources.insert("on air".into(), Source::Program); + program.scenes.push(crate::Scene { + items: vec![Item::new("on air", fill)], + ..crate::Scene::new("multiview") + }); + program.validate().unwrap(); + + for (case, collection, preview) in [ + ("direct", direct, scene.as_str()), + ("nested", nested, scene.as_str()), + ("program", program, "multiview"), + ] { + let (frame, (passes, copies)) = cost(collection, preview).await; + println!("{sources:>7} {case:<8} {frame:>7.2?} {passes:>6} {copies:>6}"); + } + } + }) + .await; } /// Sources against canvases, each further canvas placing every source diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index 91e18123..f0812d0b 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -582,14 +582,32 @@ impl Draft { }) } - /// Remove the scene at `index`, with its items. + /// Remove the scene at `index`, with its items, and every scene source + /// drawing it, as [`remove_source`](Self::remove_source) removes one. pub fn remove_scene(&mut self, index: usize) -> Result<(), Error> { + let name = self.scene_at(index)?.name; + for source in self.drawing(&name) { + self.remove_source(&source)?; + } + self.take_scene(index).map(drop) + } + + /// Take the scene at `index` out, leaving what refers to it. + fn take_scene(&mut self, index: usize) -> Result { let before = self.scene_at(index)?; self.change(Change::Scene { index, - before: Some(before), + before: Some(before.clone()), after: None, - }) + })?; + Ok(before) + } + + /// The scene sources drawing the scene called `scene`. + fn drawing(&self, scene: &str) -> Vec { + let sources = self.collection.sources.iter(); + let drawing = sources.filter(|(_, source)| matches!(source, Source::Scene { scene: drawn } if drawn == scene)); + drawing.map(|(name, _)| name.clone()).collect() } /// Replace the scene at `index`. @@ -602,19 +620,24 @@ impl Draft { }) } - /// Rename the scene at `index`, keeping what it shows. + /// Rename the scene at `index`, keeping what it shows, and every scene + /// source drawing it. pub fn rename_scene(&mut self, index: usize, name: impl Into) -> Result<(), Error> { + let before = self.scene_at(index)?; let scene = Scene { name: name.into(), - ..self.scene_at(index)? + ..before.clone() }; + for source in self.drawing(&before.name) { + let scene = scene.name.clone(); + self.set_source(&source, Some(Source::Scene { scene }))?; + } self.set_scene(index, scene) } /// Move the scene at `from` to `to`, counted after its removal. pub fn move_scene(&mut self, from: usize, to: usize) -> Result<(), Error> { - let scene = self.scene_at(from)?; - self.remove_scene(from)?; + let scene = self.take_scene(from)?; self.insert_scene(to, scene) } @@ -1146,6 +1169,40 @@ mod tests { assert_eq!(collection, before); } + /// A scene source follows its scene through a rename and a move, and goes + /// with it, with the items showing it, when it is removed, all undone whole. + #[test] + fn a_scene_source_follows_its_scene() { + let mut collection = collection(); + let fill = Rect::fill(&collection.canvas); + collection.scenes.push(Scene { + items: vec![Item::new("red", fill)], + ..Scene::new("Inner") + }); + let inner = Source::Scene { scene: "Inner".into() }; + collection.sources.insert("inner".into(), inner); + collection.scenes[0].items.push(Item::new("inner", fill)); + collection.validate().unwrap(); + let mut history = History::new(); + + let mut draft = collection.draft("Rename `Inner`"); + draft.rename_scene(1, "Nested").unwrap(); + draft.move_scene(1, 0).unwrap(); + commit(&mut collection, &mut history, draft, None); + assert_eq!(collection.sources["inner"], Source::Scene { scene: "Nested".into() }); + assert_eq!(collection.scenes[0].name, "Nested"); + let renamed = collection.clone(); + + let mut draft = collection.draft("Remove `Nested`"); + draft.remove_scene(0).unwrap(); + commit(&mut collection, &mut history, draft, None); + assert!(!collection.sources.contains_key("inner")); + assert_eq!(collection.scenes[0].items.len(), 1, "the item showing it went too"); + + history.undo(&mut collection).unwrap(); + assert_eq!(collection, renamed); + } + /// A removed source's filters go with it, from items and other sources /// alike, leaving their other filters. #[test] diff --git a/rs/moq-scene/src/effect.rs b/rs/moq-scene/src/effect.rs index b2f010eb..d345da9c 100644 --- a/rs/moq-scene/src/effect.rs +++ b/rs/moq-scene/src/effect.rs @@ -405,23 +405,30 @@ enum Command { Backdrop { texture: wgpu::Texture, area: Area }, } -/// One render's passes: the rasters items are filtered through, then the -/// targets, in the order they are drawn. +/// One render's passes: each target's after the rasters its items are +/// filtered through, and the targets in the order they are drawn, so an item +/// can show a target drawn before its own. #[derive(Default)] pub(crate) struct Frame { + /// The passes of the targets finished, in order. + done: Vec, + /// The rasters of the target drawn now. rasters: Vec, + /// The target drawn now: its passes, and the copies a blend mode makes. targets: Vec, /// The canvas of the target drawn now, which items are prepared for. canvas: Canvas, } impl Frame { - /// Start drawing a target the size of `canvas`, cleared to opaque black. - pub fn target(&mut self, target: &Picture, canvas: Canvas) { + /// Start drawing a target the size of `canvas`, cleared to `clear`. + pub fn target(&mut self, target: &Picture, canvas: Canvas, clear: wgpu::Color) { + self.done.append(&mut self.rasters); + self.done.append(&mut self.targets); self.canvas = canvas; self.targets.push(Command::Pass { view: target.view.clone(), - clear: Some(wgpu::Color::BLACK), + clear: Some(clear), draws: Vec::new(), }); } @@ -492,6 +499,11 @@ pub(crate) struct Effects { free: Vec, /// Rasters used by this render. used: Vec, + + /// The render passes and texture copies the last submit ran, for the + /// benchmarks. + #[cfg(test)] + pub(crate) ran: (usize, usize), } /// What a blend mode mixes with, `width` by `height`. @@ -656,6 +668,8 @@ impl Effects { backdrop, free: Vec::new(), used: Vec::new(), + #[cfg(test)] + ran: (0, 0), } } @@ -738,7 +752,22 @@ impl Effects { /// Run `frame`'s passes. pub fn submit(&mut self, frame: Frame) { - let commands: Vec = frame.rasters.into_iter().chain(frame.targets).collect(); + let Frame { + mut done, + rasters, + targets, + .. + } = frame; + done.extend(rasters.into_iter().chain(targets)); + let commands = done; + #[cfg(test)] + { + let copies = commands + .iter() + .filter(|command| matches!(command, Command::Backdrop { .. })) + .count(); + self.ran = (commands.len() - copies, copies); + } let draws = commands.iter().flat_map(|command| match command { Command::Pass { draws, .. } => draws.as_slice(), Command::Backdrop { .. } => &[], diff --git a/rs/moq-scene/src/effect.wgsl b/rs/moq-scene/src/effect.wgsl index 3afcab6a..0a7f58f4 100644 --- a/rs/moq-scene/src/effect.wgsl +++ b/rs/moq-scene/src/effect.wgsl @@ -306,13 +306,24 @@ fn mix_blend(backdrop: vec3, source: vec3, mode: u32) -> vec3 { } } -// The input through the ops, mixed with the opaque backdrop in `aux` by the -// draw's blend mode: (1 - αs) Cb + αs B(Cb, Cs). +// The input through the ops, mixed with the premultiplied backdrop in `aux` +// by the draw's blend mode, as the Compositing and Blending spec composites +// over a backdrop that may be transparent, as a nested scene's is: +// Cs' = (1 - αb) Cs + αb B(Cb, Cs), then Cs' over the backdrop. Over an opaque +// backdrop that is (1 - αs) Cb + αs B(Cb, Cs). @fragment fn fs_blend(in: Varyings) -> @location(0) vec4 { let source = apply(straight(in.input), in.item); - let backdrop = textureLoad(aux, vec2(in.position.xy), 0).rgb; - let mixed = mix_blend(backdrop, source.rgb, draw.mode.z); - return vec4(mix(backdrop, mixed, source.a), 1.0); + let backdrop = textureLoad(aux, vec2(in.position.xy), 0); + var straight_backdrop = vec3(0.0); + if backdrop.a > 0.0 { + straight_backdrop = backdrop.rgb / backdrop.a; + } + let mixed = mix_blend(straight_backdrop, source.rgb, draw.mode.z); + let blended = mix(source.rgb, mixed, backdrop.a); + return vec4( + source.a * blended + (1.0 - source.a) * backdrop.rgb, + source.a + (1.0 - source.a) * backdrop.a, + ); } // One direction of a separable Gaussian blur of a premultiplied raster. With diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index f71e24a5..2e2e8c68 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -20,7 +20,8 @@ //! textures on the caller's `wgpu` device, one pair per canvas from one //! upload of each source's picture, cutting or running a [`Transition`] //! between them on every canvas together, and plays each item's -//! [`Keyframe`]s from when its scene went on. `moq_video::render::export` hands the +//! [`Keyframe`]s from when its scene went on. A scene source shows another +//! scene, drawn first into a texture of its own, or the program itself. `moq_video::render::export` hands the //! program to an encoder, on the GPU where the platform allows, and a //! [`Scaler`] draws it smaller for a smaller rendition. //! - [`publish`] gives each local source its own track, in a broadcast the @@ -49,7 +50,7 @@ pub mod source; pub use clip::Clip; pub use collection::{ Audio, Blend, Canvas, Collection, Crop, Direction, Ease, Extra, Filter, Item, Keyframe, Layer, MaskMode, Mixes, - Rect, Rgba, Scene, Source, Transition, Video, + Nested, Rect, Rgba, Scene, Source, Transition, Video, }; pub use compositor::{Compositor, FORMAT}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 98748c6d..62bfe993 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -710,7 +710,9 @@ impl Driver { let plays = self.plays.clone().expect("a replay source has plays"); self.replays(plays, listing).await } - Source::Clone { .. } => unreachable!("a clone opens no feed"), + Source::Clone { .. } | Source::Scene { .. } | Source::Program => { + unreachable!("a clone or scene opens no feed") + } device => self.device(device, listing).await, } } @@ -757,7 +759,9 @@ impl Driver { | Source::Overlay { .. } | Source::Clone { .. } | Source::Lut { .. } - | Source::Replay => unreachable!("not a device source"), + | Source::Replay + | Source::Scene { .. } + | Source::Program => unreachable!("not a device source"), }; let mut config = moq_video::capture::Config::default(); @@ -1821,8 +1825,8 @@ fn open(name: &str, source: &Source, context: &Context, held: bool) -> (Feed, Dr /// A broadcast is already published by its own publisher, and a clone shows /// its source's track, its filters carried by the layout rather than encoded /// again. Web pages and overlays are seen only in the program, since H.264 -/// would encode their transparency over black, and a LUT is looked up, never -/// shown. +/// would encode their transparency over black, a LUT is looked up, never +/// shown, and a scene is drawn from other sources. fn publishes(source: &Source) -> bool { match source { Source::Camera { .. } @@ -1838,7 +1842,9 @@ fn publishes(source: &Source) -> bool { | Source::Browser { .. } | Source::Overlay { .. } | Source::Clone { .. } - | Source::Lut { .. } => false, + | Source::Lut { .. } + | Source::Scene { .. } + | Source::Program => false, } } diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index fc42fb78..2b54430d 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -185,8 +185,8 @@ impl State { /// the one drawn, once the compositor takes its canvas. The scenes in /// preview and on program follow the renames and removals `made` holds, /// each edit read against the collection it was made to, and any - /// `collection` lacks is dropped. Edits that remove a scene on air are - /// refused, changing nothing. + /// `collection` lacks is dropped. Edits that remove a scene on air, or + /// leave one drawing the program, are refused, changing nothing. fn adopt(&mut self, collection: Collection, made: &[Edit]) -> anyhow::Result<()> { let mut before = self.collection.clone(); let mut renames = Vec::with_capacity(made.len()); @@ -194,7 +194,7 @@ impl State { renames.push(before.renames(edit)); before.apply(edit)?; } - keep_on_air(self.compositor.on_air(), &renames)?; + keep_on_air(self.compositor.on_air(), &renames, &collection)?; self.compositor.set_canvases(&collection)?; for renames in &renames { self.compositor.rename(renames); @@ -221,26 +221,38 @@ impl State { /// The sources whose pictures the scenes on air place on any canvas's /// program, through clones: each visible item's of the scenes on air, /// including one leaving in a transition, and of the top layers over them, - /// and each mask's it reads. + /// and of the scenes they draw, and each mask's it reads. fn placed(&self) -> BTreeSet { let collection = &self.collection; let scenes: Vec> = self.compositor.on_air().map(Some).chain([None]).collect(); let canvases = std::iter::once(None).chain(collection.canvases.keys().map(Some)); - let items = canvases - .flat_map(|canvas| { - scenes.iter().map(move |scene| moq_scene::Layer { - canvas: canvas.cloned(), - scene: scene.map(str::to_string), - }) + let layers = canvases.flat_map(|canvas| { + scenes.iter().map(move |scene| moq_scene::Layer { + canvas: canvas.cloned(), + scene: scene.map(str::to_string), }) - .flat_map(|layer| collection.items(&layer).unwrap_or_default()) - .filter(|item| item.visible); + }); + let items = collection.drawn(layers); items + .into_iter() .flat_map(|item| std::iter::once(&item.source).chain(masks(collection.filters(item)))) .map(|source| collection.capture(source).to_string()) .collect() } + /// Put the preview on program at once; see [`Compositor::cut`]. + pub fn cut(&mut self) -> anyhow::Result<()> { + Ok(self.compositor.cut(&self.collection, &self.sources)?) + } + + /// Take the preview to program by `transition`, starting at `now`; see + /// [`Compositor::transition`]. + pub fn transition(&mut self, transition: moq_scene::Transition, now: Instant) -> anyhow::Result<()> { + Ok(self + .compositor + .transition(&self.collection, transition, &self.sources, now)?) + } + /// Restart the source called `name`, as an edit would, such as to reload a /// web page. pub fn restart(&mut self, name: &str) { @@ -290,11 +302,14 @@ impl autosave::Document for State { } /// Refuse edits, each read as its [`Collection::renames`], that remove a -/// scene `on_air` program draws, so a slip cannot put black on air. A reload -/// that leaves one out still drops it, having nothing to show instead. +/// scene `on_air` program draws, so a slip cannot put black on air, or that +/// leave one drawing the program in `collection`, which it would draw into +/// itself. A reload that leaves one out still drops it, having nothing to +/// show instead. fn keep_on_air<'a>( on_air: impl IntoIterator, renames: &'a [BTreeMap>], + collection: &Collection, ) -> anyhow::Result<()> { for mut scene in on_air { for renames in renames { @@ -304,6 +319,10 @@ fn keep_on_air<'a>( None => {} } } + anyhow::ensure!( + !collection.draws_program(scene), + "`{scene}` is on air, so it cannot draw the program, which it would draw into itself" + ); } Ok(()) } @@ -773,8 +792,13 @@ impl Engine { }); let mut compositor = Compositor::new(&gpu.device, &gpu.queue, collection.canvas)?; compositor.set_canvases(&collection)?; - compositor.set_preview(collection.scenes.first().map(|scene| scene.name.clone())); - compositor.cut(&sources); + // The first scene that can go on program starts there, and in preview. + let first = collection + .scenes + .iter() + .find(|scene| !collection.draws_program(&scene.name)); + compositor.set_preview(first.map(|scene| scene.name.clone())); + compositor.cut(&collection, &sources)?; let engine = Arc::new(Self { state: Mutex::new(State { @@ -2843,9 +2867,7 @@ mod tests { let mut state = engine.state.lock().unwrap(); let transition = state.collection().transition.clone(); let state = &mut *state; - state - .compositor - .transition(transition, &state.sources, kio::clock::now()); + state.transition(transition, kio::clock::now()).unwrap(); let shown = state.shown(); assert!(shown.contains("sting") && shown.contains("matte"), "{shown:?}"); } @@ -2914,16 +2936,36 @@ mod tests { collection.renames(&draft.finish().unwrap()) }; let removed = [remove(&collection, 0)]; - assert!(keep_on_air([live.as_str()], &removed).is_err()); - assert!(keep_on_air(["Renamed", live.as_str()], &removed).is_err(), "leaving"); - keep_on_air([live.as_str()], &[remove(&collection, 1)]).unwrap(); - keep_on_air([], &removed).unwrap(); + assert!(keep_on_air([live.as_str()], &removed, &collection).is_err()); + assert!( + keep_on_air(["Renamed", live.as_str()], &removed, &collection).is_err(), + "leaving" + ); + keep_on_air([live.as_str()], &[remove(&collection, 1)], &collection).unwrap(); + keep_on_air([], &removed, &collection).unwrap(); // Undoing both edits, newest first, removes `Other` once renamed. let undo = [collection.renames(&rename.inverse()), added.renames(&add.inverse())]; - let err = keep_on_air(["Renamed"], &undo).unwrap_err(); + let err = keep_on_air(["Renamed"], &undo, &collection).unwrap_err(); assert!(err.to_string().contains("`Other` is on air"), "{err}"); - keep_on_air([live.as_str()], &undo).unwrap(); + keep_on_air([live.as_str()], &undo, &collection).unwrap(); + + // Nor does one leave a scene on air drawing the program, which only a + // scene off air may. + let mut draft = collection.draft("Show the program"); + draft.set_source("on air", Some(moq_scene::Source::Program)).unwrap(); + let fill = moq_scene::Rect::fill(&collection.canvas); + let item = moq_scene::Item::new("on air", fill); + draft + .insert_item(&moq_scene::Layer::scene(live.clone()), 0, item) + .unwrap(); + let edit = draft.finish().unwrap(); + let mut drawing = collection.clone(); + drawing.apply(&edit).unwrap(); + let renames = [collection.renames(&edit)]; + let err = keep_on_air([live.as_str()], &renames, &drawing).unwrap_err(); + assert!(err.to_string().contains("cannot draw the program"), "{err}"); + keep_on_air(["Renamed"], &renames, &drawing).unwrap(); } async fn subscribe( diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 6a498e1a..5573a4ed 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -1008,17 +1008,16 @@ impl Panels { let mut gesture = None; ui.horizontal(|ui| { if ui.button("Cut").clicked() { - let state = &mut *state; - state.compositor.cut(&state.sources); + let cut = state.cut(); + self.check(cut); } if ui .button("Transition") .on_hover_text("Take the preview to program by the transition set beside it") .clicked() { - let state = &mut *state; - let now = kio::clock::Instant::now(); - state.compositor.transition(transition.clone(), &state.sources, now); + let taken = state.transition(transition.clone(), kio::clock::Instant::now()); + self.check(taken); } match &next { Transition::Slide { direction, .. } | Transition::Wipe { direction, .. } => self.direction = *direction, @@ -1482,6 +1481,8 @@ impl Panels { .on_hover_text(health::without(stall)); } (None, Some(feed), None) => status(ui, &feed.status()), + // A scene is drawn from other sources, so it runs nothing. + (None, None, None) if collection.nested(name).is_some() => status(ui, &Status::Live), (None, None, None) => status(ui, &Status::Starting), } // A track can fail while its source still draws, such as on a GPU @@ -1576,7 +1577,13 @@ impl Panels { }); ui.separator(); - if let Some(source) = self.add.ui(ui) { + let scenes: Vec = state + .collection() + .scenes + .iter() + .map(|scene| scene.name.clone()) + .collect(); + if let Some(source) = self.add.ui(ui, &scenes) { let name = self.add.name.trim().to_string(); let layer = self.layer(&state).ok(); let result = add(&mut state, layer, name, source, None); @@ -3601,6 +3608,8 @@ fn describe(source: &Source) -> String { Source::Clone { source } => format!("clone of {source}"), Source::Replay => "replay".into(), Source::Lut { path } => format!("LUT: {}", path.display()), + Source::Scene { scene } => format!("scene: {scene}"), + Source::Program => "program".into(), } } @@ -3661,6 +3670,8 @@ struct AddSource { /// on program. looping: bool, restart: bool, + /// The scene a scene source draws, or `None` for the program. + scene: Option, } impl Default for AddSource { @@ -3675,6 +3686,7 @@ impl Default for AddSource { height: canvas.height, looping: false, restart: true, + scene: None, } } } @@ -3695,10 +3707,11 @@ enum Kind { Overlay, Replay, Lut, + Scene, } impl Kind { - const ALL: [Self; 13] = [ + const ALL: [Self; 14] = [ Self::Color, Self::Image, Self::Media, @@ -3712,6 +3725,7 @@ impl Kind { Self::Overlay, Self::Replay, Self::Lut, + Self::Scene, ]; fn label(self) -> &'static str { @@ -3729,13 +3743,14 @@ impl Kind { Self::Overlay => "Overlay", Self::Replay => "Replay", Self::Lut => "LUT", + Self::Scene => "Scene", } } /// What the value field holds, if this kind takes one. fn value(self) -> Option<&'static str> { match self { - Self::Color | Self::SystemAudio | Self::Replay => None, + Self::Color | Self::SystemAudio | Self::Replay | Self::Scene => None, Self::Image | Self::Media => Some("file"), Self::Lut => Some(".cube file"), Self::Broadcast => Some("path"), @@ -3748,8 +3763,9 @@ impl Kind { } impl AddSource { - /// Draw the form; the source to add once the user asks for it. - fn ui(&mut self, ui: &mut Ui) -> Option { + /// Draw the form, offering `scenes` to a scene source; the source to add + /// once the user asks for it. + fn ui(&mut self, ui: &mut Ui, scenes: &[String]) -> Option { let mut add = false; egui::Grid::new("add source").show(ui, |ui| { ui.label("Name"); @@ -3787,6 +3803,23 @@ impl AddSource { egui::color_picker::color_edit_button_srgba(ui, &mut self.color, egui::color_picker::Alpha::OnlyBlend); ui.end_row(); } + if self.kind == Kind::Scene { + // A pick whose scene was removed or renamed away names no scene. + if self.scene.as_ref().is_some_and(|scene| !scenes.contains(scene)) { + self.scene = None; + } + ui.label("Draws"); + egui::ComboBox::from_id_salt("scene") + .selected_text(self.scene.as_deref().unwrap_or("Program")) + .show_ui(ui, |ui| { + ui.selectable_value(&mut self.scene, None, "Program") + .on_hover_text("Whatever is on air, transitions included"); + for scene in scenes { + ui.selectable_value(&mut self.scene, Some(scene.clone()), scene); + } + }); + ui.end_row(); + } if self.kind == Kind::Media { ui.label("Playback"); ui.horizontal(|ui| { @@ -3834,6 +3867,10 @@ impl AddSource { height: self.height, }, Kind::Replay => Source::Replay, + Kind::Scene => match &self.scene { + Some(scene) => Source::Scene { scene: scene.clone() }, + None => Source::Program, + }, }) } } diff --git a/rs/moq-studio/tests/browser.rs b/rs/moq-studio/tests/browser.rs index 29effe42..711710fa 100644 --- a/rs/moq-studio/tests/browser.rs +++ b/rs/moq-studio/tests/browser.rs @@ -87,7 +87,7 @@ impl Studio { } let mut compositor = Compositor::new(&device, &queue, collection.canvas).unwrap(); compositor.set_preview(Some("Main".into())); - compositor.cut(&sources); + compositor.cut(&collection, &sources).unwrap(); Self { collection, sources, From fab326eec8d92095495a29bcce84fed3458487a5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 11:09:36 -0800 Subject: [PATCH 3/8] fix(scene): a copied scene source passes the clipboard's own check Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/clip.rs | 58 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index 67878827..9e3f96cd 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -102,11 +102,26 @@ impl Clip { match &clip { // The rules a collection holding just the clip would break, such as // an item showing a source the clip lacks. It has whatever mixes the - // clip's sources are left out of, which pasting checks. + // clip's sources are left out of, and an empty scene for each its + // scene sources draw, which pasting checks. Self::Items(items) => { let excluded = items.audio.values().flat_map(|audio| &audio.exclude); let mut names: BTreeSet = excluded.cloned().collect(); names.insert(Mixes::DEFAULT.into()); + let drawn: BTreeSet<&str> = items + .sources + .values() + .filter_map(|source| match source { + Source::Scene { scene } => Some(scene.as_str()), + _ => None, + }) + .collect(); + // Named apart from them, so it draws none of them by name. + let clip = std::iter::once("clip".to_string()) + .chain((2..).map(|n| format!("clip {n}"))) + .find(|name| !drawn.contains(name.as_str())) + .expect("an unused name"); + let drawn = drawn.into_iter().map(Scene::new); Collection { sources: items.sources.clone(), mixes: Mixes { @@ -118,10 +133,12 @@ impl Clip { video: items.video.clone(), // In a scene, where keyframes play; pasting into the top // layer checks it holds none. - scenes: vec![Scene { + scenes: std::iter::once(Scene { items: items.items.clone(), - ..Scene::new("clip") - }], + ..Scene::new(clip) + }) + .chain(drawn) + .collect(), ..Collection::new() } .validate()? @@ -290,6 +307,39 @@ mod tests { pasted } + /// An item showing a scene pastes linked within its collection, but not + /// into one without that scene, nor into the scene it shows. + #[test] + fn a_scene_source_pastes_where_its_scene_is() { + let mut collection = collection(); + let fill = Rect::fill(&collection.canvas); + collection.scenes.push(Scene::new("clip")); + collection.scenes.push(Scene::new("Other")); + let shown = Source::Scene { scene: "clip".into() }; + collection.sources.insert("nested".into(), shown); + collection.scenes[1].items.push(Item::new("cam", fill)); + collection.scenes[2].items.push(Item::new("nested", fill)); + collection.validate().unwrap(); + let clip = Clip::items(&collection, collection.scenes[2].items.clone(), Path::new(DIR)); + let text = clip.to_toml().unwrap(); + + let pasted = paste(&collection, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.sources, collection.sources, "linked"); + + let Clip::Items(items) = Clip::from_toml(&text).unwrap() else { + panic!("not items") + }; + let refused = |collection: &Collection, layer: &Layer| { + let mut draft = collection.draft("Paste"); + draft.paste(layer, &items, Path::new(DIR)).unwrap(); + draft.finish().unwrap_err().to_string() + }; + let err = refused(&collection, &Layer::scene("clip")); + assert!(err.contains("scene `clip` draws itself"), "{err}"); + let err = refused(&Collection::new(), &Layer::scene("Scene")); + assert!(err.contains("draws scene `clip`, which is not a scene"), "{err}"); + } + /// Within a collection, a pasted item shows the very source copied. #[test] fn pastes_linked_within_a_collection() { From 4434d94498adc5810f4ddbe2aec81e40a94e97fb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 12:55:49 -0800 Subject: [PATCH 4/8] fix(scene): scene sources follow their scene by identity through a draft A draft that renamed a scene to a name another still had, then renamed or removed that other, retargeted or removed the first scene's sources too. The draft now follows each scene by place, as Collection::renames does, and each scene source by the scene it draws. Also: the top layer refuses a scene with keyframes, which it never starts; Replay no longer offers scene and program sources; removing a scene clears the item selection its cascade can shift. Co-Authored-By: Claude Opus 5.5 --- README.md | 5 +- rs/moq-scene/src/collection.rs | 58 +++++++--- rs/moq-scene/src/edit.rs | 197 +++++++++++++++++++++++++-------- rs/moq-studio/src/ui.rs | 17 ++- 4 files changed, 213 insertions(+), 64 deletions(-) diff --git a/README.md b/README.md index 0601dc32..a5d9b950 100644 --- a/README.md +++ b/README.md @@ -145,8 +145,9 @@ cropped. A nested scene is drawn at the main canvas's size, transparent where it draws nothing, once per frame however many items show it, and only while something drawn shows it. Its items' keyframes play from when the scene showing it goes -on, as that scene's own items' do, and a media file in it set to restart plays -from its start when the scene showing it goes on program. +on, as that scene's own items' do, so the top layer, which never goes on, +cannot show a scene with keyframes. A media file in a nested scene set to +restart plays from its start when the scene showing it goes on program. A scene cannot draw itself, even through other scenes, so an edit that would is refused. A scene showing the program cannot go on program, nor can the main diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 27d6f793..af83d1a6 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -226,21 +226,30 @@ impl Collection { /// a scene it draws, shown or hidden. It then cannot go on program, which /// it would draw into itself. pub fn draws_program(&self, scene: &str) -> bool { + let items = self.reached(scene); + items + .iter() + .any(|item| self.nested(&item.source) == Some(Nested::Program)) + } + + /// Every item of the scene called `scene` and of the scenes it draws, + /// shown or hidden, each scene once. + fn reached(&self, scene: &str) -> Vec<&Item> { let mut scenes = vec![scene]; let mut seen = BTreeSet::new(); + let mut items = Vec::new(); while let Some(scene) = scenes.pop() { let Some(scene) = self.scene(scene).filter(|scene| seen.insert(&scene.name)) else { continue; }; for item in &scene.items { - match self.nested(&item.source) { - Some(Nested::Program) => return true, - Some(Nested::Scene(scene)) => scenes.push(scene), - None => {} + if let Some(Nested::Scene(scene)) = self.nested(&item.source) { + scenes.push(scene); } + items.push(item); } } - false + items } /// Refuse a scene that draws itself, directly or through other scenes, @@ -377,10 +386,20 @@ impl Collection { .validate() .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; self.validate_chain(&item.filters, &on)?; - if layer.scene.is_none() && !item.keyframes.is_empty() { - return Err(Error::Invalid(format!( - "{on} has keyframes, but the top layer never goes on to start them" - ))); + if layer.scene.is_none() { + if !item.keyframes.is_empty() { + return Err(Error::Invalid(format!( + "{on} has keyframes, but the top layer never goes on to start them" + ))); + } + if let Some(Nested::Scene(scene)) = self.nested(&item.source) + && let Some(keyed) = self.reached(scene).into_iter().find(|item| !item.keyframes.is_empty()) + { + return Err(Error::Invalid(format!( + "{on} draws `{}` with keyframes, but the top layer never goes on to start them", + keyed.source + ))); + } } if !item.keyframes.is_sorted_by_key(|keyframe| keyframe.at) { return Err(Error::Invalid(format!("{on} has keyframes out of order"))); @@ -878,7 +897,8 @@ pub struct Item { /// How the item moves away from and back to where it rests, from when its /// scene goes on, in order of time. It plays once: before the first /// keyframe the item holds that one, and after the last it holds the last. - /// A top-layer item has none, since the top layer never goes on. + /// A top-layer item has none, nor does a scene it draws, since the top + /// layer never goes on. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub keyframes: Vec, } @@ -1611,8 +1631,8 @@ pub enum Source { Replay, /// Another scene, drawn whole at the main canvas's size, as a picture. Its /// items' keyframes play from when the scene drawing it goes on, as that - /// scene's own items' do. A scene cannot draw itself, even through other - /// scenes. + /// scene's own items' do, so the top layer cannot draw one with keyframes. + /// A scene cannot draw itself, even through other scenes. Scene { /// The scene drawn. scene: String, @@ -2597,7 +2617,8 @@ path = "sting.webm" /// A scene source names a scene, and no scene draws itself, even through /// others or a clone, shown or hidden. The top layer, drawn over the - /// program, cannot show it, and a mask cannot read a scene. + /// program, cannot show it, nor a scene with keyframes, which it never + /// starts, and a mask cannot read a scene. #[test] fn refuses_a_scene_drawing_itself() { let fill = Rect::fill(&Canvas::default()); @@ -2645,6 +2666,17 @@ path = "sting.webm" }, "the top layer places `a`, which draws the program", ); + refused( + &|collection| { + let source = collection.scenes[0].items[0].source.clone(); + collection.scenes[2].items.push(Item { + keyframes: vec![Keyframe::new(Duration::ZERO)], + ..Item::new(source, fill) + }); + collection.top.push(Item::new("a", fill)); + }, + "with keyframes, but the top layer never goes on to start them", + ); refused( &|collection| { let mask = Filter::Mask { diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index f0812d0b..d182e5bf 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -402,50 +402,105 @@ impl Collection { /// is the same one moved. Whatever follows a scene by name, such as the one /// in preview, looks its name up here once. pub fn renames(&self, edit: &Edit) -> BTreeMap> { - // Each place's scene: its name here, or `None` for one the edit added. - let mut scenes: Vec<(Option<&str>, &str)> = self - .scenes - .iter() - .map(|scene| (Some(scene.name.as_str()), scene.name.as_str())) - .collect(); - // Scenes the edit removed, with what each was, until one is put back. - let mut removed: Vec<(Option<&str>, &Scene)> = Vec::new(); + let mut places = Places::new(self); for change in &edit.changes { - let Change::Scene { index, before, after } = change else { - continue; - }; - match (before, after) { - (None, Some(after)) if *index <= scenes.len() => { - let back = removed.iter().position(|(_, scene)| *scene == after); - let was = back.and_then(|back| removed.remove(back).0); - scenes.insert(*index, (was, &after.name)); - } - (Some(before), None) if *index < scenes.len() => { - let (was, _) = scenes.remove(*index); - removed.push((was, before)); - } - (Some(_), Some(after)) if *index < scenes.len() => scenes[*index].1 = &after.name, - _ => {} - } + places.change(change); } - let kept = scenes - .into_iter() - .filter_map(|(was, now)| Some((was?, Some(now)))) - .filter(|(was, now)| Some(*was) != *now); - let gone = removed.into_iter().filter_map(|(was, _)| Some((was?, None))); - kept.chain(gone) - .map(|(was, now)| (was.to_string(), now.map(str::to_string))) + let now = self + .scenes + .iter() + .enumerate() + .map(|(id, scene)| (scene, places.name(id))); + now.filter(|(scene, now)| *now != Some(scene.name.as_str())) + .map(|(scene, now)| (scene.name.clone(), now.map(str::to_string))) .collect() } /// Start an edit labelled `label` against this collection. pub fn draft(&self, label: impl Into) -> Draft { + let places = Places::new(self); + let drawing = self.sources.iter().filter_map(|(name, source)| match source { + Source::Scene { scene } => Some((name.clone(), places.id(scene)?)), + _ => None, + }); Draft { collection: self.clone(), edit: Edit { label: label.into(), changes: Vec::new(), }, + drawing: drawing.collect(), + places, + } + } +} + +/// Which scene each place of a collection's scene list holds through a run of +/// changes, by an id that follows it through renames and moves: its index in +/// the list the run started from, or one past those for a scene the run adds. +struct Places { + /// The id and name of the scene at each place. + scenes: Vec<(usize, String)>, + /// Scenes taken out, until one is put back unchanged, which is then the + /// same scene moved. + removed: Vec<(usize, Scene)>, + /// The id the next scene added takes. + next: usize, +} + +impl Places { + fn new(collection: &Collection) -> Self { + let scenes: Vec<_> = collection + .scenes + .iter() + .map(|scene| scene.name.clone()) + .enumerate() + .collect(); + Self { + next: scenes.len(), + scenes, + removed: Vec::new(), + } + } + + /// Follow `change`, which leaves the places alone unless it is a scene's. + fn change(&mut self, change: &Change) { + let Change::Scene { index, before, after } = change else { + return; + }; + match (before, after) { + (None, Some(after)) if *index <= self.scenes.len() => { + let back = self.removed.iter().position(|(_, scene)| scene == after); + let id = match back { + Some(back) => self.removed.remove(back).0, + None => { + self.next += 1; + self.next - 1 + } + }; + self.scenes.insert(*index, (id, after.name.clone())); + } + (Some(before), None) if *index < self.scenes.len() => { + let (id, _) = self.scenes.remove(*index); + self.removed.push((id, before.clone())); + } + (Some(_), Some(after)) if *index < self.scenes.len() => self.scenes[*index].1 = after.name.clone(), + _ => {} + } + } + + /// The name of scene `id`, or `None` once it is removed. + fn name(&self, id: usize) -> Option<&str> { + let scene = self.scenes.iter().find(|(at, _)| *at == id); + scene.map(|(_, name)| name.as_str()) + } + + /// The id of the scene called `name`, unless no scene or more than one is. + fn id(&self, name: &str) -> Option { + let mut called = self.scenes.iter().filter(|(_, at)| at == name); + match (called.next(), called.next()) { + (Some((id, _)), None) => Some(*id), + _ => None, } } } @@ -455,6 +510,11 @@ impl Collection { pub struct Draft { collection: Collection, edit: Edit, + places: Places, + /// The scene each scene source draws, by its id in `places`, so renaming + /// or removing one reaches its own sources even while another scene has + /// the name it had or takes. + drawing: BTreeMap, } impl Draft { @@ -466,6 +526,23 @@ impl Draft { /// Make `change` and record it. pub fn change(&mut self, change: Change) -> Result<(), Error> { change.apply(&mut self.collection)?; + self.places.change(&change); + if let Change::Source { name, after, .. } = &change { + // A source still naming the scene it follows keeps following it; + // otherwise it follows the one scene by the name it gives. + let kept = + |id: &usize| matches!(after, Some(Source::Scene { scene }) if self.places.name(*id) == Some(scene)); + if !self.drawing.get(name).is_some_and(kept) { + let id = match after { + Some(Source::Scene { scene }) => self.places.id(scene), + _ => None, + }; + match id { + Some(id) => self.drawing.insert(name.clone(), id), + None => self.drawing.remove(name), + }; + } + } self.edit.changes.push(change); Ok(()) } @@ -585,8 +662,8 @@ impl Draft { /// Remove the scene at `index`, with its items, and every scene source /// drawing it, as [`remove_source`](Self::remove_source) removes one. pub fn remove_scene(&mut self, index: usize) -> Result<(), Error> { - let name = self.scene_at(index)?.name; - for source in self.drawing(&name) { + self.scene_at(index)?; + for source in self.drawing(index) { self.remove_source(&source)?; } self.take_scene(index).map(drop) @@ -603,10 +680,10 @@ impl Draft { Ok(before) } - /// The scene sources drawing the scene called `scene`. - fn drawing(&self, scene: &str) -> Vec { - let sources = self.collection.sources.iter(); - let drawing = sources.filter(|(_, source)| matches!(source, Source::Scene { scene: drawn } if drawn == scene)); + /// The scene sources drawing the scene at `index`. + fn drawing(&self, index: usize) -> Vec { + let id = self.places.scenes[index].0; + let drawing = self.drawing.iter().filter(|(_, drawn)| **drawn == id); drawing.map(|(name, _)| name.clone()).collect() } @@ -624,15 +701,19 @@ impl Draft { /// source drawing it. pub fn rename_scene(&mut self, index: usize, name: impl Into) -> Result<(), Error> { let before = self.scene_at(index)?; - let scene = Scene { - name: name.into(), - ..before.clone() - }; - for source in self.drawing(&before.name) { - let scene = scene.name.clone(); + let name = name.into(); + self.set_scene( + index, + Scene { + name: name.clone(), + ..before + }, + )?; + for source in self.drawing(index) { + let scene = name.clone(); self.set_source(&source, Some(Source::Scene { scene }))?; } - self.set_scene(index, scene) + Ok(()) } /// Move the scene at `from` to `to`, counted after its removal. @@ -1296,11 +1377,19 @@ mod tests { /// A rename keeps the scene's items and place, a move keeps its name, and /// each scene maps to its own new name, even through names the edit - /// frees and reuses. + /// frees and reuses, as the scene sources drawing it follow it. #[test] fn renames_and_reorders_scenes() { let mut collection = collection(); collection.scenes.push(Scene::new("Other")); + for (source, scene) in [("first", "Scene"), ("second", "Other")] { + let scene = scene.to_string(); + collection.sources.insert(source.into(), Source::Scene { scene }); + } + let draws = |collection: &Collection, source: &str| match collection.sources.get(source) { + Some(Source::Scene { scene }) => Some(scene.clone()), + _ => None, + }; let start = collection.clone(); let mut history = History::new(); @@ -1315,6 +1404,8 @@ mod tests { history.record(edit, None); assert_eq!(collection.scenes[0].name, "Other"); assert_eq!(collection.scenes[0].items, start.scenes[0].items); + assert_eq!(draws(&collection, "first").as_deref(), Some("Other")); + assert_eq!(draws(&collection, "second").as_deref(), Some("Scene")); let mut draft = collection.draft("Move `Other` down"); draft.move_scene(0, 1).unwrap(); @@ -1342,6 +1433,8 @@ mod tests { "an added scene renames nothing" ); collection.apply(&edit).unwrap(); + assert_eq!(draws(&collection, "first").as_deref(), Some("Other")); + assert_eq!(draws(&collection, "second").as_deref(), Some("Next")); assert_eq!( collection.renames(&edit.inverse()), renamed(&[("Other", Some("Scene")), ("Next", Some("Other")), ("New", None)]), @@ -1357,6 +1450,9 @@ mod tests { collection.renames(&edit), renamed(&[("Other", None), ("Next", Some("Other"))]) ); + collection.apply(&edit).unwrap(); + assert_eq!(draws(&collection, "first"), None, "it went with its scene"); + assert_eq!(draws(&collection, "second").as_deref(), Some("Other")); // Names stay unique and non-empty. for name in ["Other", ""] { @@ -1364,6 +1460,17 @@ mod tests { draft.rename_scene(0, name).unwrap(); assert!(matches!(draft.finish(), Err(Error::Invalid(_))), "{name:?}"); } + + // Removing a scene whose name another just took removes only its own + // scene sources. + let scene = "New".to_string(); + collection.sources.insert("third".into(), Source::Scene { scene }); + let mut draft = collection.draft("Take over"); + draft.rename_scene(0, "Other").unwrap(); + draft.remove_scene(1).unwrap(); + collection.apply(&draft.finish().unwrap()).unwrap(); + assert_eq!(draws(&collection, "second"), None); + assert_eq!(draws(&collection, "third").as_deref(), Some("Other")); } /// The canvas and the transition are edits like any other: a drag of one diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index b22cb7cd..5743c41b 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -1226,9 +1226,17 @@ impl Panels { SceneAction::Move(from, to) => edit(&mut state, format!("Reorder scene `{name}`"), None, |draft| { draft.move_scene(from, to) }), - SceneAction::Remove(index) => edit(&mut state, format!("Remove scene `{name}`"), None, |draft| { - draft.remove_scene(index) - }), + SceneAction::Remove(index) => { + let result = edit(&mut state, format!("Remove scene `{name}`"), None, |draft| { + draft.remove_scene(index) + }); + if result.is_ok() { + // Removing its scene sources removes the items showing + // them, which moves the ones after them. + self.deselect_item(); + } + result + } }; drop(state); self.check(result); @@ -2490,7 +2498,7 @@ impl Panels { }; // What a source shows or plays, a clone its source's, has a past // only if something encodes it: not an image, a color, a web page, - // an overlay, or a LUT. + // an overlay, a LUT, or a scene or the program drawn into another. let collection = state.collection(); let replayable = collection .sources @@ -2503,6 +2511,7 @@ impl Panels { Source::Image { .. } | Source::Color { .. } | Source::Browser { .. } | Source::Overlay { .. } | Source::Lut { .. } + | Source::Scene { .. } | Source::Program ) }) }) From 381ec214d9af6206cca199ff78c73d2ff2c6fd22 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 07:27:33 -0800 Subject: [PATCH 5/8] fix(scene): removing a scene takes it out before its sources' items Removing a scene's sources finds the items showing them by scene name, so a scene that took the removed one's name in the same draft lost its own. Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/edit.rs | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index d182e5bf..04ca97a0 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -663,10 +663,14 @@ impl Draft { /// drawing it, as [`remove_source`](Self::remove_source) removes one. pub fn remove_scene(&mut self, index: usize) -> Result<(), Error> { self.scene_at(index)?; - for source in self.drawing(index) { + let drawing = self.drawing(index); + // Taken first, so the items its sources' removal finds by scene name + // are another scene's, even one that took its name in this draft. + self.take_scene(index)?; + for source in drawing { self.remove_source(&source)?; } - self.take_scene(index).map(drop) + Ok(()) } /// Take the scene at `index` out, leaving what refers to it. @@ -1462,15 +1466,18 @@ mod tests { } // Removing a scene whose name another just took removes only its own - // scene sources. + // scene sources, and the items showing them from the other. let scene = "New".to_string(); collection.sources.insert("third".into(), Source::Scene { scene }); + let fill = Rect::fill(&collection.canvas); + collection.scenes[1].items.push(Item::new("third", fill)); let mut draft = collection.draft("Take over"); - draft.rename_scene(0, "Other").unwrap(); - draft.remove_scene(1).unwrap(); + draft.rename_scene(1, "New").unwrap(); + draft.remove_scene(0).unwrap(); collection.apply(&draft.finish().unwrap()).unwrap(); - assert_eq!(draws(&collection, "second"), None); - assert_eq!(draws(&collection, "third").as_deref(), Some("Other")); + assert_eq!(draws(&collection, "third"), None); + assert_eq!(draws(&collection, "second").as_deref(), Some("New")); + assert!(collection.scenes[0].items.is_empty()); } /// The canvas and the transition are edits like any other: a drag of one From 243627ef7ecef6216edffbe8e478acd35a00dfbc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 07:37:17 -0800 Subject: [PATCH 6/8] fix(scene): walk nested scenes without recursion, and a move's nested picture as its stacking Validation and the compositor walked chained scenes on the call stack, so a long enough chain aborted on a stack overflow. Both now keep a stack of their own. A move drew a paired scene source's picture at the incoming scene's moment from its first frame, jumping the nested scene back to its start. It now draws the outgoing moment until halfway, as a look that cannot be mixed does, and never shares that draw with the preview. Tests: a scene showing the program through two scene sources draws after it, a scene shown twice draws once, and a blend mode over a partly transparent backdrop mixes partly. Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/collection.rs | 66 +++++----- rs/moq-scene/src/compositor.rs | 223 ++++++++++++++++++++++++++++----- 2 files changed, 227 insertions(+), 62 deletions(-) diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 986a7523..00819bab 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -255,41 +255,43 @@ impl Collection { /// Refuse a scene that draws itself, directly or through other scenes, /// shown or hidden, since it would never finish drawing. fn validate_nesting(&self) -> Result<(), Error> { - /// Walk what `scene` draws, refusing a scene already on `path`, the - /// scenes drawing it. `done` holds the scenes walked whole. - fn visit<'a>( - collection: &'a Collection, - scene: &'a Scene, - done: &mut BTreeSet<&'a str>, - path: &mut Vec<&'a str>, - ) -> Result<(), Error> { - if done.contains(scene.name.as_str()) { - return Ok(()); - } - if let Some(at) = path.iter().position(|name| *name == scene.name) { - let through: Vec = path[at + 1..].iter().map(|name| format!("`{name}`")).collect(); - let mut err = format!("scene `{}` draws itself", scene.name); - if !through.is_empty() { - err = format!("{err} through {}", through.join(", ")); - } - return Err(Error::Invalid(err)); + // Walked with a stack of its own rather than the call stack, which a + // long enough chain of scenes would overflow. + let mut done = BTreeSet::new(); + for root in &self.scenes { + if done.contains(root.name.as_str()) { + continue; } - path.push(&scene.name); - for item in &scene.items { - if let Some(Nested::Scene(nested)) = collection.nested(&item.source) - && let Some(nested) = collection.scene(nested) - { - visit(collection, nested, done, path)?; + // The scenes drawing the one walked, each with how many of its + // items are walked. + let mut path = vec![(root, 0)]; + while let Some((scene, walked)) = path.last_mut() { + let scene: &Scene = *scene; + let Some(item) = scene.items.get(*walked) else { + done.insert(scene.name.as_str()); + path.pop(); + continue; + }; + *walked += 1; + let Some(Nested::Scene(nested)) = self.nested(&item.source) else { + continue; + }; + let Some(nested) = self.scene(nested).filter(|nested| !done.contains(nested.name.as_str())) else { + continue; + }; + if let Some(at) = path.iter().position(|(on, _)| on.name == nested.name) { + let through: Vec = path[at + 1..] + .iter() + .map(|(on, _)| format!("`{}`", on.name)) + .collect(); + let mut err = format!("scene `{}` draws itself", nested.name); + if !through.is_empty() { + err = format!("{err} through {}", through.join(", ")); + } + return Err(Error::Invalid(err)); } + path.push((nested, 0)); } - path.pop(); - done.insert(&scene.name); - Ok(()) - } - - let mut done = BTreeSet::new(); - for scene in &self.scenes { - visit(self, scene, &mut done, &mut Vec::new())?; } for item in &self.top { diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index d4b2be48..411f081a 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -968,54 +968,94 @@ impl<'a> Nesting<'a> { } /// Add the scenes `items` show, from when the scene they are in went on - /// `since` ago. Returns whether any shows the program. `walked` holds each - /// scene and moment added, or being walked. + /// `since` ago. `walked` holds each scene and moment added, or being + /// walked. fn walk( &mut self, collection: &'a Collection, items: &'a [Item], since: Duration, walked: &mut HashMap<(&'a str, Option), Option>, - ) -> bool { - let mut program = false; - for item in items.iter().filter(|item| item.visible) { + ) { + /// A run of items being walked, with the scene and moment they draw + /// unless they are the root's. + struct Walk<'a> { + nest: Option<(&'a str, Option)>, + items: &'a [Item], + next: usize, + /// Whether any item walked shows the program. + program: bool, + } + + // A stack of its own rather than the call stack, which a long enough + // chain of scenes would overflow. + let mut stack = vec![Walk { + nest: None, + items, + next: 0, + program: false, + }]; + while let Some(walk) = stack.last_mut() { + let items = walk.items; + let Some(item) = items.get(walk.next) else { + let Some(Walk { + nest: Some((scene, moment)), + program, + .. + }) = stack.pop() + else { + break; + }; + self.order.push(Nest { + scene, + since, + moment, + program, + }); + let index = self.order.len() - 1; + walked.insert((scene, moment), Some(index)); + self.drawn.insert((scene, since), index); + stack.last_mut().expect("a nest is walked from items").program |= program; + continue; + }; + walk.next += 1; + if !item.visible { + continue; + } let scene = match collection.nested(&item.source) { Some(Nested::Scene(scene)) => scene, Some(Nested::Program) => { - program = true; + walk.program = true; continue; } None => continue, }; if let Some(&index) = self.drawn.get(&(scene, since)) { - program |= self.order[index].program; + walk.program |= self.order[index].program; continue; } let moment = moment(collection, scene, since); - let index = match walked.get(&(scene, moment)) { - Some(Some(index)) => *index, + match walked.get(&(scene, moment)) { + Some(Some(index)) => { + let index = *index; + self.drawn.insert((scene, since), index); + walk.program |= self.order[index].program; + } // A scene drawing itself, which validation refuses: it shows - // nothing there rather than recursing forever. - Some(None) => continue, + // nothing there rather than walking forever. + Some(None) => {} None => { walked.insert((scene, moment), None); let items = collection.scene(scene).map_or(&[][..], |scene| scene.items.as_slice()); - let shows = self.walk(collection, items, since, walked); - self.order.push(Nest { - scene, - since, - moment, - program: shows, + stack.push(Walk { + nest: Some((scene, moment)), + items, + next: 0, + program: false, }); - let index = self.order.len() - 1; - walked.insert((scene, moment), Some(index)); - index } - }; - self.drawn.insert((scene, since), index); - program |= self.order[index].program; + } } - program } } @@ -1298,20 +1338,29 @@ impl Draw<'_> { let (old, new) = (&*leaving[at], &*entering[index]); let own = Pose::of(collection, new); let pose = Pose::of(collection, old).toward(&own, progress); + // A nested scene's picture cannot be mixed between the + // moments the two scenes are at, so it changes as the + // stacking does. + let since = match collection.nested(&new.source) { + Some(Nested::Scene(_)) if progress < 0.5 => from.1, + _ => to.1, + }; if old.filters == new.filters && old.blend == new.blend { - if pose == own { + // The preview never draws the incoming scene's item + // with the outgoing scene's picture. + if pose == own && since == to.1 { // Shared with the preview when it draws the same. - let key = (place.clone(), index, self.settled(&resting[index], to.1)); - self.item(target, key, new, to.1); + let key = (place.clone(), index, self.settled(&resting[index], since)); + self.item(target, key, new, since); } else { - self.posed(target, new, pose, to.1); + self.posed(target, new, pose, since); } } else if let Some(filters) = mixed(old, new, progress) { let item = Item { filters, ..new.clone() }; - self.posed(target, &item, pose, to.1); + self.posed(target, &item, pose, since); } else { // A look that cannot be mixed changes as the stacking does. - let (item, since) = if progress < 0.5 { (old, from.1) } else { (new, to.1) }; + let item = if progress < 0.5 { old } else { new }; self.posed(target, item, pose, since); } } @@ -3360,6 +3409,14 @@ mod tests { visible: false, ..Item::new("red scene", fill) }, + // Again, over the top-left quarter. + Item::new( + "red scene", + Rect { + y: 0.0, + ..CORNER + }, + ), ]); // Blue over the top-right quarter alone, transparent elsewhere. collection.scenes.push(Scene { @@ -3379,6 +3436,7 @@ mod tests { assert_eq!(pixel(&program, 0, 7), RED, "the red scene's red"); assert_eq!(pixel(&program, 3, 4), BLUE, "the red scene's corner"); assert_eq!(pixel(&program, 6, 6), BLUE, "under it"); + assert_eq!(pixel(&program, 0, 0), RED, "shown twice"); assert_eq!(studio.compositor.nested.get("red").map(Vec::len), Some(1)); studio.compositor.set_preview(Some("blue".into())); @@ -3432,6 +3490,111 @@ mod tests { .await; } + /// A scene source showing a scene that shows the program, even through + /// another, is drawn after the program, on the same frame. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_scene_showing_the_program_is_drawn_after_it() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(Vec::new()); + collection.scenes[2].items = vec![Item::new("outer scene", fill)]; + for (name, shows) in [("inner", "on air"), ("outer", "inner scene")] { + collection.scenes.push(Scene { + items: vec![Item::new(shows, fill)], + ..Scene::new(name) + }); + let source = Source::Scene { scene: name.into() }; + collection.sources.insert(format!("{name} scene"), source); + } + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.compositor.set_preview(Some("scene".into())); + studio.render(Instant::now()); + assert_eq!(pixel(&studio.preview(), 1, 1), BLUE); + }) + .await; + } + + /// An item showing a scene source in both scenes of a move shows the + /// nested scene at the moment the scene it leaves is at until halfway, + /// then at the moment the scene it lands on is at, as the stacking + /// changes. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_move_shows_a_nested_scene_as_the_scene_it_leaves_until_halfway() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("blue scene", fill)]); + collection.scenes[1].items[0].keyframes = slide_in(); + collection.scenes.push(Scene { + items: vec![Item::new("blue scene", fill)], + ..Scene::new("again") + }); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(at(0)); + studio.render(at(2000)); + assert_eq!(pixel(&studio.program(), 7, 1), BLUE, "all the way in"); + + studio.compositor.set_preview(Some("again".into())); + let duration = Duration::from_secs(1); + studio + .compositor + .transition( + &studio.collection, + Transition::Move { duration }, + &studio.sources, + at(2000), + ) + .unwrap(); + studio.render(at(2250)); + assert_eq!(pixel(&studio.program(), 7, 1), BLUE, "as it was"); + studio.render(at(2750)); + let program = studio.program(); + assert_eq!( + (pixel(&program, 5, 1), pixel(&program, 6, 1)), + (BLUE, BLACK), + "sliding in again" + ); + }) + .await; + } + + /// A non-normal blend over a backdrop that is partly transparent, in a + /// nested scene, mixes with the backdrop as much as the backdrop covers, + /// as CSS does. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_blend_mode_over_a_partly_transparent_backdrop_mixes_partly() { + local(async { + let fill = Rect::fill(&collection().canvas); + let mut collection = nesting(vec![Item::new("blue scene", fill)]); + let half = Item { + filters: vec![Filter::Opacity { amount: 0.5 }], + ..Item::new("red", fill) + }; + collection.scenes[1].items[0].blend = Blend::Multiply; + collection.scenes[1].items.insert(0, half); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("scene".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + // Half blue, half blue multiplied by red, which is black. + let actual = pixel(&studio.program(), 1, 1); + assert!(near(actual, [0, 0, 128, 255]), "{actual:?}"); + }) + .await; + } + /// A nested scene's keyframes play from when the scene showing it went /// on, so preview and program can show it at two moments, each drawn once, /// and once both settle, drawn once for both. From c627681636788841cc660097cfed6d48a5282b08 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 07:37:46 -0800 Subject: [PATCH 7/8] style: cargo fmt Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/collection.rs | 5 +---- rs/moq-scene/src/compositor.rs | 8 +------- 2 files changed, 2 insertions(+), 11 deletions(-) diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 00819bab..83d9a1bb 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -280,10 +280,7 @@ impl Collection { continue; }; if let Some(at) = path.iter().position(|(on, _)| on.name == nested.name) { - let through: Vec = path[at + 1..] - .iter() - .map(|(on, _)| format!("`{}`", on.name)) - .collect(); + let through: Vec = path[at + 1..].iter().map(|(on, _)| format!("`{}`", on.name)).collect(); let mut err = format!("scene `{}` draws itself", nested.name); if !through.is_empty() { err = format!("{err} through {}", through.join(", ")); diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 411f081a..9efa77ca 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -3410,13 +3410,7 @@ mod tests { ..Item::new("red scene", fill) }, // Again, over the top-left quarter. - Item::new( - "red scene", - Rect { - y: 0.0, - ..CORNER - }, - ), + Item::new("red scene", Rect { y: 0.0, ..CORNER }), ]); // Blue over the top-right quarter alone, transparent elsewhere. collection.scenes.push(Scene { From c30a39b94136a85ceaf2185976ecfa5c9bc9248e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 07:48:24 -0800 Subject: [PATCH 8/8] fix(scene): satisfy clippy's explicit auto-deref Co-Authored-By: Claude Opus 5.5 --- rs/moq-scene/src/collection.rs | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 83d9a1bb..0507dac0 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -265,8 +265,7 @@ impl Collection { // The scenes drawing the one walked, each with how many of its // items are walked. let mut path = vec![(root, 0)]; - while let Some((scene, walked)) = path.last_mut() { - let scene: &Scene = *scene; + while let Some(&mut (scene, ref mut walked)) = path.last_mut() { let Some(item) = scene.items.get(*walked) else { done.insert(scene.name.as_str()); path.pop();