diff --git a/README.md b/README.md index 9af1832..b056c17 100644 --- a/README.md +++ b/README.md @@ -90,17 +90,17 @@ Every panel is a tab you can drag, split, or stack. | Panel | Use it to | | --- | --- | -| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. **Cut** puts the preview on program at once; **Transition** takes it there by the transition picked beside it, over the time set there (see [Transitions](#transitions)). **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | +| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. With more than one canvas, pick the canvas they show, and that the Items panel edits, beside the preview. **Cut** puts the preview on program at once; **Transition** takes it there by the transition picked beside it, over the time set there (see [Transitions](#transitions)). **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | | Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still leaving it in a transition, cannot be removed, even by an undo, until program cuts away from it. | | History | See every edit, and undo or redo back to any of them. | -| Items | Place, show, hide, and order what the preview scene or the top layer draws, back to front. Select an item to filter or copy it. | +| Items | Place, show, hide, and order what the preview scene or the top layer draws on the canvas shown, back to front. Select an item to filter or copy it. | | Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. A media file reads as ended once it plays to its end without looping, **Play** plays it from its start, and **Reload** reads the file again. | | Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. | | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. | | Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | | Keyframes | Animate the selected item: add, time, ease, and remove its keyframes, and set how far each moves, resizes, crops, and filters it (see [Keyframes](#keyframes)). **Play** plays the preview scene's keyframes again. | | Audio | Add, rename (double-click), and remove mixes, pick the one on program and the one to listen to and on which output, and set each audio source's gain, mute, delay, and the mixes it feeds, with a meter on every mix and source. | -| Output | Start and stop recording, pick the mix it records, set the canvas size and frame rate, and see the broadcast path, whether the program is encoding, the encoder sessions open, and each rendition's encoder work. | +| Output | Start and stop recording, pick the mix it records, set the canvas size and frame rate, add, resize, and remove further canvases, and see the broadcast path, whether the program is encoding, the encoder sessions open, and each canvas's renditions' encoder work. | | Health | See how the stream reaches viewers, relay by relay and track by track, and the cause of each problem. | | Replay | Replay the last moments of the program or a source in a replay source, slowed down if you like, or save them as a clip. | @@ -109,6 +109,25 @@ Items panel shows. The **top layer** is drawn over every scene, in preview and on program, and stays on through a cut or transition: a downstream keyer for a logo or a lower third. +### Canvases + +Beside the main canvas, the studio can draw further canvases at once, such as +a 1080x1920 portrait one for a vertical stream beside a 1920x1080 landscape +main canvas. Add one by name in the Output panel. Each scene has its own +layout on each canvas, and each canvas its own top layer; a scene without a +layout on a canvas shows only that canvas's top layer there. Every canvas +shares the main canvas's frame rate, and the canvases are linked: one scene is +in preview and one on program on all of them, and a cut or transition takes +every canvas together. An item's keyframes play on every canvas from when its +scene goes on, as on the main one. A source captures or decodes once however many canvases +show it, and a canvas draws only the items it places. + +Each further canvas publishes a broadcast of its own beneath the main one, at +`<--broadcast>/`, with renditions and encoders of its own (see +[Output](#output)). Its catalog lists the main broadcast's audio rather than +encoding it again. Removing a canvas ends its broadcast, and removes its top +layer and every scene's layout on it. + ### Transitions | Transition | What it does | @@ -415,6 +434,11 @@ width = 1920 height = 1080 framerate = 30 +[canvases.portrait] # a further canvas, at the main one's frame rate +width = 1080 +height = 1920 +top = [{ source = "logo", rect = { x = 780, y = 40, width = 280, height = 120 } }] + [transition] kind = "slide" # fade, slide, wipe, or move duration = "300ms" @@ -485,6 +509,10 @@ items = [ { source = "guest", rect = { x = 1280, y = 720, width = 640, height = 360 }, filters = [{ kind = "drop_shadow", x = 8, y = 8, blur = 12, color = "#00000080" }] }, { source = "logo", rect = { x = 0, y = 0, width = 1920, height = 1080 }, blend = "multiply", filters = [{ kind = "opacity", amount = 0.3 }] }, ] +canvases.portrait = [ # the same scene laid out on the portrait canvas + { source = "camera", rect = { x = 0, y = 0, width = 1080, height = 1920 }, crop = { left = 0.34, right = 0.34 } }, + { source = "guest", rect = { x = 0, y = 1312, width = 1080, height = 608 } }, +] [[scenes]] name = "Break" @@ -512,6 +540,11 @@ file, which no item places, but which grades what a `lut` filter names it in), and `replay`, where the Replay panel plays replays. The Devices panel fills in ids. `[[top]]` items make up the top layer. +`[canvases.]` adds a further canvas, with its `width` and `height` and +its own `top` layer, and a scene's `canvases.` lists its items on that +canvas; leave it out, rather than empty, for a scene that shows nothing there. +A canvas name is one segment of its broadcast path, so it holds no `/`. + `[mixes]` names every mix, in the order the Audio panel lists them, the one on `program`, and the one a `recording` holds, the mix on program without it; without `[mixes]`, the collection has one mix, `Program`. @@ -657,17 +690,20 @@ encode engine for them all. The Output panel counts the sessions open: |---|---| | The program's canvas size, live | 1, or none without `--connect` | | Its half size | 1 while someone watches it | +| Each further canvas's canvas size and half size | as the program's | | The recording's own canvas size | 1 while recording | | Each captured source or media file | 1 while on program or recorded | So a live scene showing a camera, a display, and a window holds 4 sessions, 5 -with the half size watched, and 6 while recording too. When the program +with the half size watched, and 6 while recording too. A portrait canvas beside +it adds 1 more, 2 with its half size watched. When the program encodes on hardware, so do the sources, and a session the GPU refuses fails that source's track with the encoder's reason in the Sources panel rather than falling back to software. The source keeps drawing in the studio's own scenes. The program opens its session before any source, even with `--on-demand`, so the sources past the cap fail rather than -the program. When the program encodes in software, a source takes hardware +the program; so does each canvas in the scene file at startup. A canvas added +later, past the cap, draws without video and says why in the Output panel. When the program encodes in software, a source takes hardware where it can and software otherwise. ### Recording diff --git a/quest/m0/multi-canvas.md b/quest/m0/multi-canvas.md index 86c4080..e7ab30d 100644 --- a/quest/m0/multi-canvas.md +++ b/quest/m0/multi-canvas.md @@ -9,13 +9,19 @@ captures or decodes once however many canvases use it. ## Plan -Each canvas publishes its own broadcast, because different content needs a -different name. Each canvas has its own top layer, which the collection's -`top` holds for the one canvas today. Transitions are per canvas, so a scene change cuts every -canvas together unless the creator unlinks one. Reuse the one canvas's resize -path per canvas: the compositor reallocates on a new canvas, and the program -video publishes a rendition at a time, ending one and listing the next. +Further canvases, their layouts and top layers, linked scene switching, and a +broadcast per canvas at `<--broadcast>/` landed in +[moq-dev/studio#42](https://github.com/moq-dev/studio/pull/42). What remains: -Benchmark source count against canvas count, including sparse reuse. Per-canvas -cost must grow only with the sources that canvas uses, never with the whole -source table. +- Unlinking: a canvas the creator unlinks keeps its own preview and program + scene, so a scene change no longer cuts it with the others. Transitions are + per canvas, so an unlinked canvas takes its own. +- Outputs per canvas: a recording, and each destination, picks the canvas it + carries, as each picks its mix. Only the main canvas records today. +- Health per canvas: the Health panel grades each canvas's broadcast, with + viewer feedback beneath it, not only the main one's. +- Preview cost: a canvas no panel shows skips drawing its preview, so an + unseen canvas costs only its program. + +Per-canvas cost must grow only with the sources that canvas uses, never with +the whole source table; `compositor::bench` measures it. diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index a647786..6787882 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -295,7 +295,7 @@ mod tests { fn pastes_linked_within_a_collection() { let collection = collection(); let clip = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)); - let pasted = paste(&collection, &Layer::Top, &clip.to_toml().unwrap(), DIR); + let pasted = paste(&collection, &Layer::top(), &clip.to_toml().unwrap(), DIR); assert_eq!(pasted.sources, collection.sources, "no source added"); assert_eq!(pasted.top, collection.scenes[0].items); } @@ -311,14 +311,14 @@ mod tests { }]; let clip = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)); let text = clip.to_toml().unwrap(); - let pasted = paste(&collection, &Layer::Scene("Scene".into()), &text, DIR); + let pasted = paste(&collection, &Layer::scene("Scene"), &text, DIR); assert_eq!(pasted.scenes[0].items[1], collection.scenes[0].items[0]); let Clip::Items(items) = Clip::from_toml(&text).unwrap() else { panic!("not items") }; let mut draft = collection.draft("Paste"); - draft.paste(&Layer::Top, &items, Path::new(DIR)).unwrap(); + draft.paste(&Layer::top(), &items, Path::new(DIR)).unwrap(); assert!(matches!(draft.finish(), Err(Error::Invalid(_)))); } @@ -333,7 +333,7 @@ mod tests { let mut other = Collection::new(); other.sources.insert("cam".into(), color([0, 255, 0, 255])); - let pasted = paste(&other, &Layer::Scene("Scene".into()), &text, DIR); + let pasted = paste(&other, &Layer::scene("Scene"), &text, DIR); assert_eq!(pasted.sources["cam"], color([0, 255, 0, 255]), "left alone"); assert_eq!(pasted.sources["cam 2"], Source::Camera { id: None }); assert_eq!(pasted.sources["cam faded"], Source::Clone { source: "cam 2".into() }); @@ -367,7 +367,7 @@ mod tests { .to_toml() .unwrap(); - let pasted = paste(&Collection::new(), &Layer::Top, &text, DIR); + let pasted = paste(&Collection::new(), &Layer::top(), &text, DIR); let expected = crate::Audio { exclude: ["Program".to_string()].into(), ..audio @@ -393,7 +393,7 @@ mod tests { .to_toml() .unwrap(); - let home = paste(&collection, &Layer::Top, &text, DIR); + let home = paste(&collection, &Layer::top(), &text, DIR); assert_eq!(home.sources, collection.sources, "linked to the same image"); // However the scene file spells the path. @@ -404,7 +404,7 @@ mod tests { path: "/shows/one/logo.png".into(), }, ); - let home = paste(&absolute, &Layer::Top, &text, DIR); + let home = paste(&absolute, &Layer::top(), &text, DIR); assert_eq!(home.sources, absolute.sources, "linked to the same image"); let mut other = Collection::new(); @@ -414,7 +414,7 @@ mod tests { path: "logo.png".into(), }, ); - let away = paste(&other, &Layer::Top, &text, "/shows/two"); + let away = paste(&other, &Layer::top(), &text, "/shows/two"); assert_eq!(away.sources["logo"], other.sources["logo"], "another file, left alone"); assert_eq!( away.sources["logo 2"], @@ -462,7 +462,7 @@ mod tests { let mut other = Collection::new(); other.sources.insert("matte".into(), color([0, 255, 0, 255])); - let pasted = paste(&other, &Layer::Top, &text, "/shows/two"); + let pasted = paste(&other, &Layer::top(), &text, "/shows/two"); assert_eq!(pasted.top[0].filters, [mask("matte 2")]); assert_eq!( pasted.sources["grade"], diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 8094c5e..4f9beda 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -12,8 +12,8 @@ use serde_with::{DeserializeFromStr, SerializeDisplay}; use crate::Error; -/// Everything a studio composes: the canvas, the sources, and the scenes that -/// place them. +/// Everything a studio composes: the canvases, the sources, and the scenes +/// that place them. /// /// Saved with [`to_toml`](Self::to_toml) and loaded with /// [`from_toml`](Self::from_toml). The file names devices, files, and broadcast @@ -21,8 +21,14 @@ use crate::Error; #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct Collection { - /// The output every scene is drawn onto. + /// The main canvas every scene is drawn onto, whose frame rate every + /// canvas shares. pub canvas: Canvas, + /// Further canvases by name, such as a portrait one beside a landscape + /// main canvas. Each draws its own layout of every scene from the same + /// sources, under its own top layer. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub canvases: BTreeMap, /// How the preview is taken to program, short of a cut. #[serde(default)] pub transition: Transition, @@ -62,6 +68,7 @@ impl Collection { pub fn new() -> Self { Self { canvas: Canvas::default(), + canvases: BTreeMap::new(), transition: Transition::default(), sources: BTreeMap::new(), mixes: Mixes::default(), @@ -92,33 +99,87 @@ impl Collection { self.scenes.iter().find(|scene| scene.name == name) } - /// The items of `layer`, or `None` for a scene that does not exist. - pub fn items(&self, layer: &Layer) -> Option<&Vec> { - match layer { - Layer::Scene(name) => self.scene(name).map(|scene| &scene.items), - Layer::Top => Some(&self.top), + /// The canvas `canvas` names: the main one for `None`, or a further one, + /// at the main canvas's frame rate. `None` for a canvas that does not + /// exist. + pub fn canvas(&self, canvas: Option<&str>) -> Option { + match canvas { + None => Some(self.canvas), + Some(name) => self.canvases.get(name).map(|extra| extra.canvas(self.canvas.framerate)), } } - /// The items of `layer`, to edit. - pub(crate) fn items_mut(&mut self, layer: &Layer) -> Option<&mut Vec> { - match layer { - Layer::Scene(name) => self - .scenes - .iter_mut() - .find(|scene| &scene.name == name) - .map(|scene| &mut scene.items), - Layer::Top => Some(&mut self.top), + /// The items of `layer`, or `None` for a scene or canvas that does not + /// exist. A scene with no layout on a further canvas has none there. + pub fn items(&self, layer: &Layer) -> Option<&[Item]> { + match (&layer.canvas, &layer.scene) { + (None, Some(scene)) => self.scene(scene).map(|scene| scene.items.as_slice()), + (None, None) => Some(&self.top), + (Some(canvas), scene) => { + let extra = self.canvases.get(canvas)?; + match scene { + Some(scene) => { + let layout = self.scene(scene)?.canvases.get(canvas); + Some(layout.map_or(&[], Vec::as_slice)) + } + None => Some(&extra.top), + } + } } } - /// Every layer: the scenes in order, then the top layer. + /// Run `edit` on the items of `layer`, or return `None` for a scene or + /// canvas that does not exist. A scene's layout on a further canvas is + /// kept only while it holds items, so a scene has one way to say it has + /// none there, as [`validate`](Self::validate) requires. + pub(crate) fn edit_items(&mut self, layer: &Layer, edit: impl FnOnce(&mut Vec) -> T) -> Option { + if let Some(canvas) = &layer.canvas + && !self.canvases.contains_key(canvas) + { + return None; + } + match (&layer.canvas, &layer.scene) { + (None, None) => Some(edit(&mut self.top)), + (Some(canvas), None) => self.canvases.get_mut(canvas).map(|extra| edit(&mut extra.top)), + (canvas, Some(name)) => { + let scene = self.scenes.iter_mut().find(|scene| &scene.name == name)?; + let Some(canvas) = canvas else { + return Some(edit(&mut scene.items)); + }; + let mut layout = scene.canvases.remove(canvas).unwrap_or_default(); + let edited = edit(&mut layout); + if !layout.is_empty() { + scene.canvases.insert(canvas.clone(), layout); + } + Some(edited) + } + } + } + + /// Every layer: the main canvas's scenes in order, then its top layer, + /// then each further canvas's the same way. pub fn layers(&self) -> impl Iterator { - let scenes = self - .scenes + let extras = self + .canvases .iter() - .map(|scene| (Layer::Scene(scene.name.clone()), scene.items.as_slice())); - scenes.chain(std::iter::once((Layer::Top, self.top.as_slice()))) + .map(|(name, extra)| (Some(name), extra.top.as_slice())); + let canvases = std::iter::once((None, self.top.as_slice())).chain(extras); + canvases.flat_map(move |(canvas, top)| { + let scenes = self.scenes.iter().map(move |scene| { + let items = match canvas { + None => scene.items.as_slice(), + Some(name) => scene.canvases.get(name).map_or(&[][..], Vec::as_slice), + }; + (Some(scene.name.clone()), items) + }); + scenes.chain([(None, top)]).map(move |(scene, items)| { + let layer = Layer { + canvas: canvas.cloned(), + scene, + }; + (layer, items) + }) + }) } /// The source whose capture `name` shows: itself, or what it clones. @@ -148,6 +209,17 @@ impl Collection { /// source of the kind it reads. pub fn validate(&self) -> Result<(), Error> { self.canvas.validate()?; + for (name, extra) in &self.canvases { + if name.is_empty() || name.contains('/') { + return Err(Error::Invalid(format!( + "canvas `{name}` must be named, without a `/`, since its name names its broadcast" + ))); + } + extra + .canvas(self.canvas.framerate) + .validate() + .map_err(|err| Error::Invalid(format!("canvas `{name}`: {err}")))?; + } let mut names = BTreeSet::new(); for scene in &self.scenes { @@ -157,6 +229,24 @@ impl Collection { if !names.insert(scene.name.as_str()) { return Err(Error::Invalid(format!("two scenes are named `{}`", scene.name))); } + if let Some(canvas) = scene + .canvases + .keys() + .find(|canvas| !self.canvases.contains_key(*canvas)) + { + return Err(Error::Invalid(format!( + "scene `{}` has a layout on `{canvas}`, which is not a canvas", + scene.name + ))); + } + // One way to say a scene shows nothing there, so an edit snapshotting + // the scene and one emptying the layout agree on what it holds. + if let Some((canvas, _)) = scene.canvases.iter().find(|(_, items)| items.is_empty()) { + return Err(Error::Invalid(format!( + "scene `{}` has an empty layout on `{canvas}`; leave it out instead", + scene.name + ))); + } } for (layer, items) in self.layers() { for item in items { @@ -178,7 +268,7 @@ impl Collection { .validate() .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; self.validate_chain(&item.filters, &on)?; - if layer == Layer::Top && !item.keyframes.is_empty() { + if layer.scene.is_none() && !item.keyframes.is_empty() { return Err(Error::Invalid(format!( "{on} has keyframes, but the top layer never goes on to start them" ))); @@ -360,6 +450,41 @@ impl Default for Canvas { } } +/// A canvas beside the main one: its size, and the items drawn over every +/// scene on it. Each scene's layout on it is the scene's own +/// [`Scene::canvases`] entry. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Extra { + /// Width in pixels. Even, since the encoder subsamples chroma. + pub width: u32, + /// Height in pixels. Even, since the encoder subsamples chroma. + pub height: u32, + /// Its top layer, as [`Collection::top`] is the main canvas's. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub top: Vec, +} + +impl Extra { + /// An empty canvas of `width` by `height`. + pub fn new(width: u32, height: u32) -> Self { + Self { + width, + height, + top: Vec::new(), + } + } + + /// This canvas drawn `framerate` times a second. + pub fn canvas(&self, framerate: u32) -> Canvas { + Canvas { + width: self.width, + height: self.height, + framerate, + } + } +} + /// How the program changes when the preview is taken to it, short of a cut. /// /// A fade runs at an even pace; the others ease in and out, so motion starts @@ -502,9 +627,13 @@ impl Direction { pub struct Scene { /// Unique among the collection's scenes. pub name: String, - /// What the scene draws, back to front. + /// What the scene draws on the main canvas, back to front. #[serde(default)] pub items: Vec, + /// What the scene draws on each further canvas, by the canvas's name, back + /// to front. A canvas without an entry shows only its top layer. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub canvases: BTreeMap>, } impl Scene { @@ -513,25 +642,53 @@ impl Scene { Self { name: name.into(), items: Vec::new(), + canvases: BTreeMap::new(), } } } -/// Where a list of items lives: a scene, or the top layer over every scene. -#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum Layer { - /// The scene with this name. - Scene(String), - /// [`Collection::top`]. - Top, +/// Where a list of items lives: a scene's layout, or the top layer over every +/// scene, on one canvas. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Layer { + /// A further canvas by name, or `None` for the main canvas. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canvas: Option, + /// The scene, or `None` for the canvas's top layer. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub scene: Option, +} + +impl Layer { + /// The scene called `name`, on the main canvas. + pub fn scene(name: impl Into) -> Self { + Self { + canvas: None, + scene: Some(name.into()), + } + } + + /// The main canvas's top layer. + pub fn top() -> Self { + Self::default() + } + + /// The same layer on `canvas` instead. + pub fn on(self, canvas: Option) -> Self { + Self { canvas, ..self } + } } impl fmt::Display for Layer { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - Self::Scene(name) => write!(f, "scene `{name}`"), - Self::Top => write!(f, "the top layer"), + match &self.scene { + Some(name) => write!(f, "scene `{name}`")?, + None => write!(f, "the top layer")?, + } + match &self.canvas { + Some(canvas) => write!(f, " on canvas `{canvas}`"), + None => Ok(()), } } } @@ -2196,6 +2353,36 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } assert!(matches!(collection.validate(), Err(Error::Invalid(_)))); } + /// A further canvas is named as a path segment, since it names its + /// broadcast, is a size the encoder can take, and is the only canvas a + /// scene lays items out on. + #[test] + fn refuses_a_further_canvas_that_cannot_be() { + for (name, width) in [("", 1080), ("a/b", 1080), ("tall", 1081)] { + let mut collection = example(); + collection.canvases.insert(name.into(), Extra::new(width, 1920)); + assert!( + matches!(collection.validate(), Err(Error::Invalid(_))), + "{name} {width}" + ); + } + let mut collection = example(); + let item = Item::new("camera", Rect::fill(&collection.canvas)); + collection.scenes[0].canvases.insert("tall".into(), vec![item]); + assert!( + matches!(collection.validate(), Err(Error::Invalid(_))), + "no such canvas" + ); + collection.canvases.insert("tall".into(), Extra::new(1080, 1920)); + collection.validate().unwrap(); + // Leaving a layout out is the one way to say a scene shows nothing there. + collection.scenes[0].canvases.insert("tall".into(), Vec::new()); + assert!( + matches!(collection.validate(), Err(Error::Invalid(_))), + "an empty layout" + ); + } + #[test] fn refuses_a_canvas_the_encoder_cannot_take() { for (width, height, framerate) in [(1921, 1080, 30), (0, 1080, 30), (1920, 1080, 0)] { diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 3770e7b..111563f 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -1,4 +1,4 @@ -//! Draw scenes onto the canvas on the GPU: a preview, a program, and the +//! Draw scenes onto each canvas on the GPU: a preview, a program, and the //! transition between them. use std::borrow::Cow; @@ -20,38 +20,35 @@ use crate::source::{self, Sources}; /// `moq_video::render` produces and what its exporter hands an encoder. pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; -/// Draws a [`Collection`]'s scenes onto its canvas: one texture for the scene -/// in preview, one for the scene on program, and the [`Transition`] from one -/// to the other. +/// Draws a [`Collection`]'s scenes onto each of its canvases: one texture for +/// the scene in preview, one for the scene on program, and the [`Transition`] +/// from one to the other. /// -/// Both textures are plain [`wgpu::Texture`]s on the caller's device, so a UI +/// The textures are plain [`wgpu::Texture`]s on the caller's device, so a UI /// shows them without a copy and an encoder reads the program from the same -/// memory. [`render`](Self::render) redraws both from the newest picture each +/// memory. [`render`](Self::render) redraws them from the newest picture each /// source has, so a source that stalls repeats its last picture rather than -/// holding up the canvas. +/// holding up the canvas. Each source's picture is uploaded once however many +/// canvases show it, and a canvas draws only the items it places. /// /// Pixels are blended in the gamma-encoded space sources arrive in, as a /// broadcast switcher and CSS do, so a 50% fade is a 50% mix of the stored /// values. /// -/// A transition and each item's keyframes run on the clock -/// [`render`](Self::render) is given, so every output drawn from the program, -/// such as the encoder and a recording, sees the same frame of them. An item's -/// keyframes start when its scene goes on program, and separately when it goes -/// into preview, for the preview. +/// The canvases are linked: one scene is in preview and one on program on +/// every canvas, and a transition and each item's keyframes run on each at +/// once, on the clock [`render`](Self::render) is given, so every output drawn +/// from a program, such as the encoder and a recording, sees the same frame of +/// them. An item's keyframes start when its scene goes on program, and +/// separately when it goes into preview, for the preview, on every canvas +/// alike. pub struct Compositor { device: wgpu::Device, queue: wgpu::Queue, - canvas: Canvas, effects: Effects, - preview: Target, - program: Target, - /// The scene coming in, drawn whole so it fades, slides, or is uncovered - /// as one picture rather than item by item. - incoming: Target, - /// The scene going out, drawn whole to slide it off the canvas. - outgoing: Target, + main: Output, + extras: BTreeMap, layers: HashMap, /// Lays out and paints overlays, started with the first one. @@ -64,11 +61,54 @@ pub struct Compositor { /// When the program scene went on program, as `preview_start` is. program_start: Option, /// How long the preview scene had been in preview as of the last render: - /// the moment of its keyframes the preview texture shows. + /// the moment of its keyframes every canvas's preview texture shows. preview_shown: Duration, switch: Option, } +/// One canvas's preview and program, as of the last +/// [`render`](Compositor::render). +pub struct Output { + canvas: Canvas, + preview: Target, + program: Target, + /// The scene coming in, drawn whole so it fades, slides, or is uncovered + /// as one picture rather than item by item. + incoming: Target, + /// The scene going out, drawn whole to slide it off the canvas. + outgoing: Target, +} + +impl Output { + fn new(device: &wgpu::Device, effects: &Effects, canvas: Canvas) -> Self { + let target = |label| Target::new(device, effects, canvas, label); + Self { + canvas, + preview: target("moq-scene preview"), + program: target("moq-scene program"), + incoming: target("moq-scene incoming"), + outgoing: target("moq-scene outgoing"), + } + } + + /// The canvas this draws. + pub fn canvas(&self) -> Canvas { + self.canvas + } + + /// The scene in preview. + pub fn preview(&self) -> &wgpu::Texture { + &self.preview.texture + } + + /// The program. Created with `COPY_SRC | TEXTURE_BINDING`, what + /// `moq_video::render::export::Exporter::copy` takes to hand it to an + /// encoder. + pub fn program(&self) -> &wgpu::Texture { + &self.program.texture + } +} + /// A transition under way from `from` to the program scene. struct Switch { from: Option, @@ -115,23 +155,20 @@ impl effect::Library for Library<'_> { } impl Compositor { - /// A compositor drawing `canvas`-sized pictures on `device`. + /// A compositor drawing a main canvas of `canvas` on `device`, and no + /// further canvas until [`set_canvases`](Self::set_canvases) adds one. /// /// Fails if the canvas is invalid or the device cannot hold a texture that /// large. pub fn new(device: &wgpu::Device, queue: &wgpu::Queue, canvas: Canvas) -> Result { fits(device, canvas)?; - let effects = Effects::new(device, queue, canvas); - let target = |label| Target::new(device, &effects, canvas, label); + let effects = Effects::new(device, queue, canvas.width, canvas.height); Ok(Self { - preview: target("moq-scene preview"), - program: target("moq-scene program"), - incoming: target("moq-scene incoming"), - outgoing: target("moq-scene outgoing"), + main: Output::new(device, &effects, canvas), + extras: BTreeMap::new(), device: device.clone(), queue: queue.clone(), - canvas, effects, layers: HashMap::new(), overlays: None, @@ -144,42 +181,80 @@ impl Compositor { }) } - /// The canvas this compositor draws. + /// The main canvas. pub fn canvas(&self) -> Canvas { - self.canvas + self.main.canvas } - /// Draw `canvas` from the next [`render`](Self::render) on. A new size - /// replaces the [`preview`](Self::preview) and [`program`](Self::program) - /// textures, and every other texture sized from the canvas, with ones of - /// that size, blank until drawn; a transition under way carries on. + /// Draw `collection`'s canvases from the next [`render`](Self::render) on: + /// its main canvas and each further one. A canvas of a new size gets new + /// textures of that size, blank until drawn; a new canvas starts blank, + /// and one `collection` no longer has is dropped. A transition under way + /// carries on. /// - /// Fails, changing nothing, if the canvas is invalid or the device cannot + /// Fails, changing nothing, if a canvas is invalid or the device cannot /// hold a texture that large. - pub fn set_canvas(&mut self, canvas: Canvas) -> Result<(), Error> { - fits(&self.device, canvas)?; - self.effects.set_canvas(canvas); - if canvas.size() != self.canvas.size() { - let target = |label| Target::new(&self.device, &self.effects, canvas, label); - self.preview = target("moq-scene preview"); - self.program = target("moq-scene program"); - self.incoming = target("moq-scene incoming"); - self.outgoing = target("moq-scene outgoing"); + pub fn set_canvases(&mut self, collection: &Collection) -> Result<(), Error> { + let canvas = collection.canvas; + let extras: BTreeMap<&str, Canvas> = collection + .canvases + .iter() + .map(|(name, extra)| (name.as_str(), extra.canvas(canvas.framerate))) + .collect(); + for canvas in std::iter::once(&canvas).chain(extras.values()) { + fits(&self.device, *canvas)?; + } + + let width = extras.values().map(|extra| extra.width).fold(canvas.width, u32::max); + let height = extras.values().map(|extra| extra.height).fold(canvas.height, u32::max); + self.effects.fit(width, height); + let (device, effects) = (&self.device, &self.effects); + let resize = |output: &mut Output, canvas: Canvas| { + if output.canvas.size() == canvas.size() { + output.canvas = canvas; + } else { + *output = Output::new(device, effects, canvas); + } + }; + resize(&mut self.main, canvas); + self.extras.retain(|name, _| extras.contains_key(name.as_str())); + for (name, canvas) in extras { + match self.extras.get_mut(name) { + Some(output) => resize(output, canvas), + None => { + self.extras + .insert(name.to_string(), Output::new(device, effects, canvas)); + } + } } - self.canvas = canvas; Ok(()) } - /// The scene in preview, as of the last [`render`](Self::render). + /// The scene in preview on the main canvas, as of the last + /// [`render`](Self::render). pub fn preview(&self) -> &wgpu::Texture { - &self.preview.texture + self.main.preview() } - /// The program output, as of the last [`render`](Self::render). Created - /// with `COPY_SRC | TEXTURE_BINDING`, what - /// `moq_video::render::export::Exporter::copy` takes to hand it to an encoder. + /// The main canvas's program, as of the last [`render`](Self::render). See + /// [`Output::program`]. pub fn program(&self) -> &wgpu::Texture { - &self.program.texture + self.main.program() + } + + /// The canvas `canvas` names, the main one for `None`, as of the last + /// [`render`](Self::render); `None` for a canvas it does not draw. + pub fn output(&self, canvas: Option<&str>) -> Option<&Output> { + match canvas { + None => Some(&self.main), + Some(name) => self.extras.get(name), + } + } + + /// Every canvas drawn, by name: the main one first, as `None`. + pub fn outputs(&self) -> impl Iterator, &Output)> { + let extras = self.extras.iter().map(|(name, output)| (Some(name.as_str()), output)); + std::iter::once((None, &self.main)).chain(extras) } /// The scene in preview. @@ -217,8 +292,9 @@ impl Compositor { } /// How long the preview scene had been in preview as of the last - /// [`render`](Self::render): the moment of its items' keyframes the - /// [`preview`](Self::preview) shows, for [`Item::posed`]. + /// [`render`](Self::render): the moment of its items' keyframes every + /// canvas's preview shows, since the canvases are linked, for + /// [`Item::posed`]. pub fn preview_since(&self) -> Duration { self.preview_shown } @@ -278,11 +354,11 @@ impl Compositor { }); } - /// Draw preview and program as of `now`, from the newest picture each of - /// `sources` has. + /// Draw preview and program on every canvas as of `now`, from the newest + /// picture each of `sources` has. /// /// A source whose picture fails to draw keeps its last one and reports why - /// through [`error`](Self::error); this fails only when the canvas itself + /// through [`error`](Self::error); this fails only when a canvas itself /// cannot be drawn. pub fn render(&mut self, collection: &Collection, sources: &Sources, now: Instant) -> Result<(), Error> { self.update(collection, sources, now); @@ -307,60 +383,22 @@ impl Compositor { effects: &mut self.effects, library: Library(&self.layers), frame: &mut frame, + canvas: None, + size: self.main.canvas, prepared: HashMap::new(), }; - - // The top layer goes over both outputs, and over a transition rather - // than through it, so it stays on across the switch. - draw.target(&self.preview); - draw.scene(&self.preview, On(self.preview_scene.as_deref(), preview_since)); - draw.layer(&self.preview, Place::Top, Duration::ZERO); - - let to = On(self.program_scene.as_deref(), program_since); - match (&self.switch, linear) { - (Some(switch), Some(linear)) => { - let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); - let progress = switch.transition.progress(linear); - let canvas = Rect::fill(&self.canvas); - match switch.transition { - Transition::Fade { .. } => { - draw.whole(&self.incoming, to); - draw.target(&self.program); - draw.scene(&self.program, from); - draw.place(&self.program, &self.incoming, canvas, canvas, progress); - } - // Uncovered where a slide would have moved it in. - Transition::Wipe { direction, .. } => { - draw.whole(&self.incoming, to); - draw.target(&self.program); - draw.scene(&self.program, from); - let (x, y) = offset(direction, progress, &canvas); - let (dx, dy) = offset(direction, 1.0, &canvas); - let uncovered = shifted(canvas, x - dx, y - dy); - draw.place(&self.program, &self.incoming, canvas, uncovered, 1.0); - } - Transition::Slide { direction, .. } => { - draw.whole(&self.incoming, to); - draw.whole(&self.outgoing, from); - draw.target(&self.program); - let (x, y) = offset(direction, progress, &canvas); - let (dx, dy) = offset(direction, 1.0, &canvas); - draw.place(&self.program, &self.outgoing, shifted(canvas, x, y), canvas, 1.0); - let entering = shifted(canvas, x - dx, y - dy); - draw.place(&self.program, &self.incoming, entering, canvas, 1.0); - } - Transition::Move { .. } => { - draw.target(&self.program); - draw.moving(&self.program, from, to, progress); - } - } - } - _ => { - draw.target(&self.program); - draw.scene(&self.program, to); - } + let preview = On(self.preview_scene.as_deref(), preview_since); + let program = On(self.program_scene.as_deref(), program_since); + let switch = self.switch.as_ref().zip(linear).map(|(switch, linear)| { + let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); + (switch, from, linear) + }); + let extras = self.extras.iter().map(|(name, output)| (Some(name.as_str()), output)); + for (canvas, output) in std::iter::once((None, &self.main)).chain(extras) { + draw.canvas = canvas; + draw.size = output.canvas; + draw.output(output, preview, program, switch); } - draw.layer(&self.program, Place::Top, Duration::ZERO); self.effects.submit(frame); Ok(()) @@ -484,26 +522,31 @@ impl Compositor { } } - /// The sources whose pictures preview and program draw: each visible - /// item's, and each mask's, through clones. + /// The sources whose pictures preview and program draw on any canvas: + /// each visible item's, and each mask's, through clones. fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { let leaving = self.switch.as_ref().and_then(|switch| switch.from.as_deref()); - let scenes = [self.preview_scene.as_deref(), self.program_scene.as_deref(), leaving] + let scenes: Vec> = [self.preview_scene.as_deref(), self.program_scene.as_deref(), leaving] .into_iter() .flatten() - .map(|name| Place::Scene(name.to_string())); + .map(|name| Some(name.to_string())) + .chain([None]) + .collect(); + let canvases = std::iter::once(None).chain(self.extras.keys().map(Some)); let mut shown = HashSet::new(); - for place in scenes.chain([Place::Top]) { - for item in collection - .items(&place) - .into_iter() - .flatten() - .filter(|item| item.visible) - { - shown.insert(collection.capture(&item.source)); - for filter in collection.filters(item) { - if let Filter::Mask { source, .. } = filter { - shown.insert(collection.capture(source)); + for canvas in canvases { + for scene in &scenes { + let place = Place { + canvas: canvas.cloned(), + scene: scene.clone(), + }; + let items = collection.items(&place).unwrap_or_default(); + for item in items.iter().filter(|item| item.visible) { + shown.insert(collection.capture(&item.source)); + for filter in collection.filters(item) { + if let Filter::Mask { source, .. } = filter { + shown.insert(collection.capture(source)); + } } } } @@ -639,22 +682,91 @@ struct Draw<'a> { effects: &'a mut Effects, library: Library<'a>, frame: &'a mut Frame, + /// The canvas drawn now: a further one by name, or `None` for the main one. + canvas: Option<&'a str>, + /// Its size and frame rate. + size: Canvas, /// Each item, filtered once though it is drawn onto several targets, by - /// where it is and its [`Item::settled`] time, as two outputs showing its - /// scene can be at different points of its keyframes. + /// where it is, on which canvas, and its [`Item::settled`] time, as two + /// outputs showing its scene can be at different points of its keyframes. prepared: HashMap<(Place, usize, Settled), Option>, } impl Draw<'_> { + /// Draw `output`'s preview, showing `preview`, and its program, showing + /// `program`, or `switch` to it from the scene it leaves, `linear` of its + /// time along. + fn output(&mut self, output: &Output, preview: On, program: On, switch: Option<(&Switch, On, f32)>) { + let top = self.place(None); + // The top layer goes over both outputs, and over a transition rather + // than through it, so it stays on across the switch. + self.target(&output.preview); + self.scene(&output.preview, preview); + self.layer(&output.preview, top.clone(), Duration::ZERO); + + let to = program; + match switch { + Some((switch, from, linear)) => { + let progress = switch.transition.progress(linear); + let canvas = Rect::fill(&self.size); + match switch.transition { + Transition::Fade { .. } => { + self.whole(&output.incoming, to); + self.target(&output.program); + self.scene(&output.program, from); + self.picture(&output.program, &output.incoming, canvas, canvas, progress); + } + // Uncovered where a slide would have moved it in. + Transition::Wipe { direction, .. } => { + self.whole(&output.incoming, to); + self.target(&output.program); + self.scene(&output.program, from); + let (x, y) = offset(direction, progress, &canvas); + let (dx, dy) = offset(direction, 1.0, &canvas); + let uncovered = shifted(canvas, x - dx, y - dy); + self.picture(&output.program, &output.incoming, canvas, uncovered, 1.0); + } + Transition::Slide { direction, .. } => { + self.whole(&output.incoming, to); + self.whole(&output.outgoing, from); + self.target(&output.program); + let (x, y) = offset(direction, progress, &canvas); + let (dx, dy) = offset(direction, 1.0, &canvas); + self.picture(&output.program, &output.outgoing, shifted(canvas, x, y), canvas, 1.0); + let entering = shifted(canvas, x - dx, y - dy); + self.picture(&output.program, &output.incoming, entering, canvas, 1.0); + } + Transition::Move { .. } => { + self.target(&output.program); + self.moving(&output.program, from, to, progress); + } + } + } + None => { + self.target(&output.program); + self.scene(&output.program, to); + } + } + self.layer(&output.program, top, Duration::ZERO); + } + + /// `scene`'s layout on the canvas drawn now, or its top layer for `None`. + fn place(&self, scene: Option<&str>) -> Place { + Place { + canvas: self.canvas.map(str::to_string), + scene: scene.map(str::to_string), + } + } + /// Start drawing `target`. fn target(&mut self, target: &Target) { - self.frame.target(&target.picture); + self.frame.target(&target.picture, self.size); } /// Draw `on`'s items onto `target`. A missing scene draws nothing. fn scene(&mut self, target: &Target, on: On) { if let Some(scene) = on.0.filter(|name| self.collection.scene(name).is_some()) { - self.layer(target, Place::Scene(scene.to_string()), on.1); + self.layer(target, self.place(Some(scene)), on.1); } } @@ -666,8 +778,8 @@ impl Draw<'_> { /// Draw `picture`, a canvas-sized target drawn this render, onto `target` /// stretched over `at`, showing only within `within`, at `opacity`. - fn place(&mut self, target: &Target, picture: &Target, at: Rect, within: Rect, opacity: f32) { - if let Some(placed) = self.effects.place(&picture.picture, at, within, opacity) { + fn picture(&mut self, target: &Target, picture: &Target, at: Rect, within: Rect, opacity: f32) { + if let Some(placed) = self.effects.place(self.frame, &picture.picture, at, within, opacity) { self.effects.draw(self.frame, &target.texture, &placed); } } @@ -677,7 +789,7 @@ impl Draw<'_> { /// item, a source with no picture yet, or a filter whose source has nothing /// yet draws nothing. fn layer(&mut self, target: &Target, place: Place, since: Duration) { - let items = self.collection.items(&place).map_or(&[][..], Vec::as_slice); + let items = self.collection.items(&place).unwrap_or_default(); for (index, item) in items.iter().enumerate().filter(|(_, item)| item.visible) { self.item(target, (place.clone(), index, item.settled(since)), &item.posed(since)); } @@ -723,9 +835,9 @@ impl Draw<'_> { /// whose blend mode or kinds of filter differ changes look halfway too. fn moving(&mut self, target: &Target, from: On, to: On, progress: f32) { let collection = self.collection; - let items = |scene: Option<&str>| { - let scene = scene.and_then(|name| collection.scene(name)); - scene.map_or(&[][..], |scene| scene.items.as_slice()) + let items = |scene: Option<&str>| match scene { + Some(scene) => collection.items(&self.place(Some(scene))).unwrap_or_default(), + None => &[], }; let posed = |on: On| -> Vec> { items(on.0).iter().map(|item| item.posed(on.1)).collect() }; let resting = items(to.0); @@ -751,7 +863,7 @@ impl Draw<'_> { slots.into_iter().map(|(to, from)| (from, to)).collect() }; - let place = Place::Scene(to.0.unwrap_or_default().to_string()); + let place = self.place(Some(to.0.unwrap_or_default())); for slot in slots { match slot { (Some(at), None) => { @@ -890,7 +1002,7 @@ mod tests { use std::time::Duration; use super::*; - use crate::collection::{Blend, MaskMode, Rgba, Scene}; + use crate::collection::{Blend, Extra, MaskMode, Rgba, Scene}; use crate::source::test::{context, local, start}; /// A fade over a second. @@ -940,12 +1052,12 @@ mod tests { }; collection.scenes = vec![ Scene { - name: "red".into(), items: vec![item("red", fill), item("blue", corner)], + ..Scene::new("red") }, Scene { - name: "blue".into(), items: vec![item("blue", fill)], + ..Scene::new("blue") }, ]; collection.validate().unwrap(); @@ -953,18 +1065,18 @@ mod tests { } /// A compositor over running sources, read back to check pixels. - struct Studio { - collection: Collection, + pub(super) struct Studio { + pub(super) collection: Collection, sources: Sources, - compositor: Compositor, - device: wgpu::Device, + pub(super) compositor: Compositor, + pub(super) device: wgpu::Device, queue: wgpu::Queue, _origin: moq_net::origin::Producer, _dir: tempfile::TempDir, } impl Studio { - async fn new(collection: Collection) -> Self { + pub(super) async fn new(collection: Collection) -> Self { Self::with_files(collection, tempfile::tempdir().unwrap()).await } @@ -987,18 +1099,18 @@ mod tests { } } - fn render(&mut self, now: Instant) { + pub(super) fn render(&mut self, now: Instant) { self.compositor.render(&self.collection, &self.sources, now).unwrap(); } /// The canvas texture's exact RGBA rows: no 4:2:0 round trip, so a test /// compares the stored values. fn read(&mut self, texture: &wgpu::Texture) -> Vec { - let canvas = self.compositor.canvas(); - let row = (canvas.width * 4).next_multiple_of(wgpu::COPY_BYTES_PER_ROW_ALIGNMENT); + let (width, height) = (texture.width(), texture.height()); + let row = (width * 4).next_multiple_of(wgpu::COPY_BYTES_PER_ROW_ALIGNMENT); let buffer = self.device.create_buffer(&wgpu::BufferDescriptor { label: None, - size: u64::from(row * canvas.height), + size: u64::from(row * height), usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ, mapped_at_creation: false, }); @@ -1018,7 +1130,7 @@ mod tests { self.queue.submit([encoder.finish()]); buffer.map_async(wgpu::MapMode::Read, .., |result| result.unwrap()); self.device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); - let width = canvas.width as usize * 4; + let width = width as usize * 4; let mapped = buffer.slice(..).get_mapped_range().unwrap(); mapped .chunks_exact(row as usize) @@ -1194,6 +1306,66 @@ mod tests { .await; } + /// A further canvas draws its own layout of each scene and its own top + /// layer, from the same upload of each source, and cuts and fades with + /// the main canvas. A scene without a layout there shows only the top + /// layer, and a canvas the collection drops is no longer drawn. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_further_canvas_draws_its_own_layout() { + local(async { + let mut collection = collection(); + let mut tall = Extra::new(4, 8); + let bottom = Rect { + x: 0.0, + y: 4.0, + width: 4.0, + height: 4.0, + }; + tall.top.push(Item::new("red", bottom)); + collection.canvases.insert("tall".into(), tall); + let top = Rect { y: 0.0, ..bottom }; + collection.scenes[0] + .canvases + .insert("tall".into(), vec![Item::new("blue", top)]); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_canvases(&studio.collection).unwrap(); + let tall = |studio: &mut Studio| { + let texture = studio.compositor.output(Some("tall")).unwrap().program().clone(); + let rgba = studio.read(&texture); + let pixel = |x: usize, y: usize| -> [u8; 4] { rgba[(y * 4 + x) * 4..][..4].try_into().unwrap() }; + (pixel(1, 1), pixel(1, 6)) + }; + + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.render(start); + assert_eq!(pixel(&studio.program(), 1, 1), RED); + assert_eq!(tall(&mut studio), (BLUE, RED)); + assert_eq!(studio.compositor.layers.len(), 2, "each source is uploaded once"); + + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.transition(FADE, start); + studio.render(start + Duration::from_millis(500)); + let (fading, kept) = tall(&mut studio); + assert!(near(fading, [0, 0, 128, 255]), "{fading:?}"); + assert_eq!(kept, RED, "the top layer does not fade"); + + studio.render(start + Duration::from_secs(1)); + assert_eq!(pixel(&studio.program(), 1, 1), BLUE); + assert_eq!(tall(&mut studio), (BLACK, RED), "no layout leaves the top layer"); + + studio.collection.canvases.clear(); + studio.collection.scenes[0].canvases.clear(); + studio.compositor.set_canvases(&studio.collection).unwrap(); + assert!(studio.compositor.output(Some("tall")).is_none()); + assert_eq!(studio.compositor.outputs().count(), 1); + }) + .await; + } + /// A cut lands at once, even mid-fade, and an empty program is black. #[tokio::test] #[ignore = "needs a Vulkan device; run by just rs render-sw"] @@ -1356,6 +1528,69 @@ mod tests { ] } + /// A further canvas's items play their keyframes as the main canvas's do: + /// from when the scene went into preview, for its preview, and from when it + /// went on program, for its program, each drawn on its own canvas though + /// both canvases are at the same moment. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_further_canvas_plays_keyframes_with_the_main_one() { + local(async { + let mut collection = collection(); + collection.scenes[1].items[0].keyframes = slide_in(); + collection.canvases.insert("tall".into(), Extra::new(4, 8)); + let bottom = Rect { + x: 0.0, + y: 4.0, + width: 4.0, + height: 4.0, + }; + let mut slide = slide_in(); + slide[0].x = -4.0; + collection.scenes[1].canvases.insert( + "tall".into(), + vec![Item { + keyframes: slide, + ..Item::new("blue", bottom) + }], + ); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.set_canvases(&studio.collection).unwrap(); + let tall = |studio: &mut Studio, preview: bool| { + let output = studio.compositor.output(Some("tall")).unwrap(); + let texture = match preview { + true => output.preview().clone(), + false => output.program().clone(), + }; + let rgba = studio.read(&texture); + let pixel = |x: usize, y: usize| -> [u8; 4] { rgba[(y * 4 + x) * 4..][..4].try_into().unwrap() }; + (pixel(1, 6), pixel(3, 6)) + }; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + + studio.render(at(0)); + studio.render(at(500)); + assert_eq!(tall(&mut studio, true), (BLUE, BLACK), "halfway in, in preview"); + assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the main canvas too"); + + studio.compositor.cut(); + studio.render(at(500)); + assert_eq!(tall(&mut studio, false), (BLACK, BLACK), "on program from the start"); + assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "the main canvas too"); + assert_eq!(tall(&mut studio, true), (BLUE, BLACK), "the preview carries on"); + + studio.render(at(1500)); + assert_eq!(tall(&mut studio, false), (BLUE, BLUE), "landed on the further canvas"); + assert_eq!(pixel(&studio.program(), 6, 6), BLUE, "and on the main one"); + }) + .await; + } + /// An item's keyframes play from when its scene went into preview, for the /// preview, and from when it went on program, for the program, so one scene /// can be at two points of them at once. @@ -1948,9 +2183,9 @@ mod tests { width: 16, ..studio.collection.canvas }; - studio.compositor.set_canvas(wide).unwrap(); - assert_ne!(studio.compositor.program(), &old, "a new texture"); studio.collection.canvas = wide; + studio.compositor.set_canvases(&studio.collection).unwrap(); + assert_ne!(studio.compositor.program(), &old, "a new texture"); let fill = Rect::fill(&wide); studio.collection.scenes[0].items = vec![ Item::new("red", Rect { width: 8.0, ..fill }), @@ -1983,9 +2218,20 @@ mod tests { width: limit + 2, ..wide }; - assert!(matches!(studio.compositor.set_canvas(huge), Err(Error::Invalid(_)))); + let refused = |canvas: Canvas| { + let mut collection = studio.collection.clone(); + collection.canvas = canvas; + collection + }; + assert!(matches!( + studio.compositor.set_canvases(&refused(huge)), + Err(Error::Invalid(_)) + )); let empty = Canvas { width: 0, ..wide }; - assert!(matches!(studio.compositor.set_canvas(empty), Err(Error::Invalid(_)))); + assert!(matches!( + studio.compositor.set_canvases(&refused(empty)), + Err(Error::Invalid(_)) + )); assert_eq!(studio.compositor.canvas(), wide); }) .await; @@ -2197,3 +2443,95 @@ mod tests { .await; } } + +/// What drawing more canvases costs, against how many sources the collection +/// has and how many each canvas places. Run by hand: +/// `cargo nextest run -p moq-scene --run-ignored only --no-capture -E 'test(/^compositor::bench::/)'`. +#[cfg(test)] +mod bench { + use std::time::Duration; + + use super::tests::Studio; + use super::*; + use crate::collection::{Extra, Rgba}; + use crate::source::test::local; + + /// Frames timed per case, after as many to warm up. + const FRAMES: u32 = 60; + + /// A collection of `sources` color sources on a 1080p main canvas and + /// `canvases - 1` portrait ones. The main canvas places every source, and + /// each further canvas the first `placed`, each item a quarter of the + /// canvas at half opacity, so every one draws. + fn collection(sources: usize, canvases: usize, placed: usize) -> Collection { + let mut collection = Collection::new(); + let item = |index: usize, canvas: &Canvas| { + let quarter = Rect { + x: (index % 4) as f32 * canvas.width as f32 / 4.0, + y: (index / 4 % 4) as f32 * canvas.height as f32 / 4.0, + width: canvas.width as f32 / 2.0, + height: canvas.height as f32 / 2.0, + }; + Item { + filters: vec![Filter::Opacity { amount: 0.5 }], + ..Item::new(format!("color {index}"), quarter) + } + }; + for index in 0..sources { + let color = Rgba([(index * 37) as u8, (index * 91) as u8, (index * 13) as u8, 255]); + collection + .sources + .insert(format!("color {index}"), Source::Color { color }); + } + let main = collection.canvas; + collection.scenes[0].items = (0..sources).map(|index| item(index, &main)).collect(); + for canvas in 1..canvases { + let name = format!("portrait {canvas}"); + let extra = Extra::new(1080, 1920); + let size = extra.canvas(main.framerate); + collection.canvases.insert(name.clone(), extra); + let items = (0..placed.min(sources)).map(|index| item(index, &size)).collect(); + collection.scenes[0].canvases.insert(name, items); + } + collection.validate().unwrap(); + collection + } + + /// The mean time to draw and finish every canvas once. + async fn frame(collection: Collection) -> Duration { + let mut studio = Studio::new(collection).await; + studio.compositor.set_canvases(&studio.collection).unwrap(); + let scene = studio.collection.scenes[0].name.clone(); + studio.compositor.set_preview(Some(scene)); + studio.compositor.cut(); + let mut total = Duration::ZERO; + for frame in 0..FRAMES * 2 { + let start = Instant::now(); + studio.render(Instant::now()); + studio.device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); + if frame >= FRAMES { + total += Instant::now().saturating_duration_since(start); + } + } + total / FRAMES + } + + /// Sources against canvases, each further canvas placing every source + /// (dense) or two (sparse). A sparse canvas should cost about the same + /// however many sources the collection has. + #[tokio::test] + #[ignore = "a benchmark; run by hand on a GPU"] + async fn sources_against_canvases() { + local(async { + println!("sources canvases dense sparse"); + for sources in [4, 16, 64] { + for canvases in [1, 2, 4] { + let dense = frame(collection(sources, canvases, sources)).await; + let sparse = frame(collection(sources, canvases, 2)).await; + println!("{sources:>7} {canvases:>8} {dense:>7.2?} {sparse:>7.2?}"); + } + } + }) + .await; + } +} diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index 230d9db..fb00dc9 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -14,7 +14,9 @@ use std::collections::{BTreeMap, VecDeque}; use serde::{Deserialize, Serialize}; use crate::Error; -use crate::collection::{Audio, Canvas, Collection, Filter, Item, Layer, Mixes, Scene, Source, Transition, Video}; +use crate::collection::{ + Audio, Canvas, Collection, Extra, Filter, Item, Layer, Mixes, Scene, Source, Transition, Video, +}; /// One part of a collection, before and after an edit. /// @@ -70,6 +72,14 @@ pub enum Change { Transition { before: Transition, after: Transition }, /// [`Collection::canvas`]. Canvas { before: Canvas, after: Canvas }, + /// A [`Collection::canvases`] entry. + Extra { + name: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + before: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + after: Option, + }, /// [`Collection::mixes`]. Mixes { before: Mixes, after: Mixes }, } @@ -117,6 +127,11 @@ impl Change { before: after, after: before, }, + Self::Extra { name, before, after } => Self::Extra { + name, + before: after, + after: before, + }, Self::Mixes { before, after } => Self::Mixes { before: after, after: before, @@ -132,6 +147,7 @@ impl Change { | Self::Item { .. } | Self::Transition { .. } | Self::Canvas { .. } + | Self::Extra { .. } | Self::Mixes { .. } => None, } } @@ -146,6 +162,7 @@ impl Change { Self::Item { before, after, .. } => before == after, Self::Transition { before, after } => before == after, Self::Canvas { before, after } => before == after, + Self::Extra { before, after, .. } => before == after, Self::Mixes { before, after } => before == after, } } @@ -232,6 +249,18 @@ impl Change { (Self::Canvas { before, after }, Self::Canvas { before: b, after: a }) if after == *b => { Self::Canvas { before, after: *a } } + ( + Self::Extra { name, before, after }, + Self::Extra { + name: n, + before: b, + after: a, + }, + ) if name == *n && after == *b => Self::Extra { + name, + before, + after: a.clone(), + }, (Self::Mixes { before, after }, Self::Mixes { before: b, after: a }) if after == *b => Self::Mixes { before, after: a.clone(), @@ -257,10 +286,9 @@ impl Change { before, after, } => { - let items = collection - .items_mut(layer) - .ok_or_else(|| conflict(format!("there is no {layer}")))?; - splice(items, *index, before, after).map_err(|err| conflict(format!("item {err} in {layer}"))) + let spliced = collection.edit_items(layer, |items| splice(items, *index, before, after)); + let spliced = spliced.ok_or_else(|| conflict(format!("there is no {layer}")))?; + spliced.map_err(|err| conflict(format!("item {err} in {layer}"))) } Self::Transition { before, after } => { if collection.transition != *before { @@ -276,6 +304,7 @@ impl Change { collection.canvas = *after; Ok(()) } + Self::Extra { name, before, after } => replace(&mut collection.canvases, name, before, after, "canvas"), Self::Mixes { before, after } => { if collection.mixes != *before { return Err(conflict("the mixes changed".into())); @@ -647,6 +676,49 @@ impl Draft { self.change(Change::Canvas { before, after: canvas }) } + /// Add or resize the further canvas called `name`, keeping its top layer + /// and every scene's layout on it. Items keep their place in canvas + /// pixels. + pub fn set_extra(&mut self, name: &str, width: u32, height: u32) -> Result<(), Error> { + let before = self.collection.canvases.get(name).cloned(); + let after = match &before { + Some(extra) => Extra { + width, + height, + ..extra.clone() + }, + None => Extra::new(width, height), + }; + self.change(Change::Extra { + name: name.to_string(), + before, + after: Some(after), + }) + } + + /// Remove the further canvas called `name`, with its top layer and every + /// scene's layout on it. + pub fn remove_extra(&mut self, name: &str) -> Result<(), Error> { + let laid: Vec<(usize, Scene)> = self + .collection + .scenes + .iter() + .enumerate() + .filter(|(_, scene)| scene.canvases.contains_key(name)) + .map(|(index, scene)| (index, scene.clone())) + .collect(); + for (index, mut scene) in laid { + scene.canvases.remove(name); + self.set_scene(index, scene)?; + } + let before = self.collection.canvases.get(name).cloned(); + self.change(Change::Extra { + name: name.to_string(), + before, + after: None, + }) + } + /// Set the mixes and which ones the program and recordings carry, and /// nothing else: see [`rename_mix`](Self::rename_mix) and /// [`remove_mix`](Self::remove_mix) for what refers to a mix. @@ -967,7 +1039,7 @@ mod tests { } fn main() -> Layer { - Layer::Scene("Scene".into()) + Layer::scene("Scene") } /// A collection with a red source shown in the one scene. @@ -1272,6 +1344,100 @@ mod tests { assert!(matches!(draft.finish(), Err(Error::Invalid(_)))); } + /// A further canvas is added, laid out, and resized as edits; removing a + /// source takes its items off every canvas; and removing the canvas takes + /// its top layer and every scene's layout on it, undoing whole. A layout + /// emptied item by item reads as no layout, and the file round-trips. + #[test] + fn a_further_canvas_edits_and_removes_whole() { + let mut collection = collection(); + collection.sources.insert("blue".into(), color([0, 0, 255, 255])); + let mut history = History::new(); + let tall = Layer::scene("Scene").on(Some("tall".into())); + let over = Layer::top().on(Some("tall".into())); + let fill = Rect::fill(&Canvas { + width: 1080, + height: 1920, + framerate: 30, + }); + + let mut draft = collection.draft("Add a canvas"); + assert!( + draft.insert_item(&tall, 0, Item::new("red", fill)).is_err(), + "no canvas yet" + ); + draft.set_extra("tall", 1080, 1920).unwrap(); + draft.insert_item(&tall, 0, Item::new("red", fill)).unwrap(); + draft.insert_item(&tall, 1, Item::new("blue", fill)).unwrap(); + draft.insert_item(&over, 0, Item::new("blue", fill)).unwrap(); + commit(&mut collection, &mut history, draft, None); + let added = collection.clone(); + assert_eq!(collection.items(&tall).unwrap().len(), 2); + assert_eq!(collection.canvas(Some("tall")).unwrap().framerate, 30); + assert_eq!( + Collection::from_toml(&collection.to_toml().unwrap()).unwrap(), + collection + ); + + let mut draft = collection.draft("Resize"); + draft.set_extra("tall", 720, 1280).unwrap(); + commit(&mut collection, &mut history, draft, None); + assert_eq!(collection.canvases["tall"].top.len(), 1, "the top layer stays"); + + let mut draft = collection.draft("Remove blue"); + draft.remove_source("blue").unwrap(); + commit(&mut collection, &mut history, draft, None); + assert_eq!(collection.items(&tall).unwrap().len(), 1); + assert!(collection.items(&over).unwrap().is_empty()); + + let mut draft = collection.draft("Remove red there"); + draft.remove_item(&tall, 0).unwrap(); + commit(&mut collection, &mut history, draft, None); + assert!(collection.scenes[0].canvases.is_empty(), "an empty layout is dropped"); + + history.seek(&mut collection, 1).unwrap(); + assert_eq!(collection, added); + let mut draft = collection.draft("Remove the canvas"); + draft.remove_extra("tall").unwrap(); + commit(&mut collection, &mut history, draft, None); + assert!(collection.canvases.is_empty()); + assert!(collection.scenes[0].canvases.is_empty()); + assert_eq!(collection.items(&tall), None); + history.undo(&mut collection).unwrap(); + assert_eq!(collection, added); + } + + /// Emptying a scene's layout on a further canvas changes only that layer, + /// so a scene snapshot taken before still undoes: rename the scene, empty + /// its layout, then undo both. A refused item edit leaves no empty layout + /// behind in the draft. + #[test] + fn emptying_a_layout_keeps_earlier_scene_edits_undoable() { + let mut collection = collection(); + let fill = Rect::fill(&collection.canvas); + collection.canvases.insert("tall".into(), Extra::new(1080, 1920)); + collection.scenes[0] + .canvases + .insert("tall".into(), vec![Item::new("red", fill)]); + collection.validate().unwrap(); + let start = collection.clone(); + let mut history = History::new(); + + let mut draft = collection.draft("Rename"); + draft.rename_scene(0, "Renamed").unwrap(); + commit(&mut collection, &mut history, draft, None); + let tall = Layer::scene("Renamed").on(Some("tall".into())); + let mut draft = collection.draft("Remove red there"); + draft.remove_item(&tall, 0).unwrap(); + assert!(draft.insert_item(&tall, 5, Item::new("red", fill)).is_err()); + commit(&mut collection, &mut history, draft, None); + assert!(collection.scenes[0].canvases.is_empty()); + + history.undo(&mut collection).unwrap(); + history.undo(&mut collection).unwrap(); + assert_eq!(collection, start); + } + /// A renamed mix keeps its place, the program's and recordings' picks, and /// every source left out of it; a removed one takes those exclusions with /// it, sends recordings back to the mix on program, and undoes whole; and diff --git a/rs/moq-scene/src/effect.rs b/rs/moq-scene/src/effect.rs index b1a0ca1..70b9344 100644 --- a/rs/moq-scene/src/effect.rs +++ b/rs/moq-scene/src/effect.rs @@ -405,11 +405,14 @@ enum Command { pub(crate) struct Frame { rasters: Vec, targets: Vec, + /// The canvas of the target drawn now, which items are prepared for. + canvas: Canvas, } impl Frame { - /// Start drawing a canvas-sized target, cleared to opaque black. - pub fn target(&mut self, target: &Picture) { + /// Start drawing a target the size of `canvas`, cleared to opaque black. + pub fn target(&mut self, target: &Picture, canvas: Canvas) { + self.canvas = canvas; self.targets.push(Command::Pass { view: target.view.clone(), clear: Some(wgpu::Color::BLACK), @@ -458,7 +461,6 @@ pub(crate) trait Library { pub(crate) struct Effects { device: wgpu::Device, queue: wgpu::Queue, - canvas: Canvas, input_layout: wgpu::BindGroupLayout, extra_layout: wgpu::BindGroupLayout, @@ -475,7 +477,8 @@ pub(crate) struct Effects { none: wgpu::BindGroup, empty_2d: wgpu::TextureView, empty_3d: wgpu::TextureView, - /// What a blend mode mixes with: a copy of its target under it. + /// What a blend mode mixes with: a copy of its target under it, as wide + /// and as tall as the largest canvas. backdrop: wgpu::Texture, backdrop_view: wgpu::TextureView, @@ -485,13 +488,13 @@ pub(crate) struct Effects { used: Vec, } -/// What a blend mode mixes with, as large as the canvas. -fn backdrop(device: &wgpu::Device, canvas: Canvas) -> wgpu::Texture { +/// What a blend mode mixes with, `width` by `height`. +fn backdrop(device: &wgpu::Device, width: u32, height: u32) -> wgpu::Texture { device.create_texture(&wgpu::TextureDescriptor { label: Some("moq-scene backdrop"), size: wgpu::Extent3d { - width: canvas.width, - height: canvas.height, + width, + height, depth_or_array_layers: 1, }, mip_level_count: 1, @@ -504,7 +507,8 @@ fn backdrop(device: &wgpu::Device, canvas: Canvas) -> wgpu::Texture { } impl Effects { - pub fn new(device: &wgpu::Device, queue: &wgpu::Queue, canvas: Canvas) -> Self { + /// Effects drawn onto canvases that fit in `width` by `height`. + pub fn new(device: &wgpu::Device, queue: &wgpu::Queue, width: u32, height: u32) -> Self { let texture = |binding, sample_type, view_dimension| wgpu::BindGroupLayoutEntry { binding, visibility: wgpu::ShaderStages::FRAGMENT, @@ -622,7 +626,7 @@ impl Effects { }; let empty_2d = empty(wgpu::TextureDimension::D2, FORMAT); let empty_3d = empty(wgpu::TextureDimension::D3, wgpu::TextureFormat::Rgba32Float); - let backdrop = backdrop(device, canvas); + let backdrop = backdrop(device, width, height); let alignment = u64::from(device.limits().min_uniform_buffer_offset_alignment); let stride = (size_of::() as u64).next_multiple_of(alignment); @@ -636,7 +640,6 @@ impl Effects { none: extra_bind(device, &extra_layout, [&empty_3d, &empty_2d, &empty_2d]), device: device.clone(), queue: queue.clone(), - canvas, input_layout, extra_layout, uniform_layout, @@ -650,14 +653,13 @@ impl Effects { } } - /// Draw onto `canvas` from the next frame on, reallocating what is sized - /// from it. - pub fn set_canvas(&mut self, canvas: Canvas) { - if canvas.width != self.canvas.width || canvas.height != self.canvas.height { - self.backdrop = backdrop(&self.device, canvas); + /// Draw onto canvases that fit in `width` by `height` from the next frame + /// on, reallocating what is sized from them. + pub fn fit(&mut self, width: u32, height: u32) { + if (width, height) != (self.backdrop.width(), self.backdrop.height()) { + self.backdrop = backdrop(&self.device, width, height); self.backdrop_view = self.backdrop.create_view(&Default::default()); } - self.canvas = canvas; } /// `texture`, a source's picture or a canvas-sized target, bound to sample. @@ -816,7 +818,7 @@ impl Effects { /// nothing in `library` yet. pub fn prepare(&mut self, frame: &mut Frame, item: Item<'_>, library: &dyn Library) -> Option { let area = Area::of(item.rect); - let canvas = Area::canvas(self.canvas); + let canvas = Area::canvas(frame.canvas); if area.is_empty() { return None; } @@ -936,9 +938,9 @@ impl Effects { /// A whole canvas-sized `picture` stretched over `at`, showing only within /// `within`, at `opacity`, to draw over another: `None` if none of it /// shows on the canvas. - pub fn place(&self, picture: &Picture, at: Rect, within: Rect, opacity: f32) -> Option { + pub fn place(&self, frame: &Frame, picture: &Picture, at: Rect, within: Rect, opacity: f32) -> Option { let area = Area::of(at); - let bounds = area.intersect(&Area::of(within)).intersect(&Area::canvas(self.canvas)); + let bounds = area.intersect(&Area::of(within)).intersect(&Area::canvas(frame.canvas)); if bounds.is_empty() || opacity <= 0.0 { return None; } @@ -970,7 +972,7 @@ impl Effects { bounds, blend, } = prepared; - let (width, height) = (self.canvas.width as f32, self.canvas.height as f32); + let (width, height) = (frame.canvas.width as f32, frame.canvas.height as f32); let x = |px: f32| px / width * 2.0 - 1.0; let y = |px: f32| 1.0 - px / height * 2.0; let mut uniform = self.uniform(input, segment, item, bounds); @@ -1028,7 +1030,7 @@ impl Effects { /// `input` through `segment`, drawn into a raster covering `area`. fn raster(&mut self, frame: &mut Frame, input: &Input, segment: &Segment, item: Area, area: Area) -> Input { - let size = self.texels(&area); + let size = self.texels(frame.canvas, &area); let raster = self.acquire(size); let draw = Draw { kind: Kind::Raster, @@ -1061,7 +1063,7 @@ impl Effects { sigma: f32, ) -> Input { let mut current = self.raster(frame, input, segment, item, area); - let mut size = self.texels(&area); + let mut size = self.texels(frame.canvas, &area); let texels = |size: (u32, u32)| (size.0 as f32 / area.width(), size.1 as f32 / area.height()); // Halve until the blur is a few texels wide; each texel of a halving // averages four, so nothing aliases. @@ -1124,7 +1126,7 @@ impl Effects { offset: (f32, f32), color: Rgba, ) -> Input { - let raster = self.acquire(self.texels(&area)); + let raster = self.acquire(self.texels(frame.canvas, &area)); let uniform = Uniform { rect: FULL, input: picture.area.map(UNIT, &area), @@ -1153,14 +1155,15 @@ impl Effects { } } - /// The texels a raster covering `area` gets: one per canvas pixel, unless - /// that is more than twice the canvas or the device allows either way. - fn texels(&self, area: &Area) -> (u32, u32) { + /// The texels a raster covering `area` of `canvas` gets: one per canvas + /// pixel, unless that is more than twice the canvas or the device allows + /// either way. + fn texels(&self, canvas: Canvas, area: &Area) -> (u32, u32) { let limit = self.device.limits().max_texture_dimension_2d; - let most = |canvas: u32| (canvas * 2).min(limit) as f32; + let most = |side: u32| (side * 2).min(limit) as f32; let scale = 1f32 - .min(most(self.canvas.width) / area.width()) - .min(most(self.canvas.height) / area.height()); + .min(most(canvas.width) / area.width()) + .min(most(canvas.height) / area.height()); let texels = |length: f32| ((length * scale).ceil() as u32).max(1); (texels(area.width()), texels(area.height())) } diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index 41d5e51..f71e24a 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -3,9 +3,9 @@ //! The UI-free half of `moq-studio`, so the same scenes can be composed by a //! windowed app, a headless renderer, or a test: //! -//! - [`Collection`] is the scene graph: a [`Canvas`], named [`Source`]s, and -//! [`Scene`]s placing them, under a top layer that stays on across scene -//! switches. It saves to a versioned TOML file that holds no credentials. +//! - [`Collection`] is the scene graph: a main [`Canvas`] and any [`Extra`] +//! ones, named [`Source`]s, and [`Scene`]s placing them on each canvas, +//! under each canvas's top layer that stays on across scene switches. It saves to a versioned TOML file that holds no credentials. //! - [`Edit`] is every change to a collection, built with a [`Draft`]. Each //! one undoes itself, so a [`History`] undoes and redoes them and saves //! beside the scene file. A [`Clip`] carries items or filters between @@ -17,9 +17,10 @@ //! compositor and sound goes to a `moq_audio::bus::Bus`, routed into its //! program mix. //! - [`Compositor`] draws the scene in preview and the scene on program into -//! textures on the caller's `wgpu` device, cutting or running a -//! [`Transition`] between them, and plays each item's [`Keyframe`]s from -//! when its scene went on. `moq_video::render::export` hands the +//! textures on the caller's `wgpu` device, one pair per canvas from one +//! upload of each source's picture, cutting or running a [`Transition`] +//! between them on every canvas together, and plays each item's +//! [`Keyframe`]s from when its scene went on. `moq_video::render::export` hands the //! program to an encoder, on the GPU where the platform allows, and a //! [`Scaler`] draws it smaller for a smaller rendition. //! - [`publish`] gives each local source its own track, in a broadcast the @@ -47,8 +48,8 @@ pub mod source; pub use clip::Clip; pub use collection::{ - Audio, Blend, Canvas, Collection, Crop, Direction, Ease, Filter, Item, Keyframe, Layer, MaskMode, Mixes, Rect, - Rgba, Scene, Source, Transition, Video, + Audio, Blend, Canvas, Collection, Crop, Direction, Ease, Extra, Filter, Item, Keyframe, Layer, MaskMode, Mixes, + Rect, Rgba, Scene, Source, Transition, Video, }; pub use compositor::{Compositor, FORMAT}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 3a6ff0b..4cdc6b7 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -194,7 +194,7 @@ impl State { before.apply(edit)?; } keep_on_air(self.compositor.on_air(), &renames)?; - self.compositor.set_canvas(collection.canvas)?; + self.compositor.set_canvases(&collection)?; for renames in &renames { self.compositor.rename(renames); } @@ -203,15 +203,21 @@ impl State { Ok(()) } - /// The sources the program shows: placed or masking in a scene on air, - /// including one leaving in a transition, or in the top layer over them, a - /// clone by the source it shows. + /// The sources whose pictures any canvas's program draws, through clones: + /// each visible item's of the scenes on air, including one leaving in a + /// transition, and of the top layers over them, and each mask's it reads. fn shown(&self) -> BTreeSet { let collection = &self.collection; - let scenes = self.compositor.on_air().filter_map(|scene| collection.scene(scene)); - let items = scenes - .flat_map(|scene| &scene.items) - .chain(&collection.top) + let scenes: Vec> = self.compositor.on_air().map(Some).chain([None]).collect(); + let canvases = std::iter::once(None).chain(collection.canvases.keys().map(Some)); + let items = canvases + .flat_map(|canvas| { + scenes.iter().map(move |scene| moq_scene::Layer { + canvas: canvas.cloned(), + scene: scene.map(str::to_string), + }) + }) + .flat_map(|layer| collection.items(&layer).unwrap_or_default()) .filter(|item| item.visible); // A mask's picture shapes what the program draws too. let masks = |item| { @@ -289,8 +295,28 @@ fn keep_on_air<'a>( pub struct Stats { /// Canvas frames drawn. pub rendered: AtomicU64, - /// What each of the program's video encoders costs: the canvas size, the - /// half size, and the recording's own canvas size. + /// The main canvas's program video. + pub video: Arc, + /// Each further canvas's program video, by the canvas's name, while it is + /// drawn. + canvases: Mutex>>, + /// The video encoders open: every canvas's program and the sources'. + pub sessions: Sessions, + /// Each program track's share of the connection's bandwidth, by name: a + /// further canvas's prefixed with its own. + grants: Mutex>, + /// Why the audio stopped, if it did. + audio_error: Mutex>, + /// Why the recording's audio stopped, if it did, which leaves the published + /// audio running. + recording_audio_error: Mutex>, +} + +/// How one canvas's program video is doing. +#[derive(Default)] +pub struct VideoStats { + /// What each of its video encoders costs: the canvas size, the half size, + /// and, for the main canvas, the recording's own canvas size. pub renditions: [Load; 3], /// Whether the encoder reads the program on the GPU, rather than each frame /// being read back to the CPU. @@ -302,22 +328,13 @@ pub struct Stats { /// while on demand and unwatched, nor while a new rendition opens, nor /// while the grant cannot fund it. pub encoding: AtomicBool, - /// The video encoders open, the program's and the sources'. - pub sessions: Sessions, - /// How many sizes the program's video has now: the canvas size, and the - /// half size unless the canvas is too small or its encoder refused it. + /// How many sizes the video has now: the canvas size, and the half size + /// unless the canvas is too small or its encoder refused it. pub sizes: std::sync::atomic::AtomicUsize, - /// Each program track's share of the connection's bandwidth, by name. - grants: Mutex>, - /// The encoder backend, once the program's current rendition opened one. + /// The encoder backend, once the current rendition opened one. encoder: Mutex, /// Why the video stopped, if it did, until a new rendition starts. - video_error: Mutex>, - /// Why the audio stopped, if it did. - audio_error: Mutex>, - /// Why the recording's audio stopped, if it did, which leaves the published - /// audio running. - recording_audio_error: Mutex>, + error: Mutex>, } /// What one of the program's renditions costs. @@ -338,7 +355,7 @@ pub struct Load { pub encode_us: AtomicU64, /// Why this encoder stopped, or could not start, while the canvas size /// carries on, until the next canvas. The canvas size's own failure stops - /// the video, which [`Stats::error`] reports. + /// the video, which [`VideoStats::error`] reports. error: Mutex>, } @@ -354,23 +371,23 @@ impl Load { } } -/// The encoder backend the program's current rendition opened, if it has. +/// The encoder backend the current rendition opened, if it has. #[derive(Default)] struct EncoderName { - /// Counts the program's renditions, so one a resize retired while its - /// encoder was still opening cannot report that encoder afterwards. + /// Counts the renditions, so one a resize retired while its encoder was + /// still opening cannot report that encoder afterwards. rendition: u64, name: Option, } -impl Stats { - /// The encoder backend, once the program's current rendition opened one. +impl VideoStats { + /// The encoder backend, once the current rendition opened one. pub fn encoder(&self) -> Option { self.encoder.lock().unwrap().name.clone() } - /// The number of the program's current rendition, which its encode thread - /// reports its encoder under. + /// The number of the current rendition, which its encode thread reports + /// its encoder under. fn rendition(&self) -> u64 { self.encoder.lock().unwrap().rendition } @@ -383,7 +400,7 @@ impl Stats { encoder.name = None; } - /// Report `name` as the program's encoder, unless `rendition` was retired. + /// Report `name` as the encoder, unless `rendition` was retired. fn opened_encoder(&self, rendition: u64, name: &str) { let mut encoder = self.encoder.lock().unwrap(); if encoder.rendition == rendition { @@ -391,6 +408,23 @@ impl Stats { } } + /// Why the video stopped, if it did. + pub fn error(&self) -> Option { + self.error.lock().unwrap().clone() + } + + fn fail(&self, err: impl std::fmt::Display) { + tracing::error!(%err, "video stopped"); + *self.error.lock().unwrap() = Some(format!("video: {err}")); + } +} + +impl Stats { + /// Each further canvas's program video, by the canvas's name. + pub fn canvases(&self) -> BTreeMap> { + self.canvases.lock().unwrap().clone() + } + /// Each program track's share of the connection's bandwidth now, by name: /// `None` while the connection has no estimate. pub fn grants(&self) -> BTreeMap> { @@ -401,9 +435,9 @@ impl Stats { .collect() } - /// Why the output stopped, if it did. + /// Why the main canvas's output stopped, if it did. pub fn error(&self) -> Option { - let video = self.video_error.lock().unwrap().clone(); + let video = self.video.error(); let audio = self.audio_error.lock().unwrap().clone(); match (video, audio) { (Some(video), Some(audio)) => Some(format!("{video}; {audio}")), @@ -411,11 +445,6 @@ impl Stats { } } - fn fail_video(&self, err: impl std::fmt::Display) { - tracing::error!(%err, "video stopped"); - *self.video_error.lock().unwrap() = Some(format!("video: {err}")); - } - /// The recording's audio encoder failed, which stops what recordings hold but /// not the published program. fn fail_recording_audio(&self, err: impl std::fmt::Display) { @@ -537,7 +566,7 @@ impl Program { /// Encode the local sources' tracks on program, whether or not anything /// reads them, so a replay of one has its past: the pictures `shown` in a - /// scene on air, and the sound the program mix plays. A media file heard + /// scene on air on any canvas, and the sound the program mix plays. A media file heard /// and not shown encodes only its sound. fn on_program(&self, state: &State, shown: BTreeSet) { let program = &state.collection.mixes.program; @@ -693,6 +722,7 @@ impl Engine { browser: browser(&gpu)?, }); let mut compositor = Compositor::new(&gpu.device, &gpu.queue, collection.canvas)?; + compositor.set_canvases(&collection)?; compositor.set_preview(collection.scenes.first().map(|scene| scene.name.clone())); compositor.cut(); @@ -725,6 +755,7 @@ impl Engine { let Publish { renditions, ladder, + extras, audio, recorded_audio, clock, @@ -743,9 +774,33 @@ impl Engine { let feeds = crate::feed::Feeds::new(room.clone(), origin.clone(), program, clock.clone(), epoch)?; *engine.feeds.lock().unwrap() = Some(feeds); } + let main = Line::new(renditions, canvas, ladder, true); + // Opened before the program clock starts the sources, as the main + // canvas's encoder is, so each holds its encode session first. + let mut lines = Lines::new(); + let canvases: Vec<(String, Canvas)> = { + let state = engine.state.lock().unwrap(); + let collection = state.collection(); + let canvases = collection.canvases.keys(); + canvases + .filter_map(|name| Some((name.clone(), collection.canvas(Some(name))?))) + .collect() + }; + for (name, canvas) in canvases { + let video = Arc::new(VideoStats::default()); + engine + .stats + .canvases + .lock() + .unwrap() + .insert(name.clone(), video.clone()); + let opened = extras.open(&name, canvas, video.clone(), &engine.program).await; + let opened = opened.inspect_err(|err| video.fail(format!("{err:#}"))).ok(); + lines.insert(name, opened); + } let program = engine.clone(); let thread = clocked("program".into(), async move || { - program.run_program(start, epoch, renditions, ladder).await + program.run_program(start, epoch, main, extras, lines).await })?; engine.threads.lock().unwrap().push(thread); let mix = engine.clone(); @@ -908,7 +963,7 @@ impl Engine { /// What only the studio knows about its stream at `now`. fn local(&self, now: Instant) -> health::Local { let stats = &self.stats; - // A stalled picture matters where the program draws it: a scene on air, + // A stalled picture matters where any canvas's program draws it: a scene on air, // including one leaving in a transition, or the top layer over them. // Stalled sound matters only if the mix on program plays it, not one // only listened to. A source that stopped matters either way. @@ -943,68 +998,57 @@ impl Engine { .collect(); (stalled, stopped) }; - let canvas = &stats.renditions[0]; + let video = &stats.video; + let canvas = &video.renditions[0]; health::Local { rendered: stats.rendered.load(Ordering::Relaxed), dropped: canvas.dropped.load(Ordering::Relaxed), encoded: canvas.encoded.load(Ordering::Relaxed), - encoding: stats.encoding.load(Ordering::Relaxed), + encoding: video.encoding.load(Ordering::Relaxed), grants: stats.grants(), - watched: stats.watched.load(Ordering::Relaxed), - encoder: stats.encoder(), + watched: video.watched.load(Ordering::Relaxed), + encoder: video.encoder(), failed: stats.error(), stalled, stopped, } } - /// Draw a canvas frame per tick of the program clock, and hand the program - /// to each rendition's encoder, stamped `epoch` past `start`, while its gate - /// lets it encode: the canvas size every frame while live and not on - /// demand, otherwise only while anyone watches it, the half size only - /// while anyone watches it, and the recording's own only while recording. + /// Draw every canvas once per tick of the program clock, and hand each + /// canvas's program to its renditions' encoders, stamped `epoch` past + /// `start`, while their gates let them encode: the canvas size every frame + /// while live and not on demand, otherwise only while anyone watches it, + /// the half size only while anyone watches it, and the main canvas's + /// recording copy only while recording. /// - /// A new canvas size or frame rate ends `ladder`, the renditions it was - /// drawn at, and opens others in `renditions`, since a track never changes - /// what it carries. Drawing outlives the encoders: a failed output stops copying - /// the program, and the preview and program keep moving. + /// `main` encodes the main canvas, and `lines` the further canvases the + /// studio started with. Each further canvas added since gets a [`Line`] of + /// its own from `extras` once drawn, in a broadcast of its own, and each + /// ends once the collection drops its canvas. Drawing outlives the encoders: a failed + /// output stops copying its program, and the canvases keep moving. /// /// Runs until the studio [stops](Self::stop), and returns once every /// rendition's encode thread has finished its track. - async fn run_program(&self, start: Instant, epoch: Duration, renditions: Renditions, mut ladder: Option) { - let mut canvas = self.state.lock().unwrap().compositor.canvas(); - let mut clock = program_clock(start, canvas); - // One probe at a time, with the canvas it is for, since a GPU caps its - // encode sessions. - let mut probing: Option<(Canvas, std::thread::JoinHandle>)> = None; - // The encode threads of renditions ended or failed since, whose - // sessions the next probe waits to be closed. - let mut ending = Vec::new(); - // Whether the canvas drawn has its renditions, or failed to open them, - // so a failure is not retried until the canvas changes. - let mut settled = true; - if let Some(ladder) = &ladder { - ladder.adopted(&self.stats); - } + async fn run_program(&self, start: Instant, epoch: Duration, mut main: Line, extras: Extras, mut lines: Lines) { + let mut framerate = self.state.lock().unwrap().compositor.canvas().framerate; + let mut clock = program_clock(start, framerate); + // The encoders of lines since ended, waited on as the studio stops so + // each finishes its track. + let mut retired: Vec = Vec::new(); loop { let tick = clock.tick().await; if self.stopped.load(Ordering::Relaxed) { // Without a sender, each encoder publishes what it was sent and // still holds, then finishes its track. - ending.extend(ladder.take().into_iter().flat_map(|ladder| ladder.end(&self.stats))); - for thread in ending { - if thread.join().is_err() { - tracing::error!("an encode thread panicked"); - } - } - // It holds the threads of the renditions it waits on. - if let Some((_, probe)) = probing { - let _ = probe.join(); + retired.push(main.end()); + retired.extend(lines.into_values().flatten().map(|(line, _)| line.end())); + for ended in retired { + ended.join(); } return; } - let (program, drawn) = { + let drawn: Vec<(Option, wgpu::Texture, Canvas)> = { let mut guard = self.state.lock().unwrap(); let state = &mut *guard; if std::mem::take(&mut state.stale) { @@ -1022,80 +1066,52 @@ impl Engine { continue; } self.program.on_program(state, shown); - (state.compositor.program().clone(), state.compositor.canvas()) + state + .compositor + .outputs() + .map(|(name, output)| (name.map(str::to_string), output.program().clone(), output.canvas())) + .collect() }; self.stats.rendered.fetch_add(1, Ordering::Relaxed); - if drawn != canvas { - if drawn.framerate != canvas.framerate { - clock = program_clock(tick + period(drawn), drawn); - } - canvas = drawn; - settled = false; - // Ended before the next open, so the two never hold encode - // sessions at once. - ending.extend(ladder.take().into_iter().flat_map(|ladder| ladder.end(&self.stats))); - self.stats.retire_encoder(); - } - // Probed beside the program clock, so drawing carries on. - if let Some((probed, probe)) = probing.take_if(|(_, probe)| probe.is_finished()) { - let probe = probe - .join() - .unwrap_or_else(|_| Err(anyhow::anyhow!("the encoder probe panicked"))); - // One for a canvas since left is dropped unlisted. - if probed == canvas { - settled = true; - match probe.and_then(|probe| renditions.start(probe)) { - Ok((started, _)) => { - started.adopted(&self.stats); - ladder = Some(started); - } - Err(err) => self.stats.fail_video(format!("{err:#}")), - } - } - } - if probing.is_none() && !settled { - let ending = std::mem::take(&mut ending); - match Probe::spawn(renditions.gpu.clone(), canvas, ending) { - Ok(probe) => probing = Some((canvas, probe)), - Err(err) => { - settled = true; - self.stats.fail_video(format!("cannot start an encoder probe: {err}")); - } - } + let timestamp = moq_net::Timestamp::try_from(epoch + tick.saturating_duration_since(start)).ok(); + // Every canvas shares the main one's frame rate. + let drawing = drawn[0].2.framerate; + if drawing != framerate { + framerate = drawing; + clock = program_clock(tick + period(framerate), framerate); } - let Some(current) = &mut ladder else { - self.stats.watched.store(false, Ordering::Relaxed); - self.stats.encoding.store(false, Ordering::Relaxed); - continue; - }; - let watched = current - .rungs - .iter() - .any(|rung| rung.role.published() && rung.gate.watched()); - self.stats.watched.store(watched, Ordering::Relaxed); - let Ok(timestamp) = moq_net::Timestamp::try_from(epoch + tick.saturating_duration_since(start)) else { - continue; - }; - let mut encoding = false; - let mut failed = Vec::new(); - for (index, rung) in current.rungs.iter_mut().enumerate() { - let role = rung.role; - match rung.feed(&self.stats.renditions[role as usize], &program, timestamp) { - Some(step) => encoding |= role == Role::Canvas && matches!(step, Step::Encode(_)), - None => failed.push((index, role)), + retired.retain(|ended| !ended.is_finished()); + let gone: Vec = lines + .keys() + .filter(|line| !drawn.iter().any(|(name, _, _)| name.as_ref() == Some(*line))) + .cloned() + .collect(); + for name in gone { + if let Some(Some((line, _))) = lines.remove(&name) { + retired.push(line.end()); } + self.stats.canvases.lock().unwrap().remove(&name); } - self.stats.encoding.store(encoding, Ordering::Relaxed); - // An encoder stops only having reported why. Without the canvas - // size the video stops; without another rendition it carries on. - if failed.iter().any(|(_, role)| *role == Role::Canvas) { - ending.extend(ladder.take().into_iter().flat_map(|ladder| ladder.end(&self.stats))); - } else { - for (index, _) in failed.into_iter().rev() { - ending.push(current.remove(index, &self.stats)); - } + + for (name, program, canvas) in drawn { + let line = match &name { + None => &mut main, + Some(name) => { + let started = lines.entry(name.clone()).or_insert_with(|| { + let video = Arc::new(VideoStats::default()); + self.stats.canvases.lock().unwrap().insert(name.clone(), video.clone()); + extras + .start(name, canvas, video.clone(), &self.program) + .inspect_err(|err| video.fail(format!("{err:#}"))) + .ok() + }); + let Some((line, _)) = started else { continue }; + line + } + }; + line.feed(&program, canvas, timestamp); } } } @@ -1211,6 +1227,8 @@ struct Publish { /// The renditions of the canvas the studio started with, unless the canvas /// could not be encoded without a relay. ladder: Option, + /// What publishes each further canvas. + extras: Extras, /// The program audio's encoder. audio: moq_audio::encode::Producer, /// The recording broadcast's own audio encoder. @@ -1252,10 +1270,10 @@ async fn publish( // Probed now, so a machine that cannot encode the canvas fails at startup // rather than once the program clock starts. Without a relay the studio // still composes, previews, and records audio, and a new canvas tries again. - let probe = match Probe::new(gpu, canvas).await { + let probe = match Probe::new(gpu, canvas, true).await { Ok(probe) => Some(probe), Err(err) if live.is_none() => { - stats.fail_video(format!("{err:#}")); + stats.video.fail(format!("{err:#}")); None } Err(err) => return Err(err), @@ -1305,16 +1323,18 @@ async fn publish( gpu: gpu.clone(), broadcast: broadcast.clone(), timebase: catalog.timebase(), - recording: Recording { + recording: Some(Recording { broadcast: recorder.clone(), timebase: recorded.timebase(), - }, + }), bandwidth: bandwidth.clone(), // Live on a relay and not on demand; otherwise only a viewer asks for // the video. on_demand: live.is_none_or(|live| live.on_demand), live: live.is_some(), stats: stats.clone(), + video: stats.video.clone(), + grants: String::new(), }; let ladder = match probe { Some(probe) => { @@ -1369,7 +1389,7 @@ async fn publish( watching.clone().follow(live.reports.clone(), path.clone()), )); health::Publisher { - observer: hang::health::Observer::new(path.clone(), Some(instance)), + observer: hang::health::Observer::new(path.clone(), Some(instance.clone())), catalog: graded, clock: clock.clone(), relays: live.links.clone(), @@ -1381,7 +1401,7 @@ async fn publish( Ok(( Program { - _origin: origin, + _origin: origin.clone(), publication, catalog: listed, clock: clock.clone(), @@ -1392,6 +1412,17 @@ async fn publish( _audio: audio_reservation, }, Publish { + extras: Extras { + origin: origin.clone(), + path: path.clone(), + clock: clock.clone(), + instance, + gpu: gpu.clone(), + bandwidth, + on_demand: renditions.on_demand, + live: renditions.live, + stats: stats.clone(), + }, renditions, ladder, audio, @@ -1441,8 +1472,8 @@ struct Renditions { /// Shared by every rendition, so each lands on the catalog's clock the same /// way and a switch keeps the video in step with the audio. timebase: moq_mux::catalog::Timebase, - /// Where the recording's own encoder writes. - recording: Recording, + /// Where the recording's own encoder writes: the main canvas's alone. + recording: Option, /// The connection's bandwidth, which each published rendition follows its /// share of. bandwidth: bandwidth::Allocator, @@ -1451,6 +1482,11 @@ struct Renditions { /// Whether the program goes out on a relay, where viewers can watch it. live: bool, stats: Arc, + /// How this canvas's video is doing. + video: Arc, + /// What each track's grant is listed under in [`Stats::grants`], before + /// its name: empty for the main canvas, and a further one's name. + grants: String, } impl Renditions { @@ -1460,10 +1496,10 @@ impl Renditions { /// recording's once recorded. fn start(&self, probe: Probe) -> anyhow::Result<(Ladder, tokio::sync::oneshot::Receiver>)> { // Before the encode threads start, which report their own failures. - *self.stats.video_error.lock().unwrap() = None; + *self.video.error.lock().unwrap() = None; for (role, load) in [Role::Half, Role::Record] .into_iter() - .map(|role| (role, &self.stats.renditions[role as usize])) + .map(|role| (role, &self.video.renditions[role as usize])) { let refused = probe.refused.iter().find(|(refused, _)| *refused == role); *load.error.lock().unwrap() = refused.map(|(_, reason)| reason.clone()); @@ -1480,7 +1516,7 @@ impl Renditions { .collect(); let (opened, open) = tokio::sync::oneshot::channel(); let mut opened = Some(opened); - let number = self.stats.rendition(); + let number = self.video.rendition(); let mut rungs = Vec::with_capacity(probe.rungs.len()); for probed in probe.rungs { let Probed { @@ -1489,8 +1525,9 @@ impl Renditions { config, video, } = probed; - let (broadcast, timebase) = match role { - Role::Record => (&self.recording.broadcast, &self.recording.timebase), + let (broadcast, timebase) = match (role, &self.recording) { + (Role::Record, Some(recording)) => (&recording.broadcast, &recording.timebase), + (Role::Record, None) => anyhow::bail!("only the main canvas is recorded"), _ => (&self.broadcast, &self.timebase), }; let reserved = timebase.reserve().with_interval(config.gop.duration(config.framerate)); @@ -1507,16 +1544,19 @@ impl Renditions { let info = reserved.track_info(priority).with_refresh(ladder::REFRESH); let track = broadcast.unique_track(suffix, info)?; let producer = moq_video::encode::Producer::with_track(track.clone(), reserved, video)?; - let gate = match role { + let (gate, grant) = match role { // Reads nothing from the connection, so it always encodes at its // ceiling, and only while recorded. - Role::Record => Gate::new( - &bandwidth::Allocator::unlimited(), - producer.demand(), - ladder::ceiling(config.size(), config.framerate), - None, - false, - ), + Role::Record => { + let gate = Gate::new( + &bandwidth::Allocator::unlimited(), + producer.demand(), + ladder::ceiling(config.size(), config.framerate), + None, + false, + ); + (gate, None) + } _ => { let gate = Gate::new( &self.bandwidth, @@ -1525,19 +1565,17 @@ impl Renditions { ceilings.get(rank + 1).copied(), role == Role::Canvas && !self.on_demand, ); - self.stats - .grants - .lock() - .unwrap() - .insert(track.name().to_string(), gate.grant()); - gate + let grant = format!("{}{}", self.grants, track.name()); + self.stats.grants.lock().unwrap().insert(grant.clone(), gate.grant()); + (gate, Some(grant)) } }; // One more than the exporter's pool, for a skip behind a full pool. let (frames, pending) = mpsc::sync_channel(Exporter::DEPTH + 1); let job = Job { - stats: self.stats.clone(), + sessions: self.stats.sessions.clone(), + video: self.video.clone(), role, rendition: (role == Role::Canvas).then_some(number), zero_copy: exporter.route() == Route::Gpu, @@ -1561,6 +1599,7 @@ impl Renditions { rungs.push(Rung { role, gate, + grant, scaler, exporter, frames, @@ -1594,16 +1633,17 @@ struct Probed { impl Probe { /// Open a throwaway encoder of each of `canvas`'s renditions on what `gpu` - /// exports, which also settles whether the encoder reads the program on - /// the GPU. The canvas size picks the encoder, which the others use too. - async fn new(gpu: &Gpu, canvas: Canvas) -> anyhow::Result { + /// exports, the recording's copy only if `record`, which also settles + /// whether the encoder reads the program on the GPU. The canvas size picks + /// the encoder, which the others use too. + async fn new(gpu: &Gpu, canvas: Canvas, record: bool) -> anyhow::Result { let framerate = canvas.framerate(); let size = moq_video::Size::new(canvas.width, canvas.height); let sizes = ladder::sizes(size); let roles = [Role::Canvas, Role::Half] .into_iter() .zip(sizes) - .chain([(Role::Record, size)]); + .chain(record.then_some((Role::Record, size))); let mut kind = moq_video::encode::Kind::Auto; let mut rungs = Vec::new(); let mut refused = Vec::new(); @@ -1639,6 +1679,7 @@ impl Probe { fn spawn( gpu: Gpu, canvas: Canvas, + record: bool, ending: Vec>, ) -> std::io::Result>> { std::thread::Builder::new().name("probe".into()).spawn(move || { @@ -1647,11 +1688,312 @@ impl Probe { let _ = thread.join(); } let runtime = tokio::runtime::Builder::new_current_thread().enable_all().build()?; - runtime.block_on(Self::new(&gpu, canvas)) + runtime.block_on(Self::new(&gpu, canvas, record)) }) } } +/// One canvas's program video as the program clock feeds it: the renditions +/// of the canvas it last drew, the probe opening the next, and the encoders +/// ending. +struct Line { + renditions: Renditions, + ladder: Option, + /// The canvas `ladder` is for, or will be once probed. + canvas: Canvas, + /// One probe at a time, with the canvas it is for, since a GPU caps its + /// encode sessions. + probing: Option<(Canvas, std::thread::JoinHandle>)>, + /// The encode threads of renditions ended or failed since, whose sessions + /// the next probe waits to be closed. + ending: Vec>, + /// Whether the canvas drawn has its renditions, or failed to open them, so + /// a failure is not retried until the canvas changes. + settled: bool, +} + +/// What a [`Line`] leaves running once ended, to wait on. +struct Ended { + threads: Vec>, + /// Holds the encode threads it was waiting on. + probe: Option>>, +} + +impl Ended { + fn is_finished(&self) -> bool { + self.threads.iter().all(std::thread::JoinHandle::is_finished) + && self.probe.as_ref().is_none_or(std::thread::JoinHandle::is_finished) + } + + fn join(self) { + for thread in self.threads { + if thread.join().is_err() { + tracing::error!("an encode thread panicked"); + } + } + if let Some(probe) = self.probe { + let _ = probe.join(); + } + } +} + +impl Line { + /// `renditions` of `canvas`: `ladder` when already open, or `None` to probe + /// for them on the first feed unless `settled`, as a canvas that failed + /// to open at startup is. + fn new(renditions: Renditions, canvas: Canvas, ladder: Option, settled: bool) -> Self { + if let Some(ladder) = &ladder { + ladder.adopted(&renditions.video); + } + Self { + renditions, + ladder, + canvas, + probing: None, + ending: Vec::new(), + settled, + } + } + + /// Hand `program`, drawn at `drawn`, to each rendition its gate lets + /// encode, stamped `timestamp`. A new canvas size or frame rate ends the + /// renditions and probes for others, since a track never changes what it + /// carries. + fn feed(&mut self, program: &wgpu::Texture, drawn: Canvas, timestamp: Option) { + let renditions = &self.renditions; + let video = &renditions.video; + if drawn != self.canvas { + self.canvas = drawn; + self.settled = false; + // Ended before the next open, so the two never hold encode sessions + // at once. + self.ending + .extend(self.ladder.take().into_iter().flat_map(|ladder| ladder.end(renditions))); + video.retire_encoder(); + } + // Probed beside the program clock, so drawing carries on. + if let Some((probed, probe)) = self.probing.take_if(|(_, probe)| probe.is_finished()) { + let probe = probe + .join() + .unwrap_or_else(|_| Err(anyhow::anyhow!("the encoder probe panicked"))); + // One for a canvas since left is dropped unlisted. + if probed == self.canvas { + self.settled = true; + match probe.and_then(|probe| renditions.start(probe)) { + Ok((started, _)) => { + started.adopted(video); + self.ladder = Some(started); + } + Err(err) => video.fail(format!("{err:#}")), + } + } + } + if self.probing.is_none() && !self.settled { + let ending = std::mem::take(&mut self.ending); + let record = renditions.recording.is_some(); + match Probe::spawn(renditions.gpu.clone(), self.canvas, record, ending) { + Ok(probe) => self.probing = Some((self.canvas, probe)), + Err(err) => { + self.settled = true; + video.fail(format!("cannot start an encoder probe: {err}")); + } + } + } + + let Some(current) = &mut self.ladder else { + video.watched.store(false, Ordering::Relaxed); + video.encoding.store(false, Ordering::Relaxed); + return; + }; + let watched = current + .rungs + .iter() + .any(|rung| rung.role.published() && rung.gate.watched()); + video.watched.store(watched, Ordering::Relaxed); + let Some(timestamp) = timestamp else { return }; + let mut encoding = false; + let mut failed = Vec::new(); + for (index, rung) in current.rungs.iter_mut().enumerate() { + let role = rung.role; + match rung.feed(&video.renditions[role as usize], program, timestamp) { + Some(step) => encoding |= role == Role::Canvas && matches!(step, Step::Encode(_)), + None => failed.push((index, role)), + } + } + video.encoding.store(encoding, Ordering::Relaxed); + // An encoder stops only having reported why. Without the canvas size + // the video stops; without another rendition it carries on. + if failed.iter().any(|(_, role)| *role == Role::Canvas) { + self.ending + .extend(self.ladder.take().into_iter().flat_map(|ladder| ladder.end(renditions))); + } else { + for (index, _) in failed.into_iter().rev() { + self.ending.push(current.remove(index, renditions)); + } + } + } + + /// Stop handing the renditions frames, so each encode thread publishes + /// what it holds and finishes its track. + fn end(mut self) -> Ended { + let mut threads = std::mem::take(&mut self.ending); + threads.extend( + self.ladder + .take() + .into_iter() + .flat_map(|ladder| ladder.end(&self.renditions)), + ); + self.renditions.video.watched.store(false, Ordering::Relaxed); + self.renditions.video.encoding.store(false, Ordering::Relaxed); + Ended { + threads, + probe: self.probing.map(|(_, probe)| probe), + } + } +} + +/// Each further canvas's line and broadcast, by name, or `None` once it failed +/// to start, which is not retried until the canvas is removed and added again. +type Lines = BTreeMap>; + +/// A further canvas's broadcast, closed when dropped. +struct Broadcast { + publication: moq_net::origin::Publication, + /// Lists the canvas's video and the main broadcast's audio. + _catalog: moq_scene::publish::Catalog, +} + +impl Drop for Broadcast { + fn drop(&mut self) { + self.publication.close(); + } +} + +/// What starts each further canvas's broadcast: beneath the main one's path, +/// on its clock, sharing its connection's bandwidth. +struct Extras { + origin: moq_net::origin::Producer, + path: moq_net::Path, + clock: moq_mux::Clock, + /// The main broadcast's, which its audio is listed by. + instance: moq_net::Instance, + gpu: Gpu, + bandwidth: bandwidth::Allocator, + on_demand: bool, + live: bool, + stats: Arc, +} + +impl Extras { + /// Publish the further canvas `name`, first drawn at `canvas`, and the + /// [`Line`] that encodes its video, reporting in `video`. Its encoders are + /// probed for on the line's first feed, off the program clock, for a canvas + /// added while the studio runs. + fn start( + &self, + name: &str, + canvas: Canvas, + video: Arc, + program: &Program, + ) -> anyhow::Result<(Line, Broadcast)> { + let (renditions, broadcast) = self.publish(name, video, program)?; + Ok((Line::new(renditions, canvas, None, false), broadcast)) + } + + /// [`start`](Self::start), opening the canvas size's encoder before + /// returning, as the main canvas does at startup, so a canvas in the scene + /// file holds its encode session before any source opens one. A canvas + /// whose encoder cannot open draws on without video, saying why in `video`. + async fn open( + &self, + name: &str, + canvas: Canvas, + video: Arc, + program: &Program, + ) -> anyhow::Result<(Line, Broadcast)> { + let (renditions, broadcast) = self.publish(name, video.clone(), program)?; + let opened = async { + let probe = Probe::new(&self.gpu, canvas, false).await?; + let (ladder, opened) = renditions.start(probe)?; + opened.await.context("the encode thread stopped")??; + anyhow::Ok(ladder) + }; + let ladder = match opened.await { + Ok(ladder) => Some(ladder), + Err(err) => { + video.fail(format!("{err:#}")); + None + } + }; + Ok((Line::new(renditions, canvas, ladder, true), broadcast)) + } + + /// Publish the further canvas `name` at its own path beneath the main + /// broadcast's, listing the main broadcast's audio by reference so it is + /// encoded once, and where its video goes, reporting in `video`. + fn publish( + &self, + name: &str, + video: Arc, + program: &Program, + ) -> anyhow::Result<(Renditions, Broadcast)> { + let path = moq_net::Path::new(name) + .and_then(|name| self.path.join(&name)) + .with_context(|| format!("cannot publish canvas `{name}`"))?; + if self.live { + let (_, echo) = health::echo(&self.path)?; + anyhow::ensure!( + path != echo, + "canvas `{name}` would publish where viewers report back; remove it and add it under another name" + ); + } + let audio = self + .path + .relative(&path) + .with_context(|| format!("canvas `{name}` cannot refer to the main broadcast"))?; + + let publication = self + .origin + .create_broadcast(&path) + .with_context(|| format!("cannot create the broadcast of canvas `{name}`"))?; + let mut broadcast = publication.broadcast().clone(); + let config = moq_mux::catalog::Config::default() + .with_catalog(Default::default()) + .with_clock(self.clock.clone()); + let mut catalog = moq_scene::publish::Catalog::new(&mut broadcast, config)?; + let listed = program.catalog.snapshot().audio.renditions; + catalog.mutate(|catalog| { + for (id, mut config) in listed { + config.broadcast = Some(audio.clone()); + config.instance = Some(self.instance.clone()); + config.track.get_or_insert_with(|| id.clone()); + catalog.audio.renditions.insert(id, config); + } + })?; + publication + .announce(moq_net::origin::Route::default().with_instance(moq_net::Instance::random(), 0)) + .with_context(|| format!("cannot announce the broadcast of canvas `{name}`"))?; + + let renditions = Renditions { + gpu: self.gpu.clone(), + broadcast, + timebase: catalog.timebase(), + recording: None, + bandwidth: self.bandwidth.clone(), + on_demand: self.on_demand, + live: self.live, + stats: self.stats.clone(), + video, + grants: format!("{name}/"), + }; + let broadcast = Broadcast { + publication, + _catalog: catalog, + }; + Ok((renditions, broadcast)) + } +} + /// The program video of one canvas: each of its renditions, the canvas size /// first. struct Ladder { @@ -1659,28 +2001,30 @@ struct Ladder { } impl Ladder { - /// Report these renditions as the program's from now on. - fn adopted(&self, stats: &Stats) { + /// Report these renditions as the canvas's from now on. + fn adopted(&self, video: &VideoStats) { let zero_copy = self.rungs[0].exporter.route() == Route::Gpu; - stats.zero_copy.store(zero_copy, Ordering::Relaxed); - self.count(stats); + video.zero_copy.store(zero_copy, Ordering::Relaxed); + self.count(video); } - /// Report how many sizes the program publishes. - fn count(&self, stats: &Stats) { + /// Report how many sizes the canvas publishes. + fn count(&self, video: &VideoStats) { let sizes = self.rungs.iter().filter(|rung| rung.role.published()).count(); - stats.sizes.store(sizes, Ordering::Relaxed); + video.sizes.store(sizes, Ordering::Relaxed); } /// Stop handing the renditions frames, so each encode thread encodes those /// it holds, finishes its track, and returns. - fn end(self, stats: &Stats) -> Vec> { - stats.sizes.store(0, Ordering::Relaxed); - let mut grants = stats.grants.lock().unwrap(); + fn end(self, renditions: &Renditions) -> Vec> { + renditions.video.sizes.store(0, Ordering::Relaxed); + let mut grants = renditions.stats.grants.lock().unwrap(); self.rungs .into_iter() .map(|rung| { - grants.remove(rung.gate.track()); + if let Some(grant) = &rung.grant { + grants.remove(grant); + } rung.thread }) .collect() @@ -1689,10 +2033,12 @@ impl Ladder { /// Stop handing rendition `index` frames, as [`end`](Self::end) does. /// The canvas size left alone no longer disables below the half size, /// since no viewer has a smaller one to move to. - fn remove(&mut self, index: usize, stats: &Stats) -> std::thread::JoinHandle<()> { + fn remove(&mut self, index: usize, renditions: &Renditions) -> std::thread::JoinHandle<()> { let rung = self.rungs.remove(index); - stats.grants.lock().unwrap().remove(rung.gate.track()); - self.count(stats); + if let Some(grant) = &rung.grant { + renditions.stats.grants.lock().unwrap().remove(grant); + } + self.count(&renditions.video); if rung.role == Role::Half { for canvas in self.rungs.iter_mut().filter(|rung| rung.role == Role::Canvas) { canvas.gate.alone(); @@ -1707,6 +2053,8 @@ impl Ladder { struct Rung { role: Role, gate: Gate, + /// Its grant's name in [`Stats::grants`]: a published rendition's alone. + grant: Option, /// Draws the program at the rendition's size; `None` at the canvas size. scaler: Option, exporter: Exporter, @@ -1755,15 +2103,15 @@ impl Rung { } } -/// How often the program is drawn at `canvas`'s frame rate, from `start`. -fn program_clock(start: Instant, canvas: Canvas) -> kio::clock::Interval { - let mut clock = kio::clock::interval_at(start, period(canvas)); +/// How often the program is drawn at `framerate`, from `start`. +fn program_clock(start: Instant, framerate: u32) -> kio::clock::Interval { + let mut clock = kio::clock::interval_at(start, period(framerate)); clock.set_missed_tick_behavior(kio::clock::MissedTickBehavior::Skip); clock } -fn period(canvas: Canvas) -> Duration { - Duration::from_secs(1) / canvas.framerate +fn period(framerate: u32) -> Duration { + Duration::from_secs(1) / framerate } /// Connect to every relay, publishing `origin` to each and feeding `subscribe` @@ -1825,7 +2173,9 @@ enum Video { /// What an encode thread runs. struct Job { - stats: Arc, + sessions: Sessions, + /// How its canvas's video is doing. + video: Arc, /// Which of the program's renditions it is. role: Role, /// The number of the program's renditions to report the encoder under: @@ -1846,9 +2196,9 @@ struct Encoder { track: Encoding, config: moq_video::encode::Config, joins: Joins, - /// Where to report each encoder opened as the program's, under the - /// number of the program's renditions: the canvas size's alone. - report: Option<(Arc, u64)>, + /// Where to report each encoder opened as its canvas's, under the + /// number of the canvas's renditions: the canvas size's alone. + report: Option<(Arc, u64)>, /// Keep the encoder, and its session, while skipping frames. hold: bool, /// Whether the open encoder can cut a keyframe for a joining viewer. @@ -1951,7 +2301,8 @@ impl Encoder { /// track that went away unfinished. fn encode(job: Job, opened: Option>>, video: mpsc::Receiver