diff --git a/README.md b/README.md index 599e07d..e91d0bf 100644 --- a/README.md +++ b/README.md @@ -76,9 +76,8 @@ canvas and no sources. The proposed scene is the first edit in the history, so undoing it leaves that empty scene. Setup writes an ordinary scene file. Change its sources and scenes in their -panels, and its canvas in the file, which takes a restart. The Output panel -shows the canvas in use, and the Health panel shows how the uplink holds up -once you are live. +panels, and its canvas in the Output panel. The Health panel shows how the +uplink holds up once you are live. ## Panels @@ -86,8 +85,8 @@ Every panel is a tab you can drag, split, or stack. | Panel | Use it to | | --- | --- | -| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. **Cut** puts the preview on program at once; **Fade** blends over the collection's fade time. | -| Scenes | Pick the scene in preview, add one, or remove one. | +| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. **Cut** puts the preview on program at once; **Fade** blends over the fade time set beside it. | +| Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still fading out, cannot be removed, even by an undo, until program cuts away from it. | | History | See every edit, and undo or redo back to any of them. | | Items | Place, show, hide, and order what the preview scene or the top layer draws, back to front. Select an item to filter or copy it. | | Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. | @@ -95,7 +94,7 @@ Every panel is a tab you can drag, split, or stack. | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. | | Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | | Audio | Set each audio source's gain and mute, and watch its meter and the program's. | -| Output | See the broadcast path, whether the program is encoding, the canvas, and the encoder's work. | +| Output | Set the canvas size and frame rate, and see the broadcast path, whether the program is encoding, and the encoder's work. | | Health | See how the stream reaches viewers, relay by relay and track by track, and the cause of each problem. | A new source, from the Sources or Devices panel, is placed over whatever the @@ -304,7 +303,7 @@ leaving half of it, and a collection that has not changed is not rewritten, so a hand-written file keeps its comments and layout until the next edit in the studio. A failed write shows in the status bar and is retried every second; until one succeeds, a crash loses every edit since the -last write. A new canvas size or frame rate takes a restart. +last write. The file can be edited by hand, or by another studio, while the studio runs. The studio looks at it every second and before each write. With no edits of @@ -333,6 +332,14 @@ with Opus stereo audio, under the `--broadcast` path. Timestamps come from the studio's own clock, so a source that stalls repeats its last picture instead of holding up the stream. +A new canvas size or frame rate, from the Output panel, an undo, or a reload, +takes effect live. A track never changes what it carries, so the old +rendition's track ends and a new one is listed in its place in a catalog +update, on the same timeline, and viewers switch to it; the video pauses while +the new encoder opens. Items keep their place in canvas pixels. A canvas the +encoder cannot take stops the video, saying why, until the canvas changes +again. + The program encodes whether or not anyone watches, so a viewer who arrives late can fetch the recent past. `--on-demand` copies and encodes the video only while someone watches it, which saves that work but leaves nothing from before diff --git a/quest/m0/README.md b/quest/m0/README.md index 461ee6b..6caa376 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -83,7 +83,6 @@ repository. - [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux - [Viewer-side composition](/quest/m0/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers, with a server-composed program as the fallback - [The program encoder drains](/quest/m0/program-drain.md) - a hardware encoder never loses the program's last frames at exit or on pause -- [Scene editing](/quest/m0/scene-editing.md) - rename and reorder scenes, and resize the canvas while live through a new rendition - [Broadcast sources play in sync](/quest/m0/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker - [Live encode control](/quest/m0/encode-control.md) - simulcast rungs that follow the bandwidth grant, live bitrate and resolution, keyframes on request diff --git a/quest/m0/encode-control.md b/quest/m0/encode-control.md index 53ade6b..dfc71f6 100644 --- a/quest/m0/encode-control.md +++ b/quest/m0/encode-control.md @@ -26,4 +26,3 @@ track's share of the allocator. - [Simulcast rung disable](https://github.com/moq-dev/moq/blob/main/quest/m1/simulcast-rung-disable.md) - grant-driven rung disable this builds on - [moq-audio follows the grant](https://github.com/moq-dev/moq/blob/main/quest/m2/2848-follow-the-bandwidth-grant-in-moq-audio-instead-of.md) - the audio half -- [Scene editing](/quest/m0/scene-editing.md) - canvas resize through a new rendition diff --git a/quest/m0/multi-canvas.md b/quest/m0/multi-canvas.md index c63e4eb..86c4080 100644 --- a/quest/m0/multi-canvas.md +++ b/quest/m0/multi-canvas.md @@ -12,7 +12,9 @@ captures or decodes once however many canvases use it. Each canvas publishes its own broadcast, because different content needs a different name. Each canvas has its own top layer, which the collection's `top` holds for the one canvas today. Transitions are per canvas, so a scene change cuts every -canvas together unless the creator unlinks one. +canvas together unless the creator unlinks one. Reuse the one canvas's resize +path per canvas: the compositor reallocates on a new canvas, and the program +video publishes a rendition at a time, ending one and listing the next. Benchmark source count against canvas count, including sparse reuse. Per-canvas cost must grow only with the sources that canvas uses, never with the whole diff --git a/quest/m0/program-drain.md b/quest/m0/program-drain.md index f8efe16..e4df0f7 100644 --- a/quest/m0/program-drain.md +++ b/quest/m0/program-drain.md @@ -3,7 +3,8 @@ ## Goal The program's encoder drains its buffered pictures before it stops at exit -and before the studio publishes the gap marker of an on-demand pause, so a +or when a new canvas ends its rendition, and before the studio publishes the +gap marker of an on-demand pause, so a pipelined hardware encoder never loses the program's last frames. ## Plan diff --git a/quest/m0/scene-editing.md b/quest/m0/scene-editing.md deleted file mode 100644 index 52f98ec..0000000 --- a/quest/m0/scene-editing.md +++ /dev/null @@ -1,23 +0,0 @@ -# [M] Scene editing - -## Goal - -In `moq-studio`, scenes can be renamed and reordered, the transition's -settings (such as the fade time) can be edited as undoable steps, and the -canvas can be resized while live, with no restart. - -## Plan - -Decided 2026-10-26 (maintainer): a resize while publishing adds a new video -rendition under a new track name in a catalog update, and viewers switch to -it. A track name never changes what it means (AGENTS.md), so the old track -ends rather than changing resolution mid-stream. The compositor reallocates -its canvas textures and the program encoder restarts on the new size. - -A rename updates every reference to the scene in the collection. Reordering -is the order of the scene file's list. Update -`README.md`. - -## Related - -- [Multiple canvases](/quest/m0/multi-canvas.md) - canvases of different sizes at once, sharing the resize path diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 93f5aea..296ea80 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -298,7 +298,7 @@ impl Canvas { moq_video::FrameRate::new(self.framerate, 1).expect("validated canvas framerate") } - fn validate(&self) -> Result<(), Error> { + pub(crate) fn validate(&self) -> Result<(), Error> { if self.width == 0 || self.height == 0 || !self.width.is_multiple_of(2) || !self.height.is_multiple_of(2) { return Err(Error::Invalid(format!( "canvas {}x{} must be non-zero and even", diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 07ab6fc..37d2b67 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -1,7 +1,7 @@ //! Draw scenes onto the canvas on the GPU: a preview, a program, and the fade //! between them. -use std::collections::HashMap; +use std::collections::{BTreeMap, HashMap}; use std::time::Duration; use kio::clock::Instant; @@ -90,15 +90,10 @@ impl effect::Library for Library<'_> { impl Compositor { /// A compositor drawing `canvas`-sized pictures on `device`. /// - /// Fails if the device cannot hold a texture that large. + /// Fails if the canvas is invalid or the device cannot hold a texture that + /// large. pub fn new(device: &wgpu::Device, queue: &wgpu::Queue, canvas: Canvas) -> Result { - let limit = device.limits().max_texture_dimension_2d; - if canvas.width > limit || canvas.height > limit { - return Err(Error::Invalid(format!( - "canvas {}x{} exceeds this GPU's {limit}px texture limit", - canvas.width, canvas.height - ))); - } + fits(device, canvas)?; let effects = Effects::new(device, queue, canvas); let target = |label| Target::new(device, &effects, canvas, label); @@ -122,6 +117,26 @@ impl Compositor { self.canvas } + /// Draw `canvas` from the next [`render`](Self::render) on. A new size + /// replaces the [`preview`](Self::preview) and [`program`](Self::program) + /// textures, and every other texture sized from the canvas, with ones of + /// that size, blank until drawn; a fade under way carries on. + /// + /// Fails, changing nothing, if the canvas is invalid or the device cannot + /// hold a texture that large. + pub fn set_canvas(&mut self, canvas: Canvas) -> Result<(), Error> { + fits(&self.device, canvas)?; + self.effects.set_canvas(canvas); + if canvas.size() != self.canvas.size() { + let target = |label| Target::new(&self.device, &self.effects, canvas, label); + self.preview = target("moq-scene preview"); + self.program = target("moq-scene program"); + self.incoming = target("moq-scene incoming"); + } + self.canvas = canvas; + Ok(()) + } + /// The scene in preview, as of the last [`render`](Self::render). pub fn preview(&self) -> &wgpu::Texture { &self.preview.texture @@ -154,11 +169,51 @@ impl Compositor { self.fade.is_some() } + /// The scenes program draws: the one on program, and the one a fade under + /// way leaves. + pub fn on_air(&self) -> impl Iterator { + let leaving = self.fade.as_ref().and_then(|fade| fade.from.as_deref()); + self.program_scene.as_deref().into_iter().chain(leaving) + } + /// Put the scene called `scene` in preview, or nothing. pub fn set_preview(&mut self, scene: Option) { self.preview_scene = scene; } + /// Follow each renamed scene to its new name, wherever it is: in preview, + /// on program, or fading out, and drop each removed one, so a scene that + /// later takes its name does not go on program without a cut. `renames` + /// maps each old name to its new one, or `None`, as + /// [`Collection::renames`] reads them, and each scene is looked up once, so + /// a name one scene gave up and another took is not followed twice. + pub fn rename(&mut self, renames: &BTreeMap>) { + for scene in self.scenes() { + if let Some(to) = scene.as_ref().and_then(|name| renames.get(name)) { + *scene = to.clone(); + } + } + } + + /// Drop each scene `collection` does not have from preview, program, and a + /// fade under way, such as one a reloaded file left out, so a scene that + /// later takes its name does not go on program without a cut. + pub fn retain(&mut self, collection: &Collection) { + for scene in self.scenes() { + if scene.as_deref().is_some_and(|name| collection.scene(name).is_none()) { + *scene = None; + } + } + } + + /// The scenes followed by name: in preview, on program, and fading out. + fn scenes(&mut self) -> impl Iterator> { + let fading = self.fade.as_mut().map(|fade| &mut fade.from); + [&mut self.preview_scene, &mut self.program_scene] + .into_iter() + .chain(fading) + } + /// Put the preview on program at once, ending any fade. pub fn cut(&mut self) { self.fade = None; @@ -284,6 +339,20 @@ impl Compositor { } } +/// Refuse an invalid canvas, or one larger than `device` can hold as a +/// texture. +fn fits(device: &wgpu::Device, canvas: Canvas) -> Result<(), Error> { + canvas.validate()?; + let limit = device.limits().max_texture_dimension_2d; + if canvas.width > limit || canvas.height > limit { + return Err(Error::Invalid(format!( + "canvas {}x{} exceeds this GPU's {limit}px texture limit", + canvas.width, canvas.height + ))); + } + Ok(()) +} + /// One render's draws onto its targets. struct Draw<'a> { collection: &'a Collection, @@ -1158,6 +1227,110 @@ mod tests { .await; } + /// A resize draws the next frame on textures of the new size, with items + /// where their pixels say, a blend mode and a blur reaching the canvas's + /// new width, and a canvas the GPU cannot hold changes nothing. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_resize_draws_the_new_canvas() { + local(async { + let mut collection = collection(); + collection.sources.insert( + "tint".into(), + Source::Color { + color: Rgba([0, 0, 255, 255]), + }, + ); + let mut studio = Studio::new(collection).await; + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.render(Instant::now()); + let old = studio.compositor.program().clone(); + + let wide = Canvas { + width: 16, + ..studio.collection.canvas + }; + studio.compositor.set_canvas(wide).unwrap(); + assert_ne!(studio.compositor.program(), &old, "a new texture"); + studio.collection.canvas = wide; + let fill = Rect::fill(&wide); + studio.collection.scenes[0].items = vec![ + Item::new("red", Rect { width: 8.0, ..fill }), + Item { + blend: Blend::Screen, + filters: vec![Filter::Blur { radius: 1.0 }], + ..Item::new("tint", fill) + }, + ]; + studio.render(Instant::now()); + let program = studio.program(); + assert_eq!(program.len(), 16 * 8 * 4); + let pixel = |x: usize, y: usize| -> [u8; 4] { + let at = (y * 16 + x) * 4; + program[at..at + 4].try_into().unwrap() + }; + assert!( + near(pixel(4, 4), [255, 0, 255, 255]), + "screened over red: {:?}", + pixel(4, 4) + ); + assert!( + near(pixel(12, 4), BLUE), + "screened past the old width: {:?}", + pixel(12, 4) + ); + + let limit = studio.compositor.device.limits().max_texture_dimension_2d; + let huge = Canvas { + width: limit + 2, + ..wide + }; + assert!(matches!(studio.compositor.set_canvas(huge), Err(Error::Invalid(_)))); + let empty = Canvas { width: 0, ..wide }; + assert!(matches!(studio.compositor.set_canvas(empty), Err(Error::Invalid(_)))); + assert_eq!(studio.compositor.canvas(), wide); + }) + .await; + } + + /// A renamed scene stays in preview, on program, and in the fade under way, + /// even when another scene takes its old name in the same edit, and a scene + /// the collection lost leaves them. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_renamed_scene_stays_where_it_is() { + local(async { + let mut studio = Studio::new(collection()).await; + let start = Instant::now(); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.fade(Duration::from_secs(1), start); + + // `blue` takes the name `red` gives up, which is followed once. + let mut draft = studio.collection.draft("Rename"); + draft.rename_scene(0, "crimson").unwrap(); + draft.rename_scene(1, "red").unwrap(); + let edit = draft.finish().unwrap(); + studio.compositor.rename(&studio.collection.renames(&edit)); + studio.collection.apply(&edit).unwrap(); + assert_eq!(studio.compositor.preview_scene(), Some("red")); + assert_eq!(studio.compositor.program_scene(), Some("red")); + studio.render(start + Duration::from_millis(500)); + let actual = pixel(&studio.program(), 1, 1); + assert!(near(actual, [128, 0, 127, 255]), "still fading from red: {actual:?}"); + + // A scene the collection no longer has, as a reload can leave, is + // dropped, so one that later takes its name is not drawn unasked. + studio.collection.scenes.retain(|scene| scene.name != "red"); + studio.compositor.retain(&studio.collection); + assert_eq!(studio.compositor.preview_scene(), None); + assert_eq!(studio.compositor.program_scene(), None); + }) + .await; + } + /// A blend mode mixes an item with what is under it by the CSS formula, /// weighted by the item's opacity. #[tokio::test] diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index e7bd3c6..44d6d40 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -9,12 +9,12 @@ //! Build an edit with a [`Draft`], which reads what each change replaces from //! the collection as the earlier changes left it. -use std::collections::VecDeque; +use std::collections::{BTreeMap, VecDeque}; use serde::{Deserialize, Serialize}; use crate::Error; -use crate::collection::{Audio, Collection, Filter, Item, Layer, Scene, Source, Transition, Video}; +use crate::collection::{Audio, Canvas, Collection, Filter, Item, Layer, Scene, Source, Transition, Video}; /// One part of a collection, before and after an edit. /// @@ -47,7 +47,8 @@ pub enum Change { after: Option