From d6b5ecdaa2aa3d0eaf60f571a031a399d2d7a42b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 17:05:02 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m0/scenes/published.md Quest-Claim: a0b315b37df4994e From 10ddf9d9f533b8b341cd7f20df7647703ec7fec2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 18:40:28 -0800 Subject: [PATCH 2/5] feat(studio)!: published scenes replace canvases Every scene has a size, the program's by default, and any scene can be published as its own broadcast at <--broadcast>/, at its own size, with full and half renditions sharing the bandwidth grant. Further canvases, their per-scene layouts, and the canvas picker are gone; a portrait stream is a published portrait scene drawing the program through a program source. The program keeps one switcher, offering only scenes of its size. The preview follows the preview scene's size and is not drawn while no tab shows it. A published or recorded scene is drawn only while one of its encoders takes the frame. Recording has its own encoder line, recording the program or one scene. Health grades each published broadcast with its own viewers. Audio stays as today: published scenes carry the program's mix, and the audio-scene picker is left to quest/m0/scenes/audio.md. BREAKING CHANGE: `[canvases]` and a scene's `canvases.` are removed from the scene file, with no migration. Co-Authored-By: Claude Opus 5.5 --- README.md | 158 +++--- quest/m0/scenes/README.md | 1 - quest/m0/scenes/published.md | 29 - rs/moq-scene/src/collection.rs | 367 +++++++------ rs/moq-scene/src/compositor.rs | 836 +++++++++++++++++------------ rs/moq-scene/src/edit.rs | 206 ++----- rs/moq-scene/src/lib.rs | 26 +- rs/moq-studio/src/engine.rs | 757 ++++++++++++++++---------- rs/moq-studio/src/health.rs | 948 ++++++++++++++++++++++++--------- rs/moq-studio/src/interact.rs | 57 +- rs/moq-studio/src/ladder.rs | 6 + rs/moq-studio/src/ui.rs | 587 ++++++++++---------- 12 files changed, 2310 insertions(+), 1668 deletions(-) delete mode 100644 quest/m0/scenes/published.md diff --git a/README.md b/README.md index 2e080328..d993e56e 100644 --- a/README.md +++ b/README.md @@ -94,44 +94,54 @@ Every panel is a tab you can drag, split, or stack. | Panel | Use it to | | --- | --- | -| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. With more than one canvas, pick the canvas they show, and that the Items panel edits, beside the preview. **Cut** puts the preview on program at once; **Transition** takes it there by the transition picked beside it, over the time set there, or for a stinger by the media source and cut point set there (see [Transitions](#transitions)). **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | -| Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still leaving it in a transition, cannot be removed, even by an undo, until program cuts away from it. | +| Preview, Program | See the scene you are editing, at its own size, and the one that is live, and drag items in the preview to place them. A preview no tab shows is not drawn. **Cut** puts the preview on program at once; **Transition** takes it there by the transition picked beside it, over the time set there, or for a stinger by the media source and cut point set there (see [Transitions](#transitions)). **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. | +| Scenes | Pick the scene in preview, and add, rename (double-click), reorder, remove, or tick **Publish** on one (see [Published scenes](#published-scenes)). A scene on program, or still leaving it in a transition, cannot be removed, even by an undo, until program cuts away from it. | | History | See every edit, and undo or redo back to any of them. | -| Items | Place, show, hide, and order what the preview scene or the top layer draws on the canvas shown, back to front. Select an item to filter or copy it. | +| Items | Place, show, hide, and order what the preview scene or the top layer draws, back to front. The top layer is edited over a scene of the program's size. Select an item to filter or copy it. | | Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. A media file reads as ended once it plays to its end without looping, **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again. | | Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. | | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. | | Filters | Remove the selected source's background, crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | | Keyframes | Animate the selected item: add, time, ease, and remove its keyframes, and set how far each moves, resizes, crops, and filters it (see [Keyframes](#keyframes)). **Play** plays the preview scene's keyframes again. | | Audio | Add, rename (double-click), and remove mixes, pick the one on program and the one to listen to and on which output, and set each audio source's gain, mute, delay, and the mixes it feeds, with a meter on every mix and source. | -| Output | Start and stop recording, pick the mix it records, set the canvas size and frame rate, add, resize, and remove further canvases, and see the broadcast path, whether the program is encoding, the encoder sessions open, and each canvas's renditions' encoder work. | -| Health | See how the stream reaches viewers, relay by relay and track by track, and the cause of each problem, and which recordings a crash left unfinished were repaired at launch. | +| Output | Start and stop recording, pick the picture and the mix it records, set the canvas size and frame rate and each scene's size, and see the broadcast path, whether the program is encoding, the encoder sessions open, and the encoder work of the program, the recording, and each published scene. | +| Health | See how the stream reaches viewers, relay by relay and, for the program and each published scene, track by track, and the cause of each problem, and which recordings a crash left unfinished were repaired at launch. | | Replay | Replay the last moments of the program or a source in a replay source, slowed down if you like, or save them as a clip. | | Hotkeys | Bind keys to cut, transition, preview a scene, replay, save a clip, and start or stop recording, and clear them (see [Hotkeys](#hotkeys)). | A new source, from the Sources or Devices panel, is placed over whatever the -Items panel shows. The **top layer** is drawn over every scene, in preview and -on program, and stays on through a cut or transition: a downstream keyer for a -logo or a lower third. - -### Canvases - -Beside the main canvas, the studio can draw further canvases at once, such as -a 1080x1920 portrait one for a vertical stream beside a 1920x1080 landscape -main canvas. Add one by name in the Output panel. Each scene has its own -layout on each canvas, and each canvas its own top layer; a scene without a -layout on a canvas shows only that canvas's top layer there. Every canvas -shares the main canvas's frame rate, and the canvases are linked: one scene is -in preview and one on program on all of them, and a cut or transition takes -every canvas together. An item's keyframes play on every canvas from when its -scene goes on, as on the main one. A source captures or decodes once however many canvases -show it, and a canvas draws only the items it places. - -Each further canvas publishes a broadcast of its own beneath the main one, at -`<--broadcast>/`, with renditions and encoders of its own (see -[Output](#output)). Its catalog lists the main broadcast's audio rather than -encoding it again. Removing a canvas ends its broadcast, and removes its top -layer and every scene's layout on it. +Items panel shows. The **top layer** is drawn over program, and over the +preview of a scene of the program's size, and stays on through a cut or +transition: a downstream keyer for a logo or a lower third. + +### Published scenes + +Every scene has a size, the program's unless the Output panel's **Scene +sizes** gives it its own, such as 1080x1920 for a portrait scene beside a +1920x1080 program. Every scene is drawn at the program's frame rate. Only a +scene of the program's size goes on program: **Cut** and **Transition** +refuse another, and an edit resizing the scene on air is refused. The preview +shows a scene at its own size. + +Tick **Publish** on a scene in the Scenes panel to put it live as a broadcast +of its own beneath the program's, at `<--broadcast>/`, at its own +size, with renditions and encoders of its own (see [Output](#output)). A +published scene is always live and never switched: it shows what its items +show, whatever is on program. For a portrait stream that follows the program, +publish a portrait scene drawing the program through a program source (see +[Scene sources](#scene-sources)), with any overlay of its own placed beside +it; the top layer goes over the program alone. Its catalog lists the +program's audio rather than encoding it again, so every published scene +carries the program's mix for now. Unticking it, or removing the scene, ends +its broadcast. A published scene's name is one segment of its broadcast path, +so it holds no `/`, and a scene named `viewers`, where viewers report back, +does not publish while live. + +A source captures or decodes once however many scenes show it. A published +scene is drawn only while one of its encoders takes the frame, so with +`--on-demand`, or without a relay, one nobody watches costs no drawing. An +item's keyframes play in a published scene from when it was published or the +studio started. ### Scene sources @@ -140,10 +150,10 @@ scales, crops, filters, and blends like any other: a lower third built once and shown in every scene, or a whole layout shrunk into a corner. Add one in the Sources panel as kind **Scene**, picking the scene it draws, or **Program**, which shows whatever is on air, a transition under way included, -on the same frame, such as for a portrait canvas that follows the program, +on the same frame, such as for a portrait scene that follows the program, cropped. -A nested scene is drawn at the main canvas's size, transparent where it draws +A nested scene is drawn at its own size, transparent where it draws nothing, once per frame however many items show it, and only while something drawn shows it. Its items' keyframes play from when the scene showing it goes on, as that scene's own items' do, so the top layer, which never goes on, @@ -151,8 +161,8 @@ cannot show a scene with keyframes. A media file or overlay in a nested scene set to restart starts over when the scene showing it goes on program. A scene cannot draw itself, even through other scenes, so an edit that would -is refused. A scene showing the program cannot go on program, nor can the main -canvas's top layer show it, since either would draw the program into itself: +is refused. A scene showing the program cannot go on program, nor can the top +layer show it, since either would draw the program into itself: **Cut** and **Transition** refuse such a scene, and an edit making the scene on air show the program is refused. Renaming a scene renames what its scene sources draw, and removing one removes them, with the items showing them. @@ -513,11 +523,6 @@ width = 1920 height = 1080 framerate = 30 -[canvases.portrait] # a further canvas, at the main one's frame rate -width = 1080 -height = 1920 -top = [{ source = "logo", rect = { x = 780, y = 40, width = 280, height = 120 } }] - [transition] kind = "slide" # fade, slide, wipe, move, or stinger duration = "300ms" # all but a stinger @@ -601,10 +606,6 @@ items = [ { source = "guest", rect = { x = 1280, y = 720, width = 640, height = 360 }, filters = [{ kind = "drop_shadow", x = 8, y = 8, blur = 12, color = "#00000080" }] }, { source = "logo", rect = { x = 0, y = 0, width = 1920, height = 1080 }, blend = "multiply", filters = [{ kind = "opacity", amount = 0.3 }] }, ] -canvases.portrait = [ # the same scene laid out on the portrait canvas - { source = "camera", rect = { x = 0, y = 0, width = 1080, height = 1920 }, crop = { left = 0.34, right = 0.34 } }, - { source = "guest", rect = { x = 0, y = 1312, width = 1080, height = 608 } }, -] [[scenes]] name = "Break" @@ -618,6 +619,16 @@ items = [ { source = "main-view", rect = { x = 80, y = 80, width = 640, height = 360 } }, ] +[[scenes]] +name = "Portrait" +size = { width = 1080, height = 1920 } # its own size; the program's without one +published = true # live at <--broadcast>/Portrait +recorded = true # recordings hold this scene rather than the program +items = [ + { source = "on-air", rect = { x = 0, y = 0, width = 1080, height = 1920 }, crop = { left = 0.34, right = 0.34 } }, + { source = "logo", rect = { x = 780, y = 40, width = 280, height = 120 } }, +] + [[top]] source = "logo" rect = { x = 1600, y = 40, width = 280, height = 120 } @@ -634,10 +645,11 @@ file, which no item places, but which grades what a `lut` filter names it in), draws), and `program` (see [Scene sources](#scene-sources)). The Devices panel fills in ids. `[[top]]` items make up the top layer. -`[canvases.]` adds a further canvas, with its `width` and `height` and -its own `top` layer, and a scene's `canvases.` lists its items on that -canvas; leave it out, rather than empty, for a scene that shows nothing there. -A canvas name is one segment of its broadcast path, so it holds no `/`. +A scene's `size` gives it a `width` and `height` of its own; without one it is +the program's. `published = true` puts it live as its own broadcast (see +[Published scenes](#published-scenes)), so its name holds no `/`, and +`recorded = true` has recordings hold it rather than the program, on one scene +at most. `[mixes]` names every mix, in the order the Audio panel lists them, the one on `program`, and the one a `recording` holds, the mix on program without it; @@ -719,7 +731,10 @@ before sharing. The program publishes as two H.264 renditions, at the canvas size and at half of it, at the canvas frame rate, with the mix on program as Opus stereo audio, -under the `--broadcast` path. A canvas under 360 lines has no half size. +under the `--broadcast` path. A canvas under 360 lines has no half size. Each +published scene publishes the same way at `<--broadcast>/`, at +its own size and half of it, its tracks sharing the connection's bandwidth +with the program's. Timestamps come from the studio's own clock, so a source that stalls repeats its last picture instead of holding up the stream. @@ -794,25 +809,27 @@ encode engine for them all. The Output panel counts the sessions open: |---|---| | The program's canvas size, live | 1, or none without `--connect` | | Its half size | 1 while someone watches it | -| Each further canvas's canvas size and half size | as the program's | -| The recording's own canvas size | 1 while recording | +| Each published scene's own size and half size | as the program's | +| The recording's own size | 1 while recording | | Each captured source or media file | 1 while on program or recorded | So a live scene showing a camera, a display, and a window holds 4 sessions, 5 -with the half size watched, and 6 while recording too. A portrait canvas beside -it adds 1 more, 2 with its half size watched. When the program +with the half size watched, and 6 while recording too. A published portrait +scene beside it adds 1 more, 2 with its half size watched. When the program encodes on hardware, so do the sources, and a session the GPU refuses fails that source's track with the encoder's reason in the Sources panel rather than falling back to software. The source keeps drawing in the studio's own scenes. The program opens its session before any source, even with `--on-demand`, so the sources past the cap fail rather than -the program; so does each canvas in the scene file at startup. A canvas added -later, past the cap, draws without video and says why in the Output panel. When the program encodes in software, a source takes hardware +the program; so does each scene published in the scene file at startup. A +scene published later, past the cap, publishes without video and says why in +the Output panel. When the program encodes in software, a source takes hardware where it can and software otherwise. ### Recording -**Start recording** in the Output panel saves the program, at the canvas size, +**Start recording** in the Output panel saves the program at the canvas size, +or the scene picked beside it at its own size, to ` .mp4` in the `--recordings` directory, live or not. The recording has an encoder of its own, opened only while it records, at full quality whatever the uplink grants the live stream, as OBS's @@ -830,8 +847,9 @@ guest who leaves finishes their file, and one who joins again goes on in a new one, numbered after the first, since each join is a new broadcast with a timeline of its own. A source added while recording starts its file then, and one removed finishes its file then. A new canvas -size or frame rate, or a capture that changes size or color space, moves the -video to a new track, so the program or that source goes on in a new file, +size or frame rate, a recorded scene of another size, or a capture that +changes size or color space, moves the video to a new track, so the recording +or that source goes on in a new file, numbered after the first, since players show only one video track. A file starts with up to 2 s from before it opened, so a recording starts a little early, and a track that goes on into the next file, such as the program @@ -967,15 +985,20 @@ out of it), a source the program shows or plays failed or, as a media file that does not loop, played to its end, or viewers are late, stalling, or cannot decode a track. A relay lost while the other serves is degraded, since viewers have nothing left to fall back to; with none serving -it is unhealthy. Below that it shows: +it is unhealthy. A problem with a published scene's broadcast names the scene. +Below that it shows: - **Relays**: each relay's state, and the protocol its session negotiated. - **Uplink**: the round trip to the first connected relay, the bandwidth the - connection estimates it can carry, the media the program's tracks write, and + connection estimates it can carry, the media every broadcast's tracks write, and the share lost and resent. Loss and headroom are graded only while someone watches, since the relay pulls nothing otherwise. Source tracks never reach the relay, so they are left out of the media. -- **Watching**: the sessions subscribed to the broadcast, from the relays' + +Then, for the program's broadcast and each published scene's, each with its +own viewers: + +- **Watching**: the sessions subscribed to it, from the relays' `.stats` broadcasts. A relay shares those only with a connection allowed to read them, so most show "unknown". In a cluster, a relay forwarding the stream to another counts as one session. Two relays whose stats use the @@ -984,18 +1007,21 @@ it is unhealthy. Below that it shows: - **Tracks**: each track's share of the connection's bandwidth (**Granted**, or "off" for a rendition the share cannot fund), its bitrate and frame rate, and, from viewers that report, the lowest bitrate any received, its worst - grade, and the latency from the program clock to the slowest viewer. The - program's tracks come first, then each source's own tracks, grouped by - source; a still image or - color carries no media to grade. Latency compares the two hosts' wall + grade, and the latency from the program clock to the slowest viewer. A + published scene lists the program's audio, graded where the program writes + it. Latency compares the two hosts' wall clocks, so keep both synced. - **Viewers**: each viewer that reports, with its round trip, latency, and - playout delay. The studio reads 64 at most, a sample the grades rest on: it - counts the rest, and a reader one frees goes to the next viewer to arrive. + playout delay. The studio reads 64 at most per broadcast, a sample the grades + rest on: it counts the rest, and a reader one frees goes to the next viewer + to arrive. + +Last, **Sources** lists each source's own tracks, grouped by source; a still +image or color carries no media to grade. -The catalog asks viewers to report under `/viewers/`: a viewer -publishes `/viewers/.echo`, as `moq export --echo ` -does. The relay decides who may publish there, as for any broadcast. +Each catalog asks viewers to report beneath its broadcast, under +`/viewers/` or `//viewers/`: a viewer publishes +`/viewers/.echo`, as `moq export --echo ` does. The relay decides who may publish there, as for any broadcast. ## Development diff --git a/quest/m0/scenes/README.md b/quest/m0/scenes/README.md index b41a9ab7..69487c52 100644 --- a/quest/m0/scenes/README.md +++ b/quest/m0/scenes/README.md @@ -51,5 +51,4 @@ reads skips drawing. Per-scene cost grows only with what that scene draws; ## Required -- [Published scenes](/quest/m0/scenes/published.md) - any scene publishes as its own broadcast at its own size, replacing canvases - [Audio scenes](/quest/m0/scenes/audio.md) - mixes become audio-only scenes that nest diff --git a/quest/m0/scenes/published.md b/quest/m0/scenes/published.md deleted file mode 100644 index 3c5514e8..00000000 --- a/quest/m0/scenes/published.md +++ /dev/null @@ -1,29 +0,0 @@ -# [L] Published scenes - -## Goal - -Any scene can be published: ticked **Publish**, it goes live as its own -broadcast at `<--broadcast>/`, at its own size, with the audio scene it -picks (the one on program by default). This replaces canvases: further -canvases, their per-scene layouts, and the canvas picker are removed, and a -portrait broadcast is a published portrait scene that draws the program -through a program source (`kind = "program"`). - -## Plan - -Decided 2026-11-05 with the maintainer (/quest-plan); see the -[epic](/quest/m0/scenes/README.md). - -- Every scene has a size, defaulting to the program's; the switcher offers - only scenes of the program's size. A scene source then draws its scene at - that size, rather than the main canvas's as it does now. -- The program keeps one switcher at `<--broadcast>`. A published scene is - always live and never switched. -- Each published scene gets the program's two sizes (full and half) and the - bandwidth grant shares across all of them, as further canvases did. -- Carried from [moq-dev/studio#54](https://github.com/moq-dev/studio/pull/54): - Health grades each published broadcast with its viewer feedback; a - recording picks a picture scene and an audio scene; a scene no panel shows - and no output reads skips drawing. -- Remove `[canvases]` from the scene file with no migration; nothing is - released. `README.md` documents published scenes. diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index b5515420..76debf86 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -12,8 +12,8 @@ use serde_with::{DeserializeFromStr, SerializeDisplay}; use crate::Error; -/// Everything a studio composes: the canvases, the sources, and the scenes -/// that place them. +/// Everything a studio composes: the program's canvas, the sources, and the +/// scenes that place them. /// /// Saved with [`to_toml`](Self::to_toml) and loaded with /// [`from_toml`](Self::from_toml). The file names devices, files, and broadcast @@ -21,14 +21,9 @@ use crate::Error; #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct Collection { - /// The main canvas every scene is drawn onto, whose frame rate every - /// canvas shares. + /// The program's size and frame rate: the size of every scene that sets + /// none, and the frame rate every scene is drawn at. pub canvas: Canvas, - /// Further canvases by name, such as a portrait one beside a landscape - /// main canvas. Each draws its own layout of every scene from the same - /// sources, under its own top layer. - #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] - pub canvases: BTreeMap, /// How the preview is taken to program, short of a cut. #[serde(default)] pub transition: Transition, @@ -50,9 +45,10 @@ pub struct Collection { /// The scenes, in the order the scene list shows them. #[serde(default)] pub scenes: Vec, - /// The top layer: items drawn over whichever scene is in preview and on - /// program, back to front, so they stay on across scene switches (a - /// downstream keyer). They do not fade with a transition. + /// The top layer: items drawn over whichever scene is on program, and in + /// preview over one of the program's size, back to front, so they stay on + /// across scene switches (a downstream keyer). They do not fade with a + /// transition. #[serde(default, skip_serializing_if = "Vec::is_empty")] pub top: Vec, } @@ -68,7 +64,6 @@ impl Collection { pub fn new() -> Self { Self { canvas: Canvas::default(), - canvases: BTreeMap::new(), transition: Transition::default(), sources: BTreeMap::new(), mixes: Mixes::default(), @@ -99,87 +94,71 @@ impl Collection { self.scenes.iter().find(|scene| scene.name == name) } - /// The canvas `canvas` names: the main one for `None`, or a further one, - /// at the main canvas's frame rate. `None` for a canvas that does not - /// exist. - pub fn canvas(&self, canvas: Option<&str>) -> Option { - match canvas { + /// The canvas the scene called `scene` is drawn on, or the top layer for + /// `None`: its own size, or the program's, at the program's frame rate. + /// `None` for a scene that does not exist. + pub fn canvas(&self, scene: Option<&str>) -> Option { + match scene { None => Some(self.canvas), - Some(name) => self.canvases.get(name).map(|extra| extra.canvas(self.canvas.framerate)), + Some(name) => self.scene(name).map(|scene| scene.canvas(self.canvas)), } } - /// The items of `layer`, or `None` for a scene or canvas that does not - /// exist. A scene with no layout on a further canvas has none there. + /// The items of `layer`, or `None` for a scene that does not exist. pub fn items(&self, layer: &Layer) -> Option<&[Item]> { - match (&layer.canvas, &layer.scene) { - (None, Some(scene)) => self.scene(scene).map(|scene| scene.items.as_slice()), - (None, None) => Some(&self.top), - (Some(canvas), scene) => { - let extra = self.canvases.get(canvas)?; - match scene { - Some(scene) => { - let layout = self.scene(scene)?.canvases.get(canvas); - Some(layout.map_or(&[], Vec::as_slice)) - } - None => Some(&extra.top), - } - } + match &layer.scene { + Some(scene) => self.scene(scene).map(|scene| scene.items.as_slice()), + None => Some(&self.top), } } - /// Run `edit` on the items of `layer`, or return `None` for a scene or - /// canvas that does not exist. A scene's layout on a further canvas is - /// kept only while it holds items, so a scene has one way to say it has - /// none there, as [`validate`](Self::validate) requires. + /// Run `edit` on the items of `layer`, or return `None` for a scene that + /// does not exist. pub(crate) fn edit_items(&mut self, layer: &Layer, edit: impl FnOnce(&mut Vec) -> T) -> Option { - if let Some(canvas) = &layer.canvas - && !self.canvases.contains_key(canvas) - { - return None; - } - match (&layer.canvas, &layer.scene) { - (None, None) => Some(edit(&mut self.top)), - (Some(canvas), None) => self.canvases.get_mut(canvas).map(|extra| edit(&mut extra.top)), - (canvas, Some(name)) => { + match &layer.scene { + None => Some(edit(&mut self.top)), + Some(name) => { let scene = self.scenes.iter_mut().find(|scene| &scene.name == name)?; - let Some(canvas) = canvas else { - return Some(edit(&mut scene.items)); - }; - let mut layout = scene.canvases.remove(canvas).unwrap_or_default(); - let edited = edit(&mut layout); - if !layout.is_empty() { - scene.canvases.insert(canvas.clone(), layout); - } - Some(edited) + Some(edit(&mut scene.items)) } } } - /// Every layer: the main canvas's scenes in order, then its top layer, - /// then each further canvas's the same way. + /// Every layer: the scenes in order, then the top layer. pub fn layers(&self) -> impl Iterator { - let extras = self - .canvases + let scenes = self + .scenes .iter() - .map(|(name, extra)| (Some(name), extra.top.as_slice())); - let canvases = std::iter::once((None, self.top.as_slice())).chain(extras); - canvases.flat_map(move |(canvas, top)| { - let scenes = self.scenes.iter().map(move |scene| { - let items = match canvas { - None => scene.items.as_slice(), - Some(name) => scene.canvases.get(name).map_or(&[][..], Vec::as_slice), - }; - (Some(scene.name.clone()), items) - }); - scenes.chain([(None, top)]).map(move |(scene, items)| { - let layer = Layer { - canvas: canvas.cloned(), - scene, - }; - (layer, items) - }) - }) + .map(|scene| (Layer::scene(&scene.name), scene.items.as_slice())); + scenes.chain([(Layer::top(), self.top.as_slice())]) + } + + /// The scene recordings hold, or `None` for the program. + pub fn recorded(&self) -> Option<&str> { + let recorded = self.scenes.iter().find(|scene| scene.recorded); + recorded.map(|scene| scene.name.as_str()) + } + + /// Whether the scene called `scene` can go on program: it is the + /// program's size, and draws no program source, which it would draw into + /// itself. Refused with the reason otherwise, or for a scene that does + /// not exist. + pub fn takes(&self, scene: &str) -> Result<(), Error> { + let canvas = self + .canvas(Some(scene)) + .ok_or_else(|| Error::Invalid(format!("there is no scene `{scene}`")))?; + if canvas.size() != self.canvas.size() { + return Err(Error::Invalid(format!( + "`{scene}` is {}x{}, not the program's {}x{}, so it cannot go on program", + canvas.width, canvas.height, self.canvas.width, self.canvas.height + ))); + } + if self.draws_program(scene) { + return Err(Error::Invalid(format!( + "`{scene}` draws the program, so it cannot go on program" + ))); + } + Ok(()) } /// The source whose capture `name` shows: itself, or what it clones. @@ -319,23 +298,13 @@ impl Collection { source.unwrap_or_default().then(&item.crop) } - /// Check what the types cannot: the canvas can be encoded, names are unique, - /// every item, clone, audio, and video entry names a source that exists, a - /// stinger names a media source, and every crop and filter's settings are - /// in range, each filter naming a source of the kind it reads. + /// Check what the types cannot: every canvas can be encoded, names are + /// unique, a published scene's names its broadcast, one scene at most is + /// recorded, every item, clone, audio, and video entry names a source that + /// exists, a stinger names a media source, and every crop and filter's + /// settings are in range, each filter naming a source of the kind it reads. pub fn validate(&self) -> Result<(), Error> { self.canvas.validate()?; - for (name, extra) in &self.canvases { - if name.is_empty() || name.contains('/') { - return Err(Error::Invalid(format!( - "canvas `{name}` must be named, without a `/`, since its name names its broadcast" - ))); - } - extra - .canvas(self.canvas.framerate) - .validate() - .map_err(|err| Error::Invalid(format!("canvas `{name}`: {err}")))?; - } let mut names = BTreeSet::new(); for scene in &self.scenes { @@ -345,25 +314,24 @@ impl Collection { if !names.insert(scene.name.as_str()) { return Err(Error::Invalid(format!("two scenes are named `{}`", scene.name))); } - if let Some(canvas) = scene - .canvases - .keys() - .find(|canvas| !self.canvases.contains_key(*canvas)) - { - return Err(Error::Invalid(format!( - "scene `{}` has a layout on `{canvas}`, which is not a canvas", - scene.name - ))); - } - // One way to say a scene shows nothing there, so an edit snapshotting - // the scene and one emptying the layout agree on what it holds. - if let Some((canvas, _)) = scene.canvases.iter().find(|(_, items)| items.is_empty()) { + scene + .canvas(self.canvas) + .validate() + .map_err(|err| Error::Invalid(format!("scene `{}`: {err}", scene.name)))?; + if scene.published && scene.name.contains('/') { return Err(Error::Invalid(format!( - "scene `{}` has an empty layout on `{canvas}`; leave it out instead", + "scene `{}` is published, so its name names its broadcast and holds no `/`", scene.name ))); } } + let mut recorded = self.scenes.iter().filter(|scene| scene.recorded); + if let (Some(first), Some(second)) = (recorded.next(), recorded.next()) { + return Err(Error::Invalid(format!( + "scenes `{}` and `{}` are both recorded, but recordings hold one", + first.name, second.name + ))); + } for (layer, items) in self.layers() { for item in items { let Some(source) = self.sources.get(&item.source) else { @@ -599,39 +567,14 @@ impl Default for Canvas { } } -/// A canvas beside the main one: its size, and the items drawn over every -/// scene on it. Each scene's layout on it is the scene's own -/// [`Scene::canvases`] entry. -#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +/// A scene's own size, in pixels. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] -pub struct Extra { +pub struct Size { /// Width in pixels. Even, since the encoder subsamples chroma. pub width: u32, /// Height in pixels. Even, since the encoder subsamples chroma. pub height: u32, - /// Its top layer, as [`Collection::top`] is the main canvas's. - #[serde(default, skip_serializing_if = "Vec::is_empty")] - pub top: Vec, -} - -impl Extra { - /// An empty canvas of `width` by `height`. - pub fn new(width: u32, height: u32) -> Self { - Self { - width, - height, - top: Vec::new(), - } - } - - /// This canvas drawn `framerate` times a second. - pub fn canvas(&self, framerate: u32) -> Canvas { - Canvas { - width: self.width, - height: self.height, - framerate, - } - } } /// How the program changes when the preview is taken to it, short of a cut. @@ -676,8 +619,8 @@ pub enum Transition { /// A media source's clip plays over the program from its start, and the /// program switches scenes under it at `cut` into the clip, while it /// covers the canvas. It lasts as long as the clip's first play, filling - /// each canvas it is drawn on, cropped to the canvas's shape, through the - /// source's own crop and filters. + /// the canvas, cropped to its shape, through the source's own crop and + /// filters. Stinger { /// The [`Source::Media`] whose clip plays. source: String, @@ -793,74 +736,83 @@ impl Direction { pub const ALL: [Self; 4] = [Self::Left, Self::Right, Self::Up, Self::Down]; } -/// One arrangement of sources on the canvas. +/// One arrangement of sources on a canvas of its own size. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct Scene { /// Unique among the collection's scenes. pub name: String, - /// What the scene draws on the main canvas, back to front. + /// Its size, or `None` for the program's. Only a scene of the program's + /// size goes on program. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub size: Option, + /// Whether it is always live as a broadcast of its own, beneath the + /// program's and named after it, whatever is on program. + #[serde(default, skip_serializing_if = "is_false")] + pub published: bool, + /// Whether recordings hold it rather than the program. One scene at most + /// is recorded. + #[serde(default, skip_serializing_if = "is_false")] + pub recorded: bool, + /// What the scene draws, back to front. #[serde(default)] pub items: Vec, - /// What the scene draws on each further canvas, by the canvas's name, back - /// to front. A canvas without an entry shows only its top layer. - #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] - pub canvases: BTreeMap>, } impl Scene { - /// An empty scene. + /// An empty scene of the program's size, neither published nor recorded. pub fn new(name: impl Into) -> Self { Self { name: name.into(), + size: None, + published: false, + recorded: false, items: Vec::new(), - canvases: BTreeMap::new(), + } + } + + /// The canvas it is drawn on: its own size, or `program`'s, at + /// `program`'s frame rate. + pub fn canvas(&self, program: Canvas) -> Canvas { + match self.size { + Some(Size { width, height }) => Canvas { + width, + height, + framerate: program.framerate, + }, + None => program, } } } -/// Where a list of items lives: a scene's layout, or the top layer over every -/// scene, on one canvas. +/// Where a list of items lives: a scene, or the top layer over the program. #[derive(Clone, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct Layer { - /// A further canvas by name, or `None` for the main canvas. - #[serde(default, skip_serializing_if = "Option::is_none")] - pub canvas: Option, - /// The scene, or `None` for the canvas's top layer. + /// The scene, or `None` for the top layer. #[serde(default, skip_serializing_if = "Option::is_none")] pub scene: Option, } impl Layer { - /// The scene called `name`, on the main canvas. + /// The scene called `name`. pub fn scene(name: impl Into) -> Self { Self { - canvas: None, scene: Some(name.into()), } } - /// The main canvas's top layer. + /// The top layer. pub fn top() -> Self { Self::default() } - - /// The same layer on `canvas` instead. - pub fn on(self, canvas: Option) -> Self { - Self { canvas, ..self } - } } impl fmt::Display for Layer { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match &self.scene { - Some(name) => write!(f, "scene `{name}`")?, - None => write!(f, "the top layer")?, - } - match &self.canvas { - Some(canvas) => write!(f, " on canvas `{canvas}`"), - None => Ok(()), + Some(name) => write!(f, "scene `{name}`"), + None => write!(f, "the top layer"), } } } @@ -870,7 +822,7 @@ impl fmt::Display for Layer { pub enum Nested<'a> { /// The scene by this name. Scene(&'a str), - /// The main canvas's program. + /// The program. Program, } @@ -1073,6 +1025,10 @@ fn yes() -> bool { true } +fn is_false(value: &bool) -> bool { + !value +} + fn is_true(value: &bool) -> bool { *value } @@ -1669,7 +1625,7 @@ pub enum Source { /// source, on demand through [`Feed::replay`](crate::source::Feed::replay). /// Transparent while nothing plays. Replay, - /// Another scene, drawn whole at the main canvas's size, as a picture. Its + /// Another scene, drawn whole at its own size, as a picture. Its /// items' keyframes play from when the scene drawing it goes on, as that /// scene's own items' do, so the top layer cannot draw one with keyframes. /// A scene cannot draw itself, even through other scenes. @@ -1677,7 +1633,7 @@ pub enum Source { /// The scene drawn. scene: String, }, - /// What the main canvas's program shows, transitions included, on the + /// What the program shows, transitions included, on the /// same frame. A scene drawing it cannot go on program, and the top layer /// cannot draw it, since either would draw the program into itself. Program, @@ -2087,6 +2043,13 @@ items = [ ] }, ] +[[scenes]] +name = "Portrait" +size = { width = 720, height = 1280 } +published = true +recorded = true +items = [{ source = "cam", rect = { x = 0, y = 0, width = 720, height = 1280 } }] + [[top]] source = "bg" rect = { x = 0, y = 680, width = 1280, height = 40 } @@ -2118,7 +2081,19 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } ); assert_eq!(collection.mixes.program, "Stream"); assert_eq!(collection.mixes.recorded(), "Recording"); + let portrait = collection.scene("Portrait").unwrap(); + assert!(portrait.published); + assert_eq!(collection.recorded(), Some("Portrait")); + assert_eq!( + collection.canvas(Some("Portrait")), + Some(Canvas { + width: 720, + height: 1280, + framerate: 60 + }) + ); let main = collection.scene("Main").unwrap(); + assert_eq!(main.size, None, "the program's"); assert!(main.items[0].visible); assert!(!main.items[1].visible); assert_eq!( @@ -2828,34 +2803,50 @@ background = "remove" assert!(!collection.draws_program("Scene")); } - /// A further canvas is named as a path segment, since it names its - /// broadcast, is a size the encoder can take, and is the only canvas a - /// scene lays items out on. + /// A scene's own size is one the encoder can take, a published scene is + /// named as a path segment, since it names its broadcast, and recordings + /// hold one scene at most. Only a scene of the program's size that draws + /// no program goes on program. #[test] - fn refuses_a_further_canvas_that_cannot_be() { - for (name, width) in [("", 1080), ("a/b", 1080), ("tall", 1081)] { - let mut collection = example(); - collection.canvases.insert(name.into(), Extra::new(width, 1920)); - assert!( - matches!(collection.validate(), Err(Error::Invalid(_))), - "{name} {width}" - ); - } + fn a_scene_has_a_size_and_outputs() { let mut collection = example(); - let item = Item::new("camera", Rect::fill(&collection.canvas)); - collection.scenes[0].canvases.insert("tall".into(), vec![item]); + collection.scenes.push(Scene::new("a/b")); + collection.validate().unwrap(); + collection.scenes[1].published = true; assert!( matches!(collection.validate(), Err(Error::Invalid(_))), - "no such canvas" + "a / in a published name" ); - collection.canvases.insert("tall".into(), Extra::new(1080, 1920)); + collection.scenes[1].name = "Tall".into(); + collection.scenes[1].size = Some(Size { + width: 1081, + height: 1920, + }); + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "an odd width"); + collection.scenes[1].size = Some(Size { + width: 1080, + height: 1920, + }); collection.validate().unwrap(); - // Leaving a layout out is the one way to say a scene shows nothing there. - collection.scenes[0].canvases.insert("tall".into(), Vec::new()); + assert_eq!( + collection.canvas(Some("Tall")).unwrap().size(), + moq_video::Size::new(1080, 1920) + ); + assert_eq!( + collection.canvas(Some("Tall")).unwrap().framerate, + collection.canvas.framerate + ); assert!( - matches!(collection.validate(), Err(Error::Invalid(_))), - "an empty layout" + matches!(collection.takes("Tall"), Err(Error::Invalid(_))), + "another size" ); + collection.takes("Scene").unwrap(); + + assert_eq!(collection.recorded(), None, "the program"); + collection.scenes[1].recorded = true; + assert_eq!(collection.recorded(), Some("Tall")); + collection.scenes[0].recorded = true; + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "two recorded"); } #[test] diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index db0df021..ee77cfda 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -1,5 +1,5 @@ -//! Draw scenes onto each canvas on the GPU: a preview, a program, and the -//! transition between them. +//! Draw scenes on the GPU: a preview, a program and the transition between +//! scenes on it, and each scene an output reads on its own. use std::borrow::Cow; use std::collections::{BTreeMap, HashMap, HashSet}; @@ -22,43 +22,50 @@ use crate::source::{self, Feed, Sources, Status}; /// `moq_video::render` produces and what its exporter hands an encoder. pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; -/// Draws a [`Collection`]'s scenes onto each of its canvases: one texture for -/// the scene in preview, one for the scene on program, and the [`Transition`] -/// from one to the other. +/// Draws a [`Collection`]'s scenes: one texture for the scene in preview, one +/// for the program, which a [`Transition`] takes from one scene to another, +/// and one for each scene an output, such as a published broadcast or a +/// recording, reads on its own. /// /// The textures are plain [`wgpu::Texture`]s on the caller's device, so a UI -/// shows them without a copy and an encoder reads the program from the same -/// memory. [`render`](Self::render) redraws them from the newest picture each -/// source has, so a source that stalls repeats its last picture rather than -/// holding up the canvas. Each source's picture is uploaded once however many -/// canvases show it, and a canvas draws only the items it places. +/// shows them without a copy and an encoder reads them from the same memory. +/// [`render`](Self::render) redraws them from the newest picture each source +/// has, so a source that stalls repeats its last picture rather than holding +/// up the canvas. Each source's picture is uploaded once however many scenes +/// show it, and each scene is drawn at its own size. /// /// Pixels are blended in the gamma-encoded space sources arrive in, as a /// broadcast switcher and CSS do, so a 50% fade is a 50% mix of the stored /// values. /// -/// The canvases are linked: one scene is in preview and one on program on -/// every canvas, and a transition and each item's keyframes run on each at -/// once, on the clock [`render`](Self::render) is given, so every output drawn -/// from a program, such as the encoder and a recording, sees the same frame of -/// them. An item's keyframes start when its scene goes on program, and -/// separately when it goes into preview, for the preview, on every canvas -/// alike. +/// Everything is drawn on the clock [`render`](Self::render) is given, so +/// every output drawn from the program, such as the encoder and a recording, +/// sees the same frame of a transition. An item's keyframes start when its +/// scene goes on program, separately when it goes into preview, for the +/// preview, and when an output starts reading it, for that output. /// -/// A scene a [`Source::Scene`] draws is drawn first, at the main canvas's -/// size, into a texture of its own that is transparent where it draws -/// nothing, once per render however many items show it, and only while one -/// that is drawn does. Its keyframes play from when the scene drawing it went -/// on, so where preview and program show it at different moments of them, it -/// is drawn once for each. A [`Source::Program`] shows the main canvas's -/// program texture itself, drawn before anything showing it. +/// A scene a [`Source::Scene`] draws is drawn first, at its own size, into a +/// texture of its own that is transparent where it draws nothing, once per +/// render however many items show it, and only while one that is drawn does. +/// Its keyframes play from when the scene drawing it went on, so where two +/// textures show it at different moments of them, it is drawn once for each. +/// A [`Source::Program`] shows the program texture itself, drawn before +/// anything showing it. pub struct Compositor { device: wgpu::Device, queue: wgpu::Queue, effects: Effects, - main: Output, - extras: BTreeMap, + program: Program, + /// The scene in preview, at its own size, as of the last render. + preview: Target, + /// Whether the next render draws the preview, which it skips while no + /// one looks. + previewing: bool, + /// Whether the last render drew the preview. + previewed: bool, + /// The scenes drawn on their own for outputs, by name. + outputs: BTreeMap, layers: HashMap, /// The textures scenes drawn through scene sources are drawn into, by @@ -75,16 +82,15 @@ pub struct Compositor { /// When the program scene went on program, as `preview_start` is. program_start: Option, /// How long the preview scene had been in preview as of the last render: - /// the moment of its keyframes every canvas's preview texture shows. + /// the moment of its keyframes the preview texture shows. preview_shown: Duration, switch: Option, } -/// One canvas's preview and program, as of the last -/// [`render`](Compositor::render). -pub struct Output { +/// The program and the scenes a transition draws whole on its way, as of the +/// last [`render`](Compositor::render). +struct Program { canvas: Canvas, - preview: Target, program: Target, /// The scene coming in, drawn whole so it fades, slides, or is uncovered /// as one picture rather than item by item. @@ -93,33 +99,47 @@ pub struct Output { outgoing: Target, } -impl Output { +impl Program { fn new(device: &wgpu::Device, effects: &Effects, canvas: Canvas) -> Self { let target = |label| Target::new(device, effects, canvas, label); Self { canvas, - preview: target("moq-scene preview"), program: target("moq-scene program"), incoming: target("moq-scene incoming"), outgoing: target("moq-scene outgoing"), } } +} - /// The canvas this draws. - pub fn canvas(&self) -> Canvas { - self.canvas - } - - /// The scene in preview. - pub fn preview(&self) -> &wgpu::Texture { - &self.preview.texture - } +/// A scene an output, such as a broadcast or a recording, reads on its own, +/// as of the last [`render`](Compositor::render). +pub struct Output { + /// What it was last drawn into, at its size then. + target: Option, + /// When it went on, which its keyframes play from, or `None` until the + /// next render, which starts it. + start: Option, + /// Whether the next render draws it. + draw: bool, + /// Whether the last render drew it. + drawn: bool, +} - /// The program. Created with `COPY_SRC | TEXTURE_BINDING`, what +impl Output { + /// The scene as the last render drew it, or `None` if it did not. Created + /// with `COPY_SRC | TEXTURE_BINDING`, what /// `moq_video::render::export::Exporter::copy` takes to hand it to an /// encoder. - pub fn program(&self) -> &wgpu::Texture { - &self.program.texture + pub fn texture(&self) -> Option<&wgpu::Texture> { + self.target + .as_ref() + .filter(|_| self.drawn) + .map(|target| &target.texture) + } + + /// The canvas the last render drew it at, or `None` if it did not. + pub fn canvas(&self) -> Option { + self.target.as_ref().filter(|_| self.drawn).map(|target| target.canvas) } } @@ -153,6 +173,8 @@ impl Switch { /// A canvas-sized texture drawn into, and bound to sample. #[derive(Clone)] struct Target { + /// Its size, at the program's frame rate. + canvas: Canvas, texture: wgpu::Texture, picture: Picture, } @@ -267,8 +289,8 @@ impl effect::Library for Library<'_> { } impl Compositor { - /// A compositor drawing a main canvas of `canvas` on `device`, and no - /// further canvas until [`set_canvases`](Self::set_canvases) adds one. + /// A compositor drawing a program of `canvas` on `device`, and every + /// scene at the program's size until [`fit`](Self::fit) says otherwise. /// /// Fails if the canvas is invalid or the device cannot hold a texture that /// large. @@ -277,8 +299,11 @@ impl Compositor { let effects = Effects::new(device, queue, canvas.width, canvas.height); Ok(Self { - main: Output::new(device, &effects, canvas), - extras: BTreeMap::new(), + program: Program::new(device, &effects, canvas), + preview: Target::new(device, &effects, canvas, "moq-scene preview"), + previewing: true, + previewed: false, + outputs: BTreeMap::new(), device: device.clone(), queue: queue.clone(), effects, @@ -294,80 +319,90 @@ impl Compositor { }) } - /// The main canvas. + /// The program's canvas. pub fn canvas(&self) -> Canvas { - self.main.canvas + self.program.canvas } - /// Draw `collection`'s canvases from the next [`render`](Self::render) on: - /// its main canvas and each further one. A canvas of a new size gets new - /// textures of that size, blank until drawn; a new canvas starts blank, - /// and one `collection` no longer has is dropped. A transition under way - /// carries on. + /// Draw `collection`'s program and scenes at their sizes from the next + /// [`render`](Self::render) on. A program of a new size gets new textures + /// of that size, blank until drawn. A transition under way carries on. /// /// Fails, changing nothing, if a canvas is invalid or the device cannot /// hold a texture that large. - pub fn set_canvases(&mut self, collection: &Collection) -> Result<(), Error> { - let canvas = collection.canvas; - let extras: BTreeMap<&str, Canvas> = collection - .canvases - .iter() - .map(|(name, extra)| (name.as_str(), extra.canvas(canvas.framerate))) - .collect(); - for canvas in std::iter::once(&canvas).chain(extras.values()) { + pub fn fit(&mut self, collection: &Collection) -> Result<(), Error> { + let program = collection.canvas; + let scenes = collection.scenes.iter().map(|scene| scene.canvas(program)); + let canvases: Vec = std::iter::once(program).chain(scenes).collect(); + for canvas in &canvases { fits(&self.device, *canvas)?; } - let width = extras.values().map(|extra| extra.width).fold(canvas.width, u32::max); - let height = extras.values().map(|extra| extra.height).fold(canvas.height, u32::max); + let width = canvases.iter().map(|canvas| canvas.width).max().unwrap_or_default(); + let height = canvases.iter().map(|canvas| canvas.height).max().unwrap_or_default(); self.effects.fit(width, height); - let (device, effects) = (&self.device, &self.effects); - let resize = |output: &mut Output, canvas: Canvas| { - if output.canvas.size() == canvas.size() { - output.canvas = canvas; - } else { - *output = Output::new(device, effects, canvas); - } - }; - resize(&mut self.main, canvas); - self.extras.retain(|name, _| extras.contains_key(name.as_str())); - for (name, canvas) in extras { - match self.extras.get_mut(name) { - Some(output) => resize(output, canvas), - None => { - self.extras - .insert(name.to_string(), Output::new(device, effects, canvas)); - } - } + if self.program.canvas.size() == program.size() { + self.program.canvas = program; + } else { + self.program = Program::new(&self.device, &self.effects, program); } Ok(()) } - /// The scene in preview on the main canvas, as of the last - /// [`render`](Self::render). + /// The scene in preview, at its own size, as of the last + /// [`render`](Self::render), under the top layer when it is the program's + /// size. pub fn preview(&self) -> &wgpu::Texture { - self.main.preview() + &self.preview.texture } - /// The main canvas's program, as of the last [`render`](Self::render). See - /// [`Output::program`]. - pub fn program(&self) -> &wgpu::Texture { - self.main.program() + /// The canvas the preview was last drawn at. + pub fn preview_canvas(&self) -> Canvas { + self.preview.canvas } - /// The canvas `canvas` names, the main one for `None`, as of the last - /// [`render`](Self::render); `None` for a canvas it does not draw. - pub fn output(&self, canvas: Option<&str>) -> Option<&Output> { - match canvas { - None => Some(&self.main), - Some(name) => self.extras.get(name), + /// Whether the last [`render`](Self::render) drew the preview. + pub fn previewed(&self) -> bool { + self.previewed + } + + /// Whether [`render`](Self::render) draws the preview from now on, which + /// costs nothing while no one looks at it. + pub fn set_previewing(&mut self, previewing: bool) { + self.previewing = previewing; + } + + /// The program, as of the last [`render`](Self::render). Created with + /// `COPY_SRC | TEXTURE_BINDING`, what + /// `moq_video::render::export::Exporter::copy` takes to hand it to an + /// encoder. + pub fn program(&self) -> &wgpu::Texture { + &self.program.program.texture + } + + /// Keep the scenes `outputs` names as outputs from now on, each with + /// whether the next [`render`](Self::render) draws it, as it does while an + /// output reads it. Each plays its keyframes from the first render after + /// it was named, drawn or not, and one left out is forgotten, so it starts + /// over when named again. + pub fn set_outputs<'a>(&mut self, outputs: impl IntoIterator) { + let mut kept = BTreeMap::new(); + for (scene, draw) in outputs { + let output = self.outputs.remove(scene).unwrap_or(Output { + target: None, + start: None, + draw, + drawn: false, + }); + kept.insert(scene.to_string(), Output { draw, ..output }); } + self.outputs = kept; } - /// Every canvas drawn, by name: the main one first, as `None`. - pub fn outputs(&self) -> impl Iterator, &Output)> { - let extras = self.extras.iter().map(|(name, output)| (Some(name.as_str()), output)); - std::iter::once((None, &self.main)).chain(extras) + /// The output reading the scene called `scene`, as of the last + /// [`render`](Self::render), or `None` if it is not one. + pub fn output(&self, scene: &str) -> Option<&Output> { + self.outputs.get(scene) } /// The scene in preview. @@ -407,6 +442,16 @@ impl Compositor { self.program_scene.as_deref().into_iter().chain(leaving) } + /// The scenes drawn this render: those the program draws, and each output + /// drawn, each once. + pub fn live(&self) -> impl Iterator { + let outputs = self.outputs.iter().filter(|(_, output)| output.draw); + let outputs = outputs.map(|(scene, _)| scene.as_str()); + let on_air: Vec<&str> = self.on_air().collect(); + let outputs = outputs.filter(move |scene| !on_air.contains(scene)); + self.on_air().chain(outputs) + } + /// Put the scene called `scene` in preview, or nothing, playing its /// keyframes from the start, even if it was already there. pub fn set_preview(&mut self, scene: Option) { @@ -415,36 +460,46 @@ impl Compositor { } /// How long the preview scene had been in preview as of the last - /// [`render`](Self::render): the moment of its items' keyframes every - /// canvas's preview shows, since the canvases are linked, for - /// [`Item::posed`]. + /// [`render`](Self::render): the moment of its items' keyframes the + /// preview shows, for [`Item::posed`]. pub fn preview_since(&self) -> Duration { self.preview_shown } /// Follow each renamed scene to its new name, wherever it is: in preview, - /// on program, or leaving in a transition, and drop each removed one, so a - /// scene that later takes its name does not go on program without a cut. - /// `renames` maps each old name to its new one, or `None`, as - /// [`Collection::renames`] reads them, and each scene is looked up once, so - /// a name one scene gave up and another took is not followed twice. + /// on program, leaving in a transition, or read by an output, and drop each + /// removed one, so a scene that later takes its name does not go on + /// program without a cut. `renames` maps each old name to its new one, or + /// `None`, as [`Collection::renames`] reads them, and each scene is looked + /// up once, so a name one scene gave up and another took is not followed + /// twice. pub fn rename(&mut self, renames: &BTreeMap>) { for scene in self.scenes() { if let Some(to) = scene.as_ref().and_then(|name| renames.get(name)) { *scene = to.clone(); } } + let outputs = std::mem::take(&mut self.outputs); + for (scene, output) in outputs { + match renames.get(&scene) { + Some(Some(to)) => self.outputs.insert(to.clone(), output), + Some(None) => None, + None => self.outputs.insert(scene, output), + }; + } } - /// Drop each scene `collection` does not have from preview, program, and a - /// transition under way, such as one a reloaded file left out, so a scene - /// that later takes its name does not go on program without a cut. + /// Drop each scene `collection` does not have from preview, program, a + /// transition under way, and the outputs, such as one a reloaded file left + /// out, so a scene that later takes its name does not go on program + /// without a cut. pub fn retain(&mut self, collection: &Collection) { for scene in self.scenes() { if scene.as_deref().is_some_and(|name| collection.scene(name).is_none()) { *scene = None; } } + self.outputs.retain(|scene, _| collection.scene(scene).is_some()); } /// The scenes followed by name: in preview, on program, and leaving. @@ -458,8 +513,8 @@ impl Compositor { /// Put the preview on program at once, ending any transition, and /// stopping a stinger's clip in `sources`. /// - /// Refused, changing nothing, if the preview scene draws the program in - /// `collection`, which it would draw into itself. + /// Refused, changing nothing, unless the preview scene can go on program + /// in `collection`, as [`Collection::takes`] says. pub fn cut(&mut self, collection: &Collection, sources: &Sources) -> Result<(), Error> { self.can_take(collection)?; if let Some(switch) = self.switch.take() { @@ -470,13 +525,11 @@ impl Compositor { Ok(()) } - /// Refuse to take a preview scene that draws the program to program. + /// Refuse to take a preview scene to program that cannot go there. fn can_take(&self, collection: &Collection) -> Result<(), Error> { match self.preview_scene.as_deref() { - Some(scene) if collection.draws_program(scene) => Err(Error::Invalid(format!( - "`{scene}` draws the program, so it cannot go on program" - ))), - _ => Ok(()), + Some(scene) => collection.takes(scene), + None => Ok(()), } } @@ -553,8 +606,9 @@ impl Compositor { progress } - /// Draw preview and program on every canvas as of `now`, from the newest - /// picture each of `sources` has. + /// Draw the program, the preview unless no one looks at it, and each + /// output being read as of `now`, from the newest picture each of + /// `sources` has. /// /// A source whose picture fails to draw keeps its last one and reports why /// through [`error`](Self::error); this fails only when a canvas itself @@ -565,6 +619,9 @@ impl Compositor { let progress = self.progress(sources, now); let preview_start = *self.preview_start.get_or_insert(now); let program_start = *self.program_start.get_or_insert(now); + for output in self.outputs.values_mut() { + output.start.get_or_insert(now); + } self.update(collection, sources, now); self.restart(collection, preview_start, program_start); self.paint(collection, now); @@ -580,26 +637,58 @@ impl Compositor { (switch, from, progress) }); - // Each canvas's scenes and top layer, and the scenes they draw. - let shown: Vec = std::iter::once(preview).chain(on_program(program, switch)).collect(); + // Each texture at its scene's size: a scene that does not exist, or no + // scene, at the program's. + let program_canvas = self.program.canvas; + let canvas_of = |scene: Option<&str>| collection.canvas(scene).unwrap_or(program_canvas); + let previewing = self.previewing; + let preview_canvas = canvas_of(preview.0); + // The top layer goes over a preview of what could go on program. + let preview_top = preview_canvas.size() == program_canvas.size(); + if previewing { + resize( + &mut self.preview, + &self.device, + &self.effects, + preview_canvas, + "moq-scene preview", + ); + } + let mut outputs = Vec::new(); + for (scene, output) in &mut self.outputs { + output.drawn = output.draw; + if !output.draw { + continue; + } + let canvas = canvas_of(Some(scene)); + match &mut output.target { + Some(target) => resize(target, &self.device, &self.effects, canvas, "moq-scene output"), + None => output.target = Some(Target::new(&self.device, &self.effects, canvas, "moq-scene output")), + } + let since = now.saturating_duration_since(output.start.unwrap_or(now)); + outputs.push(( + On(Some(scene.as_str()), since), + output.target.clone().expect("made above"), + )); + } + + // Each texture's scenes and top layer, and the scenes they draw. let mut roots = Vec::new(); - for canvas in std::iter::once(None).chain(self.extras.keys().cloned().map(Some)) { - for On(scene, since) in &shown { - if let Some(scene) = scene { - let place = Place::scene(*scene).on(canvas.clone()); - roots.push((place, *since)); - } + if previewing { + roots.extend(preview.0.map(|scene| (Place::scene(scene), preview.1))); + if preview_top { + roots.push((Place::top(), Duration::ZERO)); } - roots.push((Place::top().on(canvas), Duration::ZERO)); + } + for On(scene, since) in on_program(program, switch) { + roots.extend(scene.map(|scene| (Place::scene(scene), since))); + } + roots.push((Place::top(), Duration::ZERO)); + for (On(scene, since), _) in &outputs { + roots.extend(scene.map(|scene| (Place::scene(scene), *since))); } let nesting = Nesting::of(collection, roots); - let targets = nested_targets( - &mut self.nested, - &self.device, - &self.effects, - self.main.canvas, - &nesting, - ); + let targets = nested_targets(&mut self.nested, &self.device, &self.effects, collection, &nesting); let mut frame = self.effects.frame(); let mut draw = Draw { @@ -607,8 +696,7 @@ impl Compositor { effects: &mut self.effects, library: Library(&self.layers), frame: &mut frame, - canvas: None, - size: self.main.canvas, + size: program_canvas, prepared: HashMap::new(), nesting: &nesting, scenes: HashMap::new(), @@ -621,18 +709,20 @@ impl Compositor { for (index, (nest, target)) in early { draw.nest(index, nest, target); } - draw.draw_program(&self.main, program, switch); - draw.program = Some(self.main.program.picture.clone()); + draw.draw_program(&self.program, program, switch); + draw.program = Some(self.program.program.picture.clone()); for (index, (nest, target)) in late { draw.nest(index, nest, target); } - draw.draw_preview(&self.main, preview); - for (name, output) in &self.extras { - draw.canvas = Some(name); - draw.size = output.canvas; - draw.draw_preview(output, preview); - draw.draw_program(output, program, switch); + if previewing { + draw.draw_preview(&self.preview, preview, preview_top); } + for (on, target) in &outputs { + draw.size = target.canvas; + draw.target(target); + draw.scene(target, *on); + } + self.previewed = previewing; self.effects.submit(frame); Ok(()) @@ -750,10 +840,12 @@ impl Compositor { /// Start each overlay set to restart over when a scene showing it went on /// since its timeline started: the program scene, which `program_start` - /// went on at, or else the preview scene, at `preview_start`, as their - /// keyframes do. Preview and program share one timeline, so a preview - /// never restarts an overlay program shows, through the scene a - /// transition leaves, the top layer, or a stinger's masks included. + /// went on at, or else the output that went on first among those showing + /// it, or else the preview scene, at `preview_start`, as their keyframes + /// do. Program, outputs, and preview share one timeline, so a preview + /// never restarts an overlay program or an output shows, through the scene + /// a transition leaves, the top layer, or a stinger's masks included, nor + /// does an output going on restart one program shows. fn restart(&mut self, collection: &Collection, preview_start: Instant, program_start: Instant) { let restarts = |layer: &Layer| matches!(layer.source, Source::Overlay { restart: true, .. }); if !self @@ -766,10 +858,22 @@ impl Compositor { let program = self.drawn(collection, self.program_scene.as_deref().map(Some)); let on_air = self.on_air_sources(collection); let preview = self.drawn(collection, self.preview_scene.as_deref().map(Some)); + let outputs: Vec<(HashSet<&str>, Instant)> = self + .outputs + .iter() + .filter_map(|(scene, output)| Some((self.drawn(collection, [Some(scene.as_str())]), output.start?))) + .collect(); for (name, layer) in &mut self.layers { let name = name.as_str(); + let output = outputs + .iter() + .filter(|(shown, _)| shown.contains(name)) + .map(|(_, start)| *start) + .min(); let start = if program.contains(name) { program_start + } else if let Some(start) = output.filter(|_| !on_air.contains(name)) { + start } else if preview.contains(name) && !on_air.contains(name) { preview_start } else { @@ -784,9 +888,10 @@ impl Compositor { } } - /// Paint each overlay preview or program draws as of `now`, where it moved - /// since its last paint. One drawn nowhere waits: its CSS runs on the - /// program clock, so it lands where it should whenever it is drawn again. + /// Paint each overlay preview, program, or an output draws as of `now`, + /// where it moved since its last paint. One drawn nowhere waits: its CSS + /// runs on the program clock, so it lands where it should whenever it is + /// drawn again. fn paint(&mut self, collection: &Collection, now: Instant) { let Some(overlays) = &self.overlays else { return }; let shown = self.shown(collection); @@ -826,19 +931,23 @@ impl Compositor { } } - /// The sources whose pictures preview and program draw on any canvas: - /// each visible item's, through the scenes they draw, a stinger's clip - /// under way, and each mask's, through clones. + /// The sources whose pictures the next render draws: each visible item's + /// of preview, program, and each output drawn, through the scenes they + /// draw, a stinger's clip under way, and each mask's, through clones. fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { let mut shown = self.on_air_sources(collection); - shown.extend(self.drawn(collection, self.preview_scene.as_deref().map(Some))); + if self.previewing { + shown.extend(self.drawn(collection, [self.preview_scene.as_deref(), None])); + } + let outputs = self.outputs.iter().filter(|(_, output)| output.draw); + shown.extend(self.drawn(collection, outputs.map(|(scene, _)| Some(scene.as_str())))); shown } - /// The sources whose pictures program draws on any canvas: each visible - /// item's in the scene on program, the scene a transition leaves, and the - /// top layer, through the scenes they draw, a stinger's clip under way, - /// and each mask's, through clones. + /// The sources whose pictures program draws: each visible item's in the + /// scene on program, the scene a transition leaves, and the top layer, + /// through the scenes they draw, a stinger's clip under way, and each + /// mask's, through clones. fn on_air_sources<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { let mut shown = self.drawn(collection, self.on_air().map(Some).chain([None])); // Drawn through its source's filters alone. @@ -854,21 +963,16 @@ impl Compositor { shown } - /// The sources whose pictures `scenes` draw on any canvas, `None` being - /// the top layer: each visible item's, through the scenes they draw, and - /// each mask's, through clones. + /// The sources whose pictures `scenes` draw, `None` being the top layer: + /// each visible item's, through the scenes they draw, and each mask's, + /// through clones. fn drawn<'a, 'b>( &self, collection: &'a Collection, scenes: impl IntoIterator>, ) -> HashSet<&'a str> { - let scenes: Vec> = scenes.into_iter().collect(); - let canvases = std::iter::once(None).chain(self.extras.keys().map(Some)); - let places = canvases.flat_map(|canvas| { - scenes.iter().map(move |scene| Place { - canvas: canvas.cloned(), - scene: scene.map(str::to_string), - }) + let places = scenes.into_iter().map(|scene| Place { + scene: scene.map(str::to_string), }); let mut shown = HashSet::new(); for item in collection.drawn(places) { @@ -879,23 +983,32 @@ impl Compositor { } } -/// A texture of `canvas` for each scene `nesting` draws, reusing those in -/// `pool`, the last render's, and leaving there only those used now. +/// `target` at `canvas`'s size, made anew if it was another, labeled `label`. +fn resize(target: &mut Target, device: &wgpu::Device, effects: &Effects, canvas: Canvas, label: &str) { + if target.canvas.size() == canvas.size() { + target.canvas = canvas; + } else { + *target = Target::new(device, effects, canvas, label); + } +} + +/// A texture of its scene's size in `collection` for each scene `nesting` +/// draws, reusing those in `pool`, the last render's, and leaving there only +/// those used now. fn nested_targets( pool: &mut HashMap>, device: &wgpu::Device, effects: &Effects, - canvas: Canvas, + collection: &Collection, nesting: &Nesting, ) -> Vec { let mut free = std::mem::take(pool); let mut targets = Vec::with_capacity(nesting.order.len()); for nest in &nesting.order { + let canvas = collection.canvas(Some(nest.scene)).unwrap_or(collection.canvas); let reused = free.get_mut(nest.scene).and_then(Vec::pop); let target = match reused { - Some(target) if (target.texture.width(), target.texture.height()) == (canvas.width, canvas.height) => { - target - } + Some(target) if target.canvas.size() == canvas.size() => target, _ => Target::new(device, effects, canvas, "moq-scene nested"), }; pool.entry(nest.scene.to_string()).or_default().push(target.clone()); @@ -1248,20 +1361,18 @@ struct Draw<'a> { effects: &'a mut Effects, library: Library<'a>, frame: &'a mut Frame, - /// The canvas drawn now: a further one by name, or `None` for the main one. - canvas: Option<&'a str>, - /// Its size and frame rate. + /// The size of the target drawn now. size: Canvas, /// Each item, filtered once though it is drawn onto several targets, by - /// where it is, on which canvas, and its [`Item::settled`] time, as two - /// outputs showing its scene can be at different points of its keyframes. + /// where it is and its [`Item::settled`] time, as two textures showing its + /// scene can be at different points of its keyframes. prepared: HashMap<(Place, usize, Settled), Option>, /// The scenes drawn through scene sources this render. nesting: &'a Nesting<'a>, /// The picture of each of `nesting`'s scenes drawn so far, by its place /// in the order. scenes: HashMap, - /// The main canvas's program, once drawn. + /// The program, once drawn. program: Option, } @@ -1269,24 +1380,29 @@ impl Draw<'_> { /// Draw the scene `nest` names, the `index`th of the nesting, into /// `target`, transparent where it draws nothing, for items to show. fn nest(&mut self, index: usize, nest: &Nest, target: &Target) { + self.size = target.canvas; self.frame.target(&target.picture, self.size, wgpu::Color::TRANSPARENT); self.layer(target, Place::scene(nest.scene), nest.since); self.scenes.insert(index, target.picture.clone()); } - /// Draw `output`'s preview, showing `preview` under the top layer. - fn draw_preview(&mut self, output: &Output, preview: On) { - // The top layer goes over both outputs, and over a transition rather - // than through it, so it stays on across the switch. - self.target(&output.preview); - self.scene(&output.preview, preview); - self.layer(&output.preview, self.place(None), Duration::ZERO); + /// Draw `preview` showing the scene `on`, under the top layer if `top`. + fn draw_preview(&mut self, preview: &Target, on: On, top: bool) { + self.size = preview.canvas; + self.target(preview); + self.scene(preview, on); + // The top layer goes over both preview and program, and over a + // transition rather than through it, so it stays on across the switch. + if top { + self.layer(preview, Place::top(), Duration::ZERO); + } } /// Draw `output`'s program, showing `program`, or `switch` to it from the /// scene it leaves, `progress` of the way there, under the top layer. - fn draw_program(&mut self, output: &Output, program: On, switch: Option<(&Switch, On, f32)>) { - let top = self.place(None); + fn draw_program(&mut self, output: &Program, program: On, switch: Option<(&Switch, On, f32)>) { + self.size = output.canvas; + let top = Place::top(); let to = program; match switch { Some((switch, from, progress)) => { @@ -1337,14 +1453,6 @@ impl Draw<'_> { self.layer(&output.program, top, Duration::ZERO); } - /// `scene`'s layout on the canvas drawn now, or its top layer for `None`. - fn place(&self, scene: Option<&str>) -> Place { - Place { - canvas: self.canvas.map(str::to_string), - scene: scene.map(str::to_string), - } - } - /// Start drawing `target`, a canvas, opaque black where nothing is drawn. fn target(&mut self, target: &Target) { self.frame.target(&target.picture, self.size, wgpu::Color::BLACK); @@ -1353,7 +1461,7 @@ impl Draw<'_> { /// Draw `on`'s items onto `target`. A missing scene draws nothing. fn scene(&mut self, target: &Target, on: On) { if let Some(scene) = on.0.filter(|name| self.collection.scene(name).is_some()) { - self.layer(target, self.place(Some(scene)), on.1); + self.layer(target, Place::scene(scene), on.1); } } @@ -1461,7 +1569,7 @@ impl Draw<'_> { fn moving(&mut self, target: &Target, from: On, to: On, progress: f32) { let collection = self.collection; let items = |scene: Option<&str>| match scene { - Some(scene) => collection.items(&self.place(Some(scene))).unwrap_or_default(), + Some(scene) => collection.items(&Place::scene(scene)).unwrap_or_default(), None => &[], }; let posed = |on: On| -> Vec> { items(on.0).iter().map(|item| item.posed(on.1)).collect() }; @@ -1488,7 +1596,7 @@ impl Draw<'_> { slots.into_iter().map(|(to, from)| (from, to)).collect() }; - let place = self.place(Some(to.0.unwrap_or_default())); + let place = Place::scene(to.0.unwrap_or_default()); for slot in slots { match slot { (Some(at), None) => { @@ -1566,8 +1674,8 @@ impl Draw<'_> { impl Draw<'_> { /// The picture `item` shows, `since` its scene went on: its source's - /// newest, the scene it draws as drawn this render, or the main canvas's - /// program once drawn. + /// newest, the scene it draws as drawn this render, or the program once + /// drawn. fn shows(&self, item: &Item, since: Duration) -> Option { match self.collection.nested(&item.source) { Some(Nested::Scene(scene)) => { @@ -1641,6 +1749,7 @@ fn upload( } impl Target { + /// A target of `canvas`'s size, labeled `label`. fn new(device: &wgpu::Device, effects: &Effects, canvas: Canvas, label: &str) -> Self { let texture = device.create_texture(&wgpu::TextureDescriptor { label: Some(label), @@ -1661,7 +1770,11 @@ impl Target { }); // A canvas is opaque, which is premultiplied too. let picture = effects.picture(&texture); - Self { texture, picture } + Self { + canvas, + texture, + picture, + } } } @@ -1670,7 +1783,7 @@ mod tests { use std::time::Duration; use super::*; - use crate::collection::{Blend, Extra, MaskMode, Rgba, Scene, Video}; + use crate::collection::{Blend, MaskMode, Rgba, Scene, Size, Video}; use crate::source::test::{context, listen, local, start}; /// A fade over a second. @@ -1985,44 +2098,69 @@ mod tests { .await; } - /// A further canvas draws its own layout of each scene and its own top - /// layer, from the same upload of each source, and cuts and fades with - /// the main canvas. A scene without a layout there shows only the top - /// layer, and a canvas the collection drops is no longer drawn. + /// `collection()` with `tall`, a 4x8 scene showing the program over its + /// top half and `items` under it, through the program source `on air`. + fn tall(items: Vec) -> Collection { + let mut collection = collection(); + collection.sources.insert("on air".into(), Source::Program); + let top = Rect { + x: 0.0, + y: 0.0, + width: 4.0, + height: 4.0, + }; + collection.scenes.push(Scene { + size: Some(Size { width: 4, height: 8 }), + items: [Item::new("on air", top)].into_iter().chain(items).collect(), + ..Scene::new("tall") + }); + collection.validate().unwrap(); + collection + } + + /// The bottom half of the 4x8 scene `tall`. + const BOTTOM: Rect = Rect { + x: 0.0, + y: 4.0, + width: 4.0, + height: 4.0, + }; + + /// Two pixels of `texture`, a 4x8 one: on the left and on the right, a row + /// from the bottom. + fn tall_pixels(studio: &mut Studio, texture: &wgpu::Texture) -> ([u8; 4], [u8; 4]) { + let rgba = studio.read(texture); + let pixel = |x: usize, y: usize| -> [u8; 4] { rgba[(y * 4 + x) * 4..][..4].try_into().unwrap() }; + (pixel(1, 6), pixel(3, 6)) + } + + /// An output draws its scene at its own size, from the same upload of + /// each source as the program, following the program's fade through a + /// program source. The preview shows a scene of another size at that + /// size, without the top layer, and it cannot go on program. An output + /// not drawn has no picture, one left out is forgotten, and a preview no + /// one looks at is not drawn. #[tokio::test] #[ignore = "needs a Vulkan device; run by just rs render-sw"] - async fn a_further_canvas_draws_its_own_layout() { + async fn an_output_draws_its_scene_at_its_own_size() { local(async { - let mut collection = collection(); - let mut tall = Extra::new(4, 8); - let bottom = Rect { - x: 0.0, - y: 4.0, - width: 4.0, - height: 4.0, - }; - tall.top.push(Item::new("red", bottom)); - collection.canvases.insert("tall".into(), tall); - let top = Rect { y: 0.0, ..bottom }; - collection.scenes[0] - .canvases - .insert("tall".into(), vec![Item::new("blue", top)]); - collection.validate().unwrap(); + let mut collection = tall(vec![Item::new("red", BOTTOM)]); + collection.top.push(Item::new("blue", BOTTOM)); let mut studio = Studio::new(collection).await; - studio.compositor.set_canvases(&studio.collection).unwrap(); - let tall = |studio: &mut Studio| { - let texture = studio.compositor.output(Some("tall")).unwrap().program().clone(); - let rgba = studio.read(&texture); - let pixel = |x: usize, y: usize| -> [u8; 4] { rgba[(y * 4 + x) * 4..][..4].try_into().unwrap() }; - (pixel(1, 1), pixel(1, 6)) - }; - + studio.compositor.fit(&studio.collection).unwrap(); let start = Instant::now(); studio.compositor.set_preview(Some("red".into())); studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.compositor.set_outputs([("tall", true)]); studio.render(start); - assert_eq!(pixel(&studio.program(), 1, 1), RED); - assert_eq!(tall(&mut studio), (BLUE, RED)); + let output = studio.compositor.output("tall").unwrap(); + assert_eq!(output.canvas().unwrap().size(), moq_video::Size::new(4, 8)); + let texture = output.texture().unwrap().clone(); + let rgba = studio.read(&texture); + let pixel = |x: usize, y: usize| -> [u8; 4] { rgba[(y * 4 + x) * 4..][..4].try_into().unwrap() }; + assert_eq!(pixel(0, 0), RED, "the program's red, shrunk"); + assert_eq!(pixel(3, 0), BLUE, "and its corner"); + assert_eq!(tall_pixels(&mut studio, &texture), (RED, RED), "its own item"); assert_eq!(studio.compositor.layers.len(), 2, "each source is uploaded once"); studio.compositor.set_preview(Some("blue".into())); @@ -2031,19 +2169,67 @@ mod tests { .transition(&studio.collection, FADE, &studio.sources, start) .unwrap(); studio.render(start + Duration::from_millis(500)); - let (fading, kept) = tall(&mut studio); - assert!(near(fading, [0, 0, 128, 255]), "{fading:?}"); - assert_eq!(kept, RED, "the top layer does not fade"); + let texture = studio.compositor.output("tall").unwrap().texture().unwrap().clone(); + let rgba = studio.read(&texture); + let faded = rgba[..4].try_into().unwrap(); + assert!(near(faded, [128, 0, 127, 255]), "the program as it fades: {faded:?}"); + studio.compositor.set_preview(Some("tall".into())); studio.render(start + Duration::from_secs(1)); - assert_eq!(pixel(&studio.program(), 1, 1), BLUE); - assert_eq!(tall(&mut studio), (BLACK, RED), "no layout leaves the top layer"); + assert_eq!(studio.compositor.preview_canvas().size(), moq_video::Size::new(4, 8)); + let preview = studio.compositor.preview().clone(); + assert_eq!( + tall_pixels(&mut studio, &preview), + (RED, RED), + "its own items, not the top layer" + ); + let err = studio.compositor.cut(&studio.collection, &studio.sources).unwrap_err(); + assert!(err.to_string().contains("not the program's"), "{err}"); - studio.collection.canvases.clear(); - studio.collection.scenes[0].canvases.clear(); - studio.compositor.set_canvases(&studio.collection).unwrap(); - assert!(studio.compositor.output(Some("tall")).is_none()); - assert_eq!(studio.compositor.outputs().count(), 1); + studio.compositor.set_outputs([("tall", false)]); + studio.compositor.set_previewing(false); + studio.render(start + Duration::from_secs(1)); + assert!(studio.compositor.output("tall").unwrap().texture().is_none()); + assert!(!studio.compositor.previewed()); + studio.compositor.set_outputs([]); + assert!(studio.compositor.output("tall").is_none()); + }) + .await; + } + + /// An output's items play their keyframes from when it was first named, + /// drawn or not, and from the start again once named anew. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn an_output_plays_keyframes_from_when_it_went_on() { + local(async { + let mut slide = slide_in(); + slide[0].x = -4.0; + let collection = tall(vec![Item { + keyframes: slide, + ..Item::new("blue", BOTTOM) + }]); + let mut studio = Studio::new(collection).await; + studio.compositor.fit(&studio.collection).unwrap(); + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + let drawn = |studio: &mut Studio| { + let texture = studio.compositor.output("tall").unwrap().texture().unwrap().clone(); + tall_pixels(studio, &texture) + }; + + studio.compositor.set_outputs([("tall", false)]); + studio.render(at(0)); + studio.compositor.set_outputs([("tall", true)]); + studio.render(at(500)); + assert_eq!(drawn(&mut studio), (BLUE, BLACK), "halfway in"); + + studio.compositor.set_outputs([]); + studio.compositor.set_outputs([("tall", true)]); + studio.render(at(500)); + assert_eq!(drawn(&mut studio), (BLACK, BLACK), "from the start again"); + studio.render(at(1500)); + assert_eq!(drawn(&mut studio), (BLUE, BLUE), "landed"); }) .await; } @@ -2224,69 +2410,6 @@ mod tests { ] } - /// A further canvas's items play their keyframes as the main canvas's do: - /// from when the scene went into preview, for its preview, and from when it - /// went on program, for its program, each drawn on its own canvas though - /// both canvases are at the same moment. - #[tokio::test] - #[ignore = "needs a Vulkan device; run by just rs render-sw"] - async fn a_further_canvas_plays_keyframes_with_the_main_one() { - local(async { - let mut collection = collection(); - collection.scenes[1].items[0].keyframes = slide_in(); - collection.canvases.insert("tall".into(), Extra::new(4, 8)); - let bottom = Rect { - x: 0.0, - y: 4.0, - width: 4.0, - height: 4.0, - }; - let mut slide = slide_in(); - slide[0].x = -4.0; - collection.scenes[1].canvases.insert( - "tall".into(), - vec![Item { - keyframes: slide, - ..Item::new("blue", bottom) - }], - ); - collection.validate().unwrap(); - let mut studio = Studio::new(collection).await; - studio.compositor.set_canvases(&studio.collection).unwrap(); - let tall = |studio: &mut Studio, preview: bool| { - let output = studio.compositor.output(Some("tall")).unwrap(); - let texture = match preview { - true => output.preview().clone(), - false => output.program().clone(), - }; - let rgba = studio.read(&texture); - let pixel = |x: usize, y: usize| -> [u8; 4] { rgba[(y * 4 + x) * 4..][..4].try_into().unwrap() }; - (pixel(1, 6), pixel(3, 6)) - }; - let start = Instant::now(); - let at = |millis| start + Duration::from_millis(millis); - studio.compositor.set_preview(Some("red".into())); - studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); - studio.compositor.set_preview(Some("blue".into())); - - studio.render(at(0)); - studio.render(at(500)); - assert_eq!(tall(&mut studio, true), (BLUE, BLACK), "halfway in, in preview"); - assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the main canvas too"); - - studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); - studio.render(at(500)); - assert_eq!(tall(&mut studio, false), (BLACK, BLACK), "on program from the start"); - assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "the main canvas too"); - assert_eq!(tall(&mut studio, true), (BLUE, BLACK), "the preview carries on"); - - studio.render(at(1500)); - assert_eq!(tall(&mut studio, false), (BLUE, BLUE), "landed on the further canvas"); - assert_eq!(pixel(&studio.program(), 6, 6), BLUE, "and on the main one"); - }) - .await; - } - /// An item's keyframes play from when its scene went into preview, for the /// preview, and from when it went on program, for the program, so one scene /// can be at two points of them at once. @@ -3383,7 +3506,7 @@ mod tests { ..studio.collection.canvas }; studio.collection.canvas = wide; - studio.compositor.set_canvases(&studio.collection).unwrap(); + studio.compositor.fit(&studio.collection).unwrap(); assert_ne!(studio.compositor.program(), &old, "a new texture"); let fill = Rect::fill(&wide); studio.collection.scenes[0].items = vec![ @@ -3422,15 +3545,9 @@ mod tests { collection.canvas = canvas; collection }; - assert!(matches!( - studio.compositor.set_canvases(&refused(huge)), - Err(Error::Invalid(_)) - )); + assert!(matches!(studio.compositor.fit(&refused(huge)), Err(Error::Invalid(_)))); let empty = Canvas { width: 0, ..wide }; - assert!(matches!( - studio.compositor.set_canvases(&refused(empty)), - Err(Error::Invalid(_)) - )); + assert!(matches!(studio.compositor.fit(&refused(empty)), Err(Error::Invalid(_)))); assert_eq!(studio.compositor.canvas(), wide); }) .await; @@ -4180,8 +4297,8 @@ mod tests { } } -/// What drawing more canvases costs, against how many sources the collection -/// has and how many each canvas places. Run by hand: +/// What drawing more published scenes costs, against how many sources the +/// collection has and how many each published scene places. Run by hand: /// `cargo nextest run -p moq-scene --run-ignored only --no-capture -E 'test(/^compositor::bench::/)'`. #[cfg(test)] mod bench { @@ -4189,17 +4306,17 @@ mod bench { use super::tests::Studio; use super::*; - use crate::collection::{Extra, Rgba}; + use crate::collection::{Rgba, Size}; use crate::source::test::local; /// Frames timed per case, after as many to warm up. const FRAMES: u32 = 60; - /// A collection of `sources` color sources on a 1080p main canvas and - /// `canvases - 1` portrait ones. The main canvas places every source, and - /// each further canvas the first `placed`, each item a quarter of the - /// canvas at half opacity, so every one draws. - fn collection(sources: usize, canvases: usize, placed: usize) -> Collection { + /// A collection of `sources` color sources on a 1080p program and + /// `published` portrait scenes. The program's scene places every source, + /// and each portrait scene the first `placed`, each item a quarter of the + /// scene at half opacity, so every one draws. + fn collection(sources: usize, published: usize, placed: usize) -> Collection { let mut collection = Collection::new(); let item = |index: usize, canvas: &Canvas| { let quarter = Rect { @@ -4221,30 +4338,49 @@ mod bench { } let main = collection.canvas; collection.scenes[0].items = (0..sources).map(|index| item(index, &main)).collect(); - for canvas in 1..canvases { - let name = format!("portrait {canvas}"); - let extra = Extra::new(1080, 1920); - let size = extra.canvas(main.framerate); - collection.canvases.insert(name.clone(), extra); - let items = (0..placed.min(sources)).map(|index| item(index, &size)).collect(); - collection.scenes[0].canvases.insert(name, items); + for scene in 0..published { + let portrait = Canvas { + width: 1080, + height: 1920, + ..main + }; + collection.scenes.push(crate::Scene { + size: Some(Size { + width: portrait.width, + height: portrait.height, + }), + published: true, + items: (0..placed.min(sources)).map(|index| item(index, &portrait)).collect(), + ..crate::Scene::new(format!("portrait {scene}")) + }); } collection.validate().unwrap(); collection } - /// The mean time to draw and finish every canvas once. + /// The mean time to draw and finish the program and every published scene + /// once. async fn frame(collection: Collection) -> Duration { let scene = collection.scenes[0].name.clone(); cost(collection, &scene).await.0 } - /// The mean time to draw and finish every canvas once with the first scene - /// on program and `preview` in preview, and the render passes and texture - /// copies each frame runs. + /// The mean time to draw and finish the program and every published scene + /// once with the first scene on program and `preview` in preview, and the + /// render passes and texture copies each frame runs. async fn cost(collection: Collection, preview: &str) -> (Duration, (usize, usize)) { let mut studio = Studio::new(collection).await; - studio.compositor.set_canvases(&studio.collection).unwrap(); + studio.compositor.fit(&studio.collection).unwrap(); + let published: Vec = studio + .collection + .scenes + .iter() + .filter(|scene| scene.published) + .map(|scene| scene.name.clone()) + .collect(); + studio + .compositor + .set_outputs(published.iter().map(|scene| (scene.as_str(), true))); let scene = studio.collection.scenes[0].name.clone(); studio.compositor.set_preview(Some(scene)); studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); @@ -4271,7 +4407,7 @@ mod bench { local(async { println!("sources case frame passes copies"); for sources in [4, 16, 64] { - let direct = collection(sources, 1, sources); + let direct = collection(sources, 0, sources); let scene = direct.scenes[0].name.clone(); let mut nested = direct.clone(); @@ -4307,19 +4443,19 @@ mod bench { .await; } - /// Sources against canvases, each further canvas placing every source - /// (dense) or two (sparse). A sparse canvas should cost about the same - /// however many sources the collection has. + /// Sources against published scenes, each placing every source (dense) or + /// two (sparse). A sparse scene should cost about the same however many + /// sources the collection has. #[tokio::test] #[ignore = "a benchmark; run by hand on a GPU"] - async fn sources_against_canvases() { + async fn sources_against_published_scenes() { local(async { - println!("sources canvases dense sparse"); + println!("sources published dense sparse"); for sources in [4, 16, 64] { - for canvases in [1, 2, 4] { - let dense = frame(collection(sources, canvases, sources)).await; - let sparse = frame(collection(sources, canvases, 2)).await; - println!("{sources:>7} {canvases:>8} {dense:>7.2?} {sparse:>7.2?}"); + for published in [0, 1, 3] { + let dense = frame(collection(sources, published, sources)).await; + let sparse = frame(collection(sources, published, 2)).await; + println!("{sources:>7} {published:>9} {dense:>7.2?} {sparse:>7.2?}"); } } }) diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index 04ca97a0..105bcb1e 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -14,9 +14,7 @@ use std::collections::{BTreeMap, VecDeque}; use serde::{Deserialize, Serialize}; use crate::Error; -use crate::collection::{ - Audio, Canvas, Collection, Extra, Filter, Item, Layer, Mixes, Scene, Source, Transition, Video, -}; +use crate::collection::{Audio, Canvas, Collection, Filter, Item, Layer, Mixes, Scene, Source, Transition, Video}; /// One part of a collection, before and after an edit. /// @@ -72,14 +70,6 @@ pub enum Change { Transition { before: Transition, after: Transition }, /// [`Collection::canvas`]. Canvas { before: Canvas, after: Canvas }, - /// A [`Collection::canvases`] entry. - Extra { - name: String, - #[serde(default, skip_serializing_if = "Option::is_none")] - before: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] - after: Option, - }, /// [`Collection::mixes`]. Mixes { before: Mixes, after: Mixes }, } @@ -127,11 +117,6 @@ impl Change { before: after, after: before, }, - Self::Extra { name, before, after } => Self::Extra { - name, - before: after, - after: before, - }, Self::Mixes { before, after } => Self::Mixes { before: after, after: before, @@ -147,7 +132,6 @@ impl Change { | Self::Item { .. } | Self::Transition { .. } | Self::Canvas { .. } - | Self::Extra { .. } | Self::Mixes { .. } => None, } } @@ -162,7 +146,6 @@ impl Change { Self::Item { before, after, .. } => before == after, Self::Transition { before, after } => before == after, Self::Canvas { before, after } => before == after, - Self::Extra { before, after, .. } => before == after, Self::Mixes { before, after } => before == after, } } @@ -252,18 +235,6 @@ impl Change { (Self::Canvas { before, after }, Self::Canvas { before: b, after: a }) if after == *b => { Self::Canvas { before, after: *a } } - ( - Self::Extra { name, before, after }, - Self::Extra { - name: n, - before: b, - after: a, - }, - ) if name == *n && after == *b => Self::Extra { - name, - before, - after: a.clone(), - }, (Self::Mixes { before, after }, Self::Mixes { before: b, after: a }) if after == *b => Self::Mixes { before, after: a.clone(), @@ -307,7 +278,6 @@ impl Change { collection.canvas = *after; Ok(()) } - Self::Extra { name, before, after } => replace(&mut collection.canvases, name, before, after, "canvas"), Self::Mixes { before, after } => { if collection.mixes != *before { return Err(conflict("the mixes changed".into())); @@ -784,54 +754,31 @@ impl Draft { }) } - /// Set the canvas's size and frame rate. Items keep their place in canvas - /// pixels. + /// Set the program's size and frame rate, which every scene without a + /// size of its own follows. Items keep their place in canvas pixels. pub fn set_canvas(&mut self, canvas: Canvas) -> Result<(), Error> { let before = self.collection.canvas; self.change(Change::Canvas { before, after: canvas }) } - /// Add or resize the further canvas called `name`, keeping its top layer - /// and every scene's layout on it. Items keep their place in canvas - /// pixels. - pub fn set_extra(&mut self, name: &str, width: u32, height: u32) -> Result<(), Error> { - let before = self.collection.canvases.get(name).cloned(); - let after = match &before { - Some(extra) => Extra { - width, - height, - ..extra.clone() - }, - None => Extra::new(width, height), - }; - self.change(Change::Extra { - name: name.to_string(), - before, - after: Some(after), - }) - } - - /// Remove the further canvas called `name`, with its top layer and every - /// scene's layout on it. - pub fn remove_extra(&mut self, name: &str) -> Result<(), Error> { - let laid: Vec<(usize, Scene)> = self - .collection - .scenes - .iter() - .enumerate() - .filter(|(_, scene)| scene.canvases.contains_key(name)) - .map(|(index, scene)| (index, scene.clone())) + /// Have recordings hold the scene called `scene`, or the program with + /// `None`, rather than the one they hold now. + pub fn set_recorded(&mut self, scene: Option<&str>) -> Result<(), Error> { + let scenes = self.collection.scenes.iter().enumerate(); + let changed: Vec<(usize, Scene)> = scenes + .filter(|(_, at)| at.recorded != (Some(at.name.as_str()) == scene)) + .map(|(index, at)| (index, at.clone())) .collect(); - for (index, mut scene) in laid { - scene.canvases.remove(name); - self.set_scene(index, scene)?; + if let Some(scene) = scene + && self.collection.scene(scene).is_none() + { + return Err(conflict(format!("there is no scene `{scene}`"))); } - let before = self.collection.canvases.get(name).cloned(); - self.change(Change::Extra { - name: name.to_string(), - before, - after: None, - }) + for (index, at) in changed { + let recorded = !at.recorded; + self.set_scene(index, Scene { recorded, ..at })?; + } + Ok(()) } /// Set the mixes and which ones the program and recordings carry, and @@ -1568,97 +1515,58 @@ mod tests { assert_eq!(collection, start); } - /// A further canvas is added, laid out, and resized as edits; removing a - /// source takes its items off every canvas; and removing the canvas takes - /// its top layer and every scene's layout on it, undoing whole. A layout - /// emptied item by item reads as no layout, and the file round-trips. + /// A scene's size, whether it is published, and the scene recordings hold + /// are edits like any other, kept through a rename. Recording another + /// scene moves the pick, and removing the recorded scene sends recordings + /// back to the program, undoing whole. #[test] - fn a_further_canvas_edits_and_removes_whole() { + fn a_scene_resizes_publishes_and_records() { let mut collection = collection(); - collection.sources.insert("blue".into(), color([0, 0, 255, 255])); + collection.scenes.push(Scene::new("Tall")); + collection.validate().unwrap(); + let start = collection.clone(); let mut history = History::new(); - let tall = Layer::scene("Scene").on(Some("tall".into())); - let over = Layer::top().on(Some("tall".into())); - let fill = Rect::fill(&Canvas { + + let tall = collection.scenes[1].clone(); + let mut draft = collection.draft("Publish a portrait scene"); + let size = Some(crate::collection::Size { width: 1080, height: 1920, - framerate: 30, }); - - let mut draft = collection.draft("Add a canvas"); - assert!( - draft.insert_item(&tall, 0, Item::new("red", fill)).is_err(), - "no canvas yet" - ); - draft.set_extra("tall", 1080, 1920).unwrap(); - draft.insert_item(&tall, 0, Item::new("red", fill)).unwrap(); - draft.insert_item(&tall, 1, Item::new("blue", fill)).unwrap(); - draft.insert_item(&over, 0, Item::new("blue", fill)).unwrap(); - commit(&mut collection, &mut history, draft, None); - let added = collection.clone(); - assert_eq!(collection.items(&tall).unwrap().len(), 2); - assert_eq!(collection.canvas(Some("tall")).unwrap().framerate, 30); - assert_eq!( - Collection::from_toml(&collection.to_toml().unwrap()).unwrap(), - collection - ); - - let mut draft = collection.draft("Resize"); - draft.set_extra("tall", 720, 1280).unwrap(); + draft + .set_scene( + 1, + Scene { + size, + published: true, + ..tall + }, + ) + .unwrap(); + draft.set_recorded(Some("Tall")).unwrap(); commit(&mut collection, &mut history, draft, None); - assert_eq!(collection.canvases["tall"].top.len(), 1, "the top layer stays"); + assert_eq!(collection.canvas(Some("Tall")).unwrap().height, 1920); + assert_eq!(collection.recorded(), Some("Tall")); - let mut draft = collection.draft("Remove blue"); - draft.remove_source("blue").unwrap(); + let mut draft = collection.draft("Rename"); + draft.rename_scene(1, "Portrait").unwrap(); commit(&mut collection, &mut history, draft, None); - assert_eq!(collection.items(&tall).unwrap().len(), 1); - assert!(collection.items(&over).unwrap().is_empty()); + let portrait = collection.scene("Portrait").unwrap(); + assert!(portrait.published && portrait.size == size); + assert_eq!(collection.recorded(), Some("Portrait")); - let mut draft = collection.draft("Remove red there"); - draft.remove_item(&tall, 0).unwrap(); + let mut draft = collection.draft("Record Scene"); + draft.set_recorded(Some("Scene")).unwrap(); commit(&mut collection, &mut history, draft, None); - assert!(collection.scenes[0].canvases.is_empty(), "an empty layout is dropped"); + assert_eq!(collection.recorded(), Some("Scene"), "one scene at most"); + assert!(collection.draft("Record none").set_recorded(Some("None")).is_err()); - history.seek(&mut collection, 1).unwrap(); - assert_eq!(collection, added); - let mut draft = collection.draft("Remove the canvas"); - draft.remove_extra("tall").unwrap(); - commit(&mut collection, &mut history, draft, None); - assert!(collection.canvases.is_empty()); - assert!(collection.scenes[0].canvases.is_empty()); - assert_eq!(collection.items(&tall), None); - history.undo(&mut collection).unwrap(); - assert_eq!(collection, added); - } - - /// Emptying a scene's layout on a further canvas changes only that layer, - /// so a scene snapshot taken before still undoes: rename the scene, empty - /// its layout, then undo both. A refused item edit leaves no empty layout - /// behind in the draft. - #[test] - fn emptying_a_layout_keeps_earlier_scene_edits_undoable() { - let mut collection = collection(); - let fill = Rect::fill(&collection.canvas); - collection.canvases.insert("tall".into(), Extra::new(1080, 1920)); - collection.scenes[0] - .canvases - .insert("tall".into(), vec![Item::new("red", fill)]); - collection.validate().unwrap(); - let start = collection.clone(); - let mut history = History::new(); - - let mut draft = collection.draft("Rename"); - draft.rename_scene(0, "Renamed").unwrap(); - commit(&mut collection, &mut history, draft, None); - let tall = Layer::scene("Renamed").on(Some("tall".into())); - let mut draft = collection.draft("Remove red there"); - draft.remove_item(&tall, 0).unwrap(); - assert!(draft.insert_item(&tall, 5, Item::new("red", fill)).is_err()); + let mut draft = collection.draft("Remove"); + draft.remove_scene(0).unwrap(); commit(&mut collection, &mut history, draft, None); - assert!(collection.scenes[0].canvases.is_empty()); + assert_eq!(collection.recorded(), None, "the program"); - history.undo(&mut collection).unwrap(); - history.undo(&mut collection).unwrap(); + history.seek(&mut collection, 0).unwrap(); assert_eq!(collection, start); } diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index 913ba9a4..fed3c0d2 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -3,9 +3,11 @@ //! The UI-free half of `moq-studio`, so the same scenes can be composed by a //! windowed app, a headless renderer, or a test: //! -//! - [`Collection`] is the scene graph: a main [`Canvas`] and any [`Extra`] -//! ones, named [`Source`]s, and [`Scene`]s placing them on each canvas, -//! under each canvas's top layer that stays on across scene switches. It saves to a versioned TOML file that holds no credentials. +//! - [`Collection`] is the scene graph: the program's [`Canvas`], named +//! [`Source`]s, and [`Scene`]s placing them, each at its own [`Size`] or +//! the program's, under a top layer that stays on across scene switches. A +//! scene can draw another, or the program, and can be published or recorded +//! on its own. It saves to a versioned TOML file that holds no credentials. //! - [`Edit`] is every change to a collection, built with a [`Draft`]. Each //! one undoes itself, so a [`History`] undoes and redoes them and saves //! beside the scene file. A [`Clip`] carries items or filters between @@ -16,12 +18,12 @@ //! rendered by a browser engine (the [`browser`] module). Pictures go to the //! compositor and sound goes to a `moq_audio::bus::Bus`, routed into its //! program mix. -//! - [`Compositor`] draws the scene in preview and the scene on program into -//! textures on the caller's `wgpu` device, one pair per canvas from one -//! upload of each source's picture, cutting or running a [`Transition`] -//! between them on every canvas together, and plays each item's -//! [`Keyframe`]s from when its scene went on. A scene source shows another -//! scene, drawn first into a texture of its own, or the program itself. A +//! - [`Compositor`] draws the scene in preview, the scene on program, and each +//! scene an output reads on its own into textures on the caller's `wgpu` +//! device, from one upload of each source's picture, cutting or running a +//! [`Transition`] on program, and plays each item's [`Keyframe`]s from when +//! its scene went on. A scene source shows another scene, drawn first into a +//! texture of its own, or the program itself. A //! source whose [`Background`] is removed is cut out first, by a //! segmentation model on the CPU. `moq_video::render::export` hands the //! program to an encoder, on the GPU where the platform allows, and a @@ -52,10 +54,10 @@ pub mod source; pub use clip::Clip; pub use collection::{ - Audio, Background, Blend, Canvas, Collection, Crop, Direction, Ease, Extra, Filter, Item, Keyframe, Layer, - MaskMode, Mixes, Nested, Rect, Rgba, Scene, Source, Transition, Video, + Audio, Background, Blend, Canvas, Collection, Crop, Direction, Ease, Filter, Item, Keyframe, Layer, MaskMode, + Mixes, Nested, Rect, Rgba, Scene, Size, Source, Transition, Video, }; -pub use compositor::{Compositor, FORMAT}; +pub use compositor::{Compositor, FORMAT, Output}; pub use edit::{Change, Draft, Edit, History}; pub use scale::Scaler; diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 79ab35da..1c458185 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -1,14 +1,15 @@ //! Everything behind the window: the GPU, the sources, the program clock that //! draws the canvas and mixes the audio, and the broadcast it encodes into: -//! the program, at the canvas size and at half of it ([`crate::ladder`]). The -//! relays serve that broadcast, and recordings read it. Each local source's own +//! the program, at the canvas size and at half of it ([`crate::ladder`]), and +//! each published scene's beneath it, at its own size and half. The relays +//! serve those broadcasts, and recordings read the program or a scene. Each local source's own //! track stays in a broadcast of the studio's own, which recordings read too, //! as they read each broadcast source from the relays. //! //! Threads: -//! - **program** draws preview and program once per canvas frame and hands the -//! program to each rendition's encoder its gate lets encode, scaled to the -//! rendition and stamped with the program clock. +//! - **program** draws preview, program, and each published scene once per +//! canvas frame and hands each to each rendition's encoder its gate lets +//! encode, scaled to the rendition and stamped with the program clock. //! - **audio** advances the audio bus, and with it every mix, on the same //! clock. //! - **audio-encode** encodes the mix on program into the program audio, and @@ -188,11 +189,12 @@ impl State { } /// Make `collection`, which the edits `made` led to from the current one, - /// the one drawn, once the compositor takes its canvas. The scenes in - /// preview and on program follow the renames and removals `made` holds, + /// the one drawn, once the compositor fits its scenes' sizes. The scenes + /// in preview and on program follow the renames and removals `made` holds, /// each edit read against the collection it was made to, and any - /// `collection` lacks is dropped. Edits that remove a scene on air, or - /// leave one drawing the program, are refused, changing nothing. + /// `collection` lacks is dropped. Edits that remove a scene on air, leave + /// one drawing the program, or make one another size than the program, + /// are refused, changing nothing. fn adopt(&mut self, collection: Collection, made: &[Edit]) -> anyhow::Result<()> { let mut before = self.collection.clone(); let mut renames = Vec::with_capacity(made.len()); @@ -201,7 +203,7 @@ impl State { before.apply(edit)?; } keep_on_air(self.compositor.on_air(), &renames, &collection)?; - self.compositor.set_canvases(&collection)?; + self.compositor.fit(&collection)?; for renames in &renames { self.compositor.rename(renames); } @@ -231,9 +233,9 @@ impl State { Some(scene) } - /// The sources whose pictures any canvas's program draws, through clones: - /// those [`placed`](Self::placed) there, and the clip a stinger under way - /// plays with each mask its source's filters read. + /// The sources whose pictures program or a published scene draws, through + /// clones: those [`placed`](Self::placed) there, and the clip a stinger + /// under way plays with each mask its source's filters read. fn shown(&self) -> BTreeSet { let collection = &self.collection; let stinger = self.compositor.stinger().into_iter().flat_map(|clip| { @@ -245,19 +247,16 @@ impl State { shown } - /// The sources whose pictures the scenes on air place on any canvas's - /// program, through clones: each visible item's of the scenes on air, - /// including one leaving in a transition, and of the top layers over them, - /// and of the scenes they draw, and each mask's it reads. + /// The sources whose pictures the scenes live place, through clones: each + /// visible item's of the scenes on air, including one leaving in a + /// transition, of the top layer over them, and of each published or + /// recorded scene drawn, and of the scenes they draw, and each mask's it + /// reads. fn placed(&self) -> BTreeSet { let collection = &self.collection; - let scenes: Vec> = self.compositor.on_air().map(Some).chain([None]).collect(); - let canvases = std::iter::once(None).chain(collection.canvases.keys().map(Some)); - let layers = canvases.flat_map(|canvas| { - scenes.iter().map(move |scene| moq_scene::Layer { - canvas: canvas.cloned(), - scene: scene.map(str::to_string), - }) + let scenes = self.compositor.live().map(Some).chain([None]); + let layers = scenes.map(|scene| moq_scene::Layer { + scene: scene.map(str::to_string), }); let items = collection.drawn(layers); items @@ -329,9 +328,10 @@ impl autosave::Document for State { } /// Refuse edits, each read as its [`Collection::renames`], that remove a -/// scene `on_air` program draws, so a slip cannot put black on air, or that +/// scene `on_air` program draws, so a slip cannot put black on air, that /// leave one drawing the program in `collection`, which it would draw into -/// itself. A reload that leaves one out still drops it, having nothing to +/// itself, or that make one another size than the program, which it would +/// not fill. A reload that leaves one out still drops it, having nothing to /// show instead. fn keep_on_air<'a>( on_air: impl IntoIterator, @@ -350,6 +350,11 @@ fn keep_on_air<'a>( !collection.draws_program(scene), "`{scene}` is on air, so it cannot draw the program, which it would draw into itself" ); + let size = collection.canvas(Some(scene)).map(|canvas| canvas.size()); + anyhow::ensure!( + size.is_none_or(|size| size == collection.canvas.size()), + "`{scene}` is on air, so it stays the program's size; cut to another scene first" + ); } Ok(()) } @@ -357,17 +362,21 @@ fn keep_on_air<'a>( /// How the output is doing, for the stream panel. #[derive(Default)] pub struct Stats { - /// Canvas frames drawn. + /// Program frames drawn. pub rendered: AtomicU64, - /// The main canvas's program video. + /// The program video. pub video: Arc, - /// Each further canvas's program video, by the canvas's name, while it is - /// drawn. - canvases: Mutex>>, - /// The video encoders open: every canvas's program and the sources'. + /// The recording's own video, of the program or the scene recordings + /// hold. + pub recording: Arc, + /// Each published scene's video, by the scene's name, while it is + /// published. + published: Mutex>>, + /// The video encoders open: the program's, each published scene's, the + /// recording's, and the sources'. pub sessions: Sessions, - /// Each program track's share of the connection's bandwidth, by name: a - /// further canvas's prefixed with its own. + /// Each published track's share of the connection's bandwidth, by name: a + /// published scene's prefixed with its own. grants: Mutex>, /// Why the audio stopped, if it did. audio_error: Mutex>, @@ -376,11 +385,12 @@ pub struct Stats { recording_audio_error: Mutex>, } -/// How one canvas's program video is doing. +/// How the program's video, a published scene's, or the recording's is +/// doing. #[derive(Default)] pub struct VideoStats { - /// What each of its video encoders costs: the canvas size, the half size, - /// and, for the main canvas, the recording's own canvas size. + /// What each of its video encoders costs: a published video's full size + /// and half size, or, third, the recording's. pub renditions: [Load; 3], /// Whether the encoder reads the program on the GPU, rather than each frame /// being read back to the CPU. @@ -484,12 +494,12 @@ impl VideoStats { } impl Stats { - /// Each further canvas's program video, by the canvas's name. - pub fn canvases(&self) -> BTreeMap> { - self.canvases.lock().unwrap().clone() + /// Each published scene's video, by the scene's name. + pub fn published(&self) -> BTreeMap> { + self.published.lock().unwrap().clone() } - /// Each program track's share of the connection's bandwidth now, by name: + /// Each published track's share of the connection's bandwidth now, by name: /// `None` while the connection has no estimate. pub fn grants(&self) -> BTreeMap> { let grants = self.grants.lock().unwrap(); @@ -499,7 +509,7 @@ impl Stats { .collect() } - /// Why the main canvas's output stopped, if it did. + /// Why the program's output stopped, if it did. pub fn error(&self) -> Option { let video = self.video.error(); let audio = self.audio_error.lock().unwrap().clone(); @@ -652,8 +662,9 @@ impl Program { /// Encode the local sources' tracks on program, whether or not anything /// reads them, so a replay of one has its past: the pictures `shown` in a - /// scene on air on any canvas, and the sound the program mix plays. A media file heard - /// and not shown encodes only its sound. + /// scene on air or a published or recorded scene drawn, and the sound the + /// program mix plays. A media file heard and not shown encodes only its + /// sound. fn on_program(&self, state: &State, shown: BTreeSet) { let program = &state.collection.mixes.program; let heard = state @@ -818,12 +829,12 @@ impl Engine { browser: browser(&gpu)?, }); let mut compositor = Compositor::new(&gpu.device, &gpu.queue, collection.canvas)?; - compositor.set_canvases(&collection)?; + compositor.fit(&collection)?; // The first scene that can go on program starts there, and in preview. let first = collection .scenes .iter() - .find(|scene| !collection.draws_program(&scene.name)); + .find(|scene| collection.takes(&scene.name).is_ok()); compositor.set_preview(first.map(|scene| scene.name.clone())); compositor.cut(&collection, &sources)?; @@ -858,6 +869,7 @@ impl Engine { let Publish { renditions, ladder, + recording, extras, audio, recorded_audio, @@ -878,22 +890,21 @@ impl Engine { *engine.feeds.lock().unwrap() = Some(feeds); } let main = Line::new(renditions, canvas, ladder, true); - // Opened before the program clock starts the sources, as the main - // canvas's encoder is, so each holds its encode session first. + // Opened before the program clock starts the sources, as the + // program's encoder is, so each holds its encode session first. let mut lines = Lines::new(); - let canvases: Vec<(String, Canvas)> = { + let (published, recorded): (Vec<(String, Canvas)>, Canvas) = { let state = engine.state.lock().unwrap(); - let collection = state.collection(); - let canvases = collection.canvases.keys(); - canvases - .filter_map(|name| Some((name.clone(), collection.canvas(Some(name))?))) - .collect() + (published(state.collection()), recorded(state.collection()).1) }; - for (name, canvas) in canvases { + // Probed now too, so a scene a recording cannot encode says so before + // anyone records it. It opens its session only once recorded. + let recording = Line::open(recording, recorded).await; + for (name, canvas) in published { let video = Arc::new(VideoStats::default()); engine .stats - .canvases + .published .lock() .unwrap() .insert(name.clone(), video.clone()); @@ -906,8 +917,14 @@ impl Engine { // publication alive beside a retry's. let starting = Starting(Some(engine.clone())); let program = engine.clone(); + let encoders = Encoders { + main, + recording, + extras, + lines, + }; let thread = clocked("program".into(), async move || { - program.run_program(start, epoch, main, extras, lines).await + program.run_program(start, epoch, encoders).await })?; engine.threads.lock().unwrap().push(thread); let mix = engine.clone(); @@ -958,10 +975,10 @@ impl Engine { Ok(engine) } - /// Stop the program clock, then wait for each rendition's encoder, on - /// every canvas, to publish the copies on their way to it and what it + /// Stop the program clock, then wait for each rendition's encoder, of + /// every picture, to publish the copies on their way to it and what it /// still holds, and to end its video track, so quitting loses none of the - /// program's last pictures; a further canvas's catalog then finishes too. + /// program's last pictures; a published scene's catalog then finishes too. /// The program audio ends its track too, so a recording reading /// the program reaches the end of each. Then stop every source, which /// finishes its own track on its thread, and the browser sources' helper, @@ -987,7 +1004,7 @@ impl Engine { } } - /// Once [`stop`](Self::stop) has ended the tracks of every canvas, close + /// Once [`stop`](Self::stop) has ended the tracks of every picture, close /// each relay session once the relay has their tails, so a viewer or a /// recording on the relay receives them rather than a cut. Meanwhile wait /// for each source's thread to end, having finished its own track. Bounded @@ -1103,8 +1120,9 @@ impl Engine { /// What only the studio knows about its stream at `now`. fn local(&self, now: Instant) -> health::Local { let stats = &self.stats; - // A stalled picture matters where any canvas's program draws it: a scene on air, - // including one leaving in a transition, or the top layer over them. + // A stalled picture matters where a live scene draws it: a scene on air, + // including one leaving in a transition, the top layer over them, or a + // published or recorded scene drawn. // Stalled sound matters only if the mix on program plays it, not one // only listened to. A source that stopped matters either way. let (stalled, stopped) = { @@ -1147,38 +1165,52 @@ impl Engine { .collect(); (stalled, stopped) }; - let video = &stats.video; - let canvas = &video.renditions[0]; + let video = |video: &VideoStats, failed: Option| { + let full = &video.renditions[Role::Canvas as usize]; + health::Video { + dropped: full.dropped.load(Ordering::Relaxed), + encoded: full.encoded.load(Ordering::Relaxed), + encoding: video.encoding.load(Ordering::Relaxed), + watched: video.watched.load(Ordering::Relaxed), + encoder: video.encoder(), + failed, + } + }; + // The program's output stops with the audio too. + let program = (None, video(&stats.video, stats.error())); + let published = stats.published().into_iter(); + let published = published.map(|(name, stats)| (Some(name), video(&stats, stats.error()))); health::Local { rendered: stats.rendered.load(Ordering::Relaxed), - dropped: canvas.dropped.load(Ordering::Relaxed), - encoded: canvas.encoded.load(Ordering::Relaxed), - encoding: video.encoding.load(Ordering::Relaxed), + broadcasts: std::iter::once(program).chain(published).collect(), grants: stats.grants(), - watched: video.watched.load(Ordering::Relaxed), - encoder: video.encoder(), - failed: stats.error(), stalled, stopped, } } - /// Draw every canvas once per tick of the program clock, and hand each - /// canvas's program to its renditions' encoders, stamped `epoch` past - /// `start`, while their gates let them encode: the canvas size every frame + /// Draw the program and each published scene once per tick of the program + /// clock, and hand each to its renditions' encoders, stamped `epoch` past + /// `start`, while their gates let them encode: the full size every frame /// while live and not on demand, otherwise only while anyone watches it, - /// the half size only while anyone watches it, and the main canvas's - /// recording copy only while recording. + /// and the half size only while anyone watches it. The program, or the + /// scene recordings hold, goes to the recording's own encoder too, which + /// encodes only while recording. /// - /// `main` encodes the main canvas, and `lines` the further canvases the - /// studio started with. Each further canvas added since gets a [`Line`] of - /// its own from `extras` once drawn, in a broadcast of its own, and each - /// ends once the collection drops its canvas. Drawing outlives the encoders: a failed - /// output stops copying its program, and the canvases keep moving. + /// A published or recorded scene is drawn only while one of its encoders + /// takes the frame, so one nobody reads costs no drawing. See [`Encoders`] + /// for what encodes each. Drawing outlives the encoders: a failed output + /// stops copying its picture, and the scenes keep moving. /// /// Runs until the studio [stops](Self::stop), and returns once every /// rendition's encode thread has finished its track. - async fn run_program(&self, start: Instant, epoch: Duration, mut main: Line, extras: Extras, mut lines: Lines) { + async fn run_program(&self, start: Instant, epoch: Duration, encoders: Encoders) { + let Encoders { + mut main, + mut recording, + extras, + mut lines, + } = encoders; let mut framerate = self.state.lock().unwrap().compositor.canvas().framerate; let mut clock = program_clock(start, framerate); // The encoders of lines since ended, waited on as the studio stops so @@ -1191,6 +1223,7 @@ impl Engine { // Without a sender, each encoder publishes what it was sent and // still holds, then finishes its track. retired.push(main.end(None)); + retired.push(recording.end(None)); retired.extend( lines .into_values() @@ -1202,7 +1235,8 @@ impl Engine { } return; } - let drawn: Vec<(Option, wgpu::Texture, Canvas)> = { + retired.retain(|ended| !ended.is_finished()); + let drawn = { let mut guard = self.state.lock().unwrap(); let state = &mut *guard; if std::mem::take(&mut state.stale) { @@ -1216,6 +1250,40 @@ impl Engine { } self.pick_mixes(state); } + + // Each published scene's line starts once it is published, in a + // broadcast of its own, and ends once it no longer is. + let published = published(&state.collection); + let gone: Vec = lines + .keys() + .filter(|line| !published.iter().any(|(name, _)| name == *line)) + .cloned() + .collect(); + for name in gone { + if let Some(Some((line, broadcast))) = lines.remove(&name) { + retired.push(line.end(Some(broadcast))); + } + self.stats.published.lock().unwrap().remove(&name); + } + for (name, canvas) in &published { + lines.entry(name.clone()).or_insert_with(|| { + let video = Arc::new(VideoStats::default()); + self.stats.published.lock().unwrap().insert(name.clone(), video.clone()); + extras + .start(name, *canvas, video.clone(), &self.program) + .inspect_err(|err| video.fail(format!("{err:#}"))) + .ok() + }); + } + let (recorded, recorded_canvas) = recorded(&state.collection); + let wants = |name: &str| matches!(lines.get(name), Some(Some((line, _))) if line.wants()); + let mut outputs: BTreeMap<&str, bool> = + published.iter().map(|(name, _)| (name.as_str(), wants(name))).collect(); + if let Some(scene) = &recorded { + *outputs.entry(scene).or_default() |= recording.wants(); + } + state.compositor.set_outputs(outputs); + // Before drawing, so a media file that restarts going on program // is cleared in this frame rather than showing where it ended. let shown = state.shown(); @@ -1225,52 +1293,40 @@ impl Engine { continue; } self.program.on_program(state, shown); - state - .compositor - .outputs() - .map(|(name, output)| (name.map(str::to_string), output.program().clone(), output.canvas())) - .collect() + let compositor = &state.compositor; + let program = (compositor.program().clone(), compositor.canvas()); + let texture = |scene: &str| compositor.output(scene).and_then(moq_scene::Output::texture).cloned(); + Drawn { + published: published + .into_iter() + .map(|(name, canvas)| { + let texture = texture(&name); + (name, texture, canvas) + }) + .collect(), + recorded: match recorded { + Some(scene) => (texture(&scene), recorded_canvas), + None => (Some(program.0.clone()), program.1), + }, + program, + } }; self.stats.rendered.fetch_add(1, Ordering::Relaxed); let timestamp = moq_net::Timestamp::try_from(epoch + tick.saturating_duration_since(start)).ok(); - // Every canvas shares the main one's frame rate. - let drawing = drawn[0].2.framerate; - if drawing != framerate { - framerate = drawing; + // Every scene shares the program's frame rate. + let (program, canvas) = drawn.program; + if canvas.framerate != framerate { + framerate = canvas.framerate; clock = program_clock(tick + period(framerate), framerate); } - - retired.retain(|ended| !ended.is_finished()); - let gone: Vec = lines - .keys() - .filter(|line| !drawn.iter().any(|(name, _, _)| name.as_ref() == Some(*line))) - .cloned() - .collect(); - for name in gone { - if let Some(Some((line, broadcast))) = lines.remove(&name) { - retired.push(line.end(Some(broadcast))); + main.feed(Some(&program), canvas, timestamp); + let (texture, canvas) = drawn.recorded; + recording.feed(texture.as_ref(), canvas, timestamp); + for (name, texture, canvas) in drawn.published { + if let Some(Some((line, _))) = lines.get_mut(&name) { + line.feed(texture.as_ref(), canvas, timestamp); } - self.stats.canvases.lock().unwrap().remove(&name); - } - - for (name, program, canvas) in drawn { - let line = match &name { - None => &mut main, - Some(name) => { - let started = lines.entry(name.clone()).or_insert_with(|| { - let video = Arc::new(VideoStats::default()); - self.stats.canvases.lock().unwrap().insert(name.clone(), video.clone()); - extras - .start(name, canvas, video.clone(), &self.program) - .inspect_err(|err| video.fail(format!("{err:#}"))) - .ok() - }); - let Some((line, _)) = started else { continue }; - line - } - }; - line.feed(&program, canvas, timestamp); } } } @@ -1407,7 +1463,10 @@ struct Publish { /// The renditions of the canvas the studio started with, unless the canvas /// could not be encoded without a relay. ladder: Option, - /// What publishes each further canvas. + /// Where the recording's own video goes, of the program or the scene it + /// holds. + recording: Renditions, + /// What publishes each published scene. extras: Extras, /// The program audio's encoder. audio: moq_audio::encode::Producer, @@ -1450,7 +1509,7 @@ async fn publish( // Probed now, so a machine that cannot encode the canvas fails at startup // rather than once the program clock starts. Without a relay the studio // still composes, previews, and records audio, and a new canvas tries again. - let probe = match Probe::new(gpu, canvas, true).await { + let probe = match Probe::new(gpu, canvas, false).await { Ok(probe) => Some(probe), Err(err) if live.is_none() => { stats.video.fail(format!("{err:#}")); @@ -1471,22 +1530,14 @@ async fn publish( // both and any solicitation. let reserved = catalog.reserve(); // Viewers report back beneath the broadcast, which only a relay carries. - let viewers = match live { - Some(_) => { - let (echo, viewers) = health::echo(path)?; - let mut soliciting = catalog.clone(); - let mut section = soliciting.modify()?; - section.echo = Some(hang::catalog::Echo::new(echo)); - section.commit()?; - Some(viewers) - } - None => None, - }; - // What a recording reads the program from: its own canvas-size encoder, - // at full quality whatever the grant, and the mix it holds, in a - // broadcast no relay sees: announced on an origin of its own, which no - // relay connection carries, so a recording reads it as it would the - // published program. + if live.is_some() { + health::solicit(&mut catalog.clone(), path)?; + } + // What a recording reads the program from: its own full-size encoder, of + // the program or the scene it holds, at full quality whatever the grant, + // and the mix it holds, in a broadcast no relay sees: announced on an + // origin of its own, which no relay connection carries, so a recording + // reads it as it would the published program. let local = moq_tokio::origin::spawn(); let recording = local .create_broadcast("recording") @@ -1503,10 +1554,7 @@ async fn publish( gpu: gpu.clone(), broadcast: broadcast.clone(), timebase: catalog.timebase(), - recording: Some(Recording { - broadcast: recorder.clone(), - timebase: recorded.timebase(), - }), + record: false, bandwidth: bandwidth.clone(), // Live on a relay and not on demand; otherwise only a viewer asks for // the video. @@ -1516,6 +1564,18 @@ async fn publish( video: stats.video.clone(), grants: String::new(), }; + let recorded_video = Renditions { + gpu: gpu.clone(), + broadcast: recorder.clone(), + timebase: recorded.timebase(), + record: true, + bandwidth: moq_net::bandwidth::Allocator::unlimited(), + on_demand: true, + live: false, + stats: stats.clone(), + video: stats.recording.clone(), + grants: String::new(), + }; let ladder = match probe { Some(probe) => { let (ladder, opened) = renditions.start(probe)?; @@ -1557,27 +1617,21 @@ async fn publish( .announce(moq_net::origin::Route::default().with_instance(instance.clone(), 0)) .context("cannot announce the broadcast")?; - let health = live.zip(viewers).map(|(live, viewers)| { - let feedback = health::Feedback::default(); - let watching = health::Watching::default(); - tokio::spawn(log( - "viewer feedback", - feedback.clone().follow(live.reports.clone(), viewers), - )); - tokio::spawn(log( - "relay stats", - watching.clone().follow(live.reports.clone(), path.clone()), - )); - health::Publisher { - observer: hang::health::Observer::new(path.clone(), Some(instance.clone())), - catalog: graded, - clock: clock.clone(), - relays: live.links.clone(), - feedback, - watching, - sources: sources.clone(), + let health = match live { + Some(live) => { + let observer = hang::health::Observer::new(path.clone(), Some(instance.clone())); + Some(health::Publisher { + main: health::Broadcast::follow(observer, graded, &live.reports)?, + scenes: health::Scenes::new(live.reports.clone()), + clock: clock.clone(), + relays: live.links.clone(), + sources: sources.clone(), + }) } - }); + None => None, + }; + // Each published scene is graded beside the program, with its own viewers. + let graded = health.as_ref().map(|health| health.scenes.clone()); Ok(( Program { @@ -1602,9 +1656,11 @@ async fn publish( on_demand: renditions.on_demand, live: renditions.live, stats: stats.clone(), + graded, }, renditions, ladder, + recording: recorded_video, audio, recorded_audio, clock, @@ -1634,26 +1690,24 @@ impl Role { } } -/// Where a recording reads the program from: a broadcast the studio keeps to -/// itself, holding the recording's own canvas-size video and its own encoding -/// of the mix it holds, listed in a catalog of its own on the program's clock. -#[derive(Clone)] -struct Recording { - broadcast: moq_net::broadcast::Producer, - timebase: moq_mux::catalog::Timebase, -} - -/// Where a canvas's program video is published, a canvas at a time, each -/// rendition its own track, since a track never changes what it carries. +/// Where the program's or a published scene's video is published, a canvas +/// at a time, each rendition its own track, since a track never changes what +/// it carries; or where the recording's own video goes, of the program or the +/// scene it holds. #[derive(Clone)] struct Renditions { gpu: Gpu, + /// A published broadcast, or the recording's: one the studio keeps to + /// itself, holding the recording's video and its own encoding of the mix + /// it holds, listed in a catalog of its own on the program's clock. broadcast: moq_net::broadcast::Producer, /// Shared by every rendition, so each lands on the catalog's clock the same /// way and a switch keeps the video in step with the audio. timebase: moq_mux::catalog::Timebase, - /// Where the recording's own encoder writes: the main canvas's alone. - recording: Option, + /// Whether this is the recording's: its one rendition the full size, at + /// full quality whatever the grant, rather than the published full and + /// half sizes. + record: bool, /// The connection's bandwidth, which each published rendition follows its /// share of. bandwidth: bandwidth::Allocator, @@ -1662,14 +1716,25 @@ struct Renditions { /// Whether the program goes out on a relay, where viewers can watch it. live: bool, stats: Arc, - /// How this canvas's video is doing. + /// How this video is doing. video: Arc, /// What each track's grant is listed under in [`Stats::grants`], before - /// its name: empty for the main canvas, and a further one's name. + /// its name: empty for the program, and a published scene's name and a + /// slash. grants: String, } impl Renditions { + /// Probe for `canvas`'s renditions and [`start`](Self::start) them, waiting + /// for the full size's encoder to open, as at startup, so it holds its + /// encode session before any source opens one. + async fn open(&self, canvas: Canvas) -> anyhow::Result { + let probe = Probe::new(&self.gpu, canvas, self.record).await?; + let (ladder, opened) = self.start(probe)?; + opened.await.context("the encode thread stopped")??; + Ok(ladder) + } + /// List each of `probe`'s renditions in its catalog under a new track, and /// start their encode threads. The receiver reports once the canvas size's /// encoder opens; the half size opens its own once watched, and the @@ -1705,11 +1770,7 @@ impl Renditions { config, video, } = probed; - let (broadcast, timebase) = match (role, &self.recording) { - (Role::Record, Some(recording)) => (&recording.broadcast, &recording.timebase), - (Role::Record, None) => anyhow::bail!("only the main canvas is recorded"), - _ => (&self.broadcast, &self.timebase), - }; + let (broadcast, timebase) = (&self.broadcast, &self.timebase); let reserved = timebase.reserve().with_interval(config.gop.duration(config.framerate)); let suffix = match &video.codec { hang::catalog::VideoCodec::H264(_) => ".avc3", @@ -1813,17 +1874,19 @@ struct Probed { impl Probe { /// Open a throwaway encoder of each of `canvas`'s renditions on what `gpu` - /// exports, the recording's copy only if `record`, which also settles - /// whether the encoder reads the program on the GPU. The canvas size picks - /// the encoder, which the others use too. + /// exports, the published sizes or, if `record`, the recording's alone, + /// which also settles whether the encoder reads the program on the GPU. + /// The first picks the encoder, which the other uses too. async fn new(gpu: &Gpu, canvas: Canvas, record: bool) -> anyhow::Result { let framerate = canvas.framerate(); let size = moq_video::Size::new(canvas.width, canvas.height); - let sizes = ladder::sizes(size); - let roles = [Role::Canvas, Role::Half] - .into_iter() - .zip(sizes) - .chain(record.then_some((Role::Record, size))); + let roles: Vec<(Role, moq_video::Size)> = match record { + true => vec![(Role::Record, size)], + false => [Role::Canvas, Role::Half] + .into_iter() + .zip(ladder::sizes(size)) + .collect(), + }; let mut kind = moq_video::encode::Kind::Auto; let mut rungs = Vec::new(); let mut refused = Vec::new(); @@ -1836,7 +1899,7 @@ impl Probe { .with_context(|| format!("cannot encode a {}x{} canvas", size.width, size.height)); let (exporter, video) = match probed { Ok(probed) => probed, - Err(err) if role == Role::Canvas => return Err(err), + Err(err) if role != Role::Half => return Err(err), // The program goes on without it. Err(err) => { refused.push((role, format!("{err:#}"))); @@ -1897,7 +1960,7 @@ struct Ended { threads: Vec>, /// Holds the encode threads it was waiting on. probe: Option>>, - /// A further canvas's broadcast, dropped only after the threads, so its + /// A published scene's broadcast, dropped only after the threads, so its /// catalog finishes after the video it lists. _broadcast: Option, } @@ -1921,6 +1984,24 @@ impl Ended { } impl Line { + /// `renditions` of `canvas`, opened at once, as at startup. One whose + /// encoder cannot open draws on without video, saying why. + async fn open(renditions: Renditions, canvas: Canvas) -> Self { + let ladder = renditions.open(canvas).await; + let ladder = ladder.inspect_err(|err| renditions.video.fail(format!("{err:#}"))); + Self::new(renditions, canvas, ladder.ok(), true) + } + + /// Whether any rendition would take the next frame: one encoding always, + /// or one anyone reads. A probe done opens the renditions on the next + /// feed, so their first frame is drawn then rather than one frame late. + /// Without renditions open or opening, none would. + fn wants(&self) -> bool { + let opening = self.probing.as_ref().is_some_and(|(_, probe)| probe.is_finished()); + let rungs = self.ladder.iter().flat_map(|ladder| &ladder.rungs); + opening || rungs.into_iter().any(|rung| rung.gate.wanted()) + } + /// `renditions` of `canvas`: `ladder` when already open, or `None` to probe /// for them on the first feed unless `settled`, as a canvas that failed /// to open at startup is. @@ -1939,10 +2020,11 @@ impl Line { } /// Hand `program`, drawn at `drawn`, to each rendition its gate lets - /// encode, stamped `timestamp`. A new canvas size or frame rate ends the - /// renditions and probes for others, since a track never changes what it - /// carries. - fn feed(&mut self, program: &wgpu::Texture, drawn: Canvas, timestamp: Option) { + /// encode, stamped `timestamp`; with no `program`, as for a scene not + /// drawn since none [wanted](Self::wants) it, each skips the frame. A new + /// canvas size or frame rate ends the renditions and probes for others, + /// since a track never changes what it carries. + fn feed(&mut self, program: Option<&wgpu::Texture>, drawn: Canvas, timestamp: Option) { let renditions = &self.renditions; let video = &renditions.video; if drawn != self.canvas { @@ -1973,8 +2055,7 @@ impl Line { } if self.probing.is_none() && !self.settled { let ending = std::mem::take(&mut self.ending); - let record = renditions.recording.is_some(); - match Probe::spawn(renditions.gpu.clone(), self.canvas, record, ending) { + match Probe::spawn(renditions.gpu.clone(), self.canvas, renditions.record, ending) { Ok(probe) => self.probing = Some((self.canvas, probe)), Err(err) => { self.settled = true; @@ -2017,7 +2098,7 @@ impl Line { } /// Stop handing the renditions frames, so each encode thread publishes - /// what it holds and finishes its track, then ends a further canvas's + /// what it holds and finishes its track, then ends a published scene's /// `broadcast`. fn end(mut self, broadcast: Option) -> Ended { let mut threads = std::mem::take(&mut self.ending); @@ -2037,16 +2118,57 @@ impl Line { } } -/// Each further canvas's line and broadcast, by name, or `None` once it failed -/// to start, which is not retried until the canvas is removed and added again. +/// Each published scene's line and broadcast, by name, or `None` once it +/// failed to start, which is not retried until the scene is published again. type Lines = BTreeMap>; -/// A further canvas's broadcast, whose catalog finishes and which closes +/// What one tick drew, for the encoders. +struct Drawn { + /// The program, at its canvas. + program: (wgpu::Texture, Canvas), + /// Each published scene by name, unless not drawn, at its canvas. + published: Vec<(String, Option, Canvas)>, + /// What recordings hold, unless not drawn, at its canvas. + recorded: (Option, Canvas), +} + +/// What the program clock hands each picture to. +struct Encoders { + /// The program's. + main: Line, + /// The recording's own, fed the program or the scene recordings hold, + /// which a change of size probes for again as any line does. + recording: Line, + /// What starts the line of a scene published while the studio runs. + extras: Extras, + /// The published scenes' lines, those the studio started with first. + lines: Lines, +} + +/// Each published scene in `collection`, with its canvas. +fn published(collection: &Collection) -> Vec<(String, Canvas)> { + let published = collection.scenes.iter().filter(|scene| scene.published); + published + .map(|scene| (scene.name.clone(), scene.canvas(collection.canvas))) + .collect() +} + +/// What recordings hold in `collection`: the scene it records, or `None` for +/// the program, with its canvas. +fn recorded(collection: &Collection) -> (Option, Canvas) { + let scene = collection.recorded(); + let canvas = collection.canvas(scene).unwrap_or(collection.canvas); + (scene.map(str::to_string), canvas) +} + +/// A published scene's broadcast, whose catalog finishes and which closes /// when dropped. struct Broadcast { publication: moq_net::origin::Publication, - /// Lists the canvas's video and the main broadcast's audio. + /// Lists the scene's video and the program broadcast's audio. catalog: moq_scene::publish::Catalog, + /// Its grading, while live, which stops with it. + _graded: Option, } impl Drop for Broadcast { @@ -2057,7 +2179,7 @@ impl Drop for Broadcast { } } -/// What starts each further canvas's broadcast: beneath the main one's path, +/// What starts each published scene's broadcast: beneath the program's path, /// on its clock, sharing its connection's bandwidth. struct Extras { origin: moq_net::origin::Producer, @@ -2070,13 +2192,15 @@ struct Extras { on_demand: bool, live: bool, stats: Arc, + /// Where each published scene's broadcast is graded, while live. + graded: Option, } impl Extras { - /// Publish the further canvas `name`, first drawn at `canvas`, and the - /// [`Line`] that encodes its video, reporting in `video`. Its encoders are - /// probed for on the line's first feed, off the program clock, for a canvas - /// added while the studio runs. + /// Publish the scene `name`, drawn at `canvas`, and the [`Line`] that + /// encodes its video, reporting in `video`. Its encoders are probed for on + /// the line's first feed, off the program clock, for a scene published + /// while the studio runs. fn start( &self, name: &str, @@ -2088,10 +2212,11 @@ impl Extras { Ok((Line::new(renditions, canvas, None, false), broadcast)) } - /// [`start`](Self::start), opening the canvas size's encoder before - /// returning, as the main canvas does at startup, so a canvas in the scene - /// file holds its encode session before any source opens one. A canvas - /// whose encoder cannot open draws on without video, saying why in `video`. + /// [`start`](Self::start), opening the full size's encoder before + /// returning, as the program does at startup, so a scene published in the + /// scene file holds its encode session before any source opens one. A + /// scene whose encoder cannot open draws on without video, saying why in + /// `video`. async fn open( &self, name: &str, @@ -2099,26 +2224,13 @@ impl Extras { video: Arc, program: &Program, ) -> anyhow::Result<(Line, Broadcast)> { - let (renditions, broadcast) = self.publish(name, video.clone(), program)?; - let opened = async { - let probe = Probe::new(&self.gpu, canvas, false).await?; - let (ladder, opened) = renditions.start(probe)?; - opened.await.context("the encode thread stopped")??; - anyhow::Ok(ladder) - }; - let ladder = match opened.await { - Ok(ladder) => Some(ladder), - Err(err) => { - video.fail(format!("{err:#}")); - None - } - }; - Ok((Line::new(renditions, canvas, ladder, true), broadcast)) + let (renditions, broadcast) = self.publish(name, video, program)?; + Ok((Line::open(renditions, canvas).await, broadcast)) } - /// Publish the further canvas `name` at its own path beneath the main - /// broadcast's, listing the main broadcast's audio by reference so it is - /// encoded once, and where its video goes, reporting in `video`. + /// Publish the scene `name` at its own path beneath the program's + /// broadcast, listing the program's audio by reference so it is encoded + /// once, and where its video goes, reporting in `video`. fn publish( &self, name: &str, @@ -2127,23 +2239,23 @@ impl Extras { ) -> anyhow::Result<(Renditions, Broadcast)> { let path = moq_net::Path::new(name) .and_then(|name| self.path.join(&name)) - .with_context(|| format!("cannot publish canvas `{name}`"))?; + .with_context(|| format!("cannot publish scene `{name}`"))?; if self.live { let (_, echo) = health::echo(&self.path)?; anyhow::ensure!( path != echo, - "canvas `{name}` would publish where viewers report back; remove it and add it under another name" + "scene `{name}` would publish where viewers report back; rename it to publish it" ); } let audio = self .path .relative(&path) - .with_context(|| format!("canvas `{name}` cannot refer to the main broadcast"))?; + .with_context(|| format!("scene `{name}` cannot refer to the program's broadcast"))?; let publication = self .origin .create_broadcast(&path) - .with_context(|| format!("cannot create the broadcast of canvas `{name}`"))?; + .with_context(|| format!("cannot create the broadcast of scene `{name}`"))?; let mut broadcast = publication.broadcast().clone(); let config = moq_mux::catalog::Config::default() .with_catalog(Default::default()) @@ -2158,15 +2270,27 @@ impl Extras { catalog.audio.renditions.insert(id, config); } })?; + // Its viewers report back beneath it, as the program's do. + if self.graded.is_some() { + health::solicit(&mut catalog, &path)?; + } + let instance = moq_net::Instance::random(); publication - .announce(moq_net::origin::Route::default().with_instance(moq_net::Instance::random(), 0)) - .with_context(|| format!("cannot announce the broadcast of canvas `{name}`"))?; + .announce(moq_net::origin::Route::default().with_instance(instance.clone(), 0)) + .with_context(|| format!("cannot announce the broadcast of scene `{name}`"))?; + let graded = match &self.graded { + Some(scenes) => { + let observer = hang::health::Observer::new(path.clone(), Some(instance)); + Some(scenes.follow(name, observer, catalog.clone())?) + } + None => None, + }; let renditions = Renditions { gpu: self.gpu.clone(), broadcast, timebase: catalog.timebase(), - recording: None, + record: false, bandwidth: self.bandwidth.clone(), on_demand: self.on_demand, live: self.live, @@ -2174,7 +2298,11 @@ impl Extras { video, grants: format!("{name}/"), }; - let broadcast = Broadcast { publication, catalog }; + let broadcast = Broadcast { + publication, + catalog, + _graded: graded, + }; Ok((renditions, broadcast)) } } @@ -2251,12 +2379,16 @@ struct Rung { impl Rung { /// Hand the encoder `program` at `timestamp`, or tell it why not, counting - /// in `load`. Returns what the gate decided, or `None` once the encoder - /// stopped, having reported why. - fn feed(&mut self, load: &Load, program: &wgpu::Texture, timestamp: moq_net::Timestamp) -> Option { + /// in `load`: with no `program`, it hears nothing new. Returns what the gate decided, + /// or `None` once the encoder stopped, having reported why. + fn feed(&mut self, load: &Load, program: Option<&wgpu::Texture>, timestamp: moq_net::Timestamp) -> Option { let step = self.gate.step(); - let message = match step { - Step::Encode(bitrate) => { + let message = match (step, program) { + // Wanted only since the picture went undrawn, as when the + // renditions opened this frame: the next frame is drawn, and + // nothing is skipped meanwhile. + (Step::Encode(_), None) => None, + (Step::Encode(bitrate), Some(program)) => { self.skipped = None; let texture = match &self.scaler { Some(scaler) => scaler.scale(program), @@ -2279,7 +2411,7 @@ impl Rung { } } } - skip => (self.skipped.replace(skip) != Some(skip)).then_some(Video::Skip(skip)), + (skip, _) => (self.skipped.replace(skip) != Some(skip)).then_some(Video::Skip(skip)), }; match message { Some(message) => self.frames.send(message).ok().map(|()| step), @@ -2337,13 +2469,6 @@ fn connect( .unzip() } -/// Run a background reader, logging why it stopped. -async fn log(what: &'static str, task: impl Future>) { - if let Err(err) = task.await { - tracing::warn!(%err, "stopped reading {what}"); - } -} - /// What the program clock hands a rendition's encoder. enum Video { /// A copy of the program on its way to the encoder, to encode at this @@ -2708,9 +2833,9 @@ mod tests { } } - /// Program frames the recording's own encoder encoded. + /// Frames the recording's own encoder encoded. fn recorded(engine: &Engine) -> u64 { - engine.stats.video.renditions[Role::Record as usize] + engine.stats.recording.renditions[Role::Record as usize] .encoded .load(Ordering::Relaxed) } @@ -2773,9 +2898,9 @@ mod tests { /// Without a relay, a recording encodes the program on its own and saves it /// as a regular MP4 holding its video and audio. The color source publishes - /// a still, which has no file of its own. A new canvas goes on in another - /// file, holding the new rendition's video. A recording holding a mix - /// other than the one on program records its audio. + /// a still, which has no file of its own. Recording a scene of another + /// size goes on in another file, holding its video. A recording holding a + /// mix other than the one on program records its audio. #[tokio::test(flavor = "multi_thread")] #[ignore = "needs a Vulkan device and an H.264 encoder; run by just rs render-sw"] async fn the_program_records_without_a_relay() { @@ -2801,13 +2926,20 @@ mod tests { assert_eq!(engine.stats.sessions.open(), 1); { let mut state = engine.state.lock().unwrap(); - let mut draft = state.collection().draft("Resize the canvas"); - let canvas = moq_scene::Canvas { - width: 160, - height: 90, - framerate: 15, + let mut draft = state.collection().draft("Record a portrait scene"); + let fill = moq_scene::Rect { + x: 0.0, + y: 0.0, + width: 90.0, + height: 160.0, }; - draft.set_canvas(canvas).unwrap(); + let tall = moq_scene::Scene { + size: Some(moq_scene::Size { width: 90, height: 160 }), + items: vec![moq_scene::Item::new("red", fill)], + ..moq_scene::Scene::new("Tall") + }; + draft.insert_scene(1, tall).unwrap(); + draft.set_recorded(Some("Tall")).unwrap(); state.commit(draft, None).unwrap(); } encoded(&engine, 90 + 45).await; @@ -3038,7 +3170,7 @@ mod tests { .await } - /// The broadcast of the further canvas `tall` as `relay` routes it, its + /// The broadcast of the published scene `tall` as `relay` routes it, its /// catalogs, and the first that lists its video. async fn routed_tall( relay: &moq_net::origin::Consumer, @@ -3047,7 +3179,7 @@ mod tests { moq_mux::catalog::Consumer<()>, hang::Catalog, ) { - // Until the canvas's broadcast is announced, the main one's route covers + // Until the scene's broadcast is announced, the program's route covers // its path and cannot serve it; the follow moves on once it is. let mut following = relay.follow("studio/tall", None).unwrap(); let (broadcast, mut catalogs) = kio::clock::timeout(TIMEOUT, async { @@ -3075,7 +3207,7 @@ mod tests { (broadcast, catalogs, catalog) } - /// Read a further canvas's `video` and `catalogs` to a clean end. + /// Read a published scene's `video` and `catalogs` to a clean end. async fn ended(mut catalogs: moq_mux::catalog::Consumer<()>, mut video: moq_net::track::Subscriber) { kio::clock::timeout(TIMEOUT, async { while let Some(mut group) = video.recv_group().await.expect("the video finishes") { @@ -3087,14 +3219,15 @@ mod tests { .expect("the canvas's tracks end"); } - /// A further canvas reaches the relay in a broadcast of its own beneath the - /// main one, at its own size, with the main broadcast's audio listed by + /// A published scene reaches the relay in a broadcast of its own beneath + /// the program's, at its own size, with the program's audio listed by /// reference rather than encoded again, and holds one more encode session. - /// Removing it, or quitting, ends its video and then its catalog cleanly, - /// then its broadcast; removing it gives its session back. + /// Health grades it beside the program, with its own viewers. Unpublishing + /// it, or quitting, ends its video and then its catalog cleanly, then its + /// broadcast; unpublishing gives its session back. #[tokio::test(flavor = "multi_thread")] #[ignore = "needs a Vulkan device and an H.264 encoder; run by just rs render-sw"] - async fn the_program_of_a_further_canvas_has_its_own_broadcast() { + async fn a_published_scene_has_its_own_broadcast() { let _ = moq_tokio::crypto::install_default(); let fixture = moq_relay::test_relay().await.expect("test relay"); let ready = fixture.relay.ready(); @@ -3114,12 +3247,22 @@ mod tests { url: fixture.url.clone(), }]; let mut loaded = red_canvas(); - let tall = moq_scene::Extra::new(360, 640); - let fill = moq_scene::Rect::fill(&tall.canvas(30)); - loaded.collection.canvases.insert("tall".into(), tall); - loaded.collection.scenes[0] - .canvases - .insert("tall".into(), vec![moq_scene::Item::new("red", fill)]); + let size = moq_scene::Size { + width: 360, + height: 640, + }; + let fill = moq_scene::Rect { + x: 0.0, + y: 0.0, + width: 360.0, + height: 640.0, + }; + loaded.collection.scenes.push(moq_scene::Scene { + size: Some(size), + published: true, + items: vec![moq_scene::Item::new("red", fill)], + ..moq_scene::Scene::new("tall") + }); let engine = Engine::start(launch(gpu, relays, dir.path()), autosave, loaded) .await .expect("the studio starts"); @@ -3144,23 +3287,56 @@ mod tests { let mut pictures = subscribe(&broadcast, &video, &config).await; red(&mut pictures, moq_video::Size::new(360, 640)) .await - .expect("the canvas's own layout arrives"); - assert_eq!(engine.stats.sessions.open(), 2, "one canvas size each"); - assert!(engine.stats.canvases()["tall"].error().is_none()); + .expect("the scene arrives"); + assert_eq!(engine.stats.sessions.open(), 2, "one full size each"); + assert!(engine.stats.published()["tall"].error().is_none()); + assert!(catalog.echo.is_some(), "its viewers are asked for feedback"); + // Health grades it beside the program, from what it writes. + let graded = |engine: &Engine| { + let report = engine.health.lock().unwrap(); + let tall = report + .broadcasts + .iter() + .find(|broadcast| broadcast.name.as_deref() == Some("tall")); + tall.map(|tall| { + tall.tracks + .iter() + .any(|track| track.sent.is_some_and(|sent| sent.as_bps() > 0)) + }) + }; + kio::clock::timeout(TIMEOUT, async { + while graded(&engine) != Some(true) { + kio::clock::sleep(health::INTERVAL).await; + } + }) + .await + .expect("health grades the published scene"); let track = broadcast.track(&video).unwrap().subscribe(None).await.unwrap(); { let mut state = engine.state.lock().unwrap(); - let mut draft = state.collection().draft("Remove the canvas"); - draft.remove_extra("tall").unwrap(); + let scene = state.collection().scenes[1].clone(); + let mut draft = state.collection().draft("Stop publishing the scene"); + let scene = moq_scene::Scene { + published: false, + ..scene + }; + draft.set_scene(1, scene).unwrap(); state.commit(draft, None).unwrap(); } ended(catalogs, track).await; kio::clock::timeout(TIMEOUT, broadcast.closed()) .await - .expect("the canvas's broadcast ends"); - assert!(engine.stats.canvases().is_empty()); - assert_eq!(engine.stats.sessions.open(), 1, "the canvas's session given back"); + .expect("the scene's broadcast ends"); + assert!(engine.stats.published().is_empty()); + assert_eq!(engine.stats.sessions.open(), 1, "the scene's session given back"); + kio::clock::timeout(TIMEOUT, async { + while graded(&engine).is_some() { + kio::clock::sleep(health::INTERVAL).await; + } + }) + .await + .expect("health forgets the scene no longer published"); // Back again, quitting ends it as cleanly. engine.state.lock().unwrap().undo().unwrap(); @@ -3175,7 +3351,7 @@ mod tests { ended(catalogs, track).await; kio::clock::timeout(TIMEOUT, broadcast.closed()) .await - .expect("the canvas's broadcast ends"); + .expect("the scene's broadcast ends"); assert_eq!(engine.stats.sessions.open(), 0); } @@ -3446,9 +3622,10 @@ mod tests { loop { { let report = engine.health.lock().unwrap(); - let sending = report.tracks.len() == 2 - && report - .tracks + let tracks = report.broadcasts.first().map_or(&[][..], |program| &program.tracks); + let sending = report.sources.is_empty() + && tracks.len() == 2 + && tracks .iter() .all(|track| track.sent.is_some_and(|sent| sent.as_bps() > 0)); let connected = matches!(&report.relays[..], [relay] if relay.state == health::State::Connected); diff --git a/rs/moq-studio/src/health.rs b/rs/moq-studio/src/health.rs index 9fd11ca3..2307a5d9 100644 --- a/rs/moq-studio/src/health.rs +++ b/rs/moq-studio/src/health.rs @@ -3,12 +3,13 @@ //! //! Every [`INTERVAL`] the studio grades its stream with `hang::health`, the //! grader the rest of the stack shares, from three points of view: -//! - its own: what each track of the broadcast wrote, each relay connection's -//! state, and the serving one's round trip, estimated capacity, and loss; -//! - its viewers': the feedback each publishes under the broadcast's `echo` -//! path, which the catalog solicits (`moq export --echo` answers it); -//! - the relay's: how many sessions watch the broadcast, read from the relay's -//! stats broadcasts when it shares them with this connection. +//! - its own: what each track of every broadcast wrote, the program's and +//! each published scene's, each relay connection's state, and the serving +//! one's round trip, estimated capacity, and loss, which they all share; +//! - its viewers': the feedback each publishes under the broadcast's +//! `echo` path, which the catalog solicits (`moq export --echo` answers it); +//! - the relay's: how many sessions watch each broadcast, read from the +//! relay's stats broadcasts when it shares them with this connection. //! //! It adds what only the studio knows (program frames the encoder had no room //! for, sources that stopped delivering) and names the cause of each problem. @@ -55,15 +56,28 @@ pub struct Report { /// Each relay the broadcast goes to, the preferred first; empty when the /// studio publishes nothing. pub relays: Vec, + /// The connection every broadcast shares. pub uplink: Uplink, - pub viewers: Viewers, - /// Each track of the broadcast in catalog order, then each source's own - /// local track by source name. - pub tracks: Vec, + /// Each broadcast, the program's first; empty when the studio + /// publishes nothing. + pub broadcasts: Vec, + /// Each source's own local track, by source name. + pub sources: Vec, /// What is wrong, worst first, each naming its cause. pub problems: Vec, } +/// One broadcast: who watches it, and how each of its tracks reaches +/// them. +#[derive(Debug, Default)] +pub struct Published { + /// The published scene's name, or `None` for the program's. + pub name: Option, + pub viewers: Viewers, + /// Each track it lists, in catalog order. + pub tracks: Vec, +} + /// One relay connection. #[derive(Debug, Clone, PartialEq)] pub struct Relay { @@ -95,7 +109,7 @@ pub struct Uplink { pub rtt: Option, /// What the congestion controller estimates the connection can carry. pub capacity: Option, - /// The media the program's tracks wrote over the last interval. Source + /// The media every broadcast's tracks wrote over the last interval. Source /// tracks are left out: they stay in the studio, and never reach a relay. pub media: Option, /// Transport bytes lost per media byte written over the last interval. @@ -197,43 +211,204 @@ pub struct Problem { pub struct Local { /// Canvas frames drawn. pub rendered: u64, + /// Each broadcast's video: the program's as `None`, then each + /// published scene's by name. + pub broadcasts: BTreeMap, Video>, + /// Each program track's share of the connection's bandwidth, by name: a + /// published scene's prefixed with its name and a slash. + pub grants: BTreeMap>, + /// Sources the program shows or plays that stopped delivering, and what. + pub stalled: Vec<(String, Stall)>, + /// Sources the program shows or plays that ended or failed. + pub stopped: Vec<(String, Status)>, +} + +impl Local { + /// Whether anyone subscribes to any broadcast's video. + fn watched(&self) -> bool { + self.broadcasts.values().any(|video| video.watched) + } +} + +/// What only the studio knows about one broadcast's video. +#[derive(Debug, Default, Clone)] +pub struct Video { /// Program frames the encoder had no room for. pub dropped: u64, /// Program frames encoded. pub encoded: u64, - /// Whether the program's canvas size should be encoding: always, or while - /// watched when on demand, and while its grant funds it. + /// Whether the full size should be encoding: always, or while watched + /// when on demand, and while its grant funds it. pub encoding: bool, - /// Each program track's share of the connection's bandwidth, by name. - pub grants: BTreeMap>, - /// Whether anyone subscribes to the program video. + /// Whether anyone subscribes to the video. pub watched: bool, /// The encoder backend, once one opened. pub encoder: Option, /// Why the output stopped, if it did. pub failed: Option, - /// Sources the program shows or plays that stopped delivering, and what. - pub stalled: Vec<(String, Stall)>, - /// Sources the program shows or plays that ended or failed. - pub stopped: Vec<(String, Status)>, } /// What the studio publishes, for grading its delivery. pub struct Publisher { - /// The broadcast's path and the instance it was announced with. - pub observer: Observer, - pub catalog: moq_scene::publish::Catalog, - /// The catalog's clock, which maps a track's timestamps to the wall time + /// The program's broadcast. + pub main: Broadcast, + /// Each published scene's, while it is published. + pub scenes: Scenes, + /// The catalogs' clock, which maps a track's timestamps to the wall time /// the program drew them. pub clock: moq_mux::Clock, - /// Each relay the broadcast goes to, the preferred first. + /// Each relay the broadcasts go to, the preferred first. pub relays: Vec, - pub feedback: Feedback, - pub watching: Watching, /// Where the sources keep their own tracks, for what those wrote. pub sources: moq_scene::publish::Publish, } +/// One broadcast, graded: what it lists, and what its viewers and +/// the relays say of it. +#[derive(Clone)] +pub struct Broadcast { + /// Its path and the instance it was announced with. + pub observer: Observer, + pub catalog: moq_scene::publish::Catalog, + pub feedback: Feedback, + pub watching: Watching, + /// Read for as long as any clone is held. + followers: Arc, +} + +impl Broadcast { + /// Grade the broadcast `observer` names, which `catalog` lists, reading + /// its viewers' feedback beneath it and the relays' count of who watches + /// it from `reports`, until the last clone drops. The catalog must solicit + /// that feedback, as [`solicit`] has it. + pub fn follow( + observer: Observer, + catalog: moq_scene::publish::Catalog, + reports: &moq_net::origin::Consumer, + ) -> anyhow::Result { + let (_, viewers) = echo(&observer.path)?; + let feedback = Feedback::default(); + let watching = Watching::default(); + let followers = Followers(vec![ + tokio::spawn(log( + "viewer feedback", + feedback.clone().follow(reports.clone(), viewers), + )), + tokio::spawn(log( + "relay stats", + watching.clone().follow(reports.clone(), observer.path.clone()), + )), + ]); + Ok(Self { + observer, + catalog, + feedback, + watching, + followers: Arc::new(followers), + }) + } +} + +/// Have the broadcast at `path`, which `catalog` lists, ask its viewers for +/// feedback beneath it. +pub fn solicit(catalog: &mut moq_scene::publish::Catalog, path: &moq_net::Path) -> anyhow::Result<()> { + let (echo, _) = echo(path)?; + let mut section = catalog.modify()?; + section.echo = Some(hang::catalog::Echo::new(echo)); + section.commit()?; + Ok(()) +} + +/// Tasks that stop when dropped. +struct Followers(Vec>); + +impl Drop for Followers { + fn drop(&mut self) { + for task in &self.0 { + task.abort(); + } + } +} + +/// Run a background reader, logging why it stopped. +async fn log(what: &'static str, task: impl Future>) { + if let Err(err) = task.await { + tracing::warn!(%err, "stopped reading {what}"); + } +} + +/// The published scenes' broadcasts graded, by scene name, shared with what +/// publishes them. +#[derive(Clone)] +pub struct Scenes { + graded: Arc>>, + /// Where viewers' reports and the relays' stats arrive. + reports: moq_net::origin::Consumer, + /// Where their readers run, rather than on the thread that publishes one, + /// such as the program clock's. + runtime: tokio::runtime::Handle, +} + +impl Scenes { + /// Published scenes graded from what arrives on `reports`, read on the + /// tokio runtime this is called inside. + pub fn new(reports: moq_net::origin::Consumer) -> Self { + Self { + graded: Arc::default(), + reports, + runtime: tokio::runtime::Handle::current(), + } + } + + /// Grade scene `name`'s broadcast, which `observer` names and `catalog` + /// lists, as [`Broadcast::follow`] does, until the returned handle drops. + pub fn follow( + &self, + name: &str, + observer: Observer, + catalog: moq_scene::publish::Catalog, + ) -> anyhow::Result { + let broadcast = { + let _runtime = self.runtime.enter(); + Broadcast::follow(observer, catalog, &self.reports)? + }; + let followers = broadcast.followers.clone(); + self.graded.lock().unwrap().insert(name.to_string(), broadcast); + Ok(Graded { + graded: self.graded.clone(), + name: name.to_string(), + followers, + }) + } + + fn snapshot(&self) -> Vec<(String, Broadcast)> { + let graded = self.graded.lock().unwrap(); + graded + .iter() + .map(|(name, broadcast)| (name.clone(), broadcast.clone())) + .collect() + } +} + +/// A published scene's broadcast being graded, until dropped. +pub struct Graded { + graded: Arc>>, + name: String, + /// Its readers, which tell it from a scene published again under its name + /// before it ended. + followers: Arc, +} + +impl Drop for Graded { + fn drop(&mut self) { + let mut graded = self.graded.lock().unwrap(); + let ours = graded.get(&self.name); + if ours.is_some_and(|broadcast| Arc::ptr_eq(&broadcast.followers, &self.followers)) { + graded.remove(&self.name); + } + } +} + /// The connection to one relay. #[derive(Clone)] pub struct Link { @@ -277,16 +452,23 @@ impl Sampler { .map(|previous| now.saturating_duration_since(previous.at)); let mut report = Report::default(); - let mut graded = Graded::default(); + // Each broadcast's viewers, as graded, beside `report.broadcasts`. + let mut watched = Vec::new(); let (mut current, mut sources) = (None, None); let mut serving = None; if let Some(publisher) = &self.publisher { report.relays = publisher.relays.iter().map(relay).collect(); serving = report.relays.iter().position(|relay| relay.state == State::Connected); let link = serving.map(|index| &publisher.relays[index]); + let extras = publisher.scenes.snapshot().into_iter(); + let broadcasts: Vec<(Option, Broadcast)> = std::iter::once((None, publisher.main.clone())) + .chain(extras.map(|(name, broadcast)| (Some(name), broadcast))) + .collect(); + // Every broadcast's tracks share the connection, so the uplink is graded + // on all of them. let ours = health::Report::new( - publisher.observer.clone(), - snapshot(publisher, link.map(|link| &link.monitor)), + publisher.main.observer.clone(), + snapshot(&broadcasts, link.map(|link| &link.monitor)), ); // Another relay's link counted its transport from another start. let previous_ours = previous @@ -294,42 +476,82 @@ impl Sampler { .filter(|previous| previous.serving == serving) .and_then(|previous| previous.report.as_ref()); let graded_ours = self.grade(&ours, previous_ours.zip(interval)); - report.uplink = uplink(&ours.snapshot, &graded_ours, local.watched); + report.uplink = uplink(&ours.snapshot, &graded_ours, local.watched()); report.uplink.relay = link.map(|link| link.endpoint.clone()); - graded.publisher = Some(graded_ours); // Graded apart from the program's, so their media stays out of the // uplink's; see [`Uplink::media`]. let mut snapshot = hang::stats::Snapshot::default(); snapshot.renditions = publisher.sources.stats(); - let theirs = health::Report::new(publisher.observer.clone(), snapshot); + let theirs = health::Report::new(publisher.main.observer.clone(), snapshot); let previous_theirs = previous.as_ref().and_then(|previous| previous.sources.as_ref()); - graded.sources = Some(self.grade(&theirs, previous_theirs.zip(interval))); - graded.viewers = publisher.feedback.grade(now, &self.thresholds); - report.viewers.unread = publisher.feedback.unread(); - report.viewers.watching = publisher.watching.get(); - let listed = Listed { - program: publisher.catalog.snapshot(), - sources: publisher.sources.catalog(), - grants: &local.grants, - }; - report.tracks = tracks(&listed, &publisher.clock, &graded); - report.viewers.reports = viewers(publisher, &graded.viewers); + let graded_theirs = self.grade(&theirs, previous_theirs.zip(interval)); + + for (name, broadcast) in broadcasts { + let graded = broadcast.feedback.grade(now, &self.thresholds); + let listed = Listed { + renditions: renditions(&broadcast.catalog.snapshot()).collect(), + prefix: prefix(name.as_deref()), + source: None, + sent: &graded_ours, + viewers: &graded, + grants: &local.grants, + }; + report.broadcasts.push(Published { + viewers: Viewers { + watching: broadcast.watching.get(), + reports: viewers(&graded, &publisher.clock), + unread: broadcast.feedback.unread(), + }, + tracks: tracks(&listed, &publisher.clock), + name, + }); + watched.push(graded); + } + let listed = publisher.sources.catalog(); + for (name, catalog) in &listed.ext.studio.sources { + let listed = Listed { + renditions: renditions(catalog).collect(), + prefix: String::new(), + source: Some(name), + sent: &graded_theirs, + viewers: &[], + grants: &local.grants, + }; + report.sources.extend(tracks(&listed, &publisher.clock)); + } current = Some(ours); sources = Some(theirs); } - let drops = previous.as_ref().map(|previous| Drops { - drawn: local.rendered.saturating_sub(previous.local.rendered), - dropped: local.dropped.saturating_sub(previous.local.dropped), - encoded: local.encoded.saturating_sub(previous.local.encoded), + let drops = previous + .as_ref() + .map(|previous| { + let drawn = local.rendered.saturating_sub(previous.local.rendered); + let broadcasts = local.broadcasts.iter(); + broadcasts + .filter_map(|(name, video)| { + let before = previous.local.broadcasts.get(name)?; + let drops = Drops { + drawn, + dropped: video.dropped.saturating_sub(before.dropped), + encoded: video.encoded.saturating_sub(before.encoded), + }; + Some((name.clone(), drops)) + }) + .collect() + }) + .unwrap_or_default(); + let seen = report.broadcasts.iter().zip(&watched).map(|(broadcast, viewers)| Seen { + name: broadcast.name.as_deref(), + viewers, + tracks: &broadcast.tracks, }); report.problems = problems(&Facts { relays: &report.relays, local: &local, drops, uplink: &report.uplink, - viewers: &graded.viewers, - labels: &report.tracks, + broadcasts: seen.collect(), thresholds: &self.thresholds, }); @@ -376,11 +598,27 @@ fn relay(link: &Link) -> Relay { } } -/// What the studio's tracks have written, and the transport of the link -/// `monitor` watches. -fn snapshot(publisher: &Publisher, monitor: Option<&moq_tokio::connection::Monitor>) -> hang::stats::Snapshot { +/// What a published scene's tracks are graded under, before each ID, since +/// two broadcasts can list the same one: its name and a slash, as its grants +/// are listed; nothing for the program's. +fn prefix(scene: Option<&str>) -> String { + scene.map(|name| format!("{name}/")).unwrap_or_default() +} + +/// What every broadcast's tracks have written, each graded under its +/// [`prefix`], and the transport of the link `monitor` watches. +fn snapshot( + broadcasts: &[(Option, Broadcast)], + monitor: Option<&moq_tokio::connection::Monitor>, +) -> hang::stats::Snapshot { let mut snapshot = hang::stats::Snapshot::default(); - snapshot.renditions = publisher.catalog.stats(); + for (name, broadcast) in broadcasts { + let prefix = prefix(name.as_deref()); + let renditions = broadcast.catalog.stats().into_iter(); + snapshot + .renditions + .extend(renditions.map(|(id, rendition)| (format!("{prefix}{id}"), rendition))); + } if let Some(stats) = monitor.and_then(moq_tokio::connection::Monitor::stats) { snapshot.transport.rtt = stats.rtt; snapshot.transport.rate = stats.estimated_send_rate.map(Rate::as_bps); @@ -424,85 +662,120 @@ fn uplink(snapshot: &hang::stats::Snapshot, graded: &Health, watched: bool) -> U } } -/// What the studio's catalogs list now. +/// What one catalog lists now, to grade each of its tracks. struct Listed<'a> { - /// The program's broadcast. - program: hang::Catalog, - /// The sources' local one, nesting each source's catalog. - sources: hang::Catalog, + /// Each video and audio rendition, as [`renditions`] reads them. + renditions: Vec, + /// What its tracks are graded and granted under, before each ID. + prefix: String, + /// The source whose own catalog it is, or `None` for a broadcast's. + source: Option<&'a String>, + /// What its tracks wrote, as graded. + sent: &'a Health, + /// Its viewers, as graded: a broadcast's alone. + viewers: &'a [GradedViewer], /// Each program track's share of the connection, by name. grants: &'a BTreeMap>, } -/// Each media track `listed`, the program's and then each source's, with -/// what it sent and what its viewers received. A source's stills carry no -/// media to grade, so they are left out. -fn tracks(listed: &Listed, clock: &moq_mux::Clock, graded: &Graded) -> Vec { - let program = - renditions(&listed.program).map(|(id, label, enabled)| (None, id, label, enabled, graded.publisher.as_ref())); - let sources = listed.sources.ext.studio.sources.iter().flat_map(|(name, catalog)| { - renditions(catalog).map(move |(id, label, enabled)| (Some(name), id, label, enabled, graded.sources.as_ref())) - }); - program - .chain(sources) - .map(|(source, id, label, enabled, sent)| { - let ours = sent.and_then(|health| health.renditions.get(id)); - let interval = sent.and_then(|health| health.interval); - // A source's track is in the local broadcast, whose names can repeat - // the program's, and no relay viewer or grant reaches it. - let program = source.is_none(); - let mut track = Track { - source: source.cloned(), - id: id.clone(), - label, - granted: listed.grants.get(id).copied().flatten().filter(|_| program), - enabled, - sent: ours.and_then(|rendition| rendition.bitrate).map(Rate::from_bps), - framerate: ours - .and_then(|rendition| rendition.frames) - .zip(interval) - .map(|(frames, interval)| frames as f64 / interval.as_secs_f64()), - viewers: 0, - received: None, - verdict: Verdict::Unknown, - latency: None, - }; - for viewer in graded.viewers.iter().filter(|_| program) { - let Some(health) = &viewer.health else { continue }; - let Some(rendition) = health.renditions.get(id) else { - continue; +/// Each media track `listed`, with what it sent and what its viewers +/// received. A source's stills carry no media to grade, so they are left out. +fn tracks(listed: &Listed, clock: &moq_mux::Clock) -> Vec { + listed + .renditions + .iter() + .map( + |Rendition { + id, + label, + enabled, + main, + }| { + // One listed by reference to the program's broadcast, as a published + // scene lists the audio, is graded and granted there. + let graded = match main { + Some(track) => track.clone(), + None => format!("{}{id}", listed.prefix), }; - track.viewers += 1; - track.verdict = track.verdict.max(rendition.verdict); - if let Some(bitrate) = rendition.bitrate.map(Rate::from_bps) { - track.received = Some(track.received.map_or(bitrate, |received| received.min(bitrate))); + let ours = listed.sent.renditions.get(&graded); + let interval = listed.sent.interval; + // A source's track is in the local broadcast, whose names can repeat + // the program's, and no relay viewer or grant reaches it. + let program = listed.source.is_none(); + let mut track = Track { + source: listed.source.cloned(), + id: id.clone(), + label: label.clone(), + granted: listed.grants.get(&graded).copied().flatten().filter(|_| program), + enabled: *enabled, + sent: ours.and_then(|rendition| rendition.bitrate).map(Rate::from_bps), + framerate: ours + .and_then(|rendition| rendition.frames) + .zip(interval) + .map(|(frames, interval)| frames as f64 / interval.as_secs_f64()), + viewers: 0, + received: None, + verdict: Verdict::Unknown, + latency: None, + }; + for viewer in listed.viewers { + let Some(health) = &viewer.health else { continue }; + let Some(rendition) = health.renditions.get(id) else { + continue; + }; + track.viewers += 1; + track.verdict = track.verdict.max(rendition.verdict); + if let Some(bitrate) = rendition.bitrate.map(Rate::from_bps) { + track.received = Some(track.received.map_or(bitrate, |received| received.min(bitrate))); + } + let latency = viewer.latency(id, clock); + track.latency = track.latency.max(latency); } - let latency = viewer.latency(id, clock); - track.latency = track.latency.max(latency); - } - track - }) + track + }, + ) .collect() } +/// One video or audio rendition a catalog lists. +struct Rendition { + id: String, + /// What it carries, such as `video H.264 1920x1080`. + label: String, + enabled: bool, + /// Its track in the main broadcast, when listed there by reference: the + /// only broadcast the studio's catalogs refer to. + main: Option, +} + /// Each video and audio rendition `catalog` lists, by ID, with what it /// carries and whether it is enabled. -fn renditions(catalog: &hang::Catalog) -> impl Iterator { - let video = catalog.video.renditions.iter().map(|(id, config)| { +fn renditions(catalog: &hang::Catalog) -> impl Iterator { + let main = |id: &String, broadcast: bool, track: &Option| { + broadcast.then(|| track.clone().unwrap_or_else(|| id.clone())) + }; + let video = catalog.video.renditions.iter().map(move |(id, config)| { let size = match (config.coded_width, config.coded_height) { (Some(width), Some(height)) => format!(" {width}x{height}"), _ => String::new(), }; - (id, format!("video {}{size}", config.codec), config.enabled) + Rendition { + id: id.clone(), + label: format!("video {}{size}", config.codec), + enabled: config.enabled, + main: main(id, config.broadcast.is_some(), &config.track), + } }); - let audio = catalog.audio.renditions.iter().map(|(id, config)| { - let label = format!("audio {} {} ch", config.codec, config.channel_count); - (id, label, config.enabled) + let audio = catalog.audio.renditions.iter().map(move |(id, config)| Rendition { + id: id.clone(), + label: format!("audio {} {} ch", config.codec, config.channel_count), + enabled: config.enabled, + main: main(id, config.broadcast.is_some(), &config.track), }); video.chain(audio) } -fn viewers(publisher: &Publisher, graded: &[GradedViewer]) -> Vec { +fn viewers(graded: &[GradedViewer], clock: &moq_mux::Clock) -> Vec { graded .iter() .map(|viewer| { @@ -515,7 +788,7 @@ fn viewers(publisher: &Publisher, graded: &[GradedViewer]) -> Vec { snapshot .renditions .keys() - .filter_map(|id| viewer.latency(id, &publisher.clock)) + .filter_map(|id| viewer.latency(id, clock)) .max() }), playout: current.and_then(|snapshot| { @@ -531,16 +804,6 @@ fn viewers(publisher: &Publisher, graded: &[GradedViewer]) -> Vec { .collect() } -/// The grades of one interval. -#[derive(Default)] -struct Graded { - /// The program's tracks and the uplink. - publisher: Option, - /// The sources' own tracks. - sources: Option, - viewers: Vec, -} - /// One viewer's latest report and its grade. struct GradedViewer { name: String, @@ -573,14 +836,32 @@ struct Drops { struct Facts<'a> { relays: &'a [Relay], local: &'a Local, - drops: Option, + /// Each broadcast's frames over the last interval, once there was + /// one. + drops: BTreeMap, Drops>, uplink: &'a Uplink, - viewers: &'a [GradedViewer], - /// The tracks, for naming one a viewer reports on. - labels: &'a [Track], + /// What each broadcast's viewers report, the program's first. + broadcasts: Vec>, thresholds: &'a Thresholds, } +/// One broadcast's viewers, as graded. +struct Seen<'a> { + /// The published scene's name, or `None` for the program's. + name: Option<&'a str>, + viewers: &'a [GradedViewer], + /// Its tracks, for naming one a viewer reports on. + tracks: &'a [Track], +} + +/// `cause` said of the broadcast `scene` names: a published scene's names it. +fn on(scene: Option<&str>, cause: String) -> String { + let Some(name) = scene else { return cause }; + let mut chars = cause.chars(); + let first = chars.next().map(|first| first.to_lowercase().to_string()); + format!("On scene `{name}`, {}{}", first.unwrap_or_default(), chars.as_str()) +} + /// Name each problem the facts show, worst first. fn problems(facts: &Facts<'_>) -> Vec { let mut problems = Vec::new(); @@ -614,8 +895,13 @@ fn problems(facts: &Facts<'_>) -> Vec { (State::Connected, _) => {} } } - if let Some(failed) = &local.failed { - add(Verdict::Unhealthy, format!("The output stopped: {failed}.")); + for (name, video) in &local.broadcasts { + if let Some(failed) = &video.failed { + add( + Verdict::Unhealthy, + on(name.as_deref(), format!("The output stopped: {failed}.")), + ); + } } for (name, stall) in &local.stalled { @@ -644,24 +930,33 @@ fn problems(facts: &Facts<'_>) -> Vec { } } - if let Some(drops) = facts.drops - && local.encoding - && local.failed.is_none() - { - let encoder = local.encoder.as_deref().unwrap_or("the encoder"); + for (name, drops) in &facts.drops { + let Some(video) = local.broadcasts.get(name) else { + continue; + }; + if !video.encoding || video.failed.is_some() { + continue; + } + let encoder = video.encoder.as_deref().unwrap_or("the encoder"); let share = ratio(drops.dropped, drops.drawn); let limits = &facts.thresholds.publisher.skipped; - if drops.drawn > 0 && drops.encoded == 0 && drops.dropped == 0 && local.encoder.is_some() { + if drops.drawn > 0 && drops.encoded == 0 && drops.dropped == 0 && video.encoder.is_some() { add( Verdict::Unhealthy, - format!("The encoder is stuck: {encoder} produced nothing in the last second."), + on( + name.as_deref(), + format!("The encoder is stuck: {encoder} produced nothing in the last second."), + ), ); } else if let Some(share) = share { add( grade_above(share, limits.degraded, limits.unhealthy), - format!( - "The encoder is behind: {encoder} had no room for {} of {} program frames in the last second.", - drops.dropped, drops.drawn + on( + name.as_deref(), + format!( + "The encoder is behind: {encoder} had no room for {} of {} program frames in the last second.", + drops.dropped, drops.drawn + ), ), ); } @@ -701,20 +996,22 @@ fn problems(facts: &Facts<'_>) -> Vec { ); } - viewer_problems(facts, &mut add); + for seen in &facts.broadcasts { + viewer_problems(uplink, seen, &mut |verdict, cause| add(verdict, on(seen.name, cause))); + } problems.sort_by_key(|problem| std::cmp::Reverse(problem.verdict)); problems } -/// What viewers' reports show, one problem per kind of trouble across them. -fn viewer_problems(facts: &Facts<'_>, add: &mut impl FnMut(Verdict, String)) { - let reporting = facts.viewers.iter().filter(|viewer| viewer.health.is_some()).count(); +/// What one broadcast's viewers report, one problem per kind of trouble across +/// them, beside the `uplink` they share. +fn viewer_problems(uplink: &Uplink, seen: &Seen<'_>, add: &mut impl FnMut(Verdict, String)) { + let reporting = seen.viewers.iter().filter(|viewer| viewer.health.is_some()).count(); // A congested or lossy uplink starves every viewer, and one measured // clear of both leaves the blame with the viewers' own links. With either // unmeasured, as headroom is under a controller without an estimate, the // cause goes unnamed. - let uplink = facts.uplink; let (headroom, loss) = (uplink.grade(Input::Headroom), uplink.grade(Input::Loss)); let cause = if headroom >= Verdict::Degraded || loss >= Verdict::Degraded { ": the uplink cannot keep up" @@ -727,7 +1024,7 @@ fn viewer_problems(facts: &Facts<'_>, add: &mut impl FnMut(Verdict, String)) { // Each input's worst grade, how many viewers it troubles, and the track // the worst grade is on. let mut worst: Vec<(Input, Verdict, usize, Option<&str>)> = Vec::new(); - for viewer in facts.viewers { + for viewer in seen.viewers { let Some(health) = &viewer.health else { continue }; let transport = health.transport.evidence.iter().map(|evidence| (evidence, None)); let renditions = health.renditions.iter().flat_map(|(id, rendition)| { @@ -761,7 +1058,7 @@ fn viewer_problems(facts: &Facts<'_>, add: &mut impl FnMut(Verdict, String)) { } let track = |id: Option<&str>| { - id.and_then(|id| facts.labels.iter().find(|track| track.id == id)) + id.and_then(|id| seen.tracks.iter().find(|track| track.id == id)) .map_or_else(|| "the stream".to_string(), Track::name) }; for (input, verdict, count, id) in worst { @@ -1029,19 +1326,33 @@ fn watching(frame: &moq_stats::PrefixFrame) -> u64 { mod tests { use super::*; + /// What the program's broadcast alone shows: `viewers` reporting on `tracks`. fn facts<'a>(local: &'a Local, uplink: &'a Uplink, viewers: &'a [GradedViewer], tracks: &'a [Track]) -> Facts<'a> { static THRESHOLDS: std::sync::LazyLock = std::sync::LazyLock::new(Thresholds::default); Facts { relays: &[], local, - drops: None, + drops: BTreeMap::new(), uplink, - viewers, - labels: tracks, + broadcasts: vec![Seen { + name: None, + viewers, + tracks, + }], thresholds: &THRESHOLDS, } } + /// A studio whose program video is `main`, and the published scene + /// `tall`'s, if any, `tall`. + fn local(main: Video, tall: Option