diff --git a/README.md b/README.md index 12cdab3..327e668 100644 --- a/README.md +++ b/README.md @@ -106,6 +106,7 @@ Every panel is a tab you can drag, split, or stack. | Filters | Remove the selected source's background, crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | | Keyframes | Animate the selected item: add, time, ease, and remove its keyframes, and set how far each moves, resizes, crops, and filters it (see [Keyframes](#keyframes)). **Play** plays the preview scene's keyframes again. | | Audio | Pick the audio scene on program and the one to listen to and on which output, set each sound's gain, mute, and delay, and send each sound and audio scene into audio scenes, with a meter on every one. | +| Graph | See the sources, scenes, and audio scenes as nodes, wired left to right to what draws or holds them, and wire them there (see [Scene graph](#scene-graph)). | | Output | Start and stop recording, pick the picture and the audio scene it records, set the canvas size and frame rate and each scene's size, see the broadcast path, whether the program is encoding, the encoder sessions open, and the encoder work of the program, the recording, and each published scene, and turn the virtual camera of the program or of each published scene on. | | Health | See how the stream reaches viewers, relay by relay and, for the program and each published scene, track by track, and the cause of each problem, and which recordings a crash left unfinished were repaired at launch. | | Replay | Replay the last moments of the program or a source in a replay source, slowed down if you like, or save them as a clip. | @@ -118,6 +119,25 @@ Items panel shows. The **top layer** is drawn over program, and over the preview of a scene of the program's size, and stays on through a cut or transition: a downstream keyer for a logo or a lower third. +### Scene graph + +The Graph panel, beside Preview, shows the collection as nodes: each source +with a picture or sound, each scene and the top layer, and each audio scene. +A wire runs from each source to every scene or layer placing it, and from +each sound and audio scene to every audio scene holding it; one into a scene +or clone source shows what it draws, and follows from that source. A wire +whose items are all hidden, or whose sound is muted, is dimmed. + +Drag from a node's right edge onto a scene's, the top layer's, or an audio +scene's left edge to wire them: a picture is placed filling the scene, and a +sound or audio scene is sent into the audio scene. Click a wire, then +**Remove**, to take every item of that source out of the scene, or stop the +send. Each sound's node has its strip's mute, gain, and delay and a waveform +of its last few seconds, each audio scene's node **On program**, **Listen**, +and the waveform of its mix, and each scene's node **Publish** and the audio +scene it carries. Clicking a scene previews it, and clicking a source picks +it for Filters. Every edit lands in History, as from the other panels. + ### Published scenes Every scene has a size, the program's unless the Output panel's **Scene diff --git a/quest/m0/audio-pan.md b/quest/m0/audio-pan.md index 1d64847..3283185 100644 --- a/quest/m0/audio-pan.md +++ b/quest/m0/audio-pan.md @@ -2,10 +2,10 @@ ## Goal -Each sound's strip in the Audio panel has a pan beside its fader, saved as -`pan` in `[audio.]` and applied wherever the sound plays, in every -audio scene holding it, as gain, mute, and delay are. `README.md` documents -it. +Each sound's strip in the Audio panel, and its node in the Graph panel, has +a pan beside its fader, saved as `pan` in `[audio.]` and applied +wherever the sound plays, in every audio scene holding it, as gain, mute, and +delay are. `README.md` documents it. ## Plan @@ -23,6 +23,8 @@ the rest of audio scenes: stereo refuses a pan other than center. - Changing pan never drops or repeats audio, ramped across a block as gain is. +- Both panels draw a sound's mute, gain, and delay with `strip_controls` in + `rs/moq-studio/src/ui.rs`; a pan added there shows in both. ## Required diff --git a/quest/m1/README.md b/quest/m1/README.md index 715dd06..5b4fefc 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -24,5 +24,4 @@ than opening an m2 for them. ## Required - [Viewer-side composition](/quest/m1/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers and by a server -- [Scene graph view](/quest/m1/scene-graph.md) - a node view of picture and audio scenes, their routing, volume, pan, and waveforms - [Rune UI spike](/quest/m1/rune-spike.md) - the monitors prototyped on Rune, with a verdict on moving the UI off egui diff --git a/quest/m1/scene-graph.md b/quest/m1/scene-graph.md deleted file mode 100644 index a2244f9..0000000 --- a/quest/m1/scene-graph.md +++ /dev/null @@ -1,15 +0,0 @@ -# [M] Scene graph view - -## Goal - -A Graph panel shows the scene graph: picture and audio scenes as nodes, the -sources and scenes each one draws or sends to as edges, with each sound's -volume, pan, and a live waveform on its node. Edges and node settings can be -edited there, as in the Scenes and Audio panels. - -## Plan - -Decided 2026-11-05 with the maintainer (/quest-plan): the live controls stay -a mixer, as pro switchers keep them, and this view is for building and -inspecting routing, as node tools (TouchDesigner, Fusion) do. It reads and -edits the same data, so it adds no second model. diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 06f8792..8520ae6 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -28,6 +28,11 @@ use crate::replay::{self, Of}; use crate::snap::{self, Grab, Snap}; use crate::{mix, record}; +mod graph; +mod levels; + +use levels::{Levels, meter}; + /// The panels, each a tab the user can move, split, or stack. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] enum Tab { @@ -42,6 +47,7 @@ enum Tab { Filters, Keyframes, Audio, + Graph, Output, Health, Replay, @@ -62,6 +68,7 @@ impl Tab { Self::Filters => "Filters", Self::Keyframes => "Keyframes", Self::Audio => "Audio", + Self::Graph => "Graph", Self::Output => "Output", Self::Health => "Health", Self::Replay => "Replay", @@ -89,7 +96,8 @@ impl Studio { ) }; - let mut dock = DockState::new(vec![Tab::Preview]); + // The graph wants room, and is for building rather than for the show. + let mut dock = DockState::new(vec![Tab::Preview, Tab::Graph]); let tree = dock.main_surface_mut(); let [top, bottom] = tree.split_below(NodeIndex::root(), 0.62, vec![Tab::Scenes, Tab::History]); tree.split_right(top, 0.5, vec![Tab::Program]); @@ -152,6 +160,8 @@ impl Studio { reloads: 0, effects: false, spots: Spots::default(), + levels: Levels::default(), + graph: graph::View::default(), }, } } @@ -225,6 +235,7 @@ impl Studio { let framerate = state.compositor.canvas().framerate; ui.ctx().request_repaint_after(Duration::from_secs(1) / framerate); self.panels.follow_reloads(&state); + self.panels.levels.take(&state.sources, state.collection()); // A key bound to a scene follows it through renames, undos, and redos, // under this lock, so a key fired meanwhile names the scene it means. let renames = state.take_renames(); @@ -407,6 +418,9 @@ struct Panels { effects: bool, /// Where this frame's panels drew what effects go around. spots: Spots, + /// Each sound's and audio scene's recent levels, for the meters. + levels: Levels, + graph: graph::View, } /// Where the panels drew what effects go around, this frame. @@ -451,6 +465,7 @@ impl TabViewer for Panels { Tab::Filters => self.filters(ui), Tab::Keyframes => self.keyframes(ui), Tab::Audio => self.audio(ui), + Tab::Graph => self.graph(ui), Tab::Output => self.output(ui), Tab::Health => self.health(ui), Tab::Replay => self.replay(ui), @@ -1411,30 +1426,8 @@ impl Panels { if let Some(size) = scene.size { ui.weak(format!("{}x{}", size.width, size.height)); } - let mut published = scene.published; - if ui - .checkbox(&mut published, "Publish") - .on_hover_text("Live as its own broadcast beneath the program's, at its own size") - .changed() - { - action = Some(SceneAction::Publish(index, published)); - } - if scene.published { - let on_program = format!("on program ({program_audio})"); - let mut audio = scene.audio.clone(); - egui::ComboBox::from_id_salt(("scene audio", index)) - .selected_text(audio.as_deref().unwrap_or(&on_program)) - .show_ui(ui, |ui| { - ui.selectable_value(&mut audio, None, &on_program); - for scene in &audio_scenes { - ui.selectable_value(&mut audio, Some(scene.name.clone()), &scene.name); - } - }) - .response - .on_hover_text("The audio scene its broadcast carries"); - if audio != scene.audio { - action = Some(SceneAction::Audio(index, audio)); - } + if let Some(settings) = scene_settings(ui, index, scene, &audio_scenes, &program_audio) { + action = Some(settings); } if ui.add_enabled(index > 0, egui::Button::new("Up").small()).clicked() { action = Some(SceneAction::Move(index, index - 1)); @@ -1541,10 +1534,17 @@ impl Panels { self.check(result); } let Some(action) = action else { return }; + let result = self.scene_action(&mut state, action); + drop(state); + self.check(result); + } + + /// Make the edit the Scenes or Graph panel picked for a scene. + fn scene_action(&mut self, state: &mut State, action: SceneAction) -> anyhow::Result<()> { let name = state.collection().scenes[action.index()].name.clone(); - let result = match action { + match action { SceneAction::Rename(index, to) => { - let result = edit(&mut state, format!("Rename scene `{name}` to `{to}`"), None, |draft| { + let result = edit(state, format!("Rename scene `{name}` to `{to}`"), None, |draft| { draft.rename_scene(index, to.clone()) }); if result.is_ok() { @@ -1552,7 +1552,7 @@ impl Panels { } result } - SceneAction::Move(from, to) => edit(&mut state, format!("Reorder scene `{name}`"), None, |draft| { + SceneAction::Move(from, to) => edit(state, format!("Reorder scene `{name}`"), None, |draft| { draft.move_scene(from, to) }), SceneAction::Publish(index, published) => { @@ -1561,7 +1561,7 @@ impl Panels { true => format!("Publish scene `{name}`"), false => format!("Stop publishing scene `{name}`"), }; - edit(&mut state, label, None, |draft| { + edit(state, label, None, |draft| { draft.set_scene(index, Scene { published, ..scene }) }) } @@ -1571,12 +1571,12 @@ impl Panels { Some(audio) => format!("Have scene `{name}` carry audio scene `{audio}`"), None => format!("Have scene `{name}` carry the audio scene on program"), }; - edit(&mut state, label, None, |draft| { + edit(state, label, None, |draft| { draft.set_scene(index, Scene { audio, ..scene }) }) } SceneAction::Remove(index) => { - let result = edit(&mut state, format!("Remove scene `{name}`"), None, |draft| { + let result = edit(state, format!("Remove scene `{name}`"), None, |draft| { draft.remove_scene(index) }); if result.is_ok() { @@ -1586,9 +1586,7 @@ impl Panels { } result } - }; - drop(state); - self.check(result); + } } /// Rename, reorder, or remove one of `scenes`, the audio scenes as the @@ -2568,13 +2566,9 @@ impl Panels { for scene in &scenes { let name = &scene.name; ui.label(name); - meter(ui, state.sources.mix(name).map(|mix| mix.level()).unwrap_or_default()); + meter(ui, self.levels.mix(name)); if ui.radio(scene.program, "On program").clicked() && !scene.program { - let mut next = scenes.clone(); - for scene in &mut next { - scene.program = scene.name == *name; - } - scenes_change = Some((next, format!("Put audio scene `{name}` on program"))); + scenes_change = Some(on_program(&scenes, name)); } let heard = listening.as_ref().is_some_and(|(heard, _)| heard == name); let response = ui @@ -2601,25 +2595,8 @@ impl Panels { // A send toggled from a sound or an audio scene into an audio scene. let send = |from: Strip<'_>, to: &str, on: bool| { let mut next = scenes.clone(); - let into = next.iter_mut().find(|scene| scene.name == to).expect("listed"); - let (held, what) = match from { - Strip::Sound(name) => (&mut into.sounds, format!("`{name}`")), - Strip::Scene(name) => (&mut into.scenes, format!("audio scene `{name}`")), - }; - let name = match from { - Strip::Sound(name) | Strip::Scene(name) => name.to_string(), - }; - let label = match on { - true => { - held.insert(name); - format!("Put {what} in `{to}`") - } - false => { - held.remove(&name); - format!("Take {what} out of `{to}`") - } - }; - (next, label) + from.send(&mut next, to, on); + (next, from.label(to, on)) }; let mut change = None; egui::Grid::new("audio").striped(true).show(ui, |ui| { @@ -2633,29 +2610,13 @@ impl Panels { } ui.end_row(); for (name, feed) in state.sources.iter() { - let Some(strip) = feed.strip() else { continue }; - let mut audio = state.collection().audio.get(name).cloned().unwrap_or_default(); - let mut db = decibels(audio.gain); - ui.label(name); - if ui.checkbox(&mut audio.muted, "Mute").changed() { - let verb = if audio.muted { "Mute" } else { "Unmute" }; - change = Some((name.to_string(), audio.clone(), format!("{verb} `{name}`"), None)); - } - let response = ui.add(egui::Slider::new(&mut db, MUTE_DB..=12.0).suffix(" dB")); - let gesture = self.gestures.of(&response); - if response.changed() { - let mut audio = audio.clone(); - audio.gain = if db <= MUTE_DB { 0.0 } else { 10f32.powf(db / 20.0) }; - change = Some((name.to_string(), audio, format!("Set `{name}` gain"), gesture)); + if feed.strip().is_none() { + continue; } - let max = moq_audio::bus::MAX_DELAY.as_millis() as u64; - let mut millis = audio.delay.as_millis() as u64; - let response = ui.add(egui::DragValue::new(&mut millis).range(0..=max).suffix(" ms")); - let gesture = self.gestures.of(&response); - if response.changed() { - let mut audio = audio.clone(); - audio.delay = Duration::from_millis(millis); - change = Some((name.to_string(), audio, format!("Set `{name}` delay"), gesture)); + let audio = state.collection().audio.get(name).cloned().unwrap_or_default(); + ui.label(name); + if let Some((audio, label, gesture)) = strip_controls(ui, &mut self.gestures, name, &audio) { + change = Some((name.to_string(), audio, label, gesture)); } for scene in &scenes { let mut sent = scene.sounds.contains(name); @@ -2663,7 +2624,7 @@ impl Panels { scenes_change = Some(send(Strip::Sound(name), &scene.name, sent)); } } - meter(ui, strip.level()); + meter(ui, self.levels.sound(name)); ui.end_row(); } // Each audio scene sends into others too, never into itself, even @@ -2684,7 +2645,7 @@ impl Panels { scenes_change = Some(send(Strip::Scene(name), &scene.name, sent)); } } - meter(ui, state.sources.mix(name).map(|mix| mix.level()).unwrap_or_default()); + meter(ui, self.levels.mix(name)); ui.end_row(); } }); @@ -3634,6 +3595,127 @@ enum Strip<'a> { Scene(&'a str), } +impl Strip<'_> { + /// The strip as an edit's label names it. + fn what(self) -> String { + match self { + Self::Sound(name) => format!("`{name}`"), + Self::Scene(name) => format!("audio scene `{name}`"), + } + } + + /// What sending it into the audio scene `to`, or stopping, is called in + /// the history. + fn label(self, to: &str, on: bool) -> String { + let what = self.what(); + match on { + true => format!("Put {what} in `{to}`"), + false => format!("Take {what} out of `{to}`"), + } + } + + /// Send it into the audio scene `to` of `scenes`, or stop; whether that + /// changed them. + fn send(self, scenes: &mut [moq_scene::AudioScene], to: &str, on: bool) -> bool { + let Some(into) = scenes.iter_mut().find(|scene| scene.name == to) else { + return false; + }; + let (held, name) = match self { + Self::Sound(name) => (&mut into.sounds, name), + Self::Scene(name) => (&mut into.scenes, name), + }; + match on { + true => held.insert(name.to_string()), + false => held.remove(name), + } + } +} + +/// `scenes` with the one called `name` on program instead, and the edit's +/// label. +fn on_program(scenes: &[moq_scene::AudioScene], name: &str) -> (Vec, String) { + let mut next = scenes.to_vec(); + for scene in &mut next { + scene.program = scene.name == name; + } + (next, format!("Put audio scene `{name}` on program")) +} + +/// A sound's mute, gain, and delay, as its Audio panel strip and Graph node +/// show them, and the change made to `audio` with its label and gesture. +fn strip_controls( + ui: &mut Ui, + gestures: &mut Gestures, + name: &str, + audio: &moq_scene::Audio, +) -> Option<(moq_scene::Audio, String, Option)> { + let mut change = None; + let mut muted = audio.muted; + if ui.checkbox(&mut muted, "Mute").changed() { + let verb = if muted { "Mute" } else { "Unmute" }; + let audio = moq_scene::Audio { muted, ..audio.clone() }; + change = Some((audio, format!("{verb} `{name}`"), None)); + } + let mut db = decibels(audio.gain); + let response = ui.add(egui::Slider::new(&mut db, MUTE_DB..=12.0).suffix(" dB")); + let gesture = gestures.of(&response); + if response.changed() { + let gain = if db <= MUTE_DB { 0.0 } else { 10f32.powf(db / 20.0) }; + let audio = moq_scene::Audio { gain, ..audio.clone() }; + change = Some((audio, format!("Set `{name}` gain"), gesture)); + } + let max = moq_audio::bus::MAX_DELAY.as_millis() as u64; + let mut millis = audio.delay.as_millis() as u64; + let response = ui + .add(egui::DragValue::new(&mut millis).range(0..=max).suffix(" ms")) + .on_hover_text("Delay: hold the source back to line it up with a camera that lags it"); + let gesture = gestures.of(&response); + if response.changed() { + let delay = Duration::from_millis(millis); + let audio = moq_scene::Audio { delay, ..audio.clone() }; + change = Some((audio, format!("Set `{name}` delay"), gesture)); + } + change +} + +/// A scene's Publish setting, and the audio scene its broadcast carries +/// while published, as its Scenes panel row and Graph node show them. +fn scene_settings( + ui: &mut Ui, + index: usize, + scene: &Scene, + audio_scenes: &[moq_scene::AudioScene], + program_audio: &str, +) -> Option { + let mut action = None; + let mut published = scene.published; + if ui + .checkbox(&mut published, "Publish") + .on_hover_text("Live as its own broadcast beneath the program's, at its own size") + .changed() + { + action = Some(SceneAction::Publish(index, published)); + } + if scene.published { + let on_program = format!("on program ({program_audio})"); + let mut audio = scene.audio.clone(); + egui::ComboBox::from_id_salt(("scene audio", index)) + .selected_text(audio.as_deref().unwrap_or(&on_program)) + .show_ui(ui, |ui| { + ui.selectable_value(&mut audio, None, &on_program); + for scene in audio_scenes { + ui.selectable_value(&mut audio, Some(scene.name.clone()), &scene.name); + } + }) + .response + .on_hover_text("The audio scene its broadcast carries"); + if audio != scene.audio { + action = Some(SceneAction::Audio(index, audio)); + } + } + action +} + /// An edit the Items panel makes. enum ItemAction { /// Replace the item at an index, labelled, as part of a gesture. @@ -4343,24 +4425,6 @@ fn decibels(gain: f32) -> f32 { } } -/// A level meter: RMS filled, peak as a tick, on a decibel scale. -fn meter(ui: &mut Ui, level: moq_audio::Level) { - let (rect, _) = ui.allocate_exact_size(egui::vec2(160.0, 12.0), egui::Sense::hover()); - let position = |amplitude: f32| (1.0 - decibels(amplitude) / MUTE_DB).clamp(0.0, 1.0); - let painter = ui.painter_at(rect); - painter.rect_filled(rect, 2.0, ui.visuals().extreme_bg_color); - let color = if level.peak() >= 1.0 { - Color32::RED - } else { - Color32::from_rgb(80, 200, 120) - }; - let mut filled = rect; - filled.set_width(rect.width() * position(level.rms())); - painter.rect_filled(filled, 2.0, color); - let x = rect.left() + rect.width() * position(level.peak()); - painter.vline(x, rect.y_range(), egui::Stroke::new(2.0, color)); -} - /// The form that adds a source by hand. struct AddSource { name: String, diff --git a/rs/moq-studio/src/ui/graph.rs b/rs/moq-studio/src/ui/graph.rs new file mode 100644 index 0000000..6b1ea02 --- /dev/null +++ b/rs/moq-studio/src/ui/graph.rs @@ -0,0 +1,982 @@ +//! The Graph panel: sources, scenes, and audio scenes as nodes, wired to what +//! draws or holds them, left to right. It edits the collection the Scenes, +//! Items, and Audio panels do, for building and checking routing; the live +//! controls stay a mixer. + +use std::collections::HashMap; + +use egui::epaint::CubicBezierShape; +use egui::{Color32, Pos2, Stroke, Ui}; +use moq_scene::{Audio, AudioScene, Collection, Draft, Item, Layer, Rect, Source}; + +use super::levels; +use super::{ + Panels, SceneAction, Selection, Strip, describe, edit, on_program, scene_settings, status, strip_controls, +}; +use crate::engine::State; +use crate::mix; + +/// Something that draws, holds, or is drawn or held. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +enum Node { + /// A source with a picture or sound, by name. + Source(String), + /// A scene, by name. + Scene(String), + /// The top layer, over the program. + Top, + /// An audio scene, by name. + Audio(String), +} + +impl Node { + /// The layer a picture wired into it is placed in. + fn layer(&self) -> Option { + match self { + Self::Scene(name) => Some(Layer::scene(name)), + Self::Top => Some(Layer::top()), + Self::Source(_) | Self::Audio(_) => None, + } + } +} + +/// What a wire carries. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +enum Kind { + Picture, + Sound, +} + +impl Kind { + fn color(self) -> Color32 { + match self { + Self::Picture => Color32::from_rgb(100, 160, 240), + Self::Sound => Color32::from_rgb(80, 200, 120), + } + } +} + +/// From what is drawn or held to what draws or holds it. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +struct Wire { + from: Node, + to: Node, + kind: Kind, +} + +/// What a wire made by hand stands for in the collection. +enum Route<'a> { + /// A source placed in a layer, by every item showing it there. + Placed(&'a str, Layer), + /// A sound or audio scene sent into the audio scene by this name. + Sent(Strip<'a>, &'a str), +} + +impl Wire { + /// What the wire stands for, or `None` for one into a source, which + /// follows from what the source shows rather than being made by hand. + fn route(&self) -> Option> { + match (&self.from, &self.to, self.kind) { + (Node::Source(source), to, Kind::Picture) => to.layer().map(|layer| Route::Placed(source, layer)), + (Node::Source(source), Node::Audio(to), Kind::Sound) => Some(Route::Sent(Strip::Sound(source), to)), + (Node::Audio(from), Node::Audio(to), Kind::Sound) if from != to => { + Some(Route::Sent(Strip::Scene(from), to)) + } + _ => None, + } + } + + /// Whether a drag can make it in `collection`, as the Audio panel offers: + /// an audio scene never holds one that holds it. + fn offered(&self, collection: &Collection) -> bool { + match self.route() { + Some(Route::Sent(Strip::Scene(from), to)) => !collection.nests(from, to), + Some(_) => true, + None => false, + } + } + + /// What making it, or taking it out with `on` false, is called in the + /// history. + fn label(&self, on: bool) -> String { + match self.route() { + Some(Route::Placed(source, layer)) if on => format!("Add `{source}` to {layer}"), + Some(Route::Placed(source, layer)) => format!("Remove `{source}` from {layer}"), + Some(Route::Sent(strip, to)) => strip.label(to, on), + None => "Wire".into(), + } + } + + /// What it stands for, as the panel names the wire picked. + fn describe(&self) -> String { + match self.route() { + Some(Route::Placed(source, layer)) => format!("`{source}` in {layer}"), + Some(Route::Sent(strip, to)) => format!("{} in `{to}`", strip.what()), + None => format!("{:?} into {:?}", self.from, self.to), + } + } + + /// Make it, placing the source to fill the layer or sending it into the + /// audio scene, or take it out with `on` false, removing every item + /// showing the source there or no longer sending it. + fn apply(&self, draft: &mut Draft, on: bool) -> Result<(), moq_scene::Error> { + let conflict = moq_scene::Error::Conflict; + match self.route() { + None => Err(moq_scene::Error::Invalid(format!( + "{:?} cannot be wired into {:?} by hand", + self.from, self.to + ))), + Some(Route::Placed(source, layer)) => { + let collection = draft.collection(); + let items = collection + .items(&layer) + .ok_or_else(|| conflict(format!("there is no {layer}")))?; + let placed: Vec = (0..items.len()).filter(|&i| items[i].source == source).collect(); + match (on, placed.is_empty()) { + (true, true) => { + let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); + let item = Item::new(source, Rect::fill(&canvas)); + draft.insert_item(&layer, items.len(), item) + } + (true, false) => Err(conflict(format!("{layer} already places `{source}`"))), + (false, true) => Err(conflict(format!("{layer} does not place `{source}`"))), + (false, false) => { + for index in placed.into_iter().rev() { + draft.remove_item(&layer, index)?; + } + Ok(()) + } + } + } + Some(Route::Sent(strip, to)) => { + let mut scenes = draft.collection().audio_scenes.clone(); + if !strip.send(&mut scenes, to, on) { + let what = strip.what(); + return Err(conflict(match on { + true => format!("`{to}` already holds {what}"), + false => format!("`{to}` does not hold {what}"), + })); + } + draft.set_audio_scenes(scenes) + } + } + } +} + +/// A wire as the collection has it. +#[derive(Debug, PartialEq)] +struct Edge { + wire: Wire, + /// Whether what it carries plays: some item it stands for is shown, or + /// the sound is not muted. One that does not is dimmed. + shown: bool, +} + +/// The nodes in columns, each node right of every one wired into it, and the +/// wires between them. +#[derive(Debug)] +struct Graph { + columns: Vec>, + edges: Vec, +} + +impl Graph { + fn of(collection: &Collection) -> Self { + // Each node with the group it sorts in among those its column holds + // with nothing wired in: pictures above sounds. + let mut nodes = Vec::new(); + let mut edges = Vec::new(); + let into = |from, to, kind, shown| Edge { + wire: Wire { from, to, kind }, + shown, + }; + for (name, source) in &collection.sources { + // A LUT is only a filter's. + let group = match (source.has_video(), source.has_audio()) { + (false, false) => continue, + (true, false) => 0, + (true, true) => 1, + (false, true) => 2, + }; + let node = Node::Source(name.clone()); + match source { + Source::Scene { scene } => { + edges.push(into(Node::Scene(scene.clone()), node.clone(), Kind::Picture, true)) + } + Source::Clone { source } => { + edges.push(into(Node::Source(source.clone()), node.clone(), Kind::Picture, true)); + } + _ => {} + } + nodes.push((node, group)); + } + for (layer, items) in collection.layers() { + let to = match layer.scene { + Some(name) => Node::Scene(name), + None => Node::Top, + }; + let mut placed: Vec = Vec::new(); + for item in items { + let from = Node::Source(item.source.clone()); + match placed.iter_mut().find(|edge| edge.wire.from == from) { + Some(edge) => edge.shown |= item.visible, + None => placed.push(into(from, to.clone(), Kind::Picture, item.visible)), + } + } + edges.extend(placed); + nodes.push((to, 0)); + } + for scene in &collection.audio_scenes { + let to = Node::Audio(scene.name.clone()); + for name in &scene.sounds { + let muted = collection.audio.get(name).is_some_and(|audio| audio.muted); + edges.push(into(Node::Source(name.clone()), to.clone(), Kind::Sound, !muted)); + } + for name in &scene.scenes { + edges.push(into(Node::Audio(name.clone()), to.clone(), Kind::Sound, true)); + } + nodes.push((to, 2)); + } + let columns = columns(nodes, &edges); + Self { columns, edges } + } +} + +/// A place in a column: a node, or a lane a wire passes through on its way +/// to a column further on, by the wire's index in [`Graph::edges`]. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +enum Slot { + Node(Node), + Lane(usize), +} + +/// Put each node in the column after the longest chain of wires into it, +/// sources first and everything they go into after, with a lane in each +/// column a wire skips, so no wire runs under a node. Then order each column +/// by group and by where what feeds each slot sits, so wires cross less. +fn columns(nodes: Vec<(Node, u8)>, edges: &[Edge]) -> Vec> { + let mut rank: HashMap<&Node, usize> = nodes + .iter() + .map(|(node, _)| (node, usize::from(!matches!(node, Node::Source(_))))) + .collect(); + // Each pass settles one more link of the longest chain. A valid + // collection's wires never loop, and the bound keeps one that does from + // hanging the window. + for _ in 0..nodes.len() { + let mut moved = false; + for edge in edges { + let Some(&from) = rank.get(&edge.wire.from) else { + continue; + }; + if let Some(to) = rank.get_mut(&edge.wire.to) + && *to <= from + { + *to = from + 1; + moved = true; + } + } + if !moved { + break; + } + } + + let count = rank.values().max().map_or(0, |last| last + 1); + let mut columns: Vec> = vec![Vec::new(); count]; + for (node, group) in &nodes { + columns[rank[node]].push((Slot::Node(node.clone()), *group)); + } + // The columns each wire drawn goes from and to, by its index. + let mut spans: HashMap = HashMap::new(); + for (index, edge) in edges.iter().enumerate() { + let (Some(&from), Some(&to)) = (rank.get(&edge.wire.from), rank.get(&edge.wire.to)) else { + continue; + }; + spans.insert(index, (from, to)); + let group = match edge.wire.kind { + Kind::Picture => 0, + Kind::Sound => 2, + }; + for column in &mut columns[from + 1..to] { + column.push((Slot::Lane(index), group)); + } + } + + // Where each slot of the previous column sits, once it is ordered. + let mut previous: HashMap = HashMap::new(); + columns + .into_iter() + .enumerate() + .map(|(at, column)| { + // What feeds a slot through the wire at `index` sits in the + // previous column: the node it comes from, or the lane it last + // passed through. + let feeder = |index: usize| match spans[&index].0 + 1 == at { + true => Slot::Node(edges[index].wire.from.clone()), + false => Slot::Lane(index), + }; + let mut keyed: Vec<(u8, f32, Slot)> = column + .into_iter() + .map(|(slot, group)| { + let fed: Vec = match &slot { + Slot::Node(node) => (0..edges.len()) + .filter(|index| edges[*index].wire.to == *node && spans.contains_key(index)) + .filter_map(|index| previous.get(&feeder(index)).copied()) + .collect(), + Slot::Lane(index) => previous.get(&feeder(*index)).copied().into_iter().collect(), + }; + let mean = match fed.len() { + 0 => f32::INFINITY, + len => fed.iter().sum::() as f32 / len as f32, + }; + (group, mean, slot) + }) + .collect(); + // Stable, so ties keep the collection's order. + keyed.sort_by(|a, b| a.0.cmp(&b.0).then(a.1.total_cmp(&b.1))); + let column: Vec = keyed.into_iter().map(|(_, _, slot)| slot).collect(); + previous = column.iter().cloned().zip(0..).collect(); + column + }) + .collect() +} + +/// What the Graph panel keeps between frames. +#[derive(Default)] +pub(super) struct View { + /// The output a wire is being dragged from. + dragging: Option<(Node, Kind)>, + /// The wire clicked, which Remove takes out. + picked: Option, +} + +/// An edit or pick the Graph panel makes. +enum Action { + /// Make a wire, or take it out. + Wire(Wire, bool), + /// Set a sound's strip, labelled, as part of a gesture. + Sound(String, Audio, String, Option), + Scene(SceneAction), + AudioScenes(Vec, String), + /// Listen to the audio scene by this name, or stop. + Listen(Option), + Preview(String), +} + +/// What a frame's nodes are drawn from, and what they leave for the wires. +struct Pass<'a> { + state: &'a State, + graph: &'a Graph, + /// Each node's input as drawn, which a wire ends on. + inputs: HashMap, + /// Each node's outputs as drawn, by what they carry. + outputs: HashMap<(Node, Kind), Pos2>, + /// Each node's body, where a click is the node's rather than a wire's. + bodies: Vec, + /// Where each wire that skips columns enters and leaves each of their + /// lanes, left to right, by its index in [`Graph::edges`]. + lanes: HashMap>, + /// The output a wire was let go from this frame, and where. + released: Option<(Node, Kind, Pos2)>, + action: Option, +} + +/// How wide a node is. +const NODE_WIDTH: f32 = 200.0; +/// The space between columns, which wires cross. +const COLUMN_GAP: f32 = 72.0; +/// The space between nodes in a column. +const NODE_GAP: f32 = 16.0; +/// How tall a lane a wire passes through is. +const LANE: f32 = 10.0; +/// A port's radius. +const PORT: f32 = 5.0; +/// How near the pointer a port or wire counts as under it. +const REACH: f32 = 8.0; + +impl Panels { + /// The scene graph, laid out from the collection each frame. + pub(super) fn graph(&mut self, ui: &mut Ui) { + let engine = self.engine.clone(); + let mut state = engine.state.lock().unwrap(); + self.follow_reloads(&state); + let graph = Graph::of(state.collection()); + // An undo or reload can take the picked wire out. + if let Some(picked) = &self.graph.picked + && !graph.edges.iter().any(|edge| edge.wire == *picked) + { + self.graph.picked = None; + } + // A drag whose port went away ends with the button. + if !ui.input(|input| input.pointer.any_down()) { + self.graph.dragging = None; + } + + let mut pass = Pass { + state: &state, + graph: &graph, + inputs: HashMap::new(), + outputs: HashMap::new(), + bodies: Vec::new(), + lanes: HashMap::new(), + released: None, + action: None, + }; + ui.horizontal(|ui| match &self.graph.picked { + Some(wire) => { + ui.label(wire.describe()); + if ui.button("Remove").clicked() { + pass.action = Some(Action::Wire(wire.clone(), false)); + } + } + None => { + ui.weak( + "Drag from a node's right edge onto another's left edge to wire them; click a wire to pick it.", + ); + } + }); + ui.separator(); + + egui::ScrollArea::both().auto_shrink(false).show(ui, |ui| { + let wires = ui.painter().add(egui::Shape::Noop); + ui.horizontal_top(|ui| { + ui.add_space(REACH); + for column in &graph.columns { + let mut lanes = Vec::new(); + let drawn = ui.vertical(|ui| { + for slot in column { + match slot { + Slot::Node(node) => { + self.node(ui, &mut pass, node); + ui.add_space(NODE_GAP); + } + Slot::Lane(index) => { + let (lane, _) = ui.allocate_exact_size(egui::vec2(0.0, LANE), egui::Sense::hover()); + lanes.push((*index, lane.center().y)); + } + } + } + }); + // A lane runs the column's whole width. + let drawn = drawn.response.rect; + for (index, y) in lanes { + let through = (Pos2::new(drawn.left(), y), Pos2::new(drawn.right(), y)); + pass.lanes.entry(index).or_default().push(through); + } + ui.add_space(COLUMN_GAP); + } + }); + self.wires(ui, &mut pass, wires); + }); + + let Some(action) = pass.action else { return }; + let result = match action { + Action::Wire(wire, on) => { + // The items taking a picture out removes, which those after + // them move down past. + let removed = match wire.route() { + Some(Route::Placed(source, layer)) if !on => { + let items = state.collection().items(&layer).unwrap_or_default(); + let removed = (0..items.len()).filter(|&i| items[i].source == source); + Some((layer, removed.collect::>())) + } + _ => None, + }; + let result = edit(&mut state, wire.label(on), None, |draft| wire.apply(draft, on)); + if result.is_ok() + && let Some((layer, removed)) = removed + { + self.follow(&layer, |index| { + let before = removed.iter().filter(|&&at| at < index).count(); + (!removed.contains(&index)).then_some(index - before) + }); + } + result + } + Action::Sound(name, audio, label, gesture) => { + edit(&mut state, label, gesture, |draft| draft.set_audio(&name, Some(audio))) + } + Action::Scene(action) => self.scene_action(&mut state, action), + Action::AudioScenes(scenes, label) => edit(&mut state, label, None, |draft| draft.set_audio_scenes(scenes)), + Action::Listen(scene) => self.listen(&mut state, scene.as_deref()), + Action::Preview(name) => { + state.compositor.set_preview(Some(name)); + Ok(()) + } + }; + drop(state); + self.check(result); + } + + /// Draw `node` and its ports, and start or end a wire dragged from them. + fn node(&mut self, ui: &mut Ui, pass: &mut Pass<'_>, node: &Node) { + // Where the node's first row sits, and its waveform if it has one. + let mut header = 0.0; + let mut sound = None; + let frame = egui::Frame::group(ui.style()).fill(ui.visuals().window_fill); + let body = frame + .show(ui, |ui| { + ui.set_width(NODE_WIDTH); + (header, sound) = match node { + Node::Source(name) => self.source_node(ui, pass, name), + Node::Scene(name) => self.scene_node(ui, pass, name), + Node::Top => { + let label = ui.strong("Top layer"); + let label = label + .on_hover_text("Drawn over program, and over a scene of the program's size in preview"); + (label.rect.center().y, None) + } + Node::Audio(name) => self.audio_node(ui, pass, name), + }; + }) + .response + .rect; + pass.bodies.push(body); + + let wired_in = pass.graph.edges.iter().any(|edge| edge.wire.to == *node); + if node.layer().is_some() || matches!(node, Node::Audio(_)) || wired_in { + let at = Pos2::new(body.left(), header); + pass.inputs.insert(node.clone(), at); + } + let mut outputs = Vec::new(); + match node { + Node::Source(name) => { + let source = &pass.state.collection().sources[name]; + if source.has_video() { + outputs.push((Kind::Picture, header, true)); + } + if let Some(sound) = sound { + outputs.push((Kind::Sound, sound, true)); + } + } + // A scene goes into another through a scene source, which names it. + Node::Scene(_) if pass.graph.edges.iter().any(|edge| edge.wire.from == *node) => { + outputs.push((Kind::Picture, header, false)); + } + Node::Audio(_) => outputs.push((Kind::Sound, sound.unwrap_or(header), true)), + Node::Scene(_) | Node::Top => {} + } + for (kind, y, draggable) in outputs { + let at = Pos2::new(body.right(), y); + pass.outputs.insert((node.clone(), kind), at); + if !draggable { + continue; + } + let id = ui.id().with(("output", node, kind)); + let rect = egui::Rect::from_center_size(at, egui::Vec2::splat(REACH * 2.0)); + let response = ui.interact(rect, id, egui::Sense::drag()); + let response = response.on_hover_text("Drag onto a node's left edge to wire it there"); + if response.drag_started() { + self.graph.dragging = Some((node.clone(), kind)); + } + if response.drag_stopped() + && let Some(pointer) = ui.ctx().pointer_latest_pos() + { + pass.released = Some((node.clone(), kind, pointer)); + } + if response.hovered() || response.dragged() { + ui.ctx().set_cursor_icon(egui::CursorIcon::Grab); + } + } + } + + /// A source's name, kind, and status, and its sound's strip and waveform. + fn source_node(&mut self, ui: &mut Ui, pass: &mut Pass<'_>, name: &str) -> (f32, Option) { + let collection = pass.state.collection(); + let source = &collection.sources[name]; + let header = ui.horizontal(|ui| { + let selection = Selection::Source(name.to_string()); + let picked = self.selection.as_ref() == Some(&selection); + let label = ui.selectable_label(picked, egui::RichText::new(name).strong()); + if source.has_video() && label.on_hover_text("Pick it for Filters").clicked() { + self.selection = Some(selection); + } + if let Some(feed) = pass.state.sources.get(name) { + status(ui, &feed.status()); + } + }); + ui.add(egui::Label::new(egui::RichText::new(describe(source)).weak()).truncate()); + if !source.has_audio() { + return (header.response.rect.center().y, None); + } + let audio = collection.audio.get(name).cloned().unwrap_or_default(); + ui.horizontal_wrapped(|ui| { + ui.spacing_mut().slider_width = 90.0; + if let Some((audio, label, gesture)) = strip_controls(ui, &mut self.gestures, name, &audio) { + pass.action = Some(Action::Sound(name.to_string(), audio, label, gesture)); + } + }); + let waveform = levels::waveform(ui, self.levels.sound(name)); + (header.response.rect.center().y, Some(waveform.rect.center().y)) + } + + /// A scene's name, which picks it for preview, and its settings. + fn scene_node(&mut self, ui: &mut Ui, pass: &mut Pass<'_>, name: &str) -> (f32, Option) { + let state = pass.state; + let collection = state.collection(); + let Some(index) = collection.scenes.iter().position(|scene| scene.name == name) else { + return (ui.min_rect().top(), None); + }; + let scene = &collection.scenes[index]; + let header = ui.horizontal(|ui| { + let previewed = state.compositor.preview_scene() == Some(name); + let label = ui.selectable_label(previewed, egui::RichText::new(name).strong()); + if label.on_hover_text("Preview it").clicked() { + pass.action = Some(Action::Preview(name.to_string())); + } + if state.compositor.program_scene() == Some(name) { + ui.colored_label(Color32::RED, "on program"); + } + }); + if let Some(size) = scene.size { + ui.weak(format!("{}x{}", size.width, size.height)); + } + let program_audio = collection.program_audio().unwrap_or_default(); + ui.horizontal_wrapped(|ui| { + if let Some(action) = scene_settings(ui, index, scene, &collection.audio_scenes, program_audio) { + pass.action = Some(Action::Scene(action)); + } + }); + (header.response.rect.center().y, None) + } + + /// An audio scene's name, whether it is on program, Listen, and its + /// mix's waveform. + fn audio_node(&mut self, ui: &mut Ui, pass: &mut Pass<'_>, name: &str) -> (f32, Option) { + let scenes = &pass.state.collection().audio_scenes; + let Some(scene) = scenes.iter().find(|scene| scene.name == name) else { + return (ui.min_rect().top(), None); + }; + let header = ui.horizontal(|ui| { + ui.strong(name); + ui.weak("audio"); + }); + ui.horizontal(|ui| { + if ui.radio(scene.program, "On program").clicked() && !scene.program { + let (scenes, label) = on_program(scenes, name); + pass.action = Some(Action::AudioScenes(scenes, label)); + } + let heard = self.engine.listening().is_some_and(|(heard, _)| heard == name); + let response = ui + .add_enabled(mix::MONITOR, egui::Button::selectable(heard, "Listen")) + .on_hover_text("Play this audio scene to the speakers; use headphones") + .on_disabled_hover_text("Needs a build with the `capture` feature"); + if response.clicked() { + pass.action = Some(Action::Listen((!heard).then(|| name.to_string()))); + } + }); + let waveform = levels::waveform(ui, self.levels.mix(name)); + (header.response.rect.center().y, Some(waveform.rect.center().y)) + } + + /// Draw the wires under the nodes into `slot`, and the ports over them; + /// pick a wire clicked, and make one let go of on an input. + fn wires(&mut self, ui: &mut Ui, pass: &mut Pass<'_>, slot: egui::layers::ShapeIdx) { + let pointer = ui.ctx().pointer_latest_pos(); + let over_node = pointer.is_some_and(|pointer| pass.bodies.iter().any(|body| body.contains(pointer))); + let in_view = pointer.is_some_and(|pointer| ui.clip_rect().contains(pointer)); + // A wire from `from` to `to`, leaving and arriving level. + let bend = |from: Pos2, to: Pos2| { + let reach = ((to.x - from.x) / 2.0).max(REACH * 4.0); + let points = [from, from + egui::vec2(reach, 0.0), to - egui::vec2(reach, 0.0), to]; + CubicBezierShape::from_points_stroke(points, false, Color32::TRANSPARENT, Stroke::NONE).flatten(Some(0.5)) + }; + + // The wire made by hand under the pointer, if no node or drag has it. + let mut hovered: Option<&Wire> = None; + let mut nearest = REACH; + let mut paths = Vec::new(); + for (index, edge) in pass.graph.edges.iter().enumerate() { + let wire = &edge.wire; + let (Some(&from), Some(&to)) = ( + pass.outputs.get(&(wire.from.clone(), wire.kind)), + pass.inputs.get(&wire.to), + ) else { + continue; + }; + // Through the lanes of the columns it skips. + let mut points = Vec::new(); + let mut at = from; + for &(left, right) in pass.lanes.get(&index).into_iter().flatten() { + points.extend(bend(at, left)); + at = right; + } + points.extend(bend(at, to)); + if let Some(pointer) = pointer + && in_view && !over_node + && self.graph.dragging.is_none() + && wire.route().is_some() + { + let distance = distance(&points, pointer); + if distance < nearest { + nearest = distance; + hovered = Some(wire); + } + } + paths.push((edge, points)); + } + + let mut shapes = Vec::new(); + for (edge, points) in paths { + let picked = self.graph.picked.as_ref() == Some(&edge.wire); + if picked { + let halo = Stroke::new(8.0, ui.visuals().selection.bg_fill); + shapes.push(egui::Shape::line(points.clone(), halo)); + } + let mut color = edge.wire.kind.color(); + if !edge.shown { + color = color.gamma_multiply(0.35); + } + let width = if picked || hovered == Some(&edge.wire) { + 3.5 + } else { + 2.0 + }; + shapes.push(egui::Shape::line(points, Stroke::new(width, color))); + } + // The wire being dragged, to the pointer. + let dragging = self.graph.dragging.clone(); + if let Some((from, kind)) = &dragging + && let Some(&at) = pass.outputs.get(&(from.clone(), *kind)) + && let Some(pointer) = pointer + { + shapes.push(egui::Shape::line(bend(at, pointer), Stroke::new(2.0, kind.color()))); + } + ui.painter().set(slot, egui::Shape::Vec(shapes)); + + // The ports, with those a dragged wire can end on lit. + let collection = pass.state.collection(); + let ring = ui.visuals().window_stroke.color; + for ((_, kind), &at) in &pass.outputs { + ui.painter().circle(at, PORT, kind.color(), Stroke::new(1.0, ring)); + } + for (node, &at) in &pass.inputs { + let kind = match node { + Node::Audio(_) => Kind::Sound, + _ => Kind::Picture, + }; + let takes = dragging.as_ref().is_some_and(|(from, kind)| { + let wire = Wire { + from: from.clone(), + to: node.clone(), + kind: *kind, + }; + wire.offered(collection) + }); + let radius = if takes { PORT * 1.6 } else { PORT }; + let stroke = match takes { + true => Stroke::new(2.0, ui.visuals().selection.stroke.color), + false => Stroke::new(1.0, ring), + }; + ui.painter().circle(at, radius, kind.color(), stroke); + } + + if let Some((from, kind, at)) = pass.released.take() { + let target = pass.inputs.iter().find(|(node, input)| { + let wire = Wire { + from: from.clone(), + to: (*node).clone(), + kind, + }; + input.distance(at) <= REACH * 1.5 && wire.offered(collection) + }); + if let Some((to, _)) = target { + let wire = Wire { + from, + to: to.clone(), + kind, + }; + pass.action = Some(Action::Wire(wire, true)); + } + } + if in_view && !over_node && ui.input(|input| input.pointer.primary_clicked()) { + self.graph.picked = hovered.cloned(); + } + } +} + +/// How far `at` is from the line through `points`. +fn distance(points: &[Pos2], at: Pos2) -> f32 { + let segment = |a: Pos2, b: Pos2| { + let along = b - a; + let t = ((at - a).dot(along) / along.length_sq().max(f32::EPSILON)).clamp(0.0, 1.0); + (a + along * t).distance(at) + }; + points + .windows(2) + .map(|pair| segment(pair[0], pair[1])) + .fold(f32::INFINITY, f32::min) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn collection(toml: &str) -> Collection { + let head = "version = 1\n[canvas]\nwidth = 1280\nheight = 720\nframerate = 30\n"; + Collection::from_toml(&format!("{head}{toml}")).unwrap() + } + + fn source(name: &str) -> Node { + Node::Source(name.into()) + } + + fn wire(from: Node, to: Node, kind: Kind) -> Wire { + Wire { from, to, kind } + } + + /// Each node sits right of everything wired into it, a scene drawn + /// through a scene source two columns past it, and pictures above sounds. + #[test] + fn lays_out_left_to_right() { + let collection = collection( + r#" +[sources.cam] +kind = "camera" +[sources.mic] +kind = "microphone" +[sources.clip] +kind = "media" +path = "clip.mp4" +[sources.grade] +kind = "lut" +path = "grade.cube" +[sources.inset] +kind = "scene" +scene = "Inset" + +[[scenes]] +name = "Inset" +[[scenes.items]] +source = "cam" +rect = { x = 0, y = 0, width = 640, height = 360 } +visible = false + +[[scenes]] +name = "Main" +[[scenes.items]] +source = "inset" +rect = { x = 0, y = 0, width = 640, height = 360 } + +[[audio_scenes]] +name = "Program" +program = true +sounds = ["mic"] +scenes = ["Music"] + +[[audio_scenes]] +name = "Music" +sounds = ["clip"] +"#, + ); + let graph = Graph::of(&collection); + let name = |slot: &Slot| match slot { + Slot::Node(node) => format!("{node:?}"), + Slot::Lane(index) => format!( + "{:?} -> {:?}", + graph.edges[*index].wire.from, graph.edges[*index].wire.to + ), + }; + let columns: Vec> = graph + .columns + .iter() + .map(|column| column.iter().map(name).collect()) + .collect(); + // The microphone's wire passes the first audio scene's column in a lane. + assert_eq!( + columns, + [ + vec![r#"Source("cam")"#, r#"Source("clip")"#, r#"Source("mic")"#], + vec![ + r#"Scene("Inset")"#, + "Top", + r#"Audio("Music")"#, + r#"Source("mic") -> Audio("Program")"#, + ], + vec![r#"Source("inset")"#, r#"Audio("Program")"#], + vec![r#"Scene("Main")"#], + ] + ); + // A hidden item keeps its wire, dimmed; the LUT has none. + let hidden = graph + .edges + .iter() + .find(|edge| edge.wire.to == Node::Scene("Inset".into())); + assert!(!hidden.unwrap().shown); + assert!(graph.edges.iter().all(|edge| edge.wire.from != source("grade"))); + // Only wires into scenes, the top layer, and audio scenes are made by hand. + let fixed = wire(Node::Scene("Inset".into()), source("inset"), Kind::Picture); + assert!(graph.edges.iter().any(|edge| edge.wire == fixed)); + assert!(fixed.route().is_none()); + assert!( + wire(source("mic"), Node::Scene("Main".into()), Kind::Sound) + .route() + .is_none() + ); + } + + /// A wire places its source to fill the layer, and taking it out removes + /// every item showing the source there; sends toggle; neither twice. + #[test] + fn wires_edit_the_collection() { + let collection = collection( + r#" +[sources.cam] +kind = "camera" +[sources.mic] +kind = "microphone" + +[[scenes]] +name = "Main" +size = { width = 640, height = 360 } +[[scenes.items]] +source = "cam" +rect = { x = 0, y = 0, width = 10, height = 10 } +[[scenes.items]] +source = "cam" +rect = { x = 10, y = 0, width = 10, height = 10 } +"#, + ); + let run = |collection: &Collection, wire: &Wire, on: bool| -> Result { + let mut draft = collection.draft(wire.label(on)); + wire.apply(&mut draft, on)?; + let next = draft.collection().clone(); + draft.finish()?; + Ok(next) + }; + + let main = Node::Scene("Main".into()); + let placed = wire(source("cam"), main.clone(), Kind::Picture); + assert!(run(&collection, &placed, true).is_err(), "already placed"); + let removed = run(&collection, &placed, false).unwrap(); + assert!(removed.scene("Main").unwrap().items.is_empty()); + let added = run(&removed, &placed, true).unwrap(); + let items = &added.scene("Main").unwrap().items; + assert_eq!(items.len(), 1); + assert_eq!(items[0].rect, Rect::fill(&added.canvas(Some("Main")).unwrap())); + + let program = Node::Audio("Program".into()); + let sent = wire(source("mic"), program.clone(), Kind::Sound); + let heard = run(&collection, &sent, true).unwrap(); + assert!(heard.audio_scene("Program").unwrap().sounds.contains("mic")); + assert!(run(&heard, &sent, true).is_err(), "already sent"); + assert!(run(&collection, &sent, false).is_err(), "never sent"); + + // An audio scene into itself, even through another, is refused. + let mut nested = heard.clone(); + nested.audio_scenes.push(AudioScene::new("Music")); + let into = wire(program.clone(), Node::Audio("Music".into()), Kind::Sound); + let nested = run(&nested, &into, true).unwrap(); + let back = wire(Node::Audio("Music".into()), program.clone(), Kind::Sound); + assert!(!back.offered(&nested)); + assert!(run(&nested, &back, true).is_err()); + assert!(wire(program.clone(), program, Kind::Sound).route().is_none()); + } + + #[test] + fn distance_to_a_bent_line() { + let points = [Pos2::new(0.0, 0.0), Pos2::new(10.0, 0.0), Pos2::new(10.0, 10.0)]; + assert_eq!(distance(&points, Pos2::new(5.0, 3.0)), 3.0); + assert_eq!(distance(&points, Pos2::new(13.0, 5.0)), 3.0); + assert_eq!(distance(&points, Pos2::new(-4.0, -3.0)), 5.0); + } +} diff --git a/rs/moq-studio/src/ui/levels.rs b/rs/moq-studio/src/ui/levels.rs new file mode 100644 index 0000000..38687ef --- /dev/null +++ b/rs/moq-studio/src/ui/levels.rs @@ -0,0 +1,162 @@ +//! Each sound's and audio scene's level, taken once a step for every panel +//! that shows it, with the recent past for waveforms. + +use std::collections::{BTreeMap, VecDeque}; +use std::time::Duration; + +use egui::{Color32, Ui}; +use moq_audio::Level; +use moq_scene::Collection; +use moq_scene::source::Sources; + +use super::{MUTE_DB, decibels}; + +/// How often levels are taken. Taking one resets it, so the Audio and Graph +/// panels read these rather than each take their own and see half of it. +const STEP: Duration = Duration::from_millis(50); + +/// How many steps a waveform shows: four seconds. +const KEPT: usize = 80; + +/// The most steps missed at once that each hold the level taken after them: +/// a slow frame's worth. +const GAP: usize = 4; + +/// The levels of the last [`KEPT`] steps, oldest first. +pub(super) type History = VecDeque; + +/// Every playing sound's and audio scene's recent levels. +#[derive(Default)] +pub(super) struct Levels { + /// When the last step was taken, or `None` before the first. + taken: Option, + /// By source name. + sounds: BTreeMap, + /// By audio scene name. + mixes: BTreeMap, +} + +impl Levels { + /// Take the steps due since the last call. A step missed by a slow frame + /// holds the loudest level since the last one taken. + pub fn take(&mut self, sources: &Sources, collection: &Collection) { + let now = kio::clock::now(); + let Some(taken) = self.taken else { + self.taken = Some(now); + return; + }; + let steps = now.saturating_duration_since(taken).as_nanos() / STEP.as_nanos(); + if steps == 0 { + return; + } + // The rest of a step carries over, so steps keep their length + // whatever the frame rate. After a long gap, such as while the window + // was hidden, the waveforms start over rather than hold one level + // across it. + let steps = match usize::try_from(steps) { + Ok(steps) if steps <= GAP => { + self.taken = Some(taken + STEP * steps as u32); + steps + } + _ => { + self.taken = Some(now); + self.sounds.clear(); + self.mixes.clear(); + 1 + } + }; + + let mut sounds = BTreeMap::new(); + for (name, feed) in sources.iter() { + let Some(strip) = feed.strip() else { continue }; + let mut history = self.sounds.remove(name).unwrap_or_default(); + push(&mut history, strip.level(), steps); + sounds.insert(name.to_string(), history); + } + self.sounds = sounds; + + let mut mixes = BTreeMap::new(); + for scene in &collection.audio_scenes { + let Some(mix) = sources.mix(&scene.name) else { continue }; + let mut history = self.mixes.remove(&scene.name).unwrap_or_default(); + push(&mut history, mix.level(), steps); + mixes.insert(scene.name.clone(), history); + } + self.mixes = mixes; + } + + /// The source called `name`'s sound, after its gain and mute, if it plays. + pub fn sound(&self, name: &str) -> Option<&History> { + self.sounds.get(name) + } + + /// The audio scene called `name`'s mix, if it runs. + pub fn mix(&self, name: &str) -> Option<&History> { + self.mixes.get(name) + } +} + +fn push(history: &mut History, level: Level, steps: usize) { + for _ in 0..steps { + if history.len() == KEPT { + history.pop_front(); + } + history.push_back(level); + } +} + +/// Where `amplitude` falls on a meter, from 0 at [`MUTE_DB`] to 1 at full scale. +fn position(amplitude: f32) -> f32 { + (1.0 - decibels(amplitude) / MUTE_DB).clamp(0.0, 1.0) +} + +/// Green, or red once a level reaches full scale. +fn color(level: Level) -> Color32 { + if level.peak() >= 1.0 { + Color32::RED + } else { + Color32::from_rgb(80, 200, 120) + } +} + +/// A level meter of the latest step: RMS filled, peak as a tick, on a +/// decibel scale. +pub(super) fn meter(ui: &mut Ui, history: Option<&History>) { + let level = history.and_then(|history| history.back().copied()).unwrap_or_default(); + let (rect, _) = ui.allocate_exact_size(egui::vec2(160.0, 12.0), egui::Sense::hover()); + let painter = ui.painter_at(rect); + painter.rect_filled(rect, 2.0, ui.visuals().extreme_bg_color); + let color = color(level); + let mut filled = rect; + filled.set_width(rect.width() * position(level.rms())); + painter.rect_filled(filled, 2.0, color); + let x = rect.left() + rect.width() * position(level.peak()); + painter.vline(x, rect.y_range(), egui::Stroke::new(2.0, color)); +} + +/// The last few seconds, newest on the right, mirrored about the middle on a +/// decibel scale: each step's peak faint, its RMS solid. +pub(super) fn waveform(ui: &mut Ui, history: Option<&History>) -> egui::Response { + let size = egui::vec2(ui.available_width(), 28.0); + let (rect, response) = ui.allocate_exact_size(size, egui::Sense::hover()); + let painter = ui.painter_at(rect); + painter.rect_filled(rect, 2.0, ui.visuals().extreme_bg_color); + let Some(history) = history else { + return response; + }; + let width = rect.width() / KEPT as f32; + let middle = rect.center().y; + let half = rect.height() / 2.0; + for (age, level) in history.iter().rev().enumerate() { + let x = rect.right() - width * (age as f32 + 0.5); + let color = color(*level); + for (amplitude, color) in [(level.peak(), color.gamma_multiply(0.4)), (level.rms(), color)] { + let reach = half * position(amplitude); + if reach > 0.0 { + let stroke = egui::Stroke::new(width.max(1.0), color); + painter.vline(x, middle - reach..=middle + reach, stroke); + } + } + } + response +}