diff --git a/README.md b/README.md index e5a3f4a..ee9ae4f 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,7 @@ Every panel is a tab you can drag, split, or stack. | Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. | | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. | | Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | +| Keyframes | Animate the selected item: add, time, ease, and remove its keyframes, and set how far each moves, resizes, crops, and filters it (see [Keyframes](#keyframes)). **Play** plays the preview scene's keyframes again. | | Audio | Add, rename (double-click), and remove mixes, pick the one on program and the one to listen to and on which output, and set each audio source's gain, mute, delay, and the mixes it feeds, with a meter on every mix and source. | | Output | Start and stop recording, pick the mix it records, set the canvas size and frame rate, and see the broadcast path, whether the program is encoding, the encoder sessions open, and each rendition's encoder work. | | Health | See how the stream reaches viewers, relay by relay and track by track, and the cause of each problem. | @@ -122,6 +123,29 @@ the stream and a recording show the same frames of it, and a second **Transition** while one runs starts from the scene it was heading to. A transition set to 0 ms is a cut. +### Keyframes + +An item's keyframes animate it: an item can fly in, fade in, or blur into +focus, and a camera can push in. Each keyframe is a time after the item's +scene goes on and how far the item is from where it rests then: moved and +resized by canvas pixels, cropped further, and drawn through filters after its +own. Between two keyframes the item eases from one to the next, linearly or +easing in, out, or (by default) both. Filters of the same kinds, in order, +mix their settings, and others change halfway. A keyframe's filters cannot be +a LUT or a mask. + +Keyframes play once, from when the item's scene goes on program, and hold the +last keyframe; before the first, the item holds the first. A cut or transition +starts them again, even to the scene already on program, while a scene leaving +in a transition carries on with its own. The preview plays them from when the +scene went into preview; pick the scene again, or press **Play**, to watch them +again. They are drawn on the program clock, so the stream, a recording, and a +replay of the program show the same frames. Top-layer items have none, since +the top layer never goes on. + +Dragging an item in the preview grabs it where its keyframes have it and moves +where it rests, so its whole animation moves with it. + In the preview, click an item of whatever the Items panel shows to select it, drag it to move it, and drag its edges or corners to resize it. A drag snaps edges and centers to the canvas's edges and center, to the other items shown, @@ -429,6 +453,11 @@ items = [ name = "Break" items = [ { source = "camera-faded", rect = { x = 0, y = 0, width = 1920, height = 1080 } }, + # Flies in from the right, blurred, over half a second, landing where it rests. + { source = "guest", rect = { x = 1280, y = 720, width = 640, height = 360 }, keyframes = [ + { at = "0s", x = 700, filters = [{ kind = "blur", radius = 12 }] }, + { at = "500ms", ease = "out", filters = [{ kind = "blur", radius = 0 }] }, + ] }, ] [[top]] @@ -477,6 +506,12 @@ An item's `blend` mixes it with what is under it by a CSS `mix-blend-mode`: `saturation`, `color`, or `luminosity`. Removing a source removes the filters that name it. +An item's `keyframes`, in order of time, each have an `at`, how long after the +scene goes on, and optionally an `ease` into it from the one before (`linear`, +`in`, `out`, or `in_out`, the default), `x`, `y`, `width`, and `height` added +to the item's `rect`, a `crop` of what the item's crop leaves, and `filters` +applied after the item's own. + The studio writes the file a second after the last change, so a burst such as a slider drag is one write, and once more on quit. Edits that never pause for a second are written five seconds after the first, so a crash loses at most the diff --git a/quest/m0/README.md b/quest/m0/README.md index eb29ff6..0786410 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -91,7 +91,6 @@ repository. - [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux - [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker -- [Keyframes](/quest/m0/animation.md) - keyframed item properties with easing, starting when their scene goes on program - [Media file source](/quest/m0/media-file.md) - a local video or audio file plays as a source, looping or from the start on demand - [Stinger transitions](/quest/m0/stinger.md) - a clip played over the program that switches scenes at its cut point - [ML video effects](/quest/m0/effects.md) - background removal and other ML effects, baked into their source's track diff --git a/quest/m0/animation.md b/quest/m0/animation.md deleted file mode 100644 index a2c69ed..0000000 --- a/quest/m0/animation.md +++ /dev/null @@ -1,35 +0,0 @@ -# [M] Keyframes - -## Goal - -Any item property in `moq-studio` can be keyframed with easing, so an item -can fly in or a camera can push in, and the same animation plays in the -program, a recording, and a replay. - -## Plan - -Fade, slide, wipe, and move transitions are drawn by moq-scene's compositor -on the program clock -([moq-dev/studio#33](https://github.com/moq-dev/studio/pull/33)). Animate in -moq-scene on the media clock, so the program, a recording, and a replay match. -Carrying keyframes in the layout track, for viewers to compose, waits for the -m1 [composition epic](/quest/m1/compose/README.md) (changed 2026-11-04, -maintainer: the program comes first). Overlays animate through their own CSS. -No Lottie runtime for now (decided 2026-10-26): the open ones either lack text -(velato) or add a C++ build (ThorVG). Keyframes change through moq-scene's -`Edit`, so they undo like any edit. - -Decided 2026-11-04 (maintainer): - -- **Start:** an item's keyframes run on the time since its scene went on - program (and into preview, for the preview), play once, and hold the last - keyframe. Rejected: a loop on the program clock, and a trigger or hotkey. -- **Editing:** a drag in the preview moves the item's resting place, and - keyframes are offsets from it, so editing needs no timeline scrubber. - Rejected: setting the keyframe at the preview's time. - -Stingers are their own quest, [Stinger transitions](/quest/m0/stinger.md). - -## Related - -- [Layout track](/quest/m1/compose/layout.md) - where keyframes travel to viewers in m1 diff --git a/quest/m0/stinger.md b/quest/m0/stinger.md index 9c6847e..3efb4f2 100644 --- a/quest/m0/stinger.md +++ b/quest/m0/stinger.md @@ -7,7 +7,7 @@ program that switches scenes at its cut point, as broadcast graphics do. ## Plan -Split from [Keyframes](/quest/m0/animation.md) on 2026-11-04 (maintainer). A +Split from the keyframes quest on 2026-11-04 (maintainer). A stinger is a transition kind beside fade, slide, wipe, and move ([moq-dev/studio#33](https://github.com/moq-dev/studio/pull/33)), drawn on the program clock, whose clip is a media file source. It must start its clip as diff --git a/quest/m1/compose/layout.md b/quest/m1/compose/layout.md index f201481..017a940 100644 --- a/quest/m1/compose/layout.md +++ b/quest/m1/compose/layout.md @@ -44,6 +44,10 @@ themselves; off, viewers see the page only in the program. A replay source layout says when it is hidden and when its span has ended, rather than leaving a viewer on its last frame. +Items carry their keyframes (`moq_scene::Keyframe`), which play from when +their scene went on program, so the layout says when that was on the media +clock, and a viewer that joins late lands at the same point of them. + Only sources the creator publishes independently ([Published independently](/quest/m0/publish-independently.md)) have tracks a viewer can fetch; the rest reach viewers only in the program. diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index 8facac9..a647786 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -7,7 +7,9 @@ use std::path::Path; use serde::{Deserialize, Serialize}; use crate::Error; -use crate::collection::{Audio, Collection, Filter, Item, Layer, Mixes, Source, Video, validate_filters, versioned}; +use crate::collection::{ + Audio, Collection, Filter, Item, Layer, Mixes, Scene, Source, Video, validate_filters, versioned, +}; use crate::edit::Draft; /// What a copy holds: items, or a filter chain. @@ -114,8 +116,12 @@ impl Clip { }, audio: items.audio.clone(), video: items.video.clone(), - scenes: Vec::new(), - top: items.items.clone(), + // In a scene, where keyframes play; pasting into the top + // layer checks it holds none. + scenes: vec![Scene { + items: items.items.clone(), + ..Scene::new("clip") + }], ..Collection::new() } .validate()? @@ -294,6 +300,28 @@ mod tests { assert_eq!(pasted.top, collection.scenes[0].items); } + /// An animated item copies with its keyframes into a scene, and the top + /// layer, which never goes on to play them, refuses it. + #[test] + fn pastes_keyframes_into_a_scene_only() { + let mut collection = collection(); + collection.scenes[0].items[0].keyframes = vec![crate::Keyframe { + x: -100.0, + ..crate::Keyframe::new(std::time::Duration::ZERO) + }]; + let clip = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)); + let text = clip.to_toml().unwrap(); + let pasted = paste(&collection, &Layer::Scene("Scene".into()), &text, DIR); + assert_eq!(pasted.scenes[0].items[1], collection.scenes[0].items[0]); + + let Clip::Items(items) = Clip::from_toml(&text).unwrap() else { + panic!("not items") + }; + let mut draft = collection.draft("Paste"); + draft.paste(&Layer::Top, &items, Path::new(DIR)).unwrap(); + assert!(matches!(draft.finish(), Err(Error::Invalid(_)))); + } + /// Into another collection, a clip brings what its items show, renaming a /// source whose name is taken by another, and its clone follows the rename. #[test] diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index b83fc69..7cf2a8c 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -1,5 +1,6 @@ //! The scene graph, and the TOML file a [`Collection`] saves to. +use std::borrow::Cow; use std::collections::{BTreeMap, BTreeSet}; use std::fmt; use std::path::PathBuf; @@ -177,6 +178,19 @@ impl Collection { .validate() .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; self.validate_chain(&item.filters, &on)?; + if layer == Layer::Top && !item.keyframes.is_empty() { + return Err(Error::Invalid(format!( + "{on} has keyframes, but the top layer never goes on to start them" + ))); + } + if !item.keyframes.is_sorted_by_key(|keyframe| keyframe.at) { + return Err(Error::Invalid(format!("{on} has keyframes out of order"))); + } + for keyframe in &item.keyframes { + keyframe + .validate() + .map_err(|err| Error::Invalid(format!("{on}: {err}")))?; + } } } @@ -411,14 +425,48 @@ impl Transition { /// How far along it looks once `linear` of its time has passed, both from /// 0 to 1: as is for a fade, and eased in and out otherwise. pub(crate) fn progress(&self, linear: f32) -> f32 { + match self { + Self::Fade { .. } => Ease::Linear.apply(linear), + _ => Ease::InOut.apply(linear), + } + } +} + +/// How a change speeds up and slows down over its time. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Ease { + /// At an even pace. + Linear, + /// Starting slowly (cubic). + In, + /// Landing slowly (cubic). + Out, + /// Starting and landing slowly (cubic). + #[default] + InOut, +} + +impl Ease { + /// Every curve. + pub const ALL: [Self; 4] = [Self::Linear, Self::In, Self::Out, Self::InOut]; + + /// How far along the change looks once `linear` of its time has passed, + /// both from 0 to 1. + pub(crate) fn apply(self, linear: f32) -> f32 { let linear = linear.clamp(0.0, 1.0); match self { - Self::Fade { .. } => linear, - // The cubic ease in and out. - _ if linear < 0.5 => 4.0 * linear.powi(3), - _ => 1.0 - (2.0 - 2.0 * linear).powi(3) / 2.0, + Self::Linear => linear, + Self::In => linear.powi(3), + Self::Out => 1.0 - (1.0 - linear).powi(3), + Self::InOut if linear < 0.5 => 4.0 * linear.powi(3), + Self::InOut => 1.0 - (2.0 - 2.0 * linear).powi(3) / 2.0, } } + + fn is_default(&self) -> bool { + *self == Self::default() + } } impl Default for Transition { @@ -512,10 +560,16 @@ pub struct Item { /// How the filtered picture mixes with what is under it. #[serde(default, skip_serializing_if = "Blend::is_normal")] pub blend: Blend, + /// How the item moves away from and back to where it rests, from when its + /// scene goes on, in order of time. It plays once: before the first + /// keyframe the item holds that one, and after the last it holds the last. + /// A top-layer item has none, since the top layer never goes on. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub keyframes: Vec, } impl Item { - /// `source` shown, uncropped and unfiltered, at `rect`. + /// `source` shown, uncropped, unfiltered, and still, at `rect`. pub fn new(source: impl Into, rect: Rect) -> Self { Self { source: source.into(), @@ -524,8 +578,156 @@ impl Item { crop: Crop::default(), filters: Vec::new(), blend: Blend::Normal, + keyframes: Vec::new(), + } + } + + /// The item as it is drawn `since` its scene went on: moved, resized, + /// cropped, and filtered by its keyframes, which the result has none of. + pub fn posed(&self, since: Duration) -> Cow<'_, Self> { + let Some(offset) = self.offset(since) else { + return Cow::Borrowed(self); + }; + let mut filters = self.filters.clone(); + filters.extend(offset.filters); + Cow::Owned(Self { + source: self.source.clone(), + rect: Rect { + x: self.rect.x + offset.x, + y: self.rect.y + offset.y, + // A size shrunk past nothing draws nothing. + width: (self.rect.width + offset.width).max(0.0), + height: (self.rect.height + offset.height).max(0.0), + }, + visible: self.visible, + crop: self.crop.then(&offset.crop), + filters, + blend: self.blend, + keyframes: Vec::new(), + }) + } + + /// Where `since` falls among the keyframes, so two times that settle the + /// same pose the item the same. + pub(crate) fn settled(&self, since: Duration) -> Settled { + match (self.keyframes.first(), self.keyframes.last()) { + (Some(first), _) if since < first.at => Settled::Before, + (Some(_), Some(last)) if since < last.at => Settled::At(since), + _ => Settled::After, } } + + /// How far the keyframes take the item from where it rests `since` its + /// scene went on, or `None` if it has none. + fn offset(&self, since: Duration) -> Option { + let next = self.keyframes.iter().position(|keyframe| keyframe.at > since); + let (from, to) = match next { + None => return self.keyframes.last().cloned(), + Some(0) => return self.keyframes.first().cloned(), + Some(next) => (&self.keyframes[next - 1], &self.keyframes[next]), + }; + // `to` is later than `since`, and `from` no later, so the span is not empty. + let linear = (since - from.at).as_secs_f32() / (to.at - from.at).as_secs_f32(); + Some(from.toward(to, to.ease.apply(linear))) + } +} + +/// Where a moment falls among an item's keyframes: holding the first, between +/// two at that moment, or holding the last. Keyframes at the same moment jump, +/// so the first keyframe's moment is not the same as before it. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub(crate) enum Settled { + Before, + At(Duration), + After, +} + +/// Where an item is at a moment after its scene goes on, as an offset from +/// where it rests, so moving the item moves its whole animation. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Keyframe { + /// How long after the item's scene goes on. + #[serde(with = "humantime_serde")] + pub at: Duration, + /// How the item eases into this keyframe from the one before. + #[serde(default, skip_serializing_if = "Ease::is_default")] + pub ease: Ease, + /// Canvas pixels added to the item's left edge. + #[serde(default, skip_serializing_if = "is_zero")] + pub x: f32, + /// Canvas pixels added to the item's top edge. + #[serde(default, skip_serializing_if = "is_zero")] + pub y: f32, + /// Canvas pixels added to the item's width. + #[serde(default, skip_serializing_if = "is_zero")] + pub width: f32, + /// Canvas pixels added to the item's height. + #[serde(default, skip_serializing_if = "is_zero")] + pub height: f32, + /// Trimmed from what the item's crop leaves, as a push-in does. + #[serde(default, skip_serializing_if = "Crop::is_none")] + pub crop: Crop, + /// Applied after the item's own. Each one's settings mix into the next + /// keyframe's when the two lists pair up by kind, in order, and otherwise + /// change halfway. None reads another source. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub filters: Vec, +} + +impl Keyframe { + /// The item where it rests, `at` after its scene goes on. + pub fn new(at: Duration) -> Self { + Self { + at, + ease: Ease::default(), + x: 0.0, + y: 0.0, + width: 0.0, + height: 0.0, + crop: Crop::default(), + filters: Vec::new(), + } + } + + /// `progress` of the way from this keyframe's offset to `to`'s. + fn toward(&self, to: &Self, progress: f32) -> Self { + let mix = |from: f32, to: f32| mix(from, to, progress); + let paired = self.filters.len() == to.filters.len(); + let pairs = self.filters.iter().zip(&to.filters); + let filters = paired.then(|| pairs.map(|(from, to)| from.toward(to, progress)).collect()); + Self { + at: to.at, + ease: to.ease, + x: mix(self.x, to.x), + y: mix(self.y, to.y), + width: mix(self.width, to.width), + height: mix(self.height, to.height), + crop: self.crop.toward(&to.crop, progress), + filters: filters + .flatten() + .unwrap_or_else(|| if progress < 0.5 { &self.filters } else { &to.filters }.clone()), + } + } + + fn validate(&self) -> Result<(), String> { + let offsets = [self.x, self.y, self.width, self.height]; + if !offsets.iter().all(|offset| offset.is_finite()) { + return Err(format!("a keyframe at {:?} offsets by {offsets:?}", self.at)); + } + self.crop.validate()?; + if let Some(source) = self.filters.iter().find_map(Filter::source) { + return Err(format!( + "a keyframe at {:?} has a filter reading `{source}`; a keyframe's filters read no other source", + self.at + )); + } + self.filters.iter().try_for_each(Filter::validate) + } +} + +fn is_zero(value: &f32) -> bool { + *value == 0.0 } fn visible() -> bool { @@ -899,6 +1101,18 @@ impl Crop { } } + /// `progress` of the way from this crop to `other`, which a mix of two + /// valid crops is. + pub(crate) fn toward(&self, other: &Self, progress: f32) -> Self { + let mix = |from: f32, to: f32| mix(from, to, progress); + Self { + left: mix(self.left, other.left), + top: mix(self.top, other.top), + right: mix(self.right, other.right), + bottom: mix(self.bottom, other.bottom), + } + } + fn validate(&self) -> Result<(), String> { let edges = [self.left, self.top, self.right, self.bottom]; if edges.iter().any(|edge| !(0.0..1.0).contains(edge)) @@ -1337,6 +1551,22 @@ mod tests { ..Crop::default() }, blend: Blend::Screen, + keyframes: vec![ + Keyframe { + x: -740.0, + crop: Crop { + right: 0.5, + ..Crop::default() + }, + filters: vec![Filter::Blur { radius: 8.0 }], + ..Keyframe::new(Duration::ZERO) + }, + Keyframe { + ease: Ease::Out, + filters: vec![Filter::Blur { radius: 0.0 }], + ..Keyframe::new(Duration::from_millis(400)) + }, + ], ..Item::new("camera", rect) }); collection @@ -1402,7 +1632,10 @@ path = "grade.cube" name = "Main" items = [ { source = "bg", rect = { x = 0, y = 0, width = 1280, height = 720 } }, - { source = "cam", rect = { x = 10, y = 10, width = 320, height = 180 }, visible = false }, + { source = "cam", rect = { x = 10, y = 10, width = 320, height = 180 }, visible = false, keyframes = [ + { at = "0s", x = -330, crop = { left = 0.2 }, filters = [{ kind = "opacity", amount = 0 }] }, + { at = "1.5s", ease = "out", filters = [{ kind = "opacity", amount = 1 }] }, + ] }, { source = "cam-faded", rect = { x = 0, y = 0, width = 640, height = 360 }, filters = [{ kind = "opacity", amount = 0.8 }] }, { source = "cam", rect = { x = 0, y = 0, width = 640, height = 360 }, crop = { left = 0.1, right = 0.1 }, blend = "soft_light", filters = [ { kind = "chroma_key", color = "#00ff00", similarity = 0.4, smoothness = 0.08, spill = 0.1 }, @@ -1446,6 +1679,25 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } let main = collection.scene("Main").unwrap(); assert!(main.items[0].visible); assert!(!main.items[1].visible); + assert_eq!( + main.items[1].keyframes, + [ + Keyframe { + x: -330.0, + crop: Crop { + left: 0.2, + ..Crop::default() + }, + filters: vec![Filter::Opacity { amount: 0.0 }], + ..Keyframe::new(Duration::ZERO) + }, + Keyframe { + ease: Ease::Out, + filters: vec![Filter::Opacity { amount: 1.0 }], + ..Keyframe::new(Duration::from_millis(1500)) + }, + ] + ); assert_eq!(collection.capture("cam-faded"), "cam"); let filters: Vec<_> = collection.filters(&main.items[2]).collect(); assert_eq!( @@ -1735,6 +1987,172 @@ rect = { x = 0, y = 680, width = 1280, height = 40 } assert_eq!(Crop::default().then(&item), item); } + /// An item with keyframes `keyframes`, resting at (100, 50), 200x100. + fn animated(keyframes: Vec) -> Item { + let rect = Rect { + x: 100.0, + y: 50.0, + width: 200.0, + height: 100.0, + }; + Item { + filters: vec![Filter::Grayscale { amount: 1.0 }], + keyframes, + ..Item::new("camera", rect) + } + } + + /// Keyframes offset the item from where it rests, ease between each other, + /// and hold the first before it and the last after it. + #[test] + fn keyframes_offset_the_item_and_hold_at_the_ends() { + let item = animated(vec![ + Keyframe { + x: -300.0, + width: -250.0, + crop: Crop { + left: 0.5, + ..Crop::default() + }, + filters: vec![Filter::Opacity { amount: 0.0 }], + ..Keyframe::new(Duration::from_secs(1)) + }, + Keyframe { + ease: Ease::Linear, + filters: vec![Filter::Opacity { amount: 1.0 }], + ..Keyframe::new(Duration::from_secs(2)) + }, + ]); + let at = |millis| item.posed(Duration::from_millis(millis)).into_owned(); + + let first = at(0); + assert_eq!(first, at(1000), "held before the first keyframe"); + assert_eq!(first.rect.x, -200.0); + assert_eq!(first.rect.width, 0.0, "a size shrunk past nothing is nothing"); + assert_eq!(first.crop.left, 0.5); + assert_eq!( + first.filters, + [Filter::Grayscale { amount: 1.0 }, Filter::Opacity { amount: 0.0 }], + "after the item's own" + ); + assert!(first.keyframes.is_empty()); + + let half = at(1500); + assert_eq!(half.rect.x, -50.0); + assert_eq!(half.rect.width, 75.0); + assert_eq!(half.crop.left, 0.25); + assert_eq!(half.filters[1], Filter::Opacity { amount: 0.5 }); + + let last = at(5000); + assert_eq!(last, at(2000), "held after the last keyframe"); + assert_eq!(last.rect, item.rect); + assert_eq!(item.settled(Duration::from_secs(5)), Settled::After); + assert_eq!(item.settled(Duration::from_secs(2)), Settled::After); + assert_eq!(item.settled(Duration::ZERO), Settled::Before); + assert_eq!( + item.settled(Duration::from_secs(1)), + Settled::At(Duration::from_secs(1)) + ); + + let still = Item::new("camera", item.rect); + assert!(matches!(still.posed(Duration::from_secs(1)), Cow::Borrowed(_))); + } + + /// Keyframes at the same moment jump there, so the moment itself poses the + /// item as the later one, and before it as the earlier one. + #[test] + fn keyframes_at_one_moment_jump() { + let at = Duration::from_secs(1); + let item = animated(vec![ + Keyframe { + x: -300.0, + ..Keyframe::new(at) + }, + Keyframe::new(at), + ]); + assert_eq!(item.posed(Duration::ZERO).rect.x, -200.0); + assert_eq!(item.posed(at).rect.x, 100.0); + assert_ne!(item.settled(Duration::ZERO), item.settled(at)); + } + + #[test] + fn eases_start_and_land_slowly() { + assert_eq!(Ease::Linear.apply(0.5), 0.5); + assert_eq!(Ease::In.apply(0.5), 0.125); + assert_eq!(Ease::Out.apply(0.5), 0.875); + assert_eq!(Ease::InOut.apply(0.25), 0.0625); + assert_eq!(Ease::InOut.apply(0.5), 0.5); + for ease in Ease::ALL { + assert_eq!((ease.apply(-1.0), ease.apply(2.0)), (0.0, 1.0), "{ease:?}"); + } + } + + /// Filters that do not pair up by kind change halfway rather than mixing. + #[test] + fn unpaired_keyframe_filters_change_halfway() { + let item = animated(vec![ + Keyframe { + filters: vec![Filter::Sepia { amount: 1.0 }], + ..Keyframe::new(Duration::ZERO) + }, + Keyframe { + ease: Ease::Linear, + filters: vec![Filter::Invert { amount: 1.0 }], + ..Keyframe::new(Duration::from_secs(1)) + }, + ]); + let filters = |millis| item.posed(Duration::from_millis(millis)).filters[1].clone(); + assert_eq!(filters(499), Filter::Sepia { amount: 1.0 }); + assert_eq!(filters(500), Filter::Invert { amount: 1.0 }); + } + + #[test] + fn refuses_keyframes_it_cannot_play() { + let at = |millis| Keyframe::new(Duration::from_millis(millis)); + let cases = [ + vec![at(500), at(100)], + vec![Keyframe { x: f32::NAN, ..at(0) }], + vec![Keyframe { + crop: Crop { + left: 0.6, + right: 0.6, + ..Crop::default() + }, + ..at(0) + }], + vec![Keyframe { + filters: vec![Filter::Opacity { amount: 2.0 }], + ..at(0) + }], + vec![Keyframe { + filters: vec![Filter::Lut { + source: "grade".into(), + amount: 1.0, + }], + ..at(0) + }], + ]; + for keyframes in cases { + let mut collection = example(); + collection.sources.insert( + "grade".into(), + Source::Lut { + path: "grade.cube".into(), + }, + ); + collection.scenes[0].items[0].keyframes = keyframes.clone(); + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "{keyframes:?}"); + } + + let mut collection = example(); + collection.top[0].keyframes = vec![at(0)]; + assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "the top layer"); + // The same moment twice jumps. + collection.top[0].keyframes.clear(); + collection.scenes[0].items[0].keyframes = vec![at(100), at(100)]; + collection.validate().unwrap(); + } + #[test] fn refuses_duplicate_scenes() { let mut collection = example(); diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index 492c612..3770e7b 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -1,13 +1,17 @@ //! Draw scenes onto the canvas on the GPU: a preview, a program, and the //! transition between them. +use std::borrow::Cow; use std::collections::{BTreeMap, HashMap, HashSet}; +use std::time::Duration; use kio::clock::Instant; use moq_video::render::{Renderer, wgpu}; use crate::Error; -use crate::collection::{Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Rect, Source, Transition}; +use crate::collection::{ + Canvas, Collection, Crop, Direction, Filter, Item, Layer as Place, Rect, Settled, Source, Transition, +}; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; use crate::overlay::{Overlay, Overlays}; use crate::source::{self, Sources}; @@ -30,9 +34,11 @@ pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm; /// broadcast switcher and CSS do, so a 50% fade is a 50% mix of the stored /// values. /// -/// A transition runs on the clock [`render`](Self::render) is given, so every -/// output drawn from the program, such as the encoder and a recording, sees the -/// same frame of it. +/// A transition and each item's keyframes run on the clock +/// [`render`](Self::render) is given, so every output drawn from the program, +/// such as the encoder and a recording, sees the same frame of them. An item's +/// keyframes start when its scene goes on program, and separately when it goes +/// into preview, for the preview. pub struct Compositor { device: wgpu::Device, queue: wgpu::Queue, @@ -52,12 +58,22 @@ pub struct Compositor { overlays: Option, preview_scene: Option, program_scene: Option, + /// When the preview scene went into preview, or `None` until the next + /// render, which starts it. + preview_start: Option, + /// When the program scene went on program, as `preview_start` is. + program_start: Option, + /// How long the preview scene had been in preview as of the last render: + /// the moment of its keyframes the preview texture shows. + preview_shown: Duration, switch: Option, } /// A transition under way from `from` to the program scene. struct Switch { from: Option, + /// When `from` went on program. + from_start: Instant, start: Instant, transition: Transition, } @@ -121,6 +137,9 @@ impl Compositor { overlays: None, preview_scene: None, program_scene: None, + preview_start: None, + program_start: None, + preview_shown: Duration::ZERO, switch: None, }) } @@ -190,9 +209,18 @@ impl Compositor { self.program_scene.as_deref().into_iter().chain(leaving) } - /// Put the scene called `scene` in preview, or nothing. + /// Put the scene called `scene` in preview, or nothing, playing its + /// keyframes from the start, even if it was already there. pub fn set_preview(&mut self, scene: Option) { self.preview_scene = scene; + self.preview_start = None; + } + + /// How long the preview scene had been in preview as of the last + /// [`render`](Self::render): the moment of its items' keyframes the + /// [`preview`](Self::preview) shows, for [`Item::posed`]. + pub fn preview_since(&self) -> Duration { + self.preview_shown } /// Follow each renamed scene to its new name, wherever it is: in preview, @@ -232,6 +260,7 @@ impl Compositor { pub fn cut(&mut self) { self.switch = None; self.program_scene = self.preview_scene.clone(); + self.program_start = None; } /// Take the preview to program by `transition`, starting at `now`. @@ -240,8 +269,10 @@ impl Compositor { /// starts from what that one was heading to. pub fn transition(&mut self, transition: Transition, now: Instant) { let from = std::mem::replace(&mut self.program_scene, self.preview_scene.clone()); + let from_start = self.program_start.replace(now).unwrap_or(now); self.switch = (!transition.duration().is_zero()).then_some(Switch { from, + from_start, start: now, transition, }); @@ -257,6 +288,11 @@ impl Compositor { self.update(collection, sources, now); self.paint(collection, now); + let since = |start: &mut Option| now.saturating_duration_since(*start.get_or_insert(now)); + let preview_since = since(&mut self.preview_start); + self.preview_shown = preview_since; + let program_since = since(&mut self.program_start); + let linear = self.switch.as_ref().map(|switch| { let elapsed = now.saturating_duration_since(switch.start); (elapsed.as_secs_f32() / switch.transition.duration().as_secs_f32()).min(1.0) @@ -277,13 +313,13 @@ impl Compositor { // The top layer goes over both outputs, and over a transition rather // than through it, so it stays on across the switch. draw.target(&self.preview); - draw.scene(&self.preview, self.preview_scene.as_deref()); - draw.layer(&self.preview, Place::Top); + draw.scene(&self.preview, On(self.preview_scene.as_deref(), preview_since)); + draw.layer(&self.preview, Place::Top, Duration::ZERO); - let to = self.program_scene.as_deref(); + let to = On(self.program_scene.as_deref(), program_since); match (&self.switch, linear) { (Some(switch), Some(linear)) => { - let from = switch.from.as_deref(); + let from = On(switch.from.as_deref(), now.saturating_duration_since(switch.from_start)); let progress = switch.transition.progress(linear); let canvas = Rect::fill(&self.canvas); match switch.transition { @@ -324,7 +360,7 @@ impl Compositor { draw.scene(&self.program, to); } } - draw.layer(&self.program, Place::Top); + draw.layer(&self.program, Place::Top, Duration::ZERO); self.effects.submit(frame); Ok(()) @@ -526,12 +562,7 @@ impl Pose { width: mix(self.rect.width, other.rect.width), height: mix(self.rect.height, other.rect.height), }, - crop: Crop { - left: mix(self.crop.left, other.crop.left), - top: mix(self.crop.top, other.crop.top), - right: mix(self.crop.right, other.crop.right), - bottom: mix(self.crop.bottom, other.crop.bottom), - }, + crop: self.crop.toward(&other.crop, progress), opacity: mix(self.opacity, other.opacity), } } @@ -554,9 +585,9 @@ fn mixed(old: &Item, new: &Item, progress: f32) -> Option> { /// `other` goes just over the paired item it was over in `other`, or under /// every slot if it was under every paired item. fn stack( - lead: &[Item], + lead: &[Cow<'_, Item>], lead_pairs: &[Option], - other: &[Item], + other: &[Cow<'_, Item>], other_pairs: &[Option], ) -> Vec<(Option, Option)> { let mut over: HashMap, Vec> = HashMap::new(); @@ -597,14 +628,21 @@ fn fits(device: &wgpu::Device, canvas: Canvas) -> Result<(), Error> { Ok(()) } +/// A scene to draw, or none, and how long it has been on: where its items' +/// keyframes are. +#[derive(Clone, Copy)] +struct On<'a>(Option<&'a str>, Duration); + /// One render's draws onto its targets. struct Draw<'a> { collection: &'a Collection, effects: &'a mut Effects, library: Library<'a>, frame: &'a mut Frame, - /// Each item, filtered once though it is drawn onto several targets. - prepared: HashMap<(Place, usize), Option>, + /// Each item, filtered once though it is drawn onto several targets, by + /// where it is and its [`Item::settled`] time, as two outputs showing its + /// scene can be at different points of its keyframes. + prepared: HashMap<(Place, usize, Settled), Option>, } impl Draw<'_> { @@ -613,17 +651,17 @@ impl Draw<'_> { self.frame.target(&target.picture); } - /// Draw `scene`'s items onto `target`. A missing scene draws nothing. - fn scene(&mut self, target: &Target, scene: Option<&str>) { - if let Some(scene) = scene.filter(|name| self.collection.scene(name).is_some()) { - self.layer(target, Place::Scene(scene.to_string())); + /// Draw `on`'s items onto `target`. A missing scene draws nothing. + fn scene(&mut self, target: &Target, on: On) { + if let Some(scene) = on.0.filter(|name| self.collection.scene(name).is_some()) { + self.layer(target, Place::Scene(scene.to_string()), on.1); } } - /// Start drawing `target`, and draw `scene` onto it whole. - fn whole(&mut self, target: &Target, scene: Option<&str>) { + /// Start drawing `target`, and draw `on` onto it whole. + fn whole(&mut self, target: &Target, on: On) { self.target(target); - self.scene(target, scene); + self.scene(target, on); } /// Draw `picture`, a canvas-sized target drawn this render, onto `target` @@ -634,19 +672,20 @@ impl Draw<'_> { } } - /// Draw `place`'s items onto `target`, back to front, each through its crop, - /// filters, and blend mode. A hidden item, a source with no picture yet, or - /// a filter whose source has nothing yet draws nothing. - fn layer(&mut self, target: &Target, place: Place) { + /// Draw `place`'s items onto `target`, back to front, each posed `since` + /// `place` went on, through its crop, filters, and blend mode. A hidden + /// item, a source with no picture yet, or a filter whose source has nothing + /// yet draws nothing. + fn layer(&mut self, target: &Target, place: Place, since: Duration) { let items = self.collection.items(&place).map_or(&[][..], Vec::as_slice); for (index, item) in items.iter().enumerate().filter(|(_, item)| item.visible) { - self.item(target, &place, index, item); + self.item(target, (place.clone(), index, item.settled(since)), &item.posed(since)); } } - /// Draw `item`, at `index` of `place`, onto `target` where it is placed. - fn item(&mut self, target: &Target, place: &Place, index: usize, item: &Item) { - let key = (place.clone(), index); + /// Draw `item`, posed, where it is placed onto `target`, shared with every + /// other draw of it under `key`. + fn item(&mut self, target: &Target, key: (Place, usize, Settled), item: &Item) { let prepared = match self.prepared.get(&key) { Some(prepared) => prepared.clone(), None => { @@ -671,7 +710,7 @@ impl Draw<'_> { } /// Draw `progress` of a move from scene `from` to scene `to` onto - /// `target`. + /// `target`, each item posed by its keyframes as its scene is. /// /// Each visible item of `to` is paired with the first visible item of /// `from` showing the same source that no earlier one took, and glides @@ -682,13 +721,15 @@ impl Draw<'_> { /// unpaired item just over the paired item it was over, so the move /// starts from `from` and lands on `to` as each is drawn. A paired item /// whose blend mode or kinds of filter differ changes look halfway too. - fn moving(&mut self, target: &Target, from: Option<&str>, to: Option<&str>, progress: f32) { + fn moving(&mut self, target: &Target, from: On, to: On, progress: f32) { let collection = self.collection; let items = |scene: Option<&str>| { let scene = scene.and_then(|name| collection.scene(name)); scene.map_or(&[][..], |scene| scene.items.as_slice()) }; - let (leaving, entering) = (items(from), items(to)); + let posed = |on: On| -> Vec> { items(on.0).iter().map(|item| item.posed(on.1)).collect() }; + let resting = items(to.0); + let (leaving, entering) = (posed(from), posed(to)); // Which item of the other scene each visible item pairs with. let mut left = vec![None; leaving.len()]; @@ -704,13 +745,13 @@ impl Draw<'_> { // Each slot is an index into `leaving`, `entering`, or both. let slots: Vec<(Option, Option)> = if progress < 0.5 { - stack(leaving, &left, entering, &entered) + stack(&leaving, &left, &entering, &entered) } else { - let slots = stack(entering, &entered, leaving, &left); + let slots = stack(&entering, &entered, &leaving, &left); slots.into_iter().map(|(to, from)| (from, to)).collect() }; - let place = Place::Scene(to.unwrap_or_default().to_string()); + let place = Place::Scene(to.0.unwrap_or_default().to_string()); for slot in slots { match slot { (Some(at), None) => { @@ -730,13 +771,14 @@ impl Draw<'_> { ); } (Some(at), Some(index)) => { - let (old, new) = (&leaving[at], &entering[index]); + let (old, new) = (&*leaving[at], &*entering[index]); let own = Pose::of(collection, new); let pose = Pose::of(collection, old).toward(&own, progress); if old.filters == new.filters && old.blend == new.blend { if pose == own { - // Shared with the preview, which draws it the same. - self.item(target, &place, index, new); + // Shared with the preview when it draws the same. + let key = (place.clone(), index, resting[index].settled(to.1)); + self.item(target, key, new); } else { self.posed(target, new, pose); } @@ -1300,6 +1342,128 @@ mod tests { .await; } + /// Keyframes sliding an item in from off the canvas's left over a second. + fn slide_in() -> Vec { + vec![ + crate::Keyframe { + x: -8.0, + ..crate::Keyframe::new(Duration::ZERO) + }, + crate::Keyframe { + ease: crate::Ease::Linear, + ..crate::Keyframe::new(Duration::from_secs(1)) + }, + ] + } + + /// An item's keyframes play from when its scene went into preview, for the + /// preview, and from when it went on program, for the program, so one scene + /// can be at two points of them at once. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn keyframes_play_from_when_the_scene_goes_on() { + local(async { + let mut collection = collection(); + collection.scenes[1].items[0].keyframes = slide_in(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + studio.compositor.set_preview(Some("red".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("blue".into())); + + studio.render(at(0)); + assert_eq!(pixel(&studio.preview(), 1, 6), BLACK, "off the canvas"); + studio.render(at(500)); + assert_eq!( + studio.compositor.preview_since(), + Duration::from_millis(500), + "the frame shown" + ); + let preview = studio.preview(); + assert_eq!(pixel(&preview, 1, 6), BLUE, "halfway in"); + assert_eq!(pixel(&preview, 6, 6), BLACK); + + studio.compositor.cut(); + studio.render(at(500)); + assert_eq!(pixel(&studio.program(), 1, 6), BLACK, "on program from the start"); + assert_eq!(pixel(&studio.preview(), 1, 6), BLUE, "the preview carries on"); + + studio.render(at(1000)); + let program = studio.program(); + assert_eq!(pixel(&program, 1, 6), BLUE); + assert_eq!(pixel(&program, 6, 6), BLACK); + assert_eq!(pixel(&studio.preview(), 6, 6), BLUE, "landed where it rests"); + studio.render(at(9000)); + assert_eq!(pixel(&studio.program(), 6, 6), BLUE, "holding the last keyframe"); + + // Picking the scene again plays it again. + studio.compositor.set_preview(Some("blue".into())); + studio.render(at(9000)); + assert_eq!(pixel(&studio.preview(), 1, 6), BLACK); + }) + .await; + } + + /// Two outputs at either side of a jump, keyframes at the same moment, each + /// draw their own side of it rather than share one. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_jump_draws_each_side_on_its_own_output() { + local(async { + let mut collection = collection(); + let jump = Duration::from_secs(1); + collection.scenes[1].items[0].keyframes = vec![ + crate::Keyframe { + x: -8.0, + ..crate::Keyframe::new(jump) + }, + crate::Keyframe::new(jump), + ]; + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(); + studio.render(start); + // Program has reached the jump; preview starts over before it. + studio.compositor.set_preview(Some("blue".into())); + studio.render(start + jump); + assert_eq!(pixel(&studio.preview(), 1, 6), BLACK, "before the jump"); + assert_eq!(pixel(&studio.program(), 1, 6), BLUE, "after the jump"); + }) + .await; + } + + /// A transition starts the incoming scene's keyframes, however long it was + /// in preview, while the outgoing scene's carry on. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn a_transition_starts_the_incoming_keyframes() { + local(async { + let mut collection = collection(); + collection.scenes[0].items[0].keyframes = slide_in(); + collection.scenes[1].items[0].keyframes = slide_in(); + let mut studio = Studio::new(collection).await; + let start = Instant::now(); + let at = |millis| start + Duration::from_millis(millis); + studio.compositor.set_preview(Some("blue".into())); + studio.compositor.cut(); + studio.compositor.set_preview(Some("red".into())); + studio.render(at(0)); + + studio.compositor.transition(FADE, at(9000)); + studio.render(at(9500)); + let program = studio.program(); + // Red has slid halfway in over the left half, blue is in place, and + // the fade is halfway. + let actual = pixel(&program, 1, 6); + assert!(near(actual, [128, 0, 128, 255]), "red half over blue: {actual:?}"); + let actual = pixel(&program, 6, 6); + assert!(near(actual, [0, 0, 128, 255]), "nothing half over blue: {actual:?}"); + }) + .await; + } + /// A move starts from the outgoing scene's stacking and switches to the /// incoming scene's halfway, and an item's filter settings mix from one /// scene's to the other's as it goes. diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index dfed418..33cc911 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -18,7 +18,8 @@ //! program mix. //! - [`Compositor`] draws the scene in preview and the scene on program into //! textures on the caller's `wgpu` device, cutting or running a -//! [`Transition`] between them. `moq_video::render::export` hands the +//! [`Transition`] between them, and plays each item's [`Keyframe`]s from +//! when its scene went on. `moq_video::render::export` hands the //! program to an encoder, on the GPU where the platform allows, and a //! [`Scaler`] draws it smaller for a smaller rendition. //! - [`publish`] gives each local source its own track, in a broadcast the @@ -45,8 +46,8 @@ pub mod source; pub use clip::Clip; pub use collection::{ - Audio, Blend, Canvas, Collection, Crop, Direction, Filter, Item, Layer, MaskMode, Mixes, Rect, Rgba, Scene, Source, - Transition, Video, + Audio, Blend, Canvas, Collection, Crop, Direction, Ease, Filter, Item, Keyframe, Layer, MaskMode, Mixes, Rect, + Rgba, Scene, Source, Transition, Video, }; pub use compositor::{Compositor, FORMAT}; pub use edit::{Change, Draft, Edit, History}; diff --git a/rs/moq-studio/src/interact.rs b/rs/moq-studio/src/interact.rs index 5d287b3..a49081e 100644 --- a/rs/moq-studio/src/interact.rs +++ b/rs/moq-studio/src/interact.rs @@ -1,12 +1,13 @@ //! Interaction with browser sources through the preview: the pointer goes to //! the page under it, and the keyboard to the page last clicked. +use std::borrow::Cow; use std::cell::RefCell; use std::time::Duration; use kio::clock::Instant; use moq_scene::browser::{Button, Input, Modifiers}; -use moq_scene::{Collection, Crop, Rect, Source}; +use moq_scene::{Collection, Crop, Item, Rect, Source}; /// How close in time and space a press must follow the last to count toward a /// double or triple click. @@ -39,8 +40,12 @@ struct Press { struct Capture { page: String, - /// Where the pressed item places the page, since the same page can be in - /// several. + /// The pressed item, by its index among the scene's items then the top + /// layer's, since the same page can be in several, and its keyframes move + /// it during the drag. + slot: usize, + /// Where the pressed item placed the page, for when the slot no longer + /// shows it. place: Place, /// The last point the page heard, where a release lands that comes from /// no pointer. @@ -49,9 +54,10 @@ struct Capture { held: Vec