From a2ba6834a844b880f0e20140af8ac6834e518b75 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 11:55:52 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m0/audio-source-tracks.md Quest-Claim: adcd8b881db6efc1 From 09622f0cc09437c59a2d69185fcc05ea01788adb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 12:30:27 -0800 Subject: [PATCH 2/5] feat(scene): microphones and system audio publish their own Opus tracks Each microphone and system-audio source encodes its capture, before gain and mute, as Opus in its nested catalog while it is used: while the mix on program plays it, or while something reads its track. Isolated recording and replay of audio sources read that track. Co-Authored-By: Claude Opus 5.5 --- README.md | 34 ++-- quest/m0/README.md | 1 - quest/m0/audio-filters.md | 6 + quest/m0/audio-source-tracks.md | 44 ----- quest/m0/publish-independently.md | 6 +- quest/m1/compose/README.md | 1 - quest/m1/compose/layout.md | 11 +- rs/moq-scene/src/publish.rs | 270 +++++++++++++++++++++++++++--- rs/moq-scene/src/source.rs | 99 +++++++---- rs/moq-studio/src/engine.rs | 33 ++-- rs/moq-studio/src/replay.rs | 9 +- rs/moq-studio/src/ui.rs | 30 ++-- 12 files changed, 394 insertions(+), 150 deletions(-) delete mode 100644 quest/m0/audio-source-tracks.md diff --git a/README.md b/README.md index 0353a07..7af941c 100644 --- a/README.md +++ b/README.md @@ -514,9 +514,9 @@ stream. ### Source tracks -Each camera, display, window, image, and color also has a track of its own, so -the studio can replay or record it on its own. These tracks stay in the studio: the -program is the only video it publishes to a relay. They are listed in a local +Each camera, display, window, image, color, microphone, and system audio also +has a track of its own, so the studio can replay or record it on its own. These +tracks stay in the studio: the program is all it publishes to a relay. They are listed in a local catalog under the `dev.moq.studio` section's `sources`, by source name, each as a hang catalog of its own on the program's clock. @@ -529,9 +529,17 @@ changes size or color space moves to a new track, which its catalog entry follows. Images and colors keep their picture once, as the image file and as a one-pixel PNG. Removing or editing a source ends its track. A track carries the plain capture, before any filter, and a clone has none, since it shows its -source's capture. Microphones and system audio have no tracks of their own yet, -nor does a browser source or an overlay, and a broadcast source is already its -publisher's. +source's capture. A browser source or an overlay has no track of its own, and a +broadcast source is already its publisher's. + +A microphone or system audio encodes its track as Opus, from the same device +open the mixes hear, only while it is used: while the mix on program plays it +(unmuted and not left out of it), so a replay of it has its recent past, or +while a recording reads its track. The track carries the sound as captured, +before its gain, mute, and delay, which stay the mixes'. A buffer the capture +lost, or a stretch the source was out of use, is a gap on its track. Opus +encodes on the CPU, about 1% of one Apple M4 core for a stereo source and +half that for mono, and holds no encode session. Every encoder holds an encode session, and consumer GPUs allow only a few hardware sessions at once: 8 on a GeForce card, while a base Apple M4 has one @@ -562,10 +570,11 @@ live or not. The recording has an encoder of its own, opened only while it records, at full quality whatever the uplink grants the live stream, as OBS's advanced output mode records apart from streaming. It costs one more encode session and one more copy of each program frame while recording. With -**each source too**, every camera, display, and window records its -own track into a file of its own beside it, +**each source too**, every camera, display, window, microphone, and system +audio records its own track into a file of its own beside it, ` .mp4`, holding the capture before -any filter; a source being recorded encodes whether or not it is on program. +any filter, or the sound before its gain and mute; a source being recorded +encodes whether or not it is on program. Broadcast sources, guests included, have no track of the studio's, so they get no file of their own. A source added while recording starts its file then, and one removed finishes its file then. A new canvas @@ -604,7 +613,9 @@ scene of its own. In the Replay panel, pick the program or a source, a length up to 30 seconds, and a speed of 100%, 50%, or 25%, then **Replay** plays that many seconds up to the click in the replay source, which is transparent before and after. **Cut** to its scene to put it on program. Sound replays only at -100%, through the replay source's strip in the Audio panel. A new replay cuts +100%, through the replay source's strip in the Audio panel, so a replay of a +microphone or system audio, which has only sound, plays only at 100%. A new +replay cuts off the one playing, and a stretch the source was offline or paused replays as the gap it was. @@ -620,7 +631,8 @@ it is written, shortly after the click. Nothing records in the background for this. The program's tracks keep their last 30 seconds for late viewers anyway, and each source's track while it is on -program or recorded, and a replay copies from those, so a replay needs +program (shown in a scene on air, or for a microphone or system audio, played by +the mix on program) or recorded, and a replay copies from those, so a replay needs `--connect` and goes back at most that far. With `--on-demand`, the program has no past from before someone watched it, and a source has none from before it was last on program or recorded; a replay of nothing is refused, saying why. A diff --git a/quest/m0/README.md b/quest/m0/README.md index e02b0c6..5ffdde1 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -90,7 +90,6 @@ repository. - [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux - [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux -- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks, for isolated recording and replay - [Broadcast sources play in sync](/quest/m0/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker - [Keyframes and stingers](/quest/m0/animation.md) - keyframed item properties with easing, and stinger transitions diff --git a/quest/m0/audio-filters.md b/quest/m0/audio-filters.md index 86ae72c..e1e8402 100644 --- a/quest/m0/audio-filters.md +++ b/quest/m0/audio-filters.md @@ -16,6 +16,12 @@ itself, its filters, and the out-of-process plugin host are moq-audio work in This quest is the studio's side: the panel, the scene-file shape, and wiring each source's chain into the program bus. +Each microphone's and system audio's own track encodes its capture, before +the bus applies gain and mute (`Driver::listen` in +`rs/moq-scene/src/source.rs`). The chain goes before that tap, so the track +carries filtered sound, as video bakes in its ML effects (decided 2026-11-02, +maintainer). + The scene file names filters and their settings, never a plugin's absolute path, so a collection stays portable; a missing plugin loads as failed rather than refusing the file. diff --git a/quest/m0/audio-source-tracks.md b/quest/m0/audio-source-tracks.md deleted file mode 100644 index ec78d62..0000000 --- a/quest/m0/audio-source-tracks.md +++ /dev/null @@ -1,44 +0,0 @@ -# [S] Audio source tracks - -## Goal - -Each microphone and system-audio source in a `moq-studio` scene has its own -capture as an Opus track in the studio's broadcast, listed in that source's -catalog under the studio catalog's `dev.moq.studio` section, beside the -program, as the picture sources already are. An isolated recording and a -replay of an audio source read that track. - -## Plan - -Cameras, displays, windows, images, and colors already publish this way -(`rs/moq-scene/src/publish.rs`): each source's renditions sit in a nested hang -catalog keyed by source name, so the root `video` and `audio` sections stay -the program's. An audio source adds an `audio` rendition to its own nested -catalog the same way. - -moq-audio's capture Producer/Consumer split (`capture::Producer`) feeds both -the bus and an encoder from one device open, rather than opening each device -twice. -Publish the capture after the source's filters but before its gain and mute -(decided 2026-11-02, maintainer): a recording or replay holds filtered audio, -as video bakes in its ML effects, and the levels stay the program's. - -Changed 2026-11-03 (maintainer, on -[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)): a source's -own track encodes only while something in the studio uses it, and reaches -the relay only when the creator publishes the source independently -([Published independently](/quest/m0/publish-independently.md)). A source -on program counts as used (decided 2026-11-04, maintainer), so a replay finds -its past from when it went on: a picture source while the program scene shows -it, and an audio source while it feeds the program mix. A source published -independently counts as used too, since its track is what it publishes. -Otherwise a source encodes only while an isolated recording or a running -replay reads it, and replaying a source with no history is refused with the -reason. -This replaces the earlier always-on decision (2026-11-01, on -[moq-dev/moq#7554](https://github.com/moq-dev/moq/pull/7554)), because -encoding every source costs encoder sessions a consumer machine does not -have; an audio source follows the picture sources' rule. - -Track names stay stable for a source's life, and an edited source that -changes what it shows gets a new name. diff --git a/quest/m0/publish-independently.md b/quest/m0/publish-independently.md index a656aeb..e51363a 100644 --- a/quest/m0/publish-independently.md +++ b/quest/m0/publish-independently.md @@ -16,7 +16,11 @@ not published to the relay by default, since every encoding costs an encoder session and consumer GPUs have few. A source published independently gets one rendition, as every source track does, while the program keeps its two sizes. It counts as used, so it encodes for as long as it is published -([Audio source tracks](/quest/m0/audio-source-tracks.md) states the rule). +(decided 2026-11-04, maintainer): a source encodes while it is on program +(shown in a scene on air, or, for a microphone or system audio, played by the +mix on program) or while something reads its track, and naming a published +source among those on program (`Publish::set_on_program` in +`rs/moq-scene/src/publish.rs`) keeps it encoding. Low priority: this is for services that move webcams and overlays themselves, and none exists yet. The option lives on the source, in the scene file, so it diff --git a/quest/m1/compose/README.md b/quest/m1/compose/README.md index a801760..f8c9b08 100644 --- a/quest/m1/compose/README.md +++ b/quest/m1/compose/README.md @@ -58,7 +58,6 @@ independent sources layered over it) and replan those bullets. ## Required -- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks beside the picture sources (an m0 quest), encoded while the studio uses them and published to the relay only when published independently - [Layout track](/quest/m1/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft - [Browser composition](/quest/m1/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build - [Server composition](/quest/m1/compose/server.md) - a headless compositor renders the program from sources and layout, headless diff --git a/quest/m1/compose/layout.md b/quest/m1/compose/layout.md index 3c725cb..f201481 100644 --- a/quest/m1/compose/layout.md +++ b/quest/m1/compose/layout.md @@ -30,11 +30,10 @@ blend modes. Only a video filter that cannot render the same way in every viewer is baked into a track of its own. A mask names a picture source's track, and a LUT a `lut` source, whose `.cube` table publishes no track yet, so a viewer needs one carrying it, as an image's file is. Audio is the other -way round: an audio source's track is published after its audio filters ([Audio source tracks](/quest/m0/audio-source-tracks.md)), so +way round: an audio source's track carries its sound after its audio filters +and before its gain and mute (decided 2026-11-02, maintainer), so the layout carries only what is applied after that point, such as gain and -mute, and never names an audio filter for a viewer to apply again. The -picture source tracks are enough to start; audio source tracks join the -layout when they land. +mute, and never names an audio filter for a viewer to apply again. A browser source publishes no track of its own (decided 2026-11-03, maintainer, on [moq-dev/studio#8](https://github.com/moq-dev/studio/pull/8)). @@ -48,7 +47,3 @@ a viewer on its last frame. Only sources the creator publishes independently ([Published independently](/quest/m0/publish-independently.md)) have tracks a viewer can fetch; the rest reach viewers only in the program. - -## Related - -- [Audio source tracks](/quest/m0/audio-source-tracks.md) - the audio tracks a layout also names once they land diff --git a/rs/moq-scene/src/publish.rs b/rs/moq-scene/src/publish.rs index 8ee7ee2..665c388 100644 --- a/rs/moq-scene/src/publish.rs +++ b/rs/moq-scene/src/publish.rs @@ -6,9 +6,10 @@ //! sessions a GPU allows. Its renditions are listed in a hang catalog of its //! own, nested by source name under the local catalog's [`SECTION`] section. //! -//! A source encodes only while it is used: while it shows on program -//! ([`Publish::show`]), so a replay of it has its recent past, or while -//! something in the studio subscribes to its track, such as a recording. +//! A source encodes only while it is used: while it is on program +//! ([`Publish::set_on_program`]), shown in a scene on air or heard in the +//! program mix, so a replay of it has its recent past, or while something in +//! the studio subscribes to its track, such as a recording. use std::collections::{BTreeMap, BTreeSet}; use std::sync::atomic::{AtomicUsize, Ordering}; @@ -86,8 +87,8 @@ pub struct Publish { kind: moq_video::encode::Kind, sessions: Sessions, owners: Arc>, - /// The sources on program, which each source's video follows. - shown: Arc>>, + /// The sources on program, which each source's track follows. + on_program: Arc>>, } /// Which [`Claim`] owns each source name's entry in the local catalog, and @@ -119,7 +120,7 @@ impl Publish { kind, sessions, owners: Default::default(), - shown: Default::default(), + on_program: Default::default(), }) } @@ -133,13 +134,13 @@ impl Publish { self.catalog.snapshot() } - /// Encode the sources named in `shown`, those on program, whether or not + /// Encode the sources named in `names`, those on program, whether or not /// anything reads their tracks; any other encodes only while read. - pub fn show(&self, shown: BTreeSet) { + pub fn set_on_program(&self, names: BTreeSet) { // Only a change wakes the sources, since this runs every frame. - self.shown.send_if_modified(|current| { - let changed = *current != shown; - *current = shown; + self.on_program.send_if_modified(|current| { + let changed = *current != names; + *current = names; changed }); } @@ -204,7 +205,7 @@ pub(crate) struct Listing<'a> { claim: &'a Claim, kind: moq_video::encode::Kind, sessions: Sessions, - shown: tokio::sync::watch::Receiver>, + on_program: tokio::sync::watch::Receiver>, broadcast: moq_net::broadcast::Producer, studio: Catalog, catalog: moq_mux::catalog::Producer, @@ -224,7 +225,7 @@ impl<'a> Listing<'a> { claim, kind: publish.kind.clone(), sessions: publish.sessions.clone(), - shown: publish.shown.subscribe(), + on_program: publish.on_program.subscribe(), broadcast: publish.broadcast.clone(), studio: publish.catalog.clone(), catalog, @@ -470,7 +471,7 @@ pub(crate) struct Video<'a> { track: Encoding, claim: &'a Claim, /// The sources on program. - shown: tokio::sync::watch::Receiver>, + on_program: tokio::sync::watch::Receiver>, config: moq_video::encode::Config, clock: moq_mux::Clock, /// Where the capture's private timeline starts on the studio clock, in @@ -501,7 +502,7 @@ impl<'a> Video<'a> { Ok(Self { track: Encoding::new(producer, listing.sessions.clone()), claim: listing.claim, - shown: listing.shown.clone(), + on_program: listing.on_program.clone(), config, clock: listing.studio.clock(), epoch: None, @@ -545,7 +546,7 @@ impl<'a> Video<'a> { /// Whether the source is used: shown on program, or its track read. fn used(&self) -> bool { - self.shown.borrow().contains(&self.claim.name) || self.track.watched() + self.on_program.borrow().contains(&self.claim.name) || self.track.watched() } /// Wait until whether the source is used disagrees with whether its @@ -561,12 +562,12 @@ impl<'a> Video<'a> { )] pub(crate) async fn settle(&self) { let mut demand = self.track.demand(); - let mut shown = self.shown.clone(); - shown.mark_unchanged(); + let mut on_program = self.on_program.clone(); + on_program.mark_unchanged(); while self.used() == self.track.is_open() { // The first look at the demand reports it as it is now. tokio::select! { - changed = shown.changed() => if changed.is_err() { std::future::pending::<()>().await }, + changed = on_program.changed() => if changed.is_err() { std::future::pending::<()>().await }, wanted = demand.wanted_changed() => if wanted.is_err() { std::future::pending::<()>().await }, } } @@ -616,6 +617,91 @@ fn encodable(size: moq_video::Size) -> moq_video::Size { moq_video::Size::new((size.width & !1).max(2), (size.height & !1).max(2)) } +/// A captured source's sound, encoded as Opus while the source is used, as +/// captured: before the gain and mute the program mix applies to it. +#[cfg_attr( + not(any(feature = "capture", test)), + expect(dead_code, reason = "only a capture publishes sound") +)] +pub(crate) struct Audio<'a> { + /// Taken only as the track ends. + producer: Option, + claim: &'a Claim, + /// The sources on program. + on_program: tokio::sync::watch::Receiver>, + clock: moq_mux::Clock, + /// Whether sound went to the encoder since the source last went out of + /// use, so leaving use marks one gap. + encoding: bool, +} + +#[cfg_attr( + not(any(feature = "capture", test)), + expect(dead_code, reason = "only a capture publishes sound") +)] +impl<'a> Audio<'a> { + /// Advertise an Opus rendition of interleaved `f32` sound in `input`'s + /// sample rate and layout. + pub(crate) fn new(listing: &Listing<'a>, input: moq_audio::encode::Input) -> Result { + let mut options = moq_audio::encode::Options::default(); + options.config = moq_audio::encode::Config::from_input(moq_audio::encode::Codec::Opus, &input); + let mut broadcast = listing.broadcast.clone(); + let producer = moq_audio::encode::Producer::new(&mut broadcast, listing.catalog.clone(), input, &options)?; + Ok(Self { + producer: Some(producer), + claim: listing.claim, + on_program: listing.on_program.clone(), + clock: listing.studio.clock(), + encoding: false, + }) + } + + /// Encode and publish `samples`, whose first was captured at `captured`, + /// while the source is used: heard on program, or its track read. `gap` + /// says sound was lost just before them. Leaving use marks a gap, and the + /// sound after one is stamped afresh, so the track's timeline holds it. + pub(crate) fn write(&mut self, samples: &[f32], captured: kio::clock::Instant, gap: bool) -> Result<(), Error> { + let producer = self.producer.as_mut().expect("open until it ends"); + let used = self.on_program.borrow().contains(&self.claim.name) || producer.demand().wanted().is_some(); + if !used { + if std::mem::take(&mut self.encoding) { + producer.discontinuity()?; + } + return Ok(()); + } + if gap && self.encoding { + producer.reanchor(); + } + self.encoding = true; + // A device may stamp a buffer a moment ahead of the clock, which the + // clock refuses as not yet captured. + let timestamp = self.clock.capture(captured.min(kio::clock::now()))?; + let data = bytes::Bytes::copy_from_slice(bytemuck::cast_slice(samples)); + producer.write(&moq_audio::Frame::new(data, timestamp))?; + Ok(()) + } + + /// End the track, failed with `err` so viewers see why. + #[cfg_attr(not(feature = "capture"), expect(dead_code, reason = "only a capture fails"))] + pub(crate) fn abort(mut self, err: &Error) { + if let Some(producer) = self.producer.take() { + producer.abort(moq_net::Error::Transport(err.to_string())); + } + } +} + +impl Drop for Audio<'_> { + /// End the track cleanly, after publishing what the encoder still holds, + /// or failed with the reason that could not be done. + fn drop(&mut self) { + if let Some(mut producer) = self.producer.take() + && let Err(err) = producer.finish() + { + producer.abort(moq_net::Error::Transport(err.to_string())); + } + } +} + /// A still picture, published once as the image file it is. pub(crate) struct Still { track: moq_net::track::Producer, @@ -814,7 +900,7 @@ mod tests { let snapshot = listed(&catalog, |sources| width(sources) == Some(64)).await; assert!(snapshot.video.renditions.is_empty(), "no source is a program rendition"); - publish.show(["camera".to_string()].into()); + publish.set_on_program(["camera".to_string()].into()); for millis in [66, 100, 133] { video.encode(&mut gray(small, millis)).unwrap(); } @@ -827,7 +913,7 @@ mod tests { let snapshot = listed(&catalog, |sources| width(sources) == Some(80)).await; assert_eq!(publish.sessions.open(), 1, "the resize closed the old encoder"); - publish.show(Default::default()); + publish.set_on_program(Default::default()); video.encode(&mut gray(large, 233)).unwrap(); assert_eq!((written(), publish.sessions.open()), (5, 0), "off program"); @@ -876,7 +962,7 @@ mod tests { kio::clock::timeout(wait, video.settle()).await.is_err(), "unused and closed" ); - publish.show(["display".to_string()].into()); + publish.set_on_program(["display".to_string()].into()); kio::clock::timeout(wait, video.settle()) .await .expect("on program without a picture, so the capture brings one"); @@ -884,7 +970,7 @@ mod tests { assert_eq!(publish.sessions.open(), 1); assert!(kio::clock::timeout(wait, video.settle()).await.is_err(), "on program"); - publish.show(Default::default()); + publish.set_on_program(Default::default()); kio::clock::timeout(wait, video.settle()).await.expect("off program"); video.release().unwrap(); assert_eq!(publish.sessions.open(), 0); @@ -894,7 +980,7 @@ mod tests { // lands on the program's clock now rather than a minute back. kio::clock::sleep(std::time::Duration::from_secs(60)).await; video.reopened(); - publish.show(["display".to_string()].into()); + publish.set_on_program(["display".to_string()].into()); let now = publish.catalog.clock().now(); let mut picture = gray(size, 0); video.encode(&mut picture).unwrap(); @@ -956,6 +1042,140 @@ mod tests { .await; } + /// Ten milliseconds of a quiet 440 Hz tone, in stereo at 48 kHz. + fn tone() -> Vec { + (0..480) + .flat_map(|index| { + let sample = (index as f32 * 440.0 * std::f32::consts::TAU / 48_000.0).sin() * 0.25; + [sample, sample] + }) + .collect() + } + + /// What a viewer of a finished track reads, in order: each packet's + /// timestamp, and `None` where the media stops for a gap. + async fn packets( + broadcast: &moq_net::broadcast::Consumer, + name: &str, + config: &hang::catalog::AudioConfig, + ) -> Vec> { + let subscription = + moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(None)); + let track = broadcast + .track(hang::catalog::Reference::track(config, name)) + .unwrap() + .subscribe(subscription) + .await + .unwrap(); + let container = moq_mux::catalog::hang::Container::try_from(config).unwrap(); + let mut reader = moq_mux::container::Consumer::new(track, container); + let mut read = Vec::new(); + while let Some(next) = reader.read().await.unwrap() { + read.push(match next { + moq_mux::container::Read::Frame { frame, .. } => Some(frame.timestamp), + moq_mux::container::Read::End { .. } | moq_mux::container::Read::Gap { .. } => None, + }); + } + read + } + + /// A microphone's sound encodes into its own listing only while the + /// source is used, heard on program or its track read, stamped on the + /// program's clock as it was captured. A gap in the capture, or the + /// source leaving use, is a gap on the track too, and a replay of it holds + /// what was encoded, refusing a span with nothing in it. + #[tokio::test(start_paused = true)] + async fn an_audio_encodes_while_used() { + local(async { + let (publish, catalog, consumer) = studio(); + let claim = publish.claim("mic"); + let listing = Listing::new(&claim, &publish).unwrap(); + let clock = publish.catalog.clock(); + let written = || publish.stats().values().map(|rendition| rendition.frames).sum::(); + let samples = tone(); + let tick = std::time::Duration::from_millis(10); + + tokio::select! { + result = listing.mirror() => panic!("the mirror ended: {result:?}"), + () = async { + let input = moq_audio::encode::Input::new(48_000, moq_audio::Layout::Stereo); + let mut audio = Audio::new(&listing, input).unwrap(); + // A tenth of a second, captured as the clock goes. + let capture = async |audio: &mut Audio<'_>, gap: bool| { + for index in 0..10 { + audio.write(&samples, kio::clock::now(), gap && index == 0).unwrap(); + kio::clock::sleep(tick).await; + } + }; + + // Advertised before anything encodes, so a recording can find it. + let rendition = |sources: &BTreeMap| { + sources.get("mic").is_some_and(|mic| !mic.audio.renditions.is_empty()) + }; + let snapshot = listed(&catalog, rendition).await; + let (name, config) = snapshot.ext.studio.sources["mic"].audio.renditions.first_key_value().unwrap(); + assert_eq!(config.codec, hang::catalog::AudioCodec::Opus); + assert!(snapshot.audio.renditions.is_empty(), "no source is a program rendition"); + let target = || crate::replay::Target { + broadcast: consumer.clone(), + video: None, + audio: Some((name.clone(), config.clone())), + end: clock.now(), + }; + + capture(&mut audio, false).await; + assert_eq!(written(), 0, "unused"); + let refused = crate::replay::Take::new(&target(), std::time::Duration::from_secs(1)); + assert!(refused.is_err(), "nothing to replay"); + + publish.set_on_program(["mic".to_string()].into()); + let heard = clock.now(); + capture(&mut audio, false).await; + assert!(written() > 0, "on program"); + // The capture lost half a second, which the track keeps. + kio::clock::sleep(std::time::Duration::from_millis(500)).await; + let resumed = clock.now(); + capture(&mut audio, true).await; + let take = crate::replay::Take::new(&target(), std::time::Duration::from_secs(5)).unwrap(); + + // Off program, past the gap the first buffer marks. + publish.set_on_program(Default::default()); + capture(&mut audio, false).await; + let before = written(); + capture(&mut audio, false).await; + assert_eq!(written(), before, "off program"); + + // A reader, such as a recording, uses it too. + let mut track = consumer.track(name).unwrap().subscribe(None).await.unwrap(); + capture(&mut audio, false).await; + assert!(written() > before, "read"); + + // Removing the source ends its track cleanly. + drop(audio); + while let Some(mut group) = track.recv_group().await.unwrap() { + while group.read_frame().await.unwrap().is_some() {} + } + + // The replay holds the sound from where the source went on + // program, a codec frame of priming aside, with the lost half + // second kept as a gap. + let (_, config) = take.audio.as_ref().unwrap(); + let read = packets(&take.consume(), name, config).await; + let near = |at: moq_net::Timestamp, to: moq_net::Timestamp| { + at.as_micros().abs_diff(to.as_micros()) <= 20_000 + }; + let first = read.first().copied().flatten().expect("a packet first"); + assert!(near(first, heard), "{first:?} for {heard:?}"); + let gap = read.iter().position(Option::is_none).expect("a gap"); + let after = read[gap + 1].expect("a packet after the gap"); + assert!(near(after, resumed), "{after:?} for {resumed:?}"); + assert!(read[..gap].iter().all(|at| at.is_some_and(|at| at < resumed))); + } => {} + } + }) + .await; + } + /// Holds its newest pictures back the way a pipelined hardware encoder /// does, handing them over only on a flush or finish. openh264, the encoder /// on every machine, holds nothing. @@ -1133,7 +1353,7 @@ mod tests { let dir = tempfile::tempdir().unwrap(); let (publish, catalog, consumer) = studio(); // On program, so its track encodes. - publish.show(["replay".to_string()].into()); + publish.set_on_program(["replay".to_string()].into()); let (mut context, _origin) = context(dir.path()); context.publish = Some(publish); let mut collection = Collection::new(); diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 96ad01c..0078b67 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -5,9 +5,10 @@ //! capture driver is `!Send` on macOS); a driver stops once its feed is //! dropped, which happens when the source is removed or edited. //! -//! With [`Context::publish`] set, each local source that shows a picture also -//! publishes it as its own track in the studio's local broadcast, web pages -//! aside; see [`crate::publish`]. +//! With [`Context::publish`] set, each local source that shows a picture, +//! and each microphone and system audio, also publishes it as its own track in +//! the studio's local broadcast, web pages and overlays aside; see +//! [`crate::publish`]. use std::collections::BTreeMap; use std::future::Future; @@ -310,7 +311,8 @@ impl Feed { /// of whatever this replay source plays now. Its sound plays only in real /// time. /// - /// Refused by any other kind of source, and by a replay source that stopped. + /// Refused by any other kind of source, by a replay source that stopped, + /// and for a take of sound alone, slowed down, which would play nothing. pub fn replay(&self, take: Take, speed: f32) -> Result<(), Error> { let Some(plays) = &self.plays else { return Err(Error::Unsupported("only a replay source plays replays".into())); @@ -320,6 +322,11 @@ impl Feed { "a replay plays at {MIN_SPEED} to 1 times real time, not {speed}" ))); } + if take.video.is_none() && speed != 1.0 { + return Err(Error::Unsupported( + "a replay of sound alone plays only in real time".into(), + )); + } let play = Play { take, speed }; // Fails once the driver is gone, which leaves the reason in the status. plays.send(Arc::new(Mutex::new(Some(play)))).map_err(|_| { @@ -341,10 +348,11 @@ impl Feed { } /// What the source's own track is doing, or `None` when it publishes none: - /// nothing is published, or the source is another broadcast, a web page, or - /// only sound. + /// nothing is published, or the source is another broadcast, a web page, + /// an overlay, or a LUT. /// - /// A failed track leaves the source drawing in the studio's own scenes. + /// A failed track leaves the source drawing and playing in the studio's + /// own scenes and mixes. pub fn track(&self) -> Option { self.track.as_ref().map(|track| track.read().clone()) } @@ -560,8 +568,8 @@ impl Driver { Source::Camera { id } => Video::Camera(id.clone()), Source::Display { id } => Video::Display(id.clone()), Source::Window { id } => Video::Window(id.clone()), - Source::Microphone { id } => return self.listen(Audio::Microphone(id.clone())).await, - Source::SystemAudio => return self.listen(Audio::System).await, + Source::Microphone { id } => return self.listen(Audio::Microphone(id.clone()), listing).await, + Source::SystemAudio => return self.listen(Audio::System, listing).await, Source::Color { .. } | Source::Image { .. } | Source::Broadcast { .. } @@ -655,24 +663,49 @@ impl Driver { } } + /// Capture a microphone or system audio into the bus, and with `listing`, + /// onto the source's own track as well, from the one device open. #[cfg(feature = "capture")] - async fn listen(&self, source: moq_audio::capture::Source) -> Result<(), Error> { + async fn listen(&self, source: moq_audio::capture::Source, listing: Option<&Listing<'_>>) -> Result<(), Error> { let strip = self.strip.clone().expect("audio sources have a strip"); let capture = capture(source).await?; let channel = self.context.bus.capture(capture.consume(), strip)?; - let heard = self.hear(capture.consume()); + let mut track = listing.and_then(|listing| { + let input = capture.consume().input().clone(); + self.publish(crate::publish::Audio::new(listing, input)) + }); + let heard = self.hear(capture.consume(), &mut track); set(&self.status, Status::Live); - tokio::try_join!(capture.run(), channel.run(), heard)?; - Ok(()) + let result = tokio::try_join!( + async { capture.run().await.map_err(Error::from) }, + async { channel.run().await.map_err(Error::from) }, + heard + ); + if let (Some(audio), Err(err)) = (track, &result) { + audio.abort(err); + } + result.map(|_| ()) } /// Note each buffer `capture` delivers, so one that goes quiet reads as - /// stalled. Ends with the capture's error. + /// stalled, and encode it onto `track` while the source is used. Ends with + /// the capture's error. #[cfg(feature = "capture")] - async fn hear(&self, mut capture: moq_audio::capture::Consumer) -> Result<(), moq_audio::Error> { + async fn hear( + &self, + mut capture: moq_audio::capture::Consumer, + track: &mut Option>, + ) -> Result<(), Error> { loop { - capture.read().await?; + let buffer = capture.read().await?; *self.delivered.lock().unwrap() = Some(kio::clock::now()); + if let Some(audio) = track + && let Err(err) = audio.write(&buffer.data, buffer.captured, buffer.gap) + { + tracing::warn!(source = %self.name, %err, "cannot encode the source"); + self.fail_track(Status::Failed(err.to_string())); + track.take().expect("encoding").abort(&err); + } } } @@ -1112,20 +1145,30 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) { } /// Whether `source` publishes its own track: a picture the studio captures or -/// loads itself, before any filter. A broadcast is already published by its own -/// publisher, and a clone shows its source's track, its filters carried by the -/// layout rather than encoded again. +/// loads itself, before the scene's filters, or the sound of a microphone or +/// system audio as captured, before its gain and mute. /// -/// Microphones and system audio do not yet: moq-audio offers no capture that -/// feeds both the program bus and an encoder. Nor do web pages and overlays, -/// which viewers see only in the program: H.264 would encode their -/// transparency over black. +/// A broadcast is already published by its own publisher, and a clone shows +/// its source's track, its filters carried by the layout rather than encoded +/// again. Web pages and overlays are seen only in the program, since H.264 +/// would encode their transparency over black, and a LUT is looked up, never +/// shown. fn publishes(source: &Source) -> bool { - source.has_video() - && !matches!( - source, - Source::Broadcast { .. } | Source::Clone { .. } | Source::Browser { .. } | Source::Overlay { .. } - ) + match source { + Source::Camera { .. } + | Source::Display { .. } + | Source::Window { .. } + | Source::Microphone { .. } + | Source::SystemAudio + | Source::Color { .. } + | Source::Image { .. } + | Source::Replay => true, + Source::Broadcast { .. } + | Source::Browser { .. } + | Source::Overlay { .. } + | Source::Clone { .. } + | Source::Lut { .. } => false, + } } /// The frame rate a capture is encoded at when the device reports none. diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index da4dc9e..9932be3 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -479,16 +479,20 @@ impl Program { } } - /// What a replay of the local source `name` copies: the video of its own - /// track, up to now, or `None` while it publishes no video. + /// What a replay of the local source `name` copies: the video or sound of + /// its own track, up to now, or `None` while it publishes neither. pub fn source(&self, name: &str) -> Option { let mut catalog = self.sources.catalog(); let listed = catalog.ext.studio.sources.remove(name)?; - let video = listed.video.renditions.into_iter().next()?; + let video = listed.video.renditions.into_iter().next(); + let audio = listed.audio.renditions.into_iter().next(); + if video.is_none() && audio.is_none() { + return None; + } Some(moq_scene::replay::Target { broadcast: self.sources.consume(), - video: Some(video), - audio: None, + video, + audio, end: self.clock.now(), }) } @@ -506,18 +510,23 @@ impl Program { } } - /// Encode the local sources the program shows, from `state`, whether or - /// not anything reads their tracks, so a replay of one has its past. - fn show(&self, state: &State) { + /// Encode the local sources on program, from `state`, whether or not + /// anything reads their tracks, so a replay of one has its past: those a + /// scene on air shows, and those the program mix plays. + fn on_program(&self, state: &State) { let collection = &state.collection; let scenes = state.compositor.on_air().filter_map(|scene| collection.scene(scene)); let shown = scenes .flat_map(|scene| &scene.items) .chain(&collection.top) .filter(|item| item.visible) - .map(|item| collection.capture(&item.source).to_string()) - .collect(); - self.sources.show(shown); + .map(|item| collection.capture(&item.source).to_string()); + let heard = state + .sources + .iter() + .filter(|(name, _)| state.sources.plays(&collection.mixes.program, name)) + .map(|(name, _)| name.to_string()); + self.sources.set_on_program(shown.chain(heard).collect()); } } @@ -942,7 +951,7 @@ impl Engine { tracing::error!(%err, "cannot draw the canvas"); continue; } - self.program.show(state); + self.program.on_program(state); (state.compositor.program().clone(), state.compositor.canvas()) }; self.stats.rendered.fetch_add(1, Ordering::Relaxed); diff --git a/rs/moq-studio/src/replay.rs b/rs/moq-studio/src/replay.rs index aa5ca60..ca24014 100644 --- a/rs/moq-studio/src/replay.rs +++ b/rs/moq-studio/src/replay.rs @@ -47,7 +47,8 @@ impl Engine { /// /// The program has a past only while it is encoded, which without a relay /// is only while recorded, a local source only from while it was on - /// program or recorded, and a broadcast source once it shows a picture. + /// program, shown or heard, or recorded, and a broadcast source once it + /// shows a picture. pub fn targets(&self, of: &Of) -> anyhow::Result> { let output = || { self.output @@ -77,10 +78,10 @@ impl Engine { "`{name}` is an overlay, which publishes no track of its own, so it has no past to replay" ) } - Some(source) if source.has_video() => output()? + Some(Source::Lut { .. }) => anyhow::bail!("`{name}` has no picture or sound to replay"), + Some(_) => output()? .source(capture) - .with_context(|| format!("`{name}` publishes no video to replay")), - Some(_) => anyhow::bail!("`{name}` has no picture to replay"), + .with_context(|| format!("`{name}` publishes no video or sound to replay")), None => anyhow::bail!("there is no source named `{name}`"), }; Ok(vec![target?]) diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index f31be18..c4dacda 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -2051,7 +2051,7 @@ impl Panels { /// Replay the recent past of the program or a source in a replay source, /// or save it as a clip. fn replay(&mut self, ui: &mut Ui) { - let (pictures, replays) = { + let (replayable, replays) = { let state = self.engine.state.lock().unwrap(); let named = |kind: fn(&Source) -> bool| -> Vec { let sources = &state.collection().sources; @@ -2061,27 +2061,27 @@ impl Panels { .map(|(name, _)| name.clone()) .collect() }; - // What a source shows, a clone its source's, has a past only if - // something encodes it: not an image, a color, or a web page. + // What a source shows or plays, a clone its source's, has a past + // only if something encodes it: not an image, a color, a web page, + // an overlay, or a LUT. let collection = state.collection(); - let pictures = collection + let replayable = collection .sources .keys() .filter(|name| { - let shown = collection.sources.get(collection.capture(name)); - shown.is_some_and(|source| { - source.has_video() - && !matches!( - source, - Source::Image { .. } - | Source::Color { .. } | Source::Browser { .. } - | Source::Overlay { .. } - ) + let captured = collection.sources.get(collection.capture(name)); + captured.is_some_and(|source| { + !matches!( + source, + Source::Image { .. } + | Source::Color { .. } | Source::Browser { .. } + | Source::Overlay { .. } | Source::Lut { .. } + ) }) }) .cloned() .collect::>(); - (pictures, named(|source| matches!(source, Source::Replay))) + (replayable, named(|source| matches!(source, Source::Replay))) }; let panel = &mut self.replay; if panel.into.as_ref().is_none_or(|into| !replays.contains(into)) { @@ -2094,7 +2094,7 @@ impl Panels { .selected_text(panel.of.to_string()) .show_ui(ui, |ui| { ui.selectable_value(&mut panel.of, Of::Program, Of::Program.to_string()); - for name in pictures { + for name in replayable { let of = Of::Source(name); let label = of.to_string(); ui.selectable_value(&mut panel.of, of, label); From 3595a77883c6b0e9df4e3d7982b138839cd2195f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 13:11:28 -0800 Subject: [PATCH 3/5] fix(studio): only microphones and system audio count as heard on program Co-Authored-By: Claude Opus 5.5 --- rs/moq-studio/src/engine.rs | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 9932be3..7115ca4 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -512,7 +512,9 @@ impl Program { /// Encode the local sources on program, from `state`, whether or not /// anything reads their tracks, so a replay of one has its past: those a - /// scene on air shows, and those the program mix plays. + /// scene on air shows, and the microphones and system audio the program + /// mix plays. A replay source's track is its pictures, so being heard + /// alone does not put it on program. fn on_program(&self, state: &State) { let collection = &state.collection; let scenes = state.compositor.on_air().filter_map(|scene| collection.scene(scene)); @@ -524,7 +526,12 @@ impl Program { let heard = state .sources .iter() - .filter(|(name, _)| state.sources.plays(&collection.mixes.program, name)) + .filter(|(name, feed)| { + matches!( + feed.source(), + moq_scene::Source::Microphone { .. } | moq_scene::Source::SystemAudio + ) && state.sources.plays(&collection.mixes.program, name) + }) .map(|(name, _)| name.to_string()); self.sources.set_on_program(shown.chain(heard).collect()); } From c2d45a71d1f0fb32aa3664af8b9962d7e70f96cc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 14:27:25 -0800 Subject: [PATCH 4/5] fix(studio): replay a local source without --connect Its track is kept in the local broadcast with or without a relay; only the program's past needs one. Co-Authored-By: Claude Opus 5.5 --- README.md | 5 +++-- rs/moq-studio/src/replay.rs | 24 ++++++++++++------------ 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index c271185..c554247 100644 --- a/README.md +++ b/README.md @@ -638,8 +638,9 @@ it is written, shortly after the click. Nothing records in the background for this. The program's tracks keep their last 30 seconds for late viewers anyway, and each source's track while it is on program (shown in a scene on air, or for a microphone or system audio, played by -the mix on program) or recorded, and a replay copies from those, so a replay needs -`--connect` and goes back at most that far. With `--on-demand`, the program has +the mix on program) or recorded, and a replay copies from those, so a replay +goes back at most that far, and one of the program needs `--connect`. A local +source keeps its track without a relay too. With `--on-demand`, the program has no past from before someone watched it, and a source has none from before it was last on program or recorded; a replay of nothing is refused, saying why. A broadcast source replays from what the studio received of it, and only over a diff --git a/rs/moq-studio/src/replay.rs b/rs/moq-studio/src/replay.rs index ca24014..717bc69 100644 --- a/rs/moq-studio/src/replay.rs +++ b/rs/moq-studio/src/replay.rs @@ -45,19 +45,17 @@ impl Engine { /// What a replay of `of` could copy, ending now, the most wanted first: /// the program at each of its sizes, largest first, or a source's one. /// - /// The program has a past only while it is encoded, which without a relay - /// is only while recorded, a local source only from while it was on - /// program, shown or heard, or recorded, and a broadcast source once it - /// shows a picture. + /// The program has a past only with a relay, from while it was encoded, a + /// local source, relay or not, only from while it was on program, shown or + /// heard, or recorded, and a broadcast source once it shows a picture. pub fn targets(&self, of: &Of) -> anyhow::Result> { - let output = || { - self.output - .as_ref() - .map(|_| &self.program) - .context("without --connect, nothing holds a past to replay") - }; let name = match of { - Of::Program => return Ok(output()?.program()), + Of::Program => { + self.output + .as_ref() + .context("without --connect, the program holds no past to replay")?; + return Ok(self.program.program()); + } Of::Source(name) => name, }; let state = self.state.lock().unwrap(); @@ -79,7 +77,9 @@ impl Engine { ) } Some(Source::Lut { .. }) => anyhow::bail!("`{name}` has no picture or sound to replay"), - Some(_) => output()? + // A local source's track is kept with or without a relay. + Some(_) => self + .program .source(capture) .with_context(|| format!("`{name}` publishes no video or sound to replay")), None => anyhow::bail!("there is no source named `{name}`"), From 254d19737b14b315f18f83b325a1557a92920cd6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 15:19:31 -0800 Subject: [PATCH 5/5] docs: say plainly that only a replay of the program needs --connect Co-Authored-By: Claude Opus 5.5 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c554247..e796589 100644 --- a/README.md +++ b/README.md @@ -639,8 +639,8 @@ Nothing records in the background for this. The program's tracks keep their last 30 seconds for late viewers anyway, and each source's track while it is on program (shown in a scene on air, or for a microphone or system audio, played by the mix on program) or recorded, and a replay copies from those, so a replay -goes back at most that far, and one of the program needs `--connect`. A local -source keeps its track without a relay too. With `--on-demand`, the program has +goes back at most that far. A replay of the program needs `--connect`; a local +source keeps its track without a relay. With `--on-demand`, the program has no past from before someone watched it, and a source has none from before it was last on program or recorded; a replay of nothing is refused, saying why. A broadcast source replays from what the studio received of it, and only over a