Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
484 changes: 271 additions & 213 deletions Cargo.lock

Large diffs are not rendered by default.

16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ cargo run --release -p moq-studio --features capture,openh264 -- \
- Without `--connect`, the studio composes and previews but publishes nothing,
and broadcast sources stay offline.
- `--failover <url>` publishes the same broadcast to a second relay under the
same publisher epoch, so a viewer linked to both resumes mid-group from it
when the first fails. Both relays must speak the one `--connect-version`,
which must carry publisher epochs (`moq-lite-07-wip` today, which a relay
same broadcast instance, so a viewer linked to both resumes mid-group from
it when the first fails. Both relays must speak the one `--connect-version`,
which must carry instances (`moq-lite-07-wip` today, which a relay
accepts only when configured to), or the studio refuses to start; it also
refuses a fixed `--connect-bind` port, since each relay's connection binds
its own. Its credential rides its URL's `?jwt=`; `--connect-token` goes to
Expand Down Expand Up @@ -130,10 +130,10 @@ comes along, renamed if its name is taken. Copied filters paste onto the
selected source or item, after its own. The clipboard holds text in the scene
file's format.

A broadcast source subscribes through the same relay connections. It plays the
newest instance of its broadcast through either relay, preferring the first
relay among routes to the same instance, so a publisher restarted behind only
the failover replaces the first relay's copy. It plays the
A broadcast source subscribes through the same relay connections. It plays its
broadcast through the first relay while that relay has a route to it, and
through the failover otherwise, so a publisher restarted behind only the
failover replaces the first relay's copy once that copy ends. It plays the
best video and audio its catalog lists, switches when the catalog changes, and
comes back on its own when the publisher restarts. Audio from every source is mixed into the program whether
or not its scene is live; mute a source to leave it out.
Expand Down Expand Up @@ -479,7 +479,7 @@ it is unhealthy. Below that it shows:
read them, so most show "unknown". In a cluster, a relay forwarding the
stream to another counts as one session. Two relays whose stats use the
same node name publish them at the same path, so only one of them counts:
the newer instance, or the first relay's when they are one instance.
the first relay's while it has them.
- **Tracks**: each track's bitrate and frame rate, and, from viewers that
report, the lowest bitrate any received, its worst grade, and the latency
from the program clock to the slowest viewer. The program's tracks come
Expand Down
2 changes: 1 addition & 1 deletion moq
Submodule moq updated 1842 files
2 changes: 0 additions & 2 deletions quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,7 @@ repository.
- [Virtual camera](/quest/m0/vcam/README.md) - a canvas's program exposed as a system camera on Linux, Windows, and macOS
- [Signing identities are available](/quest/m0/signing.md) - the Apple and Windows code-signing credentials packaging and the macOS camera need
- [Packaging](/quest/m0/packaging.md) - signed installers for macOS and Windows and a Flatpak for Linux
- [One audio capture feeding the bus and an encoder is pinned](/quest/m0/pin-audio-capture.md) - moq-audio's capture split, for audio source tracks
- [moq-mux's presentation timeline is pinned](/quest/m0/pin-presentation-timeline.md) - shared playout timing, for broadcast sync
- [moq-audio's mix router is pinned](/quest/m0/pin-audio-mix-router.md) - several mixes and mix-minus, for audio mixes
- [The persistable portal restore token is pinned](/quest/m0/pin-portal-token.md) - moq-video's restore token, for Wayland sources
- [Platform AAC encoders are pinned](/quest/m0/pin-aac-encode.md) - moq-audio's AAC encoders, for destinations
- [moq-audio's filter chain is pinned](/quest/m0/pin-audio-filters.md) - the chain audio filters configure
Expand Down
12 changes: 3 additions & 9 deletions quest/m0/audio-stall.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,8 @@ after a second without a picture.

## Plan

Audio capture runs inside the audio bus today, which reports no last-delivery
time per source, so the stall check video sources have cannot see it. The
capture Producer/Consumer split that the audio-capture pin brings gives the
studio its own handle on each capture; read delivery from there rather than
The audio bus reports no last-delivery time per source, so the stall check
video sources have cannot see it. The studio opens each capture as a
`capture::Producer`, so read delivery from a consumer of its own rather than
adding a side channel in the bus. Raised on
[#5](https://github.com/moq-dev/studio/pull/5).

## Required

- [moq-audio's capture split is pinned](/quest/m0/pin-audio-capture.md) - the
per-capture handle this reads
12 changes: 3 additions & 9 deletions quest/m0/compose/sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,9 @@ catalog keyed by source name, so the root `video` and `audio` sections stay
the program's. An audio source adds an `audio` rendition to its own nested
catalog the same way.

Blocked upstream: moq-audio's capture stream is crate-private, and the only
public captures feed either the program bus (`bus::Capture`) or an encoder
(`encode::publish_capture`), never both. The
[pin](/quest/m0/pin-audio-capture.md) brings a capture Producer/Consumer split
that feeds both from one device open, rather than opening each device twice.
moq-audio's capture Producer/Consumer split (`capture::Producer`) feeds both
the bus and an encoder from one device open, rather than opening each device
twice.
Publish the capture after the source's filters but before its gain and mute
(decided 2026-11-02, maintainer): viewers hear filtered audio, as video bakes
in its ML effects, and a composing viewer mixes the levels itself.
Expand All @@ -37,7 +35,3 @@ always-on. Rejected: sources on demand, which would make replaying a source
need an armed local subscriber again.
Track names stay stable for a source's life, and an edited source that
changes what it shows gets a new name.

## Required

- [One audio capture feeding the bus and an encoder is pinned](/quest/m0/pin-audio-capture.md) - one device open feeds the mix and the source's own track
12 changes: 4 additions & 8 deletions quest/m0/mixes.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,13 @@ Each guest hears a mix-minus: the program without their own voice.
## Plan

Decided 2026-11-02 (maintainer): routing, mix-minus, and sync offsets live in
moq-audio as a multi-mix router rather than one `Bus` per mix in the studio,
so this quest waits on its [pin](/quest/m0/pin-audio-mix-router.md). The
studio adds the routing UI, meters, the headphone monitor, clap-to-measure
offsets, and a mix pick per output and recording.
moq-audio as a multi-mix router rather than one `Bus` per mix in the studio:
`bus::Mix`, which the program audio already is. The studio adds the routing
UI, meters, the headphone monitor, clap-to-measure offsets, and a mix pick per
output and recording.

The mixer runs on the audio clock and never blocks on a slow output. An
output that falls behind drops its own audio, not the bus's.

Benchmark source count against mix count, including sparse routing. A mix
must cost only what is routed into it, never a scan of every source.

## Required

- [moq-audio's mix router is pinned](/quest/m0/pin-audio-mix-router.md) - routing, mix-minus, and sync offsets
11 changes: 0 additions & 11 deletions quest/m0/pin-audio-capture.md

This file was deleted.

12 changes: 0 additions & 12 deletions quest/m0/pin-audio-mix-router.md

This file was deleted.

3 changes: 2 additions & 1 deletion rs/moq-scene/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@
//! microphone, or system-audio capture (with the `capture` feature), a still
//! image, a solid color, a broadcast subscribed over MoQ, or a web page
//! rendered by a browser engine (the [`browser`] module). Pictures go to the
//! compositor and sound goes to a `moq_audio::bus::Bus`, the program audio.
//! compositor and sound goes to a `moq_audio::bus::Bus`, routed into its
//! program mix.
//! - [`Compositor`] draws the scene in preview and the scene on program into
//! textures on the caller's `wgpu` device, cutting or fading between them.
//! `moq_video::render::export` hands the program to an encoder, on the GPU
Expand Down
8 changes: 5 additions & 3 deletions rs/moq-scene/src/publish.rs
Original file line number Diff line number Diff line change
Expand Up @@ -733,8 +733,8 @@ mod tests {
let (name, rendition) = listing.catalog.snapshot().video.renditions.pop_first().unwrap();
// Every group, however far behind the newest, since this reads only once
// the track has ended.
let mut subscription = moq_net::track::Subscription::default();
subscription.max_delay = None;
let subscription =
moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(None));
let subscriber = consumer.track(&name).unwrap().subscribe(subscription).await.unwrap();
let container = moq_mux::catalog::hang::Container::try_from(&rendition).unwrap();
let mut reader = moq_mux::container::Consumer::new(subscriber, container);
Expand Down Expand Up @@ -884,7 +884,9 @@ mod tests {
let track = consumer
.track(hang::catalog::Reference::track(config, name))
.unwrap()
.subscribe(moq_net::track::Subscription::default().with_max_delay(None))
.subscribe(
moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(None)),
)
.await
.unwrap();
let container = moq_mux::catalog::hang::Container::try_from(config).unwrap();
Expand Down
6 changes: 4 additions & 2 deletions rs/moq-scene/src/replay.rs
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ fn copy(
// stale.
let subscription = moq_net::track::Subscription::default()
.with_ranges([range])
.with_max_delay(None)
.with_delay(moq_net::track::Delay::default().with_max(None))
.with_order(moq_net::track::Order::Ascending);
let subscribing = target.broadcast.track(name)?.subscribe(subscription);
// Never woken, like the reads below: a publisher that never describes the
Expand Down Expand Up @@ -362,7 +362,9 @@ mod tests {
.consume()
.track("video")
.unwrap()
.subscribe(moq_net::track::Subscription::default().with_max_delay(None))
.subscribe(
moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(None)),
)
.await
.unwrap();
let mut groups = Vec::new();
Expand Down
85 changes: 66 additions & 19 deletions rs/moq-scene/src/source.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ use std::time::Duration;

use hang::catalog::{AudioConfig, Reference, VideoConfig};
use kio::clock::Instant;
use moq_audio::bus::{Bus, Strip};
use moq_audio::bus::{Bus, Mix, Strip};
use moq_mux::catalog::Stream;
use tokio::sync::watch;

Expand Down Expand Up @@ -109,8 +109,10 @@ pub enum Status {
/// What every source shares.
#[derive(Clone)]
pub struct Context {
/// The program audio that microphones, system audio, and broadcasts feed.
/// Where microphones, system audio, and broadcasts feed their sound.
pub bus: Bus,
/// The program audio: every running source with sound is routed into it.
pub program: Mix,
/// Where broadcast sources are announced: the studio's relay connection.
pub origin: moq_net::origin::Consumer,
/// What a relative image or LUT path is relative to: the scene file's directory.
Expand Down Expand Up @@ -321,6 +323,16 @@ struct Track {
status: kio::Producer<Status>,
}

/// Takes the source out of the program, which [`open`] routed it into, so a
/// removed or edited source leaves nothing behind in the mix.
impl Drop for Driver {
fn drop(&mut self) {
if let Some(strip) = &self.strip {
self.context.program.unroute(strip);
}
}
}

impl Driver {
/// The source's name in the collection.
pub fn name(&self) -> &str {
Expand Down Expand Up @@ -510,11 +522,11 @@ impl Driver {
#[cfg(feature = "capture")]
async fn listen(&self, source: moq_audio::capture::Source) -> Result<(), Error> {
let strip = self.strip.clone().expect("audio sources have a strip");
let mut config = moq_audio::capture::Config::default();
config.source = source;
let capture = self.context.bus.capture(config, strip);
let capture = capture(source).await?;
let channel = self.context.bus.capture(capture.consume(), strip)?;
set(&self.status, Status::Live);
Ok(capture.run().await?)
tokio::try_join!(capture.run(), channel.run())?;
Ok(())
}

#[cfg(not(feature = "capture"))]
Expand Down Expand Up @@ -868,8 +880,8 @@ impl Driver {

/// How a replay reads its take: from the first group, where the take's tracks
/// declare they start, skipping nothing as late, since a take is history.
const REPLAY: moq_mux::container::Delay = moq_mux::container::Delay {
min: Duration::ZERO,
const REPLAY: moq_net::track::Delay = moq_net::track::Delay {
min: Some(Duration::ZERO),
max: None,
};

Expand Down Expand Up @@ -902,6 +914,9 @@ fn open(name: &str, source: &Source, context: &Context) -> (Feed, Driver) {
let picture = Mailbox::default();
let shown = Shown::default();
let strip = source.has_audio().then(Strip::default);
if let Some(strip) = &strip {
context.program.route(strip);
}
let status = kio::Producer::new(Status::Starting);
let page = matches!(source, Source::Browser { .. }).then(Page::default);
// Claimed here, in the order sources open, so an edited source's
Expand Down Expand Up @@ -981,6 +996,29 @@ fn set(status: &kio::Producer<Status>, value: Status) {
}
}

/// Open `source` in a layout the bus can mix. Past stereo a device's channels
/// carry no speaker positions to downmix by, such as a microphone array's or an
/// interface's inputs, so a device that defaults to more is asked for stereo,
/// then mono, and fails with the formats it offers if it has neither.
#[cfg(feature = "capture")]
async fn capture(source: moq_audio::capture::Source) -> Result<moq_audio::capture::Producer, moq_audio::Error> {
let mut config = moq_audio::capture::Config::default();
config.source = source;
let capture = moq_audio::capture::Producer::open(config.clone()).await?;
if !matches!(capture.consume().input().layout, moq_audio::Layout::Discrete(_)) {
return Ok(capture);
}
drop(capture);
config.channels = Some(2);
match moq_audio::capture::Producer::open(config.clone()).await {
Err(moq_audio::Error::Unsupported(_)) => {
config.channels = Some(1);
moq_audio::capture::Producer::open(config).await
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
}
result => result,
}
}

/// Read a still image, decoded to straight-alpha RGBA, and the file it came from.
fn image(path: &Path) -> Result<(Picture, bytes::Bytes), Error> {
let bytes = std::fs::read(path).map_err(|err| Error::Image(format!("{}: {err}", path.display())))?;
Expand All @@ -1004,8 +1042,10 @@ pub(crate) mod test {
pub(crate) fn context(dir: &Path) -> (Context, moq_net::origin::Producer) {
let (origin, driver) = moq_net::origin::Producer::new(Default::default());
tokio::task::spawn_local(moq_net::time::run(driver));
let (bus, _) = Bus::new(Default::default()).unwrap();
let context = Context {
bus: Bus::new(Default::default()).unwrap(),
program: bus.mix(),
bus,
origin: origin.consume(),
dir: dir.to_path_buf(),
output: moq_video::Output::Cpu,
Expand Down Expand Up @@ -1187,7 +1227,8 @@ mod tests {
}

/// Editing a source restarts it, removing one stops it, and an untouched one
/// keeps running. Gain changes reach the strip without restarting anything.
/// keeps running. Gain changes reach the strip without restarting anything,
/// and a source plays in the program until it is removed.
#[tokio::test]
async fn sync_restarts_only_what_changed() {
local(async {
Expand All @@ -1198,6 +1239,8 @@ mod tests {
let drivers = sources.sync(&collection);
start(&sources, drivers).await;
let guest = sources.get("guest").unwrap().strip().unwrap().clone();
let program = sources.context().program.clone();
assert!(program.routes(&guest));

collection.audio.insert(
"guest".into(),
Expand Down Expand Up @@ -1227,6 +1270,16 @@ mod tests {
collection.sources.remove("red");
assert!(sources.sync(&collection).is_empty());
assert!(sources.get("red").is_none());

collection.sources.remove("guest");
assert!(sources.sync(&collection).is_empty());
for _ in 0..1000 {
if !program.routes(&guest) {
return;
}
tokio::task::yield_now().await;
}
panic!("a removed source stays in the program");
})
.await;
}
Expand Down Expand Up @@ -1257,9 +1310,7 @@ mod tests {
.unwrap()
};
let first = audio("first", None);
publication
.announce(moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()))
.unwrap();
publication.announce(moq_net::origin::Route::default()).unwrap();

let mut sources = Sources::new(context);
let drivers = sources.sync(&collection());
Expand Down Expand Up @@ -1300,9 +1351,7 @@ mod tests {
.unwrap();
};
point(&mut catalog, "first");
publication
.announce(moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()))
.unwrap();
publication.announce(moq_net::origin::Route::default()).unwrap();

let mut sources = Sources::new(context);
let drivers = sources.sync(&collection());
Expand Down Expand Up @@ -1337,9 +1386,7 @@ mod tests {
catalog.audio.renditions.insert("audio".into(), config);
})
.unwrap();
publication
.announce(moq_net::origin::Route::default().with_epoch(moq_net::Epoch::mint()))
.unwrap();
publication.announce(moq_net::origin::Route::default()).unwrap();

let mut sources = Sources::new(context);
let drivers = sources.sync(&collection());
Expand Down
Loading