diff --git a/README.md b/README.md index a70ccd5..19573f4 100644 --- a/README.md +++ b/README.md @@ -106,7 +106,7 @@ Every panel is a tab you can drag, split, or stack. | Filters | Remove the selected source's background, crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. | | Keyframes | Animate the selected item: add, time, ease, and remove its keyframes, and set how far each moves, resizes, crops, and filters it (see [Keyframes](#keyframes)). **Play** plays the preview scene's keyframes again. | | Audio | Pick the audio scene on program and the one to listen to and on which output, set each sound's gain, mute, and delay, and send each sound and audio scene into audio scenes, with a meter on every one. | -| Output | Start and stop recording, pick the picture and the audio scene it records, set the canvas size and frame rate and each scene's size, see the broadcast path, whether the program is encoding, the encoder sessions open, and the encoder work of the program, the recording, and each published scene, and turn the virtual camera on. | +| Output | Start and stop recording, pick the picture and the audio scene it records, set the canvas size and frame rate and each scene's size, see the broadcast path, whether the program is encoding, the encoder sessions open, and the encoder work of the program, the recording, and each published scene, and turn the virtual camera of the program or of each published scene on. | | Health | See how the stream reaches viewers, relay by relay and, for the program and each published scene, track by track, and the cause of each problem, and which recordings a crash left unfinished were repaired at launch. | | Replay | Replay the last moments of the program or a source in a replay source, slowed down if you like, or save them as a clip. | | Hotkeys | Bind keys to cut, transition, preview a scene, replay, save a clip, and start or stop recording, and clear them (see [Hotkeys](#hotkeys)). | @@ -1074,19 +1074,26 @@ Each catalog asks viewers to report beneath its broadcast, under The **Virtual camera** toggle in the Output panel shows the program as a camera that video-call and conferencing apps can pick, at the canvas size and -frame rate. It stays on until you turn it off or quit; it is not saved in the -scene file. The program is copied for the camera only while an app reads it. -Changing the canvas size or frame rate reopens the camera at the new one, so -an app reading it opens it again. The Devices panel leaves the camera out -while it is on, so it cannot be added as a source of the program it shows. - -- **Linux:** a PipeWire camera named "MoQ Studio" (needs the `pipewire` - feature), which browsers and other PipeWire-aware apps list beside your - other cameras. Without PipeWire, the studio writes to the first +frame rate. Each published scene has a toggle of its own beneath it in the +panel, showing that scene at its own size as a camera of its own, so an app +can pick the program or any published scene. A camera stays on until you +turn it off or quit; it is not saved in the scene file. A scene's camera +turns off when you stop publishing or rename the scene. A picture is copied +for a camera only while an app reads it, and a scene's camera has the scene +drawn only then. Changing the size or frame rate of what a camera +shows reopens it at the new one, so an app reading it opens it again. The +Devices panel leaves the studio's cameras out while they are on, so one +cannot be added as a source of the picture it shows. + +- **Linux:** a PipeWire camera named "MoQ Studio" for the program, and + "MoQ Studio: " for a scene (needs the `pipewire` feature), which + browsers and other PipeWire-aware apps list beside your other cameras. + Without PipeWire, each camera writes to the first [v4l2loopback](https://github.com/v4l2loopback/v4l2loopback) device that - takes the canvas size, for apps that open `/dev/video*` directly. The - studio never loads that module, since it needs root: - `sudo modprobe v4l2loopback exclusive_caps=1 card_label="MoQ Studio"`. + takes its size and that no other camera of the studio holds, for apps that + open `/dev/video*` directly. The studio never loads that module, since it + needs root. Load a device for each camera you turn on, such as two: + `sudo modprobe v4l2loopback devices=2 exclusive_caps=1,1 card_label="MoQ Studio","MoQ Studio 2"`. - **macOS and Windows:** not available yet. ## Development diff --git a/quest/m0/vcam/README.md b/quest/m0/vcam/README.md index bfd50ca..a7b19cc 100644 --- a/quest/m0/vcam/README.md +++ b/quest/m0/vcam/README.md @@ -18,14 +18,13 @@ Decided 2026-10-22: - **macOS:** a CoreMediaIO Camera Extension, which needs signing. - **Order:** the first backend to land adds the studio toggle. -The Linux backend landed the toggle: **Virtual camera** in the Output panel, -for the program, not saved in the scene file. Each OS backend is a -`vcam::backend` in `rs/moq-studio`, fed tightly packed RGBA program frames on -the `vcam` thread only while an app reads the camera. Each published scene -gets a toggle of its own, starting on Linux. +The Linux backend landed the toggles: **Virtual camera** in the Output panel, +one for the program and one for each published scene, not saved in the scene +file. Each OS backend is a `vcam::backend` in `rs/moq-studio`, fed tightly +packed RGBA frames of what its camera shows on a `vcam` thread only while an +app reads the camera, and names its camera after the scene it shows. ## Required -- [Linux virtual camera for each published scene](/quest/m0/vcam/linux.md) - a PipeWire camera per published scene, beside the program's - [Windows virtual camera](/quest/m0/vcam/windows.md) - a Media Foundation virtual camera on Windows 11 - [macOS virtual camera](/quest/m0/vcam/macos.md) - a CoreMediaIO Camera Extension inside the app bundle diff --git a/quest/m0/vcam/linux.md b/quest/m0/vcam/linux.md deleted file mode 100644 index 0dc2a36..0000000 --- a/quest/m0/vcam/linux.md +++ /dev/null @@ -1,27 +0,0 @@ -# [S] Linux virtual camera for each published scene - -## Goal - -Each published scene has a **Virtual camera** toggle of its own, as the -program does, and shows as its own PipeWire camera named after the scene, or -else its own v4l2loopback device. An app can then pick the program or any -published scene. The Devices panel leaves out every camera the studio -publishes, so picking one cannot feed it back into itself. - -## Plan - -The program's camera landed in -[moq-dev/studio#9](https://github.com/moq-dev/studio/pull/9): `vcam::Camera` -in `rs/moq-studio` reads the program back through moq-video's -`render::readback` and reopens when the canvas changes size or frame rate. -Its PipeWire node name (`moq-studio`) and description ("MoQ Studio") are -fixed, so a camera per scene needs a name of its own, such as -`moq-studio-`. - -Published scenes landed in -[moq-dev/studio#71](https://github.com/moq-dev/studio/pull/71): the program -clock names each published scene as a compositor output -(`Compositor::set_outputs`), drawn at the scene's own size only while -something wants its frame, and reads its texture from `Compositor::output`. -A scene's camera is one more reader: it asks for the scene to be drawn while -an app reads it, and reopens when the scene's size changes. diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index f003e44..039dde6 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -20,8 +20,9 @@ //! - **encode**: one per program rendition, which waits for each program copy //! and encodes it. A new canvas size or frame rate starts new renditions. //! - Recordings write their files on the tokio runtime. -//! - **vcam** waits for each program copy the virtual camera takes, while it is -//! on, and hands it to the camera's backend. +//! - **vcam**: one per virtual camera on, the program's or a published +//! scene's, which waits for each copy the camera takes and hands it to the +//! camera's backend. //! - **source-***: one per running source, since a capture stream may not //! leave the thread that opened it. //! - **autosave** writes the collection and its history to their files once @@ -100,8 +101,9 @@ pub struct State { pub reloads: u64, pub sources: Sources, pub compositor: Compositor, - /// The program's virtual camera, while it is on. - camera: Option, + /// Each virtual camera on: the program's under `None`, and each published + /// scene's under its name. + cameras: BTreeMap, crate::vcam::Camera>, /// Edited since the sources were last synced. stale: bool, /// For the autosave, which writes the collection once edits settle. @@ -114,9 +116,17 @@ pub struct State { } impl State { - /// How the virtual camera is doing, or `None` while it is off. - pub fn camera(&self) -> Option { - self.camera.as_ref().map(crate::vcam::Camera::status) + /// How the program's virtual camera, or the published scene `scene`'s, + /// is doing, or `None` while it is off. + pub fn camera(&self, scene: Option<&str>) -> Option { + self.cameras + .get(&scene.map(str::to_string)) + .map(crate::vcam::Camera::status) + } + + /// How each virtual camera on is doing. + pub fn cameras(&self) -> impl Iterator { + self.cameras.values().map(crate::vcam::Camera::status) } pub fn collection(&self) -> &Collection { @@ -897,7 +907,7 @@ impl Engine { reloads: 0, sources, compositor, - camera: None, + cameras: BTreeMap::new(), stale: true, edits: autosave.edits(), renamed: Vec::new(), @@ -1292,17 +1302,32 @@ impl Engine { } } - /// Turn the program's virtual camera on or off. Turning it on fails, - /// saying why, when this machine has no way to publish one. - pub fn set_camera(&self, on: bool) -> anyhow::Result<()> { - let canvas = self.state.lock().unwrap().compositor.canvas(); + /// Turn the program's virtual camera, or the published scene `scene`'s, + /// on or off. Turning it on fails, saying why, when the scene is not + /// published or this machine has no way to publish a camera. A scene's + /// camera turns off once the scene is no longer published under its name. + pub fn set_camera(&self, scene: Option<&str>, on: bool) -> anyhow::Result<()> { + let canvas = { + let state = self.state.lock().unwrap(); + match scene { + None => state.compositor.canvas(), + Some(scene) => published(&state.collection) + .into_iter() + .find(|(name, _)| name == scene) + .map(|(_, canvas)| canvas) + .with_context(|| format!("scene `{scene}` is not published"))?, + } + }; // Opened and closed outside the lock, so the program clock does not // wait on the camera's backend. - let camera = match on { - true => Some(crate::vcam::Camera::open(&self.gpu.device, &self.gpu.queue, canvas)?), - false => None, + let key = scene.map(str::to_string); + let previous = match on { + true => { + let camera = crate::vcam::Camera::open(&self.gpu.device, &self.gpu.queue, scene, canvas)?; + self.state.lock().unwrap().cameras.insert(key, camera) + } + false => self.state.lock().unwrap().cameras.remove(&key), }; - let previous = std::mem::replace(&mut self.state.lock().unwrap().camera, camera); drop(previous); Ok(()) } @@ -1316,10 +1341,12 @@ impl Engine { /// encodes only while recording. /// /// A published or recorded scene is drawn only while one of its encoders - /// takes the frame, so one nobody reads costs no drawing. See [`Encoders`] - /// for what encodes each. Drawing outlives the encoders: a failed output - /// stops copying its picture, and the scenes keep moving. The virtual - /// camera, while on, takes every program frame an app reads. + /// or virtual camera takes the frame, so one nobody reads costs no + /// drawing. See [`Encoders`] for what encodes each. Drawing outlives the + /// encoders: a failed output stops copying its picture, and the scenes + /// keep moving. Each virtual camera on takes every frame of what it shows + /// while an app reads it, and one whose scene is no longer published + /// turns off. /// /// Runs until the studio [stops](Self::stop), and returns once every /// rendition's encode thread has finished its track. @@ -1338,6 +1365,9 @@ impl Engine { loop { let tick = clock.tick().await; + // The cameras of scenes no longer published, closed once the lock + // is let go, so the UI does not wait on their backends. + let mut closed: Vec = Vec::new(); if self.stopped.load(Ordering::Relaxed) { // Without a sender, each encoder publishes what it was sent and // still holds, then finishes its track. @@ -1404,8 +1434,24 @@ impl Engine { broadcast.audio.set(mix.clone()); } } + closed.extend( + state + .cameras + .extract_if(.., |scene, _| { + scene + .as_ref() + .is_some_and(|scene| !published.iter().any(|(name, _)| name == scene)) + }) + .map(|(_, camera)| camera), + ); let (recorded, recorded_canvas) = recorded(&state.collection); - let wants = |name: &str| matches!(lines.get(name), Some(Some((line, _))) if line.wants()); + let cameras = &state.cameras; + let wants = |name: &str| { + matches!(lines.get(name), Some(Some((line, _))) if line.wants()) + || cameras + .get(&Some(name.to_string())) + .is_some_and(crate::vcam::Camera::wanted) + }; let mut outputs: BTreeMap<&str, bool> = published.iter().map(|(name, _)| (name.as_str(), wants(name))).collect(); if let Some(scene) = &recorded { @@ -1424,10 +1470,19 @@ impl Engine { continue; } self.program.on_program(state, shown); - if let Some(camera) = &mut state.camera { - camera.offer(state.compositor.program(), state.compositor.canvas()); - } let compositor = &state.compositor; + for (scene, camera) in &mut state.cameras { + match scene { + None => camera.offer(Some(compositor.program()), compositor.canvas()), + Some(scene) => { + let Some((_, canvas)) = published.iter().find(|(name, _)| name == scene) else { + continue; + }; + let picture = compositor.output(scene).and_then(moq_scene::Output::texture); + camera.offer(picture, *canvas); + } + } + } let program = (compositor.program().clone(), compositor.canvas()); let texture = |scene: &str| compositor.output(scene).and_then(moq_scene::Output::texture).cloned(); Drawn { @@ -1445,6 +1500,7 @@ impl Engine { program, } }; + drop(closed); self.stats.rendered.fetch_add(1, Ordering::Relaxed); let timestamp = moq_net::Timestamp::try_from(epoch + tick.saturating_duration_since(start)).ok(); @@ -4153,63 +4209,129 @@ mod tests { } } - /// The virtual camera reaches an app that opens PipeWire cameras as the - /// picture the studio composed, and follows the canvas to a new size. + /// Each virtual camera reaches an app that opens PipeWire cameras: the + /// program's as the picture the studio composed, and a published scene's, + /// beside it, as that scene at its own size, each named after what it + /// shows. Each follows what it shows to a new size, and a scene's turns + /// off once the scene is no longer published. #[cfg(all(target_os = "linux", feature = "pipewire"))] #[tokio::test(flavor = "multi_thread")] #[ignore = "needs a PipeWire session and a Vulkan device; run by just render-sw"] async fn the_virtual_camera_reaches_a_pipewire_app() { - /// Open the camera the way an app opens a PipeWire camera, by role and - /// class, and wait for the red picture the studio composes. - async fn red(engine: &Engine, width: u32, height: u32) -> moq_video::capture::Stream { + /// Open the camera of the program, or of `scene`, the way an app opens + /// a PipeWire camera, and wait for the `color` picture the studio + /// composes at `size`. + async fn shows( + engine: &Engine, + scene: Option<&str>, + size: moq_video::Size, + color: [u8; 3], + ) -> moq_video::capture::Stream { + let id = engine.state.lock().unwrap().camera(scene).expect("a camera").id; let mut config = moq_video::capture::Config::default(); - config.source = moq_video::capture::Source::Camera(Some("pipewire:moq-studio".into())); - config.width = Some(width); - config.height = Some(height); + config.source = moq_video::capture::Source::Camera(Some(id)); + config.width = Some(size.width); + config.height = Some(size.height); let mut stream = kio::clock::timeout(TIMEOUT, moq_video::capture::open(&config)) .await .expect("the camera opens") .expect("a camera stream"); - assert_eq!((stream.width(), stream.height()), (width, height)); + assert_eq!((stream.width(), stream.height()), (size.width, size.height)); // The canvas is black until the color source delivers its first // picture, so wait for the composed one rather than take the first. kio::clock::timeout(TIMEOUT, async { loop { let frame = stream.read().await.expect("capture").expect("the camera ended"); let rgba = frame.surface.to_rgba(&Default::default()).expect("rgba"); - let center = ((height / 2 * width + width / 2) * 4) as usize; - let [r, g, b, _] = rgba.data()[center..center + 4] else { - unreachable!() - }; + let center = ((size.height / 2 * size.width + size.width / 2) * 4) as usize; + let got = &rgba.data()[center..center + 3]; // The reader converts to I420, which shifts a primary a little. - if r > 200 && g < 60 && b < 60 { + if got.iter().zip(color).all(|(got, want)| got.abs_diff(want) < 60) { return; } } }) .await .expect("the composed picture arrives"); - let status = engine.state.lock().unwrap().camera().expect("a camera"); + let status = engine.state.lock().unwrap().camera(scene).expect("a camera"); assert!(status.reading, "the camera does not know an app reads it"); assert!(status.sent > 0); assert_eq!(status.error, None); stream } + /// Wait until the camera of the program, or of `scene`, shows `canvas`. + async fn reopened(engine: &Engine, scene: Option<&str>, canvas: moq_scene::Canvas) { + kio::clock::timeout(TIMEOUT, async { + while engine.state.lock().unwrap().camera(scene).expect("a camera").canvas != canvas { + kio::clock::sleep(Duration::from_millis(100)).await; + } + }) + .await + .expect("the camera reopens at the new size"); + } + + /// Make the scene `tall` `size`, published or not. + fn set_tall(engine: &Engine, size: moq_scene::Size, published: bool) { + let mut state = engine.state.lock().unwrap(); + let scene = state.collection().scenes[1].clone(); + let mut draft = state.collection().draft("Change the scene"); + let scene = moq_scene::Scene { + size: Some(size), + published, + ..scene + }; + draft.set_scene(1, scene).unwrap(); + state.commit(draft, None).unwrap(); + } + + const RED: [u8; 3] = [255, 0, 0]; + const BLUE: [u8; 3] = [0, 0, 255]; let dir = tempfile::tempdir().unwrap(); let gpu = Gpu::new().await.expect("a GPU"); let (autosave, _) = Autosave::open(dir.path().join("show.toml")).unwrap(); - let engine = Engine::start(launch(gpu, Vec::new(), dir.path()), autosave, red_canvas()) + let mut loaded = red_canvas(); + let blue = moq_scene::Source::Color { + color: moq_scene::Rgba([0, 0, 255, 255]), + }; + loaded.collection.sources.insert("blue".into(), blue); + let fill = moq_scene::Rect { + x: 0.0, + y: 0.0, + width: 360.0, + height: 360.0, + }; + let tall = moq_scene::Size { + width: 180, + height: 320, + }; + loaded.collection.scenes.push(moq_scene::Scene { + size: Some(tall), + published: true, + items: vec![moq_scene::Item::new("blue", fill)], + ..moq_scene::Scene::new("tall") + }); + let engine = Engine::start(launch(gpu, Vec::new(), dir.path()), autosave, loaded) .await .expect("the studio starts"); - engine.set_camera(true).expect("the camera turns on"); - let status = engine.state.lock().unwrap().camera().expect("a camera"); - assert_eq!(status.backend, "PipeWire"); - assert_eq!(status.id, "pipewire:moq-studio"); - assert!(!status.reading, "nothing reads the camera yet"); + engine.set_camera(None, true).expect("the camera turns on"); + engine + .set_camera(Some("tall"), true) + .expect("the scene's camera turns on"); + for (scene, id) in [ + (None, "pipewire:moq-studio"), + (Some("tall"), "pipewire:moq-studio-tall"), + ] { + let status = engine.state.lock().unwrap().camera(scene).expect("a camera"); + assert_eq!(status.backend, "PipeWire"); + assert_eq!(status.id, id); + assert!(!status.reading, "nothing reads the camera yet"); + } - let stream = red(&engine, 640, 360).await; - drop(stream); + // Both at once, each showing its own picture. + let program = shows(&engine, None, moq_video::Size::new(640, 360), RED).await; + let scene = shows(&engine, Some("tall"), moq_video::Size::new(180, 320), BLUE).await; + drop((program, scene)); let canvas = moq_scene::Canvas { width: 320, @@ -4222,18 +4344,41 @@ mod tests { draft.set_canvas(canvas).unwrap(); state.commit(draft, None).unwrap(); } + reopened(&engine, None, canvas).await; + drop(shows(&engine, None, moq_video::Size::new(320, 180), RED).await); + + let square = moq_scene::Size { + width: 320, + height: 320, + }; + set_tall(&engine, square, true); + let canvas = moq_scene::Canvas { + width: 320, + height: 320, + framerate: 30, + }; + reopened(&engine, Some("tall"), canvas).await; + drop(shows(&engine, Some("tall"), moq_video::Size::new(320, 320), BLUE).await); + + set_tall(&engine, square, false); kio::clock::timeout(TIMEOUT, async { - while engine.state.lock().unwrap().camera().expect("a camera").canvas != canvas { + while engine.state.lock().unwrap().camera(Some("tall")).is_some() { kio::clock::sleep(Duration::from_millis(100)).await; } }) .await - .expect("the camera reopens at the new size"); - let stream = red(&engine, 320, 180).await; + .expect("the scene's camera turns off once it is not published"); + assert!( + engine.set_camera(Some("tall"), true).is_err(), + "the scene is not published" + ); + assert!( + engine.state.lock().unwrap().camera(None).is_some(), + "the program's stays on" + ); - drop(stream); - engine.set_camera(false).expect("the camera turns off"); - assert!(engine.state.lock().unwrap().camera().is_none()); + engine.set_camera(None, false).expect("the camera turns off"); + assert_eq!(engine.state.lock().unwrap().cameras().count(), 0); engine.stop(); } } diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index cd7c5e0..98c3739 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -1947,8 +1947,15 @@ impl Panels { let refreshed = self.devices.refresh(&self.runtime).context("cannot list devices"); self.check(refreshed); } - let camera = self.engine.state.lock().unwrap().camera(); - let Some(picked) = self.devices.ui(ui, camera.as_ref().map(|camera| camera.id.as_str())) else { + let own: Vec = self + .engine + .state + .lock() + .unwrap() + .cameras() + .map(|camera| camera.id) + .collect(); + let Some(picked) = self.devices.ui(ui, &own) else { return; }; let mut state = self.engine.state.lock().unwrap(); @@ -2820,27 +2827,33 @@ impl Panels { if let Some(error) = video.audio_error() { ui.colored_label(ui.visuals().error_fg_color, format!("audio stopped: {error}")); } + self.camera(ui, Some(&name)); } ui.separator(); - self.camera(ui); + self.camera(ui, None); } - /// The program's virtual camera: a toggle, and how it is doing while on. - fn camera(&mut self, ui: &mut Ui) { - let camera = self.engine.state.lock().unwrap().camera(); + /// The virtual camera of the program, or of the published scene `scene`: + /// a toggle, and how it is doing while on. + fn camera(&mut self, ui: &mut Ui, scene: Option<&str>) { + let camera = self.engine.state.lock().unwrap().camera(scene); let mut on = camera.is_some(); + let (shows, named) = match scene { + None => ("the program".to_string(), "virtual camera".to_string()), + Some(scene) => (format!("scene `{scene}`"), format!("scene `{scene}`'s virtual camera")), + }; if ui .checkbox(&mut on, "Virtual camera") - .on_hover_text("Show the program as a camera that video-call apps can pick") + .on_hover_text(format!("Show {shows} as a camera that video-call apps can pick")) .changed() { - let result = self.engine.set_camera(on); - let message = if on { "virtual camera on" } else { "virtual camera off" }; - self.report(result.map(|()| message.to_string())); + let result = self.engine.set_camera(scene, on); + let message = format!("{named} {}", if on { "on" } else { "off" }); + self.report(result.map(|()| message)); } let Some(camera) = camera else { return }; - egui::Grid::new("camera").show(ui, |ui| { + egui::Grid::new(("camera", scene)).show(ui, |ui| { ui.label("Camera"); ui.label(&camera.backend); ui.end_row(); @@ -4579,10 +4592,10 @@ impl Devices { } } - /// Draw the listing, leaving out the camera `own` is the id of, the - /// studio's own virtual camera, so the program is not fed back into + /// Draw the listing, leaving out the cameras `own` holds the ids of, the + /// studio's own virtual cameras, so what one shows is not fed back into /// itself; the device to add once the user picks one. - fn ui(&mut self, ui: &mut Ui, own: Option<&str>) -> Option { + fn ui(&mut self, ui: &mut Ui, own: &[String]) -> Option { if self.pending.is_some() { ui.spinner(); } @@ -4602,7 +4615,7 @@ impl Devices { Ok(devices) => { for (index, device) in devices.iter().enumerate() { if let Source::Camera { id: Some(id) } = &device.source - && Some(id.as_str()) == own + && own.contains(id) { continue; } diff --git a/rs/moq-studio/src/vcam.rs b/rs/moq-studio/src/vcam.rs index cfcb475..d59bca6 100644 --- a/rs/moq-studio/src/vcam.rs +++ b/rs/moq-studio/src/vcam.rs @@ -1,14 +1,15 @@ -//! The program as a system camera, for video-call and conferencing apps that -//! offer no ingest. +//! The program, or a published scene, as a system camera, for video-call and +//! conferencing apps that offer no ingest. //! //! On Linux the camera is a PipeWire video node (with the `pipewire` feature), //! which PipeWire-aware apps such as browsers list as a camera. Without //! PipeWire it is an existing v4l2loopback device, since that kernel module //! needs root to load and so is never the first choice. //! -//! The program clock offers every program frame; [`Camera::offer`] -//! has moq-video read it back only while an app reads the camera, and the -//! `vcam` thread waits for the copy and hands the pixels to the backend. +//! The program clock offers every frame it draws of what a camera shows; +//! [`Camera::offer`] has moq-video read it back only while an app reads the +//! camera, and the `vcam` thread waits for the copy and hands the pixels to +//! the backend. use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, Mutex, mpsc}; @@ -22,7 +23,7 @@ mod pipewire; #[cfg(target_os = "linux")] mod v4l2; -/// The program, published as a camera other apps can open. +/// The program, or a published scene, as a camera other apps can open. /// /// Dropping it removes the camera. pub struct Camera { @@ -39,11 +40,11 @@ pub struct Camera { /// What the program clock hands the `vcam` thread. enum Work { - /// A program copy on its way back from the GPU. Unbounded in the channel + /// A copy on its way back from the GPU. Unbounded in the channel /// only in name: the readback lends out at most [`Readback::DEPTH`] at once. Picture(Pending), - /// The canvas changed size or frame rate: close the backend and open it - /// again for this one. + /// What the camera shows changed size or frame rate: close the backend + /// and open it again for this canvas. Reopen(Canvas), } @@ -116,21 +117,28 @@ trait Sink: Send { } impl Camera { - /// Publish the `canvas` programs drawn on `device` as a camera. + /// Publish the program, or the published scene called `scene`, drawn on + /// `device` at `canvas`, as a camera named after it. /// /// Fails, naming why each backend refused, when this machine has no way to /// publish one. - pub fn open(device: &wgpu::Device, queue: &wgpu::Queue, canvas: Canvas) -> anyhow::Result { + pub fn open( + device: &wgpu::Device, + queue: &wgpu::Queue, + scene: Option<&str>, + canvas: Canvas, + ) -> anyhow::Result { let readback = Readback::new(device, queue, size(canvas))?; let shared = Arc::new(Shared::default()); - let sink = open(canvas, &shared)?; + let scene = scene.map(str::to_string); + let sink = open(scene.as_deref(), canvas, &shared)?; let (work, pending) = mpsc::channel(); let worker = { let shared = shared.clone(); std::thread::Builder::new() .name("vcam".into()) - .spawn(move || run(pending, sink, &shared)) + .spawn(move || run(pending, sink, scene.as_deref(), &shared)) .context("cannot start the virtual camera thread")? }; Ok(Self { @@ -144,14 +152,21 @@ impl Camera { }) } - /// Copy `program`, drawn at `canvas`, to the camera if an app reads it. + /// Whether an app reads the camera, so what it shows is worth drawing. + pub fn wanted(&self) -> bool { + self.shared.wanted() + } + + /// Copy `picture`, what the camera shows as drawn at `canvas`, to the + /// camera if an app reads it, or show nothing new for `None`, when it was + /// not drawn. /// /// A canvas of a new size or frame rate reopens the camera for it, so an /// app reading it opens it again. /// /// The copy is submitted before this returns, so the caller draws into - /// `program` again freely. Never waits on the GPU. - pub fn offer(&mut self, program: &wgpu::Texture, canvas: Canvas) { + /// `picture` again freely. Never waits on the GPU. + pub fn offer(&mut self, picture: Option<&wgpu::Texture>, canvas: Canvas) { let Some(work) = &self.work else { return }; if canvas != self.canvas { self.canvas = canvas; @@ -162,10 +177,10 @@ impl Camera { // A send fails only once the thread stopped, having said why. let _ = work.send(Work::Reopen(canvas)); } - if !self.shared.wanted() { + let Some(picture) = picture.filter(|_| self.shared.wanted()) else { return; - } - match self.readback.copy(program) { + }; + match self.readback.copy(picture) { Ok(Some(pending)) => { let _ = work.send(Work::Picture(pending)); } @@ -205,33 +220,41 @@ fn size(canvas: Canvas) -> moq_video::Size { moq_video::Size::new(canvas.width, canvas.height) } -/// Open the first backend that takes `canvas`, and say where it lives in -/// `shared`. -fn open(canvas: Canvas, shared: &Arc) -> anyhow::Result> { - let (sink, backend, id) = backend(size(canvas), canvas.framerate, shared)?; +/// Open the first backend that takes `canvas` for the camera of the program, +/// or of `scene`, and say where it lives in `shared`. +fn open(scene: Option<&str>, canvas: Canvas, shared: &Arc) -> anyhow::Result> { + let (sink, backend, id) = backend(scene, size(canvas), canvas.framerate, shared)?; tracing::info!(%backend, "virtual camera on"); *shared.opened.lock().unwrap() = Opened { backend, id, canvas }; Ok(sink) } -/// The first backend that takes a `size` camera at `framerate`, with where it -/// lives and its capture id. +/// The first backend that takes a `size` camera of the program, or of +/// `scene`, at `framerate`, with where it lives and its capture id. #[cfg(target_os = "linux")] fn backend( + scene: Option<&str>, size: moq_video::Size, framerate: u32, shared: &Arc, ) -> anyhow::Result<(Box, String, String)> { #[cfg(feature = "pipewire")] - let pipewire = match pipewire::Node::open(size, framerate, shared.clone()) { - Ok(node) => return Ok((Box::new(node), "PipeWire".to_string(), pipewire::ID.to_string())), + let pipewire = match pipewire::Node::open(scene, size, framerate, shared.clone()) { + Ok(node) => { + let id = node.id(); + return Ok((Box::new(node), "PipeWire".to_string(), id)); + } Err(err) => { tracing::warn!(err = %format!("{err:#}"), "no PipeWire camera, trying v4l2loopback"); format!("{err:#}") } }; #[cfg(not(feature = "pipewire"))] - let pipewire = "this build has no `pipewire` feature".to_string(); + let pipewire = { + // A loopback device is named when its module loads, not by the studio. + let _ = scene; + "this build has no `pipewire` feature".to_string() + }; match v4l2::Loopback::open(size, framerate) { Ok(loopback) => { @@ -246,27 +269,29 @@ fn backend( #[cfg(not(target_os = "linux"))] fn backend( + scene: Option<&str>, size: moq_video::Size, framerate: u32, shared: &Arc, ) -> anyhow::Result<(Box, String, String)> { - let _ = (size, framerate, shared); + let _ = (scene, size, framerate, shared); anyhow::bail!("the virtual camera is not available on this platform yet") } -/// Hand each copy to `sink` as it lands, until the camera is dropped or fails. -fn run(pending: mpsc::Receiver, mut sink: Box, shared: &Arc) { +/// Hand each copy to `sink`, the camera of the program or of `scene`, as it +/// lands, until the camera is dropped or fails. +fn run(pending: mpsc::Receiver, mut sink: Box, scene: Option<&str>, shared: &Arc) { while let Ok(work) = pending.recv() { let result = match work { Work::Picture(pending) => pending .wait() - .context("reading the program back") + .context("reading the picture back") .and_then(|rgba| sink.write(rgba.data())), Work::Reopen(canvas) => { // Closed first, so the new one finds the device free. drop(sink); shared.reading.store(false, Ordering::Relaxed); - match open(canvas, shared) { + match open(scene, canvas, shared) { Ok(opened) => { sink = opened; continue; diff --git a/rs/moq-studio/src/vcam/pipewire.rs b/rs/moq-studio/src/vcam/pipewire.rs index 47b75cf..df52a7f 100644 --- a/rs/moq-studio/src/vcam/pipewire.rs +++ b/rs/moq-studio/src/vcam/pipewire.rs @@ -3,7 +3,7 @@ //! A `Video/Source` node with the `Camera` role is what PipeWire-aware apps //! (browsers through the camera portal, conferencing apps) list as a camera. //! It offers one format, the canvas's RGBA at its size and frame rate, so the -//! program reaches an app without a conversion here. +//! picture reaches an app without a conversion here. //! //! PipeWire objects stay on the thread that made them, so the node runs its //! own main loop. The `vcam` thread leaves the newest picture in a [`Slot`] @@ -23,16 +23,16 @@ use spa::param::video::{VideoFormat, VideoInfoRaw}; use super::{Shared, Sink}; -/// The node's name, which `pw-cli` and pickers that show ids print. +/// The program camera's node name, which `pw-cli` and pickers that show ids +/// print. A scene's camera adds `-`. const NAME: &str = "moq-studio"; -/// The id moq-video's camera capture lists the node by: `pipewire:` and its name. -pub const ID: &str = "pipewire:moq-studio"; - -/// What apps list the camera as. +/// What apps list the program camera as. A scene's camera adds `: `. const DESCRIPTION: &str = "MoQ Studio"; pub struct Node { + /// The node's name, for its capture id. + name: String, slot: Arc, wake: pw::channel::Sender, thread: Option>, @@ -55,21 +55,32 @@ enum Message { } impl Node { - /// Add a `size` camera node at `framerate` to the session's PipeWire. + /// Add a `size` camera node at `framerate`, for the program or the scene + /// called `scene`, to the session's PipeWire. /// /// Fails when there is no PipeWire session to add it to. - pub fn open(size: Size, framerate: u32, shared: Arc) -> anyhow::Result { + pub fn open(scene: Option<&str>, size: Size, framerate: u32, shared: Arc) -> anyhow::Result { + let (name, description) = match scene { + None => (NAME.to_string(), DESCRIPTION.to_string()), + Some(scene) => (format!("{NAME}-{scene}"), format!("{DESCRIPTION}: {scene}")), + }; let slot = Arc::new(Slot::default()); let (wake, messages) = pw::channel::channel(); let (ready, opened) = mpsc::channel(); let thread = { let slot = slot.clone(); let shared = shared.clone(); + let node = Props { + name: name.clone(), + description, + size, + framerate, + }; std::thread::Builder::new() .name("vcam-pipewire".into()) .spawn(move || { let ready = Rc::new(RefCell::new(Some(ready))); - if let Err(err) = run(size, framerate, slot, shared, messages, ready.clone()) + if let Err(err) = run(node, slot, shared, messages, ready.clone()) && let Some(ready) = ready.take() { let _ = ready.send(Err(format!("{err:#}"))); @@ -79,6 +90,7 @@ impl Node { }; match opened.recv() { Ok(Ok(())) => Ok(Self { + name, slot, wake, thread: Some(thread), @@ -96,6 +108,14 @@ impl Node { } } +impl Node { + /// The id moq-video's camera capture lists the node by: `pipewire:` and + /// its name. + pub fn id(&self) -> String { + format!("pipewire:{}", self.name) + } +} + impl Sink for Node { fn write(&mut self, rgba: &[u8]) -> anyhow::Result<()> { if self.shared.failed() { @@ -125,12 +145,20 @@ impl Drop for Node { /// Whoever waits for the node to be added, until it is or PipeWire refuses it. type Ready = Rc>>>>; +/// The node to add: what it is called, and the one picture format it offers. +struct Props { + name: String, + /// What apps list the camera as. + description: String, + size: Size, + framerate: u32, +} + /// Add the node and run it until it is dropped or PipeWire ends it, saying on /// `ready` whether PipeWire added it. A failure before then goes to `ready` /// rather than `shared`, so the caller can still fall back. fn run( - size: Size, - framerate: u32, + node: Props, slot: Arc, shared: Arc, messages: pw::channel::Receiver, @@ -140,16 +168,22 @@ fn run( let mainloop = pw::main_loop::MainLoopRc::new(None).context("PipeWire main loop")?; let context = pw::context::ContextRc::new(&mainloop, None).context("PipeWire context")?; let core = context.connect_rc(None).context("connecting to PipeWire")?; + let Props { + name, + description, + size, + framerate, + } = node; let stream = pw::stream::StreamRc::new( core, - NAME, + &name, pw::properties::properties! { *pw::keys::MEDIA_TYPE => "Video", *pw::keys::MEDIA_CATEGORY => "Capture", *pw::keys::MEDIA_CLASS => "Video/Source", *pw::keys::MEDIA_ROLE => "Camera", - *pw::keys::NODE_NAME => NAME, - *pw::keys::NODE_DESCRIPTION => DESCRIPTION, + *pw::keys::NODE_NAME => name.as_str(), + *pw::keys::NODE_DESCRIPTION => description.as_str(), }, ) .context("PipeWire stream")?; diff --git a/rs/moq-studio/src/vcam/v4l2.rs b/rs/moq-studio/src/vcam/v4l2.rs index 6d80dc2..9405fce 100644 --- a/rs/moq-studio/src/vcam/v4l2.rs +++ b/rs/moq-studio/src/vcam/v4l2.rs @@ -1,10 +1,13 @@ //! The fallback camera: an existing v4l2loopback device, written to as I420. //! //! v4l2loopback is a kernel module, so the studio never creates a device; it -//! takes the first one it finds that accepts the program's size. +//! takes the first one it finds that accepts the picture's size and that none +//! of its other cameras holds. +use std::collections::BTreeSet; use std::io::Write; use std::path::PathBuf; +use std::sync::Mutex; use anyhow::Context; use moq_v4l::video::Output; @@ -18,6 +21,10 @@ const DRIVER: &str = "v4l2 loopback"; /// I420, which every V4L2 camera reader takes. const FOURCC: &[u8; 4] = b"YU12"; +/// The devices this process's cameras write to. The kernel lets a second +/// writer set the same format, which would show two cameras' pictures in turn. +static HELD: Mutex> = Mutex::new(BTreeSet::new()); + pub struct Loopback { device: moq_v4l::Device, path: PathBuf, @@ -25,9 +32,13 @@ pub struct Loopback { } impl Loopback { - /// Open the first v4l2loopback device that takes `size` I420 frames. + /// Open the first v4l2loopback device that takes `size` I420 frames and + /// that no other camera holds. pub fn open(size: Size, framerate: u32) -> anyhow::Result { let mut refused = Vec::new(); + // Held across the search, so two cameras opening at once never pick + // the same device. + let mut held = HELD.lock().unwrap(); for path in devices()? { let Ok(device) = moq_v4l::Device::with_path(&path) else { continue; @@ -36,14 +47,21 @@ impl Loopback { Ok(caps) if caps.driver == DRIVER => {} _ => continue, } + if held.contains(&path) { + refused.push(format!("{}: another virtual camera holds it", path.display())); + continue; + } match configure(&device, size, framerate) { - Ok(()) => return Ok(Self { device, path, size }), + Ok(()) => { + held.insert(path.clone()); + return Ok(Self { device, path, size }); + } Err(err) => refused.push(format!("{}: {err:#}", path.display())), } } match refused.is_empty() { true => anyhow::bail!("no device; load the v4l2loopback module to create one"), - false => anyhow::bail!("no device takes the program ({})", refused.join("; ")), + false => anyhow::bail!("no device takes the picture ({})", refused.join("; ")), } } @@ -52,6 +70,12 @@ impl Loopback { } } +impl Drop for Loopback { + fn drop(&mut self) { + HELD.lock().unwrap().remove(&self.path); + } +} + impl Sink for Loopback { fn write(&mut self, rgba: &[u8]) -> anyhow::Result<()> { let i420 = moq_video::Surface::rgba(rgba, self.size)?.into_i420()?; @@ -118,7 +142,8 @@ mod tests { use super::*; - /// A V4L2 app reading the loopback device gets the picture written to it. + /// A V4L2 app reading the loopback device gets the picture written to it, + /// and another camera never writes to the same device. #[test] #[ignore = "needs a v4l2loopback device; run by just v4l2loopback"] fn a_v4l2_app_reads_the_program() { @@ -146,5 +171,12 @@ mod tests { y < 100 && u < 128 && v > 200, "the reader got y={y} u={u} v={v}, not red" ); + + if let Ok(other) = Loopback::open(size, 30) { + assert_ne!(other.path, loopback.path, "two cameras share a device"); + } + let path = loopback.path.clone(); + drop(loopback); + assert_eq!(Loopback::open(size, 30).expect("the device is free again").path, path); } }