feat(scene): remove a source's background with a person segmentation model

Cargo.toml

@@ -88,6 +88,18 @@ opt-level = 3
 [profile.dev.package.rubato]
 opt-level = 3
 
+# Background removal's model, which runs on every picture of a source and is
+# ten times slower unoptimized: a camera would cut out a few pictures a
+# second rather than every one.
+[profile.dev.package.tract-core]
+opt-level = 3
+
+[profile.dev.package.tract-data]
+opt-level = 3
+
+[profile.dev.package.tract-linalg]
+opt-level = 3
+
 [profile.release]
 lto = "fat"
 codegen-units = 1

README.md

@@ -101,7 +101,7 @@ Every panel is a tab you can drag, split, or stack.
 | Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track, and **Reload** restarts its page. A media file reads as ended once it plays to its end without looping, **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again. |
 | Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. |
 | Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. |
-| Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. |
+| Filters | Remove the selected source's background, crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. |
 | Keyframes | Animate the selected item: add, time, ease, and remove its keyframes, and set how far each moves, resizes, crops, and filters it (see [Keyframes](#keyframes)). **Play** plays the preview scene's keyframes again. |
 | Audio | Add, rename (double-click), and remove mixes, pick the one on program and the one to listen to and on which output, and set each audio source's gain, mute, delay, and the mixes it feeds, with a meter on every mix and source. |
 | Output | Start and stop recording, pick the mix it records, set the canvas size and frame rate, add, resize, and remove further canvases, and see the broadcast path, whether the program is encoding, the encoder sessions open, and each canvas's renditions' encoder work. |
@@ -543,6 +543,7 @@ kind = "lut"
 path = "teal-orange.cube"
 
 [video.camera]
+background = "remove" # cut the person out, leaving the rest transparent
 crop = { left = 0.1, right = 0.1 } # trim 10% of the picture off each side
 filters = [
     { kind = "chroma_key", color = "#00ff00", similarity = 0.4, smoothness = 0.08, spill = 0.1 },
@@ -598,8 +599,16 @@ without `[mixes]`, the collection has one mix, `Program`.
 `[audio.<source>]` sets a source's linear `gain`, `muted`, the mixes it is
 left out of (`exclude`), and its `delay`.
 
-`[video.<source>]` sets a source's `crop` and `filters`, wherever it is shown,
-and an item's `crop`, `filters`, and `blend` its own. A `crop` trims a fraction
+`[video.<source>]` sets a source's `background`, `crop`, and `filters`,
+wherever it is shown, and an item's `crop`, `filters`, and `blend` its own.
+`background = "remove"` cuts the person out of a camera, display, window,
+image, media, broadcast, browser, or replay source's picture before its crop
+and filters, leaving the rest transparent. A segmentation model finds them in
+each picture on the CPU, so each picture shows only once its cut-out is
+ready, a frame or more late, and a model that falls behind skips pictures
+rather than queue them. It is MediaPipe's selfie segmentation model, made for
+one or two people near the camera. A clone shows its source's cut-out, so it
+cannot remove a background of its own. A `crop` trims a fraction
 (0 to 1) of the picture off its `left`, `top`, `right`, and `bottom`, opposite
 sides together trimming less than the whole, and stretches the rest over the
 item; an item's crop trims what its source's
@@ -717,7 +726,9 @@ changes size or color space moves to a new track, which its catalog entry
 follows. Images and colors keep their picture once, as the image file and as a
 one-pixel PNG. Removing or editing a source ends its track. A track carries the
 plain capture, before any filter, and a clone has none, since it shows its
-source's capture. A browser source or an overlay has no track of its own, and a
+source's capture. A removed background stays in the track, since H.264 carries
+no transparency, so a replay of the source shows it unless the replay source
+removes it too. A browser source or an overlay has no track of its own, and a
 broadcast source is already its publisher's.
 
 A microphone, system audio, or media file's sound encodes its track as Opus,

quest/m0/README.md

@@ -91,7 +91,6 @@ repository.
 - [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux
 - [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux
 - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker
-- [ML video effects](/quest/m0/effects.md) - background removal and other ML effects, baked into their source's track
 - [Destinations](/quest/m0/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse
 - [Composable scenes](/quest/m0/scenes/README.md) - scenes nest scenes and the program, any scene publishes at its own size, and mixes become audio scenes
 - [The join page is live](/quest/m0/join-page-live.md) - `join.studio.moq.dev` serves the join page `--join-page` defaults to

quest/m2/publish-independently.md

@@ -25,6 +25,11 @@ source among those on program (`Publish::set_on_program` in
 Moved to m2 on 2026-11-05 (maintainer, /quest-plan): no service places
 sources itself yet, so it waits behind m0 and m1.
 
+A source with its background removed (`background = "remove"`) keeps the
+background in its track, since H.264 carries no transparency. Publishing one
+independently needs its cut-out in the track, such as moq-video encoding
+alpha, or the option refused for it.
+
 Low priority: this is for services that move webcams and overlays themselves,
 and none exists yet. The option lives on the source, in the scene file, so it
 travels with the scene. The option and its effect on the program change what a

rs/moq-scene/Cargo.toml

@@ -74,6 +74,9 @@ thiserror = { workspace = true }
 tokio = { workspace = true, features = ["macros", "sync", "time"] }
 toml = { workspace = true }
 tracing = { workspace = true }
+# Runs background removal's person segmentation model on the CPU. Pure Rust, so
+# it builds wherever the studio does with no runtime library to download.
+tract-onnx = { version = "0.23.10", default-features = false }
 url = { workspace = true }
 vello = "0.11"
 # Decompresses an overlay's WOFF and WOFF2 fonts to check them. Already in the

rs/moq-scene/src/browser/run.rs

@@ -64,10 +64,7 @@ pub(crate) async fn run(
 	drop(opened);
 	// A stopped page leaves the canvas rather than freezing on its last
 	// picture, which for an alert or a ticker would show stale news.
-	*picture.lock().unwrap() = Some(Picture::Rgba {
-		size: moq_video::Size::new(1, 1),
-		pixels: vec![0; 4],
-	});
+	picture.lock().unwrap().clear();
 	Err(Error::Browser(reason))
 }
 
@@ -167,7 +164,7 @@ impl Routes {
 			Ok(None) => return,
 			Err(reason) => return self.end(id, reason),
 		};
-		*route.picture.lock().unwrap() = Some(Picture::Texture(texture));
+		route.picture.lock().unwrap().show(Picture::Texture(texture));
 		if !std::mem::replace(&mut route.live, true)
 			&& let Ok(mut status) = route.status.write()
 		{

rs/moq-scene/src/collection.rs

@@ -315,6 +315,12 @@ impl Collection {
 					"video settings for `{name}`, which is not a source with a picture"
 				)));
 			}
+			if video.background == Background::Remove && !self.sources[name].removes_background() {
+				return Err(Error::Invalid(format!(
+					"`{name}` cannot have its background removed: only a camera, display, window, image, media, \
+					 broadcast, browser, or replay source can, and a clone shows its source's"
+				)));
+			}
 			let on = format!("`{name}`");
 			video
 				.crop
@@ -1335,6 +1341,10 @@ pub(crate) fn validate_filters(filters: &[Filter], on: &str) -> Result<(), Error
 #[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
 #[serde(deny_unknown_fields)]
 pub struct Video {
+	/// What happens behind the person in the picture, before the crop and
+	/// every filter.
+	#[serde(default, skip_serializing_if = "Background::is_kept")]
+	pub background: Background,
 	/// Trimmed before every filter and each item's own crop.
 	#[serde(default, skip_serializing_if = "Crop::is_none")]
 	pub crop: Crop,
@@ -1343,6 +1353,27 @@ pub struct Video {
 	pub filters: Vec<Filter>,
 }
 
+/// What happens to the background behind a person in a source's picture.
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum Background {
+	/// Shown as captured.
+	#[default]
+	Keep,
+	/// Made transparent, leaving the person a segmentation model finds in each
+	/// picture. Each picture draws once its own cut-out is ready, a frame or
+	/// more late, skipping pictures while the model is behind. The
+	/// source's own track keeps the background, since its encoding carries no
+	/// transparency.
+	Remove,
+}
+
+impl Background {
+	fn is_kept(&self) -> bool {
+		*self == Self::Keep
+	}
+}
+
 /// A rectangle in canvas pixels, from the top-left corner.
 #[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)]
 #[serde(deny_unknown_fields)]
@@ -1513,6 +1544,22 @@ impl Source {
 		!matches!(self, Self::Clone { .. })
 	}
 
+	/// Whether the source's picture can have its background removed: a
+	/// picture of its own that may show a person.
+	pub fn removes_background(&self) -> bool {
+		matches!(
+			self,
+			Self::Camera { .. }
+				| Self::Display { .. }
+				| Self::Window { .. }
+				| Self::Image { .. }
+				| Self::Media { .. }
+				| Self::Broadcast { .. }
+				| Self::Browser { .. }
+				| Self::Replay
+		)
+	}
+
 	/// Whether the source feeds the program audio.
 	pub fn has_audio(&self) -> bool {
 		matches!(
@@ -2103,6 +2150,42 @@ path = "sting.webm"
 		}
 	}
 
+	/// A background is removed only from a picture of the source's own that
+	/// may show a person.
+	#[test]
+	fn removes_a_background_only_from_a_picture_of_its_own() {
+		let text = r#"
+version = 1
+
+[canvas]
+width = 1280
+height = 720
+framerate = 30
+
+[sources.cam]
+kind = "camera"
+
+[video.cam]
+background = "remove"
+"#;
+		let collection = Collection::from_toml(text).unwrap();
+		assert_eq!(collection.video["cam"].background, Background::Remove);
+		assert_eq!(
+			Collection::from_toml(&collection.to_toml().unwrap()).unwrap(),
+			collection
+		);
+
+		let color = Source::Color {
+			color: Rgba([0, 0, 0, 255]),
+		};
+		for source in [Source::Clone { source: "cam".into() }, color] {
+			let mut collection = collection.clone();
+			collection.sources.insert("other".into(), source.clone());
+			collection.video.insert("other".into(), collection.video["cam"].clone());
+			assert!(matches!(collection.validate(), Err(Error::Invalid(_))), "{source:?}");
+		}
+	}
+
 	#[test]
 	fn refuses_a_filter_out_of_range() {
 		let red = Rgba([255, 0, 0, 255]);