diff --git a/quest/m0/README.md b/quest/m0/README.md index 872b7dc..e407c0f 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -5,7 +5,7 @@ A native, cross-platform streaming studio (`rs/moq-studio`) for creators and teams who rely on OBS today, plus the same app built to wasm. It composites cameras, screens, media, browser sources, and remote MoQ guests into scenes -on the GPU with moq-video and moq-audio, across several canvases at once +on the GPU with moq-video and moq-audio, publishing several scenes at once (landscape and portrait), and goes live to a MoQ relay and directly to RTMP, RTMPS, and WHIP ingests. @@ -37,7 +37,7 @@ Decided 2026-10-22 with the maintainer (/quest-plan): destination itself. It remuxes its own broadcast through a UI-free egress library that a hosted service can also run server-side. There is no gateway service and no moq-cli dependency, because a studio user does not - run a server. Destinations that take the same canvas and audio mix with + run a server. Destinations that take the same picture scene and audio scene with identical encode settings share one encoder. The studio picks codecs per destination: Opus by default, AAC where an ingest requires it, from platform encoders only. - **Extensibility:** first-party features plus browser sources (CEF @@ -93,13 +93,13 @@ repository. - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker - [ML video effects](/quest/m0/effects.md) - background removal and other ML effects, baked into their source's track - [Destinations](/quest/m0/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse -- [Multiple canvases](/quest/m0/multi-canvas.md) - landscape and portrait canvases composite at once from shared sources +- [Composable scenes](/quest/m0/scenes/README.md) - scenes nest scenes and the program, any scene publishes at its own size, and mixes become audio scenes - [The join page is live](/quest/m0/join-page-live.md) - `join.studio.moq.dev` serves the join page `--join-page` defaults to - [Clap-to-measure sync delay](/quest/m0/sync-clap.md) - a clap sets a microphone's delay against its camera - [Audio filters](/quest/m0/audio-filters.md) - per-source noise suppression, gate, compressor, limiter, and plugins, configured from the studio - [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops - [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository -- [Virtual camera](/quest/m0/vcam/README.md) - a canvas's program exposed as a system camera on Linux, Windows, and macOS +- [Virtual camera](/quest/m0/vcam/README.md) - the program or a published scene exposed as a system camera on Linux, Windows, and macOS - [Browser sources without an X display](/quest/m0/cef-headless.md) - CEF runs headless, so Wayland sessions and tests need no X server - [Signing identities are available](/quest/m0/signing.md) - the Apple and Windows code-signing credentials packaging and the macOS camera need - [Packaging](/quest/m0/packaging.md) - signed installers for macOS and Windows and a Flatpak for Linux diff --git a/quest/m0/destinations.md b/quest/m0/destinations.md index dfd3034..8230f8c 100644 --- a/quest/m0/destinations.md +++ b/quest/m0/destinations.md @@ -4,7 +4,7 @@ A creator adds any number of RTMP, RTMPS, and WHIP destinations alongside MoQ, each with its own codec and bitrate, and goes live to all of them at -once. Destinations that take the same canvas and audio mix with identical +once. Destinations that take the same picture scene and audio scene with identical encode settings share one encoder. Each destination's status and errors show in the UI. @@ -54,7 +54,7 @@ Decided 2026-10-22 with the maintainer: - **Benchmarks:** - Sweep concurrent broadcasts against destinations per broadcast in the library. - - Sweep distinct encoder-sharing keys (canvas, mix, and settings) against + - Sweep distinct encoder-sharing keys (picture scene, audio scene, and settings) against destinations per key in the studio. Finding a destination's encoder must not scan every encoder group, and identical keys must never spawn a duplicate encoder. diff --git a/quest/m0/multi-canvas.md b/quest/m0/multi-canvas.md deleted file mode 100644 index e7ab30d..0000000 --- a/quest/m0/multi-canvas.md +++ /dev/null @@ -1,27 +0,0 @@ -# [M] Multiple canvases - -## Goal - -The studio composites several canvases at once from shared sources, such as -a 1920x1080 landscape canvas and a 1080x1920 portrait one. Each canvas has -its own layout per scene, its own preview, and its own outputs. A source -captures or decodes once however many canvases use it. - -## Plan - -Further canvases, their layouts and top layers, linked scene switching, and a -broadcast per canvas at `<--broadcast>/` landed in -[moq-dev/studio#42](https://github.com/moq-dev/studio/pull/42). What remains: - -- Unlinking: a canvas the creator unlinks keeps its own preview and program - scene, so a scene change no longer cuts it with the others. Transitions are - per canvas, so an unlinked canvas takes its own. -- Outputs per canvas: a recording, and each destination, picks the canvas it - carries, as each picks its mix. Only the main canvas records today. -- Health per canvas: the Health panel grades each canvas's broadcast, with - viewer feedback beneath it, not only the main one's. -- Preview cost: a canvas no panel shows skips drawing its preview, so an - unseen canvas costs only its program. - -Per-canvas cost must grow only with the sources that canvas uses, never with -the whole source table; `compositor::bench` measures it. diff --git a/quest/m0/scenes/README.md b/quest/m0/scenes/README.md new file mode 100644 index 0000000..db94bf4 --- /dev/null +++ b/quest/m0/scenes/README.md @@ -0,0 +1,56 @@ +# Composable scenes + +## Goal + +Scenes form a graph. A scene can draw another scene, or the live program, as +a source. Any scene can be published as its own broadcast at its own size, and +the old audio mixes are audio-only scenes that nest the same way. This +replaces separate canvases: a portrait broadcast is a published 1080x1920 +scene that draws the program, cropped, with its own overlays. + +This epic's own work is an end-to-end test: one studio publishes the program +at `<--broadcast>` and a portrait scene drawing it at +`<--broadcast>/`, each carrying its picked audio scene, and a viewer of +each sees and hears the same cut at the same moment. + +## Plan + +Decided 2026-11-05 with the maintainer (/quest-plan), replacing the canvases +of [moq-dev/studio#42](https://github.com/moq-dev/studio/pull/42) and closing +[moq-dev/studio#54](https://github.com/moq-dev/studio/pull/54): + +- ✅ **Published scenes + scene source** rather than canvases with a layout + per scene and linked or unlinked switching. One concept (a scene) instead of + two, and no unlinking. Rejected: published scenes that never follow cuts; + keeping canvases. +- ✅ **A scene source draws any scene, or `program`** (whatever is on air, + transitions included), scaled and cropped like any picture. Rejected: a + program-only source. +- ✅ **The program stays one switcher at `<--broadcast>`**: Preview, Program + and Cut work as today. Other published scenes are always live at + `<--broadcast>/` and follow cuts only through a scene source. + Rejected: every published scene switchable on its own. +- ✅ **Every scene has a size**, defaulting to the program's. The switcher + offers only scenes of the program's size. +- ✅ **Mixes become audio scenes** that hold only sounds and nest, so a + "Mics" scene can feed "Stream" and "Headphones". Audio stays independent of + cuts, as today. Each published output pairs a picture scene with an audio + scene, defaulting to the one on program. Rejected: scenes carrying their own + audio, so a cut switches audio; keeping `[mixes]` separate. +- ✅ **UI:** the Scenes panel lists picture and audio scenes, tagged by kind. + The Audio panel stays a mixer (fader, pan, meter, and sends to audio scenes) + for live work, as pro switchers do. A graph view of the same data is + [its own quest](/quest/m1/scene-graph.md). +- Nothing is released, so the scene file breaks freely: `[canvases]` and + `[mixes]` go away, with no migration. + +Carried over from #54: Health grades each published broadcast; a recording +picks a picture scene and an audio scene; a scene no panel shows and no output +reads skips drawing. Per-scene cost grows only with what that scene draws; +`compositor::bench` measures it. + +## Required + +- [Scene source](/quest/m0/scenes/scene-source.md) - a scene draws another scene, or the live program, as a source +- [Published scenes](/quest/m0/scenes/published.md) - any scene publishes as its own broadcast at its own size, replacing canvases +- [Audio scenes](/quest/m0/scenes/audio.md) - mixes become audio-only scenes that nest diff --git a/quest/m0/scenes/audio.md b/quest/m0/scenes/audio.md new file mode 100644 index 0000000..119f86e --- /dev/null +++ b/quest/m0/scenes/audio.md @@ -0,0 +1,29 @@ +# [M] Audio scenes + +## Goal + +The audio mixes become audio-only scenes. An audio scene holds sounds +(sources' audio, with gain, pan, mute, and delay) and other audio scenes, so a +"Mics" scene can feed both "Stream" and "Headphones". **On program** picks the +audio scene the broadcast carries; each published scene and each recording +picks one. Audio stays independent of cuts. + +## Plan + +Decided 2026-11-05 with the maintainer (/quest-plan); see the +[epic](/quest/m0/scenes/README.md). + +- Replaces `[mixes]` and `[audio.]` in the scene file with no + migration; nothing is released. +- An audio scene cannot contain itself, directly or through others. +- The Audio panel stays a mixer: a strip per sound with fader, pan, meter, + and a send to each audio scene. Listen and On program work as today. +- The guest return feed (a mix-minus) and the sync delay keep working; a + guest hears the audio scene on program without their own sound. +- Switching or editing an audio scene never drops or repeats audio, as with + mixes today. +- `README.md` documents audio scenes. + +## Required + +- [Scene source](/quest/m0/scenes/scene-source.md) - shares the scene graph and its cycle check diff --git a/quest/m0/scenes/published.md b/quest/m0/scenes/published.md new file mode 100644 index 0000000..ef6b281 --- /dev/null +++ b/quest/m0/scenes/published.md @@ -0,0 +1,32 @@ +# [L] Published scenes + +## Goal + +Any scene can be published: ticked **Publish**, it goes live as its own +broadcast at `<--broadcast>/`, at its own size, with the audio scene it +picks (the one on program by default). This replaces canvases: further +canvases, their per-scene layouts, and the canvas picker are removed, and a +portrait broadcast is a published portrait scene that draws the program +through a [scene source](/quest/m0/scenes/scene-source.md). + +## Plan + +Decided 2026-11-05 with the maintainer (/quest-plan); see the +[epic](/quest/m0/scenes/README.md). + +- Every scene has a size, defaulting to the program's; the switcher offers + only scenes of the program's size. +- The program keeps one switcher at `<--broadcast>`. A published scene is + always live and never switched. +- Each published scene gets the program's two sizes (full and half) and the + bandwidth grant shares across all of them, as further canvases did. +- Carried from [moq-dev/studio#54](https://github.com/moq-dev/studio/pull/54): + Health grades each published broadcast with its viewer feedback; a + recording picks a picture scene and an audio scene; a scene no panel shows + and no output reads skips drawing. +- Remove `[canvases]` from the scene file with no migration; nothing is + released. `README.md` documents published scenes. + +## Required + +- [Scene source](/quest/m0/scenes/scene-source.md) - a portrait scene follows the program through it diff --git a/quest/m0/scenes/scene-source.md b/quest/m0/scenes/scene-source.md new file mode 100644 index 0000000..ab40477 --- /dev/null +++ b/quest/m0/scenes/scene-source.md @@ -0,0 +1,23 @@ +# [M] Scene source + +## Goal + +A new source kind, `scene`, draws another scene, or `program` (whatever is on +air, including a transition in progress), as a picture that an item places, +scales, crops, and filters like any other. A scene cannot contain itself, +directly or through other scenes; such an edit is refused with an error. + +## Plan + +Decided 2026-11-05 with the maintainer (/quest-plan); see the +[epic](/quest/m0/scenes/README.md) for the model. + +- A nested scene renders at its own size into a texture the compositor + samples, once per frame however many items draw it, and only while + something that is drawn reads it. +- `program` reads the frame the program output draws, so a scene drawing it + shows the same cut on the same frame, with no extra composite. +- Keyframes, stingers, and restarts in a nested scene start when the scene + that draws it goes on program, as its items would. +- Measure frame time and GPU copies with and without a nested scene. +- `README.md` documents the source kind. diff --git a/quest/m0/vcam/README.md b/quest/m0/vcam/README.md index f77993e..a442514 100644 --- a/quest/m0/vcam/README.md +++ b/quest/m0/vcam/README.md @@ -2,9 +2,9 @@ ## Goal -The studio exposes a canvas's program as a system camera, so video-call and +The studio exposes the program or a published scene as a system camera, so video-call and conferencing apps, which offer no ingest API, can use it. The studio has one -toggle per canvas, and each OS backend lands on its own. +toggle per published scene, and each OS backend lands on its own. ## Plan diff --git a/quest/m0/vcam/linux.md b/quest/m0/vcam/linux.md index d1190d3..8a4b008 100644 --- a/quest/m0/vcam/linux.md +++ b/quest/m0/vcam/linux.md @@ -2,7 +2,7 @@ ## Goal -The studio publishes a canvas's program as a PipeWire video node that +The studio publishes the program or a published scene as a PipeWire video node that PipeWire-aware apps (browsers and conferencing apps) list as a camera. It falls back to an existing v4l2loopback device when one is present. The studio's Devices panel leaves out its own virtual camera, so picking it diff --git a/quest/m0/vcam/macos.md b/quest/m0/vcam/macos.md index d1ad1b8..4a26514 100644 --- a/quest/m0/vcam/macos.md +++ b/quest/m0/vcam/macos.md @@ -3,7 +3,7 @@ ## Goal The macOS app bundle carries a CoreMediaIO Camera Extension. Once the user -approves it, apps list a camera that carries a canvas's program. An app +approves it, apps list a camera that carries the program or a published scene. An app update installs the new extension without a reboot. ## Required diff --git a/quest/m0/vcam/windows.md b/quest/m0/vcam/windows.md index 66ac249..8b0cc04 100644 --- a/quest/m0/vcam/windows.md +++ b/quest/m0/vcam/windows.md @@ -2,7 +2,7 @@ ## Goal -On Windows 11, the studio registers a canvas's program as a session-lifetime +On Windows 11, the studio registers the program or a published scene as a session-lifetime camera through `MFCreateVirtualCamera`, backed by a software camera source. Apps list it as a camera while the studio runs. diff --git a/quest/m1/README.md b/quest/m1/README.md index a739aa7..fab4c79 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -28,3 +28,4 @@ than opening an m2 for them. - [Overlay assets](/quest/m1/overlay-assets.md) - overlays use fonts and images from data: URLs or files beside the scene - [Hotkeys and editor preferences](/quest/m1/hotkeys.md) - system-wide hotkeys, arrow nudge, aspect lock, and persisted editor settings - [Isolated guest recordings](/quest/m1/guest-recording.md) - record-each-source also writes each guest's tracks to its own file +- [Scene graph view](/quest/m1/scene-graph.md) - a node view of picture and audio scenes, their routing, volume, pan, and waveforms diff --git a/quest/m1/scene-graph.md b/quest/m1/scene-graph.md new file mode 100644 index 0000000..a6c3735 --- /dev/null +++ b/quest/m1/scene-graph.md @@ -0,0 +1,19 @@ +# [M] Scene graph view + +## Goal + +A Graph panel shows the scene graph: picture and audio scenes as nodes, the +sources and scenes each one draws or sends to as edges, with each sound's +volume, pan, and a live waveform on its node. Edges and node settings can be +edited there, as in the Scenes and Audio panels. + +## Plan + +Decided 2026-11-05 with the maintainer (/quest-plan): the live controls stay +a mixer, as pro switchers keep them, and this view is for building and +inspecting routing, as node tools (TouchDesigner, Fusion) do. It reads and +edits the same data, so it adds no second model. + +## Required + +- [Composable scenes](/quest/m0/scenes/README.md) - the graph this view shows