diff --git a/quest/README.md b/quest/README.md index cbe99bd..72961ce 100644 --- a/quest/README.md +++ b/quest/README.md @@ -19,3 +19,4 @@ there first. A quest here that needs it waits on a pin quest, which ## Required - [m0: the studio](/quest/m0/README.md) - the native and wasm studio for creators who rely on OBS today +- [m1: viewers compose](/quest/m1/README.md) - viewers and a server compose a broadcast from its sources and a layout track diff --git a/quest/m0/README.md b/quest/m0/README.md index 7ae2afb..8049c11 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -60,7 +60,7 @@ cameras, packaging, signing) that one monorepo no longer pays off. - **Pin:** moq is a `moq` git submodule whose crates are path dependencies, as in moq.pro, so the studio compiles against one moq checkout. - **moq-scene** moved here with the app, its only consumer. The - [viewer-side composition](/quest/m0/compose/README.md) epic moved whole: + [viewer-side composition](/quest/m1/compose/README.md) epic (now m1) moved whole: browser composition ships as a package from this repository layered on `@moq/watch`, and the server composite is a headless binary here rather than a moq-cli subcommand, so moq never depends on the studio. @@ -72,6 +72,15 @@ cameras, packaging, signing) that one monorepo no longer pays off. waits on a pin quest here, which `just upgrade-moq` advances. This replaces "never ahead of the studio UI that consumes them" above. +Changed 2026-11-04 with the maintainer: the program is how a studio +broadcast reaches viewers, not the fallback. Sources are not published to the +relay by default (decided 2026-11-03 on +[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), because +each encoding costs an encoder session and consumer GPUs have few, so the +[viewer-side composition](/quest/m1/compose/README.md) epic moved to m1. +Overlays and animation render into the program here, and travel to viewers +in the layout track only once that epic lands. + Dropped from the first scoping round: relay fan-out ("encode once, the relay restreams") and its upload savings. Direct push with shared encoders matches OBS multistream, and the hosted version is product work outside this @@ -81,7 +90,7 @@ repository. - [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux - [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux -- [Viewer-side composition](/quest/m0/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers, with a server-composed program as the fallback +- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks, for isolated recording and replay - [Broadcast sources play in sync](/quest/m0/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay - [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker - [Live encode control](/quest/m0/encode-control.md) - simulcast rungs that follow the bandwidth grant, live bitrate and resolution, keyframes on request @@ -100,6 +109,7 @@ repository. - [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops - [Browser sources keep a profile](/quest/m0/browser-profile.md) - one persistent profile, so logins survive restarts, and a reload action - [Browser source popups](/quest/m0/browser-popups.md) - dropdowns and other in-page popups draw and take input +- [Published independently](/quest/m0/publish-independently.md) - a source opts out of the program and publishes its own track for services that place it themselves - [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository - [Virtual camera](/quest/m0/vcam/README.md) - a canvas's program exposed as a system camera on Linux, Windows, and macOS - [Signing identities are available](/quest/m0/signing.md) - the Apple and Windows code-signing credentials packaging and the macOS camera need diff --git a/quest/m0/animation.md b/quest/m0/animation.md index c851ab7..991968c 100644 --- a/quest/m0/animation.md +++ b/quest/m0/animation.md @@ -5,19 +5,21 @@ Scene changes in `moq-studio` use transitions beyond a cut and fade: move (items glide between their positions in the two scenes), slide, wipe, and stinger (a video clip with a cut point). Any item property can be keyframed -with easing, so an item can fly in or a camera can push in, and the same -animation plays in every composition host. +with easing, so an item can fly in or a camera can push in. ## Plan Move is OBS's most-installed plugin (Move Transition), and stingers are -standard in broadcast. Animate in moq-scene on the media clock and carry the -transition and keyframes in the layout track, so viewers compose them too. +standard in broadcast. Animate in moq-scene on the media clock, so the program, a recording, and a +replay match. Carrying transitions and keyframes in the layout track, for +viewers to compose, waits for the m1 +[composition epic](/quest/m1/compose/README.md) (changed 2026-11-04, +maintainer: the program comes first). Overlays animate through their own CSS. No Lottie runtime for now (decided 2026-10-26): the open ones either lack text (velato) or add a C++ build (ThorVG). A stinger clip is a media source like any other. Transitions and keyframes change through moq-scene's `Edit`, so they undo like any edit. -## Required +## Related -- [Layout track](/quest/m0/compose/layout.md) - where transitions and keyframes travel +- [Layout track](/quest/m1/compose/layout.md) - where transitions and keyframes travel to viewers in m1 diff --git a/quest/m0/audio-source-tracks.md b/quest/m0/audio-source-tracks.md new file mode 100644 index 0000000..ec78d62 --- /dev/null +++ b/quest/m0/audio-source-tracks.md @@ -0,0 +1,44 @@ +# [S] Audio source tracks + +## Goal + +Each microphone and system-audio source in a `moq-studio` scene has its own +capture as an Opus track in the studio's broadcast, listed in that source's +catalog under the studio catalog's `dev.moq.studio` section, beside the +program, as the picture sources already are. An isolated recording and a +replay of an audio source read that track. + +## Plan + +Cameras, displays, windows, images, and colors already publish this way +(`rs/moq-scene/src/publish.rs`): each source's renditions sit in a nested hang +catalog keyed by source name, so the root `video` and `audio` sections stay +the program's. An audio source adds an `audio` rendition to its own nested +catalog the same way. + +moq-audio's capture Producer/Consumer split (`capture::Producer`) feeds both +the bus and an encoder from one device open, rather than opening each device +twice. +Publish the capture after the source's filters but before its gain and mute +(decided 2026-11-02, maintainer): a recording or replay holds filtered audio, +as video bakes in its ML effects, and the levels stay the program's. + +Changed 2026-11-03 (maintainer, on +[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)): a source's +own track encodes only while something in the studio uses it, and reaches +the relay only when the creator publishes the source independently +([Published independently](/quest/m0/publish-independently.md)). A source +on program counts as used (decided 2026-11-04, maintainer), so a replay finds +its past from when it went on: a picture source while the program scene shows +it, and an audio source while it feeds the program mix. A source published +independently counts as used too, since its track is what it publishes. +Otherwise a source encodes only while an isolated recording or a running +replay reads it, and replaying a source with no history is refused with the +reason. +This replaces the earlier always-on decision (2026-11-01, on +[moq-dev/moq#7554](https://github.com/moq-dev/moq/pull/7554)), because +encoding every source costs encoder sessions a consumer machine does not +have; an audio source follows the picture sources' rule. + +Track names stay stable for a source's life, and an edited source that +changes what it shows gets a new name. diff --git a/quest/m0/compose/sources.md b/quest/m0/compose/sources.md deleted file mode 100644 index 77dd74c..0000000 --- a/quest/m0/compose/sources.md +++ /dev/null @@ -1,37 +0,0 @@ -# [S] Audio source tracks - -## Goal - -Each microphone and system-audio source in a `moq-studio` scene publishes its -own capture as an Opus track in the studio's broadcast, listed in that -source's catalog under the studio catalog's `dev.moq.studio` section, beside -the program. Like the picture sources, it encodes while it is in the -collection, whatever the subscriber demand, and uploads only while something -subscribes to it. - -## Plan - -Cameras, displays, windows, images, and colors already publish this way -(`rs/moq-scene/src/publish.rs`): each source's renditions sit in a nested hang -catalog keyed by source name, so the root `video` and `audio` sections stay -the program's. An audio source adds an `audio` rendition to its own nested -catalog the same way. - -moq-audio's capture Producer/Consumer split (`capture::Producer`) feeds both -the bus and an encoder from one device open, rather than opening each device -twice. -Publish the capture after the source's filters but before its gain and mute -(decided 2026-11-02, maintainer): viewers hear filtered audio, as video bakes -in its ML effects, and a composing viewer mixes the levels itself. - -Encode like every first-party publisher (decided 2026-10-28, from -[#7101](https://github.com/moq-dev/moq/issues/7101)): an unwatched source -still encodes, so a late viewer or a replay -finds its recent past, but costs no upload. - -✅ 2026-11-01 (maintainer, on -[moq-dev/moq#7554](https://github.com/moq-dev/moq/pull/7554)): sources stay -always-on. Rejected: sources on demand, which would make replaying a source -need an armed local subscriber again. -Track names stay stable for a source's life, and an edited source that -changes what it shows gets a new name. diff --git a/quest/m0/effects.md b/quest/m0/effects.md index db13f87..04f5f82 100644 --- a/quest/m0/effects.md +++ b/quest/m0/effects.md @@ -14,7 +14,7 @@ composition host renders with the same WGSL and the layout carries by name and settings. A model cannot meet that parity rule: viewers would each need the model and a GPU to run it, and inference differs across backends. So an ML effect is baked into its source's track per the -[composition epic](/quest/m0/compose/README.md): it belongs to a source, never +[composition epic](/quest/m1/compose/README.md): it belongs to a source, never to one scene item, and sits before the source's crop and filters, as the encoder in `rs/moq-scene/src/publish.rs` sees the capture. diff --git a/quest/m0/overlays.md b/quest/m0/overlays.md index c9f96be..9cd63c3 100644 --- a/quest/m0/overlays.md +++ b/quest/m0/overlays.md @@ -3,10 +3,9 @@ ## Goal Titles, lower thirds, tickers, and other graphics in `moq-studio` are written -as HTML and CSS without JavaScript, travel in the layout track, and render in -all three composition hosts: through Blitz in the studio and the headless compositor, -and as real DOM over the video in the browser. CSS animations and transitions -run on the media clock, so an overlay animates on the same frame everywhere. +as HTML and CSS without JavaScript and render through Blitz into the program. +CSS animations and transitions run on the media clock, so an overlay animates +on the same frame in the program, a recording, and a replay. ## Plan @@ -20,11 +19,11 @@ Decided 2026-10-26 (maintainer): JavaScript engine, and has no clock injection. - **Browser:** the DOM layered over the composited video, so no HTML-in-Canvas and it works in every browser today. -- **Subset:** the layout track accepts only the CSS Blitz renders and refuses +- **Subset:** an overlay accepts only the CSS Blitz renders and refuses the rest at setup with the property named. Blitz is beta (no blend modes or `text-shadow`, filters limited to blur and drop-shadow), and the subset grows as it does. -- **Parity:** layout parity per the [composition epic](/quest/m0/compose/README.md); +- **Parity:** layout parity per the [composition epic](/quest/m1/compose/README.md); the fixture compares boxes and timing, and pixels only loosely. An overlay is hostile input to every viewer, so it fetches nothing from the @@ -37,6 +36,12 @@ upstream that if it cannot. Arbitrary web pages and JavaScript graphics stay browser sources (`moq_scene::browser`), baked into their own track. -## Required +Changed 2026-11-04 (maintainer): the program comes first, so this quest +renders overlays in the studio only. Carrying them in the layout track, the +browser's DOM rendering, and the headless compositor wait for the m1 +[composition epic](/quest/m1/compose/README.md); keep the overlay model free +of studio-only state so that epic can serialize it. -- [Layout track](/quest/m0/compose/layout.md) - where overlays travel +## Related + +- [Layout track](/quest/m1/compose/layout.md) - where overlays travel to viewers in m1 diff --git a/quest/m0/publish-independently.md b/quest/m0/publish-independently.md new file mode 100644 index 0000000..7f20082 --- /dev/null +++ b/quest/m0/publish-independently.md @@ -0,0 +1,29 @@ +# [S] Published independently + +## Goal + +A source in `moq-studio` has a "Published independently" option, off by +default. Ticked, the source's own track is published to the relay for viewers +and services to fetch, and the source is left out of the composited program, +so a service can place a webcam or overlay itself. Unticked, the source +reaches viewers only through the program. + +## Plan + +Decided 2026-11-03 (maintainer, on +[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)): sources are +not published to the relay by default, since every encoding costs an encoder +session and consumer GPUs have few. A source published independently gets one +rendition, as sources do in the encode ladder, while the program keeps its +two sizes. It counts as used, so it encodes for as long as it is published +([Audio source tracks](/quest/m0/audio-source-tracks.md) states the rule). + +Low priority: this is for services that move webcams and overlays themselves, +and none exists yet. The option lives on the source, in the scene file, so it +travels with the scene. The option and its effect on the program change what a +user does, so `README.md` documents them. + +## Related + +- [Live encode control](/quest/m0/encode-control.md) - the encode ladder that makes source tracks local by default +- [Viewer-side composition](/quest/m1/compose/README.md) - the m1 epic that composes from independently published sources diff --git a/quest/m1/README.md b/quest/m1/README.md new file mode 100644 index 0000000..5f8a276 --- /dev/null +++ b/quest/m1/README.md @@ -0,0 +1,20 @@ +# m1: viewers compose + +## Goal + +A studio broadcast can be composed by its viewers and by a server from the +sources' own tracks and a layout track, matching the program the studio +composites, so a viewer can toggle and prioritize elements. + +## Plan + +Moved out of m0 on 2026-11-04 (maintainer): the program is how a studio +broadcast reaches viewers, and sources are not published to the relay by +default (decided 2026-11-03 on +[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), since +encoding every source for viewers costs more encoder sessions than a consumer +GPU has. Viewer-side composition waits until m0 ships. + +## Required + +- [Viewer-side composition](/quest/m1/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers and by a server diff --git a/quest/m0/compose/README.md b/quest/m1/compose/README.md similarity index 64% rename from quest/m0/compose/README.md rename to quest/m1/compose/README.md index 1bc9cc1..a801760 100644 --- a/quest/m0/compose/README.md +++ b/quest/m1/compose/README.md @@ -42,10 +42,24 @@ Decided 2026-10-26 (maintainer), the studio's flagship and first in its epic: replay finds their recent past. Upload follows MoQ's pull model, so a fallback nobody uses costs its encode but no bandwidth. +Changed 2026-11-04 (maintainer): this epic moved to m1, and the program is +the default, not the fallback. Encoding every source for viewers would take +more encoder sessions than a consumer GPU has (decided 2026-11-03 on +[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), so sources +are not published to the relay by default: a source's own track encodes only +while something in the studio uses it, such as an isolated recording or a +replay, and reaches viewers only when the creator ticks +[Published independently](/quest/m0/publish-independently.md). The Delivery +and Cost bullets above, and this epic's parity goal, are deferred with it: a +source published independently is left out of the program, so no picture a +viewer composes from fetchable tracks matches the program today. When this +epic starts, define the reference picture (such as the program with the +independent sources layered over it) and replan those bullets. + ## Required -- [Audio source tracks](/quest/m0/compose/sources.md) - microphones and system audio publish their own tracks beside the picture sources, encoded whatever the demand and uploaded only while subscribed -- [Layout track](/quest/m0/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft -- [Browser composition](/quest/m0/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build -- [Server composition](/quest/m0/compose/server.md) - a headless compositor renders the program from sources and layout, headless -- [Viewer controls](/quest/m0/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch +- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks beside the picture sources (an m0 quest), encoded while the studio uses them and published to the relay only when published independently +- [Layout track](/quest/m1/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft +- [Browser composition](/quest/m1/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build +- [Server composition](/quest/m1/compose/server.md) - a headless compositor renders the program from sources and layout, headless +- [Viewer controls](/quest/m1/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch diff --git a/quest/m0/compose/browser.md b/quest/m1/compose/browser.md similarity index 94% rename from quest/m0/compose/browser.md rename to quest/m1/compose/browser.md index 110866f..712a986 100644 --- a/quest/m0/compose/browser.md +++ b/quest/m1/compose/browser.md @@ -23,5 +23,5 @@ studio, compared within a stated tolerance. ## Required -- [Layout track](/quest/m0/compose/layout.md) - what the browser composes +- [Layout track](/quest/m1/compose/layout.md) - what the browser composes - [moq-video's browser backends are pinned](/quest/m0/pin-browser-video.md) - browser frames into wgpu diff --git a/quest/m0/compose/controls.md b/quest/m1/compose/controls.md similarity index 93% rename from quest/m0/compose/controls.md rename to quest/m1/compose/controls.md index 55086da..2a1c8a3 100644 --- a/quest/m0/compose/controls.md +++ b/quest/m1/compose/controls.md @@ -20,4 +20,4 @@ of them is on, at the highest priority among them. ## Required -- [Browser composition](/quest/m0/compose/browser.md) - the viewer that exposes the toggles +- [Browser composition](/quest/m1/compose/browser.md) - the viewer that exposes the toggles diff --git a/quest/m0/compose/layout.md b/quest/m1/compose/layout.md similarity index 68% rename from quest/m0/compose/layout.md rename to quest/m1/compose/layout.md index 86ce8cd..3c725cb 100644 --- a/quest/m0/compose/layout.md +++ b/quest/m1/compose/layout.md @@ -30,12 +30,25 @@ blend modes. Only a video filter that cannot render the same way in every viewer is baked into a track of its own. A mask names a picture source's track, and a LUT a `lut` source, whose `.cube` table publishes no track yet, so a viewer needs one carrying it, as an image's file is. Audio is the other -way round: an audio source's track is published after its audio filters ([Audio source tracks](/quest/m0/compose/sources.md)), so +way round: an audio source's track is published after its audio filters ([Audio source tracks](/quest/m0/audio-source-tracks.md)), so the layout carries only what is applied after that point, such as gain and mute, and never names an audio filter for a viewer to apply again. The picture source tracks are enough to start; audio source tracks join the layout when they land. +A browser source publishes no track of its own (decided 2026-11-03, +maintainer, on [moq-dev/studio#8](https://github.com/moq-dev/studio/pull/8)). +Each has a "viewers load this page" option, off by default and refused for a +localhost URL: on, the layout carries the page's URL and viewers load it +themselves; off, viewers see the page only in the program. A replay source +(`rs/moq-scene/src/replay.rs`) shows a span of another source's past, so the +layout says when it is hidden and when its span has ended, rather than leaving +a viewer on its last frame. + +Only sources the creator publishes independently +([Published independently](/quest/m0/publish-independently.md)) have tracks a +viewer can fetch; the rest reach viewers only in the program. + ## Related -- [Audio source tracks](/quest/m0/compose/sources.md) - the audio tracks a layout also names once they land +- [Audio source tracks](/quest/m0/audio-source-tracks.md) - the audio tracks a layout also names once they land diff --git a/quest/m0/compose/server.md b/quest/m1/compose/server.md similarity index 93% rename from quest/m0/compose/server.md rename to quest/m1/compose/server.md index 1065fdb..4283027 100644 --- a/quest/m0/compose/server.md +++ b/quest/m1/compose/server.md @@ -22,4 +22,4 @@ and document it in `README.md`. ## Required -- [Layout track](/quest/m0/compose/layout.md) - what the server composes +- [Layout track](/quest/m1/compose/layout.md) - what the server composes