From 283c50d426c3d66fad2e5ce2baaa09984628ac53 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 09:20:36 -0800 Subject: [PATCH 1/2] quest: move viewer-side composition and publish-independently to m2 --- quest/m0/README.md | 4 ++-- quest/{m3 => m2}/compose/README.md | 12 ++++++------ quest/{m3 => m2}/compose/browser.md | 2 +- quest/{m3 => m2}/compose/controls.md | 2 +- quest/{m3 => m2}/compose/layout.md | 6 +++--- quest/{m3 => m2/compose}/publish-independently.md | 2 +- quest/{m3 => m2}/compose/server.md | 2 +- quest/m3/README.md | 4 ++-- 8 files changed, 17 insertions(+), 17 deletions(-) rename quest/{m3 => m2}/compose/README.md (90%) rename quest/{m3 => m2}/compose/browser.md (94%) rename quest/{m3 => m2}/compose/controls.md (93%) rename quest/{m3 => m2}/compose/layout.md (92%) rename quest/{m3 => m2/compose}/publish-independently.md (96%) rename quest/{m3 => m2}/compose/server.md (93%) diff --git a/quest/m0/README.md b/quest/m0/README.md index 464469e..8f25d6f 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -60,7 +60,7 @@ cameras, packaging, signing) that one monorepo no longer pays off. - **Pin:** moq is a `moq` git submodule whose crates are path dependencies, as in moq.pro, so the studio compiles against one moq checkout. - **moq-scene** moved here with the app, its only consumer. The - [viewer-side composition](/quest/m3/compose/README.md) epic (now m3) moved whole: + [viewer-side composition](/quest/m2/compose/README.md) epic (now m3) moved whole: browser composition ships as a package from this repository layered on `@moq/watch`, and the server composite is a headless binary here rather than a moq-cli subcommand, so moq never depends on the studio. @@ -77,7 +77,7 @@ broadcast reaches viewers, not the fallback. Sources are not published to the relay by default (decided 2026-11-03 on [moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), because each encoding costs an encoder session and consumer GPUs have few, so the -[viewer-side composition](/quest/m3/compose/README.md) epic moved to m1. +[viewer-side composition](/quest/m2/compose/README.md) epic moved to m1. Overlays and animation render into the program here, and travel to viewers in the layout track only once that epic lands. diff --git a/quest/m3/compose/README.md b/quest/m2/compose/README.md similarity index 90% rename from quest/m3/compose/README.md rename to quest/m2/compose/README.md index 4e6a7f7..c3cf727 100644 --- a/quest/m3/compose/README.md +++ b/quest/m2/compose/README.md @@ -49,7 +49,7 @@ more encoder sessions than a consumer GPU has (decided 2026-11-03 on are not published to the relay by default: a source's own track encodes only while something in the studio uses it, such as an isolated recording or a replay, and reaches viewers only when the creator ticks -[Published independently](/quest/m3/publish-independently.md). The Delivery +[Published independently](/quest/m2/compose/publish-independently.md). The Delivery and Cost bullets above, and this epic's parity goal, are deferred with it: a source published independently is left out of the program, so no picture a viewer composes from fetchable tracks matches the program today. When this @@ -59,12 +59,12 @@ independent sources layered over it) and replan those bullets. Moved to m2 on 2026-11-07 (maintainer, on [moq-dev/studio#97](https://github.com/moq-dev/studio/pull/97)): nothing in m1 can fetch a source's track, so nothing can read a layout. The layout waits on -[Published independently](/quest/m3/publish-independently.md), and the +[Published independently](/quest/m2/compose/publish-independently.md), and the browser, server, and controls quests wait on it through the layout. ## Required -- [Layout track](/quest/m3/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft -- [Browser composition](/quest/m3/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build -- [Server composition](/quest/m3/compose/server.md) - a headless compositor renders the program from sources and layout, headless -- [Viewer controls](/quest/m3/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch +- [Layout track](/quest/m2/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft +- [Browser composition](/quest/m2/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build +- [Server composition](/quest/m2/compose/server.md) - a headless compositor renders the program from sources and layout, headless +- [Viewer controls](/quest/m2/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch diff --git a/quest/m3/compose/browser.md b/quest/m2/compose/browser.md similarity index 94% rename from quest/m3/compose/browser.md rename to quest/m2/compose/browser.md index eac1e21..f14a32a 100644 --- a/quest/m3/compose/browser.md +++ b/quest/m2/compose/browser.md @@ -23,5 +23,5 @@ studio, compared within a stated tolerance. ## Required -- [Layout track](/quest/m3/compose/layout.md) - what the browser composes +- [Layout track](/quest/m2/compose/layout.md) - what the browser composes - [moq-video's browser backends are pinned](/quest/m0/pin-browser-video.md) - browser frames into wgpu diff --git a/quest/m3/compose/controls.md b/quest/m2/compose/controls.md similarity index 93% rename from quest/m3/compose/controls.md rename to quest/m2/compose/controls.md index 6e9406d..a4f664a 100644 --- a/quest/m3/compose/controls.md +++ b/quest/m2/compose/controls.md @@ -20,4 +20,4 @@ of them is on, at the highest priority among them. ## Required -- [Browser composition](/quest/m3/compose/browser.md) - the viewer that exposes the toggles +- [Browser composition](/quest/m2/compose/browser.md) - the viewer that exposes the toggles diff --git a/quest/m3/compose/layout.md b/quest/m2/compose/layout.md similarity index 92% rename from quest/m3/compose/layout.md rename to quest/m2/compose/layout.md index 032c2c3..7b6f29a 100644 --- a/quest/m3/compose/layout.md +++ b/quest/m2/compose/layout.md @@ -49,7 +49,7 @@ their scene went on program, so the layout says when that was on the media clock, and a viewer that joins late lands at the same point of them. Only sources the creator publishes independently -([Published independently](/quest/m3/publish-independently.md)) have tracks a +([Published independently](/quest/m2/compose/publish-independently.md)) have tracks a viewer can fetch; the rest reach viewers only in the program. Decided 2026-11-05 (maintainer, /quest-plan): the layout serializes the @@ -58,7 +58,7 @@ canvases and mixes they replaced. Decided 2026-11-07 (maintainer, on [moq-dev/studio#97](https://github.com/moq-dev/studio/pull/97)): this quest -waits on [Published independently](/quest/m3/publish-independently.md), since +waits on [Published independently](/quest/m2/compose/publish-independently.md), since until then no viewer can fetch a track the layout names, and the epic moved to m2 with it. Once that lands, replan this quest around the epic's reference picture, which decides what the layout carries. Rejected: replanning now @@ -68,4 +68,4 @@ it. ## Required -- [Published independently](/quest/m3/publish-independently.md) - the source tracks a viewer can fetch, which the layout names +- [Published independently](/quest/m2/compose/publish-independently.md) - the source tracks a viewer can fetch, which the layout names diff --git a/quest/m3/publish-independently.md b/quest/m2/compose/publish-independently.md similarity index 96% rename from quest/m3/publish-independently.md rename to quest/m2/compose/publish-independently.md index d528633..7a165cf 100644 --- a/quest/m3/publish-independently.md +++ b/quest/m2/compose/publish-independently.md @@ -38,4 +38,4 @@ user does, so `README.md` documents them. ## Related -- [Viewer-side composition](/quest/m3/compose/README.md) - the epic that composes from independently published sources +- [Viewer-side composition](/quest/m2/compose/README.md) - the epic that composes from independently published sources diff --git a/quest/m3/compose/server.md b/quest/m2/compose/server.md similarity index 93% rename from quest/m3/compose/server.md rename to quest/m2/compose/server.md index c0ab5e0..296ae95 100644 --- a/quest/m3/compose/server.md +++ b/quest/m2/compose/server.md @@ -22,4 +22,4 @@ and document it in `README.md`. ## Required -- [Layout track](/quest/m3/compose/layout.md) - what the server composes +- [Layout track](/quest/m2/compose/layout.md) - what the server composes diff --git a/quest/m3/README.md b/quest/m3/README.md index f47bc5e..35cea68 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -13,5 +13,5 @@ composition epic that requires it moved here from m2. ## Required -- [Published independently](/quest/m3/publish-independently.md) - a source opts out of the program and publishes its own track for services that place it themselves -- [Viewer-side composition](/quest/m3/compose/README.md) - independently published sources plus a layout track, composed by viewers and by a server +- [Published independently](/quest/m2/compose/publish-independently.md) - a source opts out of the program and publishes its own track for services that place it themselves +- [Viewer-side composition](/quest/m2/compose/README.md) - independently published sources plus a layout track, composed by viewers and by a server From e2d67864e1856d3ca8ae0cf356af518f9ead978c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 09:23:12 -0800 Subject: [PATCH 2/2] quest: replan viewer-side composition around players layering independent sources Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 5 +- quest/m2/README.md | 5 +- quest/m2/compose/README.md | 112 ++++++++++------------ quest/m2/compose/browser.md | 27 ------ quest/m2/compose/controls.md | 21 ++-- quest/m2/compose/hang-layers.md | 30 ++++++ quest/m2/compose/layout.md | 93 +++++++----------- quest/m2/compose/publish-independently.md | 12 +-- quest/m2/compose/server.md | 18 ++-- quest/m2/compose/watch.md | 27 ++++++ quest/m3/README.md | 7 +- 11 files changed, 180 insertions(+), 177 deletions(-) delete mode 100644 quest/m2/compose/browser.md create mode 100644 quest/m2/compose/hang-layers.md create mode 100644 quest/m2/compose/watch.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 8f25d6f..2c28106 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -60,9 +60,8 @@ cameras, packaging, signing) that one monorepo no longer pays off. - **Pin:** moq is a `moq` git submodule whose crates are path dependencies, as in moq.pro, so the studio compiles against one moq checkout. - **moq-scene** moved here with the app, its only consumer. The - [viewer-side composition](/quest/m2/compose/README.md) epic (now m3) moved whole: - browser composition ships as a package from this repository layered on - `@moq/watch`, and the server composite is a headless binary here rather + [viewer-side composition](/quest/m2/compose/README.md) epic (now m2) moved whole: + the server composite is a headless binary here rather than a moq-cli subcommand, so moq never depends on the studio. - **Egress library:** lives here first, beside its studio consumer, and moves to moq only when a hosted service adopts it. diff --git a/quest/m2/README.md b/quest/m2/README.md index 73939c8..dd2f771 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -10,8 +10,11 @@ it. Opened 2026-11-05 (maintainer, /quest-plan). A quest moves up to m0 or m1 when a consumer appears. Publishing a source independently and viewer-side composition moved on to [m3](/quest/m3/README.md) on 2026-11-07 (maintainer): -no customer asks for them. +no customer asks for them. They came back to m2 on 2026-11-08 (maintainer, +/quest-plan), replanned around players that composite independent sources +over the program. ## Required - [Overlays drop their Blitz panic guard](/quest/m2/overlay-blitz-release.md) - the next Blitz release lets overlay checks parse without `catch_unwind` +- [Viewer-side composition](/quest/m2/compose/README.md) - players composite independently published sources over the program from a layer layout track diff --git a/quest/m2/compose/README.md b/quest/m2/compose/README.md index c3cf727..905f128 100644 --- a/quest/m2/compose/README.md +++ b/quest/m2/compose/README.md @@ -2,69 +2,57 @@ ## Goal -A `moq-studio` broadcast carries each source as its own track plus a layout -track describing the live scene. Viewers compose the picture themselves: -they prioritize sources and toggle elements on and off. A composited program -remains for protocols and devices that cannot compose (RTMP and WHIP -destinations, HLS, legacy players), produced by the studio or by a server. - -This README's own work is the end-to-end test: a studio broadcast is composed -by the browser, by the headless compositor, and by the studio itself, and all three -pictures match. +A studio broadcast carries each independently published source as its own +track, plus a layer layout track that says where and when each one shows. +Players composite those sources over the program and mix their audio in, so +a viewer sees the creator's picture while those sources travel, and can be +toggled, as their own tracks. The end goal is moq's own players, `moq play` +and `@moq/watch`, playing a studio broadcast this way from a layer layout +that hang specifies. + +This README's own work is the end-to-end test: a studio broadcast with an +independently published camera and microphone plays in `moq play` and +`@moq/watch` with the camera placed and timed as the studio's preview shows +it, within a stated tolerance, and with its sound mixed in. ## Plan -Decided 2026-10-26 (maintainer), the studio's flagship and first in its epic: - -- **Delivery:** MoQ's value is per-track delivery, so the studio embraces - viewer composition. The program is the fallback, not the default. -- **One compositor, three hosts:** moq-scene renders in the studio, headless - on a server (the headless compositor), and in the browser, so all three - pictures come from one implementation. -- **Parity rule:** every scene feature (transitions, filters, effects, - overlays) renders in all three hosts with the same layout, timing, and - colors, within a tolerance. Each renderer keeps its own text antialiasing - and blur, so the browser can draw overlays as real DOM. A feature that - cannot meet this, such as a CEF browser source or an ML effect, is baked - into its source's own track before publishing rather than living in the - layout. -- **Wire layout:** local sources are tracks in the studio's one broadcast. - Guests stay their own broadcasts, named through the hang catalog's - `contributions` references. -- **Layout:** the live scene state (items, transforms, transitions, and each - element's priority and toggleable flag) as a hang JSON track on the media - clock, so any host composites the same frame the studio shows. -- **Browser build:** whichever line [JS-only @moq/net from Rust](https://github.com/moq-dev/moq/blob/main/quest/m5/rs2ts-js.md) - lands. moq's WASM support was removed on 2026-10-31. -- **Cost:** every source and the program encode whatever the subscriber - demand (decided 2026-10-28, from - [#7101](https://github.com/moq-dev/moq/issues/7101)), so a late viewer or a - replay finds their recent past. Upload follows MoQ's pull model, so a - fallback nobody uses costs its encode but no bandwidth. - -Changed 2026-11-04 (maintainer): this epic moved to m1, and the program is -the default, not the fallback. Encoding every source for viewers would take -more encoder sessions than a consumer GPU has (decided 2026-11-03 on -[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), so sources -are not published to the relay by default: a source's own track encodes only -while something in the studio uses it, such as an isolated recording or a -replay, and reaches viewers only when the creator ticks -[Published independently](/quest/m2/compose/publish-independently.md). The Delivery -and Cost bullets above, and this epic's parity goal, are deferred with it: a -source published independently is left out of the program, so no picture a -viewer composes from fetchable tracks matches the program today. When this -epic starts, define the reference picture (such as the program with the -independent sources layered over it) and replan those bullets. - -Moved to m2 on 2026-11-07 (maintainer, on -[moq-dev/studio#97](https://github.com/moq-dev/studio/pull/97)): nothing in m1 -can fetch a source's track, so nothing can read a layout. The layout waits on -[Published independently](/quest/m2/compose/publish-independently.md), and the -browser, server, and controls quests wait on it through the layout. - -## Required +Replanned 2026-11-08 (maintainer, /quest-plan). This supersedes the +2026-11-07 deferral on +[moq-dev/studio#97](https://github.com/moq-dev/studio/pull/97), which waited +for [Published independently](/quest/m2/compose/publish-independently.md) to +land before replanning: + +- **Reference picture:** ✅ the program with each independently published + source layered over it (rejected: the full scene rebuilt from every + source's track, which reverses the 2026-11-04 encoder-cost decision; both, + chosen per scene). +- **Players:** ✅ native first (rejected: a browser player only; a browser + page layering sources without the studio's placement rules). The end goal + is moq's `moq play` and `@moq/watch`. +- **Compositor:** ✅ hang's draft specifies the layer layout, and each player + implements it with its own code. Parity comes from a shared fixture within a + tolerance, not shared code. Rejected: moving moq-scene into moq, which + reverses the 2026-11-02 rule that moq never depends on this repository. + This replaces the 2026-10-26 "one compositor, three hosts" rule. +- **Prototype:** ✅ a throwaway `moq-studio --watch` on moq-scene with a + provisional layer format, so the format settles without moq round trips + (rejected: straight to `moq play`). moq's spec quest starts from it. +- **Audio:** ✅ players mix independently published audio sources with the + program's, at the gain and mute the layout carries (rejected: video only). +- **Controls:** ✅ viewer toggles and priorities land in the native player + (rejected: browser only). +- **Browser composition** (moq-scene's browser build layered on + `@moq/watch`): ✅ deleted, since `@moq/watch` layering supersedes it. + Server composition stays, replanned to flatten program and layers for + destinations that cannot compose. +- **Milestone:** ✅ m2 (rejected: m1; staying in m3). The moq quests join + moq's m3. + +Still settled from 2026-10-26: local sources are tracks in the studio's one +broadcast, and guests stay their own broadcasts, named through the hang +catalog's `contributions` references. + +moq's side, in its own quest tree (m3): hang specifies a layer layout track, +and `moq play` and `@moq/watch` composite it. -- [Layout track](/quest/m2/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft -- [Browser composition](/quest/m2/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build -- [Server composition](/quest/m2/compose/server.md) - a headless compositor renders the program from sources and layout, headless -- [Viewer controls](/quest/m2/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch diff --git a/quest/m2/compose/browser.md b/quest/m2/compose/browser.md deleted file mode 100644 index f14a32a..0000000 --- a/quest/m2/compose/browser.md +++ /dev/null @@ -1,27 +0,0 @@ -# [L] Browser composition - -## Goal - -A browser player composes a broadcast that carries a layout track: it fetches -the tracks the layout shows and draws them through moq-scene's browser build, -matching the studio's picture. A broadcast without one plays its program as -`@moq/watch` does today. - -## Plan - -The browser build comes from whichever line -[JS-only @moq/net from Rust](https://github.com/moq-dev/moq/blob/main/quest/m5/rs2ts-js.md) lands (decided 2026-10-26; moq's WASM -support was removed on 2026-10-31), so this quest does not fix the -mechanism. A thin package published from this repository (e.g. `@moq/scene`) -wraps it and layers on `@moq/watch`; `@moq/watch` never depends on it, so moq -never depends on this repository (decided 2026-11-02, when the studio left -moq). Frames reach it as WebCodecs `VideoFrame`s through the browser import -path [moq-video in the browser](https://github.com/moq-dev/moq/blob/main/quest/m5/video-web.md) adds. - -Prove parity with a fixture: the same broadcast composited here and by the -studio, compared within a stated tolerance. - -## Required - -- [Layout track](/quest/m2/compose/layout.md) - what the browser composes -- [moq-video's browser backends are pinned](/quest/m0/pin-browser-video.md) - browser frames into wgpu diff --git a/quest/m2/compose/controls.md b/quest/m2/compose/controls.md index a4f664a..fccf44d 100644 --- a/quest/m2/compose/controls.md +++ b/quest/m2/compose/controls.md @@ -2,22 +2,25 @@ ## Goal -A viewer composing a studio broadcast can switch off any element the layout +A viewer watching a studio broadcast can switch off any layer the layout marks toggleable (a facecam, a chat overlay), and stops fetching its track. -The layout's per-element priorities steer delivery, so under congestion the -least important sources degrade first. The creator sets both per element in +The layout's per-layer priorities steer delivery, so under congestion the +least important sources degrade first. The creator sets both per source in the studio. ## Plan -Map element priority onto the subscription priorities moq-lite already has +Decided 2026-11-08 (maintainer, /quest-plan): the controls land in the native +player, the [watch prototype](/quest/m2/compose/watch.md) (rejected: browser +only). moq's players take them from hang's layer spec. + +Map layer priority onto the subscription priorities moq-lite already has (`doc/concept/moq-lite.md`), rather than a second mechanism. A toggled-off -element is unsubscribed, so it costs the viewer and the relay nothing; the +layer is unsubscribed, so it costs the viewer and the relay nothing. The studio keeps encoding it, so toggling it back on can reach its recent past. -Subscriptions are per track, not per -element: a linked source shown by several items stays subscribed while any -of them is on, at the highest priority among them. +Subscriptions are per track, not per layer: a source shown by several layers +stays subscribed while any of them is on, at the highest priority among them. ## Required -- [Browser composition](/quest/m2/compose/browser.md) - the viewer that exposes the toggles +- [moq-studio watch prototype](/quest/m2/compose/watch.md) - the viewer that exposes the toggles diff --git a/quest/m2/compose/hang-layers.md b/quest/m2/compose/hang-layers.md new file mode 100644 index 0000000..2b45255 --- /dev/null +++ b/quest/m2/compose/hang-layers.md @@ -0,0 +1,30 @@ +# [S] The studio publishes hang's layer layout + +## Goal + +The `moq` submodule includes hang's layer layout track and `moq play`'s layer +compositing. The studio publishes hang's format instead of the provisional +`dev.moq.studio.layers`, and the `moq-studio --watch` prototype is gone, with +`README.md` pointing viewers at `moq play` instead. + +## Plan + +Decided 2026-11-08 (maintainer, /quest-plan): the layer layout becomes +official moq, so `moq play` and `@moq/watch` play a studio broadcast. Once +the wait clears, advance the gitlink with `just upgrade-moq` in this PR, move +the studio to hang's types, and delete the +[watch prototype](/quest/m2/compose/watch.md) and its flag. + +## Waiting + +- moq-dev/moq: hang's layer layout spec and `moq play`'s layer compositing + land, planned in moq's m3 from the + [watch prototype](/quest/m2/compose/watch.md)'s format. Check: both quests' + PRs have merged on moq's `main`, deleting their files, and hang exposes the + layer layout. A quest file that moved, or went away without its work + landing, does not count. + +## Required + +- [moq-studio watch prototype](/quest/m2/compose/watch.md) - the format moq specifies +- [Viewer controls](/quest/m2/compose/controls.md) - toggles and priorities the spec carries diff --git a/quest/m2/compose/layout.md b/quest/m2/compose/layout.md index 7b6f29a..05a5067 100644 --- a/quest/m2/compose/layout.md +++ b/quest/m2/compose/layout.md @@ -1,71 +1,48 @@ -# [M] Layout track +# [M] Layer layout track ## Goal -A studio broadcast carries a layout track: moq-scene's live scene state -(items, the tracks they show, transforms, transitions, and each element's -priority and whether a viewer may toggle it) as a hang JSON track on the -media clock. Any host that reads it composites the same frame the studio -shows. The hang catalog and `drafts/draft-lcurley-moq-hang.md` specify it. +A studio broadcast that publishes a source independently also carries a layer +layout track. For each such source, it says where and when that source shows +over the program: its transform, crop, z-order, opacity, and visibility, its +item's keyframes and when its scene went on program, all on the media clock, +and for audio its gain and mute. Until hang specifies it, the track is the +provisional `dev.moq.studio.layers` in the studio catalog. ## Plan -Serialize moq-scene's own model rather than a second layout schema, so the -studio, server, and browser cannot drift. Version it and refuse an unknown -version loudly. Keep it free of local paths and credentials: items name -tracks and contribution references, never files. A viewer that refuses a -layout version plays the program instead and says why. A late joiner needs the -current state, so use the JSON track's snapshot mode. - -Specify the source listings in the hang draft too, then move the studio to -what it settles on. Until then sources are listed provisionally as -`dev.moq.studio.sources.` in the studio catalog, each a nested hang -catalog (decided 2026-11-02, maintainer; `rs/moq-scene/src/publish.rs`). - -Clones and video filters travel in the layout, not as tracks: a picture -source's track is its capture before any filter, and a clone publishes none, -so the layout names the source a clone shows and the video filters each -source and item applies (decided 2026-11-02, maintainer), with crops and -blend modes. Only a video filter that cannot render the same way in every -viewer is baked into a track of its own. A mask names a picture source's -track, and a LUT a `lut` source, whose `.cube` table publishes no track yet, -so a viewer needs one carrying it, as an image's file is. Audio is the other -way round: an audio source's track carries its sound after its audio filters -and before its gain and mute (decided 2026-11-02, maintainer), so -the layout carries only what is applied after that point, such as gain and -mute, and never names an audio filter for a viewer to apply again. - -A browser source publishes no track of its own (decided 2026-11-03, -maintainer, on [moq-dev/studio#8](https://github.com/moq-dev/studio/pull/8)). -Each has a "viewers load this page" option, off by default and refused for a -localhost URL: on, the layout carries the page's URL and viewers load it -themselves; off, viewers see the page only in the program. A replay source -(`rs/moq-scene/src/replay.rs`) shows a span of another source's past, so the -layout says when it is hidden and when its span has ended, rather than leaving -a viewer on its last frame. +Replanned 2026-11-08 (maintainer, /quest-plan) around the program with +independent layers over it, rather than the full scene state; see the +[epic](/quest/m2/compose/README.md). + +A JSON track in snapshot mode, so a late joiner gets the current state. +Version it, and have a player refuse an unknown version loudly and play the +program alone. Keep it free of local paths and credentials: layers name +tracks and contribution references, never files. + +Layers draw over the program in the layout's order, so the studio's preview +of a scene with an independent source shows what viewers see. A layer carries +only what every player can reproduce: transform, crop, z-order, opacity, +visibility, and keyframes. A picture source's track is its capture before any +filter (decided 2026-11-02, maintainer). So an item that applies anything +more, such as a video filter, mask, LUT, blend mode, or background removal, +needs that effect baked into its track, or the option refused for it. Settle +which here, with +[Published independently](/quest/m2/compose/publish-independently.md)'s +background-removal note. An audio source's track already carries its sound +after its filters (decided 2026-11-02), so its layer carries only gain and +mute. Items carry their keyframes (`moq_scene::Keyframe`), which play from when -their scene went on program, so the layout says when that was on the media -clock, and a viewer that joins late lands at the same point of them. - -Only sources the creator publishes independently -([Published independently](/quest/m2/compose/publish-independently.md)) have tracks a -viewer can fetch; the rest reach viewers only in the program. +their scene went on program. The layout says when that was, so a late joiner +lands at the same point. -Decided 2026-11-05 (maintainer, /quest-plan): the layout serializes the -scene model with scene sources, published scenes, and audio scenes, not the -canvases and mixes they replaced. +Each layer also carries its priority and whether a viewer may toggle it, for +[Viewer controls](/quest/m2/compose/controls.md). -Decided 2026-11-07 (maintainer, on -[moq-dev/studio#97](https://github.com/moq-dev/studio/pull/97)): this quest -waits on [Published independently](/quest/m2/compose/publish-independently.md), since -until then no viewer can fetch a track the layout names, and the epic moved to -m2 with it. Once that lands, replan this quest around the epic's reference -picture, which decides what the layout carries. Rejected: replanning now -around the program with the independently published sources layered over it, -and building the full-scene layout as written above while no viewer can use -it. +The [watch prototype](/quest/m2/compose/watch.md) is this track's first +reader. What the two settle becomes moq's hang spec draft. ## Required -- [Published independently](/quest/m2/compose/publish-independently.md) - the source tracks a viewer can fetch, which the layout names +- [Published independently](/quest/m2/compose/publish-independently.md) - the source tracks a layer names diff --git a/quest/m2/compose/publish-independently.md b/quest/m2/compose/publish-independently.md index 7a165cf..48dbe0e 100644 --- a/quest/m2/compose/publish-independently.md +++ b/quest/m2/compose/publish-independently.md @@ -24,18 +24,18 @@ source among those on program (`Publish::set_on_program` in Moved to m2 on 2026-11-05 (maintainer, /quest-plan): no service places sources itself yet, so it waits behind m0 and m1. Moved on to m3 on -2026-11-07 (maintainer, /quest-spawn): it drives no customer demand. +2026-11-07 (maintainer, /quest-spawn): it drives no customer demand. Back to +m2 on 2026-11-08 (maintainer, /quest-plan) in the +[viewer-side composition](/quest/m2/compose/README.md) epic, whose players +composite independently published sources over the program and mix their +audio back in. A source with its background removed (`background = "remove"`) keeps the background in its track, since H.264 carries no transparency. Publishing one independently needs its cut-out in the track, such as moq-video encoding alpha, or the option refused for it. -Low priority: this is for services that move webcams and overlays themselves, -and none exists yet. The option lives on the source, in the scene file, so it +The option lives on the source, in the scene file, so it travels with the scene. The option and its effect on the program change what a user does, so `README.md` documents them. -## Related - -- [Viewer-side composition](/quest/m2/compose/README.md) - the epic that composes from independently published sources diff --git a/quest/m2/compose/server.md b/quest/m2/compose/server.md index 296ae95..324cdb1 100644 --- a/quest/m2/compose/server.md +++ b/quest/m2/compose/server.md @@ -2,14 +2,20 @@ ## Goal -A headless compositor binary in this repository subscribes to a studio broadcast's sources and -layout track and publishes the composited program, headless (a GPU, or -lavapipe without one). It can push that program to RTMP and WHIP through the -studio's egress library, so a hosted service makes the fallback composite and -the creator saves the upload. +A headless compositor binary in this repository subscribes to a studio +broadcast's program, its independently published sources, and its layer +layout, and publishes the flattened program, headless (a GPU, or lavapipe +without one). It can push that program to RTMP and WHIP through the studio's +egress library. So destinations that cannot compose, such as RTMP, WHIP, HLS, +and legacy players, still see the sources the creator published +independently. ## Plan +Replanned 2026-11-08 (maintainer, /quest-plan): it flattens the program and +its layers, since a source published independently leaves the program; see +the [epic](/quest/m2/compose/README.md). + Decided 2026-10-26 (maintainer): its own command, not a mode of `moq transcode`. A hosted service embeds the same library. It publishes its own broadcast, never the studio's names: each publisher produces its own @@ -22,4 +28,4 @@ and document it in `README.md`. ## Required -- [Layout track](/quest/m2/compose/layout.md) - what the server composes +- [Layer layout track](/quest/m2/compose/layout.md) - what the server flattens diff --git a/quest/m2/compose/watch.md b/quest/m2/compose/watch.md new file mode 100644 index 0000000..c091048 --- /dev/null +++ b/quest/m2/compose/watch.md @@ -0,0 +1,27 @@ +# [M] moq-studio watch prototype + +## Goal + +`moq-studio --watch ` plays a studio broadcast: the program, with each +source the layer layout names composited over it on the media clock, and +their audio mixed with the program's. A broadcast without a layer layout plays +its program alone. + +## Plan + +Decided 2026-11-08 (maintainer, /quest-plan): a throwaway prototype, so the +layer format settles without moq round trips before hang specifies it +(rejected: going straight to `moq play`). It lives in `moq-studio`, not its +own binary, since [hang-layers](/quest/m2/compose/hang-layers.md) deletes it +once `moq play` plays layers. + +Build it on moq-scene, which already plays remote broadcasts as sources for +guests (`rs/moq-studio/src/guests.rs`): a scene that the layout drives rather +than a scene file. `README.md` documents the flag while it exists. + +Its working format is the starting point for moq's hang layer spec, which +waits on this quest. + +## Required + +- [Layer layout track](/quest/m2/compose/layout.md) - what the prototype plays diff --git a/quest/m3/README.md b/quest/m3/README.md index 35cea68..a2f5fca 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -9,9 +9,6 @@ is offered for work until a customer need moves it up. Opened 2026-11-07 (maintainer, /quest-spawn): publishing a source independently drives no customer demand, so it and the viewer-side -composition epic that requires it moved here from m2. +composition epic that requires it moved here from m2. Both went back to +[m2](/quest/m2/README.md) on 2026-11-08 (maintainer, /quest-plan). -## Required - -- [Published independently](/quest/m2/compose/publish-independently.md) - a source opts out of the program and publishes its own track for services that place it themselves -- [Viewer-side composition](/quest/m2/compose/README.md) - independently published sources plus a layout track, composed by viewers and by a server