Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions quest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,4 @@ there first. A quest here that needs it waits on a pin quest, which
## Required

- [m0: the studio](/quest/m0/README.md) - the native and wasm studio for creators who rely on OBS today
- [m1: viewers compose](/quest/m1/README.md) - viewers and a server compose a broadcast from its sources and a layout track
14 changes: 12 additions & 2 deletions quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ cameras, packaging, signing) that one monorepo no longer pays off.
- **Pin:** moq is a `moq` git submodule whose crates are path dependencies,
as in moq.pro, so the studio compiles against one moq checkout.
- **moq-scene** moved here with the app, its only consumer. The
[viewer-side composition](/quest/m0/compose/README.md) epic moved whole:
[viewer-side composition](/quest/m1/compose/README.md) epic (now m1) moved whole:
browser composition ships as a package from this repository layered on
`@moq/watch`, and the server composite is a headless binary here rather
than a moq-cli subcommand, so moq never depends on the studio.
Expand All @@ -72,6 +72,15 @@ cameras, packaging, signing) that one monorepo no longer pays off.
waits on a pin quest here, which `just upgrade-moq` advances. This replaces
"never ahead of the studio UI that consumes them" above.

Changed 2026-11-04 with the maintainer: the program is how a studio
broadcast reaches viewers, not the fallback. Sources are not published to the
relay by default (decided 2026-11-03 on
[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), because
each encoding costs an encoder session and consumer GPUs have few, so the
[viewer-side composition](/quest/m1/compose/README.md) epic moved to m1.
Overlays and animation render into the program here, and travel to viewers
in the layout track only once that epic lands.

Dropped from the first scoping round: relay fan-out ("encode once, the relay
restreams") and its upload savings. Direct push with shared encoders matches
OBS multistream, and the hosted version is product work outside this
Expand All @@ -81,7 +90,7 @@ repository.

- [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux
- [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux
- [Viewer-side composition](/quest/m0/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers, with a server-composed program as the fallback
- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks, for isolated recording and replay
- [Broadcast sources play in sync](/quest/m0/broadcast-sync.md) - subscribed broadcasts and guests play audio and video on the publisher's timestamps, with `moq play`'s delay
- [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker
- [Live encode control](/quest/m0/encode-control.md) - simulcast rungs that follow the bandwidth grant, live bitrate and resolution, keyframes on request
Expand All @@ -100,6 +109,7 @@ repository.
- [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops
- [Browser sources keep a profile](/quest/m0/browser-profile.md) - one persistent profile, so logins survive restarts, and a reload action
- [Browser source popups](/quest/m0/browser-popups.md) - dropdowns and other in-page popups draw and take input
- [Published independently](/quest/m0/publish-independently.md) - a source opts out of the program and publishes its own track for services that place it themselves
- [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository
- [Virtual camera](/quest/m0/vcam/README.md) - a canvas's program exposed as a system camera on Linux, Windows, and macOS
- [Signing identities are available](/quest/m0/signing.md) - the Apple and Windows code-signing credentials packaging and the macOS camera need
Expand Down
14 changes: 8 additions & 6 deletions quest/m0/animation.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,21 @@
Scene changes in `moq-studio` use transitions beyond a cut and fade: move
(items glide between their positions in the two scenes), slide, wipe, and
stinger (a video clip with a cut point). Any item property can be keyframed
with easing, so an item can fly in or a camera can push in, and the same
animation plays in every composition host.
with easing, so an item can fly in or a camera can push in.

## Plan

Move is OBS's most-installed plugin (Move Transition), and stingers are
standard in broadcast. Animate in moq-scene on the media clock and carry the
transition and keyframes in the layout track, so viewers compose them too.
standard in broadcast. Animate in moq-scene on the media clock, so the program, a recording, and a
replay match. Carrying transitions and keyframes in the layout track, for
viewers to compose, waits for the m1
[composition epic](/quest/m1/compose/README.md) (changed 2026-11-04,
maintainer: the program comes first).
Overlays animate through their own CSS. No Lottie runtime for now (decided
2026-10-26): the open ones either lack text (velato) or add a C++ build
(ThorVG). A stinger clip is a media source like any other. Transitions and
keyframes change through moq-scene's `Edit`, so they undo like any edit.

## Required
## Related

- [Layout track](/quest/m0/compose/layout.md) - where transitions and keyframes travel
- [Layout track](/quest/m1/compose/layout.md) - where transitions and keyframes travel to viewers in m1
44 changes: 44 additions & 0 deletions quest/m0/audio-source-tracks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# [S] Audio source tracks

## Goal

Each microphone and system-audio source in a `moq-studio` scene has its own
capture as an Opus track in the studio's broadcast, listed in that source's
catalog under the studio catalog's `dev.moq.studio` section, beside the
program, as the picture sources already are. An isolated recording and a
replay of an audio source read that track.

## Plan

Cameras, displays, windows, images, and colors already publish this way
(`rs/moq-scene/src/publish.rs`): each source's renditions sit in a nested hang
catalog keyed by source name, so the root `video` and `audio` sections stay
the program's. An audio source adds an `audio` rendition to its own nested
catalog the same way.

moq-audio's capture Producer/Consumer split (`capture::Producer`) feeds both
the bus and an encoder from one device open, rather than opening each device
twice.
Publish the capture after the source's filters but before its gain and mute
(decided 2026-11-02, maintainer): a recording or replay holds filtered audio,
as video bakes in its ML effects, and the levels stay the program's.

Changed 2026-11-03 (maintainer, on
[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)): a source's
own track encodes only while something in the studio uses it, and reaches
the relay only when the creator publishes the source independently
([Published independently](/quest/m0/publish-independently.md)). A source
on program counts as used (decided 2026-11-04, maintainer), so a replay finds
its past from when it went on: a picture source while the program scene shows
it, and an audio source while it feeds the program mix. A source published
independently counts as used too, since its track is what it publishes.
Otherwise a source encodes only while an isolated recording or a running
replay reads it, and replaying a source with no history is refused with the
reason.
This replaces the earlier always-on decision (2026-11-01, on
[moq-dev/moq#7554](https://github.com/moq-dev/moq/pull/7554)), because
encoding every source costs encoder sessions a consumer machine does not
have; an audio source follows the picture sources' rule.
Comment thread
kixelated marked this conversation as resolved.
Resolved

Track names stay stable for a source's life, and an edited source that
changes what it shows gets a new name.
37 changes: 0 additions & 37 deletions quest/m0/compose/sources.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m0/effects.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ composition host renders with the same WGSL and the layout carries by name and
settings. A model cannot meet that parity rule: viewers would each need the
model and a GPU to run it, and inference differs across backends. So an ML
effect is baked into its source's track per the
[composition epic](/quest/m0/compose/README.md): it belongs to a source, never
[composition epic](/quest/m1/compose/README.md): it belongs to a source, never
to one scene item, and sits before the source's crop and filters, as the
encoder in `rs/moq-scene/src/publish.rs` sees the capture.

Expand Down
21 changes: 13 additions & 8 deletions quest/m0/overlays.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@
## Goal

Titles, lower thirds, tickers, and other graphics in `moq-studio` are written
as HTML and CSS without JavaScript, travel in the layout track, and render in
all three composition hosts: through Blitz in the studio and the headless compositor,
and as real DOM over the video in the browser. CSS animations and transitions
run on the media clock, so an overlay animates on the same frame everywhere.
as HTML and CSS without JavaScript and render through Blitz into the program.
CSS animations and transitions run on the media clock, so an overlay animates
on the same frame in the program, a recording, and a replay.

## Plan

Expand All @@ -20,11 +19,11 @@ Decided 2026-10-26 (maintainer):
JavaScript engine, and has no clock injection.
- **Browser:** the DOM layered over the composited video, so no HTML-in-Canvas
and it works in every browser today.
- **Subset:** the layout track accepts only the CSS Blitz renders and refuses
- **Subset:** an overlay accepts only the CSS Blitz renders and refuses
the rest at setup with the property named. Blitz is beta (no blend modes or
`text-shadow`, filters limited to blur and drop-shadow), and the subset
grows as it does.
- **Parity:** layout parity per the [composition epic](/quest/m0/compose/README.md);
- **Parity:** layout parity per the [composition epic](/quest/m1/compose/README.md);
the fixture compares boxes and timing, and pixels only loosely.

An overlay is hostile input to every viewer, so it fetches nothing from the
Expand All @@ -37,6 +36,12 @@ upstream that if it cannot. Arbitrary web pages and JavaScript graphics stay
browser sources (`moq_scene::browser`), baked into their own
track.

## Required
Changed 2026-11-04 (maintainer): the program comes first, so this quest
renders overlays in the studio only. Carrying them in the layout track, the
browser's DOM rendering, and the headless compositor wait for the m1
[composition epic](/quest/m1/compose/README.md); keep the overlay model free
of studio-only state so that epic can serialize it.

- [Layout track](/quest/m0/compose/layout.md) - where overlays travel
## Related

- [Layout track](/quest/m1/compose/layout.md) - where overlays travel to viewers in m1
29 changes: 29 additions & 0 deletions quest/m0/publish-independently.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# [S] Published independently

## Goal

A source in `moq-studio` has a "Published independently" option, off by
default. Ticked, the source's own track is published to the relay for viewers
and services to fetch, and the source is left out of the composited program,
so a service can place a webcam or overlay itself. Unticked, the source
reaches viewers only through the program.

## Plan

Decided 2026-11-03 (maintainer, on
[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)): sources are
not published to the relay by default, since every encoding costs an encoder
session and consumer GPUs have few. A source published independently gets one
rendition, as sources do in the encode ladder, while the program keeps its
two sizes. It counts as used, so it encodes for as long as it is published
([Audio source tracks](/quest/m0/audio-source-tracks.md) states the rule).

Low priority: this is for services that move webcams and overlays themselves,
and none exists yet. The option lives on the source, in the scene file, so it
travels with the scene. The option and its effect on the program change what a
user does, so `README.md` documents them.

## Related

- [Live encode control](/quest/m0/encode-control.md) - the encode ladder that makes source tracks local by default
- [Viewer-side composition](/quest/m1/compose/README.md) - the m1 epic that composes from independently published sources
20 changes: 20 additions & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# m1: viewers compose

## Goal

A studio broadcast can be composed by its viewers and by a server from the
sources' own tracks and a layout track, matching the program the studio
composites, so a viewer can toggle and prioritize elements.

## Plan

Moved out of m0 on 2026-11-04 (maintainer): the program is how a studio
broadcast reaches viewers, and sources are not published to the relay by
default (decided 2026-11-03 on
[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), since
encoding every source for viewers costs more encoder sessions than a consumer
GPU has. Viewer-side composition waits until m0 ships.

## Required

- [Viewer-side composition](/quest/m1/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers and by a server
24 changes: 19 additions & 5 deletions quest/m0/compose/README.md → quest/m1/compose/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,24 @@ Decided 2026-10-26 (maintainer), the studio's flagship and first in its epic:
replay finds their recent past. Upload follows MoQ's pull model, so a
fallback nobody uses costs its encode but no bandwidth.

Changed 2026-11-04 (maintainer): this epic moved to m1, and the program is
the default, not the fallback. Encoding every source for viewers would take
more encoder sessions than a consumer GPU has (decided 2026-11-03 on
[moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), so sources
are not published to the relay by default: a source's own track encodes only
while something in the studio uses it, such as an isolated recording or a
replay, and reaches viewers only when the creator ticks
[Published independently](/quest/m0/publish-independently.md). The Delivery
and Cost bullets above, and this epic's parity goal, are deferred with it: a
source published independently is left out of the program, so no picture a
viewer composes from fetchable tracks matches the program today. When this
epic starts, define the reference picture (such as the program with the
independent sources layered over it) and replan those bullets.

## Required

- [Audio source tracks](/quest/m0/compose/sources.md) - microphones and system audio publish their own tracks beside the picture sources, encoded whatever the demand and uploaded only while subscribed
- [Layout track](/quest/m0/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft
- [Browser composition](/quest/m0/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build
- [Server composition](/quest/m0/compose/server.md) - a headless compositor renders the program from sources and layout, headless
- [Viewer controls](/quest/m0/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch
- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks beside the picture sources (an m0 quest), encoded while the studio uses them and published to the relay only when published independently
- [Layout track](/quest/m1/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft
- [Browser composition](/quest/m1/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build
- [Server composition](/quest/m1/compose/server.md) - a headless compositor renders the program from sources and layout, headless
- [Viewer controls](/quest/m1/compose/controls.md) - viewers toggle elements and the layout's priorities steer what they fetch
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@ studio, compared within a stated tolerance.

## Required

- [Layout track](/quest/m0/compose/layout.md) - what the browser composes
- [Layout track](/quest/m1/compose/layout.md) - what the browser composes
- [moq-video's browser backends are pinned](/quest/m0/pin-browser-video.md) - browser frames into wgpu
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ of them is on, at the highest priority among them.

## Required

- [Browser composition](/quest/m0/compose/browser.md) - the viewer that exposes the toggles
- [Browser composition](/quest/m1/compose/browser.md) - the viewer that exposes the toggles
17 changes: 15 additions & 2 deletions quest/m0/compose/layout.md → quest/m1/compose/layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,25 @@ blend modes. Only a video filter that cannot render the same way in every
viewer is baked into a track of its own. A mask names a picture source's
track, and a LUT a `lut` source, whose `.cube` table publishes no track yet,
so a viewer needs one carrying it, as an image's file is. Audio is the other
way round: an audio source's track is published after its audio filters ([Audio source tracks](/quest/m0/compose/sources.md)), so
way round: an audio source's track is published after its audio filters ([Audio source tracks](/quest/m0/audio-source-tracks.md)), so
the layout carries only what is applied after that point, such as gain and
mute, and never names an audio filter for a viewer to apply again. The
picture source tracks are enough to start; audio source tracks join the
layout when they land.

A browser source publishes no track of its own (decided 2026-11-03,
maintainer, on [moq-dev/studio#8](https://github.com/moq-dev/studio/pull/8)).
Each has a "viewers load this page" option, off by default and refused for a
localhost URL: on, the layout carries the page's URL and viewers load it
themselves; off, viewers see the page only in the program. A replay source
(`rs/moq-scene/src/replay.rs`) shows a span of another source's past, so the
layout says when it is hidden and when its span has ended, rather than leaving
a viewer on its last frame.

Only sources the creator publishes independently
([Published independently](/quest/m0/publish-independently.md)) have tracks a
viewer can fetch; the rest reach viewers only in the program.

## Related

- [Audio source tracks](/quest/m0/compose/sources.md) - the audio tracks a layout also names once they land
- [Audio source tracks](/quest/m0/audio-source-tracks.md) - the audio tracks a layout also names once they land
2 changes: 1 addition & 1 deletion quest/m0/compose/server.md → quest/m1/compose/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ and document it in `README.md`.

## Required

- [Layout track](/quest/m0/compose/layout.md) - what the server composes
- [Layout track](/quest/m1/compose/layout.md) - what the server composes