Repository navigation
quest: program first; viewer-side composition moves to m1 #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
2d450b2
quest: program first; viewer-side composition moves to m1
kixelated 7d1f7c1
quest: state the new source-track rule in the epic's Required list
kixelated 04709f0
Merge remote-tracking branch 'origin/main' into plan/program-first
kixelated f2ad6fe
quest: arm a source for replay while it is on program; defer composit…
kixelated 6b9c94e
quest: an audio source on program feeds the mix; an independent sourc…
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # [S] Audio source tracks | ||
|
|
||
| ## Goal | ||
|
|
||
| Each microphone and system-audio source in a `moq-studio` scene has its own | ||
| capture as an Opus track in the studio's broadcast, listed in that source's | ||
| catalog under the studio catalog's `dev.moq.studio` section, beside the | ||
| program, as the picture sources already are. An isolated recording and a | ||
| replay of an audio source read that track. | ||
|
|
||
| ## Plan | ||
|
|
||
| Cameras, displays, windows, images, and colors already publish this way | ||
| (`rs/moq-scene/src/publish.rs`): each source's renditions sit in a nested hang | ||
| catalog keyed by source name, so the root `video` and `audio` sections stay | ||
| the program's. An audio source adds an `audio` rendition to its own nested | ||
| catalog the same way. | ||
|
|
||
| moq-audio's capture Producer/Consumer split (`capture::Producer`) feeds both | ||
| the bus and an encoder from one device open, rather than opening each device | ||
| twice. | ||
| Publish the capture after the source's filters but before its gain and mute | ||
| (decided 2026-11-02, maintainer): a recording or replay holds filtered audio, | ||
| as video bakes in its ML effects, and the levels stay the program's. | ||
|
|
||
| Changed 2026-11-03 (maintainer, on | ||
| [moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)): a source's | ||
| own track encodes only while something in the studio uses it, and reaches | ||
| the relay only when the creator publishes the source independently | ||
| ([Published independently](/quest/m0/publish-independently.md)). A source | ||
| on program counts as used (decided 2026-11-04, maintainer), so a replay finds | ||
| its past from when it went on: a picture source while the program scene shows | ||
| it, and an audio source while it feeds the program mix. A source published | ||
| independently counts as used too, since its track is what it publishes. | ||
| Otherwise a source encodes only while an isolated recording or a running | ||
| replay reads it, and replaying a source with no history is refused with the | ||
| reason. | ||
| This replaces the earlier always-on decision (2026-11-01, on | ||
| [moq-dev/moq#7554](https://github.com/moq-dev/moq/pull/7554)), because | ||
| encoding every source costs encoder sessions a consumer machine does not | ||
| have; an audio source follows the picture sources' rule. | ||
|
|
||
| Track names stay stable for a source's life, and an edited source that | ||
| changes what it shows gets a new name. | ||
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| # [S] Published independently | ||
|
|
||
| ## Goal | ||
|
|
||
| A source in `moq-studio` has a "Published independently" option, off by | ||
| default. Ticked, the source's own track is published to the relay for viewers | ||
| and services to fetch, and the source is left out of the composited program, | ||
| so a service can place a webcam or overlay itself. Unticked, the source | ||
| reaches viewers only through the program. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-03 (maintainer, on | ||
| [moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)): sources are | ||
| not published to the relay by default, since every encoding costs an encoder | ||
| session and consumer GPUs have few. A source published independently gets one | ||
| rendition, as sources do in the encode ladder, while the program keeps its | ||
| two sizes. It counts as used, so it encodes for as long as it is published | ||
| ([Audio source tracks](/quest/m0/audio-source-tracks.md) states the rule). | ||
|
|
||
| Low priority: this is for services that move webcams and overlays themselves, | ||
| and none exists yet. The option lives on the source, in the scene file, so it | ||
| travels with the scene. The option and its effect on the program change what a | ||
| user does, so `README.md` documents them. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Live encode control](/quest/m0/encode-control.md) - the encode ladder that makes source tracks local by default | ||
| - [Viewer-side composition](/quest/m1/compose/README.md) - the m1 epic that composes from independently published sources |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # m1: viewers compose | ||
|
|
||
| ## Goal | ||
|
|
||
| A studio broadcast can be composed by its viewers and by a server from the | ||
| sources' own tracks and a layout track, matching the program the studio | ||
| composites, so a viewer can toggle and prioritize elements. | ||
|
|
||
| ## Plan | ||
|
|
||
| Moved out of m0 on 2026-11-04 (maintainer): the program is how a studio | ||
| broadcast reaches viewers, and sources are not published to the relay by | ||
| default (decided 2026-11-03 on | ||
| [moq-dev/studio#21](https://github.com/moq-dev/studio/pull/21)), since | ||
| encoding every source for viewers costs more encoder sessions than a consumer | ||
| GPU has. Viewer-side composition waits until m0 ships. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Viewer-side composition](/quest/m1/compose/README.md) - sources as their own tracks plus a layout track, composed by viewers and by a server |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.