Skip to content
Merged
37 changes: 25 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -532,9 +532,9 @@ stream.

### Source tracks

Each camera, display, window, image, and color also has a track of its own, so
the studio can replay or record it on its own. These tracks stay in the studio: the
program is the only video it publishes to a relay. They are listed in a local
Each camera, display, window, image, color, microphone, and system audio also
has a track of its own, so the studio can replay or record it on its own. These
tracks stay in the studio: the program is all it publishes to a relay. They are listed in a local
catalog under the `dev.moq.studio` section's `sources`, by source name, each
as a hang catalog of its own on the program's clock.

Expand All @@ -547,9 +547,17 @@ changes size or color space moves to a new track, which its catalog entry
follows. Images and colors keep their picture once, as the image file and as a
one-pixel PNG. Removing or editing a source ends its track. A track carries the
plain capture, before any filter, and a clone has none, since it shows its
source's capture. Microphones and system audio have no tracks of their own yet,
nor does a browser source or an overlay, and a broadcast source is already its
publisher's.
source's capture. A browser source or an overlay has no track of its own, and a
broadcast source is already its publisher's.

A microphone or system audio encodes its track as Opus, from the same device
open the mixes hear, only while it is used: while the mix on program plays it
(unmuted and not left out of it), so a replay of it has its recent past, or
while a recording reads its track. The track carries the sound as captured,
before its gain, mute, and delay, which stay the mixes'. A buffer the capture
lost, or a stretch the source was out of use, is a gap on its track. Opus
encodes on the CPU, about 1% of one Apple M4 core for a stereo source and
half that for mono, and holds no encode session.

Every encoder holds an encode session, and consumer GPUs allow only a few
hardware sessions at once: 8 on a GeForce card, while a base Apple M4 has one
Expand Down Expand Up @@ -580,10 +588,11 @@ live or not. The recording has an encoder of its own, opened only while it
records, at full quality whatever the uplink grants the live stream, as OBS's
advanced output mode records apart from streaming. It costs one more encode
session and one more copy of each program frame while recording. With
**each source too**, every camera, display, and window records its
own track into a file of its own beside it,
**each source too**, every camera, display, window, microphone, and system
audio records its own track into a file of its own beside it,
`<scene file name> <UTC start time> <source>.mp4`, holding the capture before
any filter; a source being recorded encodes whether or not it is on program.
any filter, or the sound before its gain and mute; a source being recorded
encodes whether or not it is on program.
Broadcast sources, guests included, have no track of the studio's, so they get
no file of their own. A source added while recording
starts its file then, and one removed finishes its file then. A new canvas
Expand Down Expand Up @@ -623,7 +632,9 @@ scene of its own. In the Replay panel, pick the program or a source, a length
up to 30 seconds, and a speed of 100%, 50%, or 25%, then **Replay** plays that
many seconds up to the click in the replay source, which is transparent before
and after. **Cut** to its scene to put it on program. Sound replays only at
100%, through the replay source's strip in the Audio panel. A new replay cuts
100%, through the replay source's strip in the Audio panel, so a replay of a
microphone or system audio, which has only sound, plays only at 100%. A new
replay cuts
off the one playing, and a stretch the source was offline or paused replays
as the gap it was.

Expand All @@ -639,8 +650,10 @@ it is written, shortly after the click.

Nothing records in the background for this. The program's tracks keep their
last 30 seconds for late viewers anyway, and each source's track while it is on
program or recorded, and a replay copies from those, so a replay needs
`--connect` and goes back at most that far. With `--on-demand`, the program has
program (shown in a scene on air, or for a microphone or system audio, played by
the mix on program) or recorded, and a replay copies from those, so a replay
goes back at most that far. A replay of the program needs `--connect`; a local
source keeps its track without a relay. With `--on-demand`, the program has
no past from before someone watched it, and a source has none from before it
was last on program or recorded; a replay of nothing is refused, saying why. A
broadcast source replays from what the studio received of it, and only over a
Expand Down
1 change: 0 additions & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,6 @@ repository.

- [Studio on macOS with capture granted](/quest/m0/app-macos.md) - camera, screen, and audio capture run on a Mac, not only Linux
- [Studio on Windows](/quest/m0/app-windows.md) - the studio runs on Windows, not only Linux
- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks, for isolated recording and replay
- [Wayland display and window sources](/quest/m0/wayland-sources.md) - portal-picked sources that reopen without the picker
- [Keyframes](/quest/m0/animation.md) - keyframed item properties with easing, starting when their scene goes on program
- [Media file source](/quest/m0/media-file.md) - a local video or audio file plays as a source, looping or from the start on demand
Expand Down
6 changes: 6 additions & 0 deletions quest/m0/audio-filters.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ itself, its filters, and the out-of-process plugin host are moq-audio work in
This quest is the studio's side: the panel, the scene-file shape, and wiring
each source's chain into the program bus.

Each microphone's and system audio's own track encodes its capture, before
the bus applies gain and mute (`Driver::listen` in
`rs/moq-scene/src/source.rs`). The chain goes before that tap, so the track
carries filtered sound, as video bakes in its ML effects (decided 2026-11-02,
maintainer).

The scene file names filters and their settings, never a plugin's absolute
path, so a collection stays portable; a missing plugin loads as failed rather
than refusing the file.
Expand Down
44 changes: 0 additions & 44 deletions quest/m0/audio-source-tracks.md

This file was deleted.

6 changes: 5 additions & 1 deletion quest/m0/publish-independently.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@ not published to the relay by default, since every encoding costs an encoder
session and consumer GPUs have few. A source published independently gets one
rendition, as every source track does, while the program keeps its
two sizes. It counts as used, so it encodes for as long as it is published
([Audio source tracks](/quest/m0/audio-source-tracks.md) states the rule).
(decided 2026-11-04, maintainer): a source encodes while it is on program
(shown in a scene on air, or, for a microphone or system audio, played by the
mix on program) or while something reads its track, and naming a published
source among those on program (`Publish::set_on_program` in
`rs/moq-scene/src/publish.rs`) keeps it encoding.

Low priority: this is for services that move webcams and overlays themselves,
and none exists yet. The option lives on the source, in the scene file, so it
Expand Down
1 change: 0 additions & 1 deletion quest/m1/compose/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ independent sources layered over it) and replan those bullets.

## Required

- [Audio source tracks](/quest/m0/audio-source-tracks.md) - microphones and system audio get their own tracks beside the picture sources (an m0 quest), encoded while the studio uses them and published to the relay only when published independently
- [Layout track](/quest/m1/compose/layout.md) - the live scene state as a hang JSON track on the media clock, specified in the hang draft
- [Browser composition](/quest/m1/compose/browser.md) - `@moq/watch` composes a broadcast with a layout track through moq-scene's browser build
- [Server composition](/quest/m1/compose/server.md) - a headless compositor renders the program from sources and layout, headless
Expand Down
11 changes: 3 additions & 8 deletions quest/m1/compose/layout.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,10 @@ blend modes. Only a video filter that cannot render the same way in every
viewer is baked into a track of its own. A mask names a picture source's
track, and a LUT a `lut` source, whose `.cube` table publishes no track yet,
so a viewer needs one carrying it, as an image's file is. Audio is the other
way round: an audio source's track is published after its audio filters ([Audio source tracks](/quest/m0/audio-source-tracks.md)), so
way round: an audio source's track carries its sound after its audio filters
and before its gain and mute (decided 2026-11-02, maintainer), so
the layout carries only what is applied after that point, such as gain and
mute, and never names an audio filter for a viewer to apply again. The
picture source tracks are enough to start; audio source tracks join the
layout when they land.
mute, and never names an audio filter for a viewer to apply again.

A browser source publishes no track of its own (decided 2026-11-03,
maintainer, on [moq-dev/studio#8](https://github.com/moq-dev/studio/pull/8)).
Expand All @@ -48,7 +47,3 @@ a viewer on its last frame.
Only sources the creator publishes independently
([Published independently](/quest/m0/publish-independently.md)) have tracks a
viewer can fetch; the rest reach viewers only in the program.

## Related

- [Audio source tracks](/quest/m0/audio-source-tracks.md) - the audio tracks a layout also names once they land
Loading