Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
58e51d6
chore(quest): claim quest/m0/browser-source.md
kixelated Nov 2, 2026
d7cf5c4
wip: browser sources through a CEF helper process
kixelated Nov 2, 2026
1f85c72
feat(scene): browser sources through a CEF helper process
kixelated Nov 2, 2026
b5d04cd
fix(scene): keep a drag with its page, refuse popups, and stop the he…
kixelated Nov 2, 2026
c840cd7
fix(studio): hold a page's capture until every button it saw comes up
kixelated Nov 2, 2026
3ac29a1
Merge remote-tracking branch 'origin/main' into quest/m0/browser-source
kixelated Nov 3, 2026
6c46414
fix(scene): build on macOS, and pass the canvas rate to main's stall …
kixelated Nov 3, 2026
b24583f
fix(studio): keep a captured drag in the item it was pressed in
kixelated Nov 3, 2026
4929f02
Merge remote-tracking branch 'origin/main' into quest/m0/browser-source
kixelated Nov 3, 2026
0969ed8
Merge remote-tracking branch 'origin/main' into quest/m0/browser-source
kixelated Nov 3, 2026
9c3fde3
fix(studio): pass held keys and clipboard shortcuts to an interacted …
kixelated Nov 3, 2026
ceca566
fix(studio): send copy and cut to a page as commands
kixelated Nov 3, 2026
bf33346
Merge origin/main into quest/m0/browser-source
kixelated Nov 3, 2026
0603deb
fix(scene): never block on a browser helper that stops reading input
kixelated Nov 3, 2026
0dfc436
test(studio): run the browser tests under Xvfb in render-sw
kixelated Nov 3, 2026
d4c7ef2
fix(studio): leave a focused page its keys, and let go of it on focus…
kixelated Nov 3, 2026
bfb0fe6
Merge origin/main into quest/m0/browser-source
kixelated Nov 3, 2026
aa21795
Merge origin/main into quest/m0/browser-source
kixelated Nov 3, 2026
e1f1548
fix(scene): fail a browser source loud when its threads cannot start
kixelated Nov 3, 2026
fcf2c7d
docs: a browser source reaches viewers only in the program
kixelated Nov 3, 2026
6377a24
Merge origin/main into quest/m0/browser-source
kixelated Nov 3, 2026
8c6f6ce
Merge origin/main into quest/m0/browser-source
kixelated Nov 4, 2026
fabec03
fix(scene): deliver a page's releases once its helper catches up
kixelated Nov 4, 2026
724d438
fix(studio): map preview input through an item's crop
kixelated Nov 4, 2026
1eb1e94
chore(scene): drop the import the crop helper left unused
kixelated Nov 4, 2026
a69ab36
fix(studio): let go of the pages when the preview is hidden
kixelated Nov 4, 2026
0d74662
Merge origin/main into quest/m0/browser-source
kixelated Nov 4, 2026
e94b936
fix(scene): a page follows a live change to the canvas frame rate
kixelated Nov 4, 2026
a84b06c
fix(studio): forget a page that stops or restarts under the preview
kixelated Nov 4, 2026
2cd4f9c
Merge origin/main into quest/m0/browser-source
kixelated Nov 4, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
325 changes: 325 additions & 0 deletions Cargo.lock

Large diffs are not rendered by default.

50 changes: 36 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# MoQ Studio

`moq-studio` is a native streaming studio for Linux, macOS, and Windows. It
composes cameras, screens, windows, images, colors, and broadcasts from a relay
into scenes on the GPU, mixes their audio, and publishes the result to a
composes cameras, screens, windows, images, colors, web pages, and broadcasts
from a relay into scenes on the GPU, mixes their audio, and publishes the result to a
[MoQ](https://github.com/moq-dev/moq) relay. It is early and not packaged yet,
so build it from source. The moq crates come from the `moq` submodule:

Expand Down Expand Up @@ -31,6 +31,9 @@ cargo run --release -p moq-studio --features capture,openh264 -- \
- `capture` adds camera, display, window, microphone, and system-audio sources.
On Linux it links ALSA; add `pipewire` for PipeWire cameras. `openh264`
is the software H.264 fallback for a machine without a hardware encoder.
- `cef` adds browser sources, on Linux only for now. The build takes CEF from
`CEF_PATH`, which the dev shell sets, or else downloads it, and the binary
loads `libcef.so` from beside itself or from that `CEF_PATH`.
- On Linux, display and window sources capture X11 and XWayland only; a
Wayland screen or window cannot be picked yet.
- On macOS, the studio is not an app bundle yet, so the camera, microphone,
Expand Down Expand Up @@ -85,11 +88,11 @@ Every panel is a tab you can drag, split, or stack.

| Panel | Use it to |
| --- | --- |
| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. **Cut** puts the preview on program at once; **Fade** blends over the fade time set beside it. |
| Preview, Program | See the scene you are editing and the one that is live, and drag items in the preview to place them. **Cut** puts the preview on program at once; **Fade** blends over the fade time set beside it. **Interact** passes the mouse over the preview to the browser source under it instead, and the keyboard to the one last clicked. |
| Scenes | Pick the scene in preview, and add, rename (double-click), reorder, or remove one. A scene on program, or still fading out, cannot be removed, even by an undo, until program cuts away from it. |
| History | See every edit, and undo or redo back to any of them. |
| Items | Place, show, hide, and order what the preview scene or the top layer draws, back to front. Select an item to filter or copy it. |
| Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. |
| Sources | Add a source by hand, see whether each and its own track are live and where it is shown, clone one, and remove one. Select a source to filter it. A browser source shows whether its page is copied on the GPU or, as a fallback, uploaded from memory (CPU), in place of a track. |
| Devices | List this machine's cameras, displays, windows, and audio inputs, and add one as a source. |
| Guests | Invite a guest, show their join link as a QR code, see who joined, and take a guest out of every scene. |
| Filters | Crop the selected source or item, add, order, and set its filters, set an item's blend mode, and copy or paste filters. |
Expand Down Expand Up @@ -135,6 +138,15 @@ best video and audio its catalog lists, switches when the catalog changes, and
comes back on its own when the publisher restarts. Audio from every source is mixed into the program whether
or not its scene is live; mute a source to leave it out.

A browser source renders a web page, such as an overlay, alert box, or chat
widget, at the canvas frame rate (CEF caps it at 60) over a transparent
background, and its sound joins the program audio. Each page runs in a
process of its own, so a page or browser crash fails that source, with the
reason, while the stream carries on; edit or re-add the source to restart it.
Every start is a fresh profile: no cookies or logins persist. A page opens
no other windows (`window.open` and new-tab links do nothing), and its
dropdown lists, such as a `<select>`'s, are not drawn yet.

## Guests

Guests and phones join a room the studio owns on its relay, from a web page,
Expand Down Expand Up @@ -222,6 +234,12 @@ path = "logo.png" # relative to the scene file
kind = "broadcast"
path = "room/guest"

[sources.alerts]
kind = "browser"
url = "https://example.com/alerts"
width = 800 # the page's own size, whatever its items stretch it to
height = 600

[audio.guest]
gain = 0.5

Expand Down Expand Up @@ -265,10 +283,11 @@ rect = { x = 1600, y = 40, width = 280, height = 120 }
Source kinds are `camera`, `display`, and `microphone` (each with an optional
`id`, the default device without one), `window` (with an `id`), `system_audio`
(macOS), `image` (PNG or JPEG), `color` (`#rrggbb` or `#rrggbbaa`),
`broadcast`, `clone` (with the `source` it shows), `lut` (a 3D Adobe or
Resolve `.cube` file, which no item places, but which grades what a `lut`
filter names it in), and `replay`, where the Replay panel plays replays. The
Devices panel fills in ids. `[[top]]` items make up the top layer.
`broadcast`, `browser` (a `url`, and a `width` and `height` up to 4096),
`clone` (with the `source` it shows), `lut` (a 3D Adobe or Resolve `.cube`
file, which no item places, but which grades what a `lut` filter names it in),
and `replay`, where the Replay panel plays replays. The Devices panel fills in
ids. `[[top]]` items make up the top layer.

`[video.<source>]` sets a source's `crop` and `filters`, wherever it is shown,
and an item's `crop`, `filters`, and `blend` its own. A `crop` trims a fraction
Expand Down Expand Up @@ -382,7 +401,8 @@ color space moves to a new track, which its catalog entry follows. Images and
colors publish once, as the image file and as a one-pixel PNG. Removing or
editing a source ends its track. A track carries the plain capture, before any
filter, and a clone publishes none, since it shows its source's capture.
Microphones and system audio do not publish tracks of their own yet, and a
Microphones and system audio do not publish tracks of their own yet. A browser
source publishes none, so viewers see its page only in the program, and a
broadcast source is already its publisher's.

Every captured source opens an encode session as it starts, and consumer GPUs
Expand All @@ -400,9 +420,10 @@ otherwise.
Sources make the opposite trip. A frame already on the GPU is drawn where it
lies: PipeWire camera and VA-API decode DMA-BUFs on Linux, given a Vulkan
driver that imports them, and NV12 capture and VideoToolbox decode surfaces on
macOS. Every other source, every source on Windows or on a GPU that cannot
import, and every still image delivers its frames from the CPU, which the
compositor uploads.
macOS. A browser source's page is copied on the GPU from the shared texture CEF
paints, given the same Vulkan DMA-BUF import. Every other source, every source
on Windows or on a GPU that cannot import, and every still image delivers its
frames from the CPU, which the compositor uploads.

## Replay

Expand Down Expand Up @@ -431,8 +452,9 @@ from those, so a replay needs `--connect` and goes back at most that far. With
`--on-demand`, the program has no past from before someone watched it. A
broadcast source replays from what the studio received of it, and only over a
relay connection whose protocol version can address a track by time; an older
one refuses with the reason in the panel. An image or a color has no past to
replay. A replay source publishes each replay as its own track while it plays,
one refuses with the reason in the panel. An image, a color, or a web page has
no past to replay: a page publishes no track of its own, so replay the program
to show it. A replay source publishes each replay as its own track while it plays,
like a camera, so viewers composing the scene themselves see it too.

## Health
Expand Down
77 changes: 77 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,80 @@
]
);

# CEF for browser sources (`cef` feature), at the version the `cef`
# crate in Cargo.lock binds: bump them together. cef-dll-sys builds
# its wrapper from this instead of downloading CEF, and a dev build
# loads libcef.so from it. Patched like nixpkgs' cef-binary, whose
# install step expects GL libraries this release no longer ships.
cefArchive = {
x86_64-linux = {
name = "cef_binary_156.0.15+gcf671c0+chromium-156.0.8078.80_linux64_minimal";
hash = "sha256-aHP5dbsEcxZoDqGDmYDtW9TDPIzUzkV2HJQhGFZgU4g=";
sha1 = "09235427222e026a4f34399742df27c5d98a4f8a";
};
aarch64-linux = {
name = "cef_binary_156.0.15+gcf671c0+chromium-156.0.8078.80_linuxarm64_minimal";
hash = "sha256-gGZjqeyiqPe+GFSAzn53nFhktc3VK9d0iQU72xkz/rg=";
sha1 = "85602db8e813ad824b45ae54218476aa3d91d561";
};
};
cef = lib.optionalAttrs isLinux (
let
archive = cefArchive.${system};
in
pkgs.stdenv.mkDerivation {
pname = "cef";
version = "156.0.15";
src = pkgs.fetchurl {
url = "https://cef-builds.spotifycdn.com/${lib.escapeURL archive.name}.tar.bz2";
inherit (archive) hash;
};
nativeBuildInputs = [ pkgs.autoPatchelfHook ];
buildInputs = with pkgs; [
alsa-lib
at-spi2-atk
at-spi2-core
atk
cairo
cups
dbus
expat
glib
gtk3
libdrm
libgbm
libx11
libxcb
libxcomposite
libxdamage
libxext
libxfixes
libxkbcommon
libxrandr
libxshmfence
nspr
nss
pango
stdenv.cc.cc
];
# Opened at run time rather than linked.
runtimeDependencies = with pkgs; [
libGL
udev
vulkan-loader
];
dontStrip = true;
installPhase = ''
runHook preInstall
mkdir $out
cp -r Release/. Resources/. CMakeLists.txt cmake include libcef_dll $out/
# What cef-dll-sys checks the version against.
echo '{"type":"minimal","name":"${archive.name}.tar.bz2","sha1":"${archive.sha1}"}' > $out/archive.json
runHook postInstall
'';
}
);

# Where the shell's libasound looks for PCM plugins. ALSA's default
# device is a plugin on any desktop running a sound server, and the
# shell's alsa-lib only searches its own store path, so without this
Expand All @@ -108,6 +182,8 @@
jq
cmake
nasm
# cef-dll-sys builds CEF's C++ wrapper with Ninja.
ninja
pkg-config
# libclang for the bindgen builds (pipewire, VAAPI).
rustPlatform.bindgenHook
Expand Down Expand Up @@ -159,6 +235,7 @@

env = lib.optionalAttrs isLinux {
ALSA_PLUGIN_DIR = "${alsaPlugins}/lib/alsa-lib";
CEF_PATH = "${cef}";
LD_LIBRARY_PATH = lib.makeLibraryPath runtimeLibs;
};
};
Expand Down
12 changes: 8 additions & 4 deletions justfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
set shell := ["bash", "-euo", "pipefail", "-c"]

# The device and codec features off by default, which every check covers too.
features := "moq-scene/capture,moq-studio/capture,moq-studio/openh264"
# The device, codec, and browser features off by default, which every check
# covers too.
features := "moq-scene/capture,moq-studio/capture,moq-studio/openh264,moq-studio/cef"

# List the available development commands.
default:
Expand Down Expand Up @@ -44,8 +45,8 @@ fix:
bun biome check --write .
cargo fmt --all

# Composite scenes and publish a program through wgpu on Mesa's software Vulkan
# driver, so no GPU is needed.
# Composite scenes, render browser sources, and publish a program through wgpu
# on Mesa's software Vulkan driver, so no GPU is needed.
render-sw:
#!/usr/bin/env bash
set -euo pipefail
Expand All @@ -55,6 +56,9 @@ render-sw:
export LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}"
cargo nextest run --locked -p moq-scene --lib --run-ignored only -E 'test(/^compositor::tests::/)'
cargo nextest run --locked -p moq-studio --features openh264 --run-ignored only -E 'test(=engine::tests::the_program_reaches_a_relay) | test(=setup::tests::times_every_canvas)'
# CEF's browser process opens an X display even to paint offscreen.
xvfb=$(nix build --inputs-from . --no-link --print-out-paths nixpkgs#xvfb-run)
"$xvfb/bin/xvfb-run" -a cargo nextest run --locked -p moq-studio --features cef --run-ignored only -E 'binary(browser)'

# Move the moq submodule to moq's latest main (or a given commit) and refresh Cargo.lock.
upgrade-moq rev="origin/main":
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ repository.
- [Audio mixes](/quest/m0/mixes.md) - per-output mixes, meters, monitoring, and sync offsets
- [Audio filters](/quest/m0/audio-filters.md) - per-source noise suppression, gate, compressor, limiter, and plugins, configured from the studio
- [Recording](/quest/m0/recording.md) - each canvas's program, and optionally each source, recorded to disk alongside live outputs
- [Browser sources](/quest/m0/browser-source.md) - web overlays rendered by out-of-process CEF into the compositor through shared GPU textures
- [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops
- [Browser sources keep a profile](/quest/m0/browser-profile.md) - one persistent profile, so logins survive restarts, and a reload action
- [Browser source popups](/quest/m0/browser-popups.md) - dropdowns and other in-page popups draw and take input
- [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository
Expand Down
33 changes: 26 additions & 7 deletions quest/m0/browser-source.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# [L] Browser sources
# [L] Browser sources on macOS and Windows

## Goal

A browser source renders a URL (an overlay, alert box, or chat widget) into
the compositor at the canvas frame rate. The creator can interact with it,
and its audio joins the bus. A crash in the browser process drops that
source, never the stream.
Browser sources run on macOS and Windows as they do on Linux: a URL (an
overlay, alert box, or chat widget) renders into the compositor at the canvas
frame rate, the creator can interact with it, its audio joins the bus, and a
crash in the browser process drops that source, never the stream.

## Plan

Expand All @@ -19,5 +19,24 @@ Decided 2026-10-22: use CEF through `cef-rs`, desktop only.
Windows, an IOSurface on macOS, and a dmabuf on Linux. The CPU `OnPaint`
path is the fallback and shows in stats.
- **Process:** CEF runs out of process.
- **First check:** whether `cef-rs` exposes accelerated offscreen painting.
If it does not, upstream that before building on it.

Linux landed first (`moq_scene::browser`): one helper process per page,
which is the studio's executable started again, talking over a Unix socket
on its stdin that carries the DMA-BUF descriptors, and `cef-rs`'s
`osr_texture_import` copies each texture on the studio's device. What the
other platforms need on top:
- **Windows:** a channel without Unix sockets, such as the helper's stdio
pipes. The studio duplicates CEF's NT handle out of the helper with
`DuplicateHandle` before `cef-rs` imports it.
- **macOS:** the helper is CEF's browser process, so it needs an app bundle
with CEF's helper apps (`cef-rs`'s `bundle-cef-app`) and its own
NSApplication. An IOSurface crosses processes only as a Mach port
(`IOSurfaceCreateMachPort`), so the channel needs a Mach port rendezvous.

## Required

- [Studio on Windows](/quest/m0/app-windows.md) - the studio runs there at all; it already runs on macOS

## Related

- [Studio on macOS with capture granted](/quest/m0/app-macos.md) - the rest of the macOS app, which pages do not need
2 changes: 1 addition & 1 deletion quest/m0/overlays.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ sandboxed frame with scripts off and a content security policy that loads
only those carried resources. First check
whether Blitz can draw into a caller's wgpu texture without a readback, and
upstream that if it cannot. Arbitrary web pages and JavaScript graphics stay
[browser sources](/quest/m0/browser-source.md), baked into their own
browser sources (`moq_scene::browser`), baked into their own
track.

## Required
Expand Down
14 changes: 14 additions & 0 deletions rs/moq-scene/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@ capture = ["moq-video/capture", "moq-audio/capture"]
# moq-video's `pipewire`: it links libpipewire-0.3 at build time, so a Linux
# package turns it on where the build host has it.
pipewire = ["capture", "moq-video/pipewire", "moq-audio/pipewire"]
# Browser sources, rendered by CEF (the Chromium Embedded Framework) in a helper
# process, on Linux. Off by default: the build downloads the CEF runtime, or
# takes it from `CEF_PATH`, and compiles its C++ wrapper. Without it, a browser
# source reports that this build cannot render web pages.
cef = ["dep:cef"]

[dependencies]
bytemuck = { version = "1", features = ["derive"] }
Expand All @@ -48,6 +53,15 @@ tokio = { workspace = true, features = ["macros", "sync", "time"] }
toml = { workspace = true }
tracing = { workspace = true }

# The helper process a browser source runs CEF in, and the socket between them.
[target.'cfg(target_os = "linux")'.dependencies]
# `accelerated_osr` copies the page's shared texture into the studio's device.
cef = { version = "156.3", default-features = false, features = ["accelerated_osr"], optional = true }
postcard = { version = "1", default-features = false, features = ["use-std"] }
rustix = { version = "1", features = ["net", "process", "stdio"] }
# The helper's CEF profile, removed once the helper has exited.
tempfile = { workspace = true }

[dev-dependencies]
# An encoder on every machine, for the source track and replay tests.
moq-video = { workspace = true, features = ["openh264"] }
Expand Down
Loading