Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -789,7 +789,9 @@ paints, given the same Vulkan DMA-BUF import. Every other source, every source
on Windows or on a GPU that cannot import, and every still image delivers its
frames from the CPU, which the compositor uploads. A browser source on that
path renders its page in software, WebGL included, so a WebGL page runs
slower there.
slower there, and needs no X server or `$DISPLAY`: it runs under Wayland
without XWayland, and with no display at all. A page copied on the GPU still
opens the X display.

## Replay

Expand Down
4 changes: 1 addition & 3 deletions justfile
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,7 @@ render-sw:
export LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}"
cargo nextest run --locked -p moq-scene --lib --run-ignored only -E 'test(/^compositor::tests::/) | test(/^browser::layers::tests::/)'
cargo nextest run --locked -p moq-studio --features openh264 --run-ignored only -E 'test(/^engine::tests::the_program_/) | test(=setup::tests::times_every_canvas)'
# CEF's browser process opens an X display even to paint offscreen.
xvfb=$(nix build --inputs-from . --no-link --print-out-paths nixpkgs#xvfb-run)
"$xvfb/bin/xvfb-run" -a cargo nextest run --locked -p moq-studio --features cef --run-ignored only -E 'binary(browser)'
cargo nextest run --locked -p moq-studio --features cef --run-ignored only -E 'binary(browser)'

# Move the moq submodule to moq's latest main (or a given commit) and refresh Cargo.lock and bun.lock.
upgrade-moq rev="origin/main":
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ repository.
- [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops
- [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository
- [Virtual camera](/quest/m0/vcam/README.md) - the program or a published scene exposed as a system camera on Linux, Windows, and macOS
- [Browser sources without an X display](/quest/m0/cef-headless.md) - CEF runs headless, so Wayland sessions and tests need no X server
- [Browser sources on the GPU without an X display](/quest/m0/cef-headless.md) - shared-texture pages run under Wayland without XWayland, and with no display the CPU path runs instead
- [Signing identities are available](/quest/m0/signing.md) - the Apple and Windows code-signing credentials packaging and the macOS camera need
- [Packaging](/quest/m0/packaging.md) - signed installers for macOS and Windows and a Flatpak for Linux
- [The persistable portal restore token is pinned](/quest/m0/pin-portal-token.md) - moq-video's restore token, for Wayland sources
Expand Down
44 changes: 25 additions & 19 deletions quest/m0/cef-headless.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,34 @@
# [S] Browser sources without an X display
# [S] Browser sources on the GPU without an X display

## Goal

Browser sources run with no X server or `$DISPLAY` on both the CPU path and
the GPU (shared DMA-BUF texture) path, so the studio's CEF helper works under a
pure Wayland session without XWayland and `just render-sw` no longer needs
`xvfb-run`. Copy and paste under Interact still reach the desktop clipboard.
Browser sources copied on the GPU (shared DMA-BUF textures) run under a pure
Wayland session without XWayland, and a studio with no display at all still
runs its browser sources, as the CPU path already does.

## Plan

From [moq-dev/studio#49](https://github.com/moq-dev/studio/pull/49), picked
2026-11-05 (maintainer, /quest-plan), m0 by the rule that fixes join m0. With
`--disable-gpu` alone CEF still fails with "Missing X server or $DISPLAY"; a
trial adding `--ozone-platform=headless` on the CPU path passed 10/10 runs
with no display at about 0.4 s, and the browser suite matched Xvfb.
2026-11-05 (maintainer, /quest-plan), m0 by the rule that fixes join m0.
[moq-dev/studio#55](https://github.com/moq-dev/studio/pull/55) moved the CPU
path to Chromium's headless Ozone platform and relayed Copy and Cut through
the studio's clipboard, so only the GPU path still opens an X display.

Check first, and record the result in the PR:
- Whether Copy and Cut under Interact still reach the desktop clipboard with
the headless platform; if not, the studio's own clipboard (egui) relays
them.
- Whether the GPU path can go headless too. If it cannot, land the CPU path
and rewrite this quest to the GPU path's remaining work, with what blocks
it.
The GPU path cannot go headless: Chromium's headless Ozone platform allocates
`TestPixmap`s with no DMA-BUF
(`ui/ozone/platform/headless/headless_surface_factory.cc`), and in a trial its
GPU process failed to start EGL without a display, so pages fell back to
painting into memory (`Paint::Cpu`).

Then drop the `xvfb-run` wrapper from the `render-sw` recipe in the
`justfile` (the nightly job inherits it through the recipe), and say in
`README.md` that browser sources need no X server.
A trial on NixOS (RTX 3070 Ti, NVIDIA 580.178, GNOME on Wayland) on
2026-11-05 found `--ozone-platform=wayland`, with only `WAYLAND_DISPLAY` set,
painted shared textures (`Paint::Gpu`), and `a_page_draws_and_plays` passed.
On the same host the GPU path under XWayland (the default X11 Ozone platform,
`DISPLAY=:0`) never painted: "Can't create buffer -- gbm device is missing".

Recommended: pick the platform at setup in `Helper::new`, Wayland when
`WAYLAND_DISPLAY` is set, X11 when only `DISPLAY` is, and the CPU path when
neither is, which the Sources panel already shows as CPU. Check popups,
input, and the shared textures on Mesa (Intel or AMD) too, since the trial
covered only NVIDIA. No CI job has a GPU, so the PR records how it was
verified.
169 changes: 152 additions & 17 deletions rs/moq-scene/src/browser/host.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,18 @@ use super::outbox::Commands;
use super::protocol::{Area, Channel, Command, Event, Format, Id, Layer, Open, Plane, Raster, Start, Texture};
use super::{Button, Input, Modifiers};

/// What the helper asks of a page's focused frame: its selection, and whether
/// to cut it.
const COPY: &str = "moq-studio-copy";
/// What the frame's renderer answers: the text it copied.
const COPIED: &str = "moq-studio-copied";

/// Run one of CEF's subprocesses (renderer, GPU, utility, zygote).
pub(super) fn subprocess() -> ExitCode {
let _ = api_hash(sys::CEF_API_VERSION_LAST, 0);
let args = args::Args::new();
let code = execute_process(Some(args.as_main_args()), None, std::ptr::null_mut());
let mut app = RendererApp::new(RendererCopy::new());
let code = execute_process(Some(args.as_main_args()), Some(&mut app), std::ptr::null_mut());
ExitCode::from(code.clamp(0, 255) as u8)
}

Expand Down Expand Up @@ -76,10 +83,15 @@ impl Paged {

thread_local! {
/// The open pages, on CEF's UI thread, where input is delivered.
static PAGES: RefCell<HashMap<Id, Browser>> = RefCell::new(HashMap::new());
static PAGES: RefCell<HashMap<Id, (Browser, Arc<Paged>)>> = RefCell::new(HashMap::new());
/// Whether the studio is gone, so the helper quits once its last page
/// closes.
static QUITTING: Cell<bool> = const { Cell::new(false) };
/// The number of the last copy asked of a renderer.
static COPIES: Cell<i32> = const { Cell::new(0) };
/// The copy whose answer the studio still wants. Only the newest is, so
/// a slow renderer never overwrites a newer copy, nor answers unasked.
static COPYING: Cell<Option<i32>> = const { Cell::new(None) };
}

fn serve() -> Result<(), String> {
Expand Down Expand Up @@ -259,6 +271,7 @@ fn open(shared: Arc<Shared>, id: Id, open: Open) {
let url = CefString::from(open.url.as_str());
let paged = Arc::new(Paged { shared, id, open });
let mut client = HelperClient::new(
paged.clone(),
HelperRender::new(paged.clone()),
HelperAudio::new(paged.clone()),
HelperLoad::new(paged.clone()),
Expand All @@ -273,7 +286,7 @@ fn open(shared: Arc<Shared>, id: Id, open: Open) {
None,
None,
) {
Some(browser) => _ = PAGES.with_borrow_mut(|pages| pages.insert(id, browser)),
Some(browser) => _ = PAGES.with_borrow_mut(|pages| pages.insert(id, (browser, paged))),
None => paged.fail("CEF failed to create the page".into()),
}
}
Expand All @@ -286,7 +299,7 @@ fn close(id: Id) {

/// Run `f` on page `id`'s host, if the page is open. Runs on the UI thread.
fn host(id: Id, f: impl FnOnce(BrowserHost)) {
if let Some(host) = PAGES.with_borrow(|pages| pages.get(&id).and_then(|browser| browser.host())) {
if let Some(host) = PAGES.with_borrow(|pages| pages.get(&id).and_then(|(browser, _)| browser.host())) {
f(host);
}
}
Expand All @@ -300,7 +313,7 @@ fn retime(id: Id, rate: u32) {

/// Hand one input event to page `id`. Runs on the UI thread.
fn deliver(id: Id, input: Input) {
let Some(browser) = PAGES.with_borrow(|pages| pages.get(&id).cloned()) else {
let Some((browser, page)) = PAGES.with_borrow(|pages| pages.get(&id).cloned()) else {
return;
};
let Some(host) = browser.host() else {
Expand Down Expand Up @@ -366,22 +379,40 @@ fn deliver(id: Id, input: Input) {
host.send_key_event(Some(&event));
}
}
// The focused frame holds the selection, which may be in an
// iframe.
Input::Copy => {
if let Some(frame) = browser.focused_frame() {
frame.copy();
}
}
Input::Cut => {
if let Some(frame) = browser.focused_frame() {
frame.cut();
}
}
Input::Copy => copy(&browser, &page, false),
Input::Cut => copy(&browser, &page, true),
Input::Focus(focused) => host.set_focus(focused.into()),
}
}

/// Copy, or cut if `cut`, the selection in the page's focused frame, which
/// may be an iframe, to the desktop's clipboard.
fn copy(browser: &Browser, page: &Paged, cut: bool) {
let Some(frame) = browser.focused_frame() else { return };
// The GPU path opens a display, whose clipboard CEF's own is, and the
// page's copy and cut handlers decide what lands there.
if page.shared.accelerated {
match cut {
true => frame.cut(),
false => frame.copy(),
}
return;
}
// The CPU path opens none, so the frame's renderer answers with its
// selection, and the studio puts that on the desktop's clipboard.
let Some(mut message) = process_message_create(Some(&COPY.into())) else {
return;
};
let number = COPIES.get().wrapping_add(1);
COPIES.set(number);
COPYING.set(Some(number));
if let Some(args) = message.argument_list() {
args.set_bool(0, cut.into());
args.set_int(1, number);
}
frame.send_process_message(ProcessId::RENDERER, Some(&mut message));
}

fn flags(modifiers: Modifiers) -> u32 {
use sys::cef_event_flags_t as F;
[
Expand Down Expand Up @@ -458,9 +489,14 @@ wrap_app! {
// WebGL then needs SwiftShader, which Chromium runs only when
// told to, since it JITs page shaders in the GPU process. A
// browser source runs the pages its user picked, not the web.
//
// Painting offscreen in software needs no display either,
// so none is opened, which lets the helper run with no X
// server, as under Wayland without XWayland.
Raster::Cpu => {
command_line.append_switch(Some(&"disable-gpu".into()));
command_line.append_switch(Some(&"enable-unsafe-swiftshader".into()));
command_line.append_switch_with_value(Some(&"ozone-platform".into()), Some(&"headless".into()));
}
}
}
Expand All @@ -483,8 +519,83 @@ wrap_browser_process_handler! {
}
}

wrap_app! {
struct RendererApp {
render: RenderProcessHandler,
}

impl App {
fn render_process_handler(&self) -> Option<RenderProcessHandler> {
Some(self.render.clone())
}
}
}

wrap_render_process_handler! {
struct RendererCopy;

impl RenderProcessHandler {
fn on_process_message_received(
&self,
_browser: Option<&mut Browser>,
frame: Option<&mut Frame>,
source_process: ProcessId,
message: Option<&mut ProcessMessage>,
) -> ::std::os::raw::c_int {
let (Some(frame), Some(message)) = (frame, message) else {
return 0;
};
if source_process != ProcessId::BROWSER || CefString::from(&message.name()).to_string() != COPY {
return 0;
}
let Some(args) = message.argument_list() else { return 1 };
let (cut, number) = (args.bool(0) != 0, args.int(1));
let copied = Arc::new(Mutex::new(String::new()));
frame.visit_dom(Some(&mut SelectionReader::new(cut, copied.clone())));
let Some(mut answer) = process_message_create(Some(&COPIED.into())) else {
return 1;
};
if let Some(args) = answer.argument_list() {
args.set_string(0, Some(&copied.lock().unwrap().as_str().into()));
args.set_int(1, number);
}
frame.send_process_message(ProcessId::BROWSER, Some(&mut answer));
// After reading the selection, which cutting removes.
if cut {
frame.cut();
}
1
}
}
}

wrap_domvisitor! {
struct SelectionReader {
cut: bool,
text: Arc<Mutex<String>>,
}

impl Domvisitor {
fn visit(&self, document: Option<&mut Domdocument>) {
let Some(document) = document else { return };
let focused = document.focused_node();
// A browser never copies a password field's text, and cuts only
// what it can delete.
let password = focused.as_ref().is_some_and(|node| {
node.is_form_control_element() != 0
&& node.form_control_element_type() == DomFormControlType::INPUT_PASSWORD
});
let editable = focused.is_some_and(|node| node.is_editable() != 0);
if !password && (editable || !self.cut) {
*self.text.lock().unwrap() = CefString::from(&document.selection_as_text()).to_string();
}
}
}
}

wrap_client! {
struct HelperClient {
page: Arc<Paged>,
render: RenderHandler,
audio: AudioHandler,
load: LoadHandler,
Expand Down Expand Up @@ -512,6 +623,30 @@ wrap_client! {
fn life_span_handler(&self) -> Option<LifeSpanHandler> {
Some(self.life.clone())
}

fn on_process_message_received(
&self,
_browser: Option<&mut Browser>,
_frame: Option<&mut Frame>,
source_process: ProcessId,
message: Option<&mut ProcessMessage>,
) -> ::std::os::raw::c_int {
let Some(message) = message else { return 0 };
if source_process != ProcessId::RENDERER || CefString::from(&message.name()).to_string() != COPIED {
return 0;
}
let Some(args) = message.argument_list() else { return 1 };
// Only the newest copy's answer, once.
if COPYING.get() != Some(args.int(1)) {
return 1;
}
COPYING.set(None);
let text = CefString::from(&args.string(0)).to_string();
if !text.is_empty() {
self.page.shared.send(&Event::Copied(text), &[]);
}
1
}
}
}

Expand Down
21 changes: 19 additions & 2 deletions rs/moq-scene/src/browser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,21 @@ impl Helper {
}
}

/// Take what a page last copied or cut, once, for the desktop's
/// clipboard: a page that paints in memory has no display to put it there
/// itself.
#[cfg(target_os = "linux")]
pub fn copied(&self) -> Option<String> {
let running = self.running.lock().unwrap();
running.as_ref()?.copied()
}

/// Take what a page last copied or cut: nothing, without browser sources.
#[cfg(not(target_os = "linux"))]
pub fn copied(&self) -> Option<String> {
None
}

/// Stop the helper, failing every page, and wait for it to exit, so the
/// profile is saved and free for the next studio. The next page to start
/// starts a new helper.
Expand Down Expand Up @@ -301,9 +316,11 @@ pub enum Input {
},
/// Text typed, after the keyboard layout and any input method.
Text(String),
/// Copy the page's selection to the clipboard.
/// Copy the page's selection to the clipboard, through
/// [`Helper::copied`] when it paints in memory.
Copy,
/// Cut the page's selection to the clipboard.
/// Cut the page's selection to the clipboard, through
/// [`Helper::copied`] when it paints in memory.
Cut,
/// The page gained or lost keyboard focus.
Focus(bool),
Expand Down
8 changes: 6 additions & 2 deletions rs/moq-scene/src/browser/protocol.rs
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,9 @@ pub(crate) enum Raster {
/// On the GPU, into shared textures the studio imports: on the DRM render
/// node named, the studio's GPU, or else on the one CEF picks.
Gpu { render_node: Option<String> },
/// In software, into memory the studio uploads. Startup never waits on a
/// GPU process CEF cannot start, and WebGL runs on SwiftShader.
/// In software, into memory the studio uploads, with no display opened.
/// Startup never waits on a GPU process CEF cannot start, and WebGL runs
/// on SwiftShader.
#[default]
Cpu,
}
Expand Down Expand Up @@ -103,6 +104,9 @@ pub(crate) enum Event {
Popup(Id, Option<Area>),
/// Interleaved samples at the [`Open`] rate and channel count.
Audio(Id, Vec<f32>),
/// The text a [`Input::Copy`] or [`Input::Cut`] took from a page, for the
/// studio's clipboard.
Copied(String),
/// A page cannot be shown, and why.
Failed(Id, String),
/// No page can be shown, and why. The helper exits next.
Expand Down
Loading