Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,10 @@ Either shows at once, since the browser's processes dump no core.
**Reload** in Sources restarts a page, failed or not. A page's dropdown
lists, such as a `<select>`'s, draw over it and take clicks under
**Interact**, but it opens no other windows (`window.open` and new-tab links
do nothing).
do nothing). Copy and Cut there put the page's selected text on the clipboard,
and nothing from a password field. Except for a page copied on the GPU through
an X display, the text goes as plain text that the page's own copy handlers
cannot change.

An overlay is a title, lower third, or other graphic written as HTML and CSS,
drawn by the studio itself (through [Blitz](https://github.com/DioxusLabs/blitz))
Expand Down Expand Up @@ -785,13 +788,13 @@ Sources make the opposite trip. A frame already on the GPU is drawn where it
lies: PipeWire camera and VA-API decode DMA-BUFs on Linux, given a Vulkan
driver that imports them, and NV12 capture and VideoToolbox decode surfaces on
macOS. A browser source's page is copied on the GPU from the shared texture CEF
paints, given the same Vulkan DMA-BUF import. Every other source, every source
on Windows or on a GPU that cannot import, and every still image delivers its
frames from the CPU, which the compositor uploads. A browser source on that
path renders its page in software, WebGL included, so a WebGL page runs
slower there, and needs no X server or `$DISPLAY`: it runs under Wayland
without XWayland, and with no display at all. A page copied on the GPU still
opens the X display.
paints, given the same Vulkan DMA-BUF import and a display to paint through:
the Wayland one when `$WAYLAND_DISPLAY` is set, needing no XWayland, or else
the X one in `$DISPLAY`. Every other source, every source on Windows or on a
GPU that cannot import, and every still image delivers its frames from the
CPU, which the compositor uploads. A browser source takes that path with no
display at all too, and there renders its page in software, WebGL included, so
a WebGL page runs slower.

## Replay

Expand Down
10 changes: 4 additions & 6 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -138,12 +138,10 @@
pango
stdenv.cc.cc
];
# Opened at run time rather than linked.
runtimeDependencies = with pkgs; [
libGL
udev
vulkan-loader
];
# libEGL, which libcef.so opens at run time rather than links:
# without it the GPU process cannot start, and pages paint in
# memory. runtimeDependencies would reach only executables.
appendRunpaths = [ "${lib.getLib pkgs.libGL}/lib" ];
dontStrip = true;
installPhase = ''
runHook preInstall
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ repository.
- [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops
- [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository
- [Virtual camera](/quest/m0/vcam/README.md) - the program or a published scene exposed as a system camera on Linux, Windows, and macOS
- [Browser sources on the GPU without an X display](/quest/m0/cef-headless.md) - shared-texture pages run under Wayland without XWayland, and with no display the CPU path runs instead
- [Browser sources on the GPU under Mesa](/quest/m0/cef-headless.md) - shared-texture pages paint on Intel and AMD under Wayland, and a page that cannot paint fails loudly
- [Signing identities are available](/quest/m0/signing.md) - the Apple and Windows code-signing credentials packaging and the macOS camera need
- [Packaging](/quest/m0/packaging.md) - signed installers for macOS and Windows and a Flatpak for Linux
- [The persistable portal restore token is pinned](/quest/m0/pin-portal-token.md) - moq-video's restore token, for Wayland sources
Expand Down
68 changes: 40 additions & 28 deletions quest/m0/cef-headless.md
Original file line number Diff line number Diff line change
@@ -1,34 +1,46 @@
# [S] Browser sources on the GPU without an X display
# [S] Browser sources on the GPU under Mesa

## Goal

Browser sources copied on the GPU (shared DMA-BUF textures) run under a pure
Wayland session without XWayland, and a studio with no display at all still
runs its browser sources, as the CPU path already does.
Browser sources copied on the GPU (shared DMA-BUF textures) paint on Mesa
drivers (Intel or AMD) under Wayland, as they do on NVIDIA, and a page whose
GPU path cannot paint fails its source saying why instead of staying
`Starting`.

## Plan

From [moq-dev/studio#49](https://github.com/moq-dev/studio/pull/49), picked
2026-11-05 (maintainer, /quest-plan), m0 by the rule that fixes join m0.
[moq-dev/studio#55](https://github.com/moq-dev/studio/pull/55) moved the CPU
path to Chromium's headless Ozone platform and relayed Copy and Cut through
the studio's clipboard, so only the GPU path still opens an X display.

The GPU path cannot go headless: Chromium's headless Ozone platform allocates
`TestPixmap`s with no DMA-BUF
(`ui/ozone/platform/headless/headless_surface_factory.cc`), and in a trial its
GPU process failed to start EGL without a display, so pages fell back to
painting into memory (`Paint::Cpu`).

A trial on NixOS (RTX 3070 Ti, NVIDIA 580.178, GNOME on Wayland) on
2026-11-05 found `--ozone-platform=wayland`, with only `WAYLAND_DISPLAY` set,
painted shared textures (`Paint::Gpu`), and `a_page_draws_and_plays` passed.
On the same host the GPU path under XWayland (the default X11 Ozone platform,
`DISPLAY=:0`) never painted: "Can't create buffer -- gbm device is missing".

Recommended: pick the platform at setup in `Helper::new`, Wayland when
`WAYLAND_DISPLAY` is set, X11 when only `DISPLAY` is, and the CPU path when
neither is, which the Sources panel already shows as CPU. Check popups,
input, and the shared textures on Mesa (Intel or AMD) too, since the trial
covered only NVIDIA. No CI job has a GPU, so the PR records how it was
verified.
[moq-dev/studio#59](https://github.com/moq-dev/studio/pull/59) picks the
display at setup: Chromium's Wayland Ozone platform when `$WAYLAND_DISPLAY` is
set, X11 when only `$DISPLAY` is, and the CPU path with neither. It passes the
browser suite on the GPU path on NVIDIA (RTX 3070 Ti, GNOME on Wayland), and
on the CPU path with no display on NVIDIA and Intel.

Mesa did not paint in its only trial, on nixos2 (Intel UHD 630, Mesa 26.2.4,
beside an RTX 2070), under a headless sway on the Intel render node, since no
desktop session was logged in:

- Wayland: CEF's GPU process starts on Mesa's EGL, but the first shared image
fails to allocate ("Unable to allocate frame for first frame capture:
OOM?") and the page never paints. The same happens with glvnd pinned to
Mesa (`__EGL_VENDOR_LIBRARY_FILENAMES`) and with `--disable-gpu-sandbox`.
Release CEF strips the allocation's own log (a `DLOG` in
`media/gfx/renderable_mappable_shared_image_video_frame_pool.cc`).
- X11, under that sway's XWayland: shared textures import, but come back
empty (transparent pages, black WebGL).
- The same headless sway on the NVIDIA render node paints shared textures, so
the harness itself works.

First check whether a real Mesa desktop session paints; the headless sway on
a hybrid host may be the cause. If it does not, find which allocation fails
(a debug CEF build, or `chrome://gpu` in a CEF client on the same host).

When the capturer cannot allocate, CEF never paints and never says so. Fail
the page loudly instead, from a signal CEF or Chromium gives (no first-paint
timeout).

## Waiting

- Maintainer: log into a Wayland desktop session (GNOME or KDE) on a Mesa GPU,
such as nixos2's Intel, so an agent can run `binary(browser)` there with that
session's `WAYLAND_DISPLAY`. An agent cannot start a logged-in session on a
greeter. Check: `loginctl list-sessions` shows a user session on `seat0`.
42 changes: 31 additions & 11 deletions rs/moq-scene/src/browser/host.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use cef::*;
use rustix::process::{Resource, Rlimit, getrlimit, setrlimit};

use super::outbox::Commands;
use super::protocol::{Area, Channel, Command, Event, Format, Id, Layer, Open, Plane, Raster, Start, Texture};
use super::protocol::{Area, Channel, Command, Display, Event, Format, Id, Layer, Open, Plane, Raster, Start, Texture};
use super::{Button, Input, Modifiers};

/// What the helper asks of a page's focused frame: its selection, and whether
Expand Down Expand Up @@ -52,6 +52,9 @@ struct Shared {
/// never another page's.
painted: Mutex<mpsc::Receiver<()>>,
accelerated: bool,
/// Whether CEF's own copy reaches the desktop's clipboard: only through
/// an X display.
native_copy: bool,
}

impl Shared {
Expand Down Expand Up @@ -115,6 +118,13 @@ fn serve() -> Result<(), String> {
writer: Mutex::new(writer),
painted: Mutex::new(painted),
accelerated: matches!(start.raster, Raster::Gpu { .. }),
native_copy: matches!(
start.raster,
Raster::Gpu {
display: Display::X11,
..
}
),
});
// The studio shows why, rather than only that the helper exited.
browse(&shared, start, reader, painted_tx).inspect_err(|reason| _ = shared.send(&Event::Fatal(reason.clone()), &[]))
Expand Down Expand Up @@ -389,17 +399,20 @@ fn deliver(id: Id, input: Input) {
/// may be an iframe, to the desktop's clipboard.
fn copy(browser: &Browser, page: &Paged, cut: bool) {
let Some(frame) = browser.focused_frame() else { return };
// The GPU path opens a display, whose clipboard CEF's own is, and the
// page's copy and cut handlers decide what lands there.
if page.shared.accelerated {
// Through an X display, CEF's clipboard is the desktop's, and the page's
// copy and cut handlers decide what lands there.
if page.shared.native_copy {
match cut {
true => frame.cut(),
false => frame.copy(),
}
return;
}
// The CPU path opens none, so the frame's renderer answers with its
// selection, and the studio puts that on the desktop's clipboard.
// Elsewhere CEF's own copy reaches no desktop: headless Ozone, on the CPU
// path, has no clipboard, and Wayland's takes a selection only with the
// serial of an input event, which an offscreen page never receives. So
// the frame's renderer answers with its selection, and the studio puts
// that on the desktop's clipboard.
let Some(mut message) = process_message_create(Some(&COPY.into())) else {
return;
};
Expand Down Expand Up @@ -477,12 +490,19 @@ wrap_app! {
Some(&"no-user-gesture-required".into()),
);
match &self.raster {
// Paint on the studio's GPU, which is the only one it imports
// from.
Raster::Gpu { render_node: Some(node) } => {
command_line.append_switch_with_value(Some(&"render-node-override".into()), Some(&node.as_str().into()));
Raster::Gpu { render_node, display } => {
let platform = match display {
Display::Wayland => "wayland",
Display::X11 => "x11",
};
command_line.append_switch_with_value(Some(&"ozone-platform".into()), Some(&platform.into()));
// Paint on the studio's GPU, which is the only one it
// imports from.
if let Some(node) = render_node {
command_line
.append_switch_with_value(Some(&"render-node-override".into()), Some(&node.as_str().into()));
}
}
Raster::Gpu { render_node: None } => {}
// The pictures come back in memory anyway. Left to find a GPU,
// Chromium walks its fallbacks, and one can hang until its
// watchdog kills it seconds later, delaying the first paint.
Expand Down
83 changes: 52 additions & 31 deletions rs/moq-scene/src/browser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@
//! executable started again, which calls [`child`] first thing in `main` to
//! become one; CEF's own subprocesses start the same way. The helper paints
//! into a shared GPU texture the studio copies on its device, or into memory
//! the studio uploads when either side cannot share textures ([`Paint`] says
//! which). Its popups, such as a `<select>`'s list, arrive the same way apart
//! from the page, and the studio draws them over it. A page's sound joins the
//! program bus at the bus's rate.
//! the studio uploads when either side cannot share textures or there is no
//! display to paint them through ([`Paint`] says which). Its popups, such as a
//! `<select>`'s list, arrive the same way apart from the page, and the studio
//! draws them over it. A page's sound joins the program bus at the bus's rate.
//!
//! Linux only for now, and only with the `cef` feature: elsewhere a browser
//! source reports that it is unsupported.
Expand Down Expand Up @@ -87,12 +87,9 @@ pub struct Helper {
profile: PathBuf,
device: wgpu::Device,
queue: wgpu::Queue,
/// Whether pages paint into shared textures the device imports, rather
/// than into memory it uploads.
shared: bool,
/// The device's DRM render node, which CEF must paint on for the studio to
/// import what it paints.
render_node: Option<PathBuf>,
/// Where pages paint.
#[cfg(target_os = "linux")]
raster: protocol::Raster,
/// The helper, once a page has started it.
#[cfg(target_os = "linux")]
running: Arc<Mutex<Option<Arc<run::Instance>>>>,
Expand All @@ -114,34 +111,22 @@ impl Helper {
device: &wgpu::Device,
queue: &wgpu::Queue,
) -> Self {
let info = adapter.get_info();
// A software device holds a shared texture in memory anyway, so there
// the page paints in software too.
let shared = cfg!(feature = "cef")
&& info.device_type != wgpu::DeviceType::Cpu
&& device
.features()
.contains(wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF);
// A machine with two GPUs would otherwise paint on whichever CEF
// picks, and the studio cannot import from another GPU.
let render_node = (!info.device_pci_bus_id.is_empty())
.then(|| std::fs::canonicalize(format!("/dev/dri/by-path/pci-{}-render", info.device_pci_bus_id)).ok())
.flatten();
#[cfg(not(target_os = "linux"))]
let _ = adapter;
Self {
program: program.into(),
profile: profile.into(),
device: device.clone(),
queue: queue.clone(),
shared,
render_node,
#[cfg(target_os = "linux")]
raster: raster(adapter, device),
#[cfg(target_os = "linux")]
running: Default::default(),
}
}

/// Take what a page last copied or cut, once, for the desktop's
/// clipboard: a page that paints in memory has no display to put it there
/// itself.
/// clipboard, which CEF reaches itself only through an X display.
#[cfg(target_os = "linux")]
pub fn copied(&self) -> Option<String> {
let running = self.running.lock().unwrap();
Expand Down Expand Up @@ -170,13 +155,49 @@ impl Helper {
}
}

/// Where pages paint, decided once from what the studio's device and session
/// offer.
#[cfg(target_os = "linux")]
fn raster(adapter: &wgpu::Adapter, device: &wgpu::Device) -> protocol::Raster {
use protocol::{Display, Raster};

let info = adapter.get_info();
// A software device holds a shared texture in memory anyway, so there the
// page paints in software too.
let shared = cfg!(feature = "cef")
&& info.device_type != wgpu::DeviceType::Cpu
&& device
.features()
.contains(wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF);
if !shared {
return Raster::Cpu;
}
let set = |name: &str| std::env::var_os(name).is_some_and(|value| !value.is_empty());
let display = if set("WAYLAND_DISPLAY") {
Display::Wayland
} else if set("DISPLAY") {
Display::X11
} else {
// CEF's GPU process cannot allocate shared textures without one.
return Raster::Cpu;
};
// A machine with two GPUs would otherwise paint on whichever CEF picks,
// and the studio cannot import from another GPU.
let render_node = (!info.device_pci_bus_id.is_empty())
.then(|| std::fs::canonicalize(format!("/dev/dri/by-path/pci-{}-render", info.device_pci_bus_id)).ok())
.flatten()
.and_then(|node| node.into_os_string().into_string().ok());
Raster::Gpu { render_node, display }
}

/// How a page's pictures reach the compositor.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Paint {
/// A shared texture, copied on the GPU.
Gpu,
/// Pixels in memory, uploaded every frame: the fallback when the page's
/// GPU process or the studio's device cannot share textures.
/// Pixels in memory, uploaded every frame: the fallback when the studio
/// runs with no display, or the page's GPU process or the studio's device
/// cannot share textures.
Cpu,
}

Expand Down Expand Up @@ -317,10 +338,10 @@ pub enum Input {
/// Text typed, after the keyboard layout and any input method.
Text(String),
/// Copy the page's selection to the clipboard, through
/// [`Helper::copied`] when it paints in memory.
/// [`Helper::copied`] unless the page paints through an X display.
Copy,
/// Cut the page's selection to the clipboard, through
/// [`Helper::copied`] when it paints in memory.
/// [`Helper::copied`] unless the page paints through an X display.
Cut,
/// The page gained or lost keyboard focus.
Focus(bool),
Expand Down
18 changes: 16 additions & 2 deletions rs/moq-scene/src/browser/protocol.rs
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,29 @@ pub(crate) struct Start {
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub(crate) enum Raster {
/// On the GPU, into shared textures the studio imports: on the DRM render
/// node named, the studio's GPU, or else on the one CEF picks.
Gpu { render_node: Option<String> },
/// node named, the studio's GPU, or else on the one CEF picks. CEF's GPU
/// process needs `display` to allocate them.
Gpu {
render_node: Option<String>,
display: Display,
},
/// In software, into memory the studio uploads, with no display opened.
/// Startup never waits on a GPU process CEF cannot start, and WebGL runs
/// on SwiftShader.
#[default]
Cpu,
}

/// The display server the GPU path connects to, the one the studio runs on.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub(crate) enum Display {
/// `$WAYLAND_DISPLAY`, preferred: under XWayland, CEF's GPU process finds
/// no GBM device on some drivers, NVIDIA's among them, and never paints.
Wayland,
/// `$DISPLAY`.
X11,
}

/// A page to open.
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub(crate) struct Open {
Expand Down
Loading