diff --git a/README.md b/README.md index 97cc2bf..7e60437 100644 --- a/README.md +++ b/README.md @@ -789,7 +789,9 @@ paints, given the same Vulkan DMA-BUF import. Every other source, every source on Windows or on a GPU that cannot import, and every still image delivers its frames from the CPU, which the compositor uploads. A browser source on that path renders its page in software, WebGL included, so a WebGL page runs -slower there. +slower there, and needs no X server or `$DISPLAY`: it runs under Wayland +without XWayland, and with no display at all. A page copied on the GPU still +opens the X display. ## Replay diff --git a/justfile b/justfile index ef0a266..8f69f8f 100644 --- a/justfile +++ b/justfile @@ -56,9 +56,7 @@ render-sw: export LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" cargo nextest run --locked -p moq-scene --lib --run-ignored only -E 'test(/^compositor::tests::/) | test(/^browser::layers::tests::/)' cargo nextest run --locked -p moq-studio --features openh264 --run-ignored only -E 'test(/^engine::tests::the_program_/) | test(=setup::tests::times_every_canvas)' - # CEF's browser process opens an X display even to paint offscreen. - xvfb=$(nix build --inputs-from . --no-link --print-out-paths nixpkgs#xvfb-run) - "$xvfb/bin/xvfb-run" -a cargo nextest run --locked -p moq-studio --features cef --run-ignored only -E 'binary(browser)' + cargo nextest run --locked -p moq-studio --features cef --run-ignored only -E 'binary(browser)' # Move the moq submodule to moq's latest main (or a given commit) and refresh Cargo.lock and bun.lock. upgrade-moq rev="origin/main": diff --git a/quest/m0/README.md b/quest/m0/README.md index e407c0f..1437e69 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -100,7 +100,7 @@ repository. - [Browser sources on macOS and Windows](/quest/m0/browser-source.md) - the out-of-process CEF pages Linux has, with shared GPU textures on the other desktops - [Web studio](/quest/m0/web.md) - the full studio in wasm on WebGPU and WebCodecs, served from this repository - [Virtual camera](/quest/m0/vcam/README.md) - the program or a published scene exposed as a system camera on Linux, Windows, and macOS -- [Browser sources without an X display](/quest/m0/cef-headless.md) - CEF runs headless, so Wayland sessions and tests need no X server +- [Browser sources on the GPU without an X display](/quest/m0/cef-headless.md) - shared-texture pages run under Wayland without XWayland, and with no display the CPU path runs instead - [Signing identities are available](/quest/m0/signing.md) - the Apple and Windows code-signing credentials packaging and the macOS camera need - [Packaging](/quest/m0/packaging.md) - signed installers for macOS and Windows and a Flatpak for Linux - [The persistable portal restore token is pinned](/quest/m0/pin-portal-token.md) - moq-video's restore token, for Wayland sources diff --git a/quest/m0/cef-headless.md b/quest/m0/cef-headless.md index 0134084..fac245e 100644 --- a/quest/m0/cef-headless.md +++ b/quest/m0/cef-headless.md @@ -1,28 +1,34 @@ -# [S] Browser sources without an X display +# [S] Browser sources on the GPU without an X display ## Goal -Browser sources run with no X server or `$DISPLAY` on both the CPU path and -the GPU (shared DMA-BUF texture) path, so the studio's CEF helper works under a -pure Wayland session without XWayland and `just render-sw` no longer needs -`xvfb-run`. Copy and paste under Interact still reach the desktop clipboard. +Browser sources copied on the GPU (shared DMA-BUF textures) run under a pure +Wayland session without XWayland, and a studio with no display at all still +runs its browser sources, as the CPU path already does. ## Plan From [moq-dev/studio#49](https://github.com/moq-dev/studio/pull/49), picked -2026-11-05 (maintainer, /quest-plan), m0 by the rule that fixes join m0. With -`--disable-gpu` alone CEF still fails with "Missing X server or $DISPLAY"; a -trial adding `--ozone-platform=headless` on the CPU path passed 10/10 runs -with no display at about 0.4 s, and the browser suite matched Xvfb. +2026-11-05 (maintainer, /quest-plan), m0 by the rule that fixes join m0. +[moq-dev/studio#55](https://github.com/moq-dev/studio/pull/55) moved the CPU +path to Chromium's headless Ozone platform and relayed Copy and Cut through +the studio's clipboard, so only the GPU path still opens an X display. -Check first, and record the result in the PR: -- Whether Copy and Cut under Interact still reach the desktop clipboard with - the headless platform; if not, the studio's own clipboard (egui) relays - them. -- Whether the GPU path can go headless too. If it cannot, land the CPU path - and rewrite this quest to the GPU path's remaining work, with what blocks - it. +The GPU path cannot go headless: Chromium's headless Ozone platform allocates +`TestPixmap`s with no DMA-BUF +(`ui/ozone/platform/headless/headless_surface_factory.cc`), and in a trial its +GPU process failed to start EGL without a display, so pages fell back to +painting into memory (`Paint::Cpu`). -Then drop the `xvfb-run` wrapper from the `render-sw` recipe in the -`justfile` (the nightly job inherits it through the recipe), and say in -`README.md` that browser sources need no X server. +A trial on NixOS (RTX 3070 Ti, NVIDIA 580.178, GNOME on Wayland) on +2026-11-05 found `--ozone-platform=wayland`, with only `WAYLAND_DISPLAY` set, +painted shared textures (`Paint::Gpu`), and `a_page_draws_and_plays` passed. +On the same host the GPU path under XWayland (the default X11 Ozone platform, +`DISPLAY=:0`) never painted: "Can't create buffer -- gbm device is missing". + +Recommended: pick the platform at setup in `Helper::new`, Wayland when +`WAYLAND_DISPLAY` is set, X11 when only `DISPLAY` is, and the CPU path when +neither is, which the Sources panel already shows as CPU. Check popups, +input, and the shared textures on Mesa (Intel or AMD) too, since the trial +covered only NVIDIA. No CI job has a GPU, so the PR records how it was +verified. diff --git a/rs/moq-scene/src/browser/host.rs b/rs/moq-scene/src/browser/host.rs index 45c3469..5625ba6 100644 --- a/rs/moq-scene/src/browser/host.rs +++ b/rs/moq-scene/src/browser/host.rs @@ -17,11 +17,18 @@ use super::outbox::Commands; use super::protocol::{Area, Channel, Command, Event, Format, Id, Layer, Open, Plane, Raster, Start, Texture}; use super::{Button, Input, Modifiers}; +/// What the helper asks of a page's focused frame: its selection, and whether +/// to cut it. +const COPY: &str = "moq-studio-copy"; +/// What the frame's renderer answers: the text it copied. +const COPIED: &str = "moq-studio-copied"; + /// Run one of CEF's subprocesses (renderer, GPU, utility, zygote). pub(super) fn subprocess() -> ExitCode { let _ = api_hash(sys::CEF_API_VERSION_LAST, 0); let args = args::Args::new(); - let code = execute_process(Some(args.as_main_args()), None, std::ptr::null_mut()); + let mut app = RendererApp::new(RendererCopy::new()); + let code = execute_process(Some(args.as_main_args()), Some(&mut app), std::ptr::null_mut()); ExitCode::from(code.clamp(0, 255) as u8) } @@ -76,10 +83,15 @@ impl Paged { thread_local! { /// The open pages, on CEF's UI thread, where input is delivered. - static PAGES: RefCell> = RefCell::new(HashMap::new()); + static PAGES: RefCell)>> = RefCell::new(HashMap::new()); /// Whether the studio is gone, so the helper quits once its last page /// closes. static QUITTING: Cell = const { Cell::new(false) }; + /// The number of the last copy asked of a renderer. + static COPIES: Cell = const { Cell::new(0) }; + /// The copy whose answer the studio still wants. Only the newest is, so + /// a slow renderer never overwrites a newer copy, nor answers unasked. + static COPYING: Cell> = const { Cell::new(None) }; } fn serve() -> Result<(), String> { @@ -259,6 +271,7 @@ fn open(shared: Arc, id: Id, open: Open) { let url = CefString::from(open.url.as_str()); let paged = Arc::new(Paged { shared, id, open }); let mut client = HelperClient::new( + paged.clone(), HelperRender::new(paged.clone()), HelperAudio::new(paged.clone()), HelperLoad::new(paged.clone()), @@ -273,7 +286,7 @@ fn open(shared: Arc, id: Id, open: Open) { None, None, ) { - Some(browser) => _ = PAGES.with_borrow_mut(|pages| pages.insert(id, browser)), + Some(browser) => _ = PAGES.with_borrow_mut(|pages| pages.insert(id, (browser, paged))), None => paged.fail("CEF failed to create the page".into()), } } @@ -286,7 +299,7 @@ fn close(id: Id) { /// Run `f` on page `id`'s host, if the page is open. Runs on the UI thread. fn host(id: Id, f: impl FnOnce(BrowserHost)) { - if let Some(host) = PAGES.with_borrow(|pages| pages.get(&id).and_then(|browser| browser.host())) { + if let Some(host) = PAGES.with_borrow(|pages| pages.get(&id).and_then(|(browser, _)| browser.host())) { f(host); } } @@ -300,7 +313,7 @@ fn retime(id: Id, rate: u32) { /// Hand one input event to page `id`. Runs on the UI thread. fn deliver(id: Id, input: Input) { - let Some(browser) = PAGES.with_borrow(|pages| pages.get(&id).cloned()) else { + let Some((browser, page)) = PAGES.with_borrow(|pages| pages.get(&id).cloned()) else { return; }; let Some(host) = browser.host() else { @@ -366,22 +379,40 @@ fn deliver(id: Id, input: Input) { host.send_key_event(Some(&event)); } } - // The focused frame holds the selection, which may be in an - // iframe. - Input::Copy => { - if let Some(frame) = browser.focused_frame() { - frame.copy(); - } - } - Input::Cut => { - if let Some(frame) = browser.focused_frame() { - frame.cut(); - } - } + Input::Copy => copy(&browser, &page, false), + Input::Cut => copy(&browser, &page, true), Input::Focus(focused) => host.set_focus(focused.into()), } } +/// Copy, or cut if `cut`, the selection in the page's focused frame, which +/// may be an iframe, to the desktop's clipboard. +fn copy(browser: &Browser, page: &Paged, cut: bool) { + let Some(frame) = browser.focused_frame() else { return }; + // The GPU path opens a display, whose clipboard CEF's own is, and the + // page's copy and cut handlers decide what lands there. + if page.shared.accelerated { + match cut { + true => frame.cut(), + false => frame.copy(), + } + return; + } + // The CPU path opens none, so the frame's renderer answers with its + // selection, and the studio puts that on the desktop's clipboard. + let Some(mut message) = process_message_create(Some(©.into())) else { + return; + }; + let number = COPIES.get().wrapping_add(1); + COPIES.set(number); + COPYING.set(Some(number)); + if let Some(args) = message.argument_list() { + args.set_bool(0, cut.into()); + args.set_int(1, number); + } + frame.send_process_message(ProcessId::RENDERER, Some(&mut message)); +} + fn flags(modifiers: Modifiers) -> u32 { use sys::cef_event_flags_t as F; [ @@ -458,9 +489,14 @@ wrap_app! { // WebGL then needs SwiftShader, which Chromium runs only when // told to, since it JITs page shaders in the GPU process. A // browser source runs the pages its user picked, not the web. + // + // Painting offscreen in software needs no display either, + // so none is opened, which lets the helper run with no X + // server, as under Wayland without XWayland. Raster::Cpu => { command_line.append_switch(Some(&"disable-gpu".into())); command_line.append_switch(Some(&"enable-unsafe-swiftshader".into())); + command_line.append_switch_with_value(Some(&"ozone-platform".into()), Some(&"headless".into())); } } } @@ -483,8 +519,83 @@ wrap_browser_process_handler! { } } +wrap_app! { + struct RendererApp { + render: RenderProcessHandler, + } + + impl App { + fn render_process_handler(&self) -> Option { + Some(self.render.clone()) + } + } +} + +wrap_render_process_handler! { + struct RendererCopy; + + impl RenderProcessHandler { + fn on_process_message_received( + &self, + _browser: Option<&mut Browser>, + frame: Option<&mut Frame>, + source_process: ProcessId, + message: Option<&mut ProcessMessage>, + ) -> ::std::os::raw::c_int { + let (Some(frame), Some(message)) = (frame, message) else { + return 0; + }; + if source_process != ProcessId::BROWSER || CefString::from(&message.name()).to_string() != COPY { + return 0; + } + let Some(args) = message.argument_list() else { return 1 }; + let (cut, number) = (args.bool(0) != 0, args.int(1)); + let copied = Arc::new(Mutex::new(String::new())); + frame.visit_dom(Some(&mut SelectionReader::new(cut, copied.clone()))); + let Some(mut answer) = process_message_create(Some(&COPIED.into())) else { + return 1; + }; + if let Some(args) = answer.argument_list() { + args.set_string(0, Some(&copied.lock().unwrap().as_str().into())); + args.set_int(1, number); + } + frame.send_process_message(ProcessId::BROWSER, Some(&mut answer)); + // After reading the selection, which cutting removes. + if cut { + frame.cut(); + } + 1 + } + } +} + +wrap_domvisitor! { + struct SelectionReader { + cut: bool, + text: Arc>, + } + + impl Domvisitor { + fn visit(&self, document: Option<&mut Domdocument>) { + let Some(document) = document else { return }; + let focused = document.focused_node(); + // A browser never copies a password field's text, and cuts only + // what it can delete. + let password = focused.as_ref().is_some_and(|node| { + node.is_form_control_element() != 0 + && node.form_control_element_type() == DomFormControlType::INPUT_PASSWORD + }); + let editable = focused.is_some_and(|node| node.is_editable() != 0); + if !password && (editable || !self.cut) { + *self.text.lock().unwrap() = CefString::from(&document.selection_as_text()).to_string(); + } + } + } +} + wrap_client! { struct HelperClient { + page: Arc, render: RenderHandler, audio: AudioHandler, load: LoadHandler, @@ -512,6 +623,30 @@ wrap_client! { fn life_span_handler(&self) -> Option { Some(self.life.clone()) } + + fn on_process_message_received( + &self, + _browser: Option<&mut Browser>, + _frame: Option<&mut Frame>, + source_process: ProcessId, + message: Option<&mut ProcessMessage>, + ) -> ::std::os::raw::c_int { + let Some(message) = message else { return 0 }; + if source_process != ProcessId::RENDERER || CefString::from(&message.name()).to_string() != COPIED { + return 0; + } + let Some(args) = message.argument_list() else { return 1 }; + // Only the newest copy's answer, once. + if COPYING.get() != Some(args.int(1)) { + return 1; + } + COPYING.set(None); + let text = CefString::from(&args.string(0)).to_string(); + if !text.is_empty() { + self.page.shared.send(&Event::Copied(text), &[]); + } + 1 + } } } diff --git a/rs/moq-scene/src/browser/mod.rs b/rs/moq-scene/src/browser/mod.rs index 560ab07..97b514f 100644 --- a/rs/moq-scene/src/browser/mod.rs +++ b/rs/moq-scene/src/browser/mod.rs @@ -139,6 +139,21 @@ impl Helper { } } + /// Take what a page last copied or cut, once, for the desktop's + /// clipboard: a page that paints in memory has no display to put it there + /// itself. + #[cfg(target_os = "linux")] + pub fn copied(&self) -> Option { + let running = self.running.lock().unwrap(); + running.as_ref()?.copied() + } + + /// Take what a page last copied or cut: nothing, without browser sources. + #[cfg(not(target_os = "linux"))] + pub fn copied(&self) -> Option { + None + } + /// Stop the helper, failing every page, and wait for it to exit, so the /// profile is saved and free for the next studio. The next page to start /// starts a new helper. @@ -301,9 +316,11 @@ pub enum Input { }, /// Text typed, after the keyboard layout and any input method. Text(String), - /// Copy the page's selection to the clipboard. + /// Copy the page's selection to the clipboard, through + /// [`Helper::copied`] when it paints in memory. Copy, - /// Cut the page's selection to the clipboard. + /// Cut the page's selection to the clipboard, through + /// [`Helper::copied`] when it paints in memory. Cut, /// The page gained or lost keyboard focus. Focus(bool), diff --git a/rs/moq-scene/src/browser/protocol.rs b/rs/moq-scene/src/browser/protocol.rs index dfacf28..dbc0195 100644 --- a/rs/moq-scene/src/browser/protocol.rs +++ b/rs/moq-scene/src/browser/protocol.rs @@ -62,8 +62,9 @@ pub(crate) enum Raster { /// On the GPU, into shared textures the studio imports: on the DRM render /// node named, the studio's GPU, or else on the one CEF picks. Gpu { render_node: Option }, - /// In software, into memory the studio uploads. Startup never waits on a - /// GPU process CEF cannot start, and WebGL runs on SwiftShader. + /// In software, into memory the studio uploads, with no display opened. + /// Startup never waits on a GPU process CEF cannot start, and WebGL runs + /// on SwiftShader. #[default] Cpu, } @@ -103,6 +104,9 @@ pub(crate) enum Event { Popup(Id, Option), /// Interleaved samples at the [`Open`] rate and channel count. Audio(Id, Vec), + /// The text a [`Input::Copy`] or [`Input::Cut`] took from a page, for the + /// studio's clipboard. + Copied(String), /// A page cannot be shown, and why. Failed(Id, String), /// No page can be shown, and why. The helper exits next. diff --git a/rs/moq-scene/src/browser/run.rs b/rs/moq-scene/src/browser/run.rs index 27e20f7..173ec33 100644 --- a/rs/moq-scene/src/browser/run.rs +++ b/rs/moq-scene/src/browser/run.rs @@ -107,6 +107,9 @@ struct Routes { pages: HashMap, /// Why the helper stopped, once it has: no page opens in it after. ended: Option, + /// What a page last copied or cut, until the studio takes it. One slot + /// for every page, so a newer copy always replaces an older one. + copied: Option, } /// One open page's source. @@ -276,6 +279,11 @@ impl Instance { Ok(opened) } + /// Take what a page last copied or cut, once. + pub(super) fn copied(&self) -> Option { + self.routes.lock().unwrap().copied.take() + } + /// Ask the helper to exit, and wait until it has. pub(super) fn stop(&self) { self.end(); @@ -454,6 +462,10 @@ fn deliver(gpu: &Gpu, events: &Channel, commands: &Commands, routes: &Mutex { + routes.lock().unwrap().copied = Some(text); + continue; + } Event::Failed(id, reason) => { routes.lock().unwrap().end(id, reason); continue; diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 6a498e1..19152c8 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -197,6 +197,17 @@ impl eframe::App for Studio { let framerate = state.compositor.canvas().framerate; ui.ctx().request_repaint_after(Duration::from_secs(1) / framerate); self.panels.follow_reloads(&state); + // A page painting in memory copies through the studio's clipboard, + // even if the preview let go of the page before the copy came back. + if let Some(text) = state + .sources + .context() + .browser + .as_ref() + .and_then(|helper| helper.copied()) + { + ui.ctx().copy_text(text); + } drop(state); self.panels.shortcuts(ui.ctx()); diff --git a/rs/moq-studio/tests/browser.rs b/rs/moq-studio/tests/browser.rs index 29effe4..bfe3baa 100644 --- a/rs/moq-studio/tests/browser.rs +++ b/rs/moq-studio/tests/browser.rs @@ -325,6 +325,92 @@ async fn a_dropdown_draws_and_takes_clicks() { .await; } +/// Copy and Cut hand the page's selection to the studio for its clipboard, +/// and Cut deletes it from the page. A password field's text is never taken. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs CEF and a Vulkan device; run by just render-sw"] +async fn copy_and_cut_reach_the_clipboard() { + local(async { + // A text box over the whole page, and a password field below it. The + // page turns yellow once the text box is selected, lime once it is + // typed in, and blue and white for the same on the password. + let html = "\ + \ + "; + let mut studio = Studio::new(&[("page", page(html))]).await; + studio.until_center([255, 0, 0, 255], &[]).await; + + let page = studio.sources.get("page").unwrap().page().unwrap().clone(); + let key = |code: u8, modifiers: Modifiers| { + for down in [true, false] { + page.send(Input::Key { + code: code.into(), + down, + modifiers, + }); + } + }; + let ctrl = Modifiers { + ctrl: true, + ..Default::default() + }; + let shift = Modifiers { + shift: true, + ..Default::default() + }; + let helper = studio.helper.clone(); + let copied = async || { + let deadline = Instant::now() + TIMEOUT; + loop { + if let Some(copied) = helper.copied() { + return copied; + } + assert!(Instant::now() < deadline, "the copy never came back"); + kio::clock::sleep(Duration::from_millis(30)).await; + } + }; + + page.send(Input::Focus(true)); + for down in [true, false] { + page.send(Input::Button { + x: 160, + y: 120, + button: Button::Left, + down, + clicks: 1, + modifiers: Modifiers::default(), + }); + } + // Copy is asked of the page only once it shows the selection, which + // the renderer then has. + key(b'A', ctrl); + studio.until_center(YELLOW, &[]).await; + page.send(Input::Copy); + assert_eq!(copied().await, "hello world"); + + // Text typed after the copy shows only once the copy has come back. + key(b'\t', Modifiers::default()); + key(b'A', ctrl); + studio.until_center(BLUE, &[]).await; + page.send(Input::Copy); + page.send(Input::Text("x".into())); + studio.until_center([255, 255, 255, 255], &[]).await; + assert_eq!(helper.copied(), None, "a password was copied"); + + key(b'\t', shift); + key(b'A', ctrl); + studio.until_center(YELLOW, &[]).await; + page.send(Input::Cut); + studio.until_center(GREEN, &[]).await; + assert_eq!(copied().await, "hello world"); + }) + .await; +} + /// A page that closes itself fails its own source and leaves the canvas, /// while the page beside it in the same helper keeps drawing. #[tokio::test(flavor = "multi_thread")] @@ -450,6 +536,7 @@ async fn cookies_survive_a_restart() { } const GREEN: [u8; 4] = [0, 255, 0, 255]; +const YELLOW: [u8; 4] = [255, 255, 0, 255]; /// Serve `html` over HTTP on loopback to every request, returning its URL, /// since a cookie needs an origin a `data:` page lacks.