fix(browser): composite background agent tabs, actionable refusals, timed pointer paths by waleedlatif1 · Pull Request #8291 · simstudioai/sim

waleedlatif1 · 2026-09-25T08:15:59Z

Summary

Keep the agent's browser tab composited when no panel shows it. A tab view that is in no window has no compositor surface, so CDP mouse input never acknowledged and screenshots never captured: when the user viewed another tab, agent clicks failed after 5 s; when the browser panel was not open, every click stalled 5 s and every screenshot failed. Each chat's agent tab is now parked invisibly in the main window, adopted in place when the panel shows it, and released on every teardown path
Make refused clicks actionable: a click covered by an overlay lists the overlay's own snapshot controls (dismiss with one, retry the same id), and a click that lands on a nested control names that control's id. Only real overlays (dialogs, modals, fixed/sticky layers) list controls
browser_drag takes optional via points and durationMs, and browser_hover can target a viewport point along the same route with no button pressed, for sliders, physics, canvas hover, and cursor-following pages. A plain drag keeps its exact previous pace; batches refuse timed paths

Type of Change

Bug fix
New feature

Testing

A/B vs staging on gemini-3.8-flash with the production 100-step budget, paired episodes:
- WebGames 87 → 91 / 104; wins replicate on fresh seeds (block-stack 0/5 → 4/5, emoji-remember 3/5 → 5/5, color-harmony 2/5 → 3/5, verify-challenge 2/5 → 3/5)
- REAL 177 vs 177 over two full seeds (242 pairs, 8 flips each way, p = 1); no loss traced to these changes
- Capability 55 → 55 / 56
- Rounds, tool time, tokens, and cache hit rate unchanged; second-tab tasks now take under half the rounds (recipe-calculator 30 → 12, tab-sync 34 → 15)
Background-tab fix verified in real Electron: clicks in a background agent tab 5 s failure → 164 ms; with the panel closed 5.2 s → 167 ms; screenshots work in both
1,919 desktop tests pass (new tests fail without their fixes); type-check, lint, block registry, check:audits, docs manifest pass

Companion: simstudioai/mothership#528

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

A tab view that is in no window has no compositor surface, so CDP mouse input on it never acknowledges and screenshots never capture. The agent's tab was left that way whenever the user viewed another tab or had not opened the browser panel: clicks failed or stalled five seconds and screenshots failed. Each chat's agent tab is now parked invisibly in the main window, adopted in place when the panel shows it, and released on every tab teardown path.

A click refused because an overlay covers the target now lists the overlay's own snapshot controls, so the agent dismisses it and retries the same id without another snapshot. A click that lands on a nested control names that control's id. Only real overlays (dialogs, modals, fixed or sticky layers) list controls.

…paths browser_drag takes optional via points and a durationMs, and browser_hover can target a viewport point with the same route, moving with no button pressed. Pages that react to pointer speed or position (sliders, physics, canvas hover, cursor-following games) can now be driven the way Claude computer use and OpenAI's CUA allow. A plain drag keeps its exact previous pace; batches refuse timed paths so they cannot outlast the batch watchdog.

vercel · 2026-09-25T08:16:05Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 25, 2026 8:57am UTC

github-actions · 2026-09-25T08:16:33Z

⚠️ Cross-repo companion check

One or more companion PRs aren't merged into staging yet. Merging this without them will leave copilot and sim out of sync — merge them in lockstep.

❌ simstudioai/mothership#528 — OPEN, not merged (targets staging) — feat(browser): timed pointer paths and REAL harness fidelity

cubic-dev-ai

All reported issues were addressed across 11 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

greptile-apps · 2026-09-25T08:20:59Z

Confidence Score: 5/5

[High risk] Adds timed pointer paths and refusal messaging to browser automation.

The PR appears safe to merge; no outstanding finding or new actionable issue was established.

Summary

This PR keeps background agent tabs composited, makes refused clicks identify usable controls, and adds routed, timed pointer movement. Since the previous review, it also adds a pre-dispatch navigation check for coordinate hover and guards against empty snapshot-registry entries. Both previous findings are resolved in the current code.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Agent[Agent browser tool] --> Session[Chat-scoped automation tab]
  Session --> Panel{Panel showing tab?}
  Panel -->|Yes| Visible[Visible composited view]
  Panel -->|No| Parked[Invisible view parked in main window]
  Visible --> CDP[Trusted pointer input and capture]
  Parked --> CDP

_{Reviews (4) · Last reviewed commit: "fix(browser): refuse a coordinate hover ..."}

…s tracked across scopes - Timed drags and hover routes stop as soon as their tool call is cancelled; a drag still releases the button through its existing cleanup. - A coordinate hover refuses durationMs without via points instead of silently moving straight to the target. - A scope migration carries the chat's agent tab registration with it, so the migrated chat's old agent tab is released when it switches tabs. - Overlay controls are found through shadow roots and with any tag casing.

waleedlatif1 · 2026-09-25T08:29:18Z

@greptile

waleedlatif1 · 2026-09-25T08:29:19Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-25T08:29:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 11 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

…tabs in the main window - An already-aborted drag sends no input, and an abort during the settle hold takes the cancellation cleanup instead of dropping. - A coordinate hover re-checks the active tab right before it moves. - Parking an agent tab hands keyboard focus back to whichever of the Sim renderer or the visible page held it. - An agent tab detached from a secondary window is parked in the main window, which outlives it. - A framed target covered by a parent-page overlay lists that overlay's controls.

waleedlatif1 · 2026-09-25T08:45:17Z

@greptile

waleedlatif1 · 2026-09-25T08:45:18Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-25T08:45:23Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 11 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

The coordinate hover captures the navigation epoch before its page probes and rejects a cross-document navigation before moving the pointer, as browser_click_at and browser_drag do. The overlay helper skips the empty registry slots earlier snapshots leave.

waleedlatif1 · 2026-09-25T08:58:20Z

@greptile

waleedlatif1 · 2026-09-25T08:58:21Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-25T08:58:27Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 11 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 added 3 commits September 25, 2026 01:12

github-actions Bot added the requires-mothership-merge Has a companion PR on the mothership/copilot side — merge in lockstep label Sep 25, 2026

cubic-dev-ai Bot reviewed Sep 25, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 25, 2026

View reviewed changes

Comment thread apps/desktop/src/main/browser-agent/cdp.ts

Comment thread apps/desktop/src/main/browser-agent/driver.ts

vercel Bot temporarily deployed to Preview September 25, 2026 08:28 Inactive

cubic-dev-ai Bot reviewed Sep 25, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview September 25, 2026 08:44 Inactive

cubic-dev-ai Bot reviewed Sep 25, 2026

View reviewed changes

Comment thread apps/desktop/src/main/browser-agent/driver.ts Outdated

Comment thread apps/desktop/src/main/browser-agent/page-functions.ts

vercel Bot temporarily deployed to Preview September 25, 2026 08:57 Inactive

cubic-dev-ai Bot reviewed Sep 25, 2026

View reviewed changes

waleedlatif1 merged commit e6badaf into staging Sep 25, 2026
37 checks passed

waleedlatif1 deleted the feat/browser-agent-sota branch September 25, 2026 09:10

waleedlatif1 mentioned this pull request Sep 25, 2026

v0.9.1: browser agent improvements #8294

Merged

Conversation

waleedlatif1 commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 25, 2026

⚠️ Cross-repo companion check

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 25, 2026

Uh oh!

waleedlatif1 commented Sep 25, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 25, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 25, 2026

Uh oh!

waleedlatif1 commented Sep 25, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 25, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 25, 2026

Uh oh!

waleedlatif1 commented Sep 25, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 25, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Sep 25, 2026 •

edited

Loading

vercel Bot commented Sep 25, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 25, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading