Skip to content

feat(cli): restore cursor overlay through page.evaluate - #2869

Open
shrey150 wants to merge 7 commits into
agent/browse-v4-4-runtimefrom
agent/browse-v4-cli-cursor-overlay
Open

feat(cli): restore cursor overlay through page.evaluate#2869
shrey150 wants to merge 7 commits into
agent/browse-v4-4-runtimefrom
agent/browse-v4-cli-cursor-overlay

Conversation

@shrey150

@shrey150 shrey150 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Restore Browse's visible cursor as a CLI-owned DOM overlay, without adding a cursor API to core Stagehand V4.

  • Keep the overlay implementation in one dedicated cursor-overlay.ts file.
  • Install it idempotently for the current document through page.evaluate(CURSOR_OVERLAY_SCRIPT) and for future navigations through page.addInitScript(...).
  • Keep injection in the top frame and update the marker from coordinate input, including when input lands inside a child frame.
  • Track mouse movement using a fixed, pointer-events-disabled element.

Stack (#2872)

  1. chore(cli): import Browse V3 baseline #2833 — exact Browse V3 baseline import
  2. refactor(cli): migrate Browse runtime and commands to Stagehand V4 #2834 — Stagehand V4 runtime and standard command parity
  3. feat(cli): restore cursor overlay through page.evaluate #2869 — CLI-owned cursor overlay
  4. fix(cli): restore V3 network capture through a CDP sidecar #2849 — CLI-private CDP sidecar; V3 network parity
  5. refactor(cli): remove coordinate XPath return flag #2835 — remove --return-xpath; supported V3 parity/release checkpoint
  6. test(evals): exercise the workspace V4 CLI #2838 — eval and packaging integration
  7. fix(cli): persist context names in Browserbase #2839 — managed Context names (fast-follow)
  8. refactor(cli): consume shared Functions core #2701 — shared Functions core consumer (fast-follow)

Why this is separate

The cursor is a self-contained optional visual feature with different review concerns from the combined V4 runtime/command migration: injected DOM/CSS, idempotency, event handling, and screenshot behavior. Keeping it additive on #2834 lets this feature be reviewed or reverted without disturbing browser lifecycle or commands.

E2E Test Matrix

The actual built CLI on final head 79d30de4b ran these flows against synthetic pages in real Browserbase browsers:

browse cursor
browse open <second-synthetic-page>
browse eval "...assert cursor registration survived navigation..."
browse mouse hover 380 270
browse eval "...read the top-level #__browse_cursor_overlay__ style..."
browse screenshot --path cursor.png
browse stop

The point (380, 270) was first asserted to be inside the iframe. The top-document DOM assertion then returned:

{ "exists": true, "left": "380px", "top": "270px", "pointerEvents": "none" }
Command / flow Observed output Confidence / sufficiency
browse cursor; navigate the same page; inspect the page-side registration Registration remained a function after navigation Proves addInitScript persisted the overlay behavior into a new document
Assert point inside iframe; browse mouse hover 380 270; inspect the top document One top-level marker at left: 380px, top: 270px, pointer-events: none Proves iframe-targeted coordinate input updates the top-level overlay
browse screenshot --path cursor.png 34,385-byte PNG with one visible cursor inside the dashed iframe area Visual proof; does not claim coverage of every iframe nesting pattern
Drag a draggable element onto a drop target whose handler navigates; inspect the landed URL mouse drag returned dragged: true and the target navigation was observed Proves the visual-only post-drag marker update cannot turn a successful navigational drag into a reported failure
browse stop Completed and the real Browserbase test session was cleaned up Covers live session cleanup
pnpm --filter browse lint Prettier, ESLint, and TypeScript passed Static verification for the exact head
pnpm --filter browse test 25 files / 387 tests passed Full Browse unit/integration suite

The screenshot visibly shows the single top-level cursor over iframe content after navigation:

Browse V4 CLI cursor overlay surviving navigation and pointing inside an iframe

The fixtures are synthetic and deterministic; no LLM path is involved.

@changeset-bot

changeset-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: f1545fe

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@shrey150
shrey150 changed the base branch from agent/browse-v4-standard-commands to agent/browse-v4-4-runtime August 31, 2026 23:07
@shrey150
shrey150 marked this pull request as ready for review September 1, 2026 17:13

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Architecture diagram
sequenceDiagram
    participant User as Browse CLI User
    participant CLI as Browse CLI
    participant Runtime as Runtime Command Handlers
    participant Manager as Page Manager
    participant Page as Active Browser Page
    participant DOM as Page DOM Overlay
    participant Browser as Browser Input and Renderer

    Note over CLI,DOM: CLI-owned cursor visualization (no Stagehand core cursor API)

    User->>CLI: browse cursor
    CLI->>Runtime: Dispatch cursor command
    Runtime->>Manager: Resolve activePage()
    Manager-->>Runtime: Active page
    Runtime->>Page: evaluate(CURSOR_OVERLAY_SCRIPT)
    Page->>DOM: Ensure fixed overlay exists

    alt Overlay already installed
        DOM-->>Page: Reuse __browse_cursor_overlay__
    else Overlay not installed
        DOM->>DOM: Create fixed SVG cursor element
        DOM->>DOM: Set aria-hidden, z-index, pointer-events none
        DOM->>DOM: Append overlay to document root
        DOM->>DOM: Install capture-phase mousemove listener
    end

    Page-->>Runtime: Evaluation complete
    Runtime-->>CLI: Return enabled=true
    CLI-->>User: Cursor overlay enabled

    User->>CLI: browse mouse hover x y
    CLI->>Browser: Move mouse to viewport coordinates
    Browser->>DOM: Dispatch mousemove event
    DOM->>DOM: Clamp clientX and clientY to non-negative values
    DOM->>DOM: Update fixed overlay left and top

    User->>CLI: browse screenshot --path cursor.png
    CLI->>Page: Capture current page
    Page->>Browser: Render DOM and overlay
    Browser-->>CLI: Screenshot including visible cursor
    CLI-->>User: Save cursor.png
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/cli/src/lib/driver/cursor-overlay.ts
Comment thread packages/cli/src/lib/driver/commands/runtime.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/cli/src/lib/driver/commands/runtime.ts
Comment thread packages/cli/src/lib/driver/commands/mouse.ts Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant