Skip to content

[Feat] Extend desktop execution to typing, keyboard input, and scrolling #15

Description

@Yunaik

Why

The desktop environment generates click candidates and executes clicks. Document editing and file-dialog tasks require text entry, keyboard input, scrolling, and dialog interaction.

Scope

Extend the existing desktop environment and driver integration with the operations required by the task suite. Keep free-text generation with the existing chat model. Refresh the observation after execution and validate targets against the current window.

Add screenshot-based target selection for one task where accessibility information is insufficient. Evaluate this path with the multimodal Cua-S1 integration. Preserve dry-run behaviour and existing execution permissions.

Acceptance criteria

  1. Behavioural tests cover text entry, keyboard shortcuts, scrolling, dialog interaction, stale targets, and execution failures.
  2. The agent completes a document-editing task and a file-dialog task with independent checks of the saved result.
  3. A test case with insufficient accessibility information exercises screenshot-based target selection and verifies the selected target.
  4. Repeated runs report completion rate, total time, and failure reasons separately for Windows and macOS; untested configurations are identified.
  5. Drag-and-drop and multi-app workflows remain deferred until a selected task requires them.

Dependencies and starting points

Use the repeatable task suite linked below. The Cua-S1 4B 0.2 issue supplies the model integration for the screenshot evaluation; text and keyboard work can proceed independently.

s1a/desktop/env.py:94, s1a/desktop/driver.py, and s1a/agents/desktop.py.

Evaluation dependency: #13.

Multimodal integration dependency: #14.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions