Why
The desktop environment generates click candidates and executes clicks. Document editing and file-dialog tasks require text entry, keyboard input, scrolling, and dialog interaction.
Scope
Extend the existing desktop environment and driver integration with the operations required by the task suite. Keep free-text generation with the existing chat model. Refresh the observation after execution and validate targets against the current window.
Add screenshot-based target selection for one task where accessibility information is insufficient. Evaluate this path with the multimodal Cua-S1 integration. Preserve dry-run behaviour and existing execution permissions.
Acceptance criteria
- Behavioural tests cover text entry, keyboard shortcuts, scrolling, dialog interaction, stale targets, and execution failures.
- The agent completes a document-editing task and a file-dialog task with independent checks of the saved result.
- A test case with insufficient accessibility information exercises screenshot-based target selection and verifies the selected target.
- Repeated runs report completion rate, total time, and failure reasons separately for Windows and macOS; untested configurations are identified.
- Drag-and-drop and multi-app workflows remain deferred until a selected task requires them.
Dependencies and starting points
Use the repeatable task suite linked below. The Cua-S1 4B 0.2 issue supplies the model integration for the screenshot evaluation; text and keyboard work can proceed independently.
s1a/desktop/env.py:94, s1a/desktop/driver.py, and s1a/agents/desktop.py.
Evaluation dependency: #13.
Multimodal integration dependency: #14.
Why
The desktop environment generates click candidates and executes clicks. Document editing and file-dialog tasks require text entry, keyboard input, scrolling, and dialog interaction.
Scope
Extend the existing desktop environment and driver integration with the operations required by the task suite. Keep free-text generation with the existing chat model. Refresh the observation after execution and validate targets against the current window.
Add screenshot-based target selection for one task where accessibility information is insufficient. Evaluate this path with the multimodal Cua-S1 integration. Preserve dry-run behaviour and existing execution permissions.
Acceptance criteria
Dependencies and starting points
Use the repeatable task suite linked below. The Cua-S1 4B 0.2 issue supplies the model integration for the screenshot evaluation; text and keyboard work can proceed independently.
s1a/desktop/env.py:94, s1a/desktop/driver.py, and s1a/agents/desktop.py.
Evaluation dependency: #13.
Multimodal integration dependency: #14.