Skip to content

[Feat] Integrate and evaluate Cua-S1 4B 0.2 #14

Description

@Yunaik

Why

The current cua backend loads Cua-S1 Nano. Cua-S1 4B 0.2 uses a Qwen3.5-4B base with separate text and multimodal adapters and requires a dedicated integration.

Scope

Integrate both 0.2 adapters through the existing decision-model interface while keeping Nano available. Map the offered options and returned probabilities through the shared validation path. Verify that screenshots from the agent observation reach the multimodal adapter.

Compare Nano, 0.2 text, 0.2 multimodal, Jev, and the chat-model baseline on identical browser and desktop tasks. Improved accuracy and speed are hypotheses to test.

Acceptance criteria

  1. Installation and run instructions specify dependencies, model revisions, hardware, device settings, and a reproducible smoke test.
  2. Both adapters run through the agent. Behavioural tests cover option mapping, invalid outputs, and image delivery.
  3. Evaluation reports completion rate, total task time, decision latency, and failure reasons for every model on the same tasks and budgets.
  4. Model loading, warm inference, and peak memory use are measured separately. API charges and local compute assumptions are stated.
  5. Results identify where each adapter is useful and document failed or unsupported configurations.

Dependencies and references

Use the repeatable browser and desktop task suite linked below. Existing click tasks can establish an initial baseline before desktop operations expand.

Current CUA backend, decision-model interface, and Cua-S1 4B 0.2 model card.

Evaluation dependency: #13.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions