Skip to content

fix(android): use clipboard paste and verify text entry in input_text - #171

Open
vishnoorath wants to merge 1 commit into
google:mainfrom
vishnoorath:fix/android-text-entry-clipboard-paste
Open

vishnoorath wants to merge 1 commit into
google:mainfrom
vishnoorath:fix/android-text-entry-clipboard-paste

Conversation

@vishnoorath

Copy link
Copy Markdown

Summary

AndroidAdbDriver.input_text() reported success unconditionally and its clipboard tier was unreachable on local runs, so a text entry that did not land looked identical to one that did. The agent re-typed, saw an unchanged field, and retried indefinitely.

Two concrete defects:

  1. Clipboard tier was dead code locally. The tier was gated on if self._ui_adb_client:, but ctx.ui_adb_client is only populated under ARTEMIS_CLOUD_MODE=1 (artemis/drivers/factory.py). On a normal local run it is None, so every entry fell straight through to adb shell input text.

  2. Every branch did an unconditional return True. input text failures are silent — the shell call returns without error even when the target app discards the synthesized events. There was no signal that anything went wrong.

Reproduced against a Flutter app (com.nitara.farmer.uat, Android 12), where input keyevent 67 does nothing, so the field could not be cleared either — digits accumulated on every retry.

Changes

  • Build a clipboard-capable client lazily via create_screen_client(), so the clipboard tier is live outside cloud mode. It routes through the Artemis Accessibility Helper, whose setClipboard can write the system clipboard on every API level (only reads are restricted since Android 10).
  • Prefer clipboard + KEYCODE_PASTE; input text is retained as a fallback.
  • Clear through the helper's clear_text RPC when available, instead of 21 synthesized backspaces.
  • Verify the entry by reading the focused editable node back and return an honest success/failure, so a silent failure surfaces to the agent instead of looping. Masked password fields and nodes that cannot be inspected stay optimistic.
  • Retry once after warming the backend — the first RPC after a cold start can lose the race with the helper attach and spuriously return False.

Verification

On an Android 12 device with a Flutter app, entering a phone number and then a 4-box OTP code now completes in a single pass (13 steps, one run) instead of looping. Both values were confirmed by screenshot: the number field showed the entered value, and a single input_text('1111') call auto-advanced across four separate OTP boxes.

pytest tests/unit/test_drivers.py and the surrounding driver/model-service tests pass (9 passed).

Notes for review

  • _verify_focused_text() intentionally stays optimistic when the field cannot be judged — masked fields render dots, and a custom WebView may expose no editable node. The strict check only fires when we can positively see that nothing landed.
  • A related bug exists in the Accessibility Helper's findInputNode, which takes the first editable field rather than the focused one, so clear_text can report success while clearing a different field. That needs a Gradle rebuild to ship in the APK and is left out of this change.
  • artemis/drivers/android/input_ime.py has a near-duplicate type_text() implementation with no clipboard tier at all. It appears to be unused (no constructor call sites), so it was left untouched to keep this PR scoped to the live path.

input_text() always returned True, and its clipboard tier was gated on
self._ui_adb_client, which is only populated in cloud mode
(drivers/factory.py). On a normal local run that attribute is None, so the
clipboard branch was skipped entirely and every entry fell through to
`adb shell input text`.

Because the function reported success unconditionally, an entry that did not
land was indistinguishable from one that did. The agent would re-type, see
an unchanged field, and retry indefinitely - observed on a Flutter app
(com.nitara.farmer.uat) where `input keyevent 67` does nothing, so the field
never cleared either.

- Build a clipboard-capable client lazily via create_screen_client() so the
  clipboard tier is live locally. It routes through the Artemis
  Accessibility Helper, whose setClipboard can write the system clipboard at
  every API level (only reads are restricted since Android 10).
- Prefer clipboard + KEYCODE_PASTE; keep `input text` as a fallback.
- Clear through the helper's clear_text RPC when available instead of
  21 synthesized backspaces, which Flutter ignores.
- Verify the result by reading the focused editable node back, and return an
  honest success/failure so a silent failure surfaces instead of looping.
  Masked password fields and un-inspectable nodes stay optimistic.
- Retry once after warming the backend, since the first RPC after a cold
  start can lose the race with the helper attach and spuriously return False.

Verified on an Android 12 device: entering a phone number and a 4-box OTP
code into the Flutter app now completes in a single pass instead of looping.
@google-cla

google-cla Bot commented Oct 6, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant