Repository navigation
fix(android): use clipboard paste and verify text entry in input_text - #171
Open
vishnoorath wants to merge 1 commit into
Open
vishnoorath wants to merge 1 commit into
vishnoorath wants to merge 1 commit into
Conversation
input_text() always returned True, and its clipboard tier was gated on self._ui_adb_client, which is only populated in cloud mode (drivers/factory.py). On a normal local run that attribute is None, so the clipboard branch was skipped entirely and every entry fell through to `adb shell input text`. Because the function reported success unconditionally, an entry that did not land was indistinguishable from one that did. The agent would re-type, see an unchanged field, and retry indefinitely - observed on a Flutter app (com.nitara.farmer.uat) where `input keyevent 67` does nothing, so the field never cleared either. - Build a clipboard-capable client lazily via create_screen_client() so the clipboard tier is live locally. It routes through the Artemis Accessibility Helper, whose setClipboard can write the system clipboard at every API level (only reads are restricted since Android 10). - Prefer clipboard + KEYCODE_PASTE; keep `input text` as a fallback. - Clear through the helper's clear_text RPC when available instead of 21 synthesized backspaces, which Flutter ignores. - Verify the result by reading the focused editable node back, and return an honest success/failure so a silent failure surfaces instead of looping. Masked password fields and un-inspectable nodes stay optimistic. - Retry once after warming the backend, since the first RPC after a cold start can lose the race with the helper attach and spuriously return False. Verified on an Android 12 device: entering a phone number and a 4-box OTP code into the Flutter app now completes in a single pass instead of looping.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AndroidAdbDriver.input_text()reported success unconditionally and its clipboard tier was unreachable on local runs, so a text entry that did not land looked identical to one that did. The agent re-typed, saw an unchanged field, and retried indefinitely.Two concrete defects:
Clipboard tier was dead code locally. The tier was gated on
if self._ui_adb_client:, butctx.ui_adb_clientis only populated underARTEMIS_CLOUD_MODE=1(artemis/drivers/factory.py). On a normal local run it isNone, so every entry fell straight through toadb shell input text.Every branch did an unconditional
return True.input textfailures are silent — the shell call returns without error even when the target app discards the synthesized events. There was no signal that anything went wrong.Reproduced against a Flutter app (
com.nitara.farmer.uat, Android 12), whereinput keyevent 67does nothing, so the field could not be cleared either — digits accumulated on every retry.Changes
create_screen_client(), so the clipboard tier is live outside cloud mode. It routes through the Artemis Accessibility Helper, whosesetClipboardcan write the system clipboard on every API level (only reads are restricted since Android 10).KEYCODE_PASTE;input textis retained as a fallback.clear_textRPC when available, instead of 21 synthesized backspaces.False.Verification
On an Android 12 device with a Flutter app, entering a phone number and then a 4-box OTP code now completes in a single pass (13 steps, one run) instead of looping. Both values were confirmed by screenshot: the number field showed the entered value, and a single
input_text('1111')call auto-advanced across four separate OTP boxes.pytest tests/unit/test_drivers.pyand the surrounding driver/model-service tests pass (9 passed).Notes for review
_verify_focused_text()intentionally stays optimistic when the field cannot be judged — masked fields render dots, and a customWebViewmay expose no editable node. The strict check only fires when we can positively see that nothing landed.findInputNode, which takes the first editable field rather than the focused one, soclear_textcan report success while clearing a different field. That needs a Gradle rebuild to ship in the APK and is left out of this change.artemis/drivers/android/input_ime.pyhas a near-duplicatetype_text()implementation with no clipboard tier at all. It appears to be unused (no constructor call sites), so it was left untouched to keep this PR scoped to the live path.