Read text in images without Google Play services

[jpavonabian]

Problem

Describe image reads text in images with ML Kit, using play-services-mlkit-text-recognition. That artifact delegates to the ML Kit module inside Google Play services. On phones without Play services, such as GrapheneOS without sandboxed Google Play, text recognition is unavailable, so Describe image can't read any text.

Change

Switch the dependency to the bundled com.google.mlkit:text-recognition:16.0.1. The text recognition model now ships inside the app, so it works whether or not Play services is installed.

The API and package names are the same, so the Java code is unchanged. The change is in gradle/libs.versions.toml, shared.gradle, a section in differences.md, and talkback/src/main/AndroidManifest.xml (see Trade-off).

The model is bundled only in the phone build. Describe image and automatic descriptions are turned off on watches (supportsImageCaption() needs supportDynamicFeatures(), which is false there), and Describe screen, which watches do have, only sends the screenshot to Gemini without text recognition. So nothing in the watch interface reaches text recognition. The watch build keeps ML Kit on its compile classpath only (wearCompileOnly), so the code still compiles, and the APK doesn't include it.

Trade-off

The phone APK grows by about 30 MB, because the model is built for all three ABIs (armeabi-v7a, arm64-v8a, x86_64). The watch APK doesn't change. Measured on debug builds, before and after this change:

APK	Before	After	Difference
phone	106 MiB	135 MiB	+29.5 MiB
watch	80 MiB	80 MiB	none

Almost all of it is libmlkit_google_ocr_pipeline.so (11 MB arm64-v8a, 6.8 MB armeabi-v7a, 11.6 MB x86_64). differences.md gives these numbers.

The phone release build (assemblePhoneRelease, unsigned) grows the same: 108.8 MiB to 138.3 MiB (+29.5 MiB). Backtalk doesn't minify, so release builds are only about 3 MiB larger than debug builds. The debug APKs are the ones published.

ML Kit records usage events and uploads them to firebaselogging.googleapis.com through Firebase datatransport. The datatransport components were already in the merged manifest before this change, through play-services-mlkit-text-recognition-common, so the upload already came from Backtalk itself. On an x86_64 emulator, with this change alone, the app stored FIREBASE_ML_SDK events, scheduled two upload jobs and tried to connect to firebaselogging.googleapis.com/v0cc/log/batch.

The second commit removes TransportBackendDiscovery, JobInfoSchedulerService and AlarmManagerSchedulerBroadcastReceiver from the manifest. With that, the emulator logs Transport backend 'cct' is not registered, schedules no upload jobs and makes no request, and text recognition still reads the same test image. I also ran the build before this change (with play-services-mlkit-text-recognition) on the same emulator: it stores the same events, schedules the same two upload jobs and tries to connect to the same address. So the upload is not new, but this change stops it for everyone.

What is left is the datatransport code in the APK, and ML Kit still creating the events in memory. Nothing is sent and nothing is stored: without a registered backend, DefaultScheduler.schedule in transport-runtime 2.2.6 logs Transport backend 'cct' is not registered and returns before it saves the event (I checked the bytecode). Removing the code too would mean excluding com.google.android.datatransport:transport-backend-cct in Gradle. I haven't tried that, because ML Kit may crash when it starts if it refers to that class directly. Left for the maintainer to decide.

Testing

• ./gradlew assembleDebug builds successfully.
• Tested on GrapheneOS without Google Play services: text in images is read correctly. (This was before the manifest change.)
• Tested on a Pixel 10 (arm64, Android 17) in a profile where Google Play services is not installed, with a debug APK built from this branch: Describe image, assigned to a one-finger gesture, focused on an image with text. Backtalk spoke "Se ha detectado texto: ..." ("Text detected: ..."). In the log, each gesture starts the text recognition (PipelineManager: Start process bitmap, then OCR process succeeded), Transport backend 'cct' is not registered shows there is no backend to upload usage events to, and no upload jobs are scheduled.
• Tested a phone with play services.
• I don't have a watch, so the watch build without ML Kit is not tested on a real watch. It builds, the app starts on an emulator, and testPhoneDebugUnitTest and :lintPhoneDebug pass.

🤖 Generated with Claude Code

[trypsynth]

Thanks, being able to read text without Play services is worth having. Some things to sort out first:

• Size. Bundled ML Kit is about 4 MB per ABI, and we build three ABIs into one APK, the watch build included. So the APK grows by roughly 10–12 MB, not a few MB. Please update differences.md, and check the actual APK size before and after.
• Telemetry. Bundled ML Kit pulls in Firebase datatransport, which may send usage metrics to Google from the app itself. On a de-Googled phone that would be new traffic. Could you check the merged manifest and dependency tree, and turn it off if it's there?
• Phones with Play services. They'd also switch to the bundled model and stop getting model updates through Play services. Using the bundled model only when Play services is missing would avoid both the change for those users and, if it can be split out, part of the size cost.
• CI hasn't run yet; I'll approve it.

[jpavonabian]

@trypsynth thanks for the review. I went through each point.

Size. It's more than a few MB, you're right. I measured debug builds before and after: the phone APK goes from 106 MiB to 135 MiB (+29.5 MiB). It's about 30 MB and not 10 to 12 because the native OCR library is bigger than 4 MB per ABI: libmlkit_google_ocr_pipeline.so is 11 MB for arm64-v8a, 6.8 MB for armeabi-v7a and 11.6 MB for x86_64. Release is the same (+29.5 MiB, we don't minify). differences.md and the PR description have the numbers.

The watch build was growing by the same amount, but nothing in the watch interface can reach text recognition (Describe image and automatic descriptions are turned off there, and Describe screen only sends the screenshot to Gemini), so f493373 bundles ML Kit only in the phone build and keeps it compile-only for the watch. The watch APK is back to 80 MiB. I don't have a watch, so that part is not tested on real hardware. It builds, the app starts on an emulator, and testPhoneDebugUnitTest and :lintPhoneDebug pass.

Telemetry. It was there. Firebase datatransport (TransportBackendDiscovery, JobInfoSchedulerService and AlarmManagerSchedulerBroadcastReceiver) is in the merged manifest, and with this change alone the app stored FIREBASE_ML_SDK events, scheduled two upload jobs and tried to connect to firebaselogging.googleapis.com. I ran the build from before this PR too, and it did exactly the same through the Play services artifact, so it isn't new traffic. Either way 0722722 removes those three components from the manifest. On a Pixel 10 in a profile without Play services, Describe image still reads text, the log says Transport backend 'cct' is not registered, and there are no upload jobs.

Phones with Play services. I couldn't find a way to do this with a single APK. The Play services artifact and the bundled one have the same classes (TextRecognition, TextRecognizerOptions), so they can't be in the same build, and the model would take the space anyway. Phones with Play services do move to the bundled model, so the text recognition model no longer gets updated through Play services and stays at the version in the APK. This is about the ML Kit model, not about updating Backtalk itself. I tried an old phone with Play services and text is read correctly.

The only option I see is two build flavors, with and without the bundled model. That means two phone APKs and the updater choosing between them, which is a bigger change in how releases are built, so I didn't want to do it without asking. Would you like me to leave it as it is, or do the flavors?

[jpavonabian]

@trypsynth one more thing on telemetry. With 0722722 nothing is sent and nothing is stored: without a registered backend, datatransport's DefaultScheduler.schedule logs Transport backend 'cct' is not registered and returns before it saves the event (I read the bytecode of transport-runtime 2.2.6 to check).

The code that could send it is still in the APK, though, and ML Kit still creates the events in memory before they are dropped. The way to remove it entirely would be to exclude com.google.android.datatransport:transport-backend-cct in Gradle. I haven't tried that: ML Kit may crash when it starts if it refers to that class directly, and I would only do it after testing text recognition on a phone.

Do you want me to try it, or is the manifest change enough?