Skip to content

Let other apps provide Braille displays - #179

Merged
trypsynth merged 3 commits into
trypsynth:masterfrom
emassey0135:braille-display-api
Oct 15, 2026
Merged

trypsynth merged 3 commits into
trypsynth:masterfrom
emassey0135:braille-display-api

Conversation

@emassey0135

@emassey0135 emassey0135 commented Oct 13, 2026 •

Copy link
Copy Markdown

Lets an app that the user allows give Backtalk a Braille display that Backtalk can't connect to on its own, such as the built-in keys and cells of a Braille notetaker, or a display behind its maker's own software. Backtalk uses it like a display connected through BRLTTY: its keys navigate and type with the same commands, and its cells show what Backtalk shows. A display with keys and no cells works too. docs/braille-display-api.md describes the API for app developers.

How it works

  • The display app provides a service with the action fyi.quin.backtalk.action.BRAILLE_DISPLAY, protected by fyi.quin.backtalk.permission.BIND_BRAILLE_DISPLAY, a signature permission that only Backtalk holds, so only Backtalk can bind it.
  • Backtalk binds the service of every app the user has allowed in Backtalk while it runs, which keeps the app running, and binds it again if it stops. It looks for displays when it starts, when an app is installed, updated or removed, when the user allows or stops allowing one, and when the user is unlocked after a reboot. An app can also send BRAILLE_DISPLAYS_CHANGED to have it look again.
  • The display calls start(name, cellCount), then sends brailleKeys(dots, space), which run the command bound to them, as on any display, or else type, and command(name, argument) for its other keys, with any Braille display command named as in BrailleInputEvent, such as NAV_ITEM_NEXT or ROUTE with a cell. Backtalk sends the cells to show with showCells. All calls are one-way.
  • One display is used at a time. A display connected through BRLTTY comes first: while one is connected, displays from apps wait, and one that is in use steps aside when it connects and comes back when it disconnects. Among displays from apps, others that start wait until the one in use stops.
  • It drives the same BdController as a BRLTTY display. Outside the new platform/external package, the changes are a small DisplayerInterface, with the three methods that BdController uses, which Displayer now implements, and BrailleDisplay giving BrailleDisplayManager a pass-through controller from ExternalBrailleDisplays. Through it, the two kinds of display take turns, so a display from an app never starts or stops the controller during a BRLTTY session, and BRLTTY stopping a connection that never started a display doesn't stop one from an app. With no display from an app in use, it passes everything through unchanged.

Approval

  • fyi.quin.backtalk.permission.BRAILLE_DISPLAY is a normal permission that only lets an app ask to be a display. Its description says that an app that's allowed can read the screen and control the device.
  • Backtalk uses an app only once the user allows it in Backtalk, after a warning that the app will be able to read everything Backtalk shows in Braille, including messages and what the user types, and to control the device as a Braille display can. Backtalk asks with a notification when it finds an app that can be a display and the user hasn't answered for it, and the notification opens the warning. An app can also open the warning itself with fyi.quin.backtalk.action.ASK_BRAILLE_DISPLAY.
  • Displays from other apps, in Braille display settings, lists every app that can be a display, to allow or stop allowing each one; allowing one there shows the warning too. Changes take effect at once, stopping a display that's no longer allowed, and Backtalk checks again whenever it connects to an app.
  • While a display from an app is in use, an ongoing notification says which app it is.
  • A display's keys can do anything that a Braille display's keys can, so there are no limits on what an allowed display sends or how quickly. On the devices this is for, it's the only way to use Backtalk at all, and a limit would get in the way.

Cost when no display app is installed

One search for display apps when Backtalk starts and when the user is unlocked, a permission check when an app changes, and a null check on each accessibility event. A BRLTTY display's calls to the controller go through one more method call. Nothing polls and nothing is bound.

Testing

Tested on an Orbit Speak (Android 11) with a display app for its built-in Braille keys, which has no cells: navigation, typing in contracted and uncontracted Braille, every chord, and the display reconnecting after Orbit's screen reader restarts and after a reboot. Phone and watch builds, unit tests and lint pass.

The approval was tested on an Orbit Speak and an Orbit Player: the notification, the warning, the notice while the display is in use, and allowing and stopping allowing it in settings. BRLTTY taking turns was tested on an Orbit Player whose own keys were the display from an app, with an Orbit Speak in its remote mode connected through BRLTTY over Bluetooth as an Orbit Reader 20: the Player's keys stopped while the Speak was connected, and came back when it disconnected.

Cell output hasn't been tested on hardware yet. I plan to test it with the Orbit Strata when that is released and I purchase one, and I might end up implementing this for the BrailleSense 7 when I get that to allow Backtalk to run on it.

@trypsynth

Copy link
Copy Markdown
Owner

Thanks for the binding-leak fix. The two concerns from before still stand, so this isn't mergeable yet:

  1. Approval. BRAILLE_DISPLAY is still a dangerous runtime permission, so one tap in a system dialog lets an app type and act anywhere, and read everything Backtalk shows in braille. That's far easier to get than an accessibility service. Please gate it with Backtalk's own approval, with a clear warning (like the speech API's), show a notice while an external display is active, and make the permission text say plainly that the app can read the screen and control the device.
  2. BRLTTY clash. activate() calls controller.onStart without checking for a connected BRLTTY display, and stopActive() / startDisplay() call controller.onStop() on the shared controller, which breaks a live BRLTTY session. Please refuse, or step aside, while a BRLTTY display is connected.

It also conflicts in differences.md now.

emassey0135 and others added 3 commits October 15, 2026 02:24
An app that the user grants fyi.quin.backtalk.permission.BRAILLE_DISPLAY, a runtime permission that can also be granted with adb shell pm grant, can give Backtalk a braille display through a service that only Backtalk can bind. Backtalk binds the service of every allowed app, which keeps it running. The app sends braille keys, which run the same commands as on any display or else type, and any display command by name, and gets the cells to show. It drives the same braille controller as a display connected through BRLTTY, through a small DisplayerInterface that Displayer now implements, so displays with keys only and displays with cells both work, one at a time. This is for hardware that Backtalk can't reach itself, such as the built-in keys and cells of a braille notetaker. docs/braille-display-api.md describes the API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
While a display app is updated, Android can deliver onBindingDied more than once, including late calls for a binding that has already been replaced. Each one removed the current binding from the map without unbinding it and bound another, so one update could leave several bindings, each connecting as a host. A replaced binding now only unbinds itself when such a late call comes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y to BRLTTY

A display from an app can read everything Backtalk shows and control the device, so a runtime permission, granted with
one tap in a system dialog, was too easy to get. The permission is now a normal one that only lets an app ask, and
Backtalk uses an app only once the user allows it in Backtalk, after a warning that says the app will be able to read
the screen and control the device. Backtalk asks with a notification when it finds such an app, an app can open the
warning itself, and Displays from other apps in braille display settings lists them, to allow or stop allowing each
one. A notification says which app is in use while one is.

A display from an app no longer starts or stops the braille controller while a display connected through BRLTTY is in
use. BRLTTY's displays drive the controller through ExternalBrailleDisplays, so that they come first: while one is
connected, displays from apps wait, one that is in use steps aside when it connects and comes back when it
disconnects, and BRLTTY stopping a connection that never started a display no longer stops one from an app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@emassey0135

Copy link
Copy Markdown
Author

I fixed both issues. Now external Braille display apps are approved the same way as apps that speak through Backtalk, and the message in Backtalk's prompt is clear that this lets another app control your device. Also, connecting a display through BRLTTY now works and stays connected in preference to an app using this interface, and an app disconnecting should not disconnect a BRLTTY display.

@trypsynth
trypsynth merged commit eab4696 into trypsynth:master Oct 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants