Skip to content

feat(announcer): pluggable speech engine with LG/Samsung device detection #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chiefcll wants to merge 1 commit into from
Closed

feat(announcer): pluggable speech engine with LG/Samsung device detection #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 146 additions & 9 deletions docs/primitives/a11y.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Announcer Library

The `Announcer` library allows for relevant information to be voiced along the focus path of the application. The `Announcer` traverses the `focusPath` of the app collecting strings or promises of strings to announce to the user. The array of information is passed to a SpeechEngine which is responsible for converting the text to speech. By default we use the [speechSynthesis API](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis), but you can replace this by overwriting `Announcer._textToSpeech`.
The `Announcer` library allows for relevant information to be voiced along the focus path of the application. The `Announcer` traverses the `focusPath` of the app collecting strings or promises of strings to announce to the user. The array of information is passed to a speech engine which is responsible for converting the text to speech. By default we use the [speechSynthesis API](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis), but you can replace it with a platform TTS via [`Announcer.setSpeechEngine`](#custom-speech-engine).

Note: The speechSynth api has some known problems:<br />
https://stackoverflow.com/questions/39391502/js-speechsynthesis-problems-with-the-cancel-method<br />
Expand Down Expand Up @@ -85,6 +85,141 @@ You may also use `PAUSE-#` to pause speech for # seconds before saying the next
['PAUSE-2.5', 'Hello there!'];
```

## Custom Speech Engine

Some TV platforms ship their own text-to-speech service rather than the Web Speech API — webOS (Luna), Tizen, or a native bridge exposed by the app shell. Pass an implementation to `Announcer.setSpeechEngine()` and every announcement is routed through it instead:

```js
import { useAnnouncer } from '@solidtv/solid/primitives';

const Announcer = useAnnouncer();

Announcer.setSpeechEngine({
speak: (phrase) => webOSLunaTTSSpeak(phrase),
cancel: () => webOSLunaTTSStop(),
});
```

Call it with no arguments to restore the default speechSynthesis engine:

```js
Announcer.setSpeechEngine();
```

### The engine interface

An engine only speaks a single phrase. The `Announcer` still owns everything around it — flattening strings, `PAUSE-#` delays, nested arrays / promises / functions, `append`, `cancel` and the network retry — so an integration is usually a few lines.

```ts
interface SpeechEngine {
speak: (
phrase: string,
options: { lang: string; voice?: string },
) => void | Promise<void>;
cancel: VoidFunction;
}
```

| member | description |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `speak` | Speaks one phrase. `options.lang` is `Announcer.lang` (or the utterance's `lang`), `options.voice` is `Announcer.voice`. Return a promise that resolves when the phrase has finished so the series waits for it; return nothing and the series moves on to the next phrase immediately. |
| `cancel` | Stops whatever is currently being spoken. Called by `Announcer.cancel()` and whenever a new announcement replaces an in-flight one. |

Reporting completion matters for pacing. If the platform gives you a callback, wrap it:

```js
Announcer.setSpeechEngine({
speak: (phrase, { lang }) =>
new Promise((resolve, reject) => {
webOSLunaTTS.speak(phrase, lang, {
onComplete: resolve,
onFailure: reject,
});
}),
cancel: () => webOSLunaTTSStop(),
});
```

### Errors

Rejecting from `speak` signals a failure. The rejection's `error` property is used to classify it, mirroring the [speechSynthesis error codes](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisErrorEvent/error):

| `error` value | behavior |
| ---------------------------- | -------------------------------------------------------------- |
| `'network'` | Retried up to 3 times with a growing back-off |
| `'canceled'`/`'interrupted'` | Treated as benign — the series ends quietly, nothing is logged |
| anything else | Propagates to the caller |

```js
const error = new Error('TTS unavailable');
error.error = 'network'; // gets retried
throw error;
```

### Notes

- Setting an engine is global and takes effect immediately; it does not need to happen before `useAnnouncer()`.
- The engine is bypassed when `Announcer.aria` is `true`, since that mode writes to an `aria-live` region for the platform screen reader instead of speaking directly.
- The default engine is the only code that touches `window.speechSynthesis`, so an engine works on devices where the Web Speech API is missing entirely.

## Device Detection (LG & Samsung)

`Announcer.detectSpeechEngine()` tests the device and switches to that TV's own speech output. It only reads globals — it never speaks — so it is safe to call during startup, and it returns the platform it picked.

```js
import { useAnnouncer } from '@solidtv/solid/primitives';

const Announcer = useAnnouncer();
const platform = Announcer.detectSpeechEngine(); // 'webos' | 'tizen' | 'default'
```

| platform | device | what gets configured |
| --------- | ------------- | --------------------------------------------------------------------- |
| `webos` | LG | Installs a Luna TTS engine and sets `Announcer.aria = false` |
| `tizen` | Samsung | Sets `Announcer.aria = true` so Voice Guide reads the announcements |
| `default` | Anything else | Restores the speechSynthesis engine and leaves `Announcer.aria` as-is |

The two platforms work in fundamentally different ways, so it is worth knowing which one you are on.

### LG (webOS) — Luna TTS

webOS exposes the same TTS service its Audio Guidance uses, so we can hand it strings directly. Announcements go through `luna://com.webos.service.tts`, and each phrase subscribes for feedback so the promise settles when the TV has actually finished speaking — which keeps `PAUSE-#` timing in the middle of a series accurate.

Two things are required on device:

1. **webOSTV.js must be loaded.** Detection is a capability test for `webOS.service.request`, not a user-agent match. If the app is on webOS but the library is missing, detection returns `default`, logs a warning, and falls back to speechSynthesis.
2. **The service must be in your app permissions.** Add it to `appinfo.json`:

```json
{
"requiredPermissions": ["com.webos.service.tts"]
}
```

Without the permission every request fails. Rather than reject on each phrase, the engine logs one warning naming the permission and stays quiet, so a config typo doesn't turn into a stream of unhandled rejections.

To pass an `appID` (which lets `stop` target only your app's messages) build the engine yourself:

```js
import { createWebOSEngine } from '@solidtv/solid/primitives';

Announcer.setSpeechEngine(
createWebOSEngine(webOS.service.request.bind(webOS.service), {
appID: 'com.example.myapp',
}),
);
```

### Samsung (Tizen) — Voice Guide

**Samsung does not provide an API for an app to speak a string.** The TTS on a Samsung TV is the built-in Voice Guide screen reader, and it only reads the DOM. So there is no engine to install — `detectSpeechEngine()` instead turns on `Announcer.aria`, which writes each announcement into an `aria-live="assertive"` region for Voice Guide to pick up.

Consequences worth planning for:

- **Nothing is audible unless the viewer has Voice Guide turned on** in the TV's accessibility settings. That is a system setting; the app cannot enable it. Detection warns in the console when it can tell that Voice Guide is off, and you can check it yourself with `isTizenVoiceGuideEnabled()` (which returns `undefined` on models where it can't be read).
- **`Announcer.voice` is ignored**, and language comes from the `lang` attribute on the injected span rather than from the engine.
- **Timing is approximate.** The Announcer can't observe when Voice Guide finishes a phrase, so a series resolves as soon as the labels are written. `PAUSE-#` still delays, but it doesn't pace against real speech the way the webOS path does.

## API

### Properties
Expand All @@ -96,11 +231,13 @@ You may also use `PAUSE-#` to pause speech for # seconds before saying the next

### Methods

| name | args | description |
| -------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| speak | | Performs a manual announce |
| &nbsp; | `announcement` | See _SpeechType_ above |
| &nbsp; | `options` | Object containing one or more boolean flags: <br/><ul><li>append - Appends announcement to the currently announcing series.</li><li>notification - Speaks out notification and then performs $announcerRefresh.</li></ul> |
| clearPrevFocus | `depth` | Clears the last known focusPath - depth can trim known focusPath |
| cancel | none | Cancels current speaking |
| setupTimers | `options` | Object containing: <br/><ul><li>focusDebounce - default amount of time to wait after last input before focus change announcing will occur.</li><li>focusChangeTimeout - Amount of time with no input before full announce will occur on next focusChange</li></ul> |
| name | args | description |
| ------------------ | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| speak | | Performs a manual announce |
| &nbsp; | `announcement` | See _SpeechType_ above |
| &nbsp; | `options` | Object containing one or more boolean flags: <br/><ul><li>append - Appends announcement to the currently announcing series.</li><li>notification - Speaks out notification and then performs $announcerRefresh.</li></ul> |
| clearPrevFocus | `depth` | Clears the last known focusPath - depth can trim known focusPath |
| cancel | none | Cancels current speaking |
| setupTimers | `options` | Object containing: <br/><ul><li>focusDebounce - default amount of time to wait after last input before focus change announcing will occur.</li><li>focusChangeTimeout - Amount of time with no input before full announce will occur on next focusChange</li></ul> |
| setSpeechEngine | `engine` | Replaces the text-to-speech backend - see [Custom Speech Engine](#custom-speech-engine). Call with no argument to restore the speechSynthesis default. |
| detectSpeechEngine | none | Tests the device and switches to its built-in speech output, returning `'webos'`, `'tizen'` or `'default'` - see [Device Detection](#device-detection-lg--samsung). |
71 changes: 69 additions & 2 deletions src/primitives/announcer/announcer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
import type { ElementNode } from '@solidtv/solid';
import { untrack } from 'solid-js';
import SpeechEngine, { type SeriesResult, type SpeechType } from './speech.js';
import speakSeries, {
setSpeechEngine,
type SeriesResult,
type SpeechEngine,
type SpeechType,
} from './speech.js';
import {
createWebOSEngine,
detectSpeechPlatform,
isTizenVoiceGuideEnabled,
isWebOS,
webOSLunaRequest,
type SpeechPlatform,
} from './platformEngines.js';
import { debounce } from '@solid-primitives/scheduled';
import { focusPath } from '../useFocusManager.js';

Expand Down Expand Up @@ -119,7 +132,7 @@ function textToSpeech(
return;
}

return (currentlySpeaking = SpeechEngine(toSpeak, aria, lang, voice));
return (currentlySpeaking = speakSeries(toSpeak, aria, lang, voice));
}

export interface Announcer {
Expand All @@ -138,6 +151,27 @@ export interface Announcer {
focusDebounce?: number;
focusChangeTimeout?: number;
}) => void;
/**
* Replace the text-to-speech backend — for platforms with their own TTS
* (webOS Luna, Tizen, a native bridge) instead of the Web Speech API.
* Call with no argument to restore the default. Has no effect while
* `Announcer.aria` is true, since that path writes to an aria live region
* rather than speaking.
*/
setSpeechEngine: (engine?: SpeechEngine | null) => void;
/**
* Detects the TV platform and switches the Announcer to that device's own
* speech output, returning what it picked. Safe to call anywhere — it reads
* globals, it doesn't speak.
*
* - LG (`webos`) — installs an engine driving the Luna TTS service, and
* clears `aria`.
* - Samsung (`tizen`) — turns `aria` on. Samsung has no speak API; its Voice
* Guide screen reader is the TTS and it reads the live region instead.
* - Anything else (`default`) — restores the Web Speech API engine and leaves
* `aria` as configured.
*/
detectSpeechEngine: () => SpeechPlatform;
onFocusChange?: DebounceWithFlushFunction<ElementNode[]>;
refresh: (depth?: number) => void;
}
Expand Down Expand Up @@ -182,6 +216,39 @@ export const Announcer: Announcer = {
Announcer.onFocusChange(untrack(() => focusPath()));
}
},
setSpeechEngine: setSpeechEngine,
detectSpeechEngine: function () {
const platform = detectSpeechPlatform();

if (platform === 'tizen') {
// Samsung's Voice Guide is the only TTS available to an app, and it
// reads the DOM rather than accepting strings.
setSpeechEngine();
Announcer.aria = true;

if (isTizenVoiceGuideEnabled() === false) {
console.warn(
'Announcer: Voice Guide is off in the TV settings, so announcements will be silent.',
);
}
return platform;
}

if (platform === 'webos') {
setSpeechEngine(createWebOSEngine(webOSLunaRequest()!));
Announcer.aria = false;
return platform;
}

if (isWebOS()) {
console.warn(
'Announcer: webOS detected but webOS.service.request is unavailable — include webOSTV.js to use the Luna TTS engine. Falling back to speechSynthesis.',
);
}

setSpeechEngine();
return platform;
},
setupTimers: function ({
focusDebounce = 400,
focusChangeTimeout = fiveMinutes,
Expand Down
Loading
Loading