Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# Changelog

## [4.41.0]

- `client.sync.transcribe()` now sends audio over the same live connection as `transcribeLive()` / `openLive()`: `POST /v1/transcribe/live` (also served at `/v1/transcribe/stream`), chunked multipart, `config` first (always present, `{}` when empty) then `audio`. A clip already held is sent as a single chunk. There is no separate buffered request any more — nothing posts to `/v1/transcribe`
- `transcribe()`'s third argument (`SyncTranscribeOptions`) is now `{ timeout, signal }`, matching the live methods: `timeout` (default 180 000 ms, was 60 000) is a total deadline from the start of the request spanning the upload and the transcription, and `signal` (an `AbortSignal`) drops the request when aborted. `warm()` is unchanged
- Raise sync config caps to match the service: `prompt` to 6000 characters (was 4096), `keyterms_prompt` to 100 terms and 8000 characters total (was 2048 characters, uncapped terms), and `conversation_context` to 500 turns / 16 000 characters (was 100 / 4096) — still trimmed oldest-first rather than rejected. Over-cap `prompt`/`keyterms_prompt` are still rejected before any request is sent
- Raise dictation config caps to match the service: `stt_prompt` to 6000 characters (was 4096) and `keyterms_prompt` to 100 terms and 8000 characters total (was 2048, uncapped terms); `llm_instruction` is unchanged at 2048
- A failure body of the form `{"error": "...", "error_code": "..."}` now surfaces that `error` string as the thrown error's message, for both `SyncTranscriptError` and `DictationError`
- Multipart file names are now escaped per the HTML5 form-encoding rules, so a `"`, backslash or control character in a file name can no longer break out of the part header

## [4.40.0]

- Add `client.dictation` — dictation transcription for short spoken notes: `transcribeLive(audio, config?, options?)` uploads audio as it is spoken and resolves with one finished transcript, `openLive(config?, options?)` is the push-style counterpart for callback-driven sources, and `warm()` opens the connection ahead of the request. Targets `dictation.assemblyai.com`, overridable with the new `dictationBaseUrl` client option. Audio is WAV or raw 16-bit PCM, up to 120 s
Expand Down
38 changes: 23 additions & 15 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,8 +185,11 @@ const vtt = await client.transcripts.subtitles(id, "vtt");

## Sync transcription (pre-recorded, single request)

`client.sync` posts a whole audio file and returns the finished transcript in one
round trip — no job id, no polling, no status enum. It targets the sync API host
`client.sync` sends audio over one live connection and returns the finished
transcript in one round trip — no job id, no polling, no status enum.
`transcribe()` sends a clip you already hold as a single chunk over that
connection; `transcribeLive()` / `openLive()` upload it as it is still being
recorded. It targets the sync API host
(`sync.assemblyai.com`, override with the `syncBaseUrl` client option), distinct
from `client.transcripts`' async job API. Use it for short clips where you want the
answer inline; use `client.transcripts` for long-form audio, URLs, or the rich
Expand All @@ -209,11 +212,11 @@ a Blob/File, or a readable stream. **Not** a URL — pass a path/bytes or use

```typescript
const result = await client.sync.transcribe("./call.wav", {
prompt: "Transcribe verbatim. Preserve disfluencies.", // max 4096 chars, rejected over
keyterms_prompt: ["AssemblyAI", "Lemur", "U3-Pro"], // max 2048 chars total, rejected over
prompt: "Transcribe verbatim. Preserve disfluencies.", // max 6000 chars, rejected over
keyterms_prompt: ["AssemblyAI", "Lemur", "U3-Pro"], // max 100 terms, 8000 chars total, rejected over
language_codes: ["es"], // or e.g. ["en", "es"] for multilingual; defaults to English; ignored when prompt is set
conversation_context: [
// prior turns oldest-first; capped at 100 turns / 4096 chars — trimmed, not rejected
// prior turns oldest-first; capped at 500 turns / 16 000 chars — trimmed, not rejected
"I'd like to book a flight to Denver.",
"Sure, what date were you thinking?",
],
Expand Down Expand Up @@ -257,10 +260,13 @@ it is still being recorded); it returns `true` once the socket is open, `false`
a transport failure. Call it shortly before `transcribe()` — the pooled connection
idles out after a few seconds.

**Live upload**: `transcribeLive()` and `openLive()` post to `/v1/transcribe/stream` with a
chunked multipart body, so the request starts before the audio exists and chunks upload as they
arrive. Authorization, the upload and every speech segment but the last resolve while the caller
is still recording; only the final segment is left to wait for. `transcribeLive()` pulls from an
**Live upload**: `transcribe()`, `transcribeLive()` and `openLive()` all send audio over one live
connection — `POST /v1/transcribe/live` (also served at `/v1/transcribe/stream`) — with a
chunked multipart body, `config` sent first (always present, `{}` when
empty) then `audio`. `transcribe()` sends a clip already held whole as a single chunk. For
`transcribeLive()`/`openLive()` the request starts before the audio exists and chunks upload as
they arrive, so authorization, the upload and every speech segment but the last resolve while the
caller is still recording; only the final segment is left to wait for. `transcribeLive()` pulls from an
async iterable (Node streams included), a sync iterable, or a web `ReadableStream` — bytes, a
Blob or a path are rejected with a TypeError pointing at `transcribe()`. `openLive()` is the
push-style counterpart for callback-driven sources (microphone library, WebRTC track, telephony
Expand Down Expand Up @@ -291,9 +297,11 @@ long enough is aborted server-side); auth, rate-limit and capacity failures can
through the upload rather than only at the end (a malformed key at once, a key that fails deeper
checks a few seconds in), and `warm()` opens the connection but does not validate the key.

**Client-side timeout**: third argument — `client.sync.transcribe(audio, {}, { timeout: 30_000 })`
(default 60 s, kept above the server's 30 s deadline). Live requests take `SyncLiveOptions`
instead — `{ timeout, signal }`.
**Client-side timeout**: third argument, `SyncTranscribeOptions` — `{ timeout, signal }`, e.g.
`client.sync.transcribe(audio, {}, { timeout: 30_000 })`. `timeout` (default 180 000 ms) is a
total deadline from the start of the request, spanning the upload and the transcription — the
same basis `transcribeLive()`/`openLive()` use via `SyncLiveOptions`. `signal` drops the request
when aborted.

## Dictation (short spoken notes, optional LLM rewrite)

Expand Down Expand Up @@ -345,8 +353,8 @@ const result = await client.dictation.transcribeLive(mic, {
sample_rate: 16_000, // raw 16-bit PCM; setting either PCM field requires both; unset for WAV
channels: 1, // 1 mono / 2 stereo
language_codes: ["es"], // ISO 639-1; or e.g. ["en", "es"]; unset leaves the language to the server
stt_prompt: "A doctor dictating a patient visit note.", // max 4096 chars; describes the recording
keyterms_prompt: ["AssemblyAI", "U3-Pro"], // whitespace stripped, empties dropped, max 2048 chars total
stt_prompt: "A doctor dictating a patient visit note.", // max 6000 chars; describes the recording
keyterms_prompt: ["AssemblyAI", "U3-Pro"], // whitespace stripped, empties dropped, max 100 terms, 8000 chars total
llm_instruction: "Format this as a SOAP note.", // max 2048 chars; follow-up LLM pass over the transcript
});
```
Expand Down Expand Up @@ -418,7 +426,7 @@ strings, when present).
- **speech_models takes an array** with fallback ordering: ["universal-3-5-pro", "universal-2"]
- **Streaming uses universal-3-5-pro** as the speech model
- **Never expose API keys client-side** — use temporary auth tokens for browser streaming
- **Live upload timeout is a total deadline** — `transcribeLive()`/`openLive()` default to 180 s covering the whole recording plus transcription (not per-response like `transcribe()`'s 60 s), and the sync API still caps audio at 120 s
- **Sync timeout is a total deadline** — `transcribe()`, `transcribeLive()` and `openLive()` all default to a 180 s deadline spanning the whole request (upload plus transcription), and the sync API still caps audio at 120 s
- **Dictation timeout is a total 300 s deadline** — `client.dictation` spans the upload, the transcription and the LLM pass in one deadline; the service caps audio at 120 s and accepts WAV or raw 16-bit PCM only
- **Node >= 18 required**
- **Only runtime dependency**: ws (WebSocket library)
Expand Down
26 changes: 14 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,11 +313,13 @@ const res = await client.transcripts.delete(transcript.id);

### Transcribe audio synchronously

`client.sync` posts a whole audio file and returns the finished transcript in one
round trip — no job id, no polling — or uploads the audio live, as it is still
being recorded (`transcribeLive()` / `openLive()`). Use it for short clips where
you want the answer inline; use `client.transcripts` for long-form audio, URLs,
or the rich audio-intelligence features the sync API doesn't expose.
`client.sync` sends audio over one live connection and returns the finished
transcript in one round trip — no job id, no polling. `transcribe()` sends a
clip you already hold as a single chunk over that connection;
`transcribeLive()` / `openLive()` upload it as it is still being recorded. Use
it for short clips where you want the answer inline; use `client.transcripts`
for long-form audio, URLs, or the rich audio-intelligence features the sync API
doesn't expose.

```typescript
const result = await client.sync.transcribe("./call.wav");
Expand All @@ -332,8 +334,8 @@ stream — but not a URL.

```typescript
const result = await client.sync.transcribe("./call.wav", {
prompt: "Transcribe verbatim. Preserve disfluencies.", // max 4096 chars
keyterms_prompt: ["AssemblyAI", "Lemur"], // max 2048 chars total
prompt: "Transcribe verbatim. Preserve disfluencies.", // max 6000 chars
keyterms_prompt: ["AssemblyAI", "Lemur"], // max 100 terms, 8000 chars total
language_codes: ["es"], // or e.g. ["en", "es"] for multilingual; defaults to English
conversation_context: [
// prior turns, oldest first
Expand Down Expand Up @@ -490,9 +492,9 @@ console.log(result.text);
```

`options.timeout` (default 180 000 ms) is a **total** deadline spanning the
whole upload, unlike `transcribe()`'s 60 s, so it has to exceed the length of
the recording as well as the transcription. `options.signal` drops the request
from outside, the way `abort()` does for a session.
whole upload, so it has to exceed the length of the recording as well as the
transcription. `options.signal` drops the request from outside, the way
`abort()` does for a session.

Caveats, for both live entry points:

Expand Down Expand Up @@ -682,8 +684,8 @@ const result = await client.dictation.transcribeLive(micStream, {
sample_rate: 16_000, // raw 16-bit PCM only; required together with channels
channels: 1, // 1 mono, 2 stereo; leave both unset for WAV
language_codes: ["es"], // ISO 639-1; or e.g. ["en", "es"]; defaults to the server's choice
stt_prompt: "A doctor dictating a patient visit note.", // max 4096 chars
keyterms_prompt: ["AssemblyAI", "Universal-3"], // max 2048 chars total
stt_prompt: "A doctor dictating a patient visit note.", // max 6000 chars
keyterms_prompt: ["AssemblyAI", "Universal-3"], // max 100 terms, 8000 chars total
llm_instruction: "Format this as a SOAP note.", // max 2048 chars
});
```
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "assemblyai",
"version": "4.40.0",
"version": "4.41.0",
"description": "The AssemblyAI JavaScript SDK provides an easy-to-use interface for interacting with the AssemblyAI API, which supports async and real-time transcription.",
"engines": {
"node": ">=18"
Expand Down
12 changes: 10 additions & 2 deletions src/services/dictation/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,11 @@ const warmEndpoint = "/v1/warm";
// total request budget; the audio itself is capped at 120 s.
const defaultLiveTimeoutMs = 300_000;
const warmTimeoutMs = 10_000;
const maxSttPromptLength = 4096;
const maxKeytermsPromptLength = 2048;
// Caps mirror the dictation service's `config` part; a field over its cap is
// rejected before any request is sent.
const maxSttPromptLength = 6000;
const maxKeytermsPromptLength = 8000;
const maxKeytermsCount = 100;
const maxLlmInstructionLength = 2048;
// What to call the config in the message when raw PCM is missing a field.
const configName = "the config";
Expand Down Expand Up @@ -411,6 +414,11 @@ function normalizeKeytermsPrompt(
const terms = keytermsPrompt
.map((term) => term.trim())
.filter((term) => term.length > 0);
if (terms.length > maxKeytermsCount) {
throw new Error(
`keyterms_prompt exceeds ${maxKeytermsCount} terms (got ${terms.length})`,
);
}
const total = terms.reduce((sum, term) => sum + term.length, 0);
if (total > maxKeytermsPromptLength) {
throw new Error(
Expand Down
Loading