Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/voice-permission-fallback.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@schlessera/brain-ui-sdk": minor
"@schlessera/brain-ui-react": minor
"@schlessera/brain-ui-server": minor
---

Negotiate host permission outcomes in the client and add scoped voice refusals with an earcon and visual fallback when exact permission speech is unproven. Keep approval controls, original-input attribution, once-only cues, terminal and reconnect receipts tied to the host's actual result. Publish posture-denied outcomes without raising a grant card.
39 changes: 39 additions & 0 deletions api-report/ui-react.txt
Original file line number Diff line number Diff line change
Expand Up @@ -833,6 +833,7 @@ signatures (every public export and the types it is made of)
};
ToolCall (packages/ui-react/src/stores/chat-state.ts)
export interface ToolCall {
voicePermission?: VoicePermissionReceipt;
policyChange?: PolicyChangeReview;
id: string;
name: string;
Expand Down Expand Up @@ -873,6 +874,21 @@ signatures (every public export and the types it is made of)
enabled?: boolean;
reload?: () => void;
}
VoicePermissionReceipt (packages/ui-react/src/voice/conversation-permissions.ts)
export interface VoicePermissionReceipt {
sessionId: string;
turnId: string;
toolUseId: string;
phase: "pending" | "sent" | "unknown" | "denied" | "granted" | "expired";
announced: boolean;
ambiguous: boolean;
byVoice: boolean;
heardAt?: number;
script: string;
word: string;
label?: string;
postureDenied?: true;
}
VoiceShellState (packages/ui-react/src/stores/shell-stores.ts)
export interface VoiceShellState {
readonly mode: VoiceMode;
Expand Down Expand Up @@ -1275,6 +1291,21 @@ signatures (every public export and the types it is made of)
frame: (message: ServerMessage) => ServerMessage | null;
leave: () => void;
mute(): void;
permissions: {
clear(): void;
dispose(): void;
frame: (message: ServerMessage) => void;
input: (event: Extract<import("@schlessera/brain-ui-sdk").ConversationWireEvent, {
kind: "input_fragment";
}>) => boolean;
lost(): void;
quiet: () => void;
refresh(): void;
state: import("zustand").StoreApi<{
line: string;
records: import("./voice/conversation-permissions.js").VoicePermissionReceipt[];
}>;
};
resume(): void;
start(): void;
state: import("zustand").StoreApi<import("./voice/conversation-client.js").ConversationClientState>;
Expand Down Expand Up @@ -1307,6 +1338,8 @@ signatures (every public export and the types it is made of)
function flushChatDeltas(): void;
frame (packages/ui-react/src/voice/conversation-client.ts)
function frame(message: ServerMessage): ServerMessage | null;
frame (packages/ui-react/src/voice/conversation-permissions.ts)
function frame(message: ServerMessage): void;
handleServerMessage (packages/ui-react/src/connection.ts)
function handleServerMessage(msg: ServerMessage): void;
hasPendingShare (packages/ui-react/src/stores/share-state.ts)
Expand All @@ -1315,10 +1348,16 @@ signatures (every public export and the types it is made of)
export function hasUnsentText(): boolean;
inlineMermaidDiagrams (packages/ui-react/src/lib/mermaid.ts)
export function inlineMermaidDiagrams(markdown: string, theme?: MermaidTheme): Promise<string>;
input (packages/ui-react/src/voice/conversation-permissions.ts)
function input(event: Extract<ConversationWireEvent, {
kind: "input_fragment";
}>): boolean;
leave (packages/ui-react/src/voice/conversation-client.ts)
function leave(): void;
logout (packages/ui-react/src/lib/answer-delivery/manager.ts)
function logout(): Promise<void>;
quiet (packages/ui-react/src/voice/conversation-permissions.ts)
function quiet(): void;
receipt (packages/ui-react/src/lib/answer-delivery/manager.ts)
function receipt(frame: ServerAskAnswerReceipt): void;
reconnectWebSocketNow (packages/ui-react/src/connection.ts)
Expand Down
2 changes: 1 addition & 1 deletion docs/decisions/answer-delivery.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ manufacture an identity, and it was not needed.
## One card per request

The store updates a request it already holds and never appends it again
(`addExchange`, `packages/ui-react/src/stores/chat-state.ts:267-298`). A
(`addExchange`, `packages/ui-react/src/stores/chat-state.ts:270-301`). A
settled card stays settled, so a late or replayed frame cannot revive a
dismissed or answered question. A pending card keeps its payload and the
exchange object it is keyed by, so a draft in progress survives a re-send.
Expand Down
6 changes: 3 additions & 3 deletions docs/decisions/design-kit/sessions-and-drafts.md
Original file line number Diff line number Diff line change
Expand Up @@ -364,7 +364,7 @@ active; Chat is in the foreground with no panel or other view over it;
`document.visibilityState` is `visible`; history has a message whose
host-proven `turnId` equals `latest.turnId`; and that message's last line is
in the viewport at the bottom, by the same `< 20px` test the transcript uses
(`const handleScroll`, `packages/ui-react/src/components/chat/chat-page.tsx:273-277`),
(`const handleScroll`, `packages/ui-react/src/components/chat/chat-page.tsx:274-278`),
with the scroll disc not drawn. An older key never clears a newer tracker.
Selecting the session, being scrolled up, a hidden tab, a background buffer
and the bottom of a replay without the linked turn do not count.
Expand Down Expand Up @@ -825,12 +825,12 @@ Read from the source, not inferred from the drawings:
`cost`. Those props are **new**, and chips get a 44px minimum under a
coarse pointer (#945).
- The scroll-to-bottom disc is a bare 32px button with only a `title`
(`{showScrollButton && (`, `packages/ui-react/src/components/chat/chat-page.tsx:741-749`).
(`{showScrollButton && (`, `packages/ui-react/src/components/chat/chat-page.tsx:742-750`).
It has no 44px box and no accessible name, so it joins `DiscButton`.
- **`DiscButton`** is a 32px paint in a 44px box, with `tone: ink | mute`
and an optional label that expands leftward. It draws exactly three
discs: the phone Search disc, New chat below 1280
(`{hasMessages && !wide && (`, `packages/ui-react/src/components/chat/chat-page.tsx:656-663`)
(`{hasMessages && !wide && (`, `packages/ui-react/src/components/chat/chat-page.tsx:657-664`)
and scroll-to-latest. It is not used for rail rows, pills or chips. Both
overlay boxes share one vertical range, so #628's resting spacer (`pt-10`
below an 888px container, not the 880px in the drawings) still clears
Expand Down
2 changes: 1 addition & 1 deletion docs/decisions/governed-writable-workers.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ still holds.
**Voice.** Ruling 1 makes voice `governed` too. Voice keeps its narrower
membership (`VOICE_ALLOWED_TOOLS`, `packages/ui-backend-claude/src/tool-policy.ts:120`):
no Bash, no raw `Write`/`Edit`, no archive, and approval returns false
(`const apply = createBrainApplication`, `packages/ui-server/src/ws/bridge.ts:577-581`).
(`const apply = createBrainApplication`, `packages/ui-server/src/ws/bridge.ts:580-584`).
A voice turn therefore writes only through curated capture and update, now
with a journal behind it. The writable mount gives voice no tool it lacked.
Pi still refuses voice turns.
Expand Down
46 changes: 46 additions & 0 deletions docs/integration-contract/wire.md
Original file line number Diff line number Diff line change
Expand Up @@ -1834,3 +1834,49 @@ working/pending row collapses during voice; on short screens it hides first when
needed to retain transcript space. Cards remain in the existing scrolling Chat
transcript. No live recognition, latency, echo or exact permission-speech
qualification is implied; #961 owns the bounded provider/device measurements.

### Conversation permission fallback (#114)

The shipped SDK client declares `client_hello.capabilities.toolResolution: true`.
The React conversation owner consumes scoped `tool_resolution` frames before
ordinary chat dispatch. A host `permission_denied` activity event from a current,
authorized voice-posture turn publishes one `denied` outcome with the fixed
reason `voice_posture`; this is a posture refusal, not a user decision, and it
raises no approval card. It uses the existing bounded host outcome ledger.

Pending ordinary-turn approvals retain their visual grant controls. A refusal
matches original, attributed user input only while approvals are pending in the
selected session's current turn. A phrase-like `brain, stop` candidate snapshots
that turn's pending requests and sends the existing `tool_denial` with explicit
`turnId` and `channel: "voice"`. Partial, missing-confidence or unproven-isolation
candidates are treated as ambiguous. Bare stop/no/cancel and affirmatives remain
ordinary input. Assistant/output transcripts and unknown-origin input never
invoke the detector. Both built-in adapters retain their unproven attribution
profiles; manual review is not evidence authorizing a spoken decision.

Sending a refusal locks only the affected controls and says `Refusal sent ·
checking`. It never claims host confirmation. A correlated host outcome reports
denied, the actual winning screen grant, expired, or unknown. Disconnect stops
capture and retains uncertainty; it never queues an old denial for a replacement
request. Host terminal state reconciles unanswered requests without depending on
`tool_result`. Receipts remain readable if reconnect replaces the transcript.

Neither built-in adapter has a proven deterministic exact-copy permission audio
path. The approved fallback therefore uses a local earcon, the existing card and
a 40px strip line. Permission scripts and confirmed outcome wording are visual
only: they are never submitted as model commentary or to a separate TTS service.
Labels come only from bounded static registry labels (four words at most), with
`An action` as the generic fallback. Payloads and unproven effect descriptions
never enter those scripts. Cues wait for quiet, survive duplicate/reconnect
identities without repeating, and are dropped if the request resolves first.
Earcons use the existing PCM owner; speech over an earcon keeps recognition live.
Headphones/isolation limitations remain visible in the entry disclosure.

The local permission view retains at most 128 records and 4096 once-only cue
identities. If pending records exhaust the bound, capture closes and the screen
remains the decision surface; a partial snapshot is never silently denied.
At 320 × 568, shared-row summaries hide first; with review and a refusal line,
the controls' eight pixels of empty vertical space give way while every target
stays at least 44px and the transcript retains 264px. Real speaker/headphone
listening, recognition, echo and latency qualification remain human/device
verification under #114 and #961.
2 changes: 1 addition & 1 deletion docs/investigations/live-conversation-investigation.md
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ is unknown, not queued against a replacement request.
Terminal state invalidates every pending exchange for that turn without waiting
for `tool_result`. Current client terminal handling finishes the assistant
message and resyncs; its message-state helper only marks streaming false
(`finishAssistantMessage`, `packages/ui-react/src/stores/chat-state.ts:1800-1805`).
(`finishAssistantMessage`, `packages/ui-react/src/stores/chat-state.ts:1818-1823`).
This observation is a required integration check, not proof that all cards
currently clear. A host terminal event plus local audio drain/discard defines
conversation completion; provider turn/audio end does not.
Expand Down
10 changes: 5 additions & 5 deletions docs/plans/navigation-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -361,15 +361,15 @@ implementation. Any PR moving these lines owns their citation repairs.

- Welcome alternatives: (`label="Start with"`, `packages/ui-react/src/components/chat/welcome-state.tsx:46-55`).

- Session selection: (`const handleSessionResume = useCallback(`, `packages/ui-react/src/components/chat/chat-page.tsx:451-458`).
- Session selection: (`const handleSessionResume = useCallback(`, `packages/ui-react/src/components/chat/chat-page.tsx:452-459`).

- New chat: (`{hasMessages && !wide && (`, `packages/ui-react/src/components/chat/chat-page.tsx:656-663`).
- New chat: (`{hasMessages && !wide && (`, `packages/ui-react/src/components/chat/chat-page.tsx:657-664`).

- Buffer eviction: (`function evictStale(`, `packages/ui-react/src/stores/chat-state.ts:1079-1095`).
- Buffer eviction: (`function evictStale(`, `packages/ui-react/src/stores/chat-state.ts:1083-1099`).

- Root namespace: (`const prefix = options.storagePrefix ??`, `packages/ui-react/src/root.ts:182-237`).

- Clear transcript selection: (`clearMessages: () => {`, `packages/ui-react/src/stores/chat-state.ts:2104-2113`).
- Clear transcript selection: (`clearMessages: () => {`, `packages/ui-react/src/stores/chat-state.ts:2122-2131`).

- Composer draft/acknowledgement: [`const [input, setInput] = useState("");`](https://github.com/schlessera/brain-kit/blob/7bfdc71074f4ff6243e0d7133e83dbb36eb2a971/packages/ui-react/src/components/chat/composer.tsx#L62-L85), before #951 moved the draft into the root's draft store (`stores/draft-state.ts`).

Expand Down Expand Up @@ -401,7 +401,7 @@ implementation. Any PR moving these lines owns their citation repairs.

- Queued acceptance: (`// Queue it as the session's next turn; report queued immediately.`, `packages/ui-server/src/ws/run-session.ts:936-981`).

- Approval target: (`function mutateLastAssistant(`, `packages/ui-react/src/stores/chat-state.ts:1220-1250`).
- Approval target: (`function mutateLastAssistant(`, `packages/ui-react/src/stores/chat-state.ts:1224-1254`).

## Bounded follow-up ownership

Expand Down
3 changes: 2 additions & 1 deletion packages/ui-react/src/components/chat/chat-page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ export function ChatPage() {
const messages = useChatStore((s) => activeChat(s).messages);
const voiceOutputs = useStore(root.connection.conversation.state, (s) => s.outputs);
const sessionId = useChatStore((s) => s.activeSessionId);
const hasVoicePermission = useStore(root.connection.conversation.permissions.state, s => s.records.some(r => r.sessionId === sessionId && !!r.word));
const hasVoiceOutput = voiceOutputs.some((output) => output.sessionId === sessionId && output.generated);

const clearMessages = useChatStore((s) => s.clearMessages);
Expand Down Expand Up @@ -666,7 +667,7 @@ export function ChatPage() {
<div className="px-4 md:px-6"><div {...{ [READING_COLUMN_ATTR]: "" }} className="mx-auto max-w-3xl"><UnconfirmedSends /></div></div>
<RestorationState onNewChat={clearMessages} />
</div>
) : messages.length === 0 && !hasVoiceOutput ? (
) : messages.length === 0 && !hasVoiceOutput && !hasVoicePermission ? (
<div ref={welcomeRef} className="h-full overflow-y-auto" data-welcome="">
<div className="px-4 pt-3 md:px-6">
<DigestCard />
Expand Down
7 changes: 4 additions & 3 deletions packages/ui-react/src/components/chat/policy-change-card.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,9 @@ export function PolicyChangeCard({ toolCall, onApproval }: {
useLayoutEffect(() => {
if (explicit.current && (busy || review.state !== "pending")) { explicit.current = false; receipt.current?.focus({ preventScroll: true }); }
}, [busy, review.state]);
const voiceLocked = toolCall.voicePermission && toolCall.voicePermission.phase !== "pending";
function decide(approved: boolean) {
if (!verified || !current() || busy || sent.current || review.state !== "pending" || !onApproval) return;
if (voiceLocked || !verified || !current() || busy || sent.current || review.state !== "pending" || !onApproval) return;
sent.current = true;
if (onApproval(toolCall.id, approved) === false) { sent.current = false; return; }
explicit.current = true; setBusy(true); setReader(false);
Expand All @@ -72,7 +73,7 @@ export function PolicyChangeCard({ toolCall, onApproval }: {
<p className="text-xs">{review.bodyChanged ? "Body text changed." : "Body text unchanged."}</p>
{review.bodyChanged && <div className="[&_[role=button]]:min-h-11"><Disclosure label="Read body diff"><div className="max-h-80 min-w-0 overflow-auto"><DiffBlock tinted variant="inset" text={review.bodyDiff} /></div></Disclosure></div>}
</>;
const pending = review.state === "pending" && !busy;
const pending = review.state === "pending" && !busy && !voiceLocked;
return <div data-policy-change data-policy-state={review.state} className="min-w-0 space-y-2 text-foreground" style={{ overflowWrap: "anywhere" }}>
{pending ? <div data-approval-card role="group" aria-label="Policy change proposed">
<ApprovalCard tool="Policy change proposed" target={review.path} badge="Review policy" wrapHeader wrapActions diff="" risk=""
Expand All @@ -87,7 +88,7 @@ export function PolicyChangeCard({ toolCall, onApproval }: {
</ApprovalCard>
<span className="sr-only" role="status">Policy change proposed. Review before approving.</span>
</div> : <div ref={receipt} tabIndex={-1} data-policy-receipt className="rounded-md border border-border-subtle p-3 outline-none focus-visible:ring-2 focus-visible:ring-primary/50">
<p role="status" className="text-sm">{busy ? review.message : readError || review.message}</p>
<p role="status" className="text-sm">{voiceLocked ? toolCall.voicePermission?.word : busy ? review.message : readError || review.message}</p>
<p className="font-mono text-xs">{review.state === "updated" ? `${review.baseHash ?? "defaults"} → ${review.proposedHash}` : `Based on ${review.baseHash ?? "no policy file"}`}</p>
{review.state === "updated" && <Button label="View policy" tone="quiet" size="md" style={{ minHeight: 44 }} onClick={() => { const opener = document.activeElement; if (opener instanceof HTMLElement) openFile(opener, opener.matches(":focus-visible")); }} />}
{!busy && (review.state === "unknown" || readError) && <Button label="Check again" tone="quiet" size="md" style={{ minHeight: 44 }} onClick={() => { explicit.current = true; void check(); }} />}
Expand Down
4 changes: 3 additions & 1 deletion packages/ui-react/src/components/chat/tool-call-timeline.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,7 @@ function ToolCallEntryView({
</span>
)}
{/* A closed card's word replaces "Approval needed": it needs nothing. */}
{!(closed && toolCall.status === "pending_approval") && (
{!(closed && toolCall.status === "pending_approval") && !toolCall.voicePermission?.word && (
<StatusIndicator status={toolCall.status} isError={toolCall.isError} />
)}
</span>
Expand All @@ -361,6 +361,8 @@ function ToolCallEntryView({
</p>
)}

{toolCall.voicePermission?.word && <p data-voice-permission-receipt="" className="px-2 text-xs text-foreground">{toolCall.voicePermission.word}{toolCall.voicePermission.heardAt && <span> · heard “brain, stop” · {new Date(toolCall.voicePermission.heardAt).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" })}</span>}</p>}

{/* Live subagent state for tool fan-outs, fed by the activity stream. */}
{renderer.subagentRows && <SubagentEntryRows agentToolUseId={toolCall.id} />}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ import { TurnHeader } from "../chat/transcript-turn.js";
export function ConversationOutputs({ sessionId }: { sessionId: string | null }) {
const root = useBrainUiRoot();
const outputs = useStore(root.connection.conversation.state, (state) => state.outputs);
return <>{outputs.filter((output) => output.sessionId === sessionId && output.generated).map((output) =>
const permission = useStore(root.connection.conversation.permissions.state);
const buffer = useStore(root.stores.chat, state => sessionId === null ? state.draft : state.buffers[sessionId]);
const tools = buffer?.messages.flatMap(m => m.toolCalls) ?? [];
return <>{permission.records.filter(r => r.sessionId === sessionId && r.word && (r.postureDenied || r.heardAt && !tools.some(t => t.id === r.toolUseId))).map(r => <p key={`posture:${r.turnId}:${r.toolUseId}`} data-voice-posture-outcome={r.postureDenied ? "" : undefined} data-voice-permission-outcome="" className="py-2 text-xs">{r.word}</p>)}{outputs.filter((output) => output.sessionId === sessionId && output.generated).map((output) =>
<div key={`${output.conversationId}:${output.epoch}:${output.outputId}`} className="py-4" data-conversation-output="">
<TurnHeader who="Brain" tone="brain" />
<p className="whitespace-pre-wrap">{output.generated}</p>
Expand Down
Loading
Loading