fix(extension): enable voice input and resume signed-in panels safely

Author: waleedlatif1

.github/scripts/http-e2e.sh

@@ -279,6 +279,7 @@ case "$group" in
     export REDIS_URL=redis://127.0.0.1:6379
     export NEXT_PUBLIC_FORCE_HOSTED=false
     export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
+    export SIM_LINKS=true
     export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true
     export API_ENCRYPTION_KEY="$ENCRYPTION_KEY"
     start_browser_install "$group" --no-shell chromium

apps/docs/content/docs/browser-extension/index.mdx

@@ -19,6 +19,8 @@ In version 0.1.4 or later, choose **+ → Page access → Allow across websites*
 
 To share a screenshot, drag the image file onto the open chat. It uploads to Sim and appears as an attachment you can remove before sending. If a website supplies only an image URL, paste the image itself or use the file picker.
 
+In version 0.1.4 or later, click the microphone beside Send to dictate a message, and allow microphone access if your browser asks. Click it again to stop. You can edit the transcript before sending. If access is blocked, **Open settings** takes you to the browser's microphone controls.
+
 For faster access, press **Command + Shift + S** on macOS or **Alt + Shift + S** on Windows and Linux. The shortcut attaches highlighted text when you have a selection. It won't substitute the whole page for an unreadable selection. With no selection, it attaches the page or uses your send-time reference setting. On an already-open sidebar, Chrome may keep keyboard focus on the website. Click the message field, or use Chrome’s pane navigation (on Mac, **Command + Option + Up**).
 
 Change or assign the shortcut under **Extensions → Keyboard shortcuts**.
@@ -36,3 +38,5 @@ If your organization has [Links](/search/sim-links), type `sim`, press **Space**
 Sim reads a page or selection when you attach it, or when you send a message with **Reference current page** enabled. With version 0.1.4 or later, **Allow across websites** provides access across tabs without repeated site approvals. Granting access doesn't start background capture. The page text goes to the hosted composer when attached and becomes part of the conversation when you send it.
 
 See [Sim’s browser extension privacy disclosure](https://www.sim.ai/privacy#browser-extension).
+
+Voice input sends microphone audio to Sim's transcription provider while recording. It does not record the tab's audio or start when you open the panel. The transcript stays in your draft until you send it.

apps/docs/content/docs/search/sim-links.mdx

@@ -21,7 +21,7 @@ To open links as `go/handbook` or `sim/handbook`, select **Enable go/ shortcuts*
 
 Open the extension sidebar’s **+ → Browser shortcuts** and enable your search provider to recover `go/handbook` or `sim/handbook` when Chrome treats it as a search. Approve access to the exact sites shown. This is optional and supports Google (`google.com`), Bing, DuckDuckGo, and Yahoo. Sim remembers the choice in this Chrome profile. Ordinary searches still use your search engine; its default is unchanged.
 
-Recovery matches a whole `go/<name>` or `sim/<name>` shortcut, including simple slash-separated values such as `go/jira/123`. For values containing spaces, query parameters, unsupported search domains, or a managed browser that blocks site access, use `sim` + **Space** or the full `https://www.sim.ai/sim/<name>` address. Turn recovery off in **Browser shortcuts**; this keeps the site permission, which you can remove through **Site access** or Chrome’s extension settings.
+Recovery matches a whole `go/<name>` or `sim/<name>` shortcut, including simple slash-separated values such as `go/jira/123`. For values containing spaces, query parameters, unsupported search domains, or a managed browser that blocks site access, use `sim` + **Space** or the full `https://www.sim.ai/sim/<name>` address. Turn recovery off in **Browser shortcuts**; this keeps site access. In version 0.1.4, open **Page access → Manage in browser** to remove access.
 
 If `sim` + **Space** sends link names to your search engine even when the address bar shows **Sim**, a Chrome site search may already use the `sim` shortcut. Enter `http://sim/handbook` to open the link directly. To restore suggestions, temporarily turn off the Sim extension, rename the old shortcut in `chrome://settings/searchEngines`, then turn Sim back on. If your organization manages the shortcut, ask your administrator to rename it.
 

apps/extension/DESIGN.md

@@ -10,6 +10,12 @@ conversation persistence, streaming recovery, tool permissions, and organization
 or workspace authorization. The panel keeps Fast mode and omits the model selector.
 A conversation can continue in the full app.
 
+Voice input uses the existing microphone control beside Send. The browser asks for
+microphone permission on first use. Transcription updates the editable draft and
+never sends it automatically. Stop ends recording; Send and conversation changes
+discard unfinished transcription. The iframe delegates microphone access only to
+the configured Sim origin, and blocked access offers the browser's microphone settings.
+
 The user signs in in a normal Sim tab and lands in their organization chat. Choosing
 a workspace is optional; members without Build access use Assistant. They can open
 or create a conversation and explicitly attach the current page or highlighted text.

apps/extension/README.md

@@ -39,9 +39,9 @@ SIM_EXTENSION_DEV=1 SIM_EXTENSION_ORIGIN=http://localhost:3000 bun run --cwd app
 - Source URLs omit credentials, query parameters, and fragments. Human-readable text can still contain sensitive information; it is the user-selected material they review and send into their conversation.
 - Snapshots stay in memory until handed to Sim and acknowledged or expired after 30 seconds. Extension storage contains only a short-lived intent with a tab/document identity and a SHA-256 URL digest, never raw URLs, captured text, or credentials. Closing a tab or navigating invalidates its pending intent. Sim applies its normal conversation retention once a message is sent.
 - No browser history, bookmarks, cookie access, debugger access, remote extension JavaScript, arbitrary network proxy, or background content collection is requested. Incognito use is disabled.
-- The iframe bridge checks both the exact configured origin and the exact frame, then validates versioned messages. The hosted composer may request only page capture, selection capture, or the native site-access and browser-shortcut dialogs. Requests require the current conversation nonce and Chrome transient user activation; arbitrary URLs, tabs, script execution, and network operations are never accepted. Permission grants still require a click in the extension-owned page-access control and browser approval. Shortcut settings are advertised separately so older installed shells keep working without the new action.
+- The iframe bridge checks both the exact configured origin and the exact frame, then validates versioned messages. The hosted composer may request only page capture, selection capture, the native page-access and browser-shortcut dialogs, or the browser's fixed microphone settings page. Requests require the current conversation nonce and Chrome transient user activation; arbitrary URLs, tabs, script execution, and network operations are never accepted. Permission grants still require a click in the extension-owned page-access control and browser approval. Shortcut settings are advertised separately so older installed shells keep working without the new action.
 - Captures and queued toolbar intents are scoped to the receiving conversation. Changing conversation, workspace, or account discards an unfinished handoff. Captured attachments remain explicit snapshots when the source tab changes.
-- The complete hosted document, including portaled source dialogs and menus, is blocked from session replay. Clipboard writing is delegated only to the configured Sim origin. Clipboard reading and microphone capture are not delegated; voice input remains available in the main app.
+- The complete hosted document, including portaled source dialogs and menus, is blocked from session replay. Clipboard writing and microphone capture are delegated only to the configured Sim origin; clipboard reading and camera access are not delegated. Voice input starts only after the user clicks the microphone and the browser grants permission. The shared Sim voice service streams microphone audio to ElevenLabs for transcription into the editable draft. Stop ends capture while allowing the final transcript to arrive; Send and conversation changes cancel unfinished transcription. There is no tab-audio capture or background recording.
 
 With **Reference current page** enabled, a composer Send requests a single fresh snapshot for that conversation. The extension stores only the preference in local storage. A tab change does not read any content. Explicit snapshots take precedence; access denial, navigation and timeouts keep the draft unsent. **Allow across websites** works even when Chrome hides the active address; no address entry is required. The grant covers HTTP and HTTPS sites across tabs until the user revokes or restricts it in browser settings. Uploaded screenshot files stay in the hosted app’s normal attachment flow.
 

apps/extension/src/search-recovery-dialog.tsx

@@ -140,8 +140,7 @@ export function SearchRecoveryDialog({ onClose }: SearchRecoveryDialogProps) {
             </ChipModalField>
           ))}
           <p className='px-2 text-[var(--text-muted)] text-caption'>
-            Chrome asks for access to the sites shown above. Page reference stays unchanged. Turning
-            a shortcut off keeps site access.
+            Turning shortcuts off keeps browser site access.
           </p>
           {loadError && (
             <div className='flex flex-wrap items-center gap-2 px-2'>

apps/extension/src/sidepanel.tsx

@@ -349,6 +349,8 @@ function SidePanel({ origin }: SidePanelProps) {
         if (!navigator.userActivation.isActive) return
         if (parsed.data.action === 'site-access') setAccessOpen(true)
         else if (parsed.data.action === 'browser-shortcuts') setShortcutsOpen(true)
+        else if (parsed.data.action === 'microphone-settings')
+          void chrome.tabs.create({ url: 'chrome://settings/content/microphone' })
         else void capture(parsed.data.action === 'capture-page' ? 'page' : 'selection')
       } else if (pendingRef.current?.id === parsed.data.snapshotId) {
         pendingRef.current = null

apps/sim/app/(landing)/privacy/privacy-content.tsx

@@ -803,6 +803,12 @@ export const PRIVACY_CONFIG: LegalPageConfig = {
       id: 'browser-extension',
       heading: '10B. Sim Browser Extension',
       blocks: [
+        {
+          kind: 'paragraph',
+          content: richText(
+            'Versions 0.1.4 and later support voice input. When You click the microphone and grant browser permission, Sim streams microphone audio to ElevenLabs for transcription into Your editable message draft. Opening the panel does not start recording, and this feature does not capture tab audio. Stopping voice input ends microphone capture; sending a message or changing conversations cancels unfinished transcription. Submitted transcripts become conversation content under the retention and processing rules below.'
+          ),
+        },
         {
           kind: 'paragraph',
           content: richText(

apps/sim/app/extension/[workspaceId]/chat.tsx

@@ -20,6 +20,7 @@ import { useChat } from '@/app/workspace/[workspaceId]/home/hooks/use-chat'
 import { fetchWorkspaceResourceRef } from '@/app/workspace/[workspaceId]/home/resolve-resource-ref'
 import type { FileAttachmentForApi } from '@/app/workspace/[workspaceId]/home/types'
 import { useMothershipChats } from '@/hooks/queries/mothership-chats'
+import { VoiceInputPermissionContext } from '@/hooks/use-voice-input'
 import type { ChatContext } from '@/stores/panel'
 
 interface ExtensionChatProps {
@@ -119,99 +120,101 @@ export function ExtensionChat({ workspaceId, workspaceName }: ExtensionChatProps
   )
 
   return (
-    <main
-      ref={fileDropTarget}
-      className='ph-no-capture flex h-dvh min-w-0 flex-col overflow-hidden bg-[var(--bg)] text-[var(--text-body)]'
-    >
-      <ExtensionHeader
-        workspaceId={workspaceId}
-        scopeName={workspaceName}
-        history={history}
-        chatId={chat.resolvedChatId}
-        onSelectChat={selectChat}
-        appHref={appHref}
-      />
-      {chat.error && (
-        <p role='alert' className='px-3 py-2 text-[var(--text-error)] text-caption'>
-          {chat.error}
-        </p>
-      )}
-      {composer.captureError && (
-        <p role='alert' className='px-3 py-2 text-[var(--text-error)] text-caption'>
-          {composer.captureError}
-        </p>
-      )}
-      {!selectedChatId && chat.messages.length === 0 && (
-        <div className='flex flex-1 items-center justify-center px-5'>
-          <h1 className='text-lg'>How can I help?</h1>
-        </div>
-      )}
-      <MothershipChat
-        onNavigateToChat={selectChat}
-        workspaceId={workspaceId}
-        userId={userId}
-        chatId={chat.resolvedChatId}
-        messages={chat.messages}
-        isSending={chat.isSending}
-        isReconnecting={chat.isReconnecting}
-        isLoading={
-          Boolean(selectedChatId) && chat.messages.length === 0 && chat.isChatHistoryPending
-        }
-        onSubmit={(message, files, contexts) => submit(message, files, contexts, false)}
-        onComposerSubmit={submit}
-        onStopGeneration={stop}
-        messageQueue={chat.messageQueue}
-        editingQueuedId={chat.editingQueuedId}
-        dispatchingHeadId={chat.dispatchingHeadId}
-        onRemoveQueuedMessage={chat.removeFromQueue}
-        onSendQueuedMessage={chat.sendNow}
-        onEditQueuedMessage={chat.editQueuedMessage}
-        onCancelQueueEdit={chat.cancelQueueEdit}
-        onViewSources={(messageId, requestId) =>
-          void setSourceParams(
-            { 'source-message': messageId, 'source-request': requestId ?? null },
-            { history: 'push' }
-          )
-        }
-        onWorkspaceResourceSelect={(resource) => void openResource(resource)}
-        composerKey={composerKey}
-        fileDropTarget={fileDropTarget}
-        draftScopeKey={`extension:${userId}:${workspaceId}:${selectedChatId ?? 'new'}`}
-        composerAttachments={composer.attachments}
-        composerResourceActions={composer.resourceActions}
-        composerToolbarActions={composer.toolbarActions}
-        composerModelSelectorOptions={composer.modelSelectorOptions}
-        layout='copilot-view'
-        className={cn('h-auto min-h-0', (selectedChatId || chat.messages.length > 0) && 'flex-1')}
-      />
-      <ChipModal
-        open={Boolean(sourceMessage)}
-        onOpenChange={(open) => {
-          if (!open) closeSources()
-        }}
-        srTitle='Cited sources'
-        size='sm'
+    <VoiceInputPermissionContext value={composer.openMicrophoneSettings}>
+      <main
+        ref={fileDropTarget}
+        className='ph-no-capture flex h-dvh min-w-0 flex-col overflow-hidden bg-[var(--bg)] text-[var(--text-body)]'
       >
-        <ChipModalHeader onClose={closeSources}>Cited sources</ChipModalHeader>
-        {sourceMessage && (
-          <div className='h-[60dvh] min-h-0'>
-            <SourcesResourceContent
-              chatId={chat.resolvedChatId}
-              resource={{
-                id: 'cited-sources',
-                type: 'sources',
-                title: 'Sources',
-                sources: {
-                  messageId: sourceMessage.id,
-                  ...(sourceParams['source-request']
-                    ? { requestId: sourceParams['source-request'] }
-                    : {}),
-                },
-              }}
-            />
+        <ExtensionHeader
+          workspaceId={workspaceId}
+          scopeName={workspaceName}
+          history={history}
+          chatId={chat.resolvedChatId}
+          onSelectChat={selectChat}
+          appHref={appHref}
+        />
+        {chat.error && (
+          <p role='alert' className='px-3 py-2 text-[var(--text-error)] text-caption'>
+            {chat.error}
+          </p>
+        )}
+        {composer.captureError && (
+          <p role='alert' className='px-3 py-2 text-[var(--text-error)] text-caption'>
+            {composer.captureError}
+          </p>
+        )}
+        {!selectedChatId && chat.messages.length === 0 && (
+          <div className='flex flex-1 items-center justify-center px-5'>
+            <h1 className='text-lg'>How can I help?</h1>
           </div>
         )}
-      </ChipModal>
-    </main>
+        <MothershipChat
+          onNavigateToChat={selectChat}
+          workspaceId={workspaceId}
+          userId={userId}
+          chatId={chat.resolvedChatId}
+          messages={chat.messages}
+          isSending={chat.isSending}
+          isReconnecting={chat.isReconnecting}
+          isLoading={
+            Boolean(selectedChatId) && chat.messages.length === 0 && chat.isChatHistoryPending
+          }
+          onSubmit={(message, files, contexts) => submit(message, files, contexts, false)}
+          onComposerSubmit={submit}
+          onStopGeneration={stop}
+          messageQueue={chat.messageQueue}
+          editingQueuedId={chat.editingQueuedId}
+          dispatchingHeadId={chat.dispatchingHeadId}
+          onRemoveQueuedMessage={chat.removeFromQueue}
+          onSendQueuedMessage={chat.sendNow}
+          onEditQueuedMessage={chat.editQueuedMessage}
+          onCancelQueueEdit={chat.cancelQueueEdit}
+          onViewSources={(messageId, requestId) =>
+            void setSourceParams(
+              { 'source-message': messageId, 'source-request': requestId ?? null },
+              { history: 'push' }
+            )
+          }
+          onWorkspaceResourceSelect={(resource) => void openResource(resource)}
+          composerKey={composerKey}
+          fileDropTarget={fileDropTarget}
+          draftScopeKey={`extension:${userId}:${workspaceId}:${selectedChatId ?? 'new'}`}
+          composerAttachments={composer.attachments}
+          composerResourceActions={composer.resourceActions}
+          composerToolbarActions={composer.toolbarActions}
+          composerModelSelectorOptions={composer.modelSelectorOptions}
+          layout='copilot-view'
+          className={cn('h-auto min-h-0', (selectedChatId || chat.messages.length > 0) && 'flex-1')}
+        />
+        <ChipModal
+          open={Boolean(sourceMessage)}
+          onOpenChange={(open) => {
+            if (!open) closeSources()
+          }}
+          srTitle='Cited sources'
+          size='sm'
+        >
+          <ChipModalHeader onClose={closeSources}>Cited sources</ChipModalHeader>
+          {sourceMessage && (
+            <div className='h-[60dvh] min-h-0'>
+              <SourcesResourceContent
+                chatId={chat.resolvedChatId}
+                resource={{
+                  id: 'cited-sources',
+                  type: 'sources',
+                  title: 'Sources',
+                  sources: {
+                    messageId: sourceMessage.id,
+                    ...(sourceParams['source-request']
+                      ? { requestId: sourceParams['source-request'] }
+                      : {}),
+                  },
+                }}
+              />
+            </div>
+          )}
+        </ChipModal>
+      </main>
+    </VoiceInputPermissionContext>
   )
 }