Skip to content

Commit e89400f

Browse files
authored
improvement(extension): simplify page access and enable voice input (#9683)
* improvement(extension): enable page access across websites * fix(extension): distinguish unreadable pages from missing access * improvement(extension): keep page-access copy concise * test(extension): wait for the page access action * fix(extension): complete access grants after dialog dismissal * test(extension): exercise voice restart and sign-in recovery * test(extension): grant microphone for the embedded app origin * fix(extension): enable voice input and resume signed-in panels safely * refactor(extension): shorten copy and verify session recovery * fix(extension): clear stale authentication when returning to Chat * fix(extension): keep recovery actions accessible and connected * test(extension): use accessible link fields and retain voice proof * test(extension): preserve link-flow failures when capturing evidence
1 parent ffa0225 commit e89400f

31 files changed

Lines changed: 746 additions & 619 deletions

File tree

‎.github/scripts/http-e2e.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -279,6 +279,7 @@ case "$group" in
279279
export REDIS_URL=redis://127.0.0.1:6379
280280
export NEXT_PUBLIC_FORCE_HOSTED=false
281281
export KNOWLEDGE_MEMBER_ACCESS=true CREDENTIAL_GROUPS=true ACCESS_CONTROL_ENABLED=true
282+
export SIM_LINKS=true
282283
export NEXT_PUBLIC_ACCESS_CONTROL_ENABLED=true
283284
export API_ENCRYPTION_KEY="$ENCRYPTION_KEY"
284285
start_browser_install "$group" --no-shell chromium
Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,48 @@
11
---
22
title: Sim browser extension
3-
description: Ask Sim about the page you’re reading, share a screenshot, and open your team’s go/ links from Chrome.
3+
description: Ask Sim about a page, dictate a message, and open your team’s go/ links from Chrome.
44
---
55

6-
[Install Sim for Chrome](https://chromewebstore.google.com/detail/sim/oanlcahddodnjpdoopcpjlcageejjpmg) to ask questions about a page without leaving it. Open Chrome’s **Extensions** menu and pin **Sim** to keep it one click away.
6+
[Install Sim for Chrome](https://chromewebstore.google.com/detail/sim/oanlcahddodnjpdoopcpjlcageejjpmg), then pin **Sim** in Chrome’s **Extensions** menu.
77

88
## Ask about a page
99

10-
1. Open a web page and click **Sim** in the Chrome toolbar. Sign in to Sim if prompted.
11-
2. In the side panel, select the attachment chip to review the captured text, or use its **×** to remove it. To attach the page again after it changes, use the globe next to the message field.
12-
3. Ask a question and send it. For example, on a release notes page, ask “Which changes affect my team?”
10+
1. Open a page and click **Sim** in the toolbar. Sign in if prompted.
11+
2. Review the attached page by clicking its chip. Use **×** to remove it.
12+
3. Type your question and send it.
1313

14-
To ask about a passage, select the text on the page, right-click, and choose **Ask Sim about selection**. You can attach up to three snapshots to compare pages. A snapshot stays as it was when you captured it. Chrome settings pages, other extensions, and pages without readable text can’t be attached.
14+
To ask about a passage, select it, right-click, and choose **Ask Sim about selection**. To add another page, open it and use **+ → Current page**. You can attach up to three snapshots; each stays as it was when captured.
1515

16-
Open **+ → Reference current page on send** to include the active tab each time you send a message. This is off by default. The setting appears above the workspace list, and Sim remembers your choice in this Chrome profile. The globe shows **Current page** while this is on. If you attached a page or selection yourself, Sim uses those snapshots instead. Turn the setting off in the same menu.
16+
Choose **+ → Page access → Allow across websites** and approve Chrome’s prompt once to use pages across websites. Your draft stays unsent during setup. To change access later, choose **Page access → Manage in browser**.
1717

18-
If Chrome needs access after you switch tabs, your message stays in the composer. Select **Request site access**. If Chrome hasn’t shared the site address with Sim, paste it into **Site address**. Check the site shown on **Allow access**, approve it in Chrome’s permission dialog, then send again. For one-time access, click the Sim toolbar icon on the active tab and retry. Open **+ → Site access** to remove a saved permission.
18+
Turn on **+ → Reference current page on send** to include the active page whenever you send. Sim remembers your choice. Manually attached pages or selections take priority.
1919

20-
To share a screenshot, drag the image file onto the open chat. It uploads to Sim and appears as an attachment you can remove before sending. If a website supplies only an image URL, paste the image itself or use the file picker.
20+
Chrome settings, other extensions, and pages without readable text can’t be attached.
2121

22-
For faster access, press **Command + Shift + S** on macOS or **Alt + Shift + S** on Windows and Linux. The shortcut attaches highlighted text when you have a selection. It won't substitute the whole page for an unreadable selection. With no selection, it attaches the page or uses your send-time reference setting. On an already-open sidebar, Chrome may keep keyboard focus on the website. Click the message field, or use Chrome’s pane navigation (on Mac, **Command + Option + Up**).
22+
## Dictate or attach a screenshot
2323

24-
If Chrome needs access to a new tab, the sidebar shows the shortcut assigned in this Chrome profile, or **Set shortcut** if none is assigned. You can change it under **Extensions → Keyboard shortcuts**.
24+
Click the microphone beside Send to dictate. Allow microphone access if Chrome asks, then click again to stop. Edit the draft and send when ready. If access is blocked, choose **Open settings**.
2525

26-
## Roll out to a team
26+
Drag a screenshot file onto Chat, or use the file picker. It uploads as an attachment you can remove before sending. Dragging an image URL from a website doesn’t attach the image itself.
2727

28-
Share the chat extension’s install link with your team. In managed Chrome, an admin can [allow the extension for an organizational unit](https://support.google.com/chrome/a/answer/6306504) so people can install it themselves.
28+
## Keyboard shortcut
2929

30-
If your organization has [Links](/search/sim-links), type `sim`, press **Space**, and enter part of a link name for suggestions. To open a shortcut by typing it directly, include the trailing slash, as in `go/handbook/` or `sim/handbook/`. Teams that already use an internal host named `go` or `sim`, or another go-links extension, should keep that conflict in mind: the extension routes those navigations to Sim, and when two extensions redirect the same address, Chrome uses the most recently installed one.
30+
Press **Command + Shift + S** on macOS or **Alt + Shift + S** on Windows and Linux. Sim attaches your selection, or the current page when **Reference current page** is off. With it on, the page is included when you send. An unreadable selection stays unsent.
3131

32-
**+ → Browser shortcuts** can recover explicit `go/handbook` and `sim/handbook` searches on Google, Bing, DuckDuckGo, and Yahoo. Enable your provider and approve the site access shown by Chrome. This leaves ordinary searches and your default search engine unchanged. See [shortcut setup and supported inputs](/search/sim-links#open-links-from-the-address-bar).
32+
Change the binding in Chrome’s **Extensions → Keyboard shortcuts**. If Chrome keeps focus on the page, click the message field or use its pane-navigation shortcut (**Command + Option + Up** on Mac).
3333

34-
## Page access and privacy
34+
## Open team links
3535

36-
Sim reads a page or selection when you attach it, or when you send a message with **Reference current page** enabled. For repeat use on one site, choose **+ → Site access** to grant or revoke that site’s permission. Granting access doesn't start background capture. The page text goes to the hosted composer when attached and becomes part of the conversation when you send it.
36+
Open **Links → Browser shortcuts** in Sim and select **Enable go/ shortcuts**. Then type a shortcut such as `go/handbook` in Chrome’s address bar. You can enable `sim/handbook` there too.
37+
38+
For suggestions, type `sim`, press **Space**, and enter part of a link name. If Chrome searches instead of opening a shortcut, enable your search engine under the extension’s **+ → Browser shortcuts**. Other searches stay unchanged.
39+
40+
See [Links](/search/sim-links#open-links-from-the-address-bar) for supported inputs, Glean conflicts, and team rollout. Administrators can [allow the extension for a Chrome organizational unit](https://support.google.com/chrome/a/answer/6306504).
41+
42+
## Privacy
43+
44+
Sim reads page text when you attach it or send with **Reference current page** on. Approval alone doesn’t start background capture. Attached text appears in the Sim composer before you send; submitted content follows your conversation’s retention settings.
45+
46+
Voice input sends microphone audio to Sim’s transcription provider while recording. It does not record tab audio or start when you open the panel. The transcript stays in your draft until you send.
3747

3848
See [Sim’s browser extension privacy disclosure](https://www.sim.ai/privacy#browser-extension).

‎apps/docs/content/docs/search/sim-links.mdx‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,25 @@ Names ignore case and punctuation, so `sim/Team-Handbook` and `sim/teamhandbook`
1515

1616
## Open links from the address bar
1717

18-
In Sim Cloud, open **Links → Browser shortcuts** to install [Sim for Chrome](https://chromewebstore.google.com/detail/sim/oanlcahddodnjpdoopcpjlcageejjpmg) in your work profile. Type `sim` in Chrome’s address bar, press **Space**, and enter part of a link name. Chrome shows matching links, most opened first. Unlisted links can still open by name.
18+
[Install Sim for Chrome](https://chromewebstore.google.com/detail/sim/oanlcahddodnjpdoopcpjlcageejjpmg) in your work profile. In Sim, open **Links → Browser shortcuts** and select **Enable go/ shortcuts**. Type `go/handbook` in the address bar to open a link. Enable **sim/ shortcuts** there if you prefer `sim/handbook`.
1919

20-
To open links as `go/handbook` or `sim/handbook`, select **Enable go/ shortcuts** and **Enable sim/ shortcuts** in that dialog once each, using the Chrome profile where Sim is installed. Each opens your Links directory in a new tab and lets Chrome remember the short host in that profile. If Chrome still sends a shortcut to search, enter `http://sim/handbook` or `sim/handbook/` with a trailing slash. Use **Copy link** in Sim to share an address that also works without the extension.
20+
For suggestions, type `sim`, press **Space**, and enter part of a name. Matching links appear with the most opened first. Unlisted links still open by name.
2121

22-
Open the extension sidebar’s **+ → Browser shortcuts** and enable your search provider to recover `go/handbook` or `sim/handbook` when Chrome treats it as a search. Approve access to the exact sites shown. This is optional and supports Google (`google.com`), Bing, DuckDuckGo, and Yahoo. Sim remembers the choice in this Chrome profile. Ordinary searches still use your search engine; its default is unchanged.
22+
### If a shortcut opens search
2323

24-
Recovery matches a whole `go/<name>` or `sim/<name>` shortcut, including simple slash-separated values such as `go/jira/123`. For values containing spaces, query parameters, unsupported search domains, or a managed browser that blocks site access, use `sim` + **Space** or the full `https://www.sim.ai/sim/<name>` address. Turn recovery off in **Browser shortcuts**; this keeps the site permission, which you can remove through **Site access** or Chrome’s extension settings.
24+
Open the extension’s **+ → Browser shortcuts** and turn on your search engine. Approve Chrome’s site-access prompt if shown. Google (`google.com`), Bing, DuckDuckGo, and Yahoo are supported. Other searches and your default search engine stay unchanged.
2525

26-
If `sim` + **Space** sends link names to your search engine even when the address bar shows **Sim**, a Chrome site search may already use the `sim` shortcut. Enter `http://sim/handbook` to open the link directly. To restore suggestions, temporarily turn off the Sim extension, rename the old shortcut in `chrome://settings/searchEngines`, then turn Sim back on. If your organization manages the shortcut, ask your administrator to rename it.
26+
This handles whole shortcuts such as `go/handbook` or `go/jira/123`. For spaces, query parameters, unsupported search sites, or blocked site access, use `sim` + **Space** or the full `https://www.sim.ai/sim/<name>` address. A trailing slash (`go/handbook/`) or `http://go/handbook` also tells Chrome to navigate directly.
27+
28+
Turning shortcuts off keeps site access. To remove it, open Chrome’s **Extensions → Manage extensions → Sim → Details**.
29+
30+
### If another tool uses go/ or sim
31+
32+
Sim routes both `go/` and `sim/` to your organization’s links. If Glean or another extension also handles `go/`, Chrome uses the most recently installed extension. Disable the other tool’s go-link handling or remove that extension to use Sim consistently. An existing intranet host named `go` or `sim` conflicts too.
33+
34+
If `sim` + **Space** still searches while the address bar shows **Sim**, an old Chrome site-search shortcut may use the same keyword. Turn off the Sim extension, rename that shortcut in `chrome://settings/searchEngines`, then turn Sim back on. Ask your administrator if the shortcut is managed.
35+
36+
### Use links without the extension
2737

2838
Without the extension, add Sim as a Chrome site search shortcut to type `sim handbook` instead:
2939

‎apps/extension/DESIGN.md‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,20 @@ conversation persistence, streaming recovery, tool permissions, and organization
1010
or workspace authorization. The panel keeps Fast mode and omits the model selector.
1111
A conversation can continue in the full app.
1212

13+
Voice input uses the existing microphone control beside Send. The browser asks for
14+
microphone permission on first use. Transcription updates the editable draft and
15+
never sends it automatically. Stop ends recording; Send and conversation changes
16+
discard unfinished transcription. The iframe delegates microphone access only to
17+
the configured Sim origin, and blocked access offers the browser's microphone settings.
18+
1319
The user signs in in a normal Sim tab and lands in their organization chat. Choosing
1420
a workspace is optional; members without Build access use Assistant. They can open
1521
or create a conversation and explicitly attach the current page or highlighted text.
1622
The composer’s + menu offers Current page and Selected text. Pending snapshots sit
1723
inside the composer as compact removable chips alongside file attachments. Opening one
1824
shows the captured text, source, capture time, and conversation-retention disclosure.
19-
Partial captures remain labeled in the attachment row. Site access lives in the
20-
composer’s Site access action, which opens an extension-owned dialog; successful delivery adds no persistent status banner.
25+
Partial captures remain labeled in the attachment row. Page access lives in the
26+
composer’s Page access action, which opens an extension-owned dialog; successful delivery adds no persistent status banner.
2127
Snapshots are immutable: navigating to another page cannot silently change an attached
2228
source. Removing an attachment, changing organization or workspace, or starting a new conversation
2329
clears pending page context. Several explicitly attached pages can support comparison.
@@ -34,7 +40,7 @@ Existing Sim server tools remain subject to their existing authorization and app
3440
## Ownership and data flow
3541

3642
1. `apps/extension`: packaged MV3 service worker, side-panel host, bounded content
37-
extraction, per-site permission controls, build artifacts, browser verification.
43+
extraction, browser-wide page permission setup, build artifacts, browser verification.
3844
2. `packages/browser-extension-protocol`: versioned, strictly validated messages and
3945
immutable page snapshots shared by the extension, app, and server.
4046
3. `apps/sim/app/extension`: session gate, organization landing, optional workspace picker, app providers,
@@ -57,9 +63,9 @@ other workspace resources or browser-control tools.
5763
and short-lived session storage for selection handoff. Required host access is
5864
limited to the configured Sim origin and to `http(s)://sim/*` and `http(s)://go/*`, which exist
5965
only so one static redirect rule can send top-level navigations to those hosts to the link resolver.
60-
- Other HTTP(S) origins are optional grants requested for the current site from a
61-
user gesture. Each grant can be revoked. Keeping the panel open is not a grant
62-
to read every subsequently selected tab.
66+
- HTTP(S) page access is one optional browser-wide grant, requested from a direct
67+
user gesture. The browser owns revocation and restrictions. Approval enables
68+
capture across sites; switching tabs alone does not extract content.
6369
- No history, bookmarks, cookie API, debugger, native messaging, external messaging,
6470
or arbitrary remote code execution. The only network interception is the static
6571
`sim/` and `go/` redirect rule above, plus opt-in search recovery.
@@ -126,7 +132,7 @@ An absent backend or account is an unverified gate, not a pass.
126132
- [Chrome activeTab](https://developer.chrome.com/docs/extensions/develop/concepts/activeTab)
127133
and [optional permissions](https://developer.chrome.com/docs/extensions/reference/api/permissions):
128134
temporary access follows user invocation; cross-origin navigation requires fresh
129-
authority. Request persistent access per site only when selected by the user.
135+
authority. Request optional access across HTTP(S) websites once, matching Glean’s all-websites page-context option.
130136
- [Chrome extension security](https://developer.chrome.com/docs/extensions/develop/security-privacy/stay-secure)
131137
and [OWASP extension guidance](https://cheatsheetseries.owasp.org/cheatsheets/Browser_Extension_Vulnerabilities_Cheat_Sheet.html):
132138
minimize privileges and treat all incoming page messages as attacker-controlled.

0 commit comments

Comments
 (0)