v0.8.41: library, coda connector, connector sync engine hardening

.github/workflows/test-build.yml

@@ -231,8 +231,10 @@ jobs:
         run: >-
           bunx vitest run --mode integration
           lib/knowledge/__integration__/search-source-progress.integration.ts
+          lib/knowledge/__integration__/organization-search-overview.integration.ts
           lib/knowledge/__integration__/search-source-pagination.integration.ts
           lib/knowledge/__integration__/search-reference-batching.integration.ts
+          lib/knowledge/__integration__/embedding-insert-batches.integration.ts
           lib/knowledge/__integration__/kb-block-search.integration.ts
           lib/core/outbox/service.integration.ts
           lib/knowledge/__integration__/connector-upload.integration.ts

apps/docs/content/docs/knowledgebase/connectors.mdx

@@ -16,11 +16,11 @@ Connectors continuously sync documents from external services into your knowledg
 
 <Image src="/static/connectors/connectors-sources.png" alt="The current Connect Source picker showing searchable connectors including Airtable, Asana, Ashby, Azure DevOps, Bitbucket, Box, and ClickUp" width={750} height={680} className="mx-auto h-auto w-full max-w-md" />
 
-Sim ships with 64 built-in connectors:
+Sim ships with 66 built-in connectors:
 
 | Category | Connectors |
 |----------|-----------|
-| **Productivity** | Notion, Confluence, Asana, Linear, Jira, Jira Service Management, Monday, Trello, ClickUp, Google Calendar, Google Sheets, Google Forms, Microsoft Excel, Typeform |
+| **Productivity** | Coda, Notion, Confluence, Asana, Linear, Jira, Jira Service Management, Monday, Trello, ClickUp, Google Calendar, Google Sheets, Google Forms, Microsoft Excel, Typeform |
 | **Cloud Storage** | Google Drive, Dropbox, OneDrive, SharePoint, Box, Amazon S3, SFTP |
 | **Documents** | Google Docs, Google Slides, Mintlify, WordPress, Webflow, DocuSign |
 | **Development** | GitHub, GitLab, Bitbucket, Azure DevOps, Sentry |
@@ -31,7 +31,7 @@ Sim ships with 64 built-in connectors:
 | **Incident Management** | incident.io, Rootly, PagerDuty |
 | **Data** | Airtable, Databricks |
 | **Note-taking** | Obsidian |
-| **Meetings** | Zoom, Google Meet, Gong, Grain, Granola, Fathom, Fireflies |
+| **Meetings** | Zoom, Google Meet, Gong, Grain, Granola, Fathom, Fireflies, Circleback |
 | **Recruiting** | Greenhouse, Ashby |
 | **HR** | Workday Help |
 | **Compliance** | Google Vault |
@@ -52,6 +52,7 @@ Other connectors use **API keys** or **personal access tokens** instead. The set
 | Connector | Where to get the key |
 |-----------|---------------------|
 | **Obsidian** | Install the [Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api) plugin, then copy the key from its settings |
+| **Coda** | API token from Account settings → API connections, saved as a Coda service account credential |
 | **Fireflies** | Generate from the Integrations page in your Fireflies account |
 | **Typeform** | Personal access token from your Typeform account settings |
 | **Azure DevOps** | Personal access token with Wiki (Read), Work Items (Read), and Code (Read) scopes |

apps/docs/content/docs/search/coda.mdx

@@ -0,0 +1,66 @@
+---
+title: Coda
+description: Connect Coda documents while preserving each teammate's access
+---
+
+import { Step, Steps } from 'fumadocs-ui/components/steps'
+import { Image } from '@/components/ui/image'
+
+An organization admin connects Coda once. Teammates search using their verified Sim email address; they do not need a personal Coda connection. Their email must match the address Coda reports.
+
+## Connect Coda
+
+<Steps>
+<Step>
+
+### Create an API token
+
+Sign in to Coda with a dedicated integration account that can read the documents and their sharing permissions. Open **Account settings → API connections → Generate API token**, name the token, and copy it. Coda's current app may be labeled **Superhuman Docs**.
+
+Use a standard REST API token. Do not select the MCP restriction.
+
+<Image className="mx-auto h-auto w-full max-w-2xl" src="/static/search/coda-api-token.jpg" alt="Coda API connections with the Generate new token dialog" width={1280} height={720} />
+
+</Step>
+<Step>
+
+### Add the source
+
+In Sim, open **Settings → Sources → Add source → Coda**. Add or select a Coda service account credential and enter the token when prompted.
+
+| Field | What to enter |
+|---|---|
+| Documents | Select documents, or switch to **Document IDs** and enter up to 100 IDs separated by commas. For `https://coda.io/d/_dAbCDeFGH`, the ID is `AbCDeFGH`. |
+| Enterprise organization ID | Leave blank for ordinary access. For organization-wide indexing, enter the ID from Coda's [List organizations API](https://coda.io/developers/apis/admin/v1#operation/listOrganizations) and use an Enterprise organization administrator's token. Workspace admin access alone is insufficient. |
+
+The picker shows accessible documents the token owner has opened. Use IDs for other accessible documents. With no selection, Sim discovers the token owner's opened documents, or the organization's documents in Enterprise mode. Use explicit IDs when deleted documents must reliably be removed from the index; discovery listings alone are not authoritative for deletion.
+
+</Step>
+<Step>
+
+### Sync and verify
+
+Select **Connect & Sync**. Open the source's **Documents** and **Sync history** to check indexing. Content and permission changes appear after synchronization.
+
+To rotate the token, add the replacement credential, update the source, and verify a sync before revoking the old token.
+
+</Step>
+</Steps>
+
+## Content and access
+
+Ordinary connections index visible canvas text and base-table rows. Enterprise connections index the Admin API's plaintext page exports; tables are included only as represented in those exports. Attachments, comments, images, and remote embeds are not extracted.
+
+| Permission | Ordinary connection | Enterprise connection |
+|---|---|---|
+| Owner or direct email share | Matching verified Sim email | Matching active organization user |
+| Group, workspace, or domain share | Not expanded | Resolved through the Admin API |
+| Anyone with the link | Does not grant Search access | Does not grant Search access |
+
+Enterprise mode excludes deactivated and deleted users. Direct-share guests absent from the organization directory are excluded; guests explicitly listed in a group or workspace may receive that membership's access. Ordinary connections cannot check Coda organization deactivation, so manage departing users' Sim membership too.
+
+Sim refreshes permissions even when content is unchanged. Unverified permissions never become broad access, and Sim admin status does not override Coda sharing. For setup failures or missing results, check the token's access, Enterprise admin role if applicable, the teammate's verified email, and **Sync history**.
+
+In a regular knowledge base, **Workspace access** intentionally shares all indexed content with that Sim workspace and does not use the Enterprise organization ID. The permissions above apply to organization Search and **Admin access** sources.
+
+See Coda's [public API](https://coda.io/developers/apis/v1) and [Enterprise Admin API](https://coda.io/developers/apis/admin/v1).

apps/docs/content/docs/search/confluence.mdx

@@ -142,7 +142,7 @@ In **Sync history**, **Continuing** means a healthy listing needs another batch.
 | Space picker is empty or fails | Check the domain, account's space access, and `read:space:confluence` scope. Manual space keys are also supported. |
 | Service-account validation fails | Check token expiry, site, Confluence app access, and the full scope list above, including `read:confluence-user`. |
 | Content syncs but Search is empty | Connect your personal Confluence identity. Check permission/directory sync errors and group-read scopes. |
-| **Some permissions could not be verified** | Open the source's **Sync history**. Check the service account's space, page, and directory access. If access is correct and the warning persists, ask your operator to inspect the connector run's permission errors. Do not broaden sharing to clear the warning. |
+| **Permission verification incomplete** | Open the source's **Sync history**. Check the service account's space, page, and directory access. If access is correct and the warning persists, ask your operator to inspect the connector run's permission errors, including incomplete or repeated permission pages. Documents without verified access stay hidden; do not broaden sharing to clear the warning. |
 | A new page, blog post, or label is missing | Confluence search can take time to update. Once the content appears in Confluence search with the selected label, sync again. |
 | A restricted page is missing | Both your account and the crawling account need access to the page and its ancestors. |
 | Embedded content is missing | Index the referenced page separately; remote macro output is excluded. |

apps/docs/content/docs/search/connect-your-account.mdx

@@ -60,6 +60,7 @@ For a source configured inside a workspace, join that workspace and connect thro
 | GitHub App installation | Connect GitHub once for this Sim organization. The App handles indexing; your account establishes which repositories you may search. |
 | Confluence service account | Connect Confluence to verify your identity; the service account handles the crawl. |
 | Google Workspace service account (Gmail, Calendar, Drive) | No personal connection is needed for that source. Your verified Sim email identifies your mailbox and calendar view, or is matched to Drive permissions. |
+| [Coda](/search/coda) | No personal connection is needed. Your verified Sim email must match a supported Coda sharing grant. |
 | GitLab | No personal connection is needed. For an administrator token, your verified Sim email must match your confirmed primary GitLab email. For a non-admin token, it must match the uploaded user mapping and have a project grant. |
 
 Gmail, Calendar, and Drive are separate Search sources. Connecting one Google service does not connect all of them. Central Google and GitLab sources remain searchable without a personal account connection.

apps/docs/content/docs/search/gmail.mdx

@@ -133,7 +133,7 @@ Search schedules syncs hourly. The first sync lists every thread in scope and ca
 
 **Member accounts:** later syncs use each mailbox's Gmail change history, unless the configuration has a search filter. A full relisting runs about weekly, or sooner if Gmail no longer retains the saved history.
 
-**Service account:** each sync revisits the selected active mailboxes and resumes unfinished listings. A failed mailbox read leaves the crawl incomplete; it does not cause existing indexed mail to be deleted from Search.
+**Service account:** each sync revisits the selected active mailboxes and resumes unfinished listings. If Google reports that a user's mailbox is not set up or returns a mailbox `failedPrecondition`, Sim records a warning and continues with the remaining users. The crawl stays incomplete and retries affected users on the next scheduled crawl; existing indexed mail is not deleted because a mailbox could not be read. Credential, delegation, and Directory failures still stop the crawl.
 
 Updates, removals, and access refresh in the background. Empty mailboxes and filters with no matches complete normally with zero documents. Threads exceeding indexing size limits are skipped and reconsidered when they change.
 
@@ -149,7 +149,7 @@ An individual thread failure does not mean the whole mailbox failed. Sim retries
 | Reconnect | Click **Reconnect** and authorize the same account again. |
 | Unavailable or needs admin attention | Ask your Sim admin to check source status and the deployment's Google OAuth configuration. |
 | Directory or delegation error | Check both central crawl scopes, the service-account key, and the Directory administrator's user-read privileges. A normal OAuth account cannot replace the central service account. |
-| Gmail access fails for a selected user | Verify delegation is authorized and [Gmail is enabled](https://knowledge.workspace.google.com/admin/gmail/control-gmail-access-for-your-organizations-users) for that primary Workspace account. Set **Users** to accounts with Gmail enabled; leaving it blank includes all active users and can stop sync on a service-access error. Aliases and external accounts cannot be selected. |
+| Gmail access fails for a selected user | Verify delegation is authorized and [Gmail is enabled](https://knowledge.workspace.google.com/admin/gmail/control-gmail-access-for-your-organizations-users) for that primary Workspace account. Check the affected users in **Sync history**. Set **Users** to accounts with Gmail enabled if some active users should not be crawled. Aliases and external accounts cannot be selected. |
 | A central source indexes mail but a teammate sees no results | Confirm their verified Sim email is the mailbox's primary email and they belong to the Sim organization. Administrators do not receive other people's mailbox access. |
 
 ## Self-hosted operator setup

apps/docs/content/docs/search/google-calendar.mdx

@@ -131,7 +131,7 @@ Sim indexes event titles, descriptions, times, locations, and the selected atten
 
 Cancelled events, attachment contents, meeting recordings, and transcripts are not indexed. Status entries such as working location, out of office, focus time, and birthdays, and automatically generated reservation events from Gmail are not indexed. Events Google returns only as free/busy blocks, without searchable details, are not indexed. Events outside the selected date window are excluded. Private event details that Google withholds are not available in Search; see [Google's calendar sharing rules](https://developers.google.com/workspace/calendar/api/concepts/sharing).
 
-Search schedules syncs hourly. Event edits, cancellations, access changes, inactive or removed users, and events moving outside the date window are reconciled during completed background syncs. Central crawls page through each selected user and resume unfinished work before removing documents no longer listed. Authorization, quota, and provider failures stop the sync rather than treating unread calendars as empty. The first sync may take longer, and results appear as indexing progresses; Search is not a live Calendar read.
+Search schedules syncs hourly. Event edits, cancellations, access changes, inactive or removed users, and events moving outside the date window are reconciled during completed background syncs. Central crawls page through each selected user and resume unfinished work before removing documents no longer listed. If an individual user's event listing returns a `403` with an explicit `forbidden` reason and no other reasons, Sim records a warning and continues with the remaining users. The crawl stays incomplete and retries affected users on the next scheduled crawl; unread calendars are not treated as empty. A `403` without a reason stops the crawl because its cause is unknown. Credential, delegation, Directory, and other provider failures still stop the crawl. The first sync may take longer, and results appear as indexing progresses; Search is not a live Calendar read.
 
 ## Troubleshooting
 
@@ -146,7 +146,7 @@ When a sync fails, **Sync history** includes the Google API operation, HTTP stat
 | Reconnect | Click **Reconnect** and complete Google authorization again. Allow pop-ups if the connection tab does not open. |
 | Unavailable or needs admin attention | Ask your Sim admin to check source status and the deployment's Google OAuth configuration. |
 | Service-account authorization or Directory error | Confirm both delegated scopes, enabled APIs, and the Directory administrator's user-read privilege. Check whether delegation still awaits approval or propagation. |
-| Calendar is disabled for a selected user | An active Workspace user may have Calendar turned off. [Enable Calendar](https://knowledge.workspace.google.com/admin/users/access/turn-calendar-on-or-off-for-users) for them, or set **Users** to accounts with Calendar enabled. Leaving **Users** blank includes all active users and can stop sync on a service-access error. |
+| Calendar access fails for a selected user | Check the affected users in **Sync history**. An active Workspace user may have Calendar turned off, but a `403` alone does not prove this. Check their access to the selected calendars and [Calendar service settings](https://knowledge.workspace.google.com/admin/users/access/turn-calendar-on-or-off-for-users), or set **Users** to accounts that should be crawled. |
 | User not found or inactive | Use an active primary email in the same Workspace customer. Aliases, external or guest accounts, suspended users, and archived users cannot be selected. |
 | A central source has no results for a teammate | Confirm their primary Workspace email matches their verified Sim email, they belong to the Sim organization, and they are included in **Users**. Check calendar IDs and **Sync history**. |
 

apps/docs/content/docs/search/google-drive.mdx

@@ -155,13 +155,14 @@ Search schedules syncs hourly. Central crawls revisit the selected users' files
 
 ## Troubleshooting
 
-**Directory permission sync failed** means Sim could not fully verify group membership. Check the Directory administrator’s access to the affected group and any nested groups; this is separate from file-download access. An incomplete membership read does not replace the last verified membership, which remains subject to freshness checks.
+Directory permission warnings mean Sim could not fully verify group membership. When other groups refresh successfully and only nested groups outside your Workspace customer deny access or cannot be found, Sim continues syncing content and reports a partial permission sync. An incomplete membership read does not replace or refresh the last verified membership, which remains subject to freshness checks. Directory-wide failures still stop the sync. Check the Directory administrator’s access to the affected group and nested groups; this is separate from file-download access.
 
 | Problem | Next step |
 | --- | --- |
 | Google rejects authorization (`unauthorized_client`) | In **Manage Domain Wide Delegation**, verify the numeric **Client ID** matches `client_id` in the JSON key uploaded to Sim and all required scopes appear under **View details**. Check pending approval and allow time for recent changes to propagate. Changing the OAuth consent screen alone does not authorize delegation. |
 | Directory access failed | Check all four delegated scopes and the **Directory administrator email** user's administrator privileges. A normal Google OAuth credential cannot supply this central Search path. |
 | Missing files in a central crawl | Check **Users**, folder and file-type filters, and whether selected active Workspace users can download the file and read its permissions. Opening a file alone does not prove either. Check Sync history for errors. Files reachable only by excluded or inactive accounts are not crawled; files with unverified permissions stay hidden. |
+| No text could be extracted | Images and scanned PDFs use OCR, but files with no extractable text cannot be indexed. Check whether the original contains readable text; a successful download does not guarantee searchable content. |
 | User not found or inactive | Use a primary email in the same Google Workspace customer. Aliases, external or guest accounts, suspended users, and archived users cannot be selected for crawling. |
 | A teammate sees no results | Confirm they have joined the Sim organization and their verified Sim email matches the Drive permission or group membership. For member accounts, finish their personal Drive connection too. |
 | A public or shared-link file is missing | Check **Openly shared files**. Link-only sharing does not grant Search access. A named user or group permission can still make the file searchable. |

apps/docs/content/docs/search/meta.json

@@ -3,6 +3,7 @@
   "pages": [
     "connect-your-account",
     "mcp",
+    "coda",
     "confluence",
     "github",
     "gitlab",