Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ src/
├── web/ # Dashboard server (server.ts) and page (dashboard.ts: do not edit)
├── lite/ # libscope/lite: createLite() preset of LibScope, createCodeChunker(), TreeSitterChunker, normalizeRawInput()
├── db/ # SQLite (schema.ts migrations, connection.ts, index-meta.ts embedding model of the index)
├── providers/ # Embedding providers (local/xenova, ollama, openai)
├── providers/ # Embedding providers (local, ollama, openai)
├── registry/ # Git-backed pack registries (config, git, sync, publish, search, resolve, checksum)
├── connectors/ # notion, slack, confluence, onenote, obsidian, docs; registry.ts (types, saved connections, sync)
├── config-schema.ts # The zod config schema: keys, defaults, env var names, docs table (getConfigKeyTable)
Expand Down
4 changes: 1 addition & 3 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# Debian (glibc) base: the local embedding model (onnxruntime-node) does not run on Alpine (musl).
# sharp and onnxruntime-node ship prebuilt binaries, so only better-sqlite3 is rebuilt. Skipping the
# onnxruntime-node install script also skips its optional CUDA download (about 270 MB).
# better-sqlite3 is the only native addon; the production stage rebuilds it.

# Build stage
FROM node:22-bookworm-slim AS builder
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ libscope ask "How do I configure OAuth2?"
libscope doctor
```

The database is created on first use. The first command that embeds text downloads the local model [all-MiniLM-L6-v2](https://huggingface.co/Xenova/all-MiniLM-L6-v2) (about 80 MB, first run only).
The database is created on first use. The first command that embeds text downloads the local model [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) (about 90 MB, first run only).

### MCP (Claude, Cursor, VS Code, ...)

Expand Down
2 changes: 1 addition & 1 deletion docs/guide/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ src/
│ ├── embedding.ts # EmbeddingProvider interface
│ ├── index.ts # createEmbeddingProvider() from config
│ ├── dimensions.ts # vector sizes of known models
│ ├── local.ts # @huggingface/transformers (all-MiniLM-L6-v2)
│ ├── local.ts # all-MiniLM-L6-v2 on onnxruntime-web (WebAssembly)
│ ├── ollama.ts # Ollama HTTP API
│ └── openai.ts # OpenAI embeddings API
├── registry/ # git pack registries: config, git, sync, search, resolve, publish, checksum
Expand Down
4 changes: 2 additions & 2 deletions docs/guide/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,11 @@ Embeddings turn text into vectors for semantic search. LibScope supports three p

| Provider | Default | Requires | Notes |
| -------- | ------- | ---------------------- | --------------------------------------------- |
| `local` | ✅ | Nothing | all-MiniLM-L6-v2, ~80MB download on first use |
| `local` | ✅ | Nothing | all-MiniLM-L6-v2, ~90MB download on first use |
| `ollama` | | Ollama running locally | Uses nomic-embed-text by default |
| `openai` | | API key | Uses text-embedding-3-small by default |

The local provider works out of the box — no API keys, no external services. It runs the model in-process using `@huggingface/transformers` and caches it in `~/.libscope/models`.
The local provider works out of the box — no API keys, no external services. It runs the model in-process on the ONNX Runtime WebAssembly build (no native addon) and caches it in `~/.libscope/models`.

```bash
# Switch to Ollama
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Requires Node.js 20 or later.
libscope doctor
```

This shows the config files, the active workspace and its database (`~/.libscope/workspaces/default/libscope.db`), the embedding model, and which LLM `ask` will use. Each warning comes with the command that fixes it. You do not have to create anything first: the database is created when you add the first document (or run `libscope doctor --fix`). The first command that embeds text downloads the local embedding model (~80MB, once) and prints one line while it does.
This shows the config files, the active workspace and its database (`~/.libscope/workspaces/default/libscope.db`), the embedding model, and which LLM `ask` will use. Each warning comes with the command that fixes it. You do not have to create anything first: the database is created when you add the first document (or run `libscope doctor --fix`). The first command that embeds text downloads the local embedding model (~90MB, once) and prints one line while it does.

## Index Some Documents

Expand Down
2 changes: 1 addition & 1 deletion docs/guide/lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ createLite(options: LiteOptions): LibScope

### Embedding providers

The default is the local `all-MiniLM-L6-v2` model (about 80 MB, downloaded on first use). To use OpenAI or Ollama, set it in `config`:
The default is the local `all-MiniLM-L6-v2` model (about 90 MB, downloaded on first use). To use OpenAI or Ollama, set it in `config`:

```ts
const scope = createLite({
Expand Down
2 changes: 1 addition & 1 deletion docs/guide/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ If you're on an unsupported platform, LibScope will fall back to keyword-only se

**Symptom:** First run hangs at "Downloading embedding model ... (first run only)..." or fails with a network error.

**Cause:** The local embedding model (~80MB) downloads from Hugging Face on first use.
**Cause:** The local embedding model (~90MB) downloads from Hugging Face on first use.

**Fix:**

Expand Down
19 changes: 17 additions & 2 deletions docs/migration-v2.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Migrating to LibScope 2.0

LibScope 2.0 removes old names without aliases. This page lists every removed or renamed name and what to use instead. Your indexed data is not changed by the upgrade.
LibScope 2.0 removes old names without aliases. This page lists every removed or renamed name and what to use instead.

- [Rebuild the vector index (2.0.1)](#rebuild-the-vector-index-201): required once
- [Configuration](#configuration): config keys, environment variables, API keys
- [Library API](#library-api): package root exports, the `LibScope` class, `libscope/lite`
- [MCP](#mcp): server startup, renamed, merged and removed tools
Expand All @@ -14,6 +15,20 @@ LibScope 2.0 removes old names without aliases. This page lists every removed or

The references list the 2.0 names: [CLI](reference/cli.md), [MCP tools](reference/mcp-tools.md), [REST API](reference/rest-api.md) and [configuration](reference/configuration.md).

## Rebuild the vector index (2.0.1)

LibScope 2.0.1 changes the vectors of the local embedding provider, so vector indexes built by earlier versions do not match it. Upgrade to 2.0.1 or later, then rebuild the vector index once:

```bash
libscope admin reindex --rebuild
```

Until you do, commands that use the vector index stop with an error that names the old and the new model. Documents, chunks and keyword search are not changed.

The local provider now runs the full-precision all-MiniLM-L6-v2 model on the ONNX Runtime WebAssembly build, and truncates input at 256 tokens as sentence-transformers does. Its vectors match the original model. Installing LibScope no longer runs a native install script for the model runtime, and no longer downloads CUDA binaries on Linux. The first command that embeds text downloads the model again (about 90 MB, once).

Vector indexes built by LibScope 1.x with any provider must also be rebuilt, because 1.x did not record which model built them.

## Configuration

### Config keys
Expand Down Expand Up @@ -50,7 +65,7 @@ These variables do not change: `LIBSCOPE_EMBEDDING_PROVIDER`, `LIBSCOPE_LLM_PROV
| `llm.provider` had no default; `ask` failed until it was set. | `llm.provider` defaults to `auto`: passthrough under MCP, else OpenAI if an OpenAI key is set, else Anthropic if an Anthropic key is set, else Ollama if `llm.url` is set or the embedding provider is `ollama`. |
| An invalid value in a config file was used as is. | An invalid value is ignored with a warning, and the default (or lower layer) applies. |
| `LIBSCOPE_ALLOW_PRIVATE_URLS=false` did not override `true` in a file. | `LIBSCOPE_INDEXING_ALLOW_PRIVATE_URLS=false` (or `0`) sets `false`. |
| The local embedding model ran on `@xenova/transformers` and was cached in `node_modules/@xenova/transformers/.cache`. | The local embedding model runs on `@huggingface/transformers` and is cached in `~/.libscope/models` (`/data/.libscope/models` in the Docker image). The first command that embeds text after the upgrade downloads the model again, once. The model (all-MiniLM-L6-v2, quantized) and its 384-dimension vectors do not change, so you do not have to reindex. |
| The local embedding model ran on `@xenova/transformers` and was cached in `node_modules/@xenova/transformers/.cache`. | The local embedding model is cached in `~/.libscope/models` (`/data/.libscope/models` in the Docker image). The first command that embeds text after the upgrade downloads the model again, once. Its vectors change in 2.0.1: [rebuild the vector index](#rebuild-the-vector-index-201). |

### Library (SDK) code

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Every result shows document IDs (and chunk IDs for search hits), so you can pass

**Errors.** A failed command prints `✗ <message>` and a next step on stderr, and exits with code 1. For example, an unknown document ID suggests `libscope docs list`, and a configuration problem suggests `libscope doctor`. Add `--verbose` to see the stack trace.

**Progress.** Long operations (directory imports, crawls, reindexing, pack installs) show a progress line on stderr when it is a terminal. The first command that needs the local embedding model prints one line while the model downloads (about 80 MB, first run only).
**Progress.** Long operations (directory imports, crawls, reindexing, pack installs) show a progress line on stderr when it is a terminal. The first command that needs the local embedding model prints one line while the model downloads (about 90 MB, first run only).

### Document filters

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Every key `section.field` has the environment variable `LIBSCOPE_<SECTION>_<FIEL
| Key | Type | Default | Environment variables | Secret | Description |
| ------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `embedding.provider` | `local` \| `ollama` \| `openai` | `local` | `LIBSCOPE_EMBEDDING_PROVIDER` | | Embedding provider. |
| `embedding.model` | string | ollama: nomic-embed-text; openai: text-embedding-3-small | `LIBSCOPE_EMBEDDING_MODEL` | | Embedding model. Ignored by the local provider (always Xenova/all-MiniLM-L6-v2). |
| `embedding.model` | string | ollama: nomic-embed-text; openai: text-embedding-3-small | `LIBSCOPE_EMBEDDING_MODEL` | | Embedding model. Ignored by the local provider (always sentence-transformers/all-MiniLM-L6-v2). |
| `embedding.url` | string | `http://localhost:11434` | `LIBSCOPE_EMBEDDING_URL` | | Server URL of the embedding provider. Used by the ollama provider. |
| `embedding.dimensions` | integer | size of the model | `LIBSCOPE_EMBEDDING_DIMENSIONS` | | Vector size, for models whose size is not built in. |
| `llm.provider` | `auto` \| `openai` \| `anthropic` \| `ollama` \| `passthrough` | `auto` | `LIBSCOPE_LLM_PROVIDER` | | LLM for "ask". "auto": passthrough under MCP, else openai if an OpenAI key is set, else anthropic if an Anthropic key is set, else ollama if llm.url is set or embedding.provider is ollama, else none. |
Expand Down
Loading
Loading