diff --git a/README.md b/README.md index a866490..753a9f2 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,11 @@ license CI node - tests +<<<<<<< HEAD + tests +======= + tests +>>>>>>> feat/serve-daemon coverage Awesome

@@ -151,7 +155,7 @@ dsh web ## 不开 DSH 也能服务（独立服务 daemon） -`dsh-mneme-serve` 把记忆库跑成常驻数据面——DSH 关着，第三方集成（网页端桥接、脚本、自有面板）照样读写同一份记忆：token 与 DSH 面板/CLI 共用，路由与「外部访问 API」同源，端口被占直接报错（与 DSH 外部访问二选一）。第一期无 LLM（巩固/蒸馏仍属 DSH 宿主），检索为关键词 + BM25，向量接入在后续版本。 +`dsh-mneme-serve` 把记忆库跑成常驻数据面——DSH 关着，第三方集成（网页端桥接、脚本、自有面板）照样读写同一份记忆：token 与 DSH 面板/CLI 共用，路由与「外部访问 API」同源，端口被占直接报错（与 DSH 外部访问二选一）。第一期无 LLM（巩固/蒸馏仍属 DSH 宿主）；检索默认 local 嵌入（首次启动自动取件模型，`--embed off` 可关）。 ```bash npm i -g @modusensus/dsh-mneme @@ -193,7 +197,11 @@ dsh-mneme-serve # 默认 ~/.dsh/memory + 127.0.0.1:8790 ```bash cd dsh-mneme && npm install -npm test # 1535 个测试 +<<<<<<< HEAD +npm test # 1537 个测试 +======= +npm test # 1537 个测试 +>>>>>>> feat/serve-daemon npm run stress # 三轴线压测 npm run sync # src → lib 同步 ``` @@ -344,7 +352,7 @@ The plugin ships a zero-dependency stdio MCP server (standalone npm package **`m ## Serve memories without DSH (standalone daemon) -`dsh-mneme-serve` runs the memory store as a long-lived data plane — with DSH closed, third-party integrations (web-bridge tools, scripts, your own panels) still read and write the same memories: the Bearer token is shared with the DSH panel/CLI, routes mirror the external API, and a busy port is a hard error (pick either the daemon or DSH's external API, not both). Phase 1 is LLM-free (consolidation/distillation stay with the DSH host); retrieval is keyword + BM25, with vector search arriving in a later release. +`dsh-mneme-serve` runs the memory store as a long-lived data plane — with DSH closed, third-party integrations (web-bridge tools, scripts, your own panels) still read and write the same memories: the Bearer token is shared with the DSH panel/CLI, routes mirror the external API, and a busy port is a hard error (pick either the daemon or DSH's external API, not both). Phase 1 is LLM-free (consolidation/distillation stay with the DSH host); retrieval defaults to local embeddings (runtime + model auto-provisioned on first boot, `--embed off` to disable). ```bash npm i -g @modusensus/dsh-mneme @@ -386,7 +394,11 @@ dsh-mneme-serve # defaults: ~/.dsh/memory + 127.0.0.1:8790 ```bash cd dsh-mneme && npm install -npm test # 1535 tests +<<<<<<< HEAD +npm test # 1537 tests +======= +npm test # 1537 tests +>>>>>>> feat/serve-daemon npm run stress # three-axis stress test npm run sync # src → lib sync ``` diff --git a/dsh-mneme/CHANGELOG.md b/dsh-mneme/CHANGELOG.md index a0ed6dc..c9e8f95 100644 --- a/dsh-mneme/CHANGELOG.md +++ b/dsh-mneme/CHANGELOG.md @@ -4,7 +4,9 @@ ## 🆕 新增 -- **独立服务 daemon（`dsh-mneme-serve`，#363）**：mneme 现在能在 DSH 宿主之外常驻——`src/serve.js` 的 `createServeRuntime` 用最小装配（store → settings → mirror → service → maintenance → standalone API，每步锚定 index.js 装配行号）把数据面跑成独立进程，第三方集成（网页端桥接等）不必为挂载记忆库而保持 DSH 开机。第一期刻意无 LLM：巩固（autoDream）与蒸馏结构性不在 daemon 内，这是与宿主「单写者」的机械保证，不靠用户自觉；检索为关键词 + BM25（向量由后续 PR 抽取 semantic 装配后接入）。token 与 DSH 面板/CLI 共用同一 kv 凭证，端口/主机解析链与外部访问一致；`createStandaloneApi` 新增 `strictPort` 选项——daemon 的配置端口被占即报错退出而非顺延（第三方把 URL 写死，静默换端口等于坏），不传该选项的宿主旁路行为不变。`/search` 照常落 recall_runs，第三方检索的复用统计不缺数。多进程共存（daemon 与宿主同库互写互读）有专门回归锁；已知限制（双进程去重竞态、镜像双写、版本偏斜）见 docs/DAEMON.md。 +- **daemon 向量检索（PR2，#363）**：`dsh-mneme-serve` 的 `/search` 接入完整语义管线——embedder/reranker 装配与 boot 自动回填从 `index.js` **纯搬移**至 `src/semantic.js`（宿主与 daemon 共用同一份，调用时序契约原样；`backfillMissingEmbeddings` 经 index.js barrel 再出口，测试调用方零改动），daemon 侧新增 `createVectorIndex` 接线与 `--embed` 参数：`local`（默认，自管 runtime/嵌入模型缺失时经 `provisionRuntime` download 档自动取件，可用 `DSH_MNEME_RUNTIME_TARBALL_DIR`/`DSH_MNEME_RUNTIME_MIRROR` 换离线/镜像来源；失败降级关键词并打可操作日志）、`ollama`、`openai`（读宿主面板 vector-config）、`off`。向量轴有注入假 embedder 的回归锁；`createServeRuntime` 因此转为 async、语义键默认值在 `daemonSemanticCfg` 逐键锚定 config.js。 +- **独立服务 daemon（`dsh-mneme-serve`，#363）**：mneme 现在能在 DSH 宿主之外常驻——`src/serve.js` 的 `createServeRuntime` 用最小装配（store → settings → mirror → service → maintenance → standalone API，每步锚定 index.js 装配行号）把数据面跑成独立进程，第三方集成（网页端桥接等）不必为挂载记忆库而保持 DSH 开机。第一期刻意无 LLM：巩固（autoDream）与蒸馏结构性不在 daemon 内，这是与宿主「单写者」的机械保证，不靠用户自觉。token 与 DSH 面板/CLI 共用同一 kv 凭证，端口/主机解析链与外部访问一致；`createStandaloneApi` 新增 `strictPort` 选项——daemon 的配置端口被占即报错退出而非顺延（第三方把 URL 写死，静默换端口等于坏），不传该选项的宿主旁路行为不变。`/search` 照常落 recall_runs，第三方检索的复用统计不缺数。多进程共存（daemon 与宿主同库互写互读）有专门回归锁；已知限制（双进程去重竞态、镜像双写、版本偏斜）见 docs/DAEMON.md。 + ## 🧹 工程 - **发布准备脚本在 CRLF 检出上不再假成功（`scripts/release-prep.mjs`）**：该脚本用 `/^(# Changelog\n\n)/` 匹配 CHANGELOG 文件头，而 Windows 检出是 CRLF——正则命中不了，`replace` 退化成空操作，**脚本却照样打印 `✓ … 占位节`**，`git status` 里看不出任何异常（CI 跑在 ubuntu 是 LF，所以只有本机发版会中招，v0.8.13 那次即如此、最后靠人工补的占位节）。规则抽成 `dsh-mneme/scripts/changelog-prep.mjs` 的纯函数：行尾两种都吃、插入内容跟随原文件行尾、带 BOM 也认；匹配不上则如实回报 `header-not-found`，入口**报错退出（exit 1）**而不是假打印成功。配 6 条回归测试（LF / CRLF / BOM / 幂等 / 回报契约 / detectEol）。 diff --git a/dsh-mneme/README.md b/dsh-mneme/README.md index fd026b1..407a064 100644 --- a/dsh-mneme/README.md +++ b/dsh-mneme/README.md @@ -5,7 +5,7 @@ [![npm version](https://img.shields.io/npm/v/@modusensus/dsh-mneme?color=blue&label=npm)](https://www.npmjs.com/package/@modusensus/dsh-mneme) [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![Awesome](https://awesome-dsh-plugin.com/badge.svg)](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin) -[![tests](https://img.shields.io/badge/tests-1535%20passed-success)](https://github.com/slow-stack/mneme) +[![tests](https://img.shields.io/badge/tests-1537%20passed-success)](https://github.com/slow-stack/mneme) [![CI](https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml)](https://github.com/slow-stack/mneme/actions) [![node](https://img.shields.io/badge/node-22%2B-blue)](https://nodejs.org) [![npm downloads](https://img.shields.io/npm/d18m/@modusensus/dsh-mneme.svg?color=blue&label=downloads)](https://www.npmjs.com/package/@modusensus/dsh-mneme) @@ -522,11 +522,12 @@ dsh-mneme config show # 查看当前配置（toke ```bash dsh-mneme-serve # 默认 ~/.dsh/memory + 8790 dsh-mneme-serve --memory-dir "D:\my mem" --port 8790 --host 127.0.0.1 +dsh-mneme-serve --embed off # 纯关键词 + BM25(不取件模型) ``` - **鉴权与端口**：Bearer token 与 DSH 面板 / CLI 共用同一份（kv `external_api`，首次启动自动生成并持久化到 `memory.db`）；端口/主机解析链与「外部访问」一致（显式参数 > 持久值 > 默认 8790/127.0.0.1）。配置端口被占会**直接报错退出**（不做端口顺延）——第三方把 URL 写死，静默换端口等于坏。因此 **daemon 与 DSH 的「外部访问」二选一**，不要同端口同开。 - **安全**：daemon 使用明文 HTTP，不提供原生 TLS。指定非回环 `--host` 时，请勿直接把服务暴露给不可信网络；远程访问请走 TLS 终止代理或 SSH 隧道。 -- **能力边界（第一期，无 LLM）**：存储 / 检索（关键词 + BM25）/ 镜像同步与人改合并 / `POST /maintenance/reclaim` / `/bootstrap` 全可用；巩固（autoDream）与蒸馏不在 daemon 内——巩固只属于 DSH 宿主进程，这是与宿主「单写者」的机械保证。向量检索暂缺（后续版本接入），`/search` 退化为关键词 + BM25 属预期。 +- **能力边界（第一期，无 LLM）**：存储 / 检索（关键词 + BM25 + 向量）/ 镜像同步与人改合并 / `POST /maintenance/reclaim` / `/bootstrap` 全可用；巩固（autoDream）与蒸馏不在 daemon 内——巩固只属于 DSH 宿主进程，这是与宿主「单写者」的机械保证。`--embed` 默认 `local`（自管 runtime 与嵌入模型缺失时自动取件，约 200MB；失败降级关键词并打日志），也可选 `ollama` / `openai`（读宿主面板的 vector-config）/ `off`。 - **检索回执**：daemon 的 `/search` 同样落 `recall_runs`，第三方检索的复用统计不缺数。 - **生命周期**：stdout 仅就绪时打一行 `dsh-mneme-serve listening on http://host:port (pid N)`（供脚本解析实际端口），日志走 stderr；SIGINT/SIGTERM 优雅收库，Windows 强杀由 WAL 回放兜底。 - **已知限制**：与 DSH 同时运行属设计内场景（WAL 多进程并发），但去重是先查后写、库层无 UNIQUE 约束，双进程并发写同一 `(type, title, scope)` 有极小概率产生重复；daemon 与插件请同版本升级。细节与坑清单见 [docs/DAEMON.md](docs/DAEMON.md)。 @@ -600,7 +601,7 @@ src/ ├── api.js # HTTP 路由（Web 面板数据通道，含 /conflicts 冲突队列） └── index.js # 插件接线 lib/ # src 的同步分发产物（npm run sync；发布前由 root prepack 的 check-sync.js 校验一致性；唯一手写例外 lib/client.js——Web 面板 bundle，sync 不覆盖） -test/ # 1535 个 node:test 测试（审计与三轴线压测不变量；src↔lib 一致性由 scripts/check-sync.js 发布闸门校验） +test/ # 1537 个 node:test 测试（审计与三轴线压测不变量；src↔lib 一致性由 scripts/check-sync.js 发布闸门校验） scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压测 · sync-lib.js 同步 · check-sync.js 发布闸门 · benchmark-recall.js / benchmark-embed.js / benchmark-rerank.js 基准 · sync-test-badge.mjs 测试徽章 · build-runtime-manifest.mjs 运行时清单 ``` @@ -609,7 +610,7 @@ scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压 ```bash cd dsh-mneme npm install # 安装 peer 依赖（以 devDependencies 形式，用于本地测试） -npm test # 运行 1535 个测试 +npm test # 运行 1537 个测试 npm run stress # 三轴线压测：长会话检索 / 冲突仲裁 / 多 Agent 并发（离线 mock LLM） npm run sync # 把 src/ 同步到 lib/（发布时由 prepack 钩子自动执行） ``` diff --git a/dsh-mneme/bin/dsh-mneme-serve.mjs b/dsh-mneme/bin/dsh-mneme-serve.mjs index 8424b62..8281376 100644 --- a/dsh-mneme/bin/dsh-mneme-serve.mjs +++ b/dsh-mneme/bin/dsh-mneme-serve.mjs @@ -23,18 +23,27 @@ const PKG = JSON.parse(readFileSync(join(PKG_ROOT, "package.json"), "utf8")); const USAGE = `${BIN_NAME} — run the mneme data plane as a standalone service (no DSH required) -Usage: dsh-mneme-serve [--memory-dir ] [--port ] [--host ] +Usage: dsh-mneme-serve [--memory-dir ] [--port ] [--host ] [--embed ] Options: --memory-dir data directory (default: ~/.dsh/memory, same as the plugin) --port HTTP port (default: persisted external_api port, else 8790) --host bind address (default: persisted external_api host, else 127.0.0.1) + --embed semantic retrieval: local (default, downloads the ONNX runtime + + embedding model on first boot) | ollama | openai (uses the + vector-config saved by the DSH panel) | off (keyword + BM25 only) -h, --help show this help -V, --version print version Auth: Bearer token is shared with the DSH panel / CLI (kv "external_api" in memory.db); it is generated on first boot. A busy configured port is a hard -error — the DSH external API and this daemon must not share a port (pick one).`; +error — the DSH external API and this daemon must not share a port (pick one). + +Environment (runtime provisioning, local provider only): + DSH_MNEME_MEMORY_DIR data directory override + DSH_MNEME_RUNTIME_DIR self-managed runtime dir (default ~/.dsh/mneme/runtime) + DSH_MNEME_RUNTIME_TARBALL_DIR offline .tgz dir preferred over the network + DSH_MNEME_RUNTIME_MIRROR npm registry mirror prefix (e.g. npmmirror)`; /** 极简 argv 解析(--k=v / --k v / 旗标);够用即可,完整 CLI 在 bin/cli.mjs。 */ function parseArgv(argv) { @@ -93,7 +102,21 @@ async function main(argv) { } const host = typeof args.host === "string" && args.host ? args.host : undefined; - const rt = createServeRuntime({ memoryDir, port, host, logger }); + let embed = "local"; + if (args.embed !== undefined) { + if (typeof args.embed !== "string" || !["off", "local", "ollama", "openai"].includes(args.embed)) { + fail(`--embed 需要 off|local|ollama|openai,收到: ${String(args.embed)}`); + } + embed = args.embed; + } + + let rt; + try { + // 装配是异步的:embed=local 时可能要先取件 runtime(download 档,失败内部降级) + rt = await createServeRuntime({ memoryDir, port, host, logger, embed }); + } catch (err) { + fail(`启动失败: ${err?.message ?? err}`); + } try { await rt.api.ready; } catch (err) { diff --git a/dsh-mneme/docs/DAEMON.md b/dsh-mneme/docs/DAEMON.md index a5e4acc..dd22bc2 100644 --- a/dsh-mneme/docs/DAEMON.md +++ b/dsh-mneme/docs/DAEMON.md @@ -6,7 +6,7 @@ daemon 是**数据面**,不是第二个宿主: -- **有**:存储(SQLite)、检索(关键词 + BM25,`/search` 统一召回)、镜像同步与人改合并、`/maintenance/reclaim`、`/bootstrap`、recall_runs 检索回执。 +- **有**:存储(SQLite)、检索(关键词 + BM25 + 向量,`/search` 统一召回,与宿主共用 `src/semantic.js` 同一套装配)、镜像同步与人改合并、`/maintenance/reclaim`、`/bootstrap`、recall_runs 检索回执。 - **没有(第一期,无 LLM)**:巩固(autoDream)、蒸馏(autoSummarize)、实体抽取、sleep、注入/工具/面板路由。前两者是**结构性缺失**而非开关——daemon 装配里没有 LLM 句柄,巩固只属于 DSH 宿主进程。这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md externalApi/autoDream 单侧纪律的 daemon 版),不依赖用户自觉。 与宿主装配(`src/index.js` apply)的关系:`src/serve.js` 只搬数据面那一半,每步注释锚定 index.js 来源行号;刻意不抽公共装配函数(apply 其余环节与宿主 ctx 纠缠,防御段纪律「最后动或不动」)。装配漂移风险由 `test/serve-bin.test.js` 的多进程共存用例兜底(两进程真开同一个库互写互读)。 @@ -18,10 +18,11 @@ daemon 是**数据面**,不是第二个宿主: CLI: ```bash -dsh-mneme-serve [--memory-dir ] [--port ] [--host ] +dsh-mneme-serve [--memory-dir ] [--port ] [--host ] [--embed ] ``` - `memoryDir`:CLI > env `DSH_MNEME_MEMORY_DIR` > `~/.dsh/memory`(与宿主 config.js 同默认,支持前导 `~`)。 +- `--embed`:语义检索提供方。`local`(默认:自管 runtime 与嵌入模型缺失时**自动取件**,download 档,约 200MB;失败降级关键词并打可操作日志)| `ollama` | `openai`(读宿主面板存的 vector-config)| `off`(纯关键词 + BM25)。取件来源可用 env 换道:`DSH_MNEME_RUNTIME_DIR`(自管 runtime 目录)、`DSH_MNEME_RUNTIME_TARBALL_DIR`(离线 .tgz 目录,优先于联网)、`DSH_MNEME_RUNTIME_MIRROR`(registry 镜像前缀)。 - port/host 解析链与宿主「外部访问」一致:显式参数 > kv `external_api` 持久值 > 默认 8790 / 127.0.0.1。 - token 与 DSH 面板 / CLI **共用同一份**(kv `external_api`,首次启动自动生成并持久化)——三方零配置互通。 - 安全:daemon 使用明文 HTTP,不提供原生 TLS。指定非回环 `--host` 时,请勿直接把服务暴露给不可信网络;远程访问请走 TLS 终止代理或 SSH 隧道。 @@ -30,7 +31,8 @@ dsh-mneme-serve [--memory-dir ] [--port ] [--host ] ## 3. 内部文件 -- `src/serve.js` — `createServeRuntime({memoryDir, port, host, logger, strictPort})`:装配链 createStore → createSettings → createMirror → createService(最小 config)→ recoverMirror → 人改镜像合并 → recall recorder → createMaintenance → createStandaloneApi,每步锚定 index.js 行号。返回 `{api, store, service, settings, maintenance, tokenExisted, dispose}`;第三方可 import 它自行托管生命周期(bin 只是薄壳)。 +- `src/serve.js` — `createServeRuntime({memoryDir, port, host, logger, strictPort, embed, embedder, reranker})`:装配链 createStore → createSettings → createMirror → createService(最小 config)→ recoverMirror → 人改镜像合并闭包 → vectorIndex + semantic → recall recorder → createMaintenance → createStandaloneApi,每步锚定 index.js 行号。返回 `{api, store, service, settings, maintenance, semantic, tokenExisted, dispose}`;第三方可 import 它自行托管生命周期(bin 只是薄壳),`embedder/reranker` 参数供注入自管嵌入。 +- `src/semantic.js` — embedder/reranker 装配 + boot 自动回填,**纯搬移自 index.js**(PR2),宿主与 daemon 共用同一份;`backfillMissingEmbeddings` 经 index.js barrel 再出口(测试照旧从 index.js import)。 - `bin/dsh-mneme-serve.mjs` — CLI 壳。独立成 bin 而非 cli.mjs 子命令:CONTRIBUTING 禁止给 cli.mjs 加 import;命名循 dsh-mneme-mcp 先例。 - `src/api-standalone.js` 的 `strictPort` 选项 — 唯一的数据面改动,默认关闭。 @@ -40,5 +42,5 @@ dsh-mneme-serve [--memory-dir ] [--port ] [--host ] 2. **双进程写并发**:与 DSH 同时运行是设计内场景(WAL + busy_timeout 先序,store.js createStore)。但 `saveWithDedupe` 的 (type,title,scope) 去重是先查后写、库层无 UNIQUE 约束,两进程并发写同一三元组有极小概率产生重复条目——已知限制,勿当强保证宣传(要不要加 UNIQUE 索引属 schema 防御段,单独决策)。 3. **镜像双写竞态**:daemon 与宿主都会渲镜像 .md;可再生物,失败由 recoverMirror 自愈,极端并发下单文件可能短暂脏,下次同步覆盖。 4. **版本偏斜**:库迁移是幂等加法式(PRAGMA 检查 + ALTER),旧代码读新 schema 一般无碍,但该组合无人测过——daemon 与插件请同版本升级。 -5. **第一期不吃宿主配置**:daemon 不加载 config schema(schemastery 是宿主 peer 依赖),面板/feature_flags 对它不生效;它只有 CLI 参数 + 上述固定最小 config(`language: zh`、document 子系统关闭)。向量检索(依赖 embedder/reranker 装配)由后续 PR 接入,接入前 `/search` 退化关键词 + BM25 属预期。 -6. **验收锚点**:`test/serve.test.js`(in-process 全链路 + token 复用 + recall_runs 回执)、`test/serve-bin.test.js`(真子进程 + 多进程互写互读)、`test/standalone-api.test.js` 的 strictPort 用例(busy → reject,默认路径仍顺延)。 +5. **第一期不吃宿主配置**:daemon 不加载 config schema(schemastery 是宿主 peer 依赖),面板/feature_flags 对它不生效;它只有 CLI 参数 + 上述固定最小 config(`language: zh`、document 子系统关闭;语义键默认值逐键锚定 config.js,见 `src/serve.js` 的 `daemonSemanticCfg`)。`--embed local` 首次启动会自动取件 runtime + 模型(均有日志);取件/嵌入失败统一降级关键词 + BM25,不影响读写。 +6. **验收锚点**:`test/serve.test.js`(in-process 全链路 + token 复用 + recall_runs 回执 + 注入假 embedder 的向量轴锁)、`test/serve-bin.test.js`(真子进程 + 多进程互写互读)、`test/standalone-api.test.js` 的 strictPort 用例(busy → reject,默认路径仍顺延)、`test/reindex-backfill.test.js`(semantic 纯搬移后宿主行为不变)。 diff --git a/dsh-mneme/lib/index.js b/dsh-mneme/lib/index.js index 21a1b64..8118cd9 100644 --- a/dsh-mneme/lib/index.js +++ b/dsh-mneme/lib/index.js @@ -19,10 +19,10 @@ import { createStandaloneApi } from "./api-standalone.js"; import { createMaintenance } from "./maintenance.js"; import { createSettings } from "./settings.js"; import { createCommandManager } from "./commands.js"; -import { createEmbedder } from "./embedding.js"; -import { createEmbedderByProvider } from "./local-embedder.js"; -import { LocalReranker } from "./reranker.js"; import { createVectorIndex } from "./vector-index.js"; +// semantic(embedder/reranker/boot 回填)纯搬移至 src/semantic.js(PR2):宿主与 +// daemon(dsh-mneme-serve,#363)共用同一套装配,原文件保留 barrel 出口。 +import { createSemantic } from "./semantic.js"; import { Config, applyLightModePreset, injectChildEnabled } from "./config.js"; import { langOf } from "./lang.js"; import { extractEntities } from "./entities/extractor.js"; @@ -149,43 +149,9 @@ export function createEntityStreamAdapter({ llm, agentDefaultModel, logger, serv }; } -/** - * Issue #128: bounded backfill of rows still missing an embedding (active rows - * only — needsEmbedding filters archived/forgotten). Exported for tests. - * - * Runs regardless of the model fingerprint: the old call-site gate returned - * early when vector_meta already held the embedder's hash, permanently - * orphaning rows whose embed failed at write time (embedder not ready / - * provider rate limit) — one successful embed was enough to never backfill - * again. markModel is idempotent when the fingerprint already matches, so - * re-running costs nothing beyond the actually-missing rows. - */ -export async function backfillMissingEmbeddings({ - store, embedder, vectorIndex, logger, - maxTotal = 500, batchSize = 10, rateLimitMs = 200 -}) { - let indexed = 0; - for (let done = 0; done < maxTotal;) { - const rows = store.needsEmbedding(batchSize); - if (!rows.length) break; - for (const row of rows) { - try { - const text = [row.title, row.content].filter(Boolean).join("\n"); - const vector = await embedder.embedSingle(text); - if (vector?.length) { - store.setEmbedding(row.id, vector); - indexed++; - } - } catch { /* skip the bad row */ } - } - done += rows.length; - // Rate limit: space out batches so the provider is not hammered. - if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); - } - if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); - logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); - return indexed; -} +// backfillMissingEmbeddings 已随语义装配整体搬至 src/semantic.js(纯搬移); +// 保留 barrel 再出口 —— test/reindex-backfill.test.js 仍从本模块 import,调用方零改动。 +export { backfillMissingEmbeddings } from "./semantic.js"; export const apply = (ctx, config) => { const rawCfg = Config(config); @@ -339,150 +305,14 @@ export const apply = (ctx, config) => { } }; - let embedder = null; - let reranker = null; - // #118: pending embedder-init retry timer, cleared on unload. - let embedRetryTimer = null; - if (lightMode) { - // Light mode: the whole vector pipeline stays off — no embedder (nothing - // pulls in ONNX/transformers), no reranker, no boot backfill (the preset - // also cleared autoReindexOnBoot). Recall degrades to keyword search and - // human mirror edits still merge on boot. - applyHumanEdits(); - } else if (cfg.embedProvider === "openai") { - // vectorIndex is passed so the legacy OpenAI embedder records the producing - // model fingerprint after each successful embed (Bug3). - embedder = createEmbedder({ store, settings, logger: ctx.logger, vectorIndex }); - service.setEmbedder(embedder); - // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 - // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 - // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 - if (embedder.configured === false) { - ctx.logger?.warn?.( - "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" - + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" - + "dream 的语义聚类会退化为全量窗口兜底。" - + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" - ); - } - // legacy OpenAI embedder needs no async init → human edits apply right away - applyHumanEdits(); - } else { - try { - embedder = createEmbedderByProvider(cfg.embedProvider, { - model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, - dimension: cfg.localEmbedDimension, - device: cfg.localEmbedDevice, - batchSize: cfg.localEmbedBatchSize, - // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， - // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 - pooling: cfg.localEmbedPooling, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - baseUrl: cfg.ollamaBaseUrl, - logger: ctx.logger - }); - service.setEmbedder(embedder); - // issue #6: wait for extractor init before applying human edits, so - // scheduled embeddings see a ready embedder. - const bootEmbedder = () => embedder.init() - .then(() => { applyHumanEdits(); return true; }) - .catch(() => false); - // #118: the old one-shot probe permanently degraded search to keyword - // when Ollama was briefly unreachable at boot (recoverable only by - // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); - // search degrades to keyword meanwhile because per-query embed failures - // are swallowed. - bootEmbedder().then((ok) => { - if (ok) return; - let tries = 4; - const retry = () => { - if (tries-- <= 0) { - ctx.logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); - service.setEmbedder(null); - applyHumanEdits(); - return; - } - embedRetryTimer = setTimeout(async () => { - if (await bootEmbedder()) return; - retry(); - }, 15_000); - }; - ctx.logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); - retry(); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); - applyHumanEdits(); - } - } - - // Cross-encoder rerank over recall candidates. Best-effort: a failed model - // load only disables reranking, never search itself. Explicit opt-in only - // (rerankEnabled defaults to false): constructing LocalReranker is what pulls - // in onnxruntime, so the default config never loads it (item ⑥). - if (cfg.rerankEnabled && cfg.rerankProvider === "local") { - try { - reranker = new LocalReranker({ - model: cfg.rerankModel, - batchSize: cfg.rerankBatchSize, - maxCandidates: cfg.rerankMaxCandidates, - scoreThreshold: cfg.rerankScoreThreshold, - device: cfg.localEmbedDevice, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； - // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 - useDtype: cfg.rerankDtype, - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - logger: ctx.logger - }); - service.setReranker(reranker); - reranker.init().catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); - service.setReranker(null); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); - } - } - - // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API - // is configured and rows still lack an embedding (e.g. written before vector - // search was enabled), the backfill runs in the background after a short - // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a - // large backlog never floods the provider. Failures degrade silently — - // search stays keyword. - function scheduleAutoReindex() { - if (cfg.autoReindexOnBoot === false) return; - const attempt = (tries) => { - try { - if (!embedder || typeof embedder.embedSingle !== "function") return; - if ("ready" in embedder && embedder.ready !== true) { - // Local/ollama embedders init asynchronously; give them a moment - // before giving up on this boot (next boot retries). - if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); - return; - } - if (!store.needsEmbedding(1).length) return; // nothing to backfill - // Issue #128: no fingerprint gate here anymore — a matching fingerprint - // used to return early and permanently orphan rows whose embed failed - // at write time. See backfillMissingEmbeddings(). - backfillMissingEmbeddings({ store, embedder, vectorIndex, logger: ctx.logger }) - .catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - } - }; - setTimeout(() => attempt(5), 5000); - } - scheduleAutoReindex(); + // embedder/reranker/boot 回填装配已整体搬至 src/semantic.js(纯搬移,宿主与 daemon + // 共用):调用时序(applyHumanEdits 在各分支的触发点、#118 重试、autoReindexOnBoot) + // 原样保留在 createSemantic 内部,这里只拿引用。init 全失败的 embedder 引用仍会进入 + // dream/sleep 的 semantic 面 —— 与搬移前一致:检索侧 setEmbedder(null) 降级关键词。 + const semantic = createSemantic({ + store, service, settings, cfg, logger: ctx.logger, vectorIndex, applyHumanEdits, lightMode + }); + const { embedder, reranker } = semantic; // Custom commands: register persisted commands into the DSH command registry // on boot; add/remove re-register live through the API. @@ -601,7 +431,7 @@ export const apply = (ctx, config) => { // #118: never let a pending embedder init retry fire after unload and touch // a torn-down context. - disposers.push(() => { if (embedRetryTimer !== null) clearTimeout(embedRetryTimer); }); + disposers.push(() => semantic.dispose()); // #118 重试计时器 + boot 回填计时器(搬入 semantic.js 后由它自持) ctx.inject(["systemPrompt"], (promptCtx) => { if (cfg.autoInject) disposers.push(createInjector(promptCtx, service, settings, cfg)); diff --git a/dsh-mneme/lib/semantic.js b/dsh-mneme/lib/semantic.js new file mode 100644 index 0000000..a236aa1 --- /dev/null +++ b/dsh-mneme/lib/semantic.js @@ -0,0 +1,229 @@ +// src/semantic.js —— embedder/reranker 装配与 boot 自动回填。 +// 纯搬移自 src/index.js(2026-10,PR2):backfillMissingEmbeddings(原 :150-186)与 +// 装配段(原 :311-477),行为逐字节对齐,仅两处已注记的机械差异(ctx.logger → 注入 +// logger;boot 回填的首查计时器纳入 dispose)。搬移原因:daemon(dsh-mneme-serve, +// #363)与宿主要共用同一套语义装配——「向量检索开箱即用」的承诺落在两侧同一份 +// 代码上,而不是 daemon 复刻一份会漂移的副本。拆法遵循 AGENTS.md 尺寸约定: +// 纯搬移独立 PR、原文件调用方零改动(backfill 经 index.js barrel 再出口,测试照旧)。 +// +// 时序契约(搬移前即如此,由 index.js 全量测试与 reindex-backfill.test.js 锁): +// lightMode / openai / 同步构造失败 → applyHumanEdits 立即; +// local|ollama → init 成功后 applyHumanEdits;#118 重试(1 + 4×15s)耗尽 → +// setEmbedder(null) 检索降级关键词,随后仍 applyHumanEdits; +// reranker 异步 init 失败只降级 rerank 自身,绝不影响 search。 +import { createEmbedder } from "./embedding.js"; +import { createEmbedderByProvider } from "./local-embedder.js"; +import { LocalReranker } from "./reranker.js"; + +/** + * Issue #128: bounded backfill of rows still missing an embedding (active rows + * only — needsEmbedding filters archived/forgotten). Exported for tests. + * + * Runs regardless of the model fingerprint: the old call-site gate returned + * early when vector_meta already held the embedder's hash, permanently + * orphaning rows whose embed failed at write time (embedder not ready / + * provider rate limit) — one successful embed was enough to never backfill + * again. markModel is idempotent when the fingerprint already matches, so + * re-running costs nothing beyond the actually-missing rows. + */ +export async function backfillMissingEmbeddings({ + store, embedder, vectorIndex, logger, + maxTotal = 500, batchSize = 10, rateLimitMs = 200 +}) { + let indexed = 0; + for (let done = 0; done < maxTotal;) { + const rows = store.needsEmbedding(batchSize); + if (!rows.length) break; + for (const row of rows) { + try { + const text = [row.title, row.content].filter(Boolean).join("\n"); + const vector = await embedder.embedSingle(text); + if (vector?.length) { + store.setEmbedding(row.id, vector); + indexed++; + } + } catch { /* skip the bad row */ } + } + done += rows.length; + // Rate limit: space out batches so the provider is not hammered. + if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); + } + if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); + logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); + return indexed; +} + +/** + * 组装语义管线(embedder + reranker)并挂到 service 上,随后调度 boot 自动回填。 + * @param {object} opts + * store/service/settings/vectorIndex — 宿主与 daemon 同形传入; + * cfg — 宿主传合并后的完整配置;daemon 传语义子集(默认值锚定 config.js); + * logger — console 形状(原代码读 ctx.logger,搬移后注入); + * applyHumanEdits— 人改镜像合并回调(index.js 闭包,读 mirror.readHumanEdits); + * 在哪个分支何时被调是时序契约的一部分,见文件头; + * lightMode — 轻量档:整条向量管线关闭,无 embedder/reranker、不回填。 + * @returns {{embedder, reranker, dispose}} embedder 构造失败(同步抛)时为 null; + * init 异步失败经 #118 重试后 service 侧降级,此处引用仍在(dream/sleep 语义面 + * 与搬移前一致)。dispose 清两个引导期计时器。 + */ +export function createSemantic({ store, service, settings, cfg, logger, vectorIndex, applyHumanEdits, lightMode = false }) { + let embedder = null; + let reranker = null; + // #118: pending embedder-init retry timer, cleared on unload. + let embedRetryTimer = null; + // boot 回填首查计时器。搬移前在 index.js 是裸 setTimeout(不参与卸载清理); + // 纳入 dispose 是修悬挂,不改变启动行为。 + let reindexTimer = null; + + if (lightMode) { + // Light mode: the whole vector pipeline stays off — no embedder (nothing + // pulls in ONNX/transformers), no reranker, no boot backfill (the preset + // also cleared autoReindexOnBoot). Recall degrades to keyword search and + // human mirror edits still merge on boot. + applyHumanEdits(); + } else if (cfg.embedProvider === "openai") { + // vectorIndex is passed so the legacy OpenAI embedder records the producing + // model fingerprint after each successful embed (Bug3). + embedder = createEmbedder({ store, settings, logger, vectorIndex }); + service.setEmbedder(embedder); + // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 + // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 + // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 + if (embedder.configured === false) { + logger?.warn?.( + "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" + + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" + + "dream 的语义聚类会退化为全量窗口兜底。" + + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" + ); + } + // legacy OpenAI embedder needs no async init → human edits apply right away + applyHumanEdits(); + } else { + try { + embedder = createEmbedderByProvider(cfg.embedProvider, { + model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, + dimension: cfg.localEmbedDimension, + device: cfg.localEmbedDevice, + batchSize: cfg.localEmbedBatchSize, + // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， + // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 + pooling: cfg.localEmbedPooling, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + baseUrl: cfg.ollamaBaseUrl, + logger + }); + service.setEmbedder(embedder); + // issue #6: wait for extractor init before applying human edits, so + // scheduled embeddings see a ready embedder. + const bootEmbedder = () => embedder.init() + .then(() => { applyHumanEdits(); return true; }) + .catch(() => false); + // #118: the old one-shot probe permanently degraded search to keyword + // when Ollama was briefly unreachable at boot (recoverable only by + // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); + // search degrades to keyword meanwhile because per-query embed failures + // are swallowed. + bootEmbedder().then((ok) => { + if (ok) return; + let tries = 4; + const retry = () => { + if (tries-- <= 0) { + logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); + service.setEmbedder(null); + applyHumanEdits(); + return; + } + embedRetryTimer = setTimeout(async () => { + if (await bootEmbedder()) return; + retry(); + }, 15_000); + }; + logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); + retry(); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); + applyHumanEdits(); + } + } + + // Cross-encoder rerank over recall candidates. Best-effort: a failed model + // load only disables reranking, never search itself. Explicit opt-in only + // (rerankEnabled defaults to false): constructing LocalReranker is what pulls + // in onnxruntime, so the default config never loads it (item ⑥). + if (cfg.rerankEnabled && cfg.rerankProvider === "local") { + try { + reranker = new LocalReranker({ + model: cfg.rerankModel, + batchSize: cfg.rerankBatchSize, + maxCandidates: cfg.rerankMaxCandidates, + scoreThreshold: cfg.rerankScoreThreshold, + device: cfg.localEmbedDevice, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； + // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 + useDtype: cfg.rerankDtype, + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + logger + }); + service.setReranker(reranker); + reranker.init().catch((error) => { + logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); + service.setReranker(null); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); + } + } + + // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API + // is configured and rows still lack an embedding (e.g. written before vector + // search was enabled), the backfill runs in the background after a short + // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a + // large backlog never floods the provider. Failures degrade silently — + // search stays keyword. + function scheduleAutoReindex() { + if (cfg.autoReindexOnBoot === false) return; + const attempt = (tries) => { + try { + if (!embedder || typeof embedder.embedSingle !== "function") return; + if ("ready" in embedder && embedder.ready !== true) { + // Local/ollama embedders init asynchronously; give them a moment + // before giving up on this boot (next boot retries). + // CodeRabbit on #365:嵌套重试计时器同样入册,否则 dispose 后仍可能 + // 对已关库跑 needsEmbedding(有 try/catch 兜底只是日志噪声,但状态要收干净)。 + if (tries > 0) reindexTimer = setTimeout(() => attempt(tries - 1), 2000); + return; + } + if (!store.needsEmbedding(1).length) return; // nothing to backfill + // Issue #128: no fingerprint gate here anymore — a matching fingerprint + // used to return early and permanently orphan rows whose embed failed + // at write time. See backfillMissingEmbeddings(). + backfillMissingEmbeddings({ store, embedder, vectorIndex, logger }) + .catch((error) => { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + } + }; + reindexTimer = setTimeout(() => attempt(5), 5000); + } + scheduleAutoReindex(); + + return { + embedder, + reranker, + dispose() { + if (embedRetryTimer !== null) { clearTimeout(embedRetryTimer); embedRetryTimer = null; } + if (reindexTimer !== null) { clearTimeout(reindexTimer); reindexTimer = null; } + } + }; +} diff --git a/dsh-mneme/lib/serve.js b/dsh-mneme/lib/serve.js index 82ffba9..ede6284 100644 --- a/dsh-mneme/lib/serve.js +++ b/dsh-mneme/lib/serve.js @@ -2,15 +2,15 @@ // (#363 承诺的「官方推荐第三方挂载姿势」)。bin/dsh-mneme-serve.mjs 是它的 CLI 壳, // Mneme Bridge 这类第三方也可直接 import 本模块自行托管生命周期。 // -// 与宿主装配(src/index.js apply)的关系:只搬数据面那一半,每步注释锚定 index.js -// 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与宿主 ctx -// 纠缠,防御段纪律是「最后动或不动」;这 60 行的漂移风险由 serve-bin 测试的多进程 -// 共存用例兜底(两侧真开同一个库互写互读)。 +// 与宿主装配(src/index.js apply)的关系:只搬数据面 + 语义管线两块,每步注释锚定 +// index.js 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与 +// 宿主 ctx 纠缠,防御段纪律是「最后动或不动」;本文件的漂移风险由 serve-bin 测试的 +// 多进程共存用例兜底(两侧真开同一个库互写互读)。 // -// 第一期无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只 -// 属于 DSH 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 -// externalApi/autoDream 单侧纪律),不依赖用户自觉。检索是关键词 + BM25(service -// 内建);向量管线由 PR2 的 semantic 抽取接入。 +// 无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只属于 DSH +// 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 externalApi/ +// autoDream 单侧纪律),不依赖用户自觉。语义检索(PR2 起)与宿主共用 src/semantic.js +// 的同一份装配;daemon 不参与做梦,但向量检索/语义去重与宿主同质。 import { mkdirSync } from "node:fs"; import { homedir } from "node:os"; import { join } from "node:path"; @@ -21,19 +21,34 @@ import { langOf } from "./lang.js"; import { createService } from "./service.js"; import { createMaintenance } from "./maintenance.js"; import { createStandaloneApi } from "./api-standalone.js"; +import { createSemantic } from "./semantic.js"; +import { createVectorIndex } from "./vector-index.js"; +import { provisionRuntime } from "./runtime/provision.js"; +import { defaultRuntimeDir, listPayloadDirs } from "./runtime/layout.js"; + +const EMBED_PROVIDERS = new Set(["off", "local", "ollama", "openai"]); /** - * 组装并启动一个独立数据面。 + * 组装并启动一个独立数据面(含语义管线)。 * @param {object} opts * memoryDir — 数据目录;缺省与宿主同默认 ~/.dsh/memory(config.js:9),支持前导 ~。 * port/host — 透传 createStandaloneApi;缺省走 kv external_api 持久值 > 8790/127.0.0.1。 * logger — console 形状(info/warn/error,收单字符串);缺省 null(全链路容缺)。 * strictPort — 默认 true:配置端口被占即失败(第三方把 URL 写死,顺延=静默打到 * 错误端口)。显式 port=0(测试/OS 分配)不受影响。 - * @returns {api, store, service, settings, maintenance, tokenExisted, dispose} + * embed — "local"(默认,自管 runtime 取件 + ONNX 嵌入)|"ollama"|"openai" + * (读宿主面板配的 vector-config)|"off"(纯关键词 + BM25)。 + * embedder/reranker — 注入式覆盖(测试/宿主方自管):给了就跳过 createSemantic, + * 直接 setEmbedder/setReranker;仅 embed="off" 之外有意为之。 + * @returns {api, store, service, settings, maintenance, semantic, tokenExisted, dispose} * tokenExisted — 启动前 kv 里是否已有 token;false 时本次为首次生成,入口层可提示。 */ -export function createServeRuntime({ memoryDir, port, host, logger = null, strictPort = true } = {}) { +export async function createServeRuntime({ + memoryDir, port, host, logger = null, strictPort = true, embed = "local", embedder = null, reranker = null +} = {}) { + if (!EMBED_PROVIDERS.has(embed)) { + throw new Error(`invalid embed provider: ${String(embed)}(允许 off/local/ollama/openai)`); + } // index.js:192-195:~ 展开只认前导;目录不存在时 node:sqlite 直接抛,先 mkdir。 const dir = String(memoryDir || join(homedir(), ".dsh", "memory")).replace(/^~(?=$|[\\/])/, homedir()); mkdirSync(dir, { recursive: true }); @@ -55,10 +70,46 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric // index.js:319-332:人改镜像先合并——镜像文件里的手工编辑每次启动都赢。 // readHumanEdits 全类型一次读齐:mergeHumanEdits 成功会重渲全部镜像,逐类型读改 // 循环会拿没读到的类型覆盖掉未合并的编辑(index.js:324-327 注释同款坑)。 + // 何时 apply 由 semantic 的分支时序契约决定(见 semantic.js 文件头),本文件只造闭包。 const humanEdits = new Map(); for (const type of Object.keys(TYPE_FILE)) humanEdits.set(type, mirror.readHumanEdits(type)); - for (const [type, edits] of humanEdits) { - if (edits.length) service.mergeHumanEdits(type, edits); + const applyHumanEdits = () => { + for (const [type, edits] of humanEdits) { + if (edits.length) service.mergeHumanEdits(type, edits); + } + }; + + // index.js:316-317:向量索引包住 store 的 embedding 列并跟踪活跃模型指纹。 + const vectorIndex = createVectorIndex({ store, logger }); + service.setVectorIndex(vectorIndex); + + // ---- 语义管线(PR2):三路二选一 ------------------------------------------ + let semantic = null; + if (embedder !== null) { + // 注入式覆盖(测试 / 宿主方自管 embedder):绕过装配,人改合并立刻做 + //(对齐宿主 openai 分支的同步语义)。 + service.setEmbedder(embedder); + if (reranker !== null) service.setReranker(reranker); + applyHumanEdits(); + } else if (embed === "off") { + // 纯关键词 + BM25:对齐宿主 lightMode 分支——不建 embedder,人改合并立即。 + applyHumanEdits(); + } else { + let semanticReady = true; + if (embed === "local") { + // 自管 runtime 缺失时取件:只走 download 档(宿主 adopt 推导对独立进程无意义 + // ——没有宿主 node_modules 可推)。失败不阻断 daemon:降级关键词 + 可操作提示。 + semanticReady = await ensureRuntimePayload(logger); + } + if (semanticReady) { + semantic = createSemantic({ + store, service, settings, + cfg: daemonSemanticCfg(embed), + logger, vectorIndex, applyHumanEdits, lightMode: false + }); + } else { + applyHumanEdits(); + } } // index.js:294-309:检索回执落 recall_runs(searchMemories 的 recordRecall 默认开, @@ -92,6 +143,7 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric service, settings, maintenance, + semantic, tokenExisted, /** * 收尾：先停收新请求、等在途请求跑完，再关库——直接同步关库会让在途的 @@ -107,7 +159,54 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric api.server.closeIdleConnections?.(); } catch { resolve(); } }); + try { semantic?.dispose(); } catch { /* 同上 */ } try { store.close(); } catch { /* 同上 */ } } }; } + +/** + * daemon 侧语义配置:默认值逐键锚定 src/config.js(309-420),daemon 不装 schemastery + * (宿主 peer 依赖),同值硬编码;config.js 改默认值时此处与 docs/DAEMON.md 同步。 + * 只放开 embedProvider(CLI --embed);rerank 暂不提供 CLI(默认关,与宿主同默认)。 + */ +function daemonSemanticCfg(embed) { + return { + embedProvider: embed, // config.js:309(宿主默认 openai;daemon 默认 local——独立部署唯一自洽路径) + localEmbedModel: "Xenova/bge-small-zh-v1.5", // :312 + localEmbedDimension: 512, // :313 + localEmbedDevice: "cpu", // :314 + localEmbedBatchSize: 8, // :315 + localEmbedPooling: "auto", // :323(BGE 系 CLS 口径,改它=换向量空间) + ollamaBaseUrl: "http://localhost:11434", // :326 + ollamaModel: "nomic-embed-text", // :331 + embedModelCacheDir: "", // :336(空=local-embedder 内部解析 ~/.dsh/mneme/models) + embedModelMirror: "https://hf-mirror.com", // :337 + resilientModelDownload: true, // :342 + runtimeDir: process.env.DSH_MNEME_RUNTIME_DIR || "", // :348(空=defaultRuntimeDir;取件/加载同源) + rerankEnabled: false, // :419 + rerankProvider: "none", // :420 + autoReindexOnBoot: true // :367 + }; +} + +/** + * 本地推理 runtime 取件(download 档)。返回 payload 是否可用: + * 已存在(含收编/早前下载)→ true 不重复取;缺失 → 取件成功 true / 失败 false(内部 + * 已打降级日志)。并发安全由 provisionRuntime 的 IN_FLIGHT 锁保证(按归一化 runtimeDir 串行)。 + */ +async function ensureRuntimePayload(logger) { + const runtimeDir = process.env.DSH_MNEME_RUNTIME_DIR || ""; + const dir = runtimeDir || defaultRuntimeDir(); + if (listPayloadDirs(dir).length > 0) return true; + logger?.info?.("[dsh-mneme] 本地推理运行时缺失,开始取件(download 档,约 200MB;离线可用 DSH_MNEME_RUNTIME_TARBALL_DIR / DSH_MNEME_RUNTIME_MIRROR)…"); + try { + const result = await provisionRuntime({ runtimeDir, localTarballDir: process.env.DSH_MNEME_RUNTIME_TARBALL_DIR || "", mirror: process.env.DSH_MNEME_RUNTIME_MIRROR || "" }); + if (!result.ok) throw new Error(result.reason); + logger?.info?.(`[dsh-mneme] 运行时就绪(${result.strategy}):${result.packages} 包 / ${result.files} 文件`); + return true; + } catch (error) { + logger?.warn?.(`[dsh-mneme] runtime provisioning failed, search degrades to keyword: ${String(error)}`); + return false; + } +} diff --git a/dsh-mneme/src/index.js b/dsh-mneme/src/index.js index 21a1b64..8118cd9 100644 --- a/dsh-mneme/src/index.js +++ b/dsh-mneme/src/index.js @@ -19,10 +19,10 @@ import { createStandaloneApi } from "./api-standalone.js"; import { createMaintenance } from "./maintenance.js"; import { createSettings } from "./settings.js"; import { createCommandManager } from "./commands.js"; -import { createEmbedder } from "./embedding.js"; -import { createEmbedderByProvider } from "./local-embedder.js"; -import { LocalReranker } from "./reranker.js"; import { createVectorIndex } from "./vector-index.js"; +// semantic(embedder/reranker/boot 回填)纯搬移至 src/semantic.js(PR2):宿主与 +// daemon(dsh-mneme-serve,#363)共用同一套装配,原文件保留 barrel 出口。 +import { createSemantic } from "./semantic.js"; import { Config, applyLightModePreset, injectChildEnabled } from "./config.js"; import { langOf } from "./lang.js"; import { extractEntities } from "./entities/extractor.js"; @@ -149,43 +149,9 @@ export function createEntityStreamAdapter({ llm, agentDefaultModel, logger, serv }; } -/** - * Issue #128: bounded backfill of rows still missing an embedding (active rows - * only — needsEmbedding filters archived/forgotten). Exported for tests. - * - * Runs regardless of the model fingerprint: the old call-site gate returned - * early when vector_meta already held the embedder's hash, permanently - * orphaning rows whose embed failed at write time (embedder not ready / - * provider rate limit) — one successful embed was enough to never backfill - * again. markModel is idempotent when the fingerprint already matches, so - * re-running costs nothing beyond the actually-missing rows. - */ -export async function backfillMissingEmbeddings({ - store, embedder, vectorIndex, logger, - maxTotal = 500, batchSize = 10, rateLimitMs = 200 -}) { - let indexed = 0; - for (let done = 0; done < maxTotal;) { - const rows = store.needsEmbedding(batchSize); - if (!rows.length) break; - for (const row of rows) { - try { - const text = [row.title, row.content].filter(Boolean).join("\n"); - const vector = await embedder.embedSingle(text); - if (vector?.length) { - store.setEmbedding(row.id, vector); - indexed++; - } - } catch { /* skip the bad row */ } - } - done += rows.length; - // Rate limit: space out batches so the provider is not hammered. - if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); - } - if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); - logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); - return indexed; -} +// backfillMissingEmbeddings 已随语义装配整体搬至 src/semantic.js(纯搬移); +// 保留 barrel 再出口 —— test/reindex-backfill.test.js 仍从本模块 import,调用方零改动。 +export { backfillMissingEmbeddings } from "./semantic.js"; export const apply = (ctx, config) => { const rawCfg = Config(config); @@ -339,150 +305,14 @@ export const apply = (ctx, config) => { } }; - let embedder = null; - let reranker = null; - // #118: pending embedder-init retry timer, cleared on unload. - let embedRetryTimer = null; - if (lightMode) { - // Light mode: the whole vector pipeline stays off — no embedder (nothing - // pulls in ONNX/transformers), no reranker, no boot backfill (the preset - // also cleared autoReindexOnBoot). Recall degrades to keyword search and - // human mirror edits still merge on boot. - applyHumanEdits(); - } else if (cfg.embedProvider === "openai") { - // vectorIndex is passed so the legacy OpenAI embedder records the producing - // model fingerprint after each successful embed (Bug3). - embedder = createEmbedder({ store, settings, logger: ctx.logger, vectorIndex }); - service.setEmbedder(embedder); - // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 - // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 - // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 - if (embedder.configured === false) { - ctx.logger?.warn?.( - "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" - + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" - + "dream 的语义聚类会退化为全量窗口兜底。" - + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" - ); - } - // legacy OpenAI embedder needs no async init → human edits apply right away - applyHumanEdits(); - } else { - try { - embedder = createEmbedderByProvider(cfg.embedProvider, { - model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, - dimension: cfg.localEmbedDimension, - device: cfg.localEmbedDevice, - batchSize: cfg.localEmbedBatchSize, - // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， - // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 - pooling: cfg.localEmbedPooling, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - baseUrl: cfg.ollamaBaseUrl, - logger: ctx.logger - }); - service.setEmbedder(embedder); - // issue #6: wait for extractor init before applying human edits, so - // scheduled embeddings see a ready embedder. - const bootEmbedder = () => embedder.init() - .then(() => { applyHumanEdits(); return true; }) - .catch(() => false); - // #118: the old one-shot probe permanently degraded search to keyword - // when Ollama was briefly unreachable at boot (recoverable only by - // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); - // search degrades to keyword meanwhile because per-query embed failures - // are swallowed. - bootEmbedder().then((ok) => { - if (ok) return; - let tries = 4; - const retry = () => { - if (tries-- <= 0) { - ctx.logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); - service.setEmbedder(null); - applyHumanEdits(); - return; - } - embedRetryTimer = setTimeout(async () => { - if (await bootEmbedder()) return; - retry(); - }, 15_000); - }; - ctx.logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); - retry(); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); - applyHumanEdits(); - } - } - - // Cross-encoder rerank over recall candidates. Best-effort: a failed model - // load only disables reranking, never search itself. Explicit opt-in only - // (rerankEnabled defaults to false): constructing LocalReranker is what pulls - // in onnxruntime, so the default config never loads it (item ⑥). - if (cfg.rerankEnabled && cfg.rerankProvider === "local") { - try { - reranker = new LocalReranker({ - model: cfg.rerankModel, - batchSize: cfg.rerankBatchSize, - maxCandidates: cfg.rerankMaxCandidates, - scoreThreshold: cfg.rerankScoreThreshold, - device: cfg.localEmbedDevice, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； - // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 - useDtype: cfg.rerankDtype, - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - logger: ctx.logger - }); - service.setReranker(reranker); - reranker.init().catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); - service.setReranker(null); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); - } - } - - // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API - // is configured and rows still lack an embedding (e.g. written before vector - // search was enabled), the backfill runs in the background after a short - // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a - // large backlog never floods the provider. Failures degrade silently — - // search stays keyword. - function scheduleAutoReindex() { - if (cfg.autoReindexOnBoot === false) return; - const attempt = (tries) => { - try { - if (!embedder || typeof embedder.embedSingle !== "function") return; - if ("ready" in embedder && embedder.ready !== true) { - // Local/ollama embedders init asynchronously; give them a moment - // before giving up on this boot (next boot retries). - if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); - return; - } - if (!store.needsEmbedding(1).length) return; // nothing to backfill - // Issue #128: no fingerprint gate here anymore — a matching fingerprint - // used to return early and permanently orphan rows whose embed failed - // at write time. See backfillMissingEmbeddings(). - backfillMissingEmbeddings({ store, embedder, vectorIndex, logger: ctx.logger }) - .catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - } - }; - setTimeout(() => attempt(5), 5000); - } - scheduleAutoReindex(); + // embedder/reranker/boot 回填装配已整体搬至 src/semantic.js(纯搬移,宿主与 daemon + // 共用):调用时序(applyHumanEdits 在各分支的触发点、#118 重试、autoReindexOnBoot) + // 原样保留在 createSemantic 内部,这里只拿引用。init 全失败的 embedder 引用仍会进入 + // dream/sleep 的 semantic 面 —— 与搬移前一致:检索侧 setEmbedder(null) 降级关键词。 + const semantic = createSemantic({ + store, service, settings, cfg, logger: ctx.logger, vectorIndex, applyHumanEdits, lightMode + }); + const { embedder, reranker } = semantic; // Custom commands: register persisted commands into the DSH command registry // on boot; add/remove re-register live through the API. @@ -601,7 +431,7 @@ export const apply = (ctx, config) => { // #118: never let a pending embedder init retry fire after unload and touch // a torn-down context. - disposers.push(() => { if (embedRetryTimer !== null) clearTimeout(embedRetryTimer); }); + disposers.push(() => semantic.dispose()); // #118 重试计时器 + boot 回填计时器(搬入 semantic.js 后由它自持) ctx.inject(["systemPrompt"], (promptCtx) => { if (cfg.autoInject) disposers.push(createInjector(promptCtx, service, settings, cfg)); diff --git a/dsh-mneme/src/semantic.js b/dsh-mneme/src/semantic.js new file mode 100644 index 0000000..a236aa1 --- /dev/null +++ b/dsh-mneme/src/semantic.js @@ -0,0 +1,229 @@ +// src/semantic.js —— embedder/reranker 装配与 boot 自动回填。 +// 纯搬移自 src/index.js(2026-10,PR2):backfillMissingEmbeddings(原 :150-186)与 +// 装配段(原 :311-477),行为逐字节对齐,仅两处已注记的机械差异(ctx.logger → 注入 +// logger;boot 回填的首查计时器纳入 dispose)。搬移原因:daemon(dsh-mneme-serve, +// #363)与宿主要共用同一套语义装配——「向量检索开箱即用」的承诺落在两侧同一份 +// 代码上,而不是 daemon 复刻一份会漂移的副本。拆法遵循 AGENTS.md 尺寸约定: +// 纯搬移独立 PR、原文件调用方零改动(backfill 经 index.js barrel 再出口,测试照旧)。 +// +// 时序契约(搬移前即如此,由 index.js 全量测试与 reindex-backfill.test.js 锁): +// lightMode / openai / 同步构造失败 → applyHumanEdits 立即; +// local|ollama → init 成功后 applyHumanEdits;#118 重试(1 + 4×15s)耗尽 → +// setEmbedder(null) 检索降级关键词,随后仍 applyHumanEdits; +// reranker 异步 init 失败只降级 rerank 自身,绝不影响 search。 +import { createEmbedder } from "./embedding.js"; +import { createEmbedderByProvider } from "./local-embedder.js"; +import { LocalReranker } from "./reranker.js"; + +/** + * Issue #128: bounded backfill of rows still missing an embedding (active rows + * only — needsEmbedding filters archived/forgotten). Exported for tests. + * + * Runs regardless of the model fingerprint: the old call-site gate returned + * early when vector_meta already held the embedder's hash, permanently + * orphaning rows whose embed failed at write time (embedder not ready / + * provider rate limit) — one successful embed was enough to never backfill + * again. markModel is idempotent when the fingerprint already matches, so + * re-running costs nothing beyond the actually-missing rows. + */ +export async function backfillMissingEmbeddings({ + store, embedder, vectorIndex, logger, + maxTotal = 500, batchSize = 10, rateLimitMs = 200 +}) { + let indexed = 0; + for (let done = 0; done < maxTotal;) { + const rows = store.needsEmbedding(batchSize); + if (!rows.length) break; + for (const row of rows) { + try { + const text = [row.title, row.content].filter(Boolean).join("\n"); + const vector = await embedder.embedSingle(text); + if (vector?.length) { + store.setEmbedding(row.id, vector); + indexed++; + } + } catch { /* skip the bad row */ } + } + done += rows.length; + // Rate limit: space out batches so the provider is not hammered. + if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); + } + if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); + logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); + return indexed; +} + +/** + * 组装语义管线(embedder + reranker)并挂到 service 上,随后调度 boot 自动回填。 + * @param {object} opts + * store/service/settings/vectorIndex — 宿主与 daemon 同形传入; + * cfg — 宿主传合并后的完整配置;daemon 传语义子集(默认值锚定 config.js); + * logger — console 形状(原代码读 ctx.logger,搬移后注入); + * applyHumanEdits— 人改镜像合并回调(index.js 闭包,读 mirror.readHumanEdits); + * 在哪个分支何时被调是时序契约的一部分,见文件头; + * lightMode — 轻量档:整条向量管线关闭,无 embedder/reranker、不回填。 + * @returns {{embedder, reranker, dispose}} embedder 构造失败(同步抛)时为 null; + * init 异步失败经 #118 重试后 service 侧降级,此处引用仍在(dream/sleep 语义面 + * 与搬移前一致)。dispose 清两个引导期计时器。 + */ +export function createSemantic({ store, service, settings, cfg, logger, vectorIndex, applyHumanEdits, lightMode = false }) { + let embedder = null; + let reranker = null; + // #118: pending embedder-init retry timer, cleared on unload. + let embedRetryTimer = null; + // boot 回填首查计时器。搬移前在 index.js 是裸 setTimeout(不参与卸载清理); + // 纳入 dispose 是修悬挂,不改变启动行为。 + let reindexTimer = null; + + if (lightMode) { + // Light mode: the whole vector pipeline stays off — no embedder (nothing + // pulls in ONNX/transformers), no reranker, no boot backfill (the preset + // also cleared autoReindexOnBoot). Recall degrades to keyword search and + // human mirror edits still merge on boot. + applyHumanEdits(); + } else if (cfg.embedProvider === "openai") { + // vectorIndex is passed so the legacy OpenAI embedder records the producing + // model fingerprint after each successful embed (Bug3). + embedder = createEmbedder({ store, settings, logger, vectorIndex }); + service.setEmbedder(embedder); + // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 + // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 + // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 + if (embedder.configured === false) { + logger?.warn?.( + "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" + + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" + + "dream 的语义聚类会退化为全量窗口兜底。" + + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" + ); + } + // legacy OpenAI embedder needs no async init → human edits apply right away + applyHumanEdits(); + } else { + try { + embedder = createEmbedderByProvider(cfg.embedProvider, { + model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, + dimension: cfg.localEmbedDimension, + device: cfg.localEmbedDevice, + batchSize: cfg.localEmbedBatchSize, + // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， + // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 + pooling: cfg.localEmbedPooling, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + baseUrl: cfg.ollamaBaseUrl, + logger + }); + service.setEmbedder(embedder); + // issue #6: wait for extractor init before applying human edits, so + // scheduled embeddings see a ready embedder. + const bootEmbedder = () => embedder.init() + .then(() => { applyHumanEdits(); return true; }) + .catch(() => false); + // #118: the old one-shot probe permanently degraded search to keyword + // when Ollama was briefly unreachable at boot (recoverable only by + // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); + // search degrades to keyword meanwhile because per-query embed failures + // are swallowed. + bootEmbedder().then((ok) => { + if (ok) return; + let tries = 4; + const retry = () => { + if (tries-- <= 0) { + logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); + service.setEmbedder(null); + applyHumanEdits(); + return; + } + embedRetryTimer = setTimeout(async () => { + if (await bootEmbedder()) return; + retry(); + }, 15_000); + }; + logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); + retry(); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); + applyHumanEdits(); + } + } + + // Cross-encoder rerank over recall candidates. Best-effort: a failed model + // load only disables reranking, never search itself. Explicit opt-in only + // (rerankEnabled defaults to false): constructing LocalReranker is what pulls + // in onnxruntime, so the default config never loads it (item ⑥). + if (cfg.rerankEnabled && cfg.rerankProvider === "local") { + try { + reranker = new LocalReranker({ + model: cfg.rerankModel, + batchSize: cfg.rerankBatchSize, + maxCandidates: cfg.rerankMaxCandidates, + scoreThreshold: cfg.rerankScoreThreshold, + device: cfg.localEmbedDevice, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； + // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 + useDtype: cfg.rerankDtype, + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + logger + }); + service.setReranker(reranker); + reranker.init().catch((error) => { + logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); + service.setReranker(null); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); + } + } + + // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API + // is configured and rows still lack an embedding (e.g. written before vector + // search was enabled), the backfill runs in the background after a short + // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a + // large backlog never floods the provider. Failures degrade silently — + // search stays keyword. + function scheduleAutoReindex() { + if (cfg.autoReindexOnBoot === false) return; + const attempt = (tries) => { + try { + if (!embedder || typeof embedder.embedSingle !== "function") return; + if ("ready" in embedder && embedder.ready !== true) { + // Local/ollama embedders init asynchronously; give them a moment + // before giving up on this boot (next boot retries). + // CodeRabbit on #365:嵌套重试计时器同样入册,否则 dispose 后仍可能 + // 对已关库跑 needsEmbedding(有 try/catch 兜底只是日志噪声,但状态要收干净)。 + if (tries > 0) reindexTimer = setTimeout(() => attempt(tries - 1), 2000); + return; + } + if (!store.needsEmbedding(1).length) return; // nothing to backfill + // Issue #128: no fingerprint gate here anymore — a matching fingerprint + // used to return early and permanently orphan rows whose embed failed + // at write time. See backfillMissingEmbeddings(). + backfillMissingEmbeddings({ store, embedder, vectorIndex, logger }) + .catch((error) => { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + } + }; + reindexTimer = setTimeout(() => attempt(5), 5000); + } + scheduleAutoReindex(); + + return { + embedder, + reranker, + dispose() { + if (embedRetryTimer !== null) { clearTimeout(embedRetryTimer); embedRetryTimer = null; } + if (reindexTimer !== null) { clearTimeout(reindexTimer); reindexTimer = null; } + } + }; +} diff --git a/dsh-mneme/src/serve.js b/dsh-mneme/src/serve.js index 82ffba9..ede6284 100644 --- a/dsh-mneme/src/serve.js +++ b/dsh-mneme/src/serve.js @@ -2,15 +2,15 @@ // (#363 承诺的「官方推荐第三方挂载姿势」)。bin/dsh-mneme-serve.mjs 是它的 CLI 壳, // Mneme Bridge 这类第三方也可直接 import 本模块自行托管生命周期。 // -// 与宿主装配(src/index.js apply)的关系:只搬数据面那一半,每步注释锚定 index.js -// 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与宿主 ctx -// 纠缠,防御段纪律是「最后动或不动」;这 60 行的漂移风险由 serve-bin 测试的多进程 -// 共存用例兜底(两侧真开同一个库互写互读)。 +// 与宿主装配(src/index.js apply)的关系:只搬数据面 + 语义管线两块,每步注释锚定 +// index.js 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与 +// 宿主 ctx 纠缠,防御段纪律是「最后动或不动」;本文件的漂移风险由 serve-bin 测试的 +// 多进程共存用例兜底(两侧真开同一个库互写互读)。 // -// 第一期无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只 -// 属于 DSH 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 -// externalApi/autoDream 单侧纪律),不依赖用户自觉。检索是关键词 + BM25(service -// 内建);向量管线由 PR2 的 semantic 抽取接入。 +// 无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只属于 DSH +// 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 externalApi/ +// autoDream 单侧纪律),不依赖用户自觉。语义检索(PR2 起)与宿主共用 src/semantic.js +// 的同一份装配;daemon 不参与做梦,但向量检索/语义去重与宿主同质。 import { mkdirSync } from "node:fs"; import { homedir } from "node:os"; import { join } from "node:path"; @@ -21,19 +21,34 @@ import { langOf } from "./lang.js"; import { createService } from "./service.js"; import { createMaintenance } from "./maintenance.js"; import { createStandaloneApi } from "./api-standalone.js"; +import { createSemantic } from "./semantic.js"; +import { createVectorIndex } from "./vector-index.js"; +import { provisionRuntime } from "./runtime/provision.js"; +import { defaultRuntimeDir, listPayloadDirs } from "./runtime/layout.js"; + +const EMBED_PROVIDERS = new Set(["off", "local", "ollama", "openai"]); /** - * 组装并启动一个独立数据面。 + * 组装并启动一个独立数据面(含语义管线)。 * @param {object} opts * memoryDir — 数据目录;缺省与宿主同默认 ~/.dsh/memory(config.js:9),支持前导 ~。 * port/host — 透传 createStandaloneApi;缺省走 kv external_api 持久值 > 8790/127.0.0.1。 * logger — console 形状(info/warn/error,收单字符串);缺省 null(全链路容缺)。 * strictPort — 默认 true:配置端口被占即失败(第三方把 URL 写死,顺延=静默打到 * 错误端口)。显式 port=0(测试/OS 分配)不受影响。 - * @returns {api, store, service, settings, maintenance, tokenExisted, dispose} + * embed — "local"(默认,自管 runtime 取件 + ONNX 嵌入)|"ollama"|"openai" + * (读宿主面板配的 vector-config)|"off"(纯关键词 + BM25)。 + * embedder/reranker — 注入式覆盖(测试/宿主方自管):给了就跳过 createSemantic, + * 直接 setEmbedder/setReranker;仅 embed="off" 之外有意为之。 + * @returns {api, store, service, settings, maintenance, semantic, tokenExisted, dispose} * tokenExisted — 启动前 kv 里是否已有 token;false 时本次为首次生成,入口层可提示。 */ -export function createServeRuntime({ memoryDir, port, host, logger = null, strictPort = true } = {}) { +export async function createServeRuntime({ + memoryDir, port, host, logger = null, strictPort = true, embed = "local", embedder = null, reranker = null +} = {}) { + if (!EMBED_PROVIDERS.has(embed)) { + throw new Error(`invalid embed provider: ${String(embed)}(允许 off/local/ollama/openai)`); + } // index.js:192-195:~ 展开只认前导;目录不存在时 node:sqlite 直接抛,先 mkdir。 const dir = String(memoryDir || join(homedir(), ".dsh", "memory")).replace(/^~(?=$|[\\/])/, homedir()); mkdirSync(dir, { recursive: true }); @@ -55,10 +70,46 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric // index.js:319-332:人改镜像先合并——镜像文件里的手工编辑每次启动都赢。 // readHumanEdits 全类型一次读齐:mergeHumanEdits 成功会重渲全部镜像,逐类型读改 // 循环会拿没读到的类型覆盖掉未合并的编辑(index.js:324-327 注释同款坑)。 + // 何时 apply 由 semantic 的分支时序契约决定(见 semantic.js 文件头),本文件只造闭包。 const humanEdits = new Map(); for (const type of Object.keys(TYPE_FILE)) humanEdits.set(type, mirror.readHumanEdits(type)); - for (const [type, edits] of humanEdits) { - if (edits.length) service.mergeHumanEdits(type, edits); + const applyHumanEdits = () => { + for (const [type, edits] of humanEdits) { + if (edits.length) service.mergeHumanEdits(type, edits); + } + }; + + // index.js:316-317:向量索引包住 store 的 embedding 列并跟踪活跃模型指纹。 + const vectorIndex = createVectorIndex({ store, logger }); + service.setVectorIndex(vectorIndex); + + // ---- 语义管线(PR2):三路二选一 ------------------------------------------ + let semantic = null; + if (embedder !== null) { + // 注入式覆盖(测试 / 宿主方自管 embedder):绕过装配,人改合并立刻做 + //(对齐宿主 openai 分支的同步语义)。 + service.setEmbedder(embedder); + if (reranker !== null) service.setReranker(reranker); + applyHumanEdits(); + } else if (embed === "off") { + // 纯关键词 + BM25:对齐宿主 lightMode 分支——不建 embedder,人改合并立即。 + applyHumanEdits(); + } else { + let semanticReady = true; + if (embed === "local") { + // 自管 runtime 缺失时取件:只走 download 档(宿主 adopt 推导对独立进程无意义 + // ——没有宿主 node_modules 可推)。失败不阻断 daemon:降级关键词 + 可操作提示。 + semanticReady = await ensureRuntimePayload(logger); + } + if (semanticReady) { + semantic = createSemantic({ + store, service, settings, + cfg: daemonSemanticCfg(embed), + logger, vectorIndex, applyHumanEdits, lightMode: false + }); + } else { + applyHumanEdits(); + } } // index.js:294-309:检索回执落 recall_runs(searchMemories 的 recordRecall 默认开, @@ -92,6 +143,7 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric service, settings, maintenance, + semantic, tokenExisted, /** * 收尾：先停收新请求、等在途请求跑完，再关库——直接同步关库会让在途的 @@ -107,7 +159,54 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric api.server.closeIdleConnections?.(); } catch { resolve(); } }); + try { semantic?.dispose(); } catch { /* 同上 */ } try { store.close(); } catch { /* 同上 */ } } }; } + +/** + * daemon 侧语义配置:默认值逐键锚定 src/config.js(309-420),daemon 不装 schemastery + * (宿主 peer 依赖),同值硬编码;config.js 改默认值时此处与 docs/DAEMON.md 同步。 + * 只放开 embedProvider(CLI --embed);rerank 暂不提供 CLI(默认关,与宿主同默认)。 + */ +function daemonSemanticCfg(embed) { + return { + embedProvider: embed, // config.js:309(宿主默认 openai;daemon 默认 local——独立部署唯一自洽路径) + localEmbedModel: "Xenova/bge-small-zh-v1.5", // :312 + localEmbedDimension: 512, // :313 + localEmbedDevice: "cpu", // :314 + localEmbedBatchSize: 8, // :315 + localEmbedPooling: "auto", // :323(BGE 系 CLS 口径,改它=换向量空间) + ollamaBaseUrl: "http://localhost:11434", // :326 + ollamaModel: "nomic-embed-text", // :331 + embedModelCacheDir: "", // :336(空=local-embedder 内部解析 ~/.dsh/mneme/models) + embedModelMirror: "https://hf-mirror.com", // :337 + resilientModelDownload: true, // :342 + runtimeDir: process.env.DSH_MNEME_RUNTIME_DIR || "", // :348(空=defaultRuntimeDir;取件/加载同源) + rerankEnabled: false, // :419 + rerankProvider: "none", // :420 + autoReindexOnBoot: true // :367 + }; +} + +/** + * 本地推理 runtime 取件(download 档)。返回 payload 是否可用: + * 已存在(含收编/早前下载)→ true 不重复取;缺失 → 取件成功 true / 失败 false(内部 + * 已打降级日志)。并发安全由 provisionRuntime 的 IN_FLIGHT 锁保证(按归一化 runtimeDir 串行)。 + */ +async function ensureRuntimePayload(logger) { + const runtimeDir = process.env.DSH_MNEME_RUNTIME_DIR || ""; + const dir = runtimeDir || defaultRuntimeDir(); + if (listPayloadDirs(dir).length > 0) return true; + logger?.info?.("[dsh-mneme] 本地推理运行时缺失,开始取件(download 档,约 200MB;离线可用 DSH_MNEME_RUNTIME_TARBALL_DIR / DSH_MNEME_RUNTIME_MIRROR)…"); + try { + const result = await provisionRuntime({ runtimeDir, localTarballDir: process.env.DSH_MNEME_RUNTIME_TARBALL_DIR || "", mirror: process.env.DSH_MNEME_RUNTIME_MIRROR || "" }); + if (!result.ok) throw new Error(result.reason); + logger?.info?.(`[dsh-mneme] 运行时就绪(${result.strategy}):${result.packages} 包 / ${result.files} 文件`); + return true; + } catch (error) { + logger?.warn?.(`[dsh-mneme] runtime provisioning failed, search degrades to keyword: ${String(error)}`); + return false; + } +} diff --git a/dsh-mneme/test/serve-bin.test.js b/dsh-mneme/test/serve-bin.test.js index 01c77d0..0143297 100644 --- a/dsh-mneme/test/serve-bin.test.js +++ b/dsh-mneme/test/serve-bin.test.js @@ -61,7 +61,8 @@ test("serve bin: spawn 冒烟;daemon 与宿主进程同库互写互读", { timeo const peer = createService({ store: seedStore, mirror: null, config: {} }); peer.saveWithDedupe({ type: "project", title: "peer 进程直写", content: "测试进程经 createStore 写入", importance: 3 }); - const child = spawn(process.execPath, [BIN, "--memory-dir", dir, "--port", "0"], { + // --embed off:CI 无 runtime payload,绝不能触发取件;多进程共存与语义无关 + const child = spawn(process.execPath, [BIN, "--memory-dir", dir, "--port", "0", "--embed", "off"], { stdio: ["ignore", "pipe", "pipe"] }); let stdout = ""; diff --git a/dsh-mneme/test/serve.test.js b/dsh-mneme/test/serve.test.js index 5d29b98..edfb47f 100644 --- a/dsh-mneme/test/serve.test.js +++ b/dsh-mneme/test/serve.test.js @@ -8,14 +8,40 @@ import { createServeRuntime } from "../src/serve.js"; // daemon 装配面锁(test/serve-bin.test.js 另有真子进程 + 多进程共存): // in-process 起真 HTTP(port 0,OS 分配),fetch 走 health / 401 / save / search / // token 持久化复用全链路。锁的是「daemon 数据面 = api-standalone 同一工厂」这一契约。 +// 基础用例一律 embed:"off"——不碰取件与模型 init,CI 确定性;向量走注入假 embedder。 function tmpDir() { return mkdtempSync(join(tmpdir(), "mneme-serve-")); } +/** 确定性 4 维伪嵌入:同词相关、异词近正交,足以让向量路径产生非零命中。 */ +function fakeEmbedder() { + return { + ready: true, + modelHash: "fake-hash-1", + dimension: 4, + embedSingle: async (text) => { + const v = [0, 0, 0, 0]; + for (let i = 0; i < text.length; i++) v[i % 4] += text.charCodeAt(i) % 7; + const n = Math.hypot(v[0], v[1], v[2], v[3]) || 1; + return v.map((x) => x / n); + } + }; +} + +async function waitFor(fn, timeoutMs, what) { + const deadline = Date.now() + timeoutMs; + let lastErr; + while (Date.now() < deadline) { + try { return await fn(); } catch (err) { lastErr = err; } + await new Promise((r) => setTimeout(r, 100)); + } + throw new Error(`timeout waiting for ${what}: ${lastErr?.message ?? lastErr}`); +} + test("serve: /health 免鉴权,业务路由无 token 401", async () => { const dir = tmpDir(); - const rt = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt.api.ready; try { const base = `http://127.0.0.1:${rt.api.port}`; @@ -34,7 +60,7 @@ test("serve: /health 免鉴权,业务路由无 token 401", async () => { test("serve: save → search 走通;token 持久化 kv;二次启动复用同一 token;检索回执落 recall_runs", async () => { const dir = tmpDir(); - const rt = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt.api.ready; const base = `http://127.0.0.1:${rt.api.port}`; const auth = { authorization: `Bearer ${rt.api.token}` }; @@ -65,7 +91,7 @@ test("serve: save → search 走通;token 持久化 kv;二次启动复用同一 rt.dispose(); // 二次启动同目录:token 复用不重新生成(firstBoot 提示只在真正首次出现) - const rt2 = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt2 = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt2.api.ready; try { assert.equal(rt2.tokenExisted, true); @@ -78,7 +104,7 @@ test("serve: save → search 走通;token 持久化 kv;二次启动复用同一 test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)", async () => { const dir = tmpDir(); - const rt = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt.api.ready; const port = rt.api.port; // dispose 现为 async:等 server.close 回调(在途请求排干)后再重绑 @@ -87,7 +113,7 @@ test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)" let rebound = null; for (let i = 0; i < 10 && !rebound; i++) { try { - const rt2 = createServeRuntime({ memoryDir: dir, port }); + const rt2 = await createServeRuntime({ memoryDir: dir, port, embed: "off" }); await rt2.api.ready; rebound = rt2; } catch (err) { @@ -103,3 +129,57 @@ test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)" try { rmSync(dir, { recursive: true, force: true }); } catch { /* 同上 */ } } }); + +test("serve: 注入假 embedder → 写入即嵌入,vector 轴接管检索;auto 检索可用", async () => { + const dir = tmpDir(); + const fake = fakeEmbedder(); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off", embedder: fake }); + await rt.api.ready; + try { + // 覆盖注入生效:semantic 被绕过,embedder 直接挂 service + assert.equal(rt.semantic, null); + + const base = `http://127.0.0.1:${rt.api.port}`; + const auth = { authorization: `Bearer ${rt.api.token}` }; + const save = await fetch(`${base}/memories`, { + method: "POST", + headers: { ...auth, "content-type": "application/json" }, + body: JSON.stringify({ type: "project", title: "向量注入冒烟", content: "daemon 侧写入即嵌入", importance: 3 }) + }); + assert.equal(save.status, 201); + + // 写入侧嵌入是异步排队的:轮询到 vector 轴命中为止。auto 模式不断言 mode 字段 —— + // 标题同词命中时融合合并对象是 keyword 行,上游 mneme 的 auto 报告语义如此 + // (vector:true 标记只在纯向量来源上保留),那是上游行为,不是 daemon 契约。 + await waitFor(async () => { + const res = await fetch(`${base}/search?q=${encodeURIComponent("向量注入冒烟")}&mode=vector`, { headers: auth }); + assert.equal(res.status, 200); + const body = await res.json(); + assert.equal(body.mode, "vector", "mode=vector must report the vector axis"); + assert.ok((body.items ?? []).some((m) => m.title === "向量注入冒烟")); + return true; + }, 8000, "vector-axis search"); + + const autoRes = await fetch(`${base}/search?q=${encodeURIComponent("向量注入冒烟")}&mode=auto`, { headers: auth }); + assert.equal(autoRes.status, 200); + assert.ok(((await autoRes.json()).items ?? []).some((m) => m.title === "向量注入冒烟")); + } finally { + rt.dispose(); + try { rmSync(dir, { recursive: true, force: true }); } catch { /* 同上 */ } + } +}); + +test("serve: 非法 embed provider 拒装配;embed=off 时 semantic 为 null", async () => { + const dir = tmpDir(); + await assert.rejects( + createServeRuntime({ memoryDir: dir, port: 0, embed: "bogus" }), + /invalid embed provider/ + ); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); + try { + assert.equal(rt.semantic, null, "off path must not assemble a semantic pipeline"); + } finally { + rt.dispose(); + try { rmSync(dir, { recursive: true, force: true }); } catch { /* 同上 */ } + } +});