From c345970f618db41105a8cf5c8fc06ca02552a8f3 Mon Sep 17 00:00:00 2001 From: modusensus Date: Mon, 5 Oct 2026 14:26:18 +0800 Subject: [PATCH 1/3] =?UTF-8?q?feat(serve):=20daemon=20=E6=8E=A5=E5=85=A5?= =?UTF-8?q?=E5=90=91=E9=87=8F=E6=A3=80=E7=B4=A2=E2=80=94=E2=80=94semantic?= =?UTF-8?q?=20=E8=A3=85=E9=85=8D=E8=87=AA=20index.js=20=E7=BA=AF=E6=90=AC?= =?UTF-8?q?=E7=A7=BB(#363=20PR2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 10 +- dsh-mneme/CHANGELOG.md | 3 +- dsh-mneme/README.md | 9 +- dsh-mneme/bin/dsh-mneme-serve.mjs | 29 +++- dsh-mneme/docs/DAEMON.md | 12 +- dsh-mneme/lib/index.js | 200 ++------------------------ dsh-mneme/lib/semantic.js | 227 ++++++++++++++++++++++++++++++ dsh-mneme/lib/serve.js | 125 ++++++++++++++-- dsh-mneme/src/index.js | 200 ++------------------------ dsh-mneme/src/semantic.js | 227 ++++++++++++++++++++++++++++++ dsh-mneme/src/serve.js | 125 ++++++++++++++-- dsh-mneme/test/serve-bin.test.js | 3 +- dsh-mneme/test/serve.test.js | 90 +++++++++++- 13 files changed, 840 insertions(+), 420 deletions(-) create mode 100644 dsh-mneme/lib/semantic.js create mode 100644 dsh-mneme/src/semantic.js diff --git a/README.md b/README.md index 04297e4..70a626c 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ license CI node - tests + tests coverage Awesome

@@ -151,7 +151,7 @@ dsh web ## 不开 DSH 也能服务（独立服务 daemon） -`dsh-mneme-serve` 把记忆库跑成常驻数据面——DSH 关着，第三方集成（网页端桥接、脚本、自有面板）照样读写同一份记忆：token 与 DSH 面板/CLI 共用，路由与「外部访问 API」同源，端口被占直接报错（与 DSH 外部访问二选一）。第一期无 LLM（巩固/蒸馏仍属 DSH 宿主），检索为关键词 + BM25，向量接入在后续版本。 +`dsh-mneme-serve` 把记忆库跑成常驻数据面——DSH 关着，第三方集成（网页端桥接、脚本、自有面板）照样读写同一份记忆：token 与 DSH 面板/CLI 共用，路由与「外部访问 API」同源，端口被占直接报错（与 DSH 外部访问二选一）。第一期无 LLM（巩固/蒸馏仍属 DSH 宿主）；检索默认 local 嵌入（首次启动自动取件模型，`--embed off` 可关）。 ```bash npm i -g @modusensus/dsh-mneme @@ -193,7 +193,7 @@ dsh-mneme-serve # 默认 ~/.dsh/memory + 127.0.0.1:8790 ```bash cd dsh-mneme && npm install -npm test # 1502 个测试 +npm test # 1504 个测试 npm run stress # 三轴线压测 npm run sync # src → lib 同步 ``` @@ -344,7 +344,7 @@ The plugin ships a zero-dependency stdio MCP server (standalone npm package **`m ## Serve memories without DSH (standalone daemon) -`dsh-mneme-serve` runs the memory store as a long-lived data plane — with DSH closed, third-party integrations (web-bridge tools, scripts, your own panels) still read and write the same memories: the Bearer token is shared with the DSH panel/CLI, routes mirror the external API, and a busy port is a hard error (pick either the daemon or DSH's external API, not both). Phase 1 is LLM-free (consolidation/distillation stay with the DSH host); retrieval is keyword + BM25, with vector search arriving in a later release. +`dsh-mneme-serve` runs the memory store as a long-lived data plane — with DSH closed, third-party integrations (web-bridge tools, scripts, your own panels) still read and write the same memories: the Bearer token is shared with the DSH panel/CLI, routes mirror the external API, and a busy port is a hard error (pick either the daemon or DSH's external API, not both). Phase 1 is LLM-free (consolidation/distillation stay with the DSH host); retrieval defaults to local embeddings (runtime + model auto-provisioned on first boot, `--embed off` to disable). ```bash npm i -g @modusensus/dsh-mneme @@ -386,7 +386,7 @@ dsh-mneme-serve # defaults: ~/.dsh/memory + 127.0.0.1:8790 ```bash cd dsh-mneme && npm install -npm test # 1502 tests +npm test # 1504 tests npm run stress # three-axis stress test npm run sync # src → lib sync ``` diff --git a/dsh-mneme/CHANGELOG.md b/dsh-mneme/CHANGELOG.md index 45a2484..4ad7ac1 100644 --- a/dsh-mneme/CHANGELOG.md +++ b/dsh-mneme/CHANGELOG.md @@ -4,7 +4,8 @@ ## 🆕 新增 -- **独立服务 daemon（`dsh-mneme-serve`，#363）**：mneme 现在能在 DSH 宿主之外常驻——`src/serve.js` 的 `createServeRuntime` 用最小装配（store → settings → mirror → service → maintenance → standalone API，每步锚定 index.js 装配行号）把数据面跑成独立进程，第三方集成（网页端桥接等）不必为挂载记忆库而保持 DSH 开机。第一期刻意无 LLM：巩固（autoDream）与蒸馏结构性不在 daemon 内，这是与宿主「单写者」的机械保证，不靠用户自觉；检索为关键词 + BM25（向量由后续 PR 抽取 semantic 装配后接入）。token 与 DSH 面板/CLI 共用同一 kv 凭证，端口/主机解析链与外部访问一致；`createStandaloneApi` 新增 `strictPort` 选项——daemon 的配置端口被占即报错退出而非顺延（第三方把 URL 写死，静默换端口等于坏），不传该选项的宿主旁路行为不变。`/search` 照常落 recall_runs，第三方检索的复用统计不缺数。多进程共存（daemon 与宿主同库互写互读）有专门回归锁；已知限制（双进程去重竞态、镜像双写、版本偏斜）见 docs/DAEMON.md。 +- **daemon 向量检索（PR2，#363）**：`dsh-mneme-serve` 的 `/search` 接入完整语义管线——embedder/reranker 装配与 boot 自动回填从 `index.js` **纯搬移**至 `src/semantic.js`（宿主与 daemon 共用同一份，调用时序契约原样；`backfillMissingEmbeddings` 经 index.js barrel 再出口，测试调用方零改动），daemon 侧新增 `createVectorIndex` 接线与 `--embed` 参数：`local`（默认，自管 runtime/嵌入模型缺失时经 `provisionRuntime` download 档自动取件，可用 `DSH_MNEME_RUNTIME_TARBALL_DIR`/`DSH_MNEME_RUNTIME_MIRROR` 换离线/镜像来源；失败降级关键词并打可操作日志）、`ollama`、`openai`（读宿主面板 vector-config）、`off`。向量轴有注入假 embedder 的回归锁；`createServeRuntime` 因此转为 async、语义键默认值在 `daemonSemanticCfg` 逐键锚定 config.js。 +- **独立服务 daemon（`dsh-mneme-serve`，#363）**：mneme 现在能在 DSH 宿主之外常驻——`src/serve.js` 的 `createServeRuntime` 用最小装配（store → settings → mirror → service → maintenance → standalone API，每步锚定 index.js 装配行号）把数据面跑成独立进程，第三方集成（网页端桥接等）不必为挂载记忆库而保持 DSH 开机。第一期刻意无 LLM：巩固（autoDream）与蒸馏结构性不在 daemon 内，这是与宿主「单写者」的机械保证，不靠用户自觉。token 与 DSH 面板/CLI 共用同一 kv 凭证，端口/主机解析链与外部访问一致；`createStandaloneApi` 新增 `strictPort` 选项——daemon 的配置端口被占即报错退出而非顺延（第三方把 URL 写死，静默换端口等于坏），不传该选项的宿主旁路行为不变。`/search` 照常落 recall_runs，第三方检索的复用统计不缺数。多进程共存（daemon 与宿主同库互写互读）有专门回归锁；已知限制（双进程去重竞态、镜像双写、版本偏斜）见 docs/DAEMON.md。 ## [0.8.12] - 2026-10-01 diff --git a/dsh-mneme/README.md b/dsh-mneme/README.md index 8fdbbb2..fa2bafd 100644 --- a/dsh-mneme/README.md +++ b/dsh-mneme/README.md @@ -5,7 +5,7 @@ [![npm version](https://img.shields.io/npm/v/@modusensus/dsh-mneme?color=blue&label=npm)](https://www.npmjs.com/package/@modusensus/dsh-mneme) [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![Awesome](https://awesome-dsh-plugin.com/badge.svg)](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin) -[![tests](https://img.shields.io/badge/tests-1502%20passed-success)](https://github.com/slow-stack/mneme) +[![tests](https://img.shields.io/badge/tests-1504%20passed-success)](https://github.com/slow-stack/mneme) [![CI](https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml)](https://github.com/slow-stack/mneme/actions) [![node](https://img.shields.io/badge/node-22%2B-blue)](https://nodejs.org) [![npm downloads](https://img.shields.io/npm/d18m/@modusensus/dsh-mneme.svg?color=blue&label=downloads)](https://www.npmjs.com/package/@modusensus/dsh-mneme) @@ -522,10 +522,11 @@ dsh-mneme config show # 查看当前配置（toke ```bash dsh-mneme-serve # 默认 ~/.dsh/memory + 8790 dsh-mneme-serve --memory-dir "D:\my mem" --port 8790 --host 127.0.0.1 +dsh-mneme-serve --embed off # 纯关键词 + BM25(不取件模型) ``` - **鉴权与端口**：Bearer token 与 DSH 面板 / CLI 共用同一份（kv `external_api`，首次启动自动生成并持久化到 `memory.db`）；端口/主机解析链与「外部访问」一致（显式参数 > 持久值 > 默认 8790/127.0.0.1）。配置端口被占会**直接报错退出**（不做端口顺延）——第三方把 URL 写死，静默换端口等于坏。因此 **daemon 与 DSH 的「外部访问」二选一**，不要同端口同开。 -- **能力边界（第一期，无 LLM）**：存储 / 检索（关键词 + BM25）/ 镜像同步与人改合并 / `POST /maintenance/reclaim` / `/bootstrap` 全可用；巩固（autoDream）与蒸馏不在 daemon 内——巩固只属于 DSH 宿主进程，这是与宿主「单写者」的机械保证。向量检索暂缺（后续版本接入），`/search` 退化为关键词 + BM25 属预期。 +- **能力边界（第一期，无 LLM）**：存储 / 检索（关键词 + BM25 + 向量）/ 镜像同步与人改合并 / `POST /maintenance/reclaim` / `/bootstrap` 全可用；巩固（autoDream）与蒸馏不在 daemon 内——巩固只属于 DSH 宿主进程，这是与宿主「单写者」的机械保证。`--embed` 默认 `local`（自管 runtime 与嵌入模型缺失时自动取件，约 200MB；失败降级关键词并打日志），也可选 `ollama` / `openai`（读宿主面板的 vector-config）/ `off`。 - **检索回执**：daemon 的 `/search` 同样落 `recall_runs`，第三方检索的复用统计不缺数。 - **生命周期**：stdout 仅就绪时打一行 `dsh-mneme-serve listening on http://host:port (pid N)`（供脚本解析实际端口），日志走 stderr；SIGINT/SIGTERM 优雅收库，Windows 强杀由 WAL 回放兜底。 - **已知限制**：与 DSH 同时运行属设计内场景（WAL 多进程并发），但去重是先查后写、库层无 UNIQUE 约束，双进程并发写同一 `(type, title, scope)` 有极小概率产生重复；daemon 与插件请同版本升级。细节与坑清单见 [docs/DAEMON.md](docs/DAEMON.md)。 @@ -599,7 +600,7 @@ src/ ├── api.js # HTTP 路由（Web 面板数据通道，含 /conflicts 冲突队列） └── index.js # 插件接线 lib/ # src 的同步分发产物（npm run sync；发布前由 root prepack 的 check-sync.js 校验一致性；唯一手写例外 lib/client.js——Web 面板 bundle，sync 不覆盖） -test/ # 1502 个 node:test 测试（审计与三轴线压测不变量；src↔lib 一致性由 scripts/check-sync.js 发布闸门校验） +test/ # 1504 个 node:test 测试（审计与三轴线压测不变量；src↔lib 一致性由 scripts/check-sync.js 发布闸门校验） scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压测 · sync-lib.js 同步 · check-sync.js 发布闸门 · benchmark-recall.js / benchmark-embed.js / benchmark-rerank.js 基准 · sync-test-badge.mjs 测试徽章 · build-runtime-manifest.mjs 运行时清单 ``` @@ -608,7 +609,7 @@ scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压 ```bash cd dsh-mneme npm install # 安装 peer 依赖（以 devDependencies 形式，用于本地测试） -npm test # 运行 1502 个测试 +npm test # 运行 1504 个测试 npm run stress # 三轴线压测：长会话检索 / 冲突仲裁 / 多 Agent 并发（离线 mock LLM） npm run sync # 把 src/ 同步到 lib/（发布时由 prepack 钩子自动执行） ``` diff --git a/dsh-mneme/bin/dsh-mneme-serve.mjs b/dsh-mneme/bin/dsh-mneme-serve.mjs index dda2943..7988f52 100644 --- a/dsh-mneme/bin/dsh-mneme-serve.mjs +++ b/dsh-mneme/bin/dsh-mneme-serve.mjs @@ -23,18 +23,27 @@ const PKG = JSON.parse(readFileSync(join(PKG_ROOT, "package.json"), "utf8")); const USAGE = `${BIN_NAME} — run the mneme data plane as a standalone service (no DSH required) -Usage: dsh-mneme-serve [--memory-dir ] [--port ] [--host ] +Usage: dsh-mneme-serve [--memory-dir ] [--port ] [--host ] [--embed ] Options: --memory-dir data directory (default: ~/.dsh/memory, same as the plugin) --port HTTP port (default: persisted external_api port, else 8790) --host bind address (default: persisted external_api host, else 127.0.0.1) + --embed semantic retrieval: local (default, downloads the ONNX runtime + + embedding model on first boot) | ollama | openai (uses the + vector-config saved by the DSH panel) | off (keyword + BM25 only) -h, --help show this help -V, --version print version Auth: Bearer token is shared with the DSH panel / CLI (kv "external_api" in memory.db); it is generated on first boot. A busy configured port is a hard -error — the DSH external API and this daemon must not share a port (pick one).`; +error — the DSH external API and this daemon must not share a port (pick one). + +Environment (runtime provisioning, local provider only): + DSH_MNEME_MEMORY_DIR data directory override + DSH_MNEME_RUNTIME_DIR self-managed runtime dir (default ~/.dsh/mneme/runtime) + DSH_MNEME_RUNTIME_TARBALL_DIR offline .tgz dir preferred over the network + DSH_MNEME_RUNTIME_MIRROR npm registry mirror prefix (e.g. npmmirror)`; /** 极简 argv 解析(--k=v / --k v / 旗标);够用即可,完整 CLI 在 bin/cli.mjs。 */ function parseArgv(argv) { @@ -90,7 +99,21 @@ async function main(argv) { } const host = typeof args.host === "string" && args.host ? args.host : undefined; - const rt = createServeRuntime({ memoryDir, port, host, logger }); + let embed = "local"; + if (args.embed !== undefined) { + if (typeof args.embed !== "string" || !["off", "local", "ollama", "openai"].includes(args.embed)) { + fail(`--embed 需要 off|local|ollama|openai,收到: ${String(args.embed)}`); + } + embed = args.embed; + } + + let rt; + try { + // 装配是异步的:embed=local 时可能要先取件 runtime(download 档,失败内部降级) + rt = await createServeRuntime({ memoryDir, port, host, logger, embed }); + } catch (err) { + fail(`启动失败: ${err?.message ?? err}`); + } try { await rt.api.ready; } catch (err) { diff --git a/dsh-mneme/docs/DAEMON.md b/dsh-mneme/docs/DAEMON.md index 369b867..3fe8b90 100644 --- a/dsh-mneme/docs/DAEMON.md +++ b/dsh-mneme/docs/DAEMON.md @@ -6,7 +6,7 @@ daemon 是**数据面**,不是第二个宿主: -- **有**:存储(SQLite)、检索(关键词 + BM25,`/search` 统一召回)、镜像同步与人改合并、`/maintenance/reclaim`、`/bootstrap`、recall_runs 检索回执。 +- **有**:存储(SQLite)、检索(关键词 + BM25 + 向量,`/search` 统一召回,与宿主共用 `src/semantic.js` 同一套装配)、镜像同步与人改合并、`/maintenance/reclaim`、`/bootstrap`、recall_runs 检索回执。 - **没有(第一期,无 LLM)**:巩固(autoDream)、蒸馏(autoSummarize)、实体抽取、sleep、注入/工具/面板路由。前两者是**结构性缺失**而非开关——daemon 装配里没有 LLM 句柄,巩固只属于 DSH 宿主进程。这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md externalApi/autoDream 单侧纪律的 daemon 版),不依赖用户自觉。 与宿主装配(`src/index.js` apply)的关系:`src/serve.js` 只搬数据面那一半,每步注释锚定 index.js 来源行号;刻意不抽公共装配函数(apply 其余环节与宿主 ctx 纠缠,防御段纪律「最后动或不动」)。装配漂移风险由 `test/serve-bin.test.js` 的多进程共存用例兜底(两进程真开同一个库互写互读)。 @@ -18,10 +18,11 @@ daemon 是**数据面**,不是第二个宿主: CLI: ```bash -dsh-mneme-serve [--memory-dir ] [--port ] [--host ] +dsh-mneme-serve [--memory-dir ] [--port ] [--host ] [--embed ] ``` - `memoryDir`:CLI > env `DSH_MNEME_MEMORY_DIR` > `~/.dsh/memory`(与宿主 config.js 同默认,支持前导 `~`)。 +- `--embed`:语义检索提供方。`local`(默认:自管 runtime 与嵌入模型缺失时**自动取件**,download 档,约 200MB;失败降级关键词并打可操作日志)| `ollama` | `openai`(读宿主面板存的 vector-config)| `off`(纯关键词 + BM25)。取件来源可用 env 换道:`DSH_MNEME_RUNTIME_DIR`(自管 runtime 目录)、`DSH_MNEME_RUNTIME_TARBALL_DIR`(离线 .tgz 目录,优先于联网)、`DSH_MNEME_RUNTIME_MIRROR`(registry 镜像前缀)。 - port/host 解析链与宿主「外部访问」一致:显式参数 > kv `external_api` 持久值 > 默认 8790 / 127.0.0.1。 - token 与 DSH 面板 / CLI **共用同一份**(kv `external_api`,首次启动自动生成并持久化)——三方零配置互通。 - stdout 只在就绪时打一行 `dsh-mneme-serve listening on http://host:port (pid N)`(机器可读,脚本/测试解析端口用);日志全走 stderr。 @@ -29,7 +30,8 @@ dsh-mneme-serve [--memory-dir ] [--port ] [--host ] ## 3. 内部文件 -- `src/serve.js` — `createServeRuntime({memoryDir, port, host, logger, strictPort})`:装配链 createStore → createSettings → createMirror → createService(最小 config)→ recoverMirror → 人改镜像合并 → recall recorder → createMaintenance → createStandaloneApi,每步锚定 index.js 行号。返回 `{api, store, service, settings, maintenance, tokenExisted, dispose}`;第三方可 import 它自行托管生命周期(bin 只是薄壳)。 +- `src/serve.js` — `createServeRuntime({memoryDir, port, host, logger, strictPort, embed, embedder, reranker})`:装配链 createStore → createSettings → createMirror → createService(最小 config)→ recoverMirror → 人改镜像合并闭包 → vectorIndex + semantic → recall recorder → createMaintenance → createStandaloneApi,每步锚定 index.js 行号。返回 `{api, store, service, settings, maintenance, semantic, tokenExisted, dispose}`;第三方可 import 它自行托管生命周期(bin 只是薄壳),`embedder/reranker` 参数供注入自管嵌入。 +- `src/semantic.js` — embedder/reranker 装配 + boot 自动回填,**纯搬移自 index.js**(PR2),宿主与 daemon 共用同一份;`backfillMissingEmbeddings` 经 index.js barrel 再出口(测试照旧从 index.js import)。 - `bin/dsh-mneme-serve.mjs` — CLI 壳。独立成 bin 而非 cli.mjs 子命令:CONTRIBUTING 禁止给 cli.mjs 加 import;命名循 dsh-mneme-mcp 先例。 - `src/api-standalone.js` 的 `strictPort` 选项 — 唯一的数据面改动,默认关闭。 @@ -39,5 +41,5 @@ dsh-mneme-serve [--memory-dir ] [--port ] [--host ] 2. **双进程写并发**:与 DSH 同时运行是设计内场景(WAL + busy_timeout 先序,store.js createStore)。但 `saveWithDedupe` 的 (type,title,scope) 去重是先查后写、库层无 UNIQUE 约束,两进程并发写同一三元组有极小概率产生重复条目——已知限制,勿当强保证宣传(要不要加 UNIQUE 索引属 schema 防御段,单独决策)。 3. **镜像双写竞态**:daemon 与宿主都会渲镜像 .md;可再生物,失败由 recoverMirror 自愈,极端并发下单文件可能短暂脏,下次同步覆盖。 4. **版本偏斜**:库迁移是幂等加法式(PRAGMA 检查 + ALTER),旧代码读新 schema 一般无碍,但该组合无人测过——daemon 与插件请同版本升级。 -5. **第一期不吃宿主配置**:daemon 不加载 config schema(schemastery 是宿主 peer 依赖),面板/feature_flags 对它不生效;它只有 CLI 参数 + 上述固定最小 config(`language: zh`、document 子系统关闭)。向量检索(依赖 embedder/reranker 装配)由后续 PR 接入,接入前 `/search` 退化关键词 + BM25 属预期。 -6. **验收锚点**:`test/serve.test.js`(in-process 全链路 + token 复用 + recall_runs 回执)、`test/serve-bin.test.js`(真子进程 + 多进程互写互读)、`test/standalone-api.test.js` 的 strictPort 用例(busy → reject,默认路径仍顺延)。 +5. **第一期不吃宿主配置**:daemon 不加载 config schema(schemastery 是宿主 peer 依赖),面板/feature_flags 对它不生效;它只有 CLI 参数 + 上述固定最小 config(`language: zh`、document 子系统关闭;语义键默认值逐键锚定 config.js,见 `src/serve.js` 的 `daemonSemanticCfg`)。`--embed local` 首次启动会自动取件 runtime + 模型(均有日志);取件/嵌入失败统一降级关键词 + BM25,不影响读写。 +6. **验收锚点**:`test/serve.test.js`(in-process 全链路 + token 复用 + recall_runs 回执 + 注入假 embedder 的向量轴锁)、`test/serve-bin.test.js`(真子进程 + 多进程互写互读)、`test/standalone-api.test.js` 的 strictPort 用例(busy → reject,默认路径仍顺延)、`test/reindex-backfill.test.js`(semantic 纯搬移后宿主行为不变)。 diff --git a/dsh-mneme/lib/index.js b/dsh-mneme/lib/index.js index b691e6c..a9267ef 100644 --- a/dsh-mneme/lib/index.js +++ b/dsh-mneme/lib/index.js @@ -17,10 +17,10 @@ import { createStandaloneApi } from "./api-standalone.js"; import { createMaintenance } from "./maintenance.js"; import { createSettings } from "./settings.js"; import { createCommandManager } from "./commands.js"; -import { createEmbedder } from "./embedding.js"; -import { createEmbedderByProvider } from "./local-embedder.js"; -import { LocalReranker } from "./reranker.js"; import { createVectorIndex } from "./vector-index.js"; +// semantic(embedder/reranker/boot 回填)纯搬移至 src/semantic.js(PR2):宿主与 +// daemon(dsh-mneme-serve,#363)共用同一套装配,原文件保留 barrel 出口。 +import { createSemantic } from "./semantic.js"; import { Config, applyLightModePreset, injectChildEnabled } from "./config.js"; import { langOf } from "./lang.js"; import { extractEntities } from "./entities/extractor.js"; @@ -147,43 +147,9 @@ export function createEntityStreamAdapter({ llm, agentDefaultModel, logger, serv }; } -/** - * Issue #128: bounded backfill of rows still missing an embedding (active rows - * only — needsEmbedding filters archived/forgotten). Exported for tests. - * - * Runs regardless of the model fingerprint: the old call-site gate returned - * early when vector_meta already held the embedder's hash, permanently - * orphaning rows whose embed failed at write time (embedder not ready / - * provider rate limit) — one successful embed was enough to never backfill - * again. markModel is idempotent when the fingerprint already matches, so - * re-running costs nothing beyond the actually-missing rows. - */ -export async function backfillMissingEmbeddings({ - store, embedder, vectorIndex, logger, - maxTotal = 500, batchSize = 10, rateLimitMs = 200 -}) { - let indexed = 0; - for (let done = 0; done < maxTotal;) { - const rows = store.needsEmbedding(batchSize); - if (!rows.length) break; - for (const row of rows) { - try { - const text = [row.title, row.content].filter(Boolean).join("\n"); - const vector = await embedder.embedSingle(text); - if (vector?.length) { - store.setEmbedding(row.id, vector); - indexed++; - } - } catch { /* skip the bad row */ } - } - done += rows.length; - // Rate limit: space out batches so the provider is not hammered. - if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); - } - if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); - logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); - return indexed; -} +// backfillMissingEmbeddings 已随语义装配整体搬至 src/semantic.js(纯搬移); +// 保留 barrel 再出口 —— test/reindex-backfill.test.js 仍从本模块 import,调用方零改动。 +export { backfillMissingEmbeddings } from "./semantic.js"; export const apply = (ctx, config) => { const rawCfg = Config(config); @@ -331,150 +297,14 @@ export const apply = (ctx, config) => { } }; - let embedder = null; - let reranker = null; - // #118: pending embedder-init retry timer, cleared on unload. - let embedRetryTimer = null; - if (lightMode) { - // Light mode: the whole vector pipeline stays off — no embedder (nothing - // pulls in ONNX/transformers), no reranker, no boot backfill (the preset - // also cleared autoReindexOnBoot). Recall degrades to keyword search and - // human mirror edits still merge on boot. - applyHumanEdits(); - } else if (cfg.embedProvider === "openai") { - // vectorIndex is passed so the legacy OpenAI embedder records the producing - // model fingerprint after each successful embed (Bug3). - embedder = createEmbedder({ store, settings, logger: ctx.logger, vectorIndex }); - service.setEmbedder(embedder); - // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 - // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 - // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 - if (embedder.configured === false) { - ctx.logger?.warn?.( - "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" - + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" - + "dream 的语义聚类会退化为全量窗口兜底。" - + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" - ); - } - // legacy OpenAI embedder needs no async init → human edits apply right away - applyHumanEdits(); - } else { - try { - embedder = createEmbedderByProvider(cfg.embedProvider, { - model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, - dimension: cfg.localEmbedDimension, - device: cfg.localEmbedDevice, - batchSize: cfg.localEmbedBatchSize, - // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， - // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 - pooling: cfg.localEmbedPooling, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - baseUrl: cfg.ollamaBaseUrl, - logger: ctx.logger - }); - service.setEmbedder(embedder); - // issue #6: wait for extractor init before applying human edits, so - // scheduled embeddings see a ready embedder. - const bootEmbedder = () => embedder.init() - .then(() => { applyHumanEdits(); return true; }) - .catch(() => false); - // #118: the old one-shot probe permanently degraded search to keyword - // when Ollama was briefly unreachable at boot (recoverable only by - // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); - // search degrades to keyword meanwhile because per-query embed failures - // are swallowed. - bootEmbedder().then((ok) => { - if (ok) return; - let tries = 4; - const retry = () => { - if (tries-- <= 0) { - ctx.logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); - service.setEmbedder(null); - applyHumanEdits(); - return; - } - embedRetryTimer = setTimeout(async () => { - if (await bootEmbedder()) return; - retry(); - }, 15_000); - }; - ctx.logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); - retry(); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); - applyHumanEdits(); - } - } - - // Cross-encoder rerank over recall candidates. Best-effort: a failed model - // load only disables reranking, never search itself. Explicit opt-in only - // (rerankEnabled defaults to false): constructing LocalReranker is what pulls - // in onnxruntime, so the default config never loads it (item ⑥). - if (cfg.rerankEnabled && cfg.rerankProvider === "local") { - try { - reranker = new LocalReranker({ - model: cfg.rerankModel, - batchSize: cfg.rerankBatchSize, - maxCandidates: cfg.rerankMaxCandidates, - scoreThreshold: cfg.rerankScoreThreshold, - device: cfg.localEmbedDevice, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； - // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 - useDtype: cfg.rerankDtype, - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - logger: ctx.logger - }); - service.setReranker(reranker); - reranker.init().catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); - service.setReranker(null); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); - } - } - - // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API - // is configured and rows still lack an embedding (e.g. written before vector - // search was enabled), the backfill runs in the background after a short - // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a - // large backlog never floods the provider. Failures degrade silently — - // search stays keyword. - function scheduleAutoReindex() { - if (cfg.autoReindexOnBoot === false) return; - const attempt = (tries) => { - try { - if (!embedder || typeof embedder.embedSingle !== "function") return; - if ("ready" in embedder && embedder.ready !== true) { - // Local/ollama embedders init asynchronously; give them a moment - // before giving up on this boot (next boot retries). - if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); - return; - } - if (!store.needsEmbedding(1).length) return; // nothing to backfill - // Issue #128: no fingerprint gate here anymore — a matching fingerprint - // used to return early and permanently orphan rows whose embed failed - // at write time. See backfillMissingEmbeddings(). - backfillMissingEmbeddings({ store, embedder, vectorIndex, logger: ctx.logger }) - .catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - } - }; - setTimeout(() => attempt(5), 5000); - } - scheduleAutoReindex(); + // embedder/reranker/boot 回填装配已整体搬至 src/semantic.js(纯搬移,宿主与 daemon + // 共用):调用时序(applyHumanEdits 在各分支的触发点、#118 重试、autoReindexOnBoot) + // 原样保留在 createSemantic 内部,这里只拿引用。init 全失败的 embedder 引用仍会进入 + // dream/sleep 的 semantic 面 —— 与搬移前一致:检索侧 setEmbedder(null) 降级关键词。 + const semantic = createSemantic({ + store, service, settings, cfg, logger: ctx.logger, vectorIndex, applyHumanEdits, lightMode + }); + const { embedder, reranker } = semantic; // Custom commands: register persisted commands into the DSH command registry // on boot; add/remove re-register live through the API. @@ -593,7 +423,7 @@ export const apply = (ctx, config) => { // #118: never let a pending embedder init retry fire after unload and touch // a torn-down context. - disposers.push(() => { if (embedRetryTimer !== null) clearTimeout(embedRetryTimer); }); + disposers.push(() => semantic.dispose()); // #118 重试计时器 + boot 回填计时器(搬入 semantic.js 后由它自持) ctx.inject(["systemPrompt"], (promptCtx) => { if (cfg.autoInject) disposers.push(createInjector(promptCtx, service, settings, cfg)); diff --git a/dsh-mneme/lib/semantic.js b/dsh-mneme/lib/semantic.js new file mode 100644 index 0000000..f2732ea --- /dev/null +++ b/dsh-mneme/lib/semantic.js @@ -0,0 +1,227 @@ +// src/semantic.js —— embedder/reranker 装配与 boot 自动回填。 +// 纯搬移自 src/index.js(2026-10,PR2):backfillMissingEmbeddings(原 :150-186)与 +// 装配段(原 :311-477),行为逐字节对齐,仅两处已注记的机械差异(ctx.logger → 注入 +// logger;boot 回填的首查计时器纳入 dispose)。搬移原因:daemon(dsh-mneme-serve, +// #363)与宿主要共用同一套语义装配——「向量检索开箱即用」的承诺落在两侧同一份 +// 代码上,而不是 daemon 复刻一份会漂移的副本。拆法遵循 AGENTS.md 尺寸约定: +// 纯搬移独立 PR、原文件调用方零改动(backfill 经 index.js barrel 再出口,测试照旧)。 +// +// 时序契约(搬移前即如此,由 index.js 全量测试与 reindex-backfill.test.js 锁): +// lightMode / openai / 同步构造失败 → applyHumanEdits 立即; +// local|ollama → init 成功后 applyHumanEdits;#118 重试(1 + 4×15s)耗尽 → +// setEmbedder(null) 检索降级关键词,随后仍 applyHumanEdits; +// reranker 异步 init 失败只降级 rerank 自身,绝不影响 search。 +import { createEmbedder } from "./embedding.js"; +import { createEmbedderByProvider } from "./local-embedder.js"; +import { LocalReranker } from "./reranker.js"; + +/** + * Issue #128: bounded backfill of rows still missing an embedding (active rows + * only — needsEmbedding filters archived/forgotten). Exported for tests. + * + * Runs regardless of the model fingerprint: the old call-site gate returned + * early when vector_meta already held the embedder's hash, permanently + * orphaning rows whose embed failed at write time (embedder not ready / + * provider rate limit) — one successful embed was enough to never backfill + * again. markModel is idempotent when the fingerprint already matches, so + * re-running costs nothing beyond the actually-missing rows. + */ +export async function backfillMissingEmbeddings({ + store, embedder, vectorIndex, logger, + maxTotal = 500, batchSize = 10, rateLimitMs = 200 +}) { + let indexed = 0; + for (let done = 0; done < maxTotal;) { + const rows = store.needsEmbedding(batchSize); + if (!rows.length) break; + for (const row of rows) { + try { + const text = [row.title, row.content].filter(Boolean).join("\n"); + const vector = await embedder.embedSingle(text); + if (vector?.length) { + store.setEmbedding(row.id, vector); + indexed++; + } + } catch { /* skip the bad row */ } + } + done += rows.length; + // Rate limit: space out batches so the provider is not hammered. + if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); + } + if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); + logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); + return indexed; +} + +/** + * 组装语义管线(embedder + reranker)并挂到 service 上,随后调度 boot 自动回填。 + * @param {object} opts + * store/service/settings/vectorIndex — 宿主与 daemon 同形传入; + * cfg — 宿主传合并后的完整配置;daemon 传语义子集(默认值锚定 config.js); + * logger — console 形状(原代码读 ctx.logger,搬移后注入); + * applyHumanEdits— 人改镜像合并回调(index.js 闭包,读 mirror.readHumanEdits); + * 在哪个分支何时被调是时序契约的一部分,见文件头; + * lightMode — 轻量档:整条向量管线关闭,无 embedder/reranker、不回填。 + * @returns {{embedder, reranker, dispose}} embedder 构造失败(同步抛)时为 null; + * init 异步失败经 #118 重试后 service 侧降级,此处引用仍在(dream/sleep 语义面 + * 与搬移前一致)。dispose 清两个引导期计时器。 + */ +export function createSemantic({ store, service, settings, cfg, logger, vectorIndex, applyHumanEdits, lightMode = false }) { + let embedder = null; + let reranker = null; + // #118: pending embedder-init retry timer, cleared on unload. + let embedRetryTimer = null; + // boot 回填首查计时器。搬移前在 index.js 是裸 setTimeout(不参与卸载清理); + // 纳入 dispose 是修悬挂,不改变启动行为。 + let reindexTimer = null; + + if (lightMode) { + // Light mode: the whole vector pipeline stays off — no embedder (nothing + // pulls in ONNX/transformers), no reranker, no boot backfill (the preset + // also cleared autoReindexOnBoot). Recall degrades to keyword search and + // human mirror edits still merge on boot. + applyHumanEdits(); + } else if (cfg.embedProvider === "openai") { + // vectorIndex is passed so the legacy OpenAI embedder records the producing + // model fingerprint after each successful embed (Bug3). + embedder = createEmbedder({ store, settings, logger, vectorIndex }); + service.setEmbedder(embedder); + // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 + // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 + // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 + if (embedder.configured === false) { + logger?.warn?.( + "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" + + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" + + "dream 的语义聚类会退化为全量窗口兜底。" + + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" + ); + } + // legacy OpenAI embedder needs no async init → human edits apply right away + applyHumanEdits(); + } else { + try { + embedder = createEmbedderByProvider(cfg.embedProvider, { + model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, + dimension: cfg.localEmbedDimension, + device: cfg.localEmbedDevice, + batchSize: cfg.localEmbedBatchSize, + // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， + // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 + pooling: cfg.localEmbedPooling, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + baseUrl: cfg.ollamaBaseUrl, + logger + }); + service.setEmbedder(embedder); + // issue #6: wait for extractor init before applying human edits, so + // scheduled embeddings see a ready embedder. + const bootEmbedder = () => embedder.init() + .then(() => { applyHumanEdits(); return true; }) + .catch(() => false); + // #118: the old one-shot probe permanently degraded search to keyword + // when Ollama was briefly unreachable at boot (recoverable only by + // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); + // search degrades to keyword meanwhile because per-query embed failures + // are swallowed. + bootEmbedder().then((ok) => { + if (ok) return; + let tries = 4; + const retry = () => { + if (tries-- <= 0) { + logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); + service.setEmbedder(null); + applyHumanEdits(); + return; + } + embedRetryTimer = setTimeout(async () => { + if (await bootEmbedder()) return; + retry(); + }, 15_000); + }; + logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); + retry(); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); + applyHumanEdits(); + } + } + + // Cross-encoder rerank over recall candidates. Best-effort: a failed model + // load only disables reranking, never search itself. Explicit opt-in only + // (rerankEnabled defaults to false): constructing LocalReranker is what pulls + // in onnxruntime, so the default config never loads it (item ⑥). + if (cfg.rerankEnabled && cfg.rerankProvider === "local") { + try { + reranker = new LocalReranker({ + model: cfg.rerankModel, + batchSize: cfg.rerankBatchSize, + maxCandidates: cfg.rerankMaxCandidates, + scoreThreshold: cfg.rerankScoreThreshold, + device: cfg.localEmbedDevice, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； + // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 + useDtype: cfg.rerankDtype, + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + logger + }); + service.setReranker(reranker); + reranker.init().catch((error) => { + logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); + service.setReranker(null); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); + } + } + + // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API + // is configured and rows still lack an embedding (e.g. written before vector + // search was enabled), the backfill runs in the background after a short + // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a + // large backlog never floods the provider. Failures degrade silently — + // search stays keyword. + function scheduleAutoReindex() { + if (cfg.autoReindexOnBoot === false) return; + const attempt = (tries) => { + try { + if (!embedder || typeof embedder.embedSingle !== "function") return; + if ("ready" in embedder && embedder.ready !== true) { + // Local/ollama embedders init asynchronously; give them a moment + // before giving up on this boot (next boot retries). + if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); + return; + } + if (!store.needsEmbedding(1).length) return; // nothing to backfill + // Issue #128: no fingerprint gate here anymore — a matching fingerprint + // used to return early and permanently orphan rows whose embed failed + // at write time. See backfillMissingEmbeddings(). + backfillMissingEmbeddings({ store, embedder, vectorIndex, logger }) + .catch((error) => { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + } + }; + reindexTimer = setTimeout(() => attempt(5), 5000); + } + scheduleAutoReindex(); + + return { + embedder, + reranker, + dispose() { + if (embedRetryTimer !== null) { clearTimeout(embedRetryTimer); embedRetryTimer = null; } + if (reindexTimer !== null) { clearTimeout(reindexTimer); reindexTimer = null; } + } + }; +} diff --git a/dsh-mneme/lib/serve.js b/dsh-mneme/lib/serve.js index 7d13902..b7f87fe 100644 --- a/dsh-mneme/lib/serve.js +++ b/dsh-mneme/lib/serve.js @@ -2,15 +2,15 @@ // (#363 承诺的「官方推荐第三方挂载姿势」)。bin/dsh-mneme-serve.mjs 是它的 CLI 壳, // Mneme Bridge 这类第三方也可直接 import 本模块自行托管生命周期。 // -// 与宿主装配(src/index.js apply)的关系:只搬数据面那一半,每步注释锚定 index.js -// 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与宿主 ctx -// 纠缠,防御段纪律是「最后动或不动」;这 60 行的漂移风险由 serve-bin 测试的多进程 -// 共存用例兜底(两侧真开同一个库互写互读)。 +// 与宿主装配(src/index.js apply)的关系:只搬数据面 + 语义管线两块,每步注释锚定 +// index.js 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与 +// 宿主 ctx 纠缠,防御段纪律是「最后动或不动」;本文件的漂移风险由 serve-bin 测试的 +// 多进程共存用例兜底(两侧真开同一个库互写互读)。 // -// 第一期无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只 -// 属于 DSH 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 -// externalApi/autoDream 单侧纪律),不依赖用户自觉。检索是关键词 + BM25(service -// 内建);向量管线由 PR2 的 semantic 抽取接入。 +// 无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只属于 DSH +// 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 externalApi/ +// autoDream 单侧纪律),不依赖用户自觉。语义检索(PR2 起)与宿主共用 src/semantic.js +// 的同一份装配;daemon 不参与做梦,但向量检索/语义去重与宿主同质。 import { mkdirSync } from "node:fs"; import { homedir } from "node:os"; import { join } from "node:path"; @@ -21,19 +21,34 @@ import { langOf } from "./lang.js"; import { createService } from "./service.js"; import { createMaintenance } from "./maintenance.js"; import { createStandaloneApi } from "./api-standalone.js"; +import { createSemantic } from "./semantic.js"; +import { createVectorIndex } from "./vector-index.js"; +import { provisionRuntime } from "./runtime/provision.js"; +import { defaultRuntimeDir, listPayloadDirs } from "./runtime/layout.js"; + +const EMBED_PROVIDERS = new Set(["off", "local", "ollama", "openai"]); /** - * 组装并启动一个独立数据面。 + * 组装并启动一个独立数据面(含语义管线)。 * @param {object} opts * memoryDir — 数据目录;缺省与宿主同默认 ~/.dsh/memory(config.js:9),支持前导 ~。 * port/host — 透传 createStandaloneApi;缺省走 kv external_api 持久值 > 8790/127.0.0.1。 * logger — console 形状(info/warn/error,收单字符串);缺省 null(全链路容缺)。 * strictPort — 默认 true:配置端口被占即失败(第三方把 URL 写死,顺延=静默打到 * 错误端口)。显式 port=0(测试/OS 分配)不受影响。 - * @returns {api, store, service, settings, maintenance, tokenExisted, dispose} + * embed — "local"(默认,自管 runtime 取件 + ONNX 嵌入)|"ollama"|"openai" + * (读宿主面板配的 vector-config)|"off"(纯关键词 + BM25)。 + * embedder/reranker — 注入式覆盖(测试/宿主方自管):给了就跳过 createSemantic, + * 直接 setEmbedder/setReranker;仅 embed="off" 之外有意为之。 + * @returns {api, store, service, settings, maintenance, semantic, tokenExisted, dispose} * tokenExisted — 启动前 kv 里是否已有 token;false 时本次为首次生成,入口层可提示。 */ -export function createServeRuntime({ memoryDir, port, host, logger = null, strictPort = true } = {}) { +export async function createServeRuntime({ + memoryDir, port, host, logger = null, strictPort = true, embed = "local", embedder = null, reranker = null +} = {}) { + if (!EMBED_PROVIDERS.has(embed)) { + throw new Error(`invalid embed provider: ${String(embed)}(允许 off/local/ollama/openai)`); + } // index.js:192-195:~ 展开只认前导;目录不存在时 node:sqlite 直接抛,先 mkdir。 const dir = String(memoryDir || join(homedir(), ".dsh", "memory")).replace(/^~(?=$|[\\/])/, homedir()); mkdirSync(dir, { recursive: true }); @@ -55,10 +70,46 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric // index.js:319-332:人改镜像先合并——镜像文件里的手工编辑每次启动都赢。 // readHumanEdits 全类型一次读齐:mergeHumanEdits 成功会重渲全部镜像,逐类型读改 // 循环会拿没读到的类型覆盖掉未合并的编辑(index.js:324-327 注释同款坑)。 + // 何时 apply 由 semantic 的分支时序契约决定(见 semantic.js 文件头),本文件只造闭包。 const humanEdits = new Map(); for (const type of Object.keys(TYPE_FILE)) humanEdits.set(type, mirror.readHumanEdits(type)); - for (const [type, edits] of humanEdits) { - if (edits.length) service.mergeHumanEdits(type, edits); + const applyHumanEdits = () => { + for (const [type, edits] of humanEdits) { + if (edits.length) service.mergeHumanEdits(type, edits); + } + }; + + // index.js:316-317:向量索引包住 store 的 embedding 列并跟踪活跃模型指纹。 + const vectorIndex = createVectorIndex({ store, logger }); + service.setVectorIndex(vectorIndex); + + // ---- 语义管线(PR2):三路二选一 ------------------------------------------ + let semantic = null; + if (embedder !== null) { + // 注入式覆盖(测试 / 宿主方自管 embedder):绕过装配,人改合并立刻做 + //(对齐宿主 openai 分支的同步语义)。 + service.setEmbedder(embedder); + if (reranker !== null) service.setReranker(reranker); + applyHumanEdits(); + } else if (embed === "off") { + // 纯关键词 + BM25:对齐宿主 lightMode 分支——不建 embedder,人改合并立即。 + applyHumanEdits(); + } else { + let semanticReady = true; + if (embed === "local") { + // 自管 runtime 缺失时取件:只走 download 档(宿主 adopt 推导对独立进程无意义 + // ——没有宿主 node_modules 可推)。失败不阻断 daemon:降级关键词 + 可操作提示。 + semanticReady = await ensureRuntimePayload(logger); + } + if (semanticReady) { + semantic = createSemantic({ + store, service, settings, + cfg: daemonSemanticCfg(embed), + logger, vectorIndex, applyHumanEdits, lightMode: false + }); + } else { + applyHumanEdits(); + } } // index.js:294-309:检索回执落 recall_runs(searchMemories 的 recordRecall 默认开, @@ -92,11 +143,59 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric service, settings, maintenance, + semantic, tokenExisted, /** 收尾:先停收请求再关库。node:sqlite 对未 finalize 语句可能抛,吞掉——WAL 会在下次打开时回放,已提交事务不丢。 */ dispose() { try { api.server.close(); } catch { /* already closed */ } + try { semantic?.dispose(); } catch { /* 同上 */ } try { store.close(); } catch { /* 同上 */ } } }; } + +/** + * daemon 侧语义配置:默认值逐键锚定 src/config.js(309-420),daemon 不装 schemastery + * (宿主 peer 依赖),同值硬编码;config.js 改默认值时此处与 docs/DAEMON.md 同步。 + * 只放开 embedProvider(CLI --embed);rerank 暂不提供 CLI(默认关,与宿主同默认)。 + */ +function daemonSemanticCfg(embed) { + return { + embedProvider: embed, // config.js:309(宿主默认 openai;daemon 默认 local——独立部署唯一自洽路径) + localEmbedModel: "Xenova/bge-small-zh-v1.5", // :312 + localEmbedDimension: 512, // :313 + localEmbedDevice: "cpu", // :314 + localEmbedBatchSize: 8, // :315 + localEmbedPooling: "auto", // :323(BGE 系 CLS 口径,改它=换向量空间) + ollamaBaseUrl: "http://localhost:11434", // :326 + ollamaModel: "nomic-embed-text", // :331 + embedModelCacheDir: "", // :336(空=local-embedder 内部解析 ~/.dsh/mneme/models) + embedModelMirror: "https://hf-mirror.com", // :337 + resilientModelDownload: true, // :342 + runtimeDir: process.env.DSH_MNEME_RUNTIME_DIR || "", // :348(空=defaultRuntimeDir;取件/加载同源) + rerankEnabled: false, // :419 + rerankProvider: "none", // :420 + autoReindexOnBoot: true // :367 + }; +} + +/** + * 本地推理 runtime 取件(download 档)。返回 payload 是否可用: + * 已存在(含收编/早前下载)→ true 不重复取;缺失 → 取件成功 true / 失败 false(内部 + * 已打降级日志)。并发安全由 provisionRuntime 的 IN_FLIGHT 锁保证(按归一化 runtimeDir 串行)。 + */ +async function ensureRuntimePayload(logger) { + const runtimeDir = process.env.DSH_MNEME_RUNTIME_DIR || ""; + const dir = runtimeDir || defaultRuntimeDir(); + if (listPayloadDirs(dir).length > 0) return true; + logger?.info?.("[dsh-mneme] 本地推理运行时缺失,开始取件(download 档,约 200MB;离线可用 DSH_MNEME_RUNTIME_TARBALL_DIR / DSH_MNEME_RUNTIME_MIRROR)…"); + try { + const result = await provisionRuntime({ runtimeDir, localTarballDir: process.env.DSH_MNEME_RUNTIME_TARBALL_DIR || "", mirror: process.env.DSH_MNEME_RUNTIME_MIRROR || "" }); + if (!result.ok) throw new Error(result.reason); + logger?.info?.(`[dsh-mneme] 运行时就绪(${result.strategy}):${result.packages} 包 / ${result.files} 文件`); + return true; + } catch (error) { + logger?.warn?.(`[dsh-mneme] runtime provisioning failed, search degrades to keyword: ${String(error)}`); + return false; + } +} diff --git a/dsh-mneme/src/index.js b/dsh-mneme/src/index.js index b691e6c..a9267ef 100644 --- a/dsh-mneme/src/index.js +++ b/dsh-mneme/src/index.js @@ -17,10 +17,10 @@ import { createStandaloneApi } from "./api-standalone.js"; import { createMaintenance } from "./maintenance.js"; import { createSettings } from "./settings.js"; import { createCommandManager } from "./commands.js"; -import { createEmbedder } from "./embedding.js"; -import { createEmbedderByProvider } from "./local-embedder.js"; -import { LocalReranker } from "./reranker.js"; import { createVectorIndex } from "./vector-index.js"; +// semantic(embedder/reranker/boot 回填)纯搬移至 src/semantic.js(PR2):宿主与 +// daemon(dsh-mneme-serve,#363)共用同一套装配,原文件保留 barrel 出口。 +import { createSemantic } from "./semantic.js"; import { Config, applyLightModePreset, injectChildEnabled } from "./config.js"; import { langOf } from "./lang.js"; import { extractEntities } from "./entities/extractor.js"; @@ -147,43 +147,9 @@ export function createEntityStreamAdapter({ llm, agentDefaultModel, logger, serv }; } -/** - * Issue #128: bounded backfill of rows still missing an embedding (active rows - * only — needsEmbedding filters archived/forgotten). Exported for tests. - * - * Runs regardless of the model fingerprint: the old call-site gate returned - * early when vector_meta already held the embedder's hash, permanently - * orphaning rows whose embed failed at write time (embedder not ready / - * provider rate limit) — one successful embed was enough to never backfill - * again. markModel is idempotent when the fingerprint already matches, so - * re-running costs nothing beyond the actually-missing rows. - */ -export async function backfillMissingEmbeddings({ - store, embedder, vectorIndex, logger, - maxTotal = 500, batchSize = 10, rateLimitMs = 200 -}) { - let indexed = 0; - for (let done = 0; done < maxTotal;) { - const rows = store.needsEmbedding(batchSize); - if (!rows.length) break; - for (const row of rows) { - try { - const text = [row.title, row.content].filter(Boolean).join("\n"); - const vector = await embedder.embedSingle(text); - if (vector?.length) { - store.setEmbedding(row.id, vector); - indexed++; - } - } catch { /* skip the bad row */ } - } - done += rows.length; - // Rate limit: space out batches so the provider is not hammered. - if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); - } - if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); - logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); - return indexed; -} +// backfillMissingEmbeddings 已随语义装配整体搬至 src/semantic.js(纯搬移); +// 保留 barrel 再出口 —— test/reindex-backfill.test.js 仍从本模块 import,调用方零改动。 +export { backfillMissingEmbeddings } from "./semantic.js"; export const apply = (ctx, config) => { const rawCfg = Config(config); @@ -331,150 +297,14 @@ export const apply = (ctx, config) => { } }; - let embedder = null; - let reranker = null; - // #118: pending embedder-init retry timer, cleared on unload. - let embedRetryTimer = null; - if (lightMode) { - // Light mode: the whole vector pipeline stays off — no embedder (nothing - // pulls in ONNX/transformers), no reranker, no boot backfill (the preset - // also cleared autoReindexOnBoot). Recall degrades to keyword search and - // human mirror edits still merge on boot. - applyHumanEdits(); - } else if (cfg.embedProvider === "openai") { - // vectorIndex is passed so the legacy OpenAI embedder records the producing - // model fingerprint after each successful embed (Bug3). - embedder = createEmbedder({ store, settings, logger: ctx.logger, vectorIndex }); - service.setEmbedder(embedder); - // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 - // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 - // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 - if (embedder.configured === false) { - ctx.logger?.warn?.( - "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" - + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" - + "dream 的语义聚类会退化为全量窗口兜底。" - + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" - ); - } - // legacy OpenAI embedder needs no async init → human edits apply right away - applyHumanEdits(); - } else { - try { - embedder = createEmbedderByProvider(cfg.embedProvider, { - model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, - dimension: cfg.localEmbedDimension, - device: cfg.localEmbedDevice, - batchSize: cfg.localEmbedBatchSize, - // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， - // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 - pooling: cfg.localEmbedPooling, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - baseUrl: cfg.ollamaBaseUrl, - logger: ctx.logger - }); - service.setEmbedder(embedder); - // issue #6: wait for extractor init before applying human edits, so - // scheduled embeddings see a ready embedder. - const bootEmbedder = () => embedder.init() - .then(() => { applyHumanEdits(); return true; }) - .catch(() => false); - // #118: the old one-shot probe permanently degraded search to keyword - // when Ollama was briefly unreachable at boot (recoverable only by - // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); - // search degrades to keyword meanwhile because per-query embed failures - // are swallowed. - bootEmbedder().then((ok) => { - if (ok) return; - let tries = 4; - const retry = () => { - if (tries-- <= 0) { - ctx.logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); - service.setEmbedder(null); - applyHumanEdits(); - return; - } - embedRetryTimer = setTimeout(async () => { - if (await bootEmbedder()) return; - retry(); - }, 15_000); - }; - ctx.logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); - retry(); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); - applyHumanEdits(); - } - } - - // Cross-encoder rerank over recall candidates. Best-effort: a failed model - // load only disables reranking, never search itself. Explicit opt-in only - // (rerankEnabled defaults to false): constructing LocalReranker is what pulls - // in onnxruntime, so the default config never loads it (item ⑥). - if (cfg.rerankEnabled && cfg.rerankProvider === "local") { - try { - reranker = new LocalReranker({ - model: cfg.rerankModel, - batchSize: cfg.rerankBatchSize, - maxCandidates: cfg.rerankMaxCandidates, - scoreThreshold: cfg.rerankScoreThreshold, - device: cfg.localEmbedDevice, - cacheDir: cfg.embedModelCacheDir, - runtimeDir: cfg.runtimeDir, - // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； - // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 - useDtype: cfg.rerankDtype, - remoteHost: cfg.embedModelMirror, - resilientModelDownload: cfg.resilientModelDownload, - logger: ctx.logger - }); - service.setReranker(reranker); - reranker.init().catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); - service.setReranker(null); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); - } - } - - // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API - // is configured and rows still lack an embedding (e.g. written before vector - // search was enabled), the backfill runs in the background after a short - // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a - // large backlog never floods the provider. Failures degrade silently — - // search stays keyword. - function scheduleAutoReindex() { - if (cfg.autoReindexOnBoot === false) return; - const attempt = (tries) => { - try { - if (!embedder || typeof embedder.embedSingle !== "function") return; - if ("ready" in embedder && embedder.ready !== true) { - // Local/ollama embedders init asynchronously; give them a moment - // before giving up on this boot (next boot retries). - if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); - return; - } - if (!store.needsEmbedding(1).length) return; // nothing to backfill - // Issue #128: no fingerprint gate here anymore — a matching fingerprint - // used to return early and permanently orphan rows whose embed failed - // at write time. See backfillMissingEmbeddings(). - backfillMissingEmbeddings({ store, embedder, vectorIndex, logger: ctx.logger }) - .catch((error) => { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - }); - } catch (error) { - ctx.logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); - } - }; - setTimeout(() => attempt(5), 5000); - } - scheduleAutoReindex(); + // embedder/reranker/boot 回填装配已整体搬至 src/semantic.js(纯搬移,宿主与 daemon + // 共用):调用时序(applyHumanEdits 在各分支的触发点、#118 重试、autoReindexOnBoot) + // 原样保留在 createSemantic 内部,这里只拿引用。init 全失败的 embedder 引用仍会进入 + // dream/sleep 的 semantic 面 —— 与搬移前一致:检索侧 setEmbedder(null) 降级关键词。 + const semantic = createSemantic({ + store, service, settings, cfg, logger: ctx.logger, vectorIndex, applyHumanEdits, lightMode + }); + const { embedder, reranker } = semantic; // Custom commands: register persisted commands into the DSH command registry // on boot; add/remove re-register live through the API. @@ -593,7 +423,7 @@ export const apply = (ctx, config) => { // #118: never let a pending embedder init retry fire after unload and touch // a torn-down context. - disposers.push(() => { if (embedRetryTimer !== null) clearTimeout(embedRetryTimer); }); + disposers.push(() => semantic.dispose()); // #118 重试计时器 + boot 回填计时器(搬入 semantic.js 后由它自持) ctx.inject(["systemPrompt"], (promptCtx) => { if (cfg.autoInject) disposers.push(createInjector(promptCtx, service, settings, cfg)); diff --git a/dsh-mneme/src/semantic.js b/dsh-mneme/src/semantic.js new file mode 100644 index 0000000..f2732ea --- /dev/null +++ b/dsh-mneme/src/semantic.js @@ -0,0 +1,227 @@ +// src/semantic.js —— embedder/reranker 装配与 boot 自动回填。 +// 纯搬移自 src/index.js(2026-10,PR2):backfillMissingEmbeddings(原 :150-186)与 +// 装配段(原 :311-477),行为逐字节对齐,仅两处已注记的机械差异(ctx.logger → 注入 +// logger;boot 回填的首查计时器纳入 dispose)。搬移原因:daemon(dsh-mneme-serve, +// #363)与宿主要共用同一套语义装配——「向量检索开箱即用」的承诺落在两侧同一份 +// 代码上,而不是 daemon 复刻一份会漂移的副本。拆法遵循 AGENTS.md 尺寸约定: +// 纯搬移独立 PR、原文件调用方零改动(backfill 经 index.js barrel 再出口,测试照旧)。 +// +// 时序契约(搬移前即如此,由 index.js 全量测试与 reindex-backfill.test.js 锁): +// lightMode / openai / 同步构造失败 → applyHumanEdits 立即; +// local|ollama → init 成功后 applyHumanEdits;#118 重试(1 + 4×15s)耗尽 → +// setEmbedder(null) 检索降级关键词,随后仍 applyHumanEdits; +// reranker 异步 init 失败只降级 rerank 自身,绝不影响 search。 +import { createEmbedder } from "./embedding.js"; +import { createEmbedderByProvider } from "./local-embedder.js"; +import { LocalReranker } from "./reranker.js"; + +/** + * Issue #128: bounded backfill of rows still missing an embedding (active rows + * only — needsEmbedding filters archived/forgotten). Exported for tests. + * + * Runs regardless of the model fingerprint: the old call-site gate returned + * early when vector_meta already held the embedder's hash, permanently + * orphaning rows whose embed failed at write time (embedder not ready / + * provider rate limit) — one successful embed was enough to never backfill + * again. markModel is idempotent when the fingerprint already matches, so + * re-running costs nothing beyond the actually-missing rows. + */ +export async function backfillMissingEmbeddings({ + store, embedder, vectorIndex, logger, + maxTotal = 500, batchSize = 10, rateLimitMs = 200 +}) { + let indexed = 0; + for (let done = 0; done < maxTotal;) { + const rows = store.needsEmbedding(batchSize); + if (!rows.length) break; + for (const row of rows) { + try { + const text = [row.title, row.content].filter(Boolean).join("\n"); + const vector = await embedder.embedSingle(text); + if (vector?.length) { + store.setEmbedding(row.id, vector); + indexed++; + } + } catch { /* skip the bad row */ } + } + done += rows.length; + // Rate limit: space out batches so the provider is not hammered. + if (store.needsEmbedding(1).length) await new Promise((r) => setTimeout(r, rateLimitMs)); + } + if (indexed > 0 && embedder.modelHash) vectorIndex.markModel?.(embedder.modelHash, embedder.dimension); + logger?.info?.(`[dsh-mneme] auto-reindex backfilled ${indexed} embeddings on boot`); + return indexed; +} + +/** + * 组装语义管线(embedder + reranker)并挂到 service 上,随后调度 boot 自动回填。 + * @param {object} opts + * store/service/settings/vectorIndex — 宿主与 daemon 同形传入; + * cfg — 宿主传合并后的完整配置;daemon 传语义子集(默认值锚定 config.js); + * logger — console 形状(原代码读 ctx.logger,搬移后注入); + * applyHumanEdits— 人改镜像合并回调(index.js 闭包,读 mirror.readHumanEdits); + * 在哪个分支何时被调是时序契约的一部分,见文件头; + * lightMode — 轻量档:整条向量管线关闭,无 embedder/reranker、不回填。 + * @returns {{embedder, reranker, dispose}} embedder 构造失败(同步抛)时为 null; + * init 异步失败经 #118 重试后 service 侧降级,此处引用仍在(dream/sleep 语义面 + * 与搬移前一致)。dispose 清两个引导期计时器。 + */ +export function createSemantic({ store, service, settings, cfg, logger, vectorIndex, applyHumanEdits, lightMode = false }) { + let embedder = null; + let reranker = null; + // #118: pending embedder-init retry timer, cleared on unload. + let embedRetryTimer = null; + // boot 回填首查计时器。搬移前在 index.js 是裸 setTimeout(不参与卸载清理); + // 纳入 dispose 是修悬挂,不改变启动行为。 + let reindexTimer = null; + + if (lightMode) { + // Light mode: the whole vector pipeline stays off — no embedder (nothing + // pulls in ONNX/transformers), no reranker, no boot backfill (the preset + // also cleared autoReindexOnBoot). Recall degrades to keyword search and + // human mirror edits still merge on boot. + applyHumanEdits(); + } else if (cfg.embedProvider === "openai") { + // vectorIndex is passed so the legacy OpenAI embedder records the producing + // model fingerprint after each successful embed (Bug3). + embedder = createEmbedder({ store, settings, logger, vectorIndex }); + service.setEmbedder(embedder); + // issue #135: 未配置时明确告警一次。此前 legacy OpenAI embedder 恒报 + // ready=true，向量层「绿的但全哑」可以静默存在很久（本机持续了数周）。 + // 只记日志、不阻断启动：轻量模式与「先跑起来再补配置」都是正当用法。 + if (embedder.configured === false) { + logger?.warn?.( + "[dsh-mneme] 向量层未配置（vector-config 的 enabled/baseUrl/apiKey/model 有缺）：" + + "语义召回、语义去重、rerank、sleep 冲突检测将静默失效，" + + "dream 的语义聚类会退化为全量窗口兜底。" + + "请在设置面板补全 embedding 端点与模型，或把 embedProvider 改为 local/ollama。" + ); + } + // legacy OpenAI embedder needs no async init → human edits apply right away + applyHumanEdits(); + } else { + try { + embedder = createEmbedderByProvider(cfg.embedProvider, { + model: cfg.embedProvider === "ollama" ? cfg.ollamaModel : cfg.localEmbedModel, + dimension: cfg.localEmbedDimension, + device: cfg.localEmbedDevice, + batchSize: cfg.localEmbedBatchSize, + // 池化方式必须与模型的训练口径一致（BGE 系 = CLS）。它既进 embed() 的调用， + // 也进 modelHash —— 池化改了就是换向量空间，既有索引会被判失配并重建。 + pooling: cfg.localEmbedPooling, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：embedModelMirror 接成 transformers 的下载镜像（此前死配置）。 + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + baseUrl: cfg.ollamaBaseUrl, + logger + }); + service.setEmbedder(embedder); + // issue #6: wait for extractor init before applying human edits, so + // scheduled embeddings see a ready embedder. + const bootEmbedder = () => embedder.init() + .then(() => { applyHumanEdits(); return true; }) + .catch(() => false); + // #118: the old one-shot probe permanently degraded search to keyword + // when Ollama was briefly unreachable at boot (recoverable only by + // restart). Retry briefly (5 attempts total: 1 initial + 4 × 15s); + // search degrades to keyword meanwhile because per-query embed failures + // are swallowed. + bootEmbedder().then((ok) => { + if (ok) return; + let tries = 4; + const retry = () => { + if (tries-- <= 0) { + logger?.warn?.("[dsh-mneme] embedder init retries exhausted, search degrades to keyword"); + service.setEmbedder(null); + applyHumanEdits(); + return; + } + embedRetryTimer = setTimeout(async () => { + if (await bootEmbedder()) return; + retry(); + }, 15_000); + }; + logger?.warn?.("[dsh-mneme] embedder init failed, retrying"); + retry(); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] embedder unavailable, search degrades to keyword: ${String(error)}`); + applyHumanEdits(); + } + } + + // Cross-encoder rerank over recall candidates. Best-effort: a failed model + // load only disables reranking, never search itself. Explicit opt-in only + // (rerankEnabled defaults to false): constructing LocalReranker is what pulls + // in onnxruntime, so the default config never loads it (item ⑥). + if (cfg.rerankEnabled && cfg.rerankProvider === "local") { + try { + reranker = new LocalReranker({ + model: cfg.rerankModel, + batchSize: cfg.rerankBatchSize, + maxCandidates: cfg.rerankMaxCandidates, + scoreThreshold: cfg.rerankScoreThreshold, + device: cfg.localEmbedDevice, + cacheDir: cfg.embedModelCacheDir, + runtimeDir: cfg.runtimeDir, + // #188：量化档默认 q8（此前不传 dtype 会去要 1GB 级 fp32 模型）； + // embedModelMirror 此前是死配置，现接成 transformers 的下载镜像。 + useDtype: cfg.rerankDtype, + remoteHost: cfg.embedModelMirror, + resilientModelDownload: cfg.resilientModelDownload, + logger + }); + service.setReranker(reranker); + reranker.init().catch((error) => { + logger?.warn?.(`[dsh-mneme] reranker init failed, rerank disabled: ${String(error)}`); + service.setReranker(null); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] reranker unavailable, rerank disabled: ${String(error)}`); + } + } + + // Bug2: lazy auto-backfill of missing embeddings on boot. When the vector API + // is configured and rows still lack an embedding (e.g. written before vector + // search was enabled), the backfill runs in the background after a short + // delay. Gated on cfg.autoReindexOnBoot; rate-limited in small batches so a + // large backlog never floods the provider. Failures degrade silently — + // search stays keyword. + function scheduleAutoReindex() { + if (cfg.autoReindexOnBoot === false) return; + const attempt = (tries) => { + try { + if (!embedder || typeof embedder.embedSingle !== "function") return; + if ("ready" in embedder && embedder.ready !== true) { + // Local/ollama embedders init asynchronously; give them a moment + // before giving up on this boot (next boot retries). + if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); + return; + } + if (!store.needsEmbedding(1).length) return; // nothing to backfill + // Issue #128: no fingerprint gate here anymore — a matching fingerprint + // used to return early and permanently orphan rows whose embed failed + // at write time. See backfillMissingEmbeddings(). + backfillMissingEmbeddings({ store, embedder, vectorIndex, logger }) + .catch((error) => { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + }); + } catch (error) { + logger?.warn?.(`[dsh-mneme] auto-reindex failed: ${String(error)}`); + } + }; + reindexTimer = setTimeout(() => attempt(5), 5000); + } + scheduleAutoReindex(); + + return { + embedder, + reranker, + dispose() { + if (embedRetryTimer !== null) { clearTimeout(embedRetryTimer); embedRetryTimer = null; } + if (reindexTimer !== null) { clearTimeout(reindexTimer); reindexTimer = null; } + } + }; +} diff --git a/dsh-mneme/src/serve.js b/dsh-mneme/src/serve.js index 7d13902..b7f87fe 100644 --- a/dsh-mneme/src/serve.js +++ b/dsh-mneme/src/serve.js @@ -2,15 +2,15 @@ // (#363 承诺的「官方推荐第三方挂载姿势」)。bin/dsh-mneme-serve.mjs 是它的 CLI 壳, // Mneme Bridge 这类第三方也可直接 import 本模块自行托管生命周期。 // -// 与宿主装配(src/index.js apply)的关系:只搬数据面那一半,每步注释锚定 index.js -// 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与宿主 ctx -// 纠缠,防御段纪律是「最后动或不动」;这 60 行的漂移风险由 serve-bin 测试的多进程 -// 共存用例兜底(两侧真开同一个库互写互读)。 +// 与宿主装配(src/index.js apply)的关系:只搬数据面 + 语义管线两块,每步注释锚定 +// index.js 来源行号。刻意不抽公共装配函数——apply 的其余环节(注入/工具/dream)与 +// 宿主 ctx 纠缠,防御段纪律是「最后动或不动」;本文件的漂移风险由 serve-bin 测试的 +// 多进程共存用例兜底(两侧真开同一个库互写互读)。 // -// 第一期无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只 -// 属于 DSH 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 -// externalApi/autoDream 单侧纪律),不依赖用户自觉。检索是关键词 + BM25(service -// 内建);向量管线由 PR2 的 semantic 抽取接入。 +// 无 LLM:巩固(autoDream)与蒸馏(autoSummarize)结构上不在这里——巩固只属于 DSH +// 宿主进程,这就是 daemon 与宿主「单写者」的机械保证(AGENTS.md 的 externalApi/ +// autoDream 单侧纪律),不依赖用户自觉。语义检索(PR2 起)与宿主共用 src/semantic.js +// 的同一份装配;daemon 不参与做梦,但向量检索/语义去重与宿主同质。 import { mkdirSync } from "node:fs"; import { homedir } from "node:os"; import { join } from "node:path"; @@ -21,19 +21,34 @@ import { langOf } from "./lang.js"; import { createService } from "./service.js"; import { createMaintenance } from "./maintenance.js"; import { createStandaloneApi } from "./api-standalone.js"; +import { createSemantic } from "./semantic.js"; +import { createVectorIndex } from "./vector-index.js"; +import { provisionRuntime } from "./runtime/provision.js"; +import { defaultRuntimeDir, listPayloadDirs } from "./runtime/layout.js"; + +const EMBED_PROVIDERS = new Set(["off", "local", "ollama", "openai"]); /** - * 组装并启动一个独立数据面。 + * 组装并启动一个独立数据面(含语义管线)。 * @param {object} opts * memoryDir — 数据目录;缺省与宿主同默认 ~/.dsh/memory(config.js:9),支持前导 ~。 * port/host — 透传 createStandaloneApi;缺省走 kv external_api 持久值 > 8790/127.0.0.1。 * logger — console 形状(info/warn/error,收单字符串);缺省 null(全链路容缺)。 * strictPort — 默认 true:配置端口被占即失败(第三方把 URL 写死,顺延=静默打到 * 错误端口)。显式 port=0(测试/OS 分配)不受影响。 - * @returns {api, store, service, settings, maintenance, tokenExisted, dispose} + * embed — "local"(默认,自管 runtime 取件 + ONNX 嵌入)|"ollama"|"openai" + * (读宿主面板配的 vector-config)|"off"(纯关键词 + BM25)。 + * embedder/reranker — 注入式覆盖(测试/宿主方自管):给了就跳过 createSemantic, + * 直接 setEmbedder/setReranker;仅 embed="off" 之外有意为之。 + * @returns {api, store, service, settings, maintenance, semantic, tokenExisted, dispose} * tokenExisted — 启动前 kv 里是否已有 token;false 时本次为首次生成,入口层可提示。 */ -export function createServeRuntime({ memoryDir, port, host, logger = null, strictPort = true } = {}) { +export async function createServeRuntime({ + memoryDir, port, host, logger = null, strictPort = true, embed = "local", embedder = null, reranker = null +} = {}) { + if (!EMBED_PROVIDERS.has(embed)) { + throw new Error(`invalid embed provider: ${String(embed)}(允许 off/local/ollama/openai)`); + } // index.js:192-195:~ 展开只认前导;目录不存在时 node:sqlite 直接抛,先 mkdir。 const dir = String(memoryDir || join(homedir(), ".dsh", "memory")).replace(/^~(?=$|[\\/])/, homedir()); mkdirSync(dir, { recursive: true }); @@ -55,10 +70,46 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric // index.js:319-332:人改镜像先合并——镜像文件里的手工编辑每次启动都赢。 // readHumanEdits 全类型一次读齐:mergeHumanEdits 成功会重渲全部镜像,逐类型读改 // 循环会拿没读到的类型覆盖掉未合并的编辑(index.js:324-327 注释同款坑)。 + // 何时 apply 由 semantic 的分支时序契约决定(见 semantic.js 文件头),本文件只造闭包。 const humanEdits = new Map(); for (const type of Object.keys(TYPE_FILE)) humanEdits.set(type, mirror.readHumanEdits(type)); - for (const [type, edits] of humanEdits) { - if (edits.length) service.mergeHumanEdits(type, edits); + const applyHumanEdits = () => { + for (const [type, edits] of humanEdits) { + if (edits.length) service.mergeHumanEdits(type, edits); + } + }; + + // index.js:316-317:向量索引包住 store 的 embedding 列并跟踪活跃模型指纹。 + const vectorIndex = createVectorIndex({ store, logger }); + service.setVectorIndex(vectorIndex); + + // ---- 语义管线(PR2):三路二选一 ------------------------------------------ + let semantic = null; + if (embedder !== null) { + // 注入式覆盖(测试 / 宿主方自管 embedder):绕过装配,人改合并立刻做 + //(对齐宿主 openai 分支的同步语义)。 + service.setEmbedder(embedder); + if (reranker !== null) service.setReranker(reranker); + applyHumanEdits(); + } else if (embed === "off") { + // 纯关键词 + BM25:对齐宿主 lightMode 分支——不建 embedder,人改合并立即。 + applyHumanEdits(); + } else { + let semanticReady = true; + if (embed === "local") { + // 自管 runtime 缺失时取件:只走 download 档(宿主 adopt 推导对独立进程无意义 + // ——没有宿主 node_modules 可推)。失败不阻断 daemon:降级关键词 + 可操作提示。 + semanticReady = await ensureRuntimePayload(logger); + } + if (semanticReady) { + semantic = createSemantic({ + store, service, settings, + cfg: daemonSemanticCfg(embed), + logger, vectorIndex, applyHumanEdits, lightMode: false + }); + } else { + applyHumanEdits(); + } } // index.js:294-309:检索回执落 recall_runs(searchMemories 的 recordRecall 默认开, @@ -92,11 +143,59 @@ export function createServeRuntime({ memoryDir, port, host, logger = null, stric service, settings, maintenance, + semantic, tokenExisted, /** 收尾:先停收请求再关库。node:sqlite 对未 finalize 语句可能抛,吞掉——WAL 会在下次打开时回放,已提交事务不丢。 */ dispose() { try { api.server.close(); } catch { /* already closed */ } + try { semantic?.dispose(); } catch { /* 同上 */ } try { store.close(); } catch { /* 同上 */ } } }; } + +/** + * daemon 侧语义配置:默认值逐键锚定 src/config.js(309-420),daemon 不装 schemastery + * (宿主 peer 依赖),同值硬编码;config.js 改默认值时此处与 docs/DAEMON.md 同步。 + * 只放开 embedProvider(CLI --embed);rerank 暂不提供 CLI(默认关,与宿主同默认)。 + */ +function daemonSemanticCfg(embed) { + return { + embedProvider: embed, // config.js:309(宿主默认 openai;daemon 默认 local——独立部署唯一自洽路径) + localEmbedModel: "Xenova/bge-small-zh-v1.5", // :312 + localEmbedDimension: 512, // :313 + localEmbedDevice: "cpu", // :314 + localEmbedBatchSize: 8, // :315 + localEmbedPooling: "auto", // :323(BGE 系 CLS 口径,改它=换向量空间) + ollamaBaseUrl: "http://localhost:11434", // :326 + ollamaModel: "nomic-embed-text", // :331 + embedModelCacheDir: "", // :336(空=local-embedder 内部解析 ~/.dsh/mneme/models) + embedModelMirror: "https://hf-mirror.com", // :337 + resilientModelDownload: true, // :342 + runtimeDir: process.env.DSH_MNEME_RUNTIME_DIR || "", // :348(空=defaultRuntimeDir;取件/加载同源) + rerankEnabled: false, // :419 + rerankProvider: "none", // :420 + autoReindexOnBoot: true // :367 + }; +} + +/** + * 本地推理 runtime 取件(download 档)。返回 payload 是否可用: + * 已存在(含收编/早前下载)→ true 不重复取;缺失 → 取件成功 true / 失败 false(内部 + * 已打降级日志)。并发安全由 provisionRuntime 的 IN_FLIGHT 锁保证(按归一化 runtimeDir 串行)。 + */ +async function ensureRuntimePayload(logger) { + const runtimeDir = process.env.DSH_MNEME_RUNTIME_DIR || ""; + const dir = runtimeDir || defaultRuntimeDir(); + if (listPayloadDirs(dir).length > 0) return true; + logger?.info?.("[dsh-mneme] 本地推理运行时缺失,开始取件(download 档,约 200MB;离线可用 DSH_MNEME_RUNTIME_TARBALL_DIR / DSH_MNEME_RUNTIME_MIRROR)…"); + try { + const result = await provisionRuntime({ runtimeDir, localTarballDir: process.env.DSH_MNEME_RUNTIME_TARBALL_DIR || "", mirror: process.env.DSH_MNEME_RUNTIME_MIRROR || "" }); + if (!result.ok) throw new Error(result.reason); + logger?.info?.(`[dsh-mneme] 运行时就绪(${result.strategy}):${result.packages} 包 / ${result.files} 文件`); + return true; + } catch (error) { + logger?.warn?.(`[dsh-mneme] runtime provisioning failed, search degrades to keyword: ${String(error)}`); + return false; + } +} diff --git a/dsh-mneme/test/serve-bin.test.js b/dsh-mneme/test/serve-bin.test.js index 5435bce..0f8f21d 100644 --- a/dsh-mneme/test/serve-bin.test.js +++ b/dsh-mneme/test/serve-bin.test.js @@ -46,7 +46,8 @@ test("serve bin: spawn 冒烟;daemon 与宿主进程同库互写互读", { timeo const peer = createService({ store: seedStore, mirror: null, config: {} }); peer.saveWithDedupe({ type: "project", title: "peer 进程直写", content: "测试进程经 createStore 写入", importance: 3 }); - const child = spawn(process.execPath, [BIN, "--memory-dir", dir, "--port", "0"], { + // --embed off:CI 无 runtime payload,绝不能触发取件;多进程共存与语义无关 + const child = spawn(process.execPath, [BIN, "--memory-dir", dir, "--port", "0", "--embed", "off"], { stdio: ["ignore", "pipe", "pipe"] }); let stdout = ""; diff --git a/dsh-mneme/test/serve.test.js b/dsh-mneme/test/serve.test.js index d1206da..a9f29dc 100644 --- a/dsh-mneme/test/serve.test.js +++ b/dsh-mneme/test/serve.test.js @@ -8,14 +8,40 @@ import { createServeRuntime } from "../src/serve.js"; // daemon 装配面锁(test/serve-bin.test.js 另有真子进程 + 多进程共存): // in-process 起真 HTTP(port 0,OS 分配),fetch 走 health / 401 / save / search / // token 持久化复用全链路。锁的是「daemon 数据面 = api-standalone 同一工厂」这一契约。 +// 基础用例一律 embed:"off"——不碰取件与模型 init,CI 确定性;向量走注入假 embedder。 function tmpDir() { return mkdtempSync(join(tmpdir(), "mneme-serve-")); } +/** 确定性 4 维伪嵌入:同词相关、异词近正交,足以让向量路径产生非零命中。 */ +function fakeEmbedder() { + return { + ready: true, + modelHash: "fake-hash-1", + dimension: 4, + embedSingle: async (text) => { + const v = [0, 0, 0, 0]; + for (let i = 0; i < text.length; i++) v[i % 4] += text.charCodeAt(i) % 7; + const n = Math.hypot(v[0], v[1], v[2], v[3]) || 1; + return v.map((x) => x / n); + } + }; +} + +async function waitFor(fn, timeoutMs, what) { + const deadline = Date.now() + timeoutMs; + let lastErr; + while (Date.now() < deadline) { + try { return await fn(); } catch (err) { lastErr = err; } + await new Promise((r) => setTimeout(r, 100)); + } + throw new Error(`timeout waiting for ${what}: ${lastErr?.message ?? lastErr}`); +} + test("serve: /health 免鉴权,业务路由无 token 401", async () => { const dir = tmpDir(); - const rt = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt.api.ready; try { const base = `http://127.0.0.1:${rt.api.port}`; @@ -34,7 +60,7 @@ test("serve: /health 免鉴权,业务路由无 token 401", async () => { test("serve: save → search 走通;token 持久化 kv;二次启动复用同一 token;检索回执落 recall_runs", async () => { const dir = tmpDir(); - const rt = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt.api.ready; const base = `http://127.0.0.1:${rt.api.port}`; const auth = { authorization: `Bearer ${rt.api.token}` }; @@ -65,7 +91,7 @@ test("serve: save → search 走通;token 持久化 kv;二次启动复用同一 rt.dispose(); // 二次启动同目录:token 复用不重新生成(firstBoot 提示只在真正首次出现) - const rt2 = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt2 = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt2.api.ready; try { assert.equal(rt2.tokenExisted, true); @@ -78,7 +104,7 @@ test("serve: save → search 走通;token 持久化 kv;二次启动复用同一 test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)", async () => { const dir = tmpDir(); - const rt = createServeRuntime({ memoryDir: dir, port: 0 }); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt.api.ready; const port = rt.api.port; rt.dispose(); @@ -86,7 +112,7 @@ test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)" let rebound = null; for (let i = 0; i < 10 && !rebound; i++) { try { - const rt2 = createServeRuntime({ memoryDir: dir, port }); + const rt2 = await createServeRuntime({ memoryDir: dir, port, embed: "off" }); await rt2.api.ready; rebound = rt2; } catch (err) { @@ -102,3 +128,57 @@ test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)" try { rmSync(dir, { recursive: true, force: true }); } catch { /* 同上 */ } } }); + +test("serve: 注入假 embedder → 写入即嵌入,vector 轴接管检索;auto 检索可用", async () => { + const dir = tmpDir(); + const fake = fakeEmbedder(); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off", embedder: fake }); + await rt.api.ready; + try { + // 覆盖注入生效:semantic 被绕过,embedder 直接挂 service + assert.equal(rt.semantic, null); + + const base = `http://127.0.0.1:${rt.api.port}`; + const auth = { authorization: `Bearer ${rt.api.token}` }; + const save = await fetch(`${base}/memories`, { + method: "POST", + headers: { ...auth, "content-type": "application/json" }, + body: JSON.stringify({ type: "project", title: "向量注入冒烟", content: "daemon 侧写入即嵌入", importance: 3 }) + }); + assert.equal(save.status, 201); + + // 写入侧嵌入是异步排队的:轮询到 vector 轴命中为止。auto 模式不断言 mode 字段 —— + // 标题同词命中时融合合并对象是 keyword 行,上游 mneme 的 auto 报告语义如此 + // (vector:true 标记只在纯向量来源上保留),那是上游行为,不是 daemon 契约。 + await waitFor(async () => { + const res = await fetch(`${base}/search?q=${encodeURIComponent("向量注入冒烟")}&mode=vector`, { headers: auth }); + assert.equal(res.status, 200); + const body = await res.json(); + assert.equal(body.mode, "vector", "mode=vector must report the vector axis"); + assert.ok((body.items ?? []).some((m) => m.title === "向量注入冒烟")); + return true; + }, 8000, "vector-axis search"); + + const autoRes = await fetch(`${base}/search?q=${encodeURIComponent("向量注入冒烟")}&mode=auto`, { headers: auth }); + assert.equal(autoRes.status, 200); + assert.ok(((await autoRes.json()).items ?? []).some((m) => m.title === "向量注入冒烟")); + } finally { + rt.dispose(); + try { rmSync(dir, { recursive: true, force: true }); } catch { /* 同上 */ } + } +}); + +test("serve: 非法 embed provider 拒装配;embed=off 时 semantic 为 null", async () => { + const dir = tmpDir(); + await assert.rejects( + createServeRuntime({ memoryDir: dir, port: 0, embed: "bogus" }), + /invalid embed provider/ + ); + const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); + try { + assert.equal(rt.semantic, null, "off path must not assemble a semantic pipeline"); + } finally { + rt.dispose(); + try { rmSync(dir, { recursive: true, force: true }); } catch { /* 同上 */ } + } +}); From 9dbeb975ef877ae955d0516a6c0cb20889db803f Mon Sep 17 00:00:00 2001 From: modusensus Date: Mon, 5 Oct 2026 18:47:32 +0800 Subject: [PATCH 2/3] =?UTF-8?q?fix(serve):=20CodeRabbit=20on=20#364=20?= =?UTF-8?q?=E5=9B=9B=E6=9D=A1=E2=80=94=E2=80=94=E5=80=BC=E6=97=97=E6=A0=87?= =?UTF-8?q?=E7=BC=BA=E5=80=BC=E6=8A=A5=E9=94=99/dispose=20=E6=8E=92?= =?UTF-8?q?=E5=B9=B2=E5=9C=A8=E9=80=94=E5=86=8D=E5=85=B3=E5=BA=93/?= =?UTF-8?q?=E9=9D=9E=E5=9B=9E=E7=8E=AF=E6=98=8E=E6=96=87=20HTTP=20?= =?UTF-8?q?=E8=AD=A6=E5=91=8A/exports=20=E6=9A=B4=E9=9C=B2=20./serve?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 12 ++++++------ dsh-mneme/README.md | 7 ++++--- dsh-mneme/bin/dsh-mneme-serve.mjs | 14 +++++++++----- dsh-mneme/docs/DAEMON.md | 1 + dsh-mneme/lib/serve.js | 17 ++++++++++++++--- dsh-mneme/package.json | 3 +++ dsh-mneme/src/serve.js | 17 ++++++++++++++--- dsh-mneme/test/serve-bin.test.js | 15 +++++++++++++++ dsh-mneme/test/serve.test.js | 7 ++++--- 9 files changed, 70 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index e83a84e..753a9f2 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,9 @@ CI node <<<<<<< HEAD - tests + tests ======= - tests + tests >>>>>>> feat/serve-daemon coverage Awesome @@ -198,9 +198,9 @@ dsh-mneme-serve # 默认 ~/.dsh/memory + 127.0.0.1:8790 ```bash cd dsh-mneme && npm install <<<<<<< HEAD -npm test # 1536 个测试 +npm test # 1537 个测试 ======= -npm test # 1536 个测试 +npm test # 1537 个测试 >>>>>>> feat/serve-daemon npm run stress # 三轴线压测 npm run sync # src → lib 同步 @@ -395,9 +395,9 @@ dsh-mneme-serve # defaults: ~/.dsh/memory + 127.0.0.1:8790 ```bash cd dsh-mneme && npm install <<<<<<< HEAD -npm test # 1536 tests +npm test # 1537 tests ======= -npm test # 1536 tests +npm test # 1537 tests >>>>>>> feat/serve-daemon npm run stress # three-axis stress test npm run sync # src → lib sync diff --git a/dsh-mneme/README.md b/dsh-mneme/README.md index a2000ab..407a064 100644 --- a/dsh-mneme/README.md +++ b/dsh-mneme/README.md @@ -5,7 +5,7 @@ [![npm version](https://img.shields.io/npm/v/@modusensus/dsh-mneme?color=blue&label=npm)](https://www.npmjs.com/package/@modusensus/dsh-mneme) [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![Awesome](https://awesome-dsh-plugin.com/badge.svg)](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin) -[![tests](https://img.shields.io/badge/tests-1536%20passed-success)](https://github.com/slow-stack/mneme) +[![tests](https://img.shields.io/badge/tests-1537%20passed-success)](https://github.com/slow-stack/mneme) [![CI](https://img.shields.io/github/actions/workflow/status/slow-stack/mneme/ci.yml)](https://github.com/slow-stack/mneme/actions) [![node](https://img.shields.io/badge/node-22%2B-blue)](https://nodejs.org) [![npm downloads](https://img.shields.io/npm/d18m/@modusensus/dsh-mneme.svg?color=blue&label=downloads)](https://www.npmjs.com/package/@modusensus/dsh-mneme) @@ -526,6 +526,7 @@ dsh-mneme-serve --embed off # 纯关键词 + BM ``` - **鉴权与端口**：Bearer token 与 DSH 面板 / CLI 共用同一份（kv `external_api`，首次启动自动生成并持久化到 `memory.db`）；端口/主机解析链与「外部访问」一致（显式参数 > 持久值 > 默认 8790/127.0.0.1）。配置端口被占会**直接报错退出**（不做端口顺延）——第三方把 URL 写死，静默换端口等于坏。因此 **daemon 与 DSH 的「外部访问」二选一**，不要同端口同开。 +- **安全**：daemon 使用明文 HTTP，不提供原生 TLS。指定非回环 `--host` 时，请勿直接把服务暴露给不可信网络；远程访问请走 TLS 终止代理或 SSH 隧道。 - **能力边界（第一期，无 LLM）**：存储 / 检索（关键词 + BM25 + 向量）/ 镜像同步与人改合并 / `POST /maintenance/reclaim` / `/bootstrap` 全可用；巩固（autoDream）与蒸馏不在 daemon 内——巩固只属于 DSH 宿主进程，这是与宿主「单写者」的机械保证。`--embed` 默认 `local`（自管 runtime 与嵌入模型缺失时自动取件，约 200MB；失败降级关键词并打日志），也可选 `ollama` / `openai`（读宿主面板的 vector-config）/ `off`。 - **检索回执**：daemon 的 `/search` 同样落 `recall_runs`，第三方检索的复用统计不缺数。 - **生命周期**：stdout 仅就绪时打一行 `dsh-mneme-serve listening on http://host:port (pid N)`（供脚本解析实际端口），日志走 stderr；SIGINT/SIGTERM 优雅收库，Windows 强杀由 WAL 回放兜底。 @@ -600,7 +601,7 @@ src/ ├── api.js # HTTP 路由（Web 面板数据通道，含 /conflicts 冲突队列） └── index.js # 插件接线 lib/ # src 的同步分发产物（npm run sync；发布前由 root prepack 的 check-sync.js 校验一致性；唯一手写例外 lib/client.js——Web 面板 bundle，sync 不覆盖） -test/ # 1536 个 node:test 测试（审计与三轴线压测不变量；src↔lib 一致性由 scripts/check-sync.js 发布闸门校验） +test/ # 1537 个 node:test 测试（审计与三轴线压测不变量；src↔lib 一致性由 scripts/check-sync.js 发布闸门校验） scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压测 · sync-lib.js 同步 · check-sync.js 发布闸门 · benchmark-recall.js / benchmark-embed.js / benchmark-rerank.js 基准 · sync-test-badge.mjs 测试徽章 · build-runtime-manifest.mjs 运行时清单 ``` @@ -609,7 +610,7 @@ scripts/ # e2e-dsh.js 端到端演示 · stress-dsh.js 三轴线压 ```bash cd dsh-mneme npm install # 安装 peer 依赖（以 devDependencies 形式，用于本地测试） -npm test # 运行 1536 个测试 +npm test # 运行 1537 个测试 npm run stress # 三轴线压测：长会话检索 / 冲突仲裁 / 多 Agent 并发（离线 mock LLM） npm run sync # 把 src/ 同步到 lib/（发布时由 prepack 钩子自动执行） ``` diff --git a/dsh-mneme/bin/dsh-mneme-serve.mjs b/dsh-mneme/bin/dsh-mneme-serve.mjs index 7988f52..8281376 100644 --- a/dsh-mneme/bin/dsh-mneme-serve.mjs +++ b/dsh-mneme/bin/dsh-mneme-serve.mjs @@ -57,8 +57,11 @@ function parseArgv(argv) { if (eq > -1) out[a.slice(2, eq)] = a.slice(eq + 1); else { const v = argv[i + 1]; - if (v !== undefined && !v.startsWith("--")) { out[a.slice(2)] = v; i++; } - else out[a.slice(2)] = true; + // 本 bin 的旗标全部取值（无泛用布尔旗标）：值缺失或下一个 token 是旗标 + // 都按「缺值」报错退出——不能落成 true（Number(true)=1 会把 --port 变成 + // 绑端口 1，报 EACCES 让用户查错方向）。帮助/版本在上方分支已提前返回。 + if (v !== undefined && !v.startsWith("-")) { out[a.slice(2)] = v; i++; } + else fail(`${a} 缺少参数值`); } } else { fail(`未知参数: ${a}\n运行 \`${BIN_NAME} --help\` 查看用法。`); @@ -130,11 +133,12 @@ async function main(argv) { } let closing = false; - const shutdown = (signal) => { - if (closing) return; + const shutdown = async (signal) => { + // 第二次信号 = 强制退出（dispose 里 server.close 等 in-flight 收尾，极端情况下会挂） + if (closing) process.exit(0); closing = true; console.error(`[dsh-mneme] ${signal} received, closing...`); - try { rt.dispose(); } catch { /* dispose 各步自吞 */ } + try { await rt.dispose(); } catch { /* dispose 各步自吞 */ } process.exit(0); }; process.on("SIGINT", () => shutdown("SIGINT")); diff --git a/dsh-mneme/docs/DAEMON.md b/dsh-mneme/docs/DAEMON.md index 3fe8b90..dd22bc2 100644 --- a/dsh-mneme/docs/DAEMON.md +++ b/dsh-mneme/docs/DAEMON.md @@ -25,6 +25,7 @@ dsh-mneme-serve [--memory-dir ] [--port ] [--host ] [--embed kv `external_api` 持久值 > 默认 8790 / 127.0.0.1。 - token 与 DSH 面板 / CLI **共用同一份**(kv `external_api`,首次启动自动生成并持久化)——三方零配置互通。 +- 安全:daemon 使用明文 HTTP,不提供原生 TLS。指定非回环 `--host` 时,请勿直接把服务暴露给不可信网络;远程访问请走 TLS 终止代理或 SSH 隧道。 - stdout 只在就绪时打一行 `dsh-mneme-serve listening on http://host:port (pid N)`(机器可读,脚本/测试解析端口用);日志全走 stderr。 - SIGINT/SIGTERM 优雅收库后 exit 0;Windows 强杀由 WAL 回放兜底。 diff --git a/dsh-mneme/lib/serve.js b/dsh-mneme/lib/serve.js index b7f87fe..ede6284 100644 --- a/dsh-mneme/lib/serve.js +++ b/dsh-mneme/lib/serve.js @@ -145,9 +145,20 @@ export async function createServeRuntime({ maintenance, semantic, tokenExisted, - /** 收尾:先停收请求再关库。node:sqlite 对未 finalize 语句可能抛,吞掉——WAL 会在下次打开时回放,已提交事务不丢。 */ - dispose() { - try { api.server.close(); } catch { /* already closed */ } + /** + * 收尾：先停收新请求、等在途请求跑完，再关库——直接同步关库会让在途的 + * PUT/POST 撞上已关的 store（500 或丢写）。closeIdleConnections 排干 + * keep-alive 空闲连接（node ≥18.2，旧版无此 API 则跳过），否则 server.close + * 的回调要等 keep-alive 超时才触发。node:sqlite 对未 finalize 语句可能抛， + * 吞掉——WAL 会在下次打开时回放，已提交事务不丢。 + */ + async dispose() { + await new Promise((resolve) => { + try { + api.server.close(() => resolve()); + api.server.closeIdleConnections?.(); + } catch { resolve(); } + }); try { semantic?.dispose(); } catch { /* 同上 */ } try { store.close(); } catch { /* 同上 */ } } diff --git a/dsh-mneme/package.json b/dsh-mneme/package.json index a1da00d..6502c46 100644 --- a/dsh-mneme/package.json +++ b/dsh-mneme/package.json @@ -23,6 +23,9 @@ ".": { "default": "./lib/index.js" }, + "./serve": { + "default": "./lib/serve.js" + }, "./client": { "default": "./lib/client.js" }, diff --git a/dsh-mneme/src/serve.js b/dsh-mneme/src/serve.js index b7f87fe..ede6284 100644 --- a/dsh-mneme/src/serve.js +++ b/dsh-mneme/src/serve.js @@ -145,9 +145,20 @@ export async function createServeRuntime({ maintenance, semantic, tokenExisted, - /** 收尾:先停收请求再关库。node:sqlite 对未 finalize 语句可能抛,吞掉——WAL 会在下次打开时回放,已提交事务不丢。 */ - dispose() { - try { api.server.close(); } catch { /* already closed */ } + /** + * 收尾：先停收新请求、等在途请求跑完，再关库——直接同步关库会让在途的 + * PUT/POST 撞上已关的 store（500 或丢写）。closeIdleConnections 排干 + * keep-alive 空闲连接（node ≥18.2，旧版无此 API 则跳过），否则 server.close + * 的回调要等 keep-alive 超时才触发。node:sqlite 对未 finalize 语句可能抛， + * 吞掉——WAL 会在下次打开时回放，已提交事务不丢。 + */ + async dispose() { + await new Promise((resolve) => { + try { + api.server.close(() => resolve()); + api.server.closeIdleConnections?.(); + } catch { resolve(); } + }); try { semantic?.dispose(); } catch { /* 同上 */ } try { store.close(); } catch { /* 同上 */ } } diff --git a/dsh-mneme/test/serve-bin.test.js b/dsh-mneme/test/serve-bin.test.js index 0f8f21d..0143297 100644 --- a/dsh-mneme/test/serve-bin.test.js +++ b/dsh-mneme/test/serve-bin.test.js @@ -34,6 +34,21 @@ async function waitFor(fn, timeoutMs, what) { throw new Error(`timeout waiting for ${what}: ${lastErr?.message ?? lastErr}`); } +test("serve bin: 值旗标缺值直接报错退出(--port 后无值不能落成 Number(true)=1)", { timeout: 30000 }, async () => { + // CodeRabbit on #364:--port 紧跟另一个旗标或结束时,旧解析把它存成 true, + // Number(true)=1 通过校验 → 静默改绑端口 1(EACCES 误导排错方向)。 + const child = spawn(process.execPath, [BIN, "--memory-dir", mkdtempSync(join(tmpdir(), "mneme-serve-arg-")), "--port"], { + stdio: ["ignore", "pipe", "pipe"] + }); + let stderr = ""; + child.stderr.on("data", (d) => { stderr += d; }); + const [code] = await new Promise((resolve) => { + child.on("exit", (c) => resolve([c])); + }); + assert.notEqual(code, 0, "missing value must exit non-zero"); + assert.ok(stderr.includes("缺少参数值"), `stderr should name the missing value, got: ${stderr.slice(-200)}`); +}); + test("serve bin: spawn 冒烟;daemon 与宿主进程同库互写互读", { timeout: 120000 }, async () => { const dir = mkdtempSync(join(tmpdir(), "mneme-serve-bin-")); diff --git a/dsh-mneme/test/serve.test.js b/dsh-mneme/test/serve.test.js index a9f29dc..edfb47f 100644 --- a/dsh-mneme/test/serve.test.js +++ b/dsh-mneme/test/serve.test.js @@ -107,8 +107,9 @@ test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)" const rt = await createServeRuntime({ memoryDir: dir, port: 0, embed: "off" }); await rt.api.ready; const port = rt.api.port; - rt.dispose(); - // dispose 同步关库;端口释放可能有内核级迟滞,重试绑定而不是假设立即可用 + // dispose 现为 async:等 server.close 回调(在途请求排干)后再重绑 + await rt.dispose(); + // 端口释放可能有内核级迟滞,重试绑定而不是假设立即可用 let rebound = null; for (let i = 0; i < 10 && !rebound; i++) { try { @@ -124,7 +125,7 @@ test("serve: dispose 后端口可复用(同端口连起两轮不踩 strictPort)" assert.ok(rebound, "port should be rebindable after dispose"); assert.equal(rebound.api.port, port); } finally { - rebound?.dispose(); + await rebound?.dispose(); try { rmSync(dir, { recursive: true, force: true }); } catch { /* 同上 */ } } }); From 32d3d06c9cc931047d2f09e31797066569011e92 Mon Sep 17 00:00:00 2001 From: modusensus Date: Mon, 5 Oct 2026 19:27:18 +0800 Subject: [PATCH 3/3] =?UTF-8?q?fix(semantic):=20=E6=B6=88=E5=8C=96=20CHANG?= =?UTF-8?q?ELOG=20=E6=AE=8B=E7=95=99=E5=86=B2=E7=AA=81=E6=A0=87=E8=AE=B0(?= =?UTF-8?q?=E4=B8=A4=E4=B8=AA=20daemon=20=E6=9D=A1=E7=9B=AE=E5=B9=B6?= =?UTF-8?q?=E6=AE=B5,=E5=8E=BB=E6=8E=89=E8=BF=87=E6=97=B6=E6=A3=80?= =?UTF-8?q?=E7=B4=A2=E5=8F=A5)+=20=E5=B5=8C=E5=A5=97=E5=9B=9E=E5=A1=AB?= =?UTF-8?q?=E9=87=8D=E8=AF=95=E8=AE=A1=E6=97=B6=E5=99=A8=E7=BA=B3=E5=85=A5?= =?UTF-8?q?=20dispose(CodeRabbit=20on=20#365)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- dsh-mneme/CHANGELOG.md | 5 +---- dsh-mneme/lib/semantic.js | 4 +++- dsh-mneme/src/semantic.js | 4 +++- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/dsh-mneme/CHANGELOG.md b/dsh-mneme/CHANGELOG.md index 5d99cab..c9e8f95 100644 --- a/dsh-mneme/CHANGELOG.md +++ b/dsh-mneme/CHANGELOG.md @@ -4,11 +4,9 @@ ## 🆕 新增 -<<<<<<< HEAD - **daemon 向量检索（PR2，#363）**：`dsh-mneme-serve` 的 `/search` 接入完整语义管线——embedder/reranker 装配与 boot 自动回填从 `index.js` **纯搬移**至 `src/semantic.js`（宿主与 daemon 共用同一份，调用时序契约原样；`backfillMissingEmbeddings` 经 index.js barrel 再出口，测试调用方零改动），daemon 侧新增 `createVectorIndex` 接线与 `--embed` 参数：`local`（默认，自管 runtime/嵌入模型缺失时经 `provisionRuntime` download 档自动取件，可用 `DSH_MNEME_RUNTIME_TARBALL_DIR`/`DSH_MNEME_RUNTIME_MIRROR` 换离线/镜像来源；失败降级关键词并打可操作日志）、`ollama`、`openai`（读宿主面板 vector-config）、`off`。向量轴有注入假 embedder 的回归锁；`createServeRuntime` 因此转为 async、语义键默认值在 `daemonSemanticCfg` 逐键锚定 config.js。 - **独立服务 daemon（`dsh-mneme-serve`，#363）**：mneme 现在能在 DSH 宿主之外常驻——`src/serve.js` 的 `createServeRuntime` 用最小装配（store → settings → mirror → service → maintenance → standalone API，每步锚定 index.js 装配行号）把数据面跑成独立进程，第三方集成（网页端桥接等）不必为挂载记忆库而保持 DSH 开机。第一期刻意无 LLM：巩固（autoDream）与蒸馏结构性不在 daemon 内，这是与宿主「单写者」的机械保证，不靠用户自觉。token 与 DSH 面板/CLI 共用同一 kv 凭证，端口/主机解析链与外部访问一致；`createStandaloneApi` 新增 `strictPort` 选项——daemon 的配置端口被占即报错退出而非顺延（第三方把 URL 写死，静默换端口等于坏），不传该选项的宿主旁路行为不变。`/search` 照常落 recall_runs，第三方检索的复用统计不缺数。多进程共存（daemon 与宿主同库互写互读）有专门回归锁；已知限制（双进程去重竞态、镜像双写、版本偏斜）见 docs/DAEMON.md。 -======= -- **独立服务 daemon（`dsh-mneme-serve`，#363）**：mneme 现在能在 DSH 宿主之外常驻——`src/serve.js` 的 `createServeRuntime` 用最小装配（store → settings → mirror → service → maintenance → standalone API，每步锚定 index.js 装配行号）把数据面跑成独立进程，第三方集成（网页端桥接等）不必为挂载记忆库而保持 DSH 开机。第一期刻意无 LLM：巩固（autoDream）与蒸馏结构性不在 daemon 内，这是与宿主「单写者」的机械保证，不靠用户自觉；检索为关键词 + BM25（向量由后续 PR 抽取 semantic 装配后接入）。token 与 DSH 面板/CLI 共用同一 kv 凭证，端口/主机解析链与外部访问一致；`createStandaloneApi` 新增 `strictPort` 选项——daemon 的配置端口被占即报错退出而非顺延（第三方把 URL 写死，静默换端口等于坏），不传该选项的宿主旁路行为不变。`/search` 照常落 recall_runs，第三方检索的复用统计不缺数。多进程共存（daemon 与宿主同库互写互读）有专门回归锁；已知限制（双进程去重竞态、镜像双写、版本偏斜）见 docs/DAEMON.md。 + ## 🧹 工程 - **发布准备脚本在 CRLF 检出上不再假成功（`scripts/release-prep.mjs`）**：该脚本用 `/^(# Changelog\n\n)/` 匹配 CHANGELOG 文件头，而 Windows 检出是 CRLF——正则命中不了，`replace` 退化成空操作，**脚本却照样打印 `✓ … 占位节`**，`git status` 里看不出任何异常（CI 跑在 ubuntu 是 LF，所以只有本机发版会中招，v0.8.13 那次即如此、最后靠人工补的占位节）。规则抽成 `dsh-mneme/scripts/changelog-prep.mjs` 的纯函数：行尾两种都吃、插入内容跟随原文件行尾、带 BOM 也认；匹配不上则如实回报 `header-not-found`，入口**报错退出（exit 1）**而不是假打印成功。配 6 条回归测试（LF / CRLF / BOM / 幂等 / 回报契约 / detectEol）。 @@ -32,7 +30,6 @@ ### 贡献者 / Thanks - **@heptaspirit** — 写入边界的密钥 / PII 判据（PR #354）与 `memory_save` 的 scope 声明指引（PR #355）。 ->>>>>>> feat/serve-daemon ## [0.8.12] - 2026-10-01 diff --git a/dsh-mneme/lib/semantic.js b/dsh-mneme/lib/semantic.js index f2732ea..a236aa1 100644 --- a/dsh-mneme/lib/semantic.js +++ b/dsh-mneme/lib/semantic.js @@ -197,7 +197,9 @@ export function createSemantic({ store, service, settings, cfg, logger, vectorIn if ("ready" in embedder && embedder.ready !== true) { // Local/ollama embedders init asynchronously; give them a moment // before giving up on this boot (next boot retries). - if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); + // CodeRabbit on #365:嵌套重试计时器同样入册,否则 dispose 后仍可能 + // 对已关库跑 needsEmbedding(有 try/catch 兜底只是日志噪声,但状态要收干净)。 + if (tries > 0) reindexTimer = setTimeout(() => attempt(tries - 1), 2000); return; } if (!store.needsEmbedding(1).length) return; // nothing to backfill diff --git a/dsh-mneme/src/semantic.js b/dsh-mneme/src/semantic.js index f2732ea..a236aa1 100644 --- a/dsh-mneme/src/semantic.js +++ b/dsh-mneme/src/semantic.js @@ -197,7 +197,9 @@ export function createSemantic({ store, service, settings, cfg, logger, vectorIn if ("ready" in embedder && embedder.ready !== true) { // Local/ollama embedders init asynchronously; give them a moment // before giving up on this boot (next boot retries). - if (tries > 0) setTimeout(() => attempt(tries - 1), 2000); + // CodeRabbit on #365:嵌套重试计时器同样入册,否则 dispose 后仍可能 + // 对已关库跑 needsEmbedding(有 try/catch 兜底只是日志噪声,但状态要收干净)。 + if (tries > 0) reindexTimer = setTimeout(() => attempt(tries - 1), 2000); return; } if (!store.needsEmbedding(1).length) return; // nothing to backfill