Skip to content

feat(app): Implement intelligent text chunking via external service - #7706

Merged
c121914yu merged 7 commits into
labring:mainfrom
sunjiaqi52777:feat/intelligent-chunking
Sep 12, 2026
Merged

feat(app): Implement intelligent text chunking via external service#7706
c121914yu merged 7 commits into
labring:mainfrom
sunjiaqi52777:feat/intelligent-chunking

Conversation

@sunjiaqi52777

@sunjiaqi52777 sunjiaqi52777 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

功能
为知识库文本切分提供可选的外部智能分块服务对接能力。默认完全关闭,未配置服务地址的平台行为零变化。

流程:
导入文档

├─ chunkSettingMode ≠ intelligent(默认 custom/auto)
│ → 原有本地分块逻辑,行为零变化

└─ chunkSettingMode = intelligent 且训练类型为 chunk
│(UI 侧:未配置 SXF_CHUNK_URL 时 show_intelligent_chunking=false,
│ 「智能分块」入口直接隐藏,无法误选)

├─ 文本未达触发阈值(chunkTriggerMinSize,默认 1000)→ 整段入库,不请求外部服务
├─ 未配置 SXF_CHUNK_URL → 显式报错进解析失败记录,不静默回退本地分块

└─ 调用 sxf 智能分块服务
POST {markdown, chunk_sizes?} → { chunks: [{text, ...}], count }
(鉴权:可选 Authorization: Bearer ;平台只消费 chunks[].text)
├─ 成功 → chunks[].text 映射为数据块(q=text, a='')→ 向量化入库
└─ 失败/格式异常/空结果 → i18n 错误码进解析失败记录

**改动: **

  • 新增环境变量 SXF_CHUNK_URL / SXF_CHUNK_KEY / SXF_CHUNK_TIMEOUT(单位分钟,默认 60,与定制版 chunkTimeout 对齐):URL 默认空 = 功能关闭,地址非法启动报错
  • 新增第三方服务客户端 chunkBySxf(packages/service/thirdProvider/sxf,与现有 thirdProvider结构一致),rawText2Chunks 按 trainingType=chunk && chunkSettingMode=intelligent 路由;未配置/失败/异常均抛错,不静默回退
  • 分块模式重构:智能分块建模为独立的 ChunkSettingModeEnum.intelligent(复用 auto 参数档:chunkSize作服务的 chunk_sizes.text,indexSize 用于建索引);chunkSettingMode 默认保持/回落custom,确保未配置用户不会被路由到外部服务
  • 3 个新错误码(externalChunkFailed / externalChunkInvalidResponse /externalChunkNotConfigured,i18n 四语言:zh-CN / zh-Hant / en / ko-KR)
  • systemTypes 暴露 show_intelligent_chunking(由 SXF_CHUNK_URL 派生),UI 据此隐藏「智能分块」入口
  • .env.template 与测试同步补充(sxf 客户端 / readExternalRouting / env 校验 / preview / training utils)

@cla-assistant

cla-assistant Bot commented Sep 10, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

Build and publish successful - Preview fastgpt Image:

ghcr.io/labring/fastgpt-pr:fastgpt_8895f49f587d32be0e7b38e8c330e96afe27e141

🕒 Time: 2026-09-12 19:39:17 (UTC+8)

Comment thread projects/app/src/pageComponents/dataset/detail/Import/Context.tsx Outdated
Comment thread packages/service/thirdProvider/sangfor/chunk.ts
Comment thread packages/web/i18n/zh-Hant/common.json
Comment thread projects/app/src/pageComponents/dataset/detail/Import/Context.tsx Outdated
Comment thread packages/service/core/dataset/read.ts Outdated
Comment thread packages/service/core/dataset/read.ts Outdated
@sunjiaqi52777
sunjiaqi52777 force-pushed the feat/intelligent-chunking branch 5 times, most recently from ce08519 to 451fcaa Compare September 11, 2026 03:05

@c121914yu c121914yu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参考刚合入的文件解析,环境变量专门设置 sxf 的接口,也不用做通用的了。

Comment thread packages/global/core/dataset/training/utils.ts Outdated
Comment thread packages/service/thirdProvider/sangfor/chunk.ts
Comment thread packages/service/env.ts Outdated
@sunjiaqi52777
sunjiaqi52777 force-pushed the feat/intelligent-chunking branch 2 times, most recently from ac12c19 to a044b15 Compare September 11, 2026 08:12
Comment thread packages/service/thirdProvider/sangfor/chunk.ts
@sunjiaqi52777
sunjiaqi52777 force-pushed the feat/intelligent-chunking branch 3 times, most recently from 2f0e792 to ffd1ac7 Compare September 12, 2026 08:34
ctlaltlaltc
ctlaltlaltc previously approved these changes Sep 12, 2026
sunjiaqi52777 and others added 3 commits September 12, 2026 18:23
CHUNKING_SERVICE_URL 配置后, chunkSettingMode=auto(智能分块) 的文档导入/解析
把「文本→chunk」委托给自定义分块服务(POST {markdown} -> chunks[]), 自定义分块
及其余切分消费方(预览/QA/backup/template)保持本地逻辑。新增 feConfigs 开关
show_intelligent_chunking 供 UI 在未配置服务时隐藏「智能分块」入口。
Follow the sangfor file-parse pattern: keep both sangfor integrations
(file parse + intelligent chunking) under thirdProvider/sangfor with
unified naming instead of a separate sxf provider.

- env: rename CHUNKING_SERVICE_URL/KEY/TIMEOUT to SANGFOR_CHUNK_URL/KEY/TIMEOUT; validate URL with UrlSchema directly so an invalid value now fails startup validation instead of degrading to unset with a warning
- thirdProvider/sangfor: add chunk.ts with chunkByIultmzh; keep the DatasetErrEnum.externalChunk* codes so parse failures stay translatable
- training/utils: drop the intelligent parameter fallback in computedCollectionChunkSettings - the UI sends the full auto parameter profile (chunkSize/indexSize) when selecting intelligent chunking
- read.ts: route through chunkByIultmzh with the renamed env keys
- sync .env.template, i18n error copy, show_intelligent_chunking feConfig and tests (sangforChunk.test.ts, readExternalRouting, env)
@c121914yu
c121914yu force-pushed the feat/intelligent-chunking branch from 57ac778 to a2c3b8a Compare September 12, 2026 10:24
@c121914yu
c121914yu merged commit 519e503 into labring:main Sep 12, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants