feat(app): Implement intelligent text chunking via external service - #7706
Merged
Conversation
|
✅ Build and publish successful - Preview fastgpt Image: 🕒 Time: 2026-09-12 19:39:17 (UTC+8) |
ctlaltlaltc
reviewed
Sep 10, 2026
c121914yu
reviewed
Sep 10, 2026
sunjiaqi52777
force-pushed
the
feat/intelligent-chunking
branch
5 times, most recently
from
September 11, 2026 03:05
ce08519 to
451fcaa
Compare
c121914yu
reviewed
Sep 11, 2026
c121914yu
left a comment
Collaborator
There was a problem hiding this comment.
参考刚合入的文件解析,环境变量专门设置 sxf 的接口,也不用做通用的了。
sunjiaqi52777
force-pushed
the
feat/intelligent-chunking
branch
2 times, most recently
from
September 11, 2026 08:12
ac12c19 to
a044b15
Compare
ctlaltlaltc
reviewed
Sep 12, 2026
sunjiaqi52777
force-pushed
the
feat/intelligent-chunking
branch
3 times, most recently
from
September 12, 2026 08:34
2f0e792 to
ffd1ac7
Compare
ctlaltlaltc
approved these changes
Sep 12, 2026
ctlaltlaltc
previously approved these changes
Sep 12, 2026
sunjiaqi52777
force-pushed
the
feat/intelligent-chunking
branch
from
September 12, 2026 09:13
ffd1ac7 to
4d2fb8c
Compare
CHUNKING_SERVICE_URL 配置后, chunkSettingMode=auto(智能分块) 的文档导入/解析
把「文本→chunk」委托给自定义分块服务(POST {markdown} -> chunks[]), 自定义分块
及其余切分消费方(预览/QA/backup/template)保持本地逻辑。新增 feConfigs 开关
show_intelligent_chunking 供 UI 在未配置服务时隐藏「智能分块」入口。
Follow the sangfor file-parse pattern: keep both sangfor integrations (file parse + intelligent chunking) under thirdProvider/sangfor with unified naming instead of a separate sxf provider. - env: rename CHUNKING_SERVICE_URL/KEY/TIMEOUT to SANGFOR_CHUNK_URL/KEY/TIMEOUT; validate URL with UrlSchema directly so an invalid value now fails startup validation instead of degrading to unset with a warning - thirdProvider/sangfor: add chunk.ts with chunkByIultmzh; keep the DatasetErrEnum.externalChunk* codes so parse failures stay translatable - training/utils: drop the intelligent parameter fallback in computedCollectionChunkSettings - the UI sends the full auto parameter profile (chunkSize/indexSize) when selecting intelligent chunking - read.ts: route through chunkByIultmzh with the renamed env keys - sync .env.template, i18n error copy, show_intelligent_chunking feConfig and tests (sangforChunk.test.ts, readExternalRouting, env)
c121914yu
force-pushed
the
feat/intelligent-chunking
branch
from
September 12, 2026 10:24
57ac778 to
a2c3b8a
Compare
c121914yu
approved these changes
Sep 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
功能:
为知识库文本切分提供可选的外部智能分块服务对接能力。默认完全关闭,未配置服务地址的平台行为零变化。
流程:
导入文档
│
├─ chunkSettingMode ≠ intelligent(默认 custom/auto)
│ → 原有本地分块逻辑,行为零变化
│
└─ chunkSettingMode = intelligent 且训练类型为 chunk
│(UI 侧:未配置 SXF_CHUNK_URL 时 show_intelligent_chunking=false,
│ 「智能分块」入口直接隐藏,无法误选)
│
├─ 文本未达触发阈值(chunkTriggerMinSize,默认 1000)→ 整段入库,不请求外部服务
├─ 未配置 SXF_CHUNK_URL → 显式报错进解析失败记录,不静默回退本地分块
│
└─ 调用 sxf 智能分块服务
POST {markdown, chunk_sizes?} → { chunks: [{text, ...}], count }
(鉴权:可选 Authorization: Bearer ;平台只消费 chunks[].text)
├─ 成功 → chunks[].text 映射为数据块(q=text, a='')→ 向量化入库
└─ 失败/格式异常/空结果 → i18n 错误码进解析失败记录
**改动: **