Feature/recognition optimize - #105
Merged
Merged
Conversation
识别后处理 (base.py):
- 分级置信度阈值:低于阈值保留原标签并标记 needs_review,不再硬砍 body_text
- 章节状态机置信度门控 (STATE_MIN_CONF=0.6):低置信伪标题不触发内容传播
- 附录区中性化:可信 heading_fulu 之后把摘要/关键词/致谢类回退 body_text
- 摘要页标题回补、目录识别、参考文献/致谢正文序列修正(均复用置信度门控,
避免把内联源码的孤立 "{" 误判为致谢标题后扫成 acknowledgements_content)
节点对齐 (structure/settings.py):
- CATEGORY_TO_CLASS 补齐 heading_mulu/heading_fulu/document_title/footer,
修复 create_node 丢弃节点导致的段落与文档树错位
推理与流水线:
- onnx_infer.py 批量推理、settings.py BATCH_SIZE、stages.py 段落对齐
工具与测试:
- scripts/eval_recognition.py:零依赖识别准确率评估(classification_report + 混淆矩阵)
- 新增后处理与门控单测;1054 passed,覆盖率 87.82%
实测:真实论文准确率 98.6%→100%,源码/内联代码附录零污染。
- 新增训练流水线(dump/fix/train 三脚本,编号增强+小类上采样) - onnx_infer 单模型化:model_info.json 驱动、[PREV] 上下文链 - 22 类标签表与 tokenizer 切换为 chinese_roberta_L-4_H-256 - 推理参数调优:BATCH_SIZE 32 + intra 2/inter 1(基准验证) - 环境变量文档化,.gitignore 排除训练数据与模型大文件
- thesis_paragraph_classifier_int8.onnx(8.6MB) 直接提交仓库分发,不再走 Release - 新增摘要内容位置门控 _gate_abstract_content(修复绪论被误吸) - 训练集/测试集/训练脚本/checkpoint 收拢到 training/,不入库 - 补充合成数据补齐训练报告
生产仅保留 int8 ONNX(8.6MB) + tokenizer + 标签表;fp32 可由 training/checkpoints/model_small_bert 重新导出恢复
- 前端 SCORE_THRESHOLD 0.8(旧BERT时代) 改为 22 类分级阈值表,与后端统一 - 后端 CONF_THRESHOLDS 补全 22 类:以正确段 p10 分档 0.35~0.6 - 留出集实测:ui 标记率 44.9% -> 2.06%,正确段误标仅 1.64%
- node-meta 加 max-width 30% + ellipsis,hover 显示完整注释 - node-content 改 flex 1 1 0,保证收缩责任落在注释上 - 消除超长注释(如位置门控说明)把正文压成单列瘦条的问题
- flex-wrap: wrap + node-meta width:100%,注释不再与正文争行内宽度 - 无注释节点不再渲染 meta 行(去掉'无注释'占位) - 上次只 build 到 dist 未同步 api/static,导致后端 serve 的仍是旧版
- utils.js abstract.english.keywords.rules 补齐 trailing_punctuation(中文侧已有) - AbstractConfig.vue 深层访问 config.english.keywords.rules.trailing_punctuation.enabled 不再 undefined - 同步 api/static 构建产物
- 前端 DocTagChecker:移除 figure_image 数据层过滤(改为显示层隐藏), 保证节点与文档段落 1:1 对应,杜绝 keywords_chinese/caption_figure/ heading_level_* 规则错位套用(正文被加"图1.1"编号、中文关键词批注 出现在论文题目等);统一过滤/原始索引换算,修正选中与详情面板 - 后端 ParagraphAlignmentStage:zip 前校验节点数与段落数一致, 不一致显式报错并给出可操作指引,不再静默错位 - 后端新增 _fix_references_content 位置门控:参考文献标题前的 references_content(正文综述段含 [n] 引用标记被误判)回退 body_text - 前端格式化前提醒低置信(needs_review)节点数量,防止误格式化 - base.py 补 score 重置与标题编号正则校准(沿用既有后处理链) - 测试:新增 alignment_guard 防错位、references 位置门控用例, 修正 mock 树与段落数不匹配的旧用例,全量 1073 通过 - 同步前端构建产物至 api/static
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 关联 Issue
🎯 变更概述
受影响的用户场景
具体变更点
迁移建议
是否提供自动迁移工具?
scripts/migrate_v02_to_v03.py)✅ 提交前自查清单
🔹 测试保障
make test(或等效命令)tests/目录)io.BytesIO,未保存任何临时.docx文件到磁盘🔹 配置与兼容性
example/中提供示例requirements.txt或pyproject.toml已更新)🔹 文档与示例
README.md已更新(如新增功能说明、命令参数)example/)能正常运行且反映最新能力python-docx底层操作的部分)🔹 安全与资源
pickle、yaml.load未指定Loader)🔹 其他
master切出,无无关提交feat: add table caption check)