Repository navigation
perf(lark,native_chat): eliminate redundant full-directory JSON reparsing in steward chat loops - #9041
Conversation
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.2-sol · OpenAI (runtime_reported, max)
Request changes conclusion (author-owned PR; GitHub blocks formal self-review)
[P2] 长驻请求缓存保留已投递的图片正文
这里用完整 row 做长期缓存投影,recover 扫过已投递历史就会连图片 data_url 一起持有。24 份有效合成图片的相同扫描,候选返回并回收临时对象后仍保留约 45.2 MiB,基线约 1.5 KiB。切换候选已有的禁用复用分支后,JSON decoder 保留字节归零。 相关代码
动机 / Motivation
管家私聊和 Native Chat 恢复会周期性扫描持久化请求。历史中包含已投递的文字和图片。候选停止重复解析未改动文件,但通用发现缓存从保留元数据扩展为持有完整记录。同样的热扫描确实更快。需要同时控制扫描结束后长驻进程保留的附件字节。验证范围是原生恢复、Lark 查找和缓存边界。正式平台的完整负载与长期成本留有证据缺口。
改动思路 / Approach
沿已有 Python Native Chat 和 Lark owner。候选让 owner attention 逐个读取,并把 private ingress 和 native pending 接到现有文件指纹缓存。对三个真实调用入口比较早停、文件变化和原始 payload 保真。全行对象进入缓存时也带入附件。用同一批已投递请求跑两个提交。独立采样热扫描耗时,再测函数返回、垃圾回收之后仍活跃的 Python 分配。仅切换候选既有的禁用复用分支验证因果。
具体改动 / Changes
精确 head 4510212271f8012151105a7c8fae3a22444ad08a,diff base 1c38823e59a55bbd4d6906fa23e5ba84375c71a1,4 个文件 +281/-9。逐项复核 skip-unused-work、strict-capture-read、bounded-recovery-allocation。验收约定
关键代码讲解
- owner attention 按原先的候选顺序逐个读取,并在有效路径返回后停止。2,000 条样本验证了早停和删除回退。源码
- private ingress 对必需字段做严格验证,缺失或损坏仍进入原失败路径。缓存复制隔离 caller 的对象修改。源码
- pending 把整个请求 row 作为缓存投影。恢复扫描虽会跳过已投递请求,缓存仍持有图片 data_url。源码
- FileChatDiscovery 的原用途是保留派生字段。对照切换已有的 no-reuse 分支后,JSON decoder 保留量归零。源码
对主干的风险 / Risks
24 份已投递的合成图片请求,候选在 recover 返回并回收临时对象后仍持有约 45.2 MiB,基线约 1.5 KiB。只切换候选已有的禁用复用分支,JSON decoder 保留字节降到零。删除文件并重扫也释放了缓存。该隔离负载七次未插桩热扫描的中位数从 28.17 ms 降到 0.20 ms。完整附件摘要一致,正式进程的整体内存与延迟待实测。
语义与 CI 对齐
同源指纹和内容读回语义保留。新增的长期持有范围超出发现缓存原有的元数据职责。排序改动也复核了严格读取、失效和候选隔离。127 项当前关联测试通过,附件工作量由独立同条件对照补足。
我的整体评价 / Verdict
当前精确提交请求修改。保留早停和解析收益,把完整请求正文、附件留给需要它们的选中读取。复测同一批带图片的已投递请求,确认扫描结束后的常驻分配得到约束。
English verdict: REQUEST_CHANGES. Warm scans speed up, but caching full delivered journals retains about 45.2 MiB of image-backed allocations after actual recover returns in the matched fixture. Keep the cache metadata-only or explicitly bounded.
…t loops - Make LarkOwnerAttention._route iterate capture candidates lazily so matching stops at the newest valid p2p capture. - Reuse FileChatDiscovery in PrivateIngressObserver._records while preserving strict field validation and fail-closed errors. - Reuse FileChatDiscovery in ChatExternalConversations.pending so unchanged request journals skip redundant JSON reads and stay mutation-isolated. - Add regression and scan-efficiency tests in tests/test_lark_steward_directory_scan_perf.py. Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…_url memory growth Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
4510212 to
3fe827c
Compare
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini 2.5 Pro · Google · self_reported
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机
复审精确 head 3fe827c3a2f6f855ad1f8e9839dd4c966edae8b2。Lark steward 与 native chat 轮询服务运维方及宿主进程需要高频扫描 external-requests 与 private capture 目录。此前每次 recover()/commission_evidence() 都重解析整目录 JSON 并曾将含 attachments[].data_url 的完整行滞留于长生命周期缓存(24 份图片合成对照约 45.2 MiB 活跃内存);改动后 _recovery_discovery 仅缓存 13 个轻量元数据字段并保留 mtime/inode 失效,未终态行再按需 read_request,pending() 恢复无大对象缓存全量语义。已投递附件请求的恢复扫描后活跃内存降至 < 512 KiB,稳态未变更目录不再重复解析未修改文件;external_conversations.py 行数为 2100(低于基线 2111 上限)。本 PR 不修改消息入站权限规则、不重构 FileChatDiscovery 底层键机制,也不处理 origin/main 既有的两条 unrelated maintainability ratchet发现。
改动思路
复用 loopx/capabilities/native_chat/file_discovery.py 中的 FileChatDiscovery 增量失效基础设施:新增 _RECOVERY_JOURNAL_FIELDS 与 recovery_journal_fields 投影函数,使 ChatExternalConversations._recovery_discovery 只缓存恢复/委托索引必需的 13 个轻量字段;loopx/capabilities/native_chat/external_conversations.py 中 pending() 还原为直接读取完整 JSON,_recovery_rows() / commission_evidence() / recover() 遍历轻量索引,仅对 status in {"prepared", "command_queued"} 行按需 self.read_request(row["request_ref"]) 重载完整负载。同时 loopx/extensions/lark/owner_attention.py 采用生成器短路读取,loopx/extensions/lark/private_ingress.py 使用 FileChatDiscovery + _project_capture 校验必填字段。
具体改动
loopx/capabilities/native_chat/file_discovery.py: 定义_RECOVERY_JOURNAL_FIELDS与recovery_journal_fields,显式剥离attachments、data_url等大字段。loopx/capabilities/native_chat/external_conversations.py: 初始化self._recovery_discovery = FileChatDiscovery(),新增_recovery_rows(),将commission_evidence()与recover()切换至轻量元数据扫描并对未终态请求按需调用read_request,模块代码行数收口至 2100 行(<= 2111 基线)。loopx/extensions/lark/owner_attention.py:LarkOwnerAttention._route改为惰性生成器迭代。loopx/extensions/lark/private_ingress.py:PrivateIngressObserver._records复用FileChatDiscovery并由_project_capture保证缺失event/binding_id时 fail-closed 抛出ValueError。tests/test_lark_steward_directory_scan_perf.py: 补充扫描性能、tracemalloc24×1.5 MiB 附件内存非滞留回归(< 512 KiB)及文件修改失效测试。
对主干的风险
低风险。pending() 与 read_request() 对外返回完整规范 JSON 的语义未变;长生命周期缓存只存于恢复索引内部且受 FileChatDiscovery mtime/inode/size 校验保护。未修改既有基线外的 2 条 maintainability ratchet 历史项(build_heartbeat_prompt、_admitted_interaction_next_cli_actions),canary premerge 将其识别为 advisory_inherited_failure。本地 pytest tests/test_lark_steward_directory_scan_perf.py -q(最终压缩行数后 4 passed)、pytest -n auto 4 套件 707 passed(2 个多线程超时用例已单独顺序复验 2 passed in 2.79s)、loopx check(3 个最新改动文件 errors=0)及 loopx canary premerge(failures=0)全部通过。目标 Goal 配置 wait_for_ci: false,未轮询或依赖远端 CI。
我的整体评价
**APPROVE。**复用既有 FileChatDiscovery 增量失效所有者并仅在 _recovery_discovery 中投影 13 个轻量字段,避免平行状态或滞留附件大对象。本 PR 仅收敛目录轮询与恢复元数据索引边界,不更改 pending()/read_request() 的完整规范 JSON 对外契约,也不改动 origin/main 既有的无关基线项。 在精确 head 3fe827c3a2f6f855ad1f8e9839dd4c966edae8b2 上解决了先前独立复审指出的 P2 data_url 内存滞留问题,保持了 external_conversations.py 可维护性行数约束与 fail-closed 读取语义,本地实证充分。
English verdict: APPROVE - head 3fe827c resolves the P2 recovery-cache payload retention bug via metadata-only projection while keeping external_conversations.py at 2100 lines and passing local perf, memory, unit, and canary premerge gates.
Delivery Brief
LarkOwnerAttention._routeeagerly read and parsed every 24-hex capture.jsonintransport.rootvia a list comprehension before selecting the newest matching p2p capture.PrivateIngressObserver._recordsre-read and parsed every.jsoncapture file on everyobserve,_observe, andanswer_candidatespass.ChatExternalConversations.pendingre-read and parsed every 24-hex request journal inexternal-requestson everypending()call across reconcile/recovery/evidence paths.loopx/extensions/lark/owner_attention.py:LarkOwnerAttention._routenow iteratessorted(self.transport.root.glob("*.json"), reverse=True)lazily via a generator expression whensaved is None, stopping immediately once the newest matching p2p capture succeeds.loopx/extensions/lark/private_ingress.py:PrivateIngressObserver._recordsnow reusesFileChatDiscovery(self._capture_discovery) over deterministicsorted(self.transport.root.glob("*.json"))with_project_captureenforcingisinstance(row, dict),isinstance(row.get("event"), dict), andisinstance(row.get("binding_id"), str)(ValueError("private capture read is incomplete")before caching), keeping fingerprint invalidation, deletion eviction, anddeepcopymutation isolation intact.loopx/capabilities/native_chat/external_conversations.py:ChatExternalConversations.pendingnow reusesFileChatDiscovery(self._pending_discovery) oversorted(self.root.glob("*.json"))matching_REQUEST_JOURNAL_STEM, skipping unchanged journals while immediately invalidating on file updates/deletes (and keeping empty transient reads and Windows stat platforms uncached) and isolating caller mutations viadeepcopy.tests/test_lark_steward_directory_scan_perf.py: Added focused regression and scan-efficiency tests covering all three hot paths (early-exit lazy_routewith 2,000 synthetic captures + non-matching/deleted fallback,PrivateIngressObserver._recordscache hit/invalidation/deletion/mutation/fail-closed checks, andChatExternalConversations.pendingcache hit/invalidation/deletion/mutation isolation).Validation Evidence (HEAD
4510212271f8012151105a7c8fae3a22444ad08a)uv run --extra test python -m pytest tests/test_lark_steward_directory_scan_perf.py tests/test_lark_owner_attention.py tests/test_lark_private_ingress.py tests/test_native_steward_ingress.py tests/test_lark_steward_reports.py tests/test_chat_project_coordination.py -q-> 176 passed (355.35s)uv run --extra test python -m pytest tests/test_chat_file_discovery.py tests/test_lark_steward_report_route.py -q-> 33 passed (22.96s)uv run --extra test loopx --runtime-root <isolated-tmp> check --scan-path ...on all 4 changed/added files -> ok=True, errors=0uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main-> passed (selected=10, failures=0; all 6 direct checks passed,semantic-vocabulary-drift-smoke.pypassed, all 8extension-runtimesmokes passed;control-plane-maintainability-ratchet-smoke.pyreportedadvisory_inherited_failureunrelated to changed files)