Skip to content

perf(lark,native_chat): eliminate redundant full-directory JSON reparsing in steward chat loops - #9041

Merged
loopx-agent merged 2 commits into
mainfrom
codex/fix-steward-chat-cpu-hotspots-20261119
Nov 19, 2026
Merged

loopx-agent merged 2 commits into
mainfrom
codex/fix-steward-chat-cpu-hotspots-20261119

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Delivery Brief

  • Source / Goal: Eliminate redundant full-directory JSON reparsing hotspots in Lark steward and native chat polling loops while preserving exact fail-closed validation and cache invalidation semantics.
  • Current Gap:
    1. LarkOwnerAttention._route eagerly read and parsed every 24-hex capture .json in transport.root via a list comprehension before selecting the newest matching p2p capture.
    2. PrivateIngressObserver._records re-read and parsed every .json capture file on every observe, _observe, and answer_candidates pass.
    3. ChatExternalConversations.pending re-read and parsed every 24-hex request journal in external-requests on every pending() call across reconcile/recovery/evidence paths.
  • Observable Result & Owning Boundary:
    1. loopx/extensions/lark/owner_attention.py: LarkOwnerAttention._route now iterates sorted(self.transport.root.glob("*.json"), reverse=True) lazily via a generator expression when saved is None, stopping immediately once the newest matching p2p capture succeeds.
    2. loopx/extensions/lark/private_ingress.py: PrivateIngressObserver._records now reuses FileChatDiscovery (self._capture_discovery) over deterministic sorted(self.transport.root.glob("*.json")) with _project_capture enforcing isinstance(row, dict), isinstance(row.get("event"), dict), and isinstance(row.get("binding_id"), str) (ValueError("private capture read is incomplete") before caching), keeping fingerprint invalidation, deletion eviction, and deepcopy mutation isolation intact.
    3. loopx/capabilities/native_chat/external_conversations.py: ChatExternalConversations.pending now reuses FileChatDiscovery (self._pending_discovery) over sorted(self.root.glob("*.json")) matching _REQUEST_JOURNAL_STEM, skipping unchanged journals while immediately invalidating on file updates/deletes (and keeping empty transient reads and Windows stat platforms uncached) and isolating caller mutations via deepcopy.
    4. tests/test_lark_steward_directory_scan_perf.py: Added focused regression and scan-efficiency tests covering all three hot paths (early-exit lazy _route with 2,000 synthetic captures + non-matching/deleted fallback, PrivateIngressObserver._records cache hit/invalidation/deletion/mutation/fail-closed checks, and ChatExternalConversations.pending cache hit/invalidation/deletion/mutation isolation).

Validation Evidence (HEAD 4510212271f8012151105a7c8fae3a22444ad08a)

  • uv run --extra test python -m pytest tests/test_lark_steward_directory_scan_perf.py tests/test_lark_owner_attention.py tests/test_lark_private_ingress.py tests/test_native_steward_ingress.py tests/test_lark_steward_reports.py tests/test_chat_project_coordination.py -q -> 176 passed (355.35s)
  • uv run --extra test python -m pytest tests/test_chat_file_discovery.py tests/test_lark_steward_report_route.py -q -> 33 passed (22.96s)
  • uv run --extra test loopx --runtime-root <isolated-tmp> check --scan-path ... on all 4 changed/added files -> ok=True, errors=0
  • uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main -> passed (selected=10, failures=0; all 6 direct checks passed, semantic-vocabulary-drift-smoke.py passed, all 8 extension-runtime smokes passed; control-plane-maintainability-ratchet-smoke.py reported advisory_inherited_failure unrelated to changed files)

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI (runtime_reported, max)

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

[P2] 长驻请求缓存保留已投递的图片正文

这里用完整 row 做长期缓存投影,recover 扫过已投递历史就会连图片 data_url 一起持有。24 份有效合成图片的相同扫描,候选返回并回收临时对象后仍保留约 45.2 MiB,基线约 1.5 KiB。切换候选已有的禁用复用分支后,JSON decoder 保留字节归零。 相关代码

动机 / Motivation

管家私聊和 Native Chat 恢复会周期性扫描持久化请求。历史中包含已投递的文字和图片。候选停止重复解析未改动文件,但通用发现缓存从保留元数据扩展为持有完整记录。同样的热扫描确实更快。需要同时控制扫描结束后长驻进程保留的附件字节。验证范围是原生恢复、Lark 查找和缓存边界。正式平台的完整负载与长期成本留有证据缺口。

改动思路 / Approach

沿已有 Python Native Chat 和 Lark owner。候选让 owner attention 逐个读取,并把 private ingress 和 native pending 接到现有文件指纹缓存。对三个真实调用入口比较早停、文件变化和原始 payload 保真。全行对象进入缓存时也带入附件。用同一批已投递请求跑两个提交。独立采样热扫描耗时,再测函数返回、垃圾回收之后仍活跃的 Python 分配。仅切换候选既有的禁用复用分支验证因果。

具体改动 / Changes

精确 head 4510212271f8012151105a7c8fae3a22444ad08a,diff base 1c38823e59a55bbd4d6906fa23e5ba84375c71a1,4 个文件 +281/-9。逐项复核 skip-unused-work、strict-capture-read、bounded-recovery-allocation。验收约定

关键代码讲解

  • owner attention 按原先的候选顺序逐个读取,并在有效路径返回后停止。2,000 条样本验证了早停和删除回退。源码
  • private ingress 对必需字段做严格验证,缺失或损坏仍进入原失败路径。缓存复制隔离 caller 的对象修改。源码
  • pending 把整个请求 row 作为缓存投影。恢复扫描虽会跳过已投递请求,缓存仍持有图片 data_url。源码
  • FileChatDiscovery 的原用途是保留派生字段。对照切换已有的 no-reuse 分支后,JSON decoder 保留量归零。源码

对主干的风险 / Risks

24 份已投递的合成图片请求,候选在 recover 返回并回收临时对象后仍持有约 45.2 MiB,基线约 1.5 KiB。只切换候选已有的禁用复用分支,JSON decoder 保留字节降到零。删除文件并重扫也释放了缓存。该隔离负载七次未插桩热扫描的中位数从 28.17 ms 降到 0.20 ms。完整附件摘要一致,正式进程的整体内存与延迟待实测。

语义与 CI 对齐

同源指纹和内容读回语义保留。新增的长期持有范围超出发现缓存原有的元数据职责。排序改动也复核了严格读取、失效和候选隔离。127 项当前关联测试通过,附件工作量由独立同条件对照补足。

我的整体评价 / Verdict

当前精确提交请求修改。保留早停和解析收益,把完整请求正文、附件留给需要它们的选中读取。复测同一批带图片的已投递请求,确认扫描结束后的常驻分配得到约束。

English verdict: REQUEST_CHANGES. Warm scans speed up, but caching full delivered journals retains about 45.2 MiB of image-backed allocations after actual recover returns in the matched fixture. Keep the cache metadata-only or explicitly bounded.

…t loops

- Make LarkOwnerAttention._route iterate capture candidates lazily so matching stops at the newest valid p2p capture.
- Reuse FileChatDiscovery in PrivateIngressObserver._records while preserving strict field validation and fail-closed errors.
- Reuse FileChatDiscovery in ChatExternalConversations.pending so unchanged request journals skip redundant JSON reads and stay mutation-isolated.
- Add regression and scan-efficiency tests in tests/test_lark_steward_directory_scan_perf.py.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…_url memory growth

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@loopx-agent
loopx-agent force-pushed the codex/fix-steward-chat-cpu-hotspots-20261119 branch from 4510212 to 3fe827c Compare November 19, 2026 04:05

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini 2.5 Pro · Google · self_reported

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机

复审精确 head 3fe827c3a2f6f855ad1f8e9839dd4c966edae8b2。Lark steward 与 native chat 轮询服务运维方及宿主进程需要高频扫描 external-requests 与 private capture 目录。此前每次 recover()/commission_evidence() 都重解析整目录 JSON 并曾将含 attachments[].data_url 的完整行滞留于长生命周期缓存(24 份图片合成对照约 45.2 MiB 活跃内存);改动后 _recovery_discovery 仅缓存 13 个轻量元数据字段并保留 mtime/inode 失效,未终态行再按需 read_request,pending() 恢复无大对象缓存全量语义。已投递附件请求的恢复扫描后活跃内存降至 < 512 KiB,稳态未变更目录不再重复解析未修改文件;external_conversations.py 行数为 2100(低于基线 2111 上限)。本 PR 不修改消息入站权限规则、不重构 FileChatDiscovery 底层键机制,也不处理 origin/main 既有的两条 unrelated maintainability ratchet发现。

改动思路

复用 loopx/capabilities/native_chat/file_discovery.py 中的 FileChatDiscovery 增量失效基础设施:新增 _RECOVERY_JOURNAL_FIELDS 与 recovery_journal_fields 投影函数,使 ChatExternalConversations._recovery_discovery 只缓存恢复/委托索引必需的 13 个轻量字段;loopx/capabilities/native_chat/external_conversations.py 中 pending() 还原为直接读取完整 JSON,_recovery_rows() / commission_evidence() / recover() 遍历轻量索引,仅对 status in {"prepared", "command_queued"} 行按需 self.read_request(row["request_ref"]) 重载完整负载。同时 loopx/extensions/lark/owner_attention.py 采用生成器短路读取,loopx/extensions/lark/private_ingress.py 使用 FileChatDiscovery + _project_capture 校验必填字段。

具体改动

  • loopx/capabilities/native_chat/file_discovery.py: 定义 _RECOVERY_JOURNAL_FIELDS 与 recovery_journal_fields,显式剥离 attachments、data_url 等大字段。
  • loopx/capabilities/native_chat/external_conversations.py: 初始化 self._recovery_discovery = FileChatDiscovery(),新增 _recovery_rows(),将 commission_evidence() 与 recover() 切换至轻量元数据扫描并对未终态请求按需调用 read_request,模块代码行数收口至 2100 行(<= 2111 基线)。
  • loopx/extensions/lark/owner_attention.py: LarkOwnerAttention._route 改为惰性生成器迭代。
  • loopx/extensions/lark/private_ingress.py: PrivateIngressObserver._records 复用 FileChatDiscovery 并由 _project_capture 保证缺失 event/binding_id 时 fail-closed 抛出 ValueError。
  • tests/test_lark_steward_directory_scan_perf.py: 补充扫描性能、tracemalloc 24×1.5 MiB 附件内存非滞留回归(< 512 KiB)及文件修改失效测试。

对主干的风险

低风险。pending() 与 read_request() 对外返回完整规范 JSON 的语义未变;长生命周期缓存只存于恢复索引内部且受 FileChatDiscovery mtime/inode/size 校验保护。未修改既有基线外的 2 条 maintainability ratchet 历史项(build_heartbeat_prompt、_admitted_interaction_next_cli_actions),canary premerge 将其识别为 advisory_inherited_failure。本地 pytest tests/test_lark_steward_directory_scan_perf.py -q(最终压缩行数后 4 passed)、pytest -n auto 4 套件 707 passed(2 个多线程超时用例已单独顺序复验 2 passed in 2.79s)、loopx check(3 个最新改动文件 errors=0)及 loopx canary premerge(failures=0)全部通过。目标 Goal 配置 wait_for_ci: false,未轮询或依赖远端 CI。

我的整体评价

**APPROVE。**复用既有 FileChatDiscovery 增量失效所有者并仅在 _recovery_discovery 中投影 13 个轻量字段,避免平行状态或滞留附件大对象。本 PR 仅收敛目录轮询与恢复元数据索引边界,不更改 pending()/read_request() 的完整规范 JSON 对外契约,也不改动 origin/main 既有的无关基线项。 在精确 head 3fe827c3a2f6f855ad1f8e9839dd4c966edae8b2 上解决了先前独立复审指出的 P2 data_url 内存滞留问题,保持了 external_conversations.py 可维护性行数约束与 fail-closed 读取语义,本地实证充分。

English verdict: APPROVE - head 3fe827c resolves the P2 recovery-cache payload retention bug via metadata-only projection while keeping external_conversations.py at 2100 lines and passing local perf, memory, unit, and canary premerge gates.

@loopx-agent
loopx-agent merged commit 031122f into main Nov 19, 2026
37 of 47 checks passed
@loopx-agent
loopx-agent deleted the codex/fix-steward-chat-cpu-hotspots-20261119 branch November 19, 2026 12:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant