Skip to content

perf(native_chat,lark): cache unchanged quota backlog health and interrupt observations - #9098

Merged
loopx-agent merged 1 commit into
mainfrom
codex/cache-quota-backlog-health-observations-20261120
Nov 19, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/cache-quota-backlog-health-observations-20261120

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Summary

Eliminates repeated full-history load_turn(), TypeScript effect_runtime_result IPC batches, and events_after .events.jsonl.lock churn during repeated LarkPrivateConversations.health() / ?view=diagnostics polls by introducing a revision-bound QuotaBacklogObservationCache:

  • Revision-bound backlog & interrupt caches (loopx/capabilities/native_chat/file_discovery.py):
    • Adds optional_file_fingerprint() (_MISSING_FILE_FP sentinel when absent) and bounded QuotaBacklogObservationCache (_base, _stopped, _interrupt_events keyed by request/Turn file fingerprints, (session_id, turn_id, status, command, session_status, evaluator_id), and events.jsonl fingerprint + empty _event_pending guard).
    • Adds observe_quota_backlog_batch() with two-phase ordering (build_inputs on uncached rows -> load_session -> cache lookup/eval) so store.load_turn hooks/concurrent session closes keep exact existing batch semantics, plus BEFORE == AFTER TOCTOU fingerprint guards on cache commit.
    • Adds observed_turn_interrupt() using flush_lock.acquire(blocking=False) + _event_pending check so idle turns avoid exclusive_file_lock(..., operation="read_chat_events") and ChatEventCache thrashing while buffered or flushed interrupts invalidate immediately.
  • Observer integration (loopx/capabilities/native_chat/external_conversations.py, loopx/extensions/lark/private_conversations.py):
    • Adds opt-in use_cache: bool = False to quota_backlog_observer(), pre-filtering command in {"stop", "new"} source controls once per observer via @cache and keeping direct/test callers uncached by default.
    • Enables use_cache=True inside LarkPrivateConversations.health().
  • Regression & invalidation tests (tests/test_lark_steward_directory_scan_perf.py):
    • Verifies zero extra effect_runtime_result, load_turn(), or events_after() calls on unchanged repeated health() polls across >8 failed turns (TERMINAL_CACHE_MAX_TURNS=8).
    • Verifies immediate invalidation on Turn status transitions, flushed/unflushed interrupt events, native_result_recovery disk writes, /stop commands (and subsequent _stopped cache hits), effect_runtime_result identity changes, session closure, and uncached use_cache=False direct calls.

Local Verification

  • ruff check loopx/capabilities/native_chat/file_discovery.py loopx/capabilities/native_chat/external_conversations.py loopx/extensions/lark/private_conversations.py tests/test_lark_steward_directory_scan_perf.py: passed
  • uv run --extra test python -m pytest -q tests/test_lark_steward_directory_scan_perf.py tests/test_private_chat_backlog_projection.py tests/test_chat_file_discovery.py tests/test_context_recovery_discovery.py tests/test_lark_private_conversations.py: 618 passed (0 failures)
  • uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main: passed (6/6 direct checks, 2/2 catalog canaries including maintainability ratchet and semantic vocabulary drift, 8/8 risk-profile smokes, failures: 0)
  • Line counts preserved strictly within ceilings (private_conversations.py: 1,999 lines; external_conversations.py: 1,935 lines)

…rrupt observations

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google (self_reported)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

  • affected_caller_or_operator: 启用了 Lark 私聊诊断页与守护进程健康检查(LarkPrivateConversations.health、ChatExternalConversations.quota_backlog_observer)的守护进程维护者及 Web/App 设置面板调用方。
  • before_after_scenario: 修改前:LarkPrivateConversations.health()(被前端 ?view=diagnostics 每 5s 轮询触发)每次都会调用 quota_backlog_observer(),对约 1,555 个已交付 Turn 逐一执行 ChatSessionStore.load_turn()(~38.86 MB JSON 解析)、打包发送给 Node/TypeScript effect_runtime_result('collaboration.conversation.request')、对约 73 个失败 Turn 重复调用 store.events_after() 导致 ChatEventCache(TERMINAL_CACHE_MAX_TURNS=8) 淘汰抖动并触发大量 read_chat_events 文件锁重写,且对 source_controls()(~1,651 条)逐条线性扫描。修改后:在 file_discovery.py 引入有界指纹缓存 QuotaBacklogObservationCache、TOCTOU 保护的 observed_turn_interrupt 与 observe_quota_backlog_batch,并在 external_conversations.py 增加 opt-in 参数 use_cache=False 及单次观察周期内的 stop/new 控制记录预筛选;仅在 LarkPrivateConversations.health() 中启用 use_cache=True,其余直接/测试调用默认保持 uncached。
  • observable_outcome: 未发生文件或会话状态变更时,LarkPrivateConversations.health() 重复轮询实现 0 次额外 effect_runtime_result IPC、0 次额外 ChatSessionStore.load_turn 与 0 次额外 events_after 文件锁读取;任意 Turn 状态、已落盘/缓冲中断事件、native_result_recovery、/stop 控制文件、会话关闭或 evaluator 替换均在下一轮立即失效并重算(618/618 单测全绿,canary premerge 10/10 ok: true)。
  • non_goals: 不改变 TypeScript 控制面 pending_observations 业务判定规则,不将默认 quota_backlog_observer(use_cache=False) 改为缓存模式,不改变 store.load_turn mock 或同批并发关停会话的执行顺序,也不持久化任何缓存索引文件。

在 PR #9096 消除 17 条历史恢复请求锁风暴后,对线上 com.loopx.chat 进行二次性能剖析确认:GET /api/chat/lark/private-conversations?view=diagnostics(由打开的面板每 5s 轮询)单次耗时仍达 ~1.69s–2.64s,且每 6s 仍在重写大量历史失败 Turn 的 .events.jsonl.lock。根因在于 LarkPrivateConversations.health() -> quota_backlog_observer() 每次全量重读 ~38.86 MB Turn JSON、发往 TS 控制面并击穿 TERMINAL_CACHE_MAX_TURNS=8 的事件缓存。本次在提交 40e999803985204d9302d37133579acffb51a176 补充按文件指纹与会话状态绑定失效的只读健康缓存。

改动思路 / Approach

  1. 有界指纹与 TOCTOU 防护缓存(file_discovery.py#L124-L280):
    • 新增 optional_file_fingerprint(文件不存在返回 _MISSING_FILE_FP 哨兵、Win32/OSError 回退 None)与有界容器 QuotaBacklogObservationCache(分别维护 _base、_stopped 与 _interrupt_events 布尔 verdicts 及 turn_status,不保留 Turn 正文或附件)。
    • 新增 observed_turn_interrupt,在 flush_lock.acquire(blocking=False) 下检查 _event_pending 为空与 events.jsonl 指纹未变才命中 _interrupt_events,否则落回完整 store.events_after() 并在提交缓存前后校验 fp_before == fp_after 与缓冲仍为空。
  2. 两阶段批次观察保持原并发语义(file_discovery.py#L282-L367):
    • 先对未命中文件前置缓存的条目执行 build_inputs(从而先运行 store.load_turn),再批量调用 load_session,最后结合 (req_fp, turn_fp, session_id, turn_id, status, command, evaluator_id, session_status) 检索/提交 _base 与 _stopped。这与原函数执行顺序完全一致,确保同批并发关停会话(concurrent-close)测试语义不变。
  3. 默认关闭隔离与预筛选(external_conversations.py#L411-L439、private_conversations.py#L193):
    • quota_backlog_observer(..., use_cache: bool = False) 默认保持 uncached;同时在单次 observer 内通过 @cache stop_controls() 仅保留 command in {"stop", "new"} 的控制项以降低线性扫描开销。仅 LarkPrivateConversations.health() 显式传入 use_cache=True。
  4. 缓存命中与全量失效回归测试(tests/test_lark_steward_directory_scan_perf.py#L581-L775):
    • 覆盖 20 个失败 + 20 个已完成 Turn 稳态零 IPC/零 load_turn/零 events_after、Turn 完成/事件落盘/buffered 未落盘中断/native_result_recovery 写盘//stop 控制与 _stopped 命中、effect_runtime_result 替换、Session 关闭及默认 use_cache=False 实时读取。

当前边界(current_pr_boundary):本 PR 仅收敛 LarkPrivateConversations.health() / quota_backlog_observer 热路径的重复读盘、IPC 与事件锁抖动;线上升版重启后在真实守护进程上测量 ?view=diagnostics 延迟、闲置 CPU 与 RSS 改善效果将在合并后于本机激活验证阶段完成。。
架构保留原因(architecture_assessment.reason):缓存结构与批次/TOCTOU 校验集中留在文件发现模块(file_discovery.py,与 FileChatDiscovery 同域),external_conversations.py 保持 1,935 行(远低于 2,000 行阈值)并通过 opt-in 参数向后兼容,private_conversations.py 仅 1 处开关改动(保持 1,999 行)。不增加第二裁决权或持久化状态。

具体改动 / Changes

关键代码讲解

  1. QuotaBacklogObservationCache & observe_quota_backlog_batch(file_discovery.py#L142-L367)
    利用 _base 与 _stopped 分层键值区分常规状态与已观察到停机控制后的覆写状态;写入前统一通过 commit() 校验前后 (req_fp, turn_fp) 一致性,若并发改动导致不一致则 drop_request(),绝不将竞态中间态写入缓存。
  2. observed_turn_interrupt(file_discovery.py#L232-L280)
    尊重 ChatSessionStore 原有锁序(flush_lock -> _event_lock),通过非阻塞 acquire(blocking=False) 与 _event_pending 判空避开正在刷盘的 Turn,并将未决事件或指纹不匹配直接回退到原始 store.events_after()。

对主干的风险 / Risks

  • 并发关停与 Mock 旁路边界:首次运行 test_lark_private_conversations.py 时暴露了先读 load_session 后调 load_turn 会导致批内并发 close() 漏读新会话状态的问题,以及提取 observe_quota_backlog_batch 保持 external_conversations.py 低于 2,000 行的必要性;调整为两阶段顺序并在真存储校验中比对 getattr(store.<method>, "__func__", None) 后,完整 575+43 项回归全部通过。
  • 文件行数监控:private_conversations.py 为 1,999 行,external_conversations.py 为 1,935 行,file_discovery.py 为 505 行,全部通过 control-plane-maintainability-ratchet-smoke.py。
  • 视觉与 UI:零 UI 改动,visual_review 为 not_applicable。
  • 本地验证记录(未依赖远端 GitHub CI):
    • ruff check(4 个改动文件):PASS,All checks passed!
    • pytest -q tests/test_lark_steward_directory_scan_perf.py tests/test_private_chat_backlog_projection.py tests/test_chat_file_discovery.py tests/test_context_recovery_discovery.py tests/test_lark_private_conversations.py:PASS,618 passed in 1331.03s(0 failures)
    • loopx canary premerge --from-git-diff --git-diff-base origin/main:PASS,ok: true,direct_checks 6/6、catalog_canaries 2/2(含 maintainability ratchet 与 semantic vocabulary drift)、risk_profile_smokes 8/8,0 失败。

我的整体评价 / Verdict

本次改动在不改变任何 TypeScript 判定规则与默认未缓存路径语义的前提下,消除了健康检查端点的全量 Turn/事件锁/IPC 重复放大;并发会话关闭顺序、TOCTOU 校验、未落盘中断事件、/stop 控制变更与代码规模门禁均有完整测试和门禁实证,满足合入要求。视觉评审说明:Backend/runtime caching optimization only; verified diff touches zero UI, HTML, CSS, or dashboard files.(unverified_scope: none)。

English Verdict: APPROVE

@loopx-agent
loopx-agent merged commit 477def5 into main Nov 19, 2026
27 of 35 checks passed
@loopx-agent
loopx-agent deleted the codex/cache-quota-backlog-health-observations-20261120 branch November 19, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant