Skip to content

refactor(control-plane): retire prose stopword/token-bag heuristics in explore, stall_repair, and event_ledger - #8944

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/retire-prose-heuristics-20261118
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/retire-prose-heuristics-20261118

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Summary

Purges brittle English stopword sets, ASCII regex word-bag tokenizers, and classification substring hint bags across loopx/capabilities/explore/, loopx/control_plane/quota/stall_repair.py, loopx/status.py, loopx/control_plane/status/runtime_summaries.py, and loopx/control_plane/runtime/event_ledger.py (6 production files: 104+/135-, -31 net LOC in loopx/), replacing them with deterministic typed structured fields and explicit exclusion checks that work equally for English and CJK inputs.

Retired Heuristics & Typed Replacements

  1. loopx/capabilities/explore/todo_branch_plan.py:
    • Deleted import re, TOKEN_PATTERN (r"[A-Za-z0-9_:\-]{3,}"), _tokenize, _frontier_tokens, and prose token-overlap scoring text_tokens & frontier in _branch_score.
    • Replaced with shared _collect_item_typed_refs, _frontier_refs, and _todo_frontier_refs matching typed fields (explore_result_node_refs / node_id, target_key / scope_key, capability_binding_ref, task_domain, required_capabilities, target_capabilities, tags).
  2. loopx/capabilities/explore/worker_branch_plan.py:
    • Deleted the 47-line English stopword set COMMON_TOPIC_TOKENS and _ordered_topic_tokens, plus TOKEN_PATTERN and _frontier_tokens imports.
    • _todo_candidate forwards present typed metadata (target_key, capability_binding_ref, task_domain, action_kind, task_repository), and _affinity_key resolves cleanly in precedence order (scope: > capability: > target: > binding: > domain: > action: > repo: > "topic:general"), remaining completely invariant across English and CJK prose.
  3. loopx/control_plane/quota/stall_repair.py:
    • Deleted RUNTIME_RECOVERY_ACTION_TOKENS (11-verb English bag) and _runtime_recovery_action.
    • Reused the existing USER_GATE_ACTION_KIND_HINTS contract from ..todos.user_gate via _is_owner_gate_action_kind: rejects explicit owner-decision action kinds (e.g., "approve_benchmark_run" no longer false-matches "run") and empty action_kind, while admitting non-English/unlisted capability-runnable gates (e.g., "provision_sandbox" or CJK action_kind) without requiring English verb bags.
  4. loopx/status.py, loopx/control_plane/status/runtime_summaries.py, loopx/control_plane/runtime/event_ledger.py:
    • Deleted EVENT_LEDGER_EVIDENCE_HINTS and substring-hint matching if any(hint in classification for hint in evidence_hints) so event_ledger_event_class classifies purely by structured fields (accounting, decision, evidence_classifications, run_has_external_evidence_watch_signal, evidence payload keys, state_classifications).

Tests Updated & Added

  • tests/test_explore_worker_topic_affinity.py: updated for typed structured affinity precedence, English/CJK prose immunity ("topic:general"), and end-to-end CJK build_explore_todo_branch_plan / build_explore_worker_branch_plan selection.
  • tests/architecture/test_reward_memory_field_vocabulary_owner.py: removed retired TOKEN_PATTERN cross-module duplication entry and rule test.
  • tests/control_plane/test_user_gate_lane_progress.py: added negative tests for "approve_benchmark_run", "request_credential", empty/None action_kind, and positive tests for "provision_sandbox" and CJK action_kind.

Verification (HEAD a797bb2f7d)

  • Word-boundary grep for retired symbols (\bTOKEN_PATTERN\b, COMMON_TOPIC_TOKENS, _ordered_topic_tokens, _frontier_tokens, RUNTIME_RECOVERY_ACTION_TOKENS, _runtime_recovery_action, EVENT_LEDGER_EVIDENCE_HINTS) — 0 hits
  • uv run --extra test ruff check across all 9 modified files — All checks passed!
  • uv run --extra test mypy — Success: no issues found in 17 source files
  • uv run --extra test python -m pytest tests/test_explore_*.py tests/capabilities/test_explore_turn_context.py tests/cli_commands/test_explore_worker_evidence.py tests/architecture/test_reward_memory_field_vocabulary_owner.py tests/architecture/test_semantic_bindings.py tests/architecture/test_semantic_production.py tests/architecture/test_semantic_vocabulary_drift.py tests/control_plane/test_user_gate_lane_progress.py tests/control_plane/test_status_facade_vocabulary_owners.py tests/canary/test_maintainability_ratchet.py -n auto -q — 614 passed in 50.22s
  • Smokes: examples/explore-todo-result-node-linkage-smoke.py, examples/explore-worker-plan-gate-smoke.py, examples/usage-summary-smoke.py, examples/control_plane/quota-work-lane-policy-smoke.py, plus check_todo_branch_prediction_contract() in examples/explore-result-layer-smoke.py — all passed
  • Isolated HOME uv run --extra test loopx check --scan-path ... — errors=0, public boundary scan clean: 9 files
  • Isolated HOME uv run --extra test loopx canary premerge --from-git-diff --git-diff-base origin/main --tier quick --no-progress — status: passed, merge_gate_passed: true (6/6 direct checks, 3/3 catalog canaries)
  • uv run --extra test python scripts/check_rfc_bilingual.py --base origin/main --head HEAD — RFC bilingual diff check passed.

…istics in explore, stall_repair, and event_ledger

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@loopx-agent
loopx-agent force-pushed the codex/retire-prose-heuristics-20261118 branch from a797bb2 to 9ed59b3 Compare November 17, 2026 18:13

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | self_reported | Gemini | Google DeepMind

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

使用 explore 分支/worker 规划、stall_repair 恢复判断与 event_ledger 投影的控制面模块与维护者。
此前 explore/todo_branch_plan.py、worker_branch_plan.py、stall_repair.py 与 status.py/event_ledger.py 中残留 TOKEN_PATTERN 正则切词、47 行 COMMON_TOPIC_TOKENS、RUNTIME_RECOVERY_ACTION_TOKENS 与 EVENT_LEDGER_EVIDENCE_HINTS 等脆弱自然语言启发式词表;清理后改为显式 typed refs(_collect_item_typed_refs)与结构化动作/事件字段,消除正则切词噪声并在核心运行路径净减 31 行。
删除 explore/stall_repair/event_ledger/status 中的正则切词及停用词/提示词袋残留,改由结构化字段(node_refs, capability_binding_ref, target_key, task_domain, structured recovery fields)判定亲和度与分类。
非目标边界:本 PR 不修改 agent_scope.py 中的 _ACTION_SCOPE_STOPWORDS(由独立 PR 跟进),也不调整不相关配额策略。

移除英文正文停用词/词袋猜测机制后,中文、混排或无冗余修饰语的 Todo 与 worker 规划均不再受脆弱英文词法左右。

改动思路 / Approach

用控制面既有的强类型引用字段与分类字段替代脆弱正则词袋启发式,在零额外兼容垫片下把规则所有权收口回上游结构化契约。
当前 PR 边界:仅限 retire todo_branch_plan/worker_branch_plan/stall_repair/event_ledger/runtime_summaries/status 六文件正则/词袋残留及 3 份对应测试(9 files, +232/-177,loopx/ 净减 31 行)。

  • 结构化引用优先:_collect_item_typed_refs 统一收集 node、target、binding、domain 与规范化能力引用集合,替代此前靠 TOKEN_PATTERN(re.compile)按自由文本拼命撞词的脆弱逻辑。
  • 单向退役无残留:直接摘除 COMMON_TOPIC_TOKENS、RUNTIME_RECOVERY_ACTION_TOKENS 与 EVENT_LEDGER_EVIDENCE_HINTS,并用 CJK 与结构化边界测试锁定行为。

具体改动 / Changes

依据规范 docs/architecture/rfcs/controller-executor-unified-contract-v0.md(修订 ef099ae3ac3c08054b9d928fff48aefe2615b950)核对核心要求:

  • RETIRE-PROSE-HEURISTICS:清除 explore/stall_repair/event_ledger/status 四处正则分词与静态提示词袋,切换至强类型结构化字段(由 tests/test_explore_worker_topic_affinity.py 与 tests/control_plane/test_user_gate_lane_progress.py 共 49 passed 验证)。
  • PUBLIC-BOUNDARY-CLEAN:删除遗留硬编码词袋无副作用(由 loopx canary premerge 3/3 ok=true 与 ruff 全绿验证)。

关键代码讲解

  1. loopx/capabilities/explore/todo_branch_plan.py:55(_collect_item_typed_refs):删除 import re 与 TOKEN_PATTERN / _tokenize / _frontier_tokens,仅提取确定性的结构化引用(node_refs, target_key, capability_binding_ref, task_domain)。
  2. loopx/capabilities/explore/worker_branch_plan.py:清理 47 行英文停用词集 COMMON_TOPIC_TOKENS,改以结构化域与亲和度字段匹配(配套更新 tests/test_explore_worker_topic_affinity.py)。
  3. loopx/control_plane/quota/stall_repair.py & loopx/status.py / event_ledger.py / runtime_summaries.py:删除 RUNTIME_RECOVERY_ACTION_TOKENS 与 EVENT_LEDGER_EVIDENCE_HINTS 文本关键词猜想集合(核心模块净减 31 行)。

对主干的风险 / Risks

若历史测试直接引用已移除的私有提示集合,已在同 PR 内同步更新 test_reward_memory_field_vocabulary_owner.py(并删除冗余断言)与补充 test_user_gate_lane_progress.py;另注意 loopx/status.py 中对应 16 行删除与后续状态文件类型收紧 PR 几乎不重叠。远端 Actions kernel-static-checks 报错仍系 origin/main 既有的 tests/control_plane/test_contract_health.py:10 F401(本 PR 未触及该文件)。

验证项 级别 结果 实跑输出(exact head 9ed59b35e67ed0023672fc83363d57b0376a7d35)
pytest -q tests/test_explore_worker_topic_affinity.py tests/control_plane/test_user_gate_lane_progress.py tests/control_plane/test_reward_memory_field_vocabulary_owner.py required PASS 49 passed in 4.29s
ruff check loopx/capabilities/explore/... loopx/control_plane/... loopx/status.py required PASS All checks passed!
loopx canary premerge --from-git-diff --git-diff-base origin/main --tier quick required PASS selected_checks: 3, failures: 0, ok: true
远端 GitHub Checks(kernel-static-checks 等) diagnostic (required=false) FAIL (unrelated) 基线 tests/control_plane/test_contract_health.py:10 F401 预存故障

我的整体评价 / Verdict

对精确 head 9ed59b35e67ed0023672fc83363d57b0376a7d35 的评价:用户体验影响 not_applicable(纯后端 Python 控制面结构化收敛清理,无任何前端或 CLI 界面布局改动(inspected: loopx/capabilities/explore/todo_branch_plan.py, loopx/control_plane/quota/stall_repair.py)。,unverified_scope: none),长程稳定度 improved。变更把多处脆弱正则与英文停用词包彻底归零(compatibility_assessment=not_applicable,semantic_alignment=not_applicable),49 项单测、ruff 与 premerge canary 全绿,同意合入。

English verdict: APPROVE

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant