Repository navigation
refactor(benchmark_toolkit): eliminate Any and add ratchet baselines for experiment_board and study_projection - #8978
Conversation
…iment_board and study_projection Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google (self_reported)
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
已针对 PR #8978 exact head 74e9f9c0251f65dbf7cf0fb85ed3308bf497d38e 完成完整审查与本地验证(审查结论:APPROVE,无 P0/P1/P2 问题)。
动机 / Motivation
受影响调用方与运维者:所有消费 loopx/capabilities/benchmark_toolkit/experiment_board.py 与 loopx/capabilities/benchmark_toolkit/study_projection.py 的基准实验与 CLI 模块(包括 loopx/cli_commands/benchmark_experiment_board.py、loopx/cli_commands/benchmark_study.py、loopx/cli_commands/benchmark_dispatch.py、loopx/capabilities/benchmark_toolkit/behavior_finding.py、loopx/capabilities/benchmark_toolkit/catalog_entry.py、loopx/capabilities/benchmark_toolkit/four_arm_contract.py、loopx/capabilities/benchmark_toolkit/terminal_followthrough.py 与 loopx/capabilities/benchmark_toolkit/__init__.py),以及 loopx/canary/ 可维护性棘轮维护者。
修改前(origin/main 513b6f168a):experiment_board.py 共 1074 行含 58 处 Any、study_projection.py 共 1386 行含 59 处 Any(合计 117 处 Any 含 dict[str, Any])、存在跨文件重复的 _reject_unknown_fields/_optional_token/_timestamp/_finite_number 校验器与内联字典拆包/章节拼接样板,且两文件均未录入 loopx/canary/module_metric_baseline.json 零 Any 棘轮。修改后(74e9f9c0251f65dbf7cf0fb85ed3308bf497d38e):experiment_board.py 降至 1073 行、两文件 any_count 与 dict_any_count 均降至 0,复用共享校验函数并写入 experiment_board.py(lines: 1088)与 study_projection.py(lines: 1401)零 Any 基线棘轮。
可观测结果:module_metrics 返回 eb: {'lines': 1073, 'any_count': 0, 'dict_any_count': 0} 与 sp: {'lines': 1386, 'any_count': 0, 'dict_any_count': 0};mypy --strict --follow-imports=silent 两文件 0 错误,全仓 mypy(17 个源码文件)0 错误,ruff 全过;11 组 benchmark/架构/棘轮单测 830 passed, 20 skipped in 39.39s(该 20 条跳过与 origin/main 基线一致);canary premerge --tier quick 6 项直接检查与 3 项目录金丝雀均通过(failures=0,因触及 benchmark_toolkit/* 按预期报告 manual_holds=1 benchmark_sensitive 人工门禁);loopx check --scan-path ... errors=0, warnings=0。
非目标边界:本 PR 不修改 benchmark 实验行、研究看板、四臂对比或上传回执的运行时 JSON schema、错误提示文案或校验语义,保留 behavior_finding.py 引用的全部兼容内部符号,亦不授予基准敏感面合入权限。
改动思路 / Approach
全部强类型收口与重复样板合并均收敛在 loopx/capabilities/benchmark_toolkit/experiment_board.py 与 loopx/capabilities/benchmark_toolkit/study_projection.py 原生边界内,未新增并行决策层,亦未修改任何下游调用方或 behavior_finding.py。
当前 PR 边界:仅触及 experiment_board.py 与 study_projection.py 的零 Any 强类型收敛/公共校验复用与局部助函数抽离,以及 loopx/canary/module_metric_baseline.json 中对应的 2 条零 Any 棘轮基线。
- 统一窄化契约(
experiment_board.py:86-98、study_projection.py:65-87,438-449):以Payload = dict[str, object]与PayloadView = Mapping[str, object]替代两文件全部Any/dict[str, Any],抽出_as_map、_as_list、_as_map_list、_as_int、_sorted_value_counts、_strip_domain_state_key、_require_bool_arg、_require_str_arg消除未窄化索引与重复枚举计数模板。 - 消除跨模块校验重复并保兼容(
study_projection.py:55-63):从.experiment_board复用_finite_number、_optional_token、_reject_unknown_fields、_timestamp,同时完整保留behavior_finding.py导入的_active_envelopes、_bounded_text、_finite_number、_reject_unknown_fields、_token内部接口;在experiment_board.py:908-930抽出_join_codes与_render_table_section,在study_projection.py:785-813抽出_resolve_upload_candidate。 - 零
Any棘轮锁定(loopx/canary/module_metric_baseline.json:17-26):按字母序在planner.py与catalog.py之间锁定experiment_board.py(any_count=0, dict_any_count=0, lines=1088)与study_projection.py(any_count=0, dict_any_count=0, lines=1401)。
具体改动 / Changes
本次 diff 仅触及 3 个文件(656 insertions(+), 647 deletions(-)):
loopx/capabilities/benchmark_toolkit/experiment_board.py(+280 / -281,1074 -> 1073行,58 Any -> 0):清除全部Any,增加Payload/PayloadView、_as_map、_as_list、_strip_domain_state_key、_join_codes、_render_table_section,保留全部既有 docstring 与注释。loopx/capabilities/benchmark_toolkit/study_projection.py(+366 / -366,1386行,59 Any -> 0):清除全部Any,复用.experiment_board共享校验器并抽离_as_map_list、_as_int、_sorted_value_counts、_require_bool_arg、_require_str_arg、_resolve_upload_candidate与Sequence[PayloadView]聚合入参。loopx/canary/module_metric_baseline.json(+10 / -0,L17-26):注册上述两文件的零Any与行数上限基线。
对主干的风险 / Risks
- 兼容性与输入窄化:
_as_map/_as_list/_as_map_list/_as_int针对边界非法类型延续原模块校验与空值回退,behavior_finding.py与 CLI/能力调用方无需改动,全仓 17 个mypy目标 0 错误、830 passed, 20 skipped(与origin/main基线跳过数相同)。 - 基准敏感面与 CI 状态:因触及
loopx/capabilities/benchmark_toolkit/*,canary premerge --tier quick按策略返回manual_review_required(manual_holds=1: benchmark_sensitive,failures=0);远端Sign-off、inspect-pr-diff、dependency-review、loopx-codex-provider-routing、loopx-repo-health、loopx-obelisk已SUCCESS,其余 checks(build、loopx-core、changes、prepare、Summary)尚在运行中,需维护者审查与授权后方可合并。 - 本地完整验证结果(exact head
74e9f9c0251f65dbf7cf0fb85ed3308bf497d38e):module_metrics:eb: {'lines': 1073, 'any_count': 0, 'dict_any_count': 0},sp: {'lines': 1386, 'any_count': 0, 'dict_any_count': 0}uv run --extra test mypy --strict --follow-imports=silent ...: Success: no issues found in 2 source filesuv run --extra test mypy: Success: no issues found in 17 source filesuv run --extra test ruff check ...: All checks passed!uv run --extra test pytest -q -n 4(11 个 benchmark/架构/棘轮/词表测试文件): 830 passed, 20 skipped in 39.39suv run --extra test loopx canary premerge --tier quick --from-git-diff --git-diff-base origin/main: 6 direct + 3 catalog canaries passed (failures=0),manual_holds=1 (benchmark_sensitive)loopx check --scan-path ...: ok=True (errors=0, warnings=0, checks=5)
我的整体评价 / Verdict
- Long-horizon 影响:
improved— 彻底清零experiment_board.py(58 处)与study_projection.py(59 处)共 117 处Any、复用共享校验并锁定棘轮基线,消除后续基准分析演进的隐式类型漂移风险。 - User-experience 影响:
not_applicable— 内部 Python 静态类型收口、校验复用与 JSON 基线登记,不改变 CLI 参数、输出结构或前端界面,visual_review为not_applicable。 - 可观测语义 / 复用 / 词表:
observable_semantics为equivalent,repository_reuse为reused,authority_semantics与semantic_alignment均为aligned(candidate_decision="reuse_existing")。
改动聚焦、严格类型检查与本地 830 条用例均充分验证且无 P0/P1/P2 发现项;因触及 benchmark_toolkit/*(benchmark_sensitive)且为作者自有 PR,合并需等待远端 CI 全绿并获得维护者明确授权。
English verdict: APPROVE exact head 74e9f9c (author-owned COMMENTED approval review); eliminates all 117 Any usages across loopx/capabilities/benchmark_toolkit/experiment_board.py (58 -> 0) and study_projection.py (59 -> 0), consolidates shared validators and upload/table helpers, and locks zero-Any baselines in module_metric_baseline.json, verified by strict/repo mypy (17 files), ruff, 830 passed / 20 pre-existing skipped pytest tests, and quick premerge canaries (0 failures, benchmark_sensitive manual hold).
Summary
Eliminate all
Anyanddict[str, Any]usage acrossloopx/capabilities/benchmark_toolkit/experiment_board.pyandloopx/capabilities/benchmark_toolkit/study_projection.py, unify shared parsing/formatting helpers, and lock zero-Anymaintainability ratchet baselines for both modules.Owning Boundary & Changes
experiment_board.py,study_projection.py):58Anyinexperiment_board.py(1074 -> 1073lines) and59Anyinstudy_projection.py(1386 -> 1386lines) with strictPayload/PayloadViewmappings and explicit runtime validators (_as_map,_as_list,_as_map_list,_as_int), leaving exported signatures, private_-prefixed symbols imported bybehavior_finding.py, dict shapes, and validation error strings unchanged._reject_unknown_fields,_optional_token,_timestamp, and_finite_numberfromloopx.capabilities.benchmark_toolkit.experiment_boardinstudy_projection.py(while preservingbehavior_finding.pycompatibility imports_active_envelopes,_bounded_text,_finite_number,_reject_unknown_fields,_token)._strip_domain_state_key,_join_codes,_render_table_section,_sorted_value_counts,_require_bool_arg,_require_str_arg, and_resolve_upload_candidateto remove repeated dict filtering, CSV joining, markdown section rendering, and upload candidate validation blocks.loopx/canary/module_metric_baseline.json):any_count: 0anddict_any_count: 0for bothloopx/capabilities/benchmark_toolkit/experiment_board.py(lines: 1088) andloopx/capabilities/benchmark_toolkit/study_projection.py(lines: 1401).Acceptance & Verification Evidence
experiment_board.py:lines=1073,any_count=0,dict_any_count=0study_projection.py:lines=1386,any_count=0,dict_any_count=0uv run --extra test mypy --strict --follow-imports=silent loopx/capabilities/benchmark_toolkit/experiment_board.py loopx/capabilities/benchmark_toolkit/study_projection.py->Success: no issues found in 2 source filesuv run --extra test mypy->Success: no issues found in 17 source filesuv run --extra test ruff check loopx/capabilities/benchmark_toolkit/experiment_board.py loopx/capabilities/benchmark_toolkit/study_projection.py tests/canary/test_maintainability_ratchet.py->All checks passed!uv run --extra test pytest -q -n 4 tests/capabilities/test_benchmark_study_projection.py tests/capabilities/test_benchmark_experiment_board.py tests/capabilities/test_benchmark_four_arm_contract.py tests/capabilities/test_benchmark_behavior_finding.py tests/capabilities/test_benchmark_terminal_followthrough.py tests/capabilities/test_benchmark_toolkit.py tests/benchmark/test_edgebench_managed_run.py tests/architecture/test_benchmark_experiment_identity_owner.py tests/architecture/test_content_digest_single_owner.py tests/canary/test_maintainability_ratchet.py tests/architecture/test_semantic_vocabulary_drift.py->830 passed, 20 skipped in 39.39s(20 pre-existing skipped cases also skipped on unmodified baseline)uv run --extra test loopx canary premerge --tier quick --from-git-diff --git-diff-base origin/main-> all 6 direct checks and all 3 catalog canaries (control-plane-maintainability-ratchet-smoke.py,semantic-vocabulary-drift-smoke.py,catalog-planner-smoke.py) passed (failures=0); exitedmanual_review_required(manual_holds: 1,benchmark_sensitive: benchmark adapter, scoring, runner, or evidence paths require explicit maintainer review before self-merge).loopx check --scan-path ...->ok: True,errors=0, warnings=0, checks=5,public boundary scan clean: 3 files.Boundary & Rollback
benchmark_toolkitplus two new entries inmodule_metric_baseline.json; zero schema, CLI, wire-protocol, or persisted receipt changes. Fully reversible by reverting this PR.