Skip to content

refactor(benchmark_toolkit): eliminate Any and add ratchet baselines for experiment_board and study_projection - #8978

Merged
huangruiteng merged 1 commit into
mainfrom
codex/benchmark-study-board-slim-20261118
Nov 18, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/benchmark-study-board-slim-20261118

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Summary

Eliminate all Any and dict[str, Any] usage across loopx/capabilities/benchmark_toolkit/experiment_board.py and loopx/capabilities/benchmark_toolkit/study_projection.py, unify shared parsing/formatting helpers, and lock zero-Any maintainability ratchet baselines for both modules.

Owning Boundary & Changes

  • Typed payload contracts (experiment_board.py, study_projection.py):
    • Replaced all 58 Any in experiment_board.py (1074 -> 1073 lines) and 59 Any in study_projection.py (1386 -> 1386 lines) with strict Payload / PayloadView mappings and explicit runtime validators (_as_map, _as_list, _as_map_list, _as_int), leaving exported signatures, private _-prefixed symbols imported by behavior_finding.py, dict shapes, and validation error strings unchanged.
  • Shared helper reuse & deduplication:
    • Reused _reject_unknown_fields, _optional_token, _timestamp, and _finite_number from loopx.capabilities.benchmark_toolkit.experiment_board in study_projection.py (while preserving behavior_finding.py compatibility imports _active_envelopes, _bounded_text, _finite_number, _reject_unknown_fields, _token).
    • Extracted _strip_domain_state_key, _join_codes, _render_table_section, _sorted_value_counts, _require_bool_arg, _require_str_arg, and _resolve_upload_candidate to remove repeated dict filtering, CSV joining, markdown section rendering, and upload candidate validation blocks.
  • Maintainability ratchet (loopx/canary/module_metric_baseline.json):
    • Added colocation baselines locking any_count: 0 and dict_any_count: 0 for both loopx/capabilities/benchmark_toolkit/experiment_board.py (lines: 1088) and loopx/capabilities/benchmark_toolkit/study_projection.py (lines: 1401).

Acceptance & Verification Evidence

  • Maintainability metrics:
    • experiment_board.py: lines=1073, any_count=0, dict_any_count=0
    • study_projection.py: lines=1386, any_count=0, dict_any_count=0
  • Static analysis:
    • uv run --extra test mypy --strict --follow-imports=silent loopx/capabilities/benchmark_toolkit/experiment_board.py loopx/capabilities/benchmark_toolkit/study_projection.py -> Success: no issues found in 2 source files
    • uv run --extra test mypy -> Success: no issues found in 17 source files
    • uv run --extra test ruff check loopx/capabilities/benchmark_toolkit/experiment_board.py loopx/capabilities/benchmark_toolkit/study_projection.py tests/canary/test_maintainability_ratchet.py -> All checks passed!
  • Targeted & architecture test suite:
    • uv run --extra test pytest -q -n 4 tests/capabilities/test_benchmark_study_projection.py tests/capabilities/test_benchmark_experiment_board.py tests/capabilities/test_benchmark_four_arm_contract.py tests/capabilities/test_benchmark_behavior_finding.py tests/capabilities/test_benchmark_terminal_followthrough.py tests/capabilities/test_benchmark_toolkit.py tests/benchmark/test_edgebench_managed_run.py tests/architecture/test_benchmark_experiment_identity_owner.py tests/architecture/test_content_digest_single_owner.py tests/canary/test_maintainability_ratchet.py tests/architecture/test_semantic_vocabulary_drift.py -> 830 passed, 20 skipped in 39.39s (20 pre-existing skipped cases also skipped on unmodified baseline)
  • Premerge & public boundary scan:
    • uv run --extra test loopx canary premerge --tier quick --from-git-diff --git-diff-base origin/main -> all 6 direct checks and all 3 catalog canaries (control-plane-maintainability-ratchet-smoke.py, semantic-vocabulary-drift-smoke.py, catalog-planner-smoke.py) passed (failures=0); exited manual_review_required (manual_holds: 1, benchmark_sensitive: benchmark adapter, scoring, runner, or evidence paths require explicit maintainer review before self-merge).
    • loopx check --scan-path ... -> ok: True, errors=0, warnings=0, checks=5, public boundary scan clean: 3 files.

Boundary & Rollback

  • Pure Python static-typing and local helper consolidation inside benchmark_toolkit plus two new entries in module_metric_baseline.json; zero schema, CLI, wire-protocol, or persisted receipt changes. Fully reversible by reverting this PR.

…iment_board and study_projection

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google (self_reported)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

已针对 PR #8978 exact head 74e9f9c0251f65dbf7cf0fb85ed3308bf497d38e 完成完整审查与本地验证(审查结论:APPROVE,无 P0/P1/P2 问题)。

动机 / Motivation

受影响调用方与运维者:所有消费 loopx/capabilities/benchmark_toolkit/experiment_board.py 与 loopx/capabilities/benchmark_toolkit/study_projection.py 的基准实验与 CLI 模块(包括 loopx/cli_commands/benchmark_experiment_board.py、loopx/cli_commands/benchmark_study.py、loopx/cli_commands/benchmark_dispatch.py、loopx/capabilities/benchmark_toolkit/behavior_finding.py、loopx/capabilities/benchmark_toolkit/catalog_entry.py、loopx/capabilities/benchmark_toolkit/four_arm_contract.py、loopx/capabilities/benchmark_toolkit/terminal_followthrough.py 与 loopx/capabilities/benchmark_toolkit/__init__.py),以及 loopx/canary/ 可维护性棘轮维护者。

修改前(origin/main 513b6f168a):experiment_board.py 共 1074 行含 58 处 Any、study_projection.py 共 1386 行含 59 处 Any(合计 117 处 Any 含 dict[str, Any])、存在跨文件重复的 _reject_unknown_fields/_optional_token/_timestamp/_finite_number 校验器与内联字典拆包/章节拼接样板,且两文件均未录入 loopx/canary/module_metric_baseline.json 零 Any 棘轮。修改后(74e9f9c0251f65dbf7cf0fb85ed3308bf497d38e):experiment_board.py 降至 1073 行、两文件 any_count 与 dict_any_count 均降至 0,复用共享校验函数并写入 experiment_board.py(lines: 1088)与 study_projection.py(lines: 1401)零 Any 基线棘轮。

可观测结果:module_metrics 返回 eb: {'lines': 1073, 'any_count': 0, 'dict_any_count': 0} 与 sp: {'lines': 1386, 'any_count': 0, 'dict_any_count': 0};mypy --strict --follow-imports=silent 两文件 0 错误,全仓 mypy(17 个源码文件)0 错误,ruff 全过;11 组 benchmark/架构/棘轮单测 830 passed, 20 skipped in 39.39s(该 20 条跳过与 origin/main 基线一致);canary premerge --tier quick 6 项直接检查与 3 项目录金丝雀均通过(failures=0,因触及 benchmark_toolkit/* 按预期报告 manual_holds=1 benchmark_sensitive 人工门禁);loopx check --scan-path ... errors=0, warnings=0。

非目标边界:本 PR 不修改 benchmark 实验行、研究看板、四臂对比或上传回执的运行时 JSON schema、错误提示文案或校验语义,保留 behavior_finding.py 引用的全部兼容内部符号,亦不授予基准敏感面合入权限。

改动思路 / Approach

全部强类型收口与重复样板合并均收敛在 loopx/capabilities/benchmark_toolkit/experiment_board.py 与 loopx/capabilities/benchmark_toolkit/study_projection.py 原生边界内,未新增并行决策层,亦未修改任何下游调用方或 behavior_finding.py。

当前 PR 边界:仅触及 experiment_board.py 与 study_projection.py 的零 Any 强类型收敛/公共校验复用与局部助函数抽离,以及 loopx/canary/module_metric_baseline.json 中对应的 2 条零 Any 棘轮基线。

  • 统一窄化契约(experiment_board.py:86-98、study_projection.py:65-87,438-449):以 Payload = dict[str, object] 与 PayloadView = Mapping[str, object] 替代两文件全部 Any / dict[str, Any],抽出 _as_map、_as_list、_as_map_list、_as_int、_sorted_value_counts、_strip_domain_state_key、_require_bool_arg、_require_str_arg 消除未窄化索引与重复枚举计数模板。
  • 消除跨模块校验重复并保兼容(study_projection.py:55-63):从 .experiment_board 复用 _finite_number、_optional_token、_reject_unknown_fields、_timestamp,同时完整保留 behavior_finding.py 导入的 _active_envelopes、_bounded_text、_finite_number、_reject_unknown_fields、_token 内部接口;在 experiment_board.py:908-930 抽出 _join_codes 与 _render_table_section,在 study_projection.py:785-813 抽出 _resolve_upload_candidate。
  • 零 Any 棘轮锁定(loopx/canary/module_metric_baseline.json:17-26):按字母序在 planner.py 与 catalog.py 之间锁定 experiment_board.py(any_count=0, dict_any_count=0, lines=1088)与 study_projection.py(any_count=0, dict_any_count=0, lines=1401)。

具体改动 / Changes

本次 diff 仅触及 3 个文件(656 insertions(+), 647 deletions(-)):

  1. loopx/capabilities/benchmark_toolkit/experiment_board.py(+280 / -281,1074 -> 1073 行,58 Any -> 0):清除全部 Any,增加 Payload/PayloadView、_as_map、_as_list、_strip_domain_state_key、_join_codes、_render_table_section,保留全部既有 docstring 与注释。
  2. loopx/capabilities/benchmark_toolkit/study_projection.py(+366 / -366,1386 行,59 Any -> 0):清除全部 Any,复用 .experiment_board 共享校验器并抽离 _as_map_list、_as_int、_sorted_value_counts、_require_bool_arg、_require_str_arg、_resolve_upload_candidate 与 Sequence[PayloadView] 聚合入参。
  3. loopx/canary/module_metric_baseline.json(+10 / -0,L17-26):注册上述两文件的零 Any 与行数上限基线。

对主干的风险 / Risks

  • 兼容性与输入窄化:_as_map/_as_list/_as_map_list/_as_int 针对边界非法类型延续原模块校验与空值回退,behavior_finding.py 与 CLI/能力调用方无需改动,全仓 17 个 mypy 目标 0 错误、830 passed, 20 skipped(与 origin/main 基线跳过数相同)。
  • 基准敏感面与 CI 状态:因触及 loopx/capabilities/benchmark_toolkit/*,canary premerge --tier quick 按策略返回 manual_review_required(manual_holds=1: benchmark_sensitive,failures=0);远端 Sign-off、inspect-pr-diff、dependency-review、loopx-codex-provider-routing、loopx-repo-health、loopx-obelisk 已 SUCCESS,其余 checks(build、loopx-core、changes、prepare、Summary)尚在运行中,需维护者审查与授权后方可合并。
  • 本地完整验证结果(exact head 74e9f9c0251f65dbf7cf0fb85ed3308bf497d38e):
    • module_metrics: eb: {'lines': 1073, 'any_count': 0, 'dict_any_count': 0},sp: {'lines': 1386, 'any_count': 0, 'dict_any_count': 0}
    • uv run --extra test mypy --strict --follow-imports=silent ...: Success: no issues found in 2 source files
    • uv run --extra test mypy: Success: no issues found in 17 source files
    • uv run --extra test ruff check ...: All checks passed!
    • uv run --extra test pytest -q -n 4(11 个 benchmark/架构/棘轮/词表测试文件): 830 passed, 20 skipped in 39.39s
    • uv run --extra test loopx canary premerge --tier quick --from-git-diff --git-diff-base origin/main: 6 direct + 3 catalog canaries passed (failures=0), manual_holds=1 (benchmark_sensitive)
    • loopx check --scan-path ...: ok=True (errors=0, warnings=0, checks=5)

我的整体评价 / Verdict

  • Long-horizon 影响:improved — 彻底清零 experiment_board.py(58 处)与 study_projection.py(59 处)共 117 处 Any、复用共享校验并锁定棘轮基线,消除后续基准分析演进的隐式类型漂移风险。
  • User-experience 影响:not_applicable — 内部 Python 静态类型收口、校验复用与 JSON 基线登记,不改变 CLI 参数、输出结构或前端界面,visual_review 为 not_applicable。
  • 可观测语义 / 复用 / 词表:observable_semantics 为 equivalent,repository_reuse 为 reused,authority_semantics 与 semantic_alignment 均为 aligned(candidate_decision="reuse_existing")。

改动聚焦、严格类型检查与本地 830 条用例均充分验证且无 P0/P1/P2 发现项;因触及 benchmark_toolkit/*(benchmark_sensitive)且为作者自有 PR,合并需等待远端 CI 全绿并获得维护者明确授权。

English verdict: APPROVE exact head 74e9f9c (author-owned COMMENTED approval review); eliminates all 117 Any usages across loopx/capabilities/benchmark_toolkit/experiment_board.py (58 -> 0) and study_projection.py (59 -> 0), consolidates shared validators and upload/table helpers, and locks zero-Any baselines in module_metric_baseline.json, verified by strict/repo mypy (17 files), ruff, 830 passed / 20 pre-existing skipped pytest tests, and quick premerge canaries (0 failures, benchmark_sensitive manual hold).

@huangruiteng
huangruiteng merged commit 05618da into main Nov 18, 2026
33 of 38 checks passed
@huangruiteng
huangruiteng deleted the codex/benchmark-study-board-slim-20261118 branch November 18, 2026 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants