Skip to content

Defer Yardstick evaluation until explicitly selected - #9059

Merged
huangruiteng merged 1 commit into
mainfrom
codex/yardstick-on-demand-20261119
Nov 19, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/yardstick-on-demand-20261119

Conversation

@cocolord

Copy link
Copy Markdown
Member

Goal and delivered behavior

An ordinary standalone Core task can use the existing C1 context and replay path without loading the optional Yardstick evaluator. C1 assembly retains the existing Explore guidance, evidence references, and context rewrite receipts through a lightweight delivery module. Explore returns the same empty response for a disabled graph and plan. Selected graph reads retain full validation and acquire the Yardstick evaluator only when the current Goal has explicitly enabled it. Paging, evidence capture, and prior records keep their existing owners.

The Core distribution and Yardstick RFC pairs record this boundary. It adds no setting, provider registration, stored-state format, or authority. Related to #5072. Base main@5cfd6ae7d571.

Verification

Verified e67b5d911a8bf44b6eab49547f2b25e31ecb4127 with source-built macOS arm64 Core wheels and local synthetic tasks.

  • In a fresh Node process, importing C1's Turn envelope loads two targeted Explore/Yardstick modules, down from seven on the base. Existing pre-work guidance and evidence-reference delivery match the base byte for byte.
  • On File and SQLite, the base wheel completes task work but fails the unused-module assertion for all four monitored modules. The candidate passes both probes, exercises independent validation, and replays the original result without a second host call, authority write, or debit. The installed Core identity and help check also passes.
  • The live handler dispatch test covers disabled reads, malformed pagination, explicit Explore with Yardstick disabled, first evaluation load, exact repeated reads, and turning the capability off again. Focused CLI fixtures cover historical recovery after disable. All 153 focused TypeScript tests pass.
  • TypeScript typecheck, mypy, focused Ruff, bilingual RFC diff, semantic advisory and quick premerge pass. Premerge reports one inherited maintainability advisory and no manual holds. A local trial merge with the current package and CLI proposals had no textual conflicts.

These checks qualify load selection and task continuity in the tested profile. Model input size, latency and outcome costs remain open.

Signed-off-by: cicolord <cicolord@outlook.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · reasoning_effort=max

Exact head e67b5d9

批准这个有界变更。实际安装、读回与拒绝路径覆盖了按需加载和原任务连续性。

动机 / Motivation

受影响的是在 standalone Core 完成普通任务,以及在已启用 Yardstick 的 Goal 上通过 Explore 读取证据的使用者。

普通 C1 启动会连带载入 Explore 图投影和 Yardstick 代码。候选由同一交付模块提供指引和精确引用,只有实际选择图及 Goal 评估时才载入 Yardstick。

File 和 SQLite 的已装 Core 都完成并重放原任务,保持一次宿主执行及一次扣额。显式选中后的读回与关掉评估后的历史证据仍可取用。

这次验证覆盖 macOS arm64 的独立 Core wheel、两个存储和隔离数据。真实模型收益、完整成本、其他平台和最终分包组合仍待各自验收。

正式分包候选还需同一版本集成读回。节省的启动耗时、常驻开销和真实任务收益留待配对测量。

改动思路 / Approach

本次边界是在既有入口按需载入 Explore 投影和 Yardstick,保留普通 C1 的上下文与重放。把已用的指引和等值引用抽到同一轻量模块,集中划分空读、图投影和 Yardstick。Goal 设置、任务范围、证据和回执继续经原 owner 校验。

只延迟最外层 Explore 仍会让 C1 通过两处静态导入加载同一链路。共享现有文字、字段和引用后,读端入口可以按请求分支,写端保存与回滚仍沿原实现。 同一交付模块也服务于纯重规划,保留先读指引和实际可用的后续命令。

具体改动 / Changes

完整差异为 15 个文件,新增 398 行,删除 256 行。两个 RFC 均同步更新中英文。

  • projectEmptyExploreTurnContext。仅在规范空读上返回原关闭视图。其他请求仍经完整参数、分页与归属检查。 当前代码
  • exploreResultWritebackAffordance。统一从同一个导出取得原指引、可选字段与模板。 当前代码
  • referenceExploreResultDelivery。成功收尾只在完全等值时替换重复附件,未完成回执保持原输入。 当前代码
  • explore.turn_context。分离空读、全量图读取,以及只有启用时加载的 Yardstick 读投影。 当前代码
  • projectExploreTurnContext。沿用 Todo、证据与版本校验,同一评估模块随精确页重读传递。 当前代码

C1 的 Turn envelope 和 capability hooks 继续使用原文、原 owner 及精确引用。写回规范化继续使用原实现。C1、C8、C11、C12 和 M1 分别核对普通工作、原轮恢复、错误版本与范围及开关读回。后续 M3 保留实际目标和成本测量。Yardstick RFC

普通安装的依赖闭包与原可用入口按 I7、I8 和 V1 检查。分包 RFC

对主干的风险 / Risks

同一 Node 24 观察的 8 个模块中,C1 冷导入由 7 个降为 3 个。禁用时空读不再载入图。只有图和 Yardstick 都启用时才首次载入评估投影。

独立生产 dispatcher 对照包含 19 个输入及 7 种拒绝,并核对 7 组指引和回执。输出和错误一致,过期页、缺少修订、坏输入与其他 Agent 的任务均拒绝。 代表性数据包含可翻页的同任务历史、一个新任务、错 Agent 任务,以及关闭后仍可读的旧测量。一并复核未知指引保持完整,成功可引用的附件与待重试附件各沿原规则。

两组 TypeScript 原路由与上下文用例共 153 项通过。真实候选 wheel 的身份及两种存储的首次和恢复用例 3 项通过。主干同组的两个普通 Turn 已完成再重放,随后在惰性加载断言处失败,能够区分候选修复的依赖链。TypeScript 类型检查通过。

在相同 Python 3.12 环境、相同 187 个测试上,主干和候选均 173 通过、14 失败。基线与候选的失败用例、位置和 KeyError 一致,两项缺 reference_file、八项缺 writeback、四项缺 command_template。 此批保留诊断,另用安装态的普通重放和上述生产 dispatcher 对照核验本次变化。quick premerge 通过 5 项直接检查与 4 项选中检查。维护性 advisory 的两项发现均来自同样的主干文件,决策点和语句数一致。工作区补齐 TypeScript 依赖后重验。

我的整体评价 / Verdict

普通用法保留原任务、证据、作用域和重放入口。显式读取时仍能取得完整评估投影,禁用时的普通 Turn 减少实际依赖加载。采纳现有评审经验中的原入口任务与结果读回。在两个真实安装的 Core 上区分首次采用及过期、拒绝和重放。此改动不调整页面或操作布局。 检视的 Journey 是 standalone Core 和原 Explore 命令。变更在上下文和模块加载,未改渲染界面,现有设置及读取入口继续使用原件。

English verdict: APPROVE. At e67b5d9, paired installed wheels preserve selected Explore output, scope refusal, captured guidance, and one-effect task replay. Yardstick loads only after the enabled graph read. The separately checked source failures also reproduce on the base.

@huangruiteng
huangruiteng merged commit c22ad13 into main Nov 19, 2026
36 of 48 checks passed
@huangruiteng
huangruiteng deleted the codex/yardstick-on-demand-20261119 branch November 19, 2026 05:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants