Skip to content

feat(progress-review): close the drift loop with typed receipts and a recorded differential - #4854

Open
songoow wants to merge 16 commits into
loopx-project:mainfrom
songoow:codex/jev-drift-shadow-upstream
Open

songoow wants to merge 16 commits into
loopx-project:mainfrom
songoow:codex/jev-drift-shadow-upstream

Conversation

@songoow

@songoow songoow commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Closes the progress-review loop through existing LoopX contracts and ships a reproducible differential. Revised on 2026-09-22 after an external review of the first closed-loop version; the four defects it found are fixed deterministically (table below), and the matrix was re-recorded with question set scoped-progress-sentinel-v2.

  • Core capability progress-review-sentinel (default off). Per-goal policy control_plane.progress_review with off / shadow / assist, a drift signal, a drift_threshold, and a pinned contract_revision. A strict progress_review_receipt_v0 contract whose drift booleans the core recomputes from the typed judgments and rejects when they disagree. A pure trigger joins receipts to run rows by turn identity (with Agent/Todo agreement), counts consecutive completed drift receipts bound to the pinned revision, skips at most two newest still-pending evaluations, and in assist raises the existing autonomous_replan_obligation after the typed repeat fuse. The refresh-state writeback judges an acknowledgement against the same obligation, so an accepted replan re-arms it. The policy vocabulary lives in the control plane; the capability re-exports it, so the control plane never imports the capability layer.
  • Policy surfaces. configure-goal --progress-review-mode|--progress-review-signal|--progress-review-drift-threshold|--progress-review-contract-revision|--clear-progress-review-configuration, the configuration catalog, the Dashboard capability editor (assets repackaged), the chat configuration API and localization.
  • Observer (packages/loopx-jev). The model receives only operator basis fields plus scoped material, never goal identity (byte-identical requests across identities are pinned by a test). Three Noul questions ask about the change between checkpoints, not the after state; drift init --runtime-root writes a pending receipt on queue and the evaluated receipt on drain; drift label records private truth; status reports observer_influence and the fact that consumption is governed by the Goal policy.
  • Differential harness and closed-loop test. loopx-jev sentinel compare replays a frozen 16-sequence matrix against committed recordings; test_closed_loop.py runs one real refresh-state sequence through off, shadow, unpinned assist (blocked, reason reported), pinned assist (obligation raised, shown in loopx status, acknowledged, re-armed).

Review findings and fixes

Finding (2026-09-21 review) Fix in this revision
Harness passed a goal_id derived from the case name into model input Model state is a whitelist of basis fields; hashed goal ids in the harness; anonymity test
noul gated on behaviour change: unrelated features passed, negative experiments were flagged Rule progress_review_signal_rule_v1 gates on serves_acceptance and evidence_increment, both asked about the marginal change; core recomputes and rejects inconsistent receipts
Receipts only had to agree with each other, not with the current goal contract assist requires a pinned contract_revision; other revisions are stale and never counted; unpinned assist reports contract_revision_unpinned
Turn-id join ignored Agent/Todo; an unevaluated newest run dissolved the streak Identity agreement required; ambiguous fallbacks unattributed; pending receipts skipped within a bound of two
worker_influence: none misdescribed assist Observer reports observer_influence + core_consumption; docs state that assist changes the work contract (required obligation) while granting no new authority

CI failures on the previous head are addressed: mypy Callable type, the control-plane import boundary, the ratchet oversized function (helper extracted), the packaged dashboard assets, and a rebase onto 0d464d2ac that picks up upstream fixes for the goal-channel and SQLite Node tests.

Recorded differential (committed v2 live recording, jev-1.13.0, 35 rounds, 34 completed, 1 failed closed)

Typed repeat fuse noul receipts choice receipts
Drift sequences flagged at their gold round 0/9 (invisible by construction) 9/9 5/9
Reaching the assist obligation at threshold 2 0/9 9/9 2/9
Real on-goal upstream commits falsely flagged 0/7 0/7 0/7
Premature flags inside mixed sequences 0 0 0

Cosmetic sequences flag at round 1; mixed sequences flag at exactly their drift round (serves_acceptance 0.06–0.13 on post-implementation churn, versus 0.62–0.90 under the v1 wording). The negative probe (evidence_increment 0.85) and the necessary failing test (serves_acceptance 0.73) stay unflagged. A second independent live run reproduced every outcome on all 16 sequences. Client latency 1.45–1.49 s median and 2.9 s p95 in the v2 runs (0.74–0.81 s median in earlier runs; network and provider time), 1890 median input tokens.

Caveats. The v2 wording was revised after seeing the v1 misses on these constructed sequences, so they are not held-out evidence for the wording; the seven real commits were not used to tune anything. Constructed gold labels come from their author. One of 35 answers was rejected by the strict decoder (selected label ≠ argmax) and fails closed. No claim of reduced wasted work or task-completion improvement: that needs a held-out set labelled with drift label and a separately authorized intervention study.

Verification

  • 242 tests across the package (real Git, real refresh-state, receipts, labels, pending lifecycle, anonymity, matrix strictness, replay determinism, closed loop) and the touched core suites (trigger, capability, configuration UI, chat API, registry, refresh gate, contract quality, architecture boundary, maintainability ratchet).
  • ruff check on the CI scope, kernel mypy (22 files), strict mypy on the new modules, docs-governance-smoke, repository-hygiene-smoke, progress-review-sentinel-smoke, packaged dashboard rebuild.

Draft exit and merge readiness

This PR asks for stage 0 of the readiness ladder in loopx/capabilities/progress_review/README.md: a default-off registered capability whose shadow records receipts and whose assist requires a pinned contract revision. Author-side conditions, all met on this head:

  • CI green on the exact head (26 pass, 5 conditional skips)
  • External review findings fixed deterministically, not by tuning the model
  • Differential replays from committed recordings without a credential; second live run reproduced it
  • Bilingual capability README, operation guide and decision record, including an adoption path and the readiness ladder
  • No new authority: no pause, gate, settlement or acceptance; the core never calls a model or reads a raw delta

Merging a control-plane change is the maintainers' decision and is never self-merged. Stages 1 (shadow with labels on real Goals) and 2 (one pinned assist Goal) are operator studies made with the tool after merge; stage 3 (defaults, escalation, pause) needs a separately authorized intervention study and is out of scope.

Boundaries

No pause, user gate, Turn settlement or Goal acceptance authority; escalation legs are future work. The core never calls a model, never reads a raw delta, never overwrites progress_observation.

@songoow songoow changed the title feat(jev): add optional D1 progress shadow observations feat(jev): add optional progress review from scoped file changes Sep 21, 2026
@songoow
songoow marked this pull request as draft September 21, 2026 09:40
@songoow
songoow force-pushed the codex/jev-drift-shadow-upstream branch from 1f2f218 to 6be1537 Compare September 21, 2026 10:59
@songoow songoow changed the title feat(jev): add optional progress review from scoped file changes feat(progress-review): close the drift loop with typed receipts and a recorded differential Sep 21, 2026
@huangruiteng

Copy link
Copy Markdown
Collaborator

Draft design review:把进展评估接入同一条 replan 闭环

针对 head 6be153723eaefddc3d251a7a2ddf5eacfa4cf534。本次是围绕“与现有 replan 融合、改进触发语义并伴随 TS 重构”的设计反馈和定向代码验证,不是全量 merge-readiness 认证。PR 当前仍为 Draft;loopx pr-review 为该 head 返回 no-action,fresh-audit 又要求已有有效结论,因此本次不伪称完成工具未生成的正式评审计划,也不发布批准。

动机

我支持研究这个问题:Agent 可以持续报告 advanced,而实际工作没有推进当前目标。精确重复检测确实不能覆盖所有这种情形。但更有价值的交付应当是增强现有 replan 的证据和触发质量,而不是让一个外部模型形成第二套规划权威。

这与总路线的长程纠偏、可恢复执行有关,也可以成为 TS 规则收敛的合理切片。参考 overall roadmapTS migration RFC 的 Delivery semantics原 semantic-assistance RFC 的 M0 讨论稿收录不能代替 assist 的产品采用决定;本评论提出整合方向,不认证 Jev 的模型质量或默认启用资格。

改动思路

目前 PR 已做对一件重要的事:external_progress_review_drift 最终进入已有 autonomous_replan_obligation,status 与 refresh writeback 也使用同一个 receipt loader。这是应该保留的方向。

建议进一步收敛为:

运行历史 / Todo·Vision 事实 / 可选外部评估 receipt
        ↓ 绑定 Goal·Agent·工作对象·当前基准,校验时效和去重
领域触发器 → 类型化触发事实 → 明确的优先级与触发策略
        ↓
现有 autonomous_replan_obligation / action packet
        ↓
现有 replan_semantics.ts 校验实际 writeback → ACK / 后续重触发

模型负责回答有限的语义问题;确定性规则决定何时要求复盘;原有 Agent、Todo/Vision 与 writeback owner 决定如何处理。外部评估不能取消已有硬义务,模型不可用也不能使原有 replan 失效。复盘允许有证据地继续原计划,不应强迫每次制造新 Todo 或改方向。

具体改动

本 head 的范围包括:内置 capability、Goal 配置和 CLI/API/Dashboard 编辑器;receipt codec/存储/投影;run-history trigger 与 refresh-state ACK 接入;可选 observer 的 capture/queue/drain/provider 调用;对照 harness、合成与真实提交 fixture、双语说明。以下意见主要针对 assist 闭环,其余表面尚未得到全量合并资格验证。

1. [P1] 先修复“原样重复也能结清漂移义务”

obligation builder 的 review 分支 只携带 review 信息,没有像 typed_progress_repeat 一样带入 progress_baseline;trigger 本身也没有提供该 baseline。

我在隔离 runtime 中经过真实 refresh_state_run 和 receipt 写入复现:两轮各有 surface/hypothesis/evidence 的 advanced 记录,被两份完成的 drift receipt 触发 assist 后,原样提交最新一轮的同一 surface、hypothesis、evidence,仍得到 semantic_delta.accepted=truenew_surface/new_hypothesis、durable ACK,随后 obligation 消失。把同一 observation 作为 baseline 调用已有 novelty evaluator,则返回 accepted=false

原因是 semantic_delta_from_writeback 找不到 baseline,semantic_progress_delta 就把已有标识算作新增。这不是模型准确率问题,而是当前闭环可以被零增量写回解除。

最小修复:将被评估窗口的进展 baseline 绑定到 obligation,并在既有 TS outcome owner 中明确该 trigger 可以接受什么结果。进一步的目标关系修复应通过已有证据关联的 Vision/path 或 successor/blocker 契约完成;不要要求“外部模型再次说好”才能 ACK。回归必须包含真实 refresh:同 observation/同 evidence 被拒绝,有实际新证据的合法处理可结清,同 Turn 重试不会多次结清。

2. [P1] 两个 drift signal 目前不是同一产品语义,不能只当可替换的评分方式

noul_drift_signalbehavior_change >= threshold 直接判为非漂移,并且不使用已经采集的 evidence_increment。使用合法响应、阈值 0.6 重放 decoder:

输入事实 Noul drift Choice drift
行为变化 0.99、服务目标 0.01、新证据 0.01;Choice=off_goal/no_new_evidence false true
行为变化 0.01、服务目标 0.01、新证据 0.99;Choice=unknown/new_evidence true false

第一行表明做了实质性但无关的改动也可逃过默认规则;第二行表明有新证据、目标关系尚待核实时,仍可被直接标成漂移。这里证明的是规则的确定性差异,不是 live 模型错误率。

建议先分清三件事:目标关系、相对上一窗口的有效增量、有价值的目标内探索/前置工作behavior_change 只能作为增量证据之一,不能单独证明方向正确;“有新证据”也不能自动洗白无关工作,必须看其与当前问题的关系。把 off-goal、on-goal-but-stalled、useful-exploration、legitimate-wait、unknown 分开,不要压成一个可被 provider 名称切换含义的 bool。Jev 的 Choice/Noul 和原始概率可留在 provider 诊断中;核心消费统一的有限语义事实。

3. [P2] receipt 彼此版本相同,不等于仍适用于当前基准

trigger 只比较被计数 receipts 之间的 contract_revisioncontext loader 没有读当前评估基准版本。

复用 PR 的真实 capture/refresh/drain fixture(注入模型答案)后,我先生成两份 completed receipts,再修改 observer basis 的 objective/acceptance。当前 basis hash 已改变,但读回的旧 receipts 仍是 completed,仍生成完全相同的 required obligation。现有版本测试只覆盖“一新一旧 receipt 不能凑成连续两轮”,未覆盖“基准已变、尚无新 receipt”的窗口。

assist 需要把 operator study basis 与已生效 Goal/acceptance 基准的绑定讲清楚,并在读取/使用时比对当前 revision。旧 receipt 可保留作历史,不应继续生成当前义务。回归应覆盖已完成结果之后的基准变更、推理过程中变更、晚到结果和 scope 变更;不能只靠 observer 在发送前后检查。

4. [P2] 共用“有效工作轮次”,不要让 bookkeeping 行截断判断

新 scanner 对首个无 receipt 的 run 直接 break,没有使用原 replan 的 neutral-classification 规则。给上述真实 runs 前加入不带 turn identity 的 quota_slot_voided bookkeeping 行后,外部 obligation 从 required 变为 None;同一类型在既有 replan policy 中属于 neutral。这是对生产 reducer 的合成边界测试,没有声称完整执行了 quota-void journey。

应复用有类型的工作轮次投影,区分配额记账、同 Turn 重试、真实未评估的工作轮次。真正 unknown 可以阻止形成新的连续证据,但不能被描述成进展正常,更不能默默充当处理既有义务的 ACK。补上 interleaved spend/void、异步 receipt 未到达和两个 Agent 交错历史的测试。

建议伴随的 TS 重构

不建议本 PR 顺手迁移整个 planner。可以做一个完整而有限的切片:先用 characterization 固定现有 run-history repeat、monitor、periodic-review 的语义和优先级,再把这些历史触发规则与外部 receipt 触发统一到 work_items 下的 TS 纯决策边界。Python 留下历史/receipt 读取和调用适配;原有 Goal-frontier 领域规则继续由原 owner 产生事实,不复制一遍。

触发事实用 discriminated union 表达各自必需字段:重复进展需要 baseline,monitor 需要 target/due facts,外部评估需要基准版本、工作身份和证据窗口。共享 scope、logical-turn 去重、身份与 ACK 匹配,保留各领域不同的阈值、等待条件和合法解除结果。status 与 writeback 调用同一边界;不要为每个 trigger 添加一次独立 RPC,也不要保留第二份 Python 规则。

这样重构的收益是消除重复的身份/时效/历史解释,后续 File、SQLite、PostgreSQL 提供相同事实即可使用相同策略;它本身不等于完成 provider 默认切换。

对主干的风险

default-off 和模型调用放在外部 consumer 是好的隔离措施。但 assist 的 required=true 已经是影响下一步执行的义务,不能仅用 advisory_evidence_only 暗示它没有控制效果。要区分“模型无直接写权限”与“用户启用的确定性策略可以据此要求复盘”。核心 action 文案也应保持 Goal 通用性,避免把“修改文件/改变运行行为”写成所有研究、写作或运营 Goal 的进展标准。

本次执行:

  • exact-head tests/capabilities/test_progress_review.pytests/control_plane/test_external_progress_review.pypackages/loopx-jev/tests97 passed,包含已有 off/shadow/assist、真实 refresh/status/ACK 测试。
  • 额外隔离反例:上述原样 ACK、信号差异、完成结果后的 basis 变更、neutral bookkeeping 都得到所述结果。
  • git diff --check 在 7 个 fixtures/sentinel/real/*/COMMIT.txt 报新增 EOF 空行,未把它报告为通过。
  • 未运行新的付费 live inference、完整 frontend packaged journey、全部 CI 或 provider 切换验收;这些不是本次设计反馈已经通过的证据。

后续对照需要包含有正常上下文的现有 Agent/replan 流程,而不仅是精确重复 fuse。衡量首次正确且可行动的触发时间、误触发率、合法探索损失、复盘消耗与触发后是否恢复有效推进。连续两次同模型判断不等于两份独立证据,不能直接换算成更高置信度。当前 recorded matrix 有实验价值,但还不能证明任务完成收益。

我的整体评价

建议沿“现有 replan 的语义证据增强 + 有界 TS 触发规则收敛”继续。先修复上面的闭环与有效性问题,再讨论 assist 采用;保留 shadow 作为校准入口。代码上可以按同一主题分两个可独立复核的阶段:① 原规则 characterization/迁移,证明 off-path parity;② 新语义事实与有效性/解除契约接入,证明正反闭环。若仍保持一个 PR,也请用这两个逻辑阶段组织 commits 和验证。

不需要为了这个方向重写整个 Goal-frontier、另造 planner 或扩大模型权限。最关键的完成标准是:触发有当前有效证据,处置对应触发原因,确认能证明问题已处理或有证据地保留原计划。

English verdict: REQUEST_CHANGES - design feedback at 6be1537, not a merge-readiness certification. Reuse the existing replan lifecycle and converge the bounded history-trigger policy in TypeScript. Fix the reproduced identical-observation ACK bypass, distinguish alignment from behavior/evidence novelty, bind completed receipts to the current approved basis, and preserve neutral bookkeeping semantics. The 97 focused tests pass, but the additional counterexamples expose gaps outside that coverage; no new live-model quality or full integration approval is claimed.

Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
Signed-off-by: song <liusongstep@gmail.com>
…n trigger

Add the default-off progress_review policy, the strict progress_review_receipt_v0 contract and a pure trigger that counts consecutive completed drift receipts joined to run rows by turn identity. In assist the trigger raises the existing autonomous replan obligation after the typed repeat fuse; unknown, abstained, failed or missing receipts break the streak, an acknowledged replan re-arms it, and a changed goal contract invalidates earlier receipts. The same context is read by status projections and by the refresh-state writeback, so an acknowledgement is judged against the obligation that status shows. The core imports nothing from the observer package and adds no pause, gate or acceptance authority.

Signed-off-by: song <liusongstep@gmail.com>
…aces

Register the builtin progress-review-sentinel capability and expose its per-goal policy (off, shadow, assist; drift signal; drift threshold) through configure-goal flags, the configuration catalog, the Dashboard capability editor, the chat configuration API and dashboard localization, with a bilingual capability README. The policy is default off and a malformed block fails closed to off.

Signed-off-by: song <liusongstep@gmail.com>
The observer asks three Noul questions (behavior_change, serves_acceptance, evidence_increment) next to the two Choice questions, derives the noul and choice drift signals with the configured label threshold, and, when bound to a runtime root, writes one progress_review_receipt_v0 per evaluated event under the goal runtime. Probabilities inside the undecided band are not decisions, so an evaluation with no decided answer still abstains. drift label records a private human truth and drift status reports agreement per signal; labels never enter a receipt.

Signed-off-by: song <liusongstep@gmail.com>
…d-loop test

loopx-jev sentinel compare replays a frozen 16-sequence matrix (6 constructed cosmetic-drift sequences, 3 mixed sequences that drift after genuine work, 7 real upstream commits labelled on-goal) against recorded provider answers and reports, per sequence, when the typed repeat fuse fires, when each receipt signal first flags drift, when assist would raise the obligation, and every false flag; --live records fresh answers. The closed-loop test drives one real refresh-state sequence through off, shadow and assist, checks loopx status, accepts a real acknowledged replan and verifies the re-arm. The smoke replays the committed recording without a credential.

Signed-off-by: song <liusongstep@gmail.com>
Update the operation guide and the decision record in both languages, link the capability from the README capability tables and the research RFC, and state what the committed live recording shows and does not show: the typed fuse fires on none of the sixteen sequences, the noul signal flags six of nine drift sequences at round one with no false flags on real commits, cosmetic churn after a landed implementation is missed, and one executed negative probe was flagged in two of three live runs.

Signed-off-by: song <liusongstep@gmail.com>
…eipt trigger

Move the policy vocabulary into loopx.control_plane.work_items.progress_review_policy so the control plane never imports the capability layer, and add a pinned contract_revision to it. The receipt contract now carries a signal_rule_version and an optional reason; the core recomputes the drift booleans from the typed judgments with rule v1 (serves_acceptance and evidence_increment, both about the change between checkpoints; behaviour change recorded only) and rejects a receipt whose booleans disagree. The trigger requires the pinned revision, requires Agent/Todo agreement on a turn-id match, never attributes an ambiguous fallback, skips at most two newest pending evaluations, and is extracted into a helper so autonomous_replan_obligation_from_runs stays within the maintainability ratchet. Unpinned assist raises nothing and the summary says why.

Signed-off-by: song <liusongstep@gmail.com>
…s-review policy

Expose --progress-review-contract-revision through configure-goal, the configuration catalog, the Dashboard editor, the chat configuration API and the capability entry. An empty value clears the pin; assist without a pin is blocked. Catalog boundaries state that assist changes the Agent's work contract while granting no new authority.

Signed-off-by: song <liusongstep@gmail.com>
… goal evidence

The request carries only operator basis fields plus scoped material, never goal_id or study bookkeeping; a test pins byte-identical requests across identities and the harness uses hashed goal ids. Question set v2 asks serves_acceptance and evidence_increment about the change between checkpoints, and the drift booleans come from the core-owned rule. The observer writes a pending receipt when it queues an event and replaces it after evaluation, reports the contract revision to pin, and describes itself as observer_influence none with consumption governed by the Goal policy.

Signed-off-by: song <liusongstep@gmail.com>
Fresh live recording of the 16-sequence matrix under scoped-progress-sentinel-v2 and its deterministic expected summary. A second independent live run reproduced every first-flag round, obligation round and false-flag count.

Signed-off-by: song <liusongstep@gmail.com>
…ization

Clean rebuild of the tracked Personal Workspace assets after adding the progress_review capability copy, so the packaged-assets check matches the source.

Signed-off-by: song <liusongstep@gmail.com>
Document rule v1, the contract pin, pending and identity semantics, the two configuration layers, and that assist changes the work contract. Record the v2 differential (noul 9/9 at the gold round, 0/7 false flags, reproduced by a second live run) together with the caveat that the wording was revised after seeing the v1 misses on the constructed sequences. Keep only the final E7 evidence row in the RFC.

Signed-off-by: song <liusongstep@gmail.com>
@songoow
songoow force-pushed the codex/jev-drift-shadow-upstream branch from 6be1537 to a49a112 Compare September 22, 2026 13:32
@songoow

songoow commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

Revised after the external review of head 6be153723. All four determinism findings are fixed in code rather than by tuning the model, and the matrix was re-recorded with a v2 question set that removes goal identity from the model input:

  • Label leak: the harness no longer passes a case-derived goal_id; the model state is a whitelist of operator basis fields, and a test pins byte-identical requests across identities.
  • Drift rule semantics: noul now gates on serves_acceptanceevidence_increment, both asked about the change between checkpoints; behavior_change is recorded only. The core recomputes the booleans from the typed judgments and rejects a receipt that disagrees.
  • Contract currency: assist requires --progress-review-contract-revision pinned to the observer basis printed by drift init; other revisions are stale, unpinned assist reports contract_revision_unpinned.
  • Identity and lifecycle: turn-id joins require Agent/Todo agreement, ambiguous fallbacks are unattributed, and up to two newest pending receipts are skipped so a streak neither grows nor dissolves while evaluation runs.

Re-recorded differential (two independent live runs agree on all 16 sequences): typed fuse 0/9, noul 9/9 flagged at the gold round and 9/9 reaching the obligation, 0/7 false flags on real commits, 0 premature; choice 5/9. Post-implementation churn now scores serves_acceptance 0.06–0.13 (was 0.62–0.90). Caveat kept in the docs: the v2 wording was revised after seeing v1 misses on the constructed sequences, so those are no longer held-out for the wording; the real commits were untouched.

CI: fixed the mypy Callable type, the control-plane→capabilities import edge (policy now lives in control_plane/work_items/progress_review_policy.py), the ratchet oversized function, and repackaged the dashboard assets; rebased onto 0d464d2ac for the upstream goal-channel and SQLite Node fixes.

Describe how an operator adopts the sentinel one Goal at a time (install, bind, wrap refresh, run the consumer, shadow, label, then pinned assist) and define the readiness ladder: stage 0 is this default-off preview, stages 1 and 2 are operator studies with labels and one pinned assist Goal, stage 3 needs a separately authorized intervention study. State that leaving Draft is an author-side condition met on this head while merging stays a maintainer decision.

Signed-off-by: song <liusongstep@gmail.com>
@songoow
songoow marked this pull request as ready for review September 22, 2026 15:10
@songoow
songoow requested a review from maxliux5 as a code owner September 22, 2026 15:10

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head review — e7afab6daedba91aee15a7f312510e50ac12245d

当前结论是 REQUEST_CHANGES。最关键的 P1 不是模型质量问题,而是 control-plane 的 discharge 语义仍有缺口:external_progress_review_drift 没有把已评估的 progress baseline 交给现有 replan 语义,因此完全相同的 observation 会被误判成新进展并解除义务。另有 7 个 fixture 文件未通过 git diff --check

动机

这个 PR 解决的是一个真实且重要的问题:agent 自报 advanced 并不能证明目标真的前进,连续的改名、格式整理或重复尝试可能掩盖 drift。PR 的目标是引入一个默认关闭、显式启用的 progress-review sentinel:外部 observer 只产生带 contract revision 的 typed receipts;shadow 只观测;assist 达到阈值后,把证据接入现有 autonomous_replan_obligation,但不获得 Todo、lease、quota 或执行权限。

这个边界总体是对的。它复用了现有 typed progress/replan vocabulary,并通过 off/shadow/assist、pinned contract revision 和 advisory receipt 把可用性、激活与权限分开;这比让 provider 直接写 Todo 或另起一套 scheduler/replan 状态机更符合当前架构。不过“触发 obligation”只是闭环的一半,另一半是 obligation 只能被真实的新语义证据解除。本 head 尚未满足这一点,所以完整交付目标还没有成立。

改动思路

主要链路是:loopx-jev 从 Goal basis、代码变化和既有 evidence 构造匿名请求并记录 typed review receipt;core 的 external_progress_review_context 只选择当前 pinned contract 的 receipts;external_progress_review_trigger 把达到阈值的 drift 变成 external_progress_review_driftautonomous_replan_obligation_from_runs 继续生成既有 obligation;最终 refresh_state_run 通过 semantic_delta_from_writeback 和 TypeScript replan semantics 判断本轮 writeback 能否 ACK。

配置与产品入口也形成了相对完整的路径:capability catalog、configure-goal、chat Goal configuration API、dashboard localization/packaged assets、双语 README/能力文档,以及独立的 packages/loopx-jev provider。默认 off;仅安装 package、存在 credential、worker 可用或已有 receipt 都不会激活 assist;shadow 不产生机器义务;assist 还要求 contract revision 被 pin。这些 counterfactual 在测试里有直接覆盖。

问题出在 core 复用边界的最后一步。现有 typed_progress_repeat trigger 会携带 progress_baseline,所以相同 surface/hypothesis/evidence 会被判为 repeat;新的 external trigger 只携带 review metadata 和 frontier_identity,没有 baseline。下游没有足够状态区分“触发 obligation 的旧 observation”和“本轮真正的新 observation”。

具体改动

  • loopx/capabilities/progress_review/**、catalog、Goal 配置 API 与 dashboard 增加 policy/config/readback,公开 offshadowassist 与 contract-revision pinning。
  • loopx/control_plane/work_items/external_progress_review.pyprogress_review_policy.py 负责读取/筛选 typed receipts 并形成 goal-neutral trigger;status.pystate_refresh.py 和 obligation projection 把它接入现有控制面。
  • packages/loopx-jev/** 实现可选 observer 的配置、协议、runner、HTTP worker、drift capture、sentinel matrix 与 receipt store,并附带 constructed/real fixtures 和 recorded responses。
  • README、双语 capability/RFC/package 文档说明激活、隐私、authority 和 rollout;tests 与 public smoke 覆盖默认关闭、shadow、assist、contract mismatch、matrix 结果和一个正向 ACK/re-arm 流程。

关键代码讲解

  1. external_progress_review_trigger:把当前 contract 的 drift receipts 压缩为一个 typed trigger。它正确保留 signal、threshold、evidence ids 和 contract revision,但没有保留触发窗口对应的 progress observation baseline。
  2. autonomous_replan_obligation_from_runs:统一拼装 typed repeat、dead monitor、blocked successor 与 external review 的 obligation。当前代码只在 typed_progress_repeat 分支写入顶层 progress_baseline;随后 external review 分支仅覆盖 frontier 和 review metadata。
  3. semantic_delta_from_writeback / semantic_progress_delta:这是实际 ACK gate。前者从 obligation/trigger 读取 baseline;读取不到时传 None,后者会把一个完整 observation 当成首次出现,于是相同 surface 与 hypothesis 被分类为 new_surface / new_hypothesis
  4. test_same_sequence_off_sees_nothing_and_assist_raises_the_obligation:很好地证明了 off/shadow/assist 与正向 re-arm,但 ACK 使用的是全新的 blocked observation,因此没有覆盖“把刚刚触发 drift 的 observation 原样提交”这个负向路径。

对主干的风险

P1 — 相同 observation 可以错误解除 external drift obligation

触发状态:assist 已因两个 current-contract drift receipts 生成 external_progress_review_drift,agent 随后提交与已评估 run 完全相同的 surface_idhypothesis_idevidence_ids

代码路径:loopx/control_plane/work_items/autonomous_replan_obligation.py 的 baseline 复制逻辑只匹配 typed_progress_repeat;external review 分支没有写 baseline。loopx/control_plane/work_items/progress_observation.py 随后以 baseline=None 调用 semantic_progress_delta

我在 exact head 上用生产函数复现到:trigger 是 external_progress_review_drift,obligation 不含 baseline,完全相同的 observation 返回 accepted: true,outcomes 为 new_surfacenew_hypothesis。这会让 agent 在没有产生新证据、blocker、successor 或 terminal boundary 的情况下 ACK 机器义务,状态表面显示“已恢复”,实际 drift 没有停止。

最小修复:把 external review 所评估窗口的 typed progress baseline 绑定到 trigger/obligation,或者在既有 TypeScript semantic owner 中定义等价的 trigger-specific discharge rule;不要把 ACK authority 下放到 loopx-jev。需要补一个真实 refresh_state_run 回归:相同 observation 必须拒绝;真正的新 evidence、blocker、successor 或 terminal outcome 分别允许。建议的最小重跑集合是相关 control-plane test、packages/loopx-jev/tests/test_closed_loop.py、typecheck 与 public sentinel smoke。

P2 — exact diff 仍未通过 whitespace gate

git diff --check origin/main...HEAD(以及 latest-main staged integration)报告 7 个 packages/loopx-jev/tests/fixtures/sentinel/real/*/COMMIT.txt 文件新增 EOF 空行:docs_vision_schema_compactionfix_closeout_preflight_latencyfix_lark_part_sequence_settlementfix_manager_refused_read_argumentfix_settled_turn_safe_bypasstest_closeout_preflight_budgettest_registry_smoke_external_evidence。请清理后重跑 git diff --check

语义与 CI 对齐

本地 exact-head 验证中,104 个 focused tests、npm run typecheck:control-plane 与 16-case public smoke 都通过;exact head 也能干净合入最新 origin/main。但这些绿灯没有覆盖上述 negative transition,而当前远端 CI 仍有运行中的 Python shards,早先 superseded run 还出现过 merge-gate failure。packet 对本 Goal 明确是 wait_for_ci=false,所以本评审没有等待 CI;结论由已复现的语义反例决定。

此外,整份变更为 170 files、+17,311/-10,主体体量来自 optional provider、real/constructed fixtures 与 34 份 recorded responses。core mechanism 本身是合理的;修掉 blocker 后仍建议做一次有证据的 durability pass:保留能保护 shipped contract 的最小矩阵,把不承担长期 regression contract 的研究记录压缩或归档。这一点目前作为维护性建议,不替代 P1/P2。

我的整体评价

方向和 authority 边界是扎实的:默认关闭、shadow/assist 分层、contract pinning、anonymous request、typed receipt、复用既有 obligation/TypeScript semantics,均比新建平行权威更稳妥;配置、UI、CLI、状态 readback 和双语文档也基本形成了完整用户路径。未来向前的 bounded refactor 应继续放在现有 replan owner:让所有 trigger producer 提供统一的 frontier/baseline contract,避免每个 provider 自己解释“新进展”。

但当前 exact head 的核心闭环仍可被重复 observation 绕过,这不是非阻塞建议;再加上确定的 diff-check 失败,我不能批准。请修复 typed baseline/discharge invariant、加入真实负向回归并清理 7 个 fixture EOF;保持同一 head 重新跑 focused suite、typecheck、public smoke、git diff --check 和 latest-main integration 后再复审。没有执行 merge。

English verdict: REQUEST_CHANGES - at e7afab6, external_progress_review_drift lacks the baseline needed to reject an identical observation, so unchanged work can falsely ACK the enforced replan obligation; seven fixture files also fail git diff --check, while 104 focused tests, typecheck, the 16-case smoke, and latest-main integration otherwise passed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants