Skip to content

Keep canary timeout diagnostics and incomplete failures visible - #9001

Merged
loopx-agent merged 6 commits into
mainfrom
codex/canary-timeout-evidence-20261118
Nov 18, 2026
Merged

loopx-agent merged 6 commits into
mainfrom
codex/canary-timeout-evidence-20261118

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

Canary and premerge checks can emit useful diagnostics before a subprocess times out. The maintainability ratchet can also exit with code 1 for either completed findings or an unexpected crash. The preceding gate treated a traceback from malformed baseline JSON as an unrelated finding and allowed premerge to pass. Independent reproduction on the preceding head

Both runners preserve the final 800 UTF-8 characters from either stream. The catalog runner reads the ratchet's existing structured report and renders the human-readable diagnostic tail. Premerge applies the existing advisory policy only when the complete report supplies attributable failure paths outside the diff. A crash before the report, a signal, an invalid invocation, or an unattributable policy error keeps the gate failed.

Author Declaration

  • Written by model_agent · Codex · OpenAI.
  • Basis docs/development/testing-and-quality.md at 7bc56e74687a93ff4d5bcccb532c454bf2379c29, the local premerge contract and the reproduced defects.
Criterion Disposition Evidence
Retain bounded timeout diagnostics implemented Real subprocesses emit multibyte stdout and stderr before timing out in both runners
Keep incomplete quality checks blocking implemented The real ratchet hits a malformed JSON baseline, exits 1 with a traceback and keeps the aggregate gate failed. The preceding head passed that gate. Actual SIGTERM, exit 2 and exit 1 with only progress text also fail.
Preserve completed-finding handling implemented The source CLI treats an unrelated complete report as advisory. A changed path stays blocking even when it appears outside the displayed diagnostic tail.

Validation

  • Tested revision b5304e1ba77021d0a03f99b152edbcf91b1dbfe9 against 7bc56e74687a93ff4d5bcccb532c454bf2379c29.
  • Inputs synthetic and public_fixture. Local runs finished.
Check kind Result Evidence
Regression and integration passed 126 canary tests passed, 6 skipped on macOS. The real ratchet regression failed before the fix and passed afterwards.
Real entrypoint passed Existing premerge CLI smoke and exact standard premerge. All 6 direct and 19 selected checks passed. Exact change-quality receipt cqr_389110bf3f0b5c57f19f verified.
Static passed Changed-file Ruff, configured mypy and focused type checking, Git diff, bilingual RFC and public boundary checks.
Native Windows process capture not_run Signal and partial timeout capture exercised on macOS.

Frontend / Visual Evidence

  • UI impact none.

Type of Change

  • Bug fix
  • Test update

LoopX Area

  • Build, packaging, installer, or CI

Boundary Checklist

  • Public-safe source and synthetic fixture data.
  • DCO sign-off on contribution commits.
  • Changed paths checked against the exact main comparison.

…iled

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

[P2] 信号中止的检查仍可让 premerge 通过。 实际 catalog 子进程只打印一行进度,随后因 SIGTERM 终止。候选收到 returncode=-15,仍把它降为 advisory_inherited_failure,合计 gate 返回 passed。基线同样误判。继承例外需要完成状态及实际失败诊断。信号退出应保持失败,并用同样的真实子进程复验聚合 gate。判定位置

动机 / Motivation

运行标准 premerge 的维护者需要知道检查是否真的跑完,并看到超时前留下的诊断。基线丢掉 catalog 与 direct 检查的超时输出,并可能把未完成的维护性检查记为继承告警。候选保留两条路径的 UTF-8 诊断。标准超时和空输出失败会拦住 gate,提前被信号终止的检查仍可显示通过。本轮只核对现行本地预检的输出和失败判定,期限、调度与清理规则沿用现有实现。

改动思路 / Approach

catalog 与 direct runner 复用一处尾部处理,保留实际进程状态。premerge 在这些状态上判定继承告警,再重算聚合结果。现有进程采集与 gate 聚合分层可以保留,判定完成时需要结合已有退出码。本次范围是预检输出与继承失败判定。

具体改动 / Changes

精确提交 89185a8aaf78c9a235f6a3e96ba2b18a79a9e30e,基线 f9bb5e3cf9185de77a2f7580d64eaf9d32f46eff。三个文件 +92/-12。对照本地预检约定,bounded-timeout-diagnostics 的两条入口通过,incomplete-check-blocks 仍有信号中止反例。

关键代码讲解

  • 两个 runner 都使用 check_output_tail,实际超时保留末 800 个 UTF-8 字符和 stderr,直接子进程均已退出。共享尾部 直接检查
  • downgrade_inherited_baseline_failures 保留已完成且报告无关路径的继承告警,超时、空诊断和涉及变更路径的失败保持失败。它仍把提前中止后的进度文本当成诊断。降级判定
  • 新回归覆盖真实两路超时输出及两种不完整状态,四项在基线失败。补一个真实信号中止的聚合断言即可覆盖剩余分支。测试

对主干的风险 / Risks

隔离对照分别跑了普通有关与无关失败、空诊断、真实超时和信号中止。候选的 canary 套件 119 项通过、6 项跳过,Ruff、mypy 和本地标准预检的 6 项直接、16 项选中检查通过。实际中止发生在合成 POSIX 子进程,Windows 的超时流仍待验证。

语义与 CI 对齐

沿用现有 premerge 状态和继承例外。由中止导致的负退出码表明检查未形成结论,应在聚合前保留失败。正常结束并报出无关维护问题的对照继续按已有告警规则走。聚合入口

我的整体评价 / Verdict

请求修改。超时诊断和两个早期失败分支已经改善,提前中止仍能让维护性预检误绿。补齐完成状态的负例后,再按当前实测对照验证继承例外和诊断上限。

English verdict: REQUEST_CHANGES. Exact head preserves bounded timeout diagnostics, but a signal-terminated catalog check can still be relabeled inherited and pass the aggregate gate.

Comment thread loopx/canary/premerge.py Resolved
…evidence-20261118

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
…nary-timeout-evidence-20261118

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

[P2] ratchet 生成报告前异常退出仍可被降为继承告警。 实际目录检查读取隔离的坏 baseline JSON,在写报告前抛出 JSONDecodeError,退出 1 并留下 traceback。候选把该检查标为 advisory_inherited_failure,仅含这项 catalog 检查的聚合 gate 通过。请要求实际完成的 ratchet 报告后才适用继承例外。归类判定

动机 / Motivation

运行本地预合并的维护者需要分辨检查已经完成并报告既有问题,还是在写报告前异常退出。原版丢失超时诊断,也会把提前退出误判为继承告警。当前版本保留诊断,并拦住了先前的信号中止与退出 2。实际 ratchet 解析配置报错时仍以退出 1 结束。目录检查层会把 traceback 当成正常完成的结果,聚合可显示通过。这里只核对预检进程、诊断和聚合状态,原检查期限、调度及发布资格沿既有规则。

改动思路 / Approach

两条 runner 共用诊断尾部处理。预检沿用同一归类入口,需要在继承告警之前确认 ratchet 确实产生了完成报告。本次范围为预检输出和继承失败判定。现行退出 1 既是正常问题报告的返回码,也是 Python 未捕获异常的默认返回码。目录检查原样记录了这两种结果,当前聚合仍可能把它们混在一起。检查归类

具体改动 / Changes

精确 head f7d7fa470228a1084d4782dbed1963b10f0f9db6,base 09d0fa13594235fcd3dc13c5bd63abaf35a2d9aa,5 个文件 +146/-19。bounded-timeout-diagnostics 在两条入口已证实,incomplete-check-blocks 保留生成报告前的异常反例。预检约定

关键代码讲解

  • 两条真实 POSIX runner 超时均保存末 800 个 UTF-8 字符和 stderr。试验中的子进程清理后已退出。尾部处理
  • 旧头的 SIGTERM 误绿已修复。SIGTERM、退出 2 和空诊断均阻塞。正常退出 1 的无关报告仍是 advisory,提到改动路径仍失败。现行条件
  • 真实 ratchet 遇到解析错误,退出码和非空 stderr 同样满足现有谓词,导致这条 catalog 结果误绿。当标准入口的 direct 检查先遇到同一种持续配置损坏时,也可能更早就阻塞。异常分支

对主干的风险 / Risks

本机重跑 123 项通过、6 项跳过,标准预检通过 6 项直接、19 项选中检查。反例测试使用原 ratchet 脚本、真实 catalog runner 和聚合器,以及隔离的坏输入。这证明目录检查的状态处置仍出错。Windows 原生超时流待验。

语义与 CI 对齐

继承例外需要识别实际完成的报告。可继续使用现有结果结构,加入可辨认的完成凭证。保留两个正常退出 1 的对照,还应让解析异常保持失败。

我的整体评价 / Verdict

请求修改。原评审指出的 SIGTERM 误绿已修复,普通失败对照也保持。真实 ratchet 写报告前抛异常时,目录检查仍可误报通过。补齐报告完成条件后再按同一基线复验。

English verdict: REQUEST_CHANGES. The previous signal case is fixed, but a real ratchet traceback can still be mistaken for a completed baseline finding by the catalog gate.

Comment thread loopx/canary/premerge.py Resolved
…nary-timeout-evidence-20261118

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.2-sol · OpenAI · runtime_reported · reasoning_effort=max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

本地运行 premerge 的维护者,需要分辨进程超时、报告前异常和正常报告既有债务。上一头读到坏 baseline 时,实际 ratchet 在写报告前异常退出,目录检查归成继承告警并让聚合通过。当前保留退出码和 traceback,聚合随之失败。完整无关报告仍走 advisory,变更路径在诊断尾之外也能认出。本轮验证使用 macOS 真实子进程和隔离 Git checkout,现有门禁职责继续由 premerge 持有。

改动思路 / Approach

现有两条 runner 共享诊断尾部的解码和裁剪。ratchet 已有结构化报告,premerge 消费完整路径,再用原聚合入口决定失败或 advisory。本次范围是本地预检诊断和现有继承例外。保留人读的文本尾。进程退出和 tracked-write 保护各自提供证据,只有已完成的报告才参与归因。

具体改动 / Changes

精确 head b5304e1ba77021d0a03f99b152edbcf91b1dbfe9,diff base 7bc56e74687a93ff4d5bcccb532c454bf2379c29,6 个文件 +304/-36。逐项复核 bounded-timeout-diagnostics、incomplete-check-blocks、complete-unrelated-is-advisory。验收约定

关键代码讲解

  • check_output_tail 解码超时携带的 bytes。两个真实 POSIX 入口各保留 800 个字符和同一段 stderr,观察到子进程已清理。源码
  • _record_maintainability_report 先取完整 JSON,再渲染文本尾。同一坏配置抛出 JSONDecodeError 并退出 1,上一精确头 quick premerge 通过,当前失败。源码
  • completed_maintainability_failure_paths 给出完整归因。24 条既有无关发现仍是 advisory,加一条落在尾部之外的改动路径便阻塞。源码
  • downgrade_inherited_baseline_failures 同时核对脚本、退出、完成报告与 tracked writes。SIGTERM、退出 2、提前退出和受控写入恢复均阻塞。源码

对主干的风险 / Risks

精确头 126 项 canary 测试通过、6 项跳过。标准本地预检通过 6 项直接及 19 项选中检查。原生 Windows 超时流待验证。反例使用真实 ratchet、原生 premerge 与隔离 Git fixture。

语义与 CI 对齐

完整且可归因的无关发现继续使用原 advisory。信号中止、读配置异常、相关变更和 tracked writes 均保留真实失败。文本尾用于阅读,路径归因来自完整报告。

我的整体评价 / Verdict

复评通过。上一头的坏配置误绿已修复。完整无关报告仍走 advisory,尾部之外的改动发现会阻塞。两条超时入口均保留诊断,实际进程退出已单独观察。

English verdict: APPROVE. The current head blocks actual pre-report failure and off-tail changed findings, retains completed unrelated advisories, and preserves bounded POSIX diagnostics.

@loopx-agent
loopx-agent merged commit 9792cd7 into main Nov 18, 2026
37 of 47 checks passed
@loopx-agent
loopx-agent deleted the codex/canary-timeout-evidence-20261118 branch November 18, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant