Skip to content

fix(engine): treat Codex no-rollout resume errors as lost CLI sessions - #1129

Open
331931009 wants to merge 1 commit into
Tencent:mainfrom
331931009:fix/codex-lost-session
Open

331931009 wants to merge 1 commit into
Tencent:mainfrom
331931009:fix/codex-lost-session

Conversation

@331931009

Copy link
Copy Markdown

Problem

When a node is retried after its sandbox was recreated, the workspace's
.agentflow/ directory is re-prepared and the Codex thread's rollout file is
gone. The runtime still passes the stored session id
(nr.ResumeSessionID / nr.SessionID, internal/engine/node_execute.go:686-689),
so Codex answers:

Error: thread/resume: thread/resume failed: no rollout found for thread id 01a1f321-… (code -32600)

lostCLISession() only matched no conversation found, session not found,
no such session and conversation does not exist, so dropLostCLISession()
never fired: the node was parked on waiting_approval instead of restarting in
a fresh session.

Reproduction (LoopForge 0.4.0, self-hosted Compose, Codex CLI on an
OpenAI-compatible gateway, generic-backend-delivery SOP):

  1. Let a SOP reach its mr node; the node commits and pushes the work branch
    (origin == HEAD, 12 files changed).
  2. Restart the control plane while the node runs → signal: killed.
  3. Press Retry.
  4. The node fails immediately with the no rollout found error above and the
    run stays waiting; the only way forward is to clear
    run_nodes.payload.sessionId by hand.

This is a user-visible contradiction: the work is already pushed, but the run
looks broken and needs database surgery to continue. Any restart of the control
plane, or any sandbox reclaim/recreate that re-prepares .agentflow, hits it.

Fix

Recognise the Codex phrasings as lost sessions so the existing recovery path
(dropLostCLISession → agent.session.dropped → restart the node with a fresh
session) takes over:

strings.Contains(msg, "no rollout found") ||
strings.Contains(msg, "thread/resume failed") ||
strings.Contains(msg, "code -32600")

plus TestLostCLISession covering every phrasing, including the full Codex
message, and a negative list (exit status 1, 503, gate REJECT) that must
not be treated as a lost session.

gofmt clean. Two files, +50/−1.

中文摘要

重试被中断的节点时,沙箱重建会把 .agentflow/ 里的 Codex rollout 清掉,但节点仍带着旧
sessionId 去 resume,Codex 返回 code -32600 / no rollout found;lostCLISession()
不认这个措辞,导致本该"丢弃会话重新开始"的自动恢复没有触发,节点被挂到人工审批。
本 PR 只扩展匹配列表并补单测,不改其它行为。

Follow-up (separate issues, not in this PR)

  1. internal/runtime/browser.go builds the Chromium sidecar with a hard-coded
    15 * time.Minute timeout and only on first use. On slow networks the build
    is killed every attempt (docker build browser image: signal: killed) while
    pulling ~177 Debian packages. Suggest AGENTFLOW_BROWSER_BUILD_TIMEOUT and
    warming it at startup like PrebuildIsolatedImage().
  2. Codex rejects rm -f style commands are not permitted, which breaks the
    usual "clean the temp classes dir and recompile" loop for JVM projects; a
    documented workaround (e.g. use mktemp -d) in the agent rules would help.

When a node is retried after its sandbox was recreated, .agentflow is
re-prepared and the Codex thread's rollout file is gone. The runtime still
passes the stored session id and Codex answers

  thread/resume: thread/resume failed: no rollout found for thread id ... (code -32600)

lostCLISession() did not recognise that phrasing, so dropLostCLISession()
never fired and the node was parked on a human gate even though its code
had already been committed and pushed. Match the Codex phrasings so the
node restarts in a fresh session instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant