fix(cli-tools): defer install execution until step completion is confirmed - #676
Open
canblmz1 wants to merge 2 commits into
Open
fix(cli-tools): defer install execution until step completion is confirmed#676canblmz1 wants to merge 2 commits into
canblmz1 wants to merge 2 commits into
Conversation
…irmed Root cause: codepilot_cli_tools_install's execute() called execAsync(command) directly and unconditionally, the moment the AI SDK finished parsing that one tool call's arguments. It had no awareness of finishReason or whether the *step* containing it (which may include other tool calls or trailing text) ever reached a safe terminal state. A tool call can be syntactically complete and still belong to a generation that was cut off by a token limit, a provider error, or a content filter immediately after — the existing permission system (permission-checker.ts) is a pure name/pattern allow-list with no visibility into stream completion, so nothing in the current codebase catches this. In "trust" mode, or for any future tool without an explicit ask rule, a truncated/unconfirmed generation could already have run a real shell command by the time anything downstream knew the response wasn't finished. execute() cannot simply await proof of that itself: the step's fullStream can't reach its own finish part until every in-flight execute() call for that step has already resolved, so waiting inside execute() would deadlock. Fix: src/lib/execution-guard.ts defers the real side effect. execute() registers it and returns an immediate "queued" result instead of running the command; agent-loop.ts — which already iterates the step's fullStream and already learns finishReason once the step ends — feeds every event into a prefix-safe-json (npm) execution guard and, once the step is over, resolves each pending registration against the guard's decision for that exact toolCallId. Only a call whose surrounding step positively confirmed completion ever actually runs; everything else is discarded before taking effect. The confirmed/rejected outcome is surfaced as a follow-up tool_result SSE event, reusing the existing event shape. Files: - src/lib/execution-guard.ts (new): registerDeferredExecution / createStepGuard / resolvePendingExecutions. - src/lib/builtin-tools/cli-tools.ts: codepilot_cli_tools_install's execute now registers via execution-guard instead of running execAsync directly; no other tool or behavior changed. - src/lib/agent-loop.ts: push every fullStream event into a per-step guard (additive — these event types were previously unhandled, falling to the existing `default: break`), resolve pending executions once the step's finishReason is known. - package.json / package-lock.json: add prefix-safe-json@0.0.1-alpha.4. - src/__tests__/unit/execution-guard.test.ts (new): 7 tests — safe completion executes exactly once; four unsafe terminal states (length, truncated arguments, provider error, unknown) never execute; an unregistered tool call is ignored; a contrast test demonstrating the pre-fix unconditional-execute pattern for comparison. Scope: only codepilot_cli_tools_install (the tool with the clearest, directly-verifiable shell-execution side effect) is converted in this patch. Other execute()-based tools (file writes, other MCP-backed tools) have the same architectural exposure and could adopt the same registerDeferredExecution pattern, but are left out here to keep this patch reviewable.
|
Someone is attempting to deploy a commit to the op7418's projects Team on Vercel. A member of the Team first needs to authorize it. |
Two follow-up fixes to the deferred-execution pattern from f783133. 1. Scope deferred execution state per turn, not by toolCallId alone. execution-guard.ts's pending map was keyed only by toolCallId, a provider-generated id with no uniqueness guarantee across concurrent Native turns (two overlapping turns can both produce e.g. "call_1") — the same identity-isolation concern runtime/native-turn-registry.ts already handles for abort controllers, and for the same underlying reason: module-local state shared across concurrent turns. A late resolution from one turn could in principle have matched a different turn's registration for the same toolCallId, or a thrown/aborted turn could leave a registration in the map indefinitely with no path to remove it. Fix: a fresh `executionScopeId` (randomUUID()) is created once per runAgentLoop invocation and threaded through the existing tool assembly chain (agent-loop.ts -> assembleTools -> getBuiltinTools -> createCliToolsTools), so the pending map is keyed by scope + toolCallId. registerDeferredExecution fails closed (refuses to register, returns an explicit error) if no scope is available. resolvePendingExecutions now also fail-closed-discards any registration in its scope left unmatched by a guard decision, instead of leaving it to linger. A new discardPendingExecutions(scopeId) is called from agent-loop.ts's teardown `finally` (which already runs on every exit path — success, abort, timeout, thrown error), so a turn that never reaches its own step-level resolution can never leave a queued shell command for a later, unrelated turn to accidentally pick up. 2. Feed the real deferred result back into the model's own history, not just the UI. The tool's execute() returns an immediate "Queued…" placeholder, which the AI SDK bakes into responseData.messages as that call's tool-result — and responseData.messages is exactly what gets appended to the conversation for the next step. Previously only the UI (via a follow-up SSE tool_result) ever learned the real outcome; the model's own transcript permanently kept "Queued…" as the result, so a later step had no way to know whether the command actually ran. Fix: ResolvedExecution now carries a single `outcomeText` (the real result text, or an explicit "Execution skipped: ..." message for a rejected call) computed once in resolvePendingExecutions, so the UI SSE and the model transcript can never disagree about what happened. applyResolvedExecutionsToMessages(responseData.messages, resolvedExecutions) replaces the queued placeholder tool-result part for each resolved toolCallId before the messages get appended to the loop's `messages` array — using the repository's existing ModelMessage tool-result shape (role: "tool", content: [{ type: "tool-result", toolCallId, toolName, output: { type: "text", value } }], matching tool-history-integrity.ts). Only the matching part is replaced; every other message and part is returned unchanged (same reference, not cloned). Scope unchanged from f783133: only Native Runtime's codepilot_cli_tools_install is affected. cli-tools-mcp.ts (the SDK Runtime's independent implementation) is not touched. prefix-safe-json stays pinned at 0.0.1-alpha.4. Tests: 16 in execution-guard.test.ts (10 from f783133 plus 6 new) — two scopes registering the identical toolCallId never cross-resolve in adversarial resolution order; a discarded scope's closure never runs even when a later, unrelated scope reuses the same toolCallId; an unmatched registration is discarded, not carried forward; and applyResolvedExecutionsToMessages never leaves "Queued" in the transcript for either the safe (real result) or unsafe (explicit skip, no false success) case, leaves unrelated tool-result parts byte-identical, and only replaces the matching toolCallId among multiple parts in the same message.
|
建议拆分成更小、聚焦的 PR,或在描述里说明为什么需要一次性改这么多——这样更好审查、风险更低。 这只是提醒,不阻塞合并。 |
canblmz1
marked this pull request as ready for review
August 21, 2026 07:28
Author
|
recheck |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
codepilot_cli_tools_installexecutesexecAsync(command, ...)as soon as theAI SDK resolves that individual tool call.
At that point, the tool itself has no visibility into whether the surrounding
generation later terminates safely. A syntactically complete tool call can still
belong to a step that ends because of a token limit, provider error, or another
unsafe terminal condition.
The existing permission system does not currently carry stream-completion
state, so it cannot make an execution-integrity decision based on whether the
surrounding generation terminated safely.
Reproduction
src/__tests__/unit/execution-guard.test.tscovers acodepilot_cli_tools_installcall whose own arguments are complete, followed bya step ending with
finishReason: "length".The contrast case demonstrates the previous execution pattern: once
execute()is entered, the real command runs without any knowledge of thestep's final termination state.
Fix
This patch uses
prefix-safe-json0.0.1-alpha.4and itscreateAiSdkExecutionGuard()API to gate the real sideeffect using the raw
fullStreamparts already consumed inagent-loop.ts.The real install command cannot wait inside the tool's
execute()for thestream's terminal state, because the stream cannot finish until in-flight tool
executions resolve.
Instead:
codepilot_cli_tools_install.execute()registers the real install closureand immediately returns a queued result.
agent-loop.tsfeeds the step's rawfullStreamevents into the executionguard.
against the guard decision for its
toolCallId.execAsyncside effect runs only when the surrounding tool call ispositively confirmed safe to execute.
The scope is intentionally narrow: only
codepilot_cli_tools_install, the directly shell-executing tool traced in thispatch, is converted. Other execute()-based tools may benefit from the same
pattern, but are intentionally out of scope here.
Safety semantics
A real install command executes only when the surrounding streamed tool call is
positively confirmed complete.
Truncated, errored, unknown, or otherwise unconfirmed terminal states discard
the pending side effect instead of executing it.
Follow-up fixes (turn isolation + model-history correctness)
Two issues found in review of the first commit, addressed in a second commit
on this branch:
Deferred state is isolated per Native turn, not by
toolCallIdalone.toolCallIdis provider-generated per call and not guaranteed unique acrossconcurrent turns — two overlapping turns can both produce e.g.
"call_1"(the same identity-isolation concern
runtime/native-turn-registry.tsalready handles for abort controllers, for the same underlying reason). A
fresh
executionScopeIdis created once perrunAgentLoopinvocation andthreaded through the existing tool-assembly chain
(
agent-loop.ts→assembleTools→getBuiltinTools→createCliToolsTools), so pending registrations are keyed byscope + toolCallIdand two turns can never resolve or discard each other'spending commands even if their
toolCallIds collide.Abort/error cleanup discards pending closures fail-closed. Every step
already resolves its own scope's registrations; a new
discardPendingExecutions(scopeId)is additionally called fromagent-loop.ts's teardownfinallyblock (which runs on every exit path —success, abort, timeout, thrown error), so a turn that never reaches its own
step-level resolution can never leave a queued shell command available for a
later, unrelated turn to accidentally resolve. Any registration left
unmatched by a guard decision at step-resolution time is discarded the same
way, not carried forward.
The real deferred result now replaces the temporary "Queued…" placeholder
in the next model-step transcript, not just the UI. The AI SDK bakes
execute()'s immediate "Queued…" return value intoresponseData.messagesas that call's
tool-result— which is exactly what gets appended to theconversation for the next step. Previously only a follow-up SSE event told
the UI the real outcome; the model's own history permanently kept
"Queued…" as the result.
applyResolvedExecutionsToMessages()now replacesthat placeholder with the real outcome (the actual result text for an
executed call, or an explicit "Execution skipped: …" message — never a
claimed success — for a rejected one) before the messages are appended,
using the repository's existing
ModelMessagetool-result shape fromtool-history-integrity.ts. Only the matchingtoolCallId's part istouched; every other message and part is unchanged. The UI SSE and the
model transcript now share the exact same outcome text, so they can never
disagree about what happened.
Scope is unchanged from the first commit: only Native Runtime's
codepilot_cli_tools_installis affected, andcli-tools-mcp.ts(the SDKRuntime's independent implementation) is not touched.
Testing
src/__tests__/unit/execution-guard.test.ts: 16/16 passingtool-callscompletion → executes exactly oncefinishReason: "length"→ does not executerun in the same
lengthscenariotoolCallIdnever cross-resolve,even resolved in adversarial order
scope reuses the same
toolCallIdcarried forward
applyResolvedExecutionsToMessages: the safe case never leaves "Queued"in the transcript and shows the real result; the unsafe case never
leaves "Queued" and never claims success; an unrelated tool-result
message is returned byte-identical; only the matching
toolCallIdamong multiple parts in one message is replaced
npx tsc --noEmit: cleannpx eslinton every changed file: 0 errors; the one new warningintroduced during review was fixed, the remaining warnings in
agent-loop.tsare pre-existing (confirmed identical against theunpatched file) and unrelated to this change
agent-loop-*,agent-tools-permission-allowlist,cli-tools-mcp,tool-history-integrity): 128/131 passing; the 3 failuresare
better-sqlite3's native binding missing in this sandbox (confirmedreproducible identically with the patch stashed), unrelated to these changes
better-sqlite3native-binding limitation described aboveprefix-safe-json@0.0.1-alpha.4was installed from the public npm registry(unchanged from the first commit).