Skip to content

fix(cli-tools): defer install execution until step completion is confirmed - #676

Open
canblmz1 wants to merge 2 commits into
op7418:mainfrom
canblmz1:fix/cli-install-execution-integrity
Open

fix(cli-tools): defer install execution until step completion is confirmed#676
canblmz1 wants to merge 2 commits into
op7418:mainfrom
canblmz1:fix/cli-install-execution-integrity

Conversation

@canblmz1

@canblmz1 canblmz1 commented Aug 21, 2026

Copy link
Copy Markdown

Problem

codepilot_cli_tools_install executes execAsync(command, ...) as soon as the
AI SDK resolves that individual tool call.

At that point, the tool itself has no visibility into whether the surrounding
generation later terminates safely. A syntactically complete tool call can still
belong to a step that ends because of a token limit, provider error, or another
unsafe terminal condition.

The existing permission system does not currently carry stream-completion
state, so it cannot make an execution-integrity decision based on whether the
surrounding generation terminated safely.

Reproduction

src/__tests__/unit/execution-guard.test.ts covers a
codepilot_cli_tools_install call whose own arguments are complete, followed by
a step ending with finishReason: "length".

The contrast case demonstrates the previous execution pattern: once
execute() is entered, the real command runs without any knowledge of the
step's final termination state.

Fix

This patch uses
prefix-safe-json
0.0.1-alpha.4 and its createAiSdkExecutionGuard() API to gate the real side
effect using the raw fullStream parts already consumed in agent-loop.ts.

The real install command cannot wait inside the tool's execute() for the
stream's terminal state, because the stream cannot finish until in-flight tool
executions resolve.

Instead:

  1. codepilot_cli_tools_install.execute() registers the real install closure
    and immediately returns a queued result.
  2. agent-loop.ts feeds the step's raw fullStream events into the execution
    guard.
  3. Once the stream is fully consumed, each pending execution is resolved
    against the guard decision for its toolCallId.
  4. The real execAsync side effect runs only when the surrounding tool call is
    positively confirmed safe to execute.

The scope is intentionally narrow: only
codepilot_cli_tools_install, the directly shell-executing tool traced in this
patch, is converted. Other execute()-based tools may benefit from the same
pattern, but are intentionally out of scope here.

Safety semantics

A real install command executes only when the surrounding streamed tool call is
positively confirmed complete.

Truncated, errored, unknown, or otherwise unconfirmed terminal states discard
the pending side effect instead of executing it.

Follow-up fixes (turn isolation + model-history correctness)

Two issues found in review of the first commit, addressed in a second commit
on this branch:

Deferred state is isolated per Native turn, not by toolCallId alone.
toolCallId is provider-generated per call and not guaranteed unique across
concurrent turns — two overlapping turns can both produce e.g. "call_1"
(the same identity-isolation concern runtime/native-turn-registry.ts
already handles for abort controllers, for the same underlying reason). A
fresh executionScopeId is created once per runAgentLoop invocation and
threaded through the existing tool-assembly chain
(agent-loop.tsassembleToolsgetBuiltinTools
createCliToolsTools), so pending registrations are keyed by
scope + toolCallId and two turns can never resolve or discard each other's
pending commands even if their toolCallIds collide.

Abort/error cleanup discards pending closures fail-closed. Every step
already resolves its own scope's registrations; a new
discardPendingExecutions(scopeId) is additionally called from
agent-loop.ts's teardown finally block (which runs on every exit path —
success, abort, timeout, thrown error), so a turn that never reaches its own
step-level resolution can never leave a queued shell command available for a
later, unrelated turn to accidentally resolve. Any registration left
unmatched by a guard decision at step-resolution time is discarded the same
way, not carried forward.

The real deferred result now replaces the temporary "Queued…" placeholder
in the next model-step transcript, not just the UI.
The AI SDK bakes
execute()'s immediate "Queued…" return value into responseData.messages
as that call's tool-result — which is exactly what gets appended to the
conversation for the next step. Previously only a follow-up SSE event told
the UI the real outcome; the model's own history permanently kept
"Queued…" as the result. applyResolvedExecutionsToMessages() now replaces
that placeholder with the real outcome (the actual result text for an
executed call, or an explicit "Execution skipped: …" message — never a
claimed success — for a rejected one) before the messages are appended,
using the repository's existing ModelMessage tool-result shape from
tool-history-integrity.ts. Only the matching toolCallId's part is
touched; every other message and part is unchanged. The UI SSE and the
model transcript now share the exact same outcome text, so they can never
disagree about what happened.

Scope is unchanged from the first commit: only Native Runtime's
codepilot_cli_tools_install is affected, and cli-tools-mcp.ts (the SDK
Runtime's independent implementation) is not touched.

Testing

  • src/__tests__/unit/execution-guard.test.ts: 16/16 passing
    • safe tool-calls completion → executes exactly once
    • finishReason: "length" → does not execute
    • truncated arguments → does not execute
    • provider error → does not execute
    • unknown terminal state → does not execute
    • unrelated unregistered tool call → ignored
    • contrast case confirms the previous unconditional execution pattern would
      run in the same length scenario
    • two scopes registering the identical toolCallId never cross-resolve,
      even resolved in adversarial order
    • a discarded scope's closure never runs, even when a later, unrelated
      scope reuses the same toolCallId
    • a registration left unmatched by any guard decision is discarded, not
      carried forward
    • applyResolvedExecutionsToMessages: the safe case never leaves "Queued"
      in the transcript and shows the real result; the unsafe case never
      leaves "Queued" and never claims success; an unrelated tool-result
      message is returned byte-identical; only the matching toolCallId
      among multiple parts in one message is replaced
  • npx tsc --noEmit: clean
  • npx eslint on every changed file: 0 errors; the one new warning
    introduced during review was fixed, the remaining warnings in
    agent-loop.ts are pre-existing (confirmed identical against the
    unpatched file) and unrelated to this change
  • Targeted relevant existing suites (agent-loop-*, agent-tools-permission-allowlist,
    cli-tools-mcp, tool-history-integrity): 128/131 passing; the 3 failures
    are better-sqlite3's native binding missing in this sandbox (confirmed
    reproducible identically with the patch stashed), unrelated to these changes
  • Full unit-suite execution in this sandbox has the same known
    better-sqlite3 native-binding limitation described above

prefix-safe-json@0.0.1-alpha.4 was installed from the public npm registry
(unchanged from the first commit).

…irmed

Root cause: codepilot_cli_tools_install's execute() called execAsync(command)
directly and unconditionally, the moment the AI SDK finished parsing that
one tool call's arguments. It had no awareness of finishReason or whether
the *step* containing it (which may include other tool calls or trailing
text) ever reached a safe terminal state. A tool call can be syntactically
complete and still belong to a generation that was cut off by a token
limit, a provider error, or a content filter immediately after — the
existing permission system (permission-checker.ts) is a pure name/pattern
allow-list with no visibility into stream completion, so nothing in the
current codebase catches this. In "trust" mode, or for any future tool
without an explicit ask rule, a truncated/unconfirmed generation could
already have run a real shell command by the time anything downstream knew
the response wasn't finished.

execute() cannot simply await proof of that itself: the step's fullStream
can't reach its own finish part until every in-flight execute() call for
that step has already resolved, so waiting inside execute() would deadlock.

Fix: src/lib/execution-guard.ts defers the real side effect. execute()
registers it and returns an immediate "queued" result instead of running
the command; agent-loop.ts — which already iterates the step's fullStream
and already learns finishReason once the step ends — feeds every event
into a prefix-safe-json (npm) execution guard and, once the step is over,
resolves each pending registration against the guard's decision for that
exact toolCallId. Only a call whose surrounding step positively confirmed
completion ever actually runs; everything else is discarded before taking
effect. The confirmed/rejected outcome is surfaced as a follow-up
tool_result SSE event, reusing the existing event shape.

Files:
- src/lib/execution-guard.ts (new): registerDeferredExecution / createStepGuard
  / resolvePendingExecutions.
- src/lib/builtin-tools/cli-tools.ts: codepilot_cli_tools_install's execute
  now registers via execution-guard instead of running execAsync directly;
  no other tool or behavior changed.
- src/lib/agent-loop.ts: push every fullStream event into a per-step guard
  (additive — these event types were previously unhandled, falling to the
  existing `default: break`), resolve pending executions once the step's
  finishReason is known.
- package.json / package-lock.json: add prefix-safe-json@0.0.1-alpha.4.
- src/__tests__/unit/execution-guard.test.ts (new): 7 tests — safe
  completion executes exactly once; four unsafe terminal states (length,
  truncated arguments, provider error, unknown) never execute; an
  unregistered tool call is ignored; a contrast test demonstrating the
  pre-fix unconditional-execute pattern for comparison.

Scope: only codepilot_cli_tools_install (the tool with the clearest,
directly-verifiable shell-execution side effect) is converted in this
patch. Other execute()-based tools (file writes, other MCP-backed tools)
have the same architectural exposure and could adopt the same
registerDeferredExecution pattern, but are left out here to keep this
patch reviewable.
@vercel

vercel Bot commented Aug 21, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the op7418's projects Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added the area:tests PR/issue 影响面: tests label Aug 21, 2026
Two follow-up fixes to the deferred-execution pattern from f783133.

1. Scope deferred execution state per turn, not by toolCallId alone.

   execution-guard.ts's pending map was keyed only by toolCallId, a
   provider-generated id with no uniqueness guarantee across concurrent
   Native turns (two overlapping turns can both produce e.g. "call_1") —
   the same identity-isolation concern runtime/native-turn-registry.ts
   already handles for abort controllers, and for the same underlying
   reason: module-local state shared across concurrent turns. A late
   resolution from one turn could in principle have matched a different
   turn's registration for the same toolCallId, or a thrown/aborted turn
   could leave a registration in the map indefinitely with no path to
   remove it.

   Fix: a fresh `executionScopeId` (randomUUID()) is created once per
   runAgentLoop invocation and threaded through the existing tool assembly
   chain (agent-loop.ts -> assembleTools -> getBuiltinTools ->
   createCliToolsTools), so the pending map is keyed by scope + toolCallId.
   registerDeferredExecution fails closed (refuses to register, returns an
   explicit error) if no scope is available. resolvePendingExecutions now
   also fail-closed-discards any registration in its scope left unmatched
   by a guard decision, instead of leaving it to linger. A new
   discardPendingExecutions(scopeId) is called from agent-loop.ts's
   teardown `finally` (which already runs on every exit path — success,
   abort, timeout, thrown error), so a turn that never reaches its own
   step-level resolution can never leave a queued shell command for a
   later, unrelated turn to accidentally pick up.

2. Feed the real deferred result back into the model's own history, not
   just the UI.

   The tool's execute() returns an immediate "Queued…" placeholder, which
   the AI SDK bakes into responseData.messages as that call's tool-result
   — and responseData.messages is exactly what gets appended to the
   conversation for the next step. Previously only the UI (via a follow-up
   SSE tool_result) ever learned the real outcome; the model's own
   transcript permanently kept "Queued…" as the result, so a later step
   had no way to know whether the command actually ran.

   Fix: ResolvedExecution now carries a single `outcomeText` (the real
   result text, or an explicit "Execution skipped: ..." message for a
   rejected call) computed once in resolvePendingExecutions, so the UI
   SSE and the model transcript can never disagree about what happened.
   applyResolvedExecutionsToMessages(responseData.messages,
   resolvedExecutions) replaces the queued placeholder tool-result part
   for each resolved toolCallId before the messages get appended to the
   loop's `messages` array — using the repository's existing
   ModelMessage tool-result shape (role: "tool", content: [{ type:
   "tool-result", toolCallId, toolName, output: { type: "text", value }
   }], matching tool-history-integrity.ts). Only the matching part is
   replaced; every other message and part is returned unchanged (same
   reference, not cloned).

Scope unchanged from f783133: only Native Runtime's
codepilot_cli_tools_install is affected. cli-tools-mcp.ts (the SDK
Runtime's independent implementation) is not touched. prefix-safe-json
stays pinned at 0.0.1-alpha.4.

Tests: 16 in execution-guard.test.ts (10 from f783133 plus 6 new) —
two scopes registering the identical toolCallId never cross-resolve in
adversarial resolution order; a discarded scope's closure never runs
even when a later, unrelated scope reuses the same toolCallId; an
unmatched registration is discarded, not carried forward; and
applyResolvedExecutionsToMessages never leaves "Queued" in the transcript
for either the safe (real result) or unsafe (explicit skip, no false
success) case, leaves unrelated tool-result parts byte-identical, and
only replaces the matching toolCallId among multiple parts in the same
message.
@github-actions github-actions Bot added the pr:large PR 过大,建议拆分或在描述说明 label Aug 21, 2026
@github-actions

Copy link
Copy Markdown

⚠️ 这个 PR 较大(8 个文件 / 814 行改动;阈值 25 文件 / 800 行)。

建议拆分成更小、聚焦的 PR,或在描述里说明为什么需要一次性改这么多——这样更好审查、风险更低。

这只是提醒,不阻塞合并。

@canblmz1
canblmz1 marked this pull request as ready for review August 21, 2026 07:28
@canblmz1

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tests PR/issue 影响面: tests pr:large PR 过大,建议拆分或在描述说明

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant