Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions EVENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,6 +302,67 @@ Emitted at step boundaries during multi-step tool use.
{ "type": "step_finish", "part": { "type": "step-finish" } }
```

#### Provider termination details

`step_finish.part.termination` is an optional additive diagnostic object. Its

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Termination block splits heading from section intro sentence.

🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #114, EVENTS.md:298-354):

Problem: Termination block splits heading from section intro sentence
Detail: The 57-line termination subsection is inserted directly under `### step_start / step_finish`, pushing the section's own one-line description "Emitted at step boundaries during multi-step tool use." to after the JSON example and policy paragraphs. Readers now hit termination-specific detail before being told what the events are, inverting the intro-then-detail structure used by surrounding sections.
Suggested fix: Move the insertion so the intro sentence stays immediately after the heading — either place the termination content after "Emitted at step boundaries during multi-step tool use." or under its own `#### step_finish.part.termination` subheading.

Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The 57-line termination subsection is inserted directly under ### step_start / step_finish, pushing the section's own one-line description "Emitted at step boundaries during multi-step tool use." to after the JSON example and policy paragraphs. Readers now hit termination-specific detail before being told what the events are, inverting the intro-then-detail structure used by surrounding sections.

`normalizedReason` mirrors `part.reason`; `providerID` and `modelID` identify the
configured provider/model. Correlate it with the enclosing `invocationID` and
`sessionID`, and `part.messageID`. Older stored parts may omit `termination`.
This diagnostic does not decide whether an invocation succeeded (see #108).

```json
{
"type": "step_finish",
"invocationID": "inv_fixture",
"sessionID": "ses_fixture",
"part": {
"type": "step-finish",
"messageID": "msg_fixture",
"reason": "error",
"termination": {
"providerID": "google-vertex",
"modelID": "gemini-2.5-flash",
"normalizedReason": "error",
"rawReason": { "status": "available", "value": "MALFORMED_FUNCTION_CALL", "truncated": false },
"requestID": { "status": "unavailable", "truncated": false },
"diagnostic": { "status": "redacted", "truncated": false }
}
}
}
```

The example omits unrelated step fields. Each diagnostic field reports
`available` (a permitted observed value), `unavailable` (missing or unsupported),
or `redacted` (present but suppressed). Values are absent when unavailable or
redacted. `truncated: true` means an oversized value was entirely suppressed;
no prefix is retained. Raw reasons and request IDs are limited to 128 characters.
Free-form provider diagnostics are always suppressed, with a 2,048-character
threshold for the oversize flag: they can contain credentials, prompt text, or
tool arguments that pattern-based redaction cannot reliably remove.

Raw reasons are currently captured only for native `@ai-sdk/google` and
`@ai-sdk/google-vertex` streams. Their pinned adapters support internal raw
chunks; middleware selects the first candidate's allowlisted `finishReason`,
records `finishMessage` presence/size, and drops every raw chunk before it reaches
AI SDK stream consumers. Unknown raw reason strings are redacted. Other adapters
still provide their normalized reason and explicitly report raw details as
unavailable. No raw reason is inferred from the normalized reason.

Request-ID availability uses only `x-request-id` or `x-goog-request-id` response
headers. All nonempty values are reported as `redacted` without retaining the
value, including ordinary IDs: arbitrary IDs cannot be reliably distinguished
from credential material by format. Values over 128 characters also set
`truncated: true`. The SDK's generated response ID is never presented as a
provider request ID. No new prompts,
reasoning, tool arguments, full responses, or response-header maps are collected
by this diagnostic path. Existing exception diagnostics are unchanged.

For #109, this delivers observed raw finish reasons and privacy-preserving
diagnostic availability. Richer diagnostic text remains a separate policy and
adapter-coverage decision; a redacted diagnostic cannot identify the exact
offending tool call. Deliver after #108 so a provider error reason is paired
with truthful process/session failure status.

## Skill Events

Skills are loaded progressively. The model first sees skill names and descriptions in the tool schema. Full skill content only enters context when the model explicitly invokes the skill tool.
Expand Down
15 changes: 15 additions & 0 deletions packages/cli/src/cli/cmd/generate.ts
Original file line number Diff line number Diff line change
@@ -1,15 +1,30 @@
import type { CommandModule } from "yargs"
import z from "zod"
import { MessageV2 } from "@/session/message-v2"
import { ProviderTermination } from "@/provider/termination"

export const GenerateCommand = {
command: "generate",
handler: async () => {
// Keep OpenAPI component identities separate from legacy Zod metadata.
const registry = z.registry<{ id: string }>()
registry.add(MessageV2.StepFinishPart, { id: "StepFinishPart" })
registry.add(ProviderTermination.Info, { id: "ProviderTermination" })
const components = z.toJSONSchema(registry, {
metadata: z.registry(),
target: "draft-2020-12",
uri: (id) => `#/components/schemas/${id}`,
})
// Component JSON pointers are references, not standalone schema resource IDs.
for (const schema of Object.values(components.schemas)) delete schema.$id
const specs = {
openapi: "3.1.1",
info: {
title: "aictrl",
version: "1.0.0",
},
paths: {},
components,
}
const json = JSON.stringify(specs, null, 2)

Expand Down
123 changes: 123 additions & 0 deletions packages/cli/src/provider/termination.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
import type { LanguageModelV2Middleware } from "@ai-sdk/provider"
import z from "zod"

export namespace ProviderTermination {
const VALUE_LIMIT = 128
const DIAGNOSTIC_LIMIT = 2048
function Field(limit: number) {
return z.object({
status: z.enum(["available", "unavailable", "redacted"]),
value: z.string().max(limit).optional(),
truncated: z.boolean(),
})
}
type Field = z.infer<ReturnType<typeof Field>>

export const Info = z
.object({
providerID: z.string(),
modelID: z.string(),
normalizedReason: z.string(),
rawReason: Field(VALUE_LIMIT),
requestID: Field(VALUE_LIMIT),
diagnostic: Field(DIAGNOSTIC_LIMIT),
})
.meta({ ref: "ProviderTermination" })
export type Info = z.infer<typeof Info>

const reasons = new Set([
"FINISH_REASON_UNSPECIFIED",
"STOP",
"MAX_TOKENS",
"SAFETY",
"RECITATION",
"LANGUAGE",
"OTHER",
"BLOCKLIST",
"PROHIBITED_CONTENT",
"SPII",
"MALFORMED_FUNCTION_CALL",
"IMAGE_SAFETY",
"UNEXPECTED_TOOL_CALL",
"NO_IMAGE",
"IMAGE_PROHIBITED_CONTENT",
"IMAGE_OTHER",
"IMAGE_RECITATION",
])

function field(value: unknown, allowed: (value: string) => boolean, limit = VALUE_LIMIT): Field {
if (typeof value !== "string" || !value) return { status: "unavailable", truncated: false }
if (value.length > limit || !allowed(value)) return { status: "redacted", truncated: value.length > limit }
return { status: "available", value, truncated: false }
}

function record(value: unknown): Record<string, unknown> | undefined {
if (typeof value !== "object" || value === null || Array.isArray(value)) return
return value as Record<string, unknown>
}

export function middleware(input: { providerID: string; modelID: string; npm: string }): LanguageModelV2Middleware {
const google = input.npm === "@ai-sdk/google" || input.npm === "@ai-sdk/google-vertex"
return {
async transformParams({ type, params }) {
return type === "stream" && google ? { ...params, includeRawChunks: true } : params
},
async wrapStream({ doStream }) {
const result = await doStream()
let rawReason = field(undefined, () => false)
let diagnostic = field(undefined, () => false)
const headers = result.response?.headers

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 result.response?.headers never exists in v2; requestID dead.

--- a/packages/cli/src/provider/termination.ts
+++ b/packages/cli/src/provider/termination.ts
@@ -61,7 +61,7 @@
         const result = await doStream()
         let rawReason = field(undefined, () => false)
         let diagnostic = field(undefined, () => false)
-        const headers = result.response?.headers
+        const headers = result.responseHeaders
         const requestID = field(
           headers?.["x-request-id"] ?? headers?.["x-goog-request-id"],
🤖 Fix with your agent
Fix this code review finding (aictrl-dev/cli PR #114, packages/cli/src/provider/termination.ts:64-68):

Problem: result.response?.headers never exists in v2; requestID dead
Detail: The middleware declares `LanguageModelV2Middleware` from `@ai-sdk/provider` (v2 / AI SDK 5), where `doStream()` resolves `LanguageModelV2StreamResult = { stream, request, responseHeaders? }` — there is no `response` wrapper property. So `result.response?.headers` is always `undefined`, `requestID` is permanently `{status:"unavailable"}`, and the feature's request-ID capture never works. The PR's own tests assert `requestID: {status:"available", value:"req_fixture"|"req_123"}` (test/cli/run-termination.test.ts, test/provider/termination.test.ts), which fail; it is also a TypeScript error (`Property 'response' does not exist`), so typecheck should flag it.
Suggested fix: Replace `const headers = result.response?.headers` with `const headers = result.responseHeaders` (the v2 `LanguageModelV2StreamResult` exposes response headers directly as `responseHeaders`).

Suggested patch:
--- a/packages/cli/src/provider/termination.ts
+++ b/packages/cli/src/provider/termination.ts
@@ -61,7 +61,7 @@
         const result = await doStream()
         let rawReason = field(undefined, () => false)
         let diagnostic = field(undefined, () => false)
-        const headers = result.response?.headers
+        const headers = result.responseHeaders
         const requestID = field(
           headers?.["x-request-id"] ?? headers?.["x-goog-request-id"],


Implement the fix on the PR head branch and add a regression test that fails before the fix and passes after.
Why this matters

The middleware declares LanguageModelV2Middleware from @ai-sdk/provider (v2 / AI SDK 5), where doStream() resolves LanguageModelV2StreamResult = { stream, request, responseHeaders? } — there is no response wrapper property. So result.response?.headers is always undefined, requestID is permanently {status:"unavailable"}, and the feature's request-ID capture never works. The PR's own tests assert requestID: {status:"available", value:"req_fixture"|"req_123"} (test/cli/run-termination.test.ts, test/provider/termination.test.ts), which fail; it is also a TypeScript error (Property 'response' does not exist), so typecheck should flag it.

      async wrapStream({ doStream }) {
        const result = await doStream()
        let rawReason = field(undefined, () => false)
        let diagnostic = field(undefined, () => false)
        const headers = result.response?.headers
        const requestID = field(
          headers?.["x-request-id"] ?? headers?.["x-goog-request-id"],
          (value) => /^[a-zA-Z0-9_-]+$/.test(value) && !/^(sk-|gh[pousr]_|github_pat_|AIza|AKIA|ASIA)/.test(value),
        )
        return {

// A provider/proxy can echo arbitrary credentials into an ID header.
// Format allowlists cannot distinguish an opaque ID from an opaque key.
const requestID = field(headers?.["x-request-id"] || headers?.["x-goog-request-id"], () => false)
return {
...result,
stream: result.stream.pipeThrough(
new TransformStream({
transform(chunk, controller) {
if (chunk.type === "raw") {
if (google) {
const candidates = record(chunk.rawValue)?.candidates
const candidate = Array.isArray(candidates) ? record(candidates[0]) : undefined
if (candidate?.finishReason != null) {
rawReason = field(candidate.finishReason, (value) => reasons.has(value))
// Provider messages can contain prompts or tool arguments. Retain only
// presence/size information; do not collect their free-form contents.
diagnostic = field(candidate.finishMessage, () => false, DIAGNOSTIC_LIMIT)
}
}
// Raw payloads must never escape this boundary to stream consumers.
return
}
if (chunk.type !== "finish") {
controller.enqueue(chunk)
return
}
const termination: Info = {
providerID: input.providerID,
modelID: input.modelID,
normalizedReason: chunk.finishReason,
rawReason,
requestID,
diagnostic,
}
controller.enqueue({
...chunk,
providerMetadata: {
...chunk.providerMetadata,
aictrl: { ...chunk.providerMetadata?.aictrl, termination },
},
})
},
}),
),
}
},
}
}

export function from(metadata: unknown): Info | undefined {
const result = Info.safeParse(record(record(metadata)?.aictrl)?.termination)
return result.success ? result.data : undefined
}
}
6 changes: 6 additions & 0 deletions packages/cli/src/session/llm.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import { SystemPrompt } from "./system"
import { Flag } from "@/flag/flag"
import { PermissionNext } from "@/permission/next"
import { Auth } from "@/auth"
import { ProviderTermination } from "@/provider/termination"

export namespace LLM {
const log = Log.create({ service: "llm" })
Expand Down Expand Up @@ -234,6 +235,11 @@ export namespace LLM {
model: wrapLanguageModel({
model: language,
middleware: [
ProviderTermination.middleware({
providerID: input.model.providerID,
modelID: input.model.id,
npm: input.model.api.npm,
}),
{
async transformParams(args) {
if (args.type === "stream") {
Expand Down
2 changes: 2 additions & 0 deletions packages/cli/src/session/message-v2.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { fn } from "@/util/fn"
import { Database, eq, desc, inArray } from "@/storage/db"
import { MessageTable, PartTable } from "./session.sql"
import { ProviderTransform } from "@/provider/transform"
import { ProviderTermination } from "@/provider/termination"
import { STATUS_CODES } from "http"
import { Storage } from "@/storage/storage"
import { ProviderError } from "@/provider/error"
Expand Down Expand Up @@ -241,6 +242,7 @@ export namespace MessageV2 {
export const StepFinishPart = PartBase.extend({
type: z.literal("step-finish"),
reason: z.string(),
termination: ProviderTermination.Info.optional(),
snapshot: z.string().optional(),
cost: z.number(),
tokens: z.object({
Expand Down
2 changes: 2 additions & 0 deletions packages/cli/src/session/processor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { SessionCompaction } from "./compaction"
import { PermissionNext } from "@/permission/next"
import { Question } from "@/question"
import { NamedError } from "@aictrl/util/error"
import { ProviderTermination } from "@/provider/termination"

export namespace SessionProcessor {
const DOOM_LOOP_THRESHOLD = 3
Expand Down Expand Up @@ -255,6 +256,7 @@ export namespace SessionProcessor {
await Session.updatePart({
id: Identifier.ascending("part"),
reason: value.finishReason,
termination: ProviderTermination.from(value.providerMetadata),
snapshot: await Snapshot.track(),
messageID: input.assistantMessage.id,
sessionID: input.assistantMessage.sessionID,
Expand Down
37 changes: 37 additions & 0 deletions packages/cli/test/cli/generate.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import { expect, test } from "bun:test"
import path from "path"

test("generate publishes the termination schema used by SDK codegen", async () => {
const proc = Bun.spawn(
["bun", "run", "--conditions=browser", path.resolve(import.meta.dir, "../../src/index.ts"), "generate"],
{
stdout: "pipe",
stderr: "pipe",
},
)
const [stdout, stderr, exit] = await Promise.all([
new Response(proc.stdout).text(),
new Response(proc.stderr).text(),
proc.exited,
])
expect(exit, stderr).toBe(0)
const spec = JSON.parse(stdout)
const part = spec.components.schemas.StepFinishPart
const diagnostic = spec.components.schemas.ProviderTermination
expect(spec.openapi).toBe("3.1.1")
expect(part.properties.termination).toEqual({ $ref: "#/components/schemas/ProviderTermination" })
expect(part.$schema).toBe("https://json-schema.org/draft/2020-12/schema")
expect(diagnostic.$schema).toBe("https://json-schema.org/draft/2020-12/schema")
expect(part.id).toBeUndefined()
expect(diagnostic.id).toBeUndefined()
expect(part.$id).toBeUndefined()
expect(diagnostic.$id).toBeUndefined()
expect(part.properties.type.const).toBe("step-finish")
expect(stdout).not.toMatch(/"ref"\s*:/)
expect(stdout).not.toMatch(/"nullable"\s*:/)
expect(part.required).not.toContain("termination")
expect(diagnostic.required).toContain("normalizedReason")
expect(diagnostic.properties.rawReason.properties.value.maxLength).toBe(128)
expect(diagnostic.properties.diagnostic.properties.value.maxLength).toBe(2048)
expect(diagnostic.properties.requestID.properties.status.enum).toEqual(["available", "unavailable", "redacted"])
})
Loading
Loading