Expose execution outcome and retry measurement inputs

[byapparov]

Context

An observed headless execution using CLI 0.4.3 and Gemini 2.5 Flash completed one file-read tool call, then emitted a terminal message with finish: "error". No final text was recorded; there was no session-error event, and the process exited 0. A downstream consumer classified the result as incomplete despite the successful process exit.

The pinned Google SDK 2.0.54 maps only MALFORMED_FUNCTION_CALL to normalized error. This is strong evidence for a malformed generated function call, but the raw response and exact invalid call were not retained. Do not describe the exact offending call or its cause as proven.

Problem / Goal

Consumers need trustworthy denominators and release/model dimensions to calculate failure rate, contradictory outcomes, diagnostic coverage, and retry effectiveness. Existing invocation/session/message identities and usage events should be reused rather than introducing a second telemetry stream.

Proposed Approach

Audit the existing event contract and add only missing CLI-owned measurement inputs: CLI version distinct from schema version, resolved model/provider identity, terminal outcome/reason correlation, and lifecycle telemetry for existing retries. Define how to derive metrics from events, including interrupted streams. Keep template/version and review completeness enrichment in the downstream application.

Acceptance Criteria

• Publish an event-to-metric contract defining invocation outcome rate, provider-turn error rate, contradiction count, diagnostic coverage, and recovery rate, with explicit denominators and unknown/censored outcomes.
• CLI release identity is available even for pre-session failures; model/provider and message identity are included where resolved. Existing invocation/session IDs remain stable.
• Existing retry attempts expose attempt identity/ordinal, reason, delay, and eventual outcome with parent correlation; no new retry behavior is introduced here.
• Additional retry latency and known usage/cost can be calculated without counting a terminal summary twice. Unavailable cost or usage stays unknown, not zero.
• Fixtures cover clean empty success, error finish without exception, thrown error, retry then success, exhausted retries, pre-session failure, cancellation, and an abruptly truncated event stream.
• A reference aggregation example or contract test yields expected counts from fixtures, distinguishes child sessions from top-level invocations, and does not treat missing terminal events as success.
• Document safe metric dimensions: release/provider/model/reason; request IDs and message IDs are correlation attributes, not metric labels. Do not attach free-text errors to metric labels.
• Coordinate the schema with Comprehensive OpenTelemetry integration for observability #22 without making an OTEL exporter or backend dashboard a dependency.

Out of Scope

Product-specific finding counts, file coverage, template metadata, GitHub status, backend ingestion/dashboards/alerts, implementing a new OTEL exporter, and introducing automatic retries.

Roadmap Alignment

• Pillar: EXEC; Q3 2026 pilot-ready executor reliability and code-review quality.
• Priority: P1 — enables reliable operational measurement.
• Milestone: Enterprise Observability (CLI repository milestone; repository-specific milestone numbering).
• Uses existing repository labels; priority and pillar are recorded here because matching labels do not exist in this repository.

References

• Parent lifecycle tracker: Epic: Make headless NDJSON lifecycle and usage telemetry reliable #89; broader OTEL/export work: Comprehensive OpenTelemetry integration for observability #22.
• CLI v0.4.3: packages/cli/src/session/processor.ts, src/session/prompt.ts, src/cli/cmd/run.ts, and src/cli/cmd/run.invocation.ts (paths relative to packages/cli except the first).
• Pinned Google finish-reason mapping.
• Related earlier work: Fix: aictrl run exits 0 on auth failure, masking broken CI workflows #70 (exception-driven exit status), NDJSON event schema gaps for downstream observability consumers #63 (event contract), Add NDJSON invocation lifecycle events before session creation #90 (invocation lifecycle), Preserve unknown usage on failed headless turns #93 (unknown usage), Add model stream idle-timeout handling #80 (stream idle timeout).

Sequencing

Depends on the outcome and diagnostic contracts: #108, #109