experiments: support outcome-backed metrics (outcome:accepted, outcome:zero_touch) #74988

Analysis

gh aw experiments analyze gets its per-run measurements from only two sources: grader artifacts and eval answers. The function that loads them is loadMetricObservationSetsForAnalysis (experiments_command.go#L286). The compiler recognizes only eval: and grader: metric references (compiler_experiments.go#L428, #L453). Graders read only the run's own trace and have no network access, so they can't see a merge that happens after the run.

daily-harness-experiment-proposer tells the agent to prefer "a labeled/merged PR outcome" as the primary metric (L274). It also requires guardrails to be grader: or eval: (L282). The analyzer can't measure an outcome metric, so the experiment stays at EXTEND. As a result, the proposer's PROMOTE/REJECT ledger can only learn from signals measured during the run.

Outcome evaluation already exists and is deterministic: collectRunOutcomes (logs_outcomes.go#L68) and evalCreatePullRequest (outcome_eval_pr.go#L16). This issue proposes feeding those results into experiment analysis.

Outcome lag is short but varies by workflow

I checked the last 1,000 PRs opened by gh-aw workflows in this repo (author app/github-actions, 2026-08-08 to 2026-11-04):

• Time to merge: median 1.7 h, 90th percentile 8.5 h, maximum 38 h.
• 97.2% were merged or closed within 24 h, and 99.9% (999 of 1,000) within 72 h.

Across the 25 workflows with at least 10 PRs, the 95th-percentile time to decision ranges from 2.2 h (ci-coach, smoke-multi-pr) to 48.7 h (jsweep). Repos with slower human review will differ even more. No single fixed window fits everyone. Too long delays analysis for fast workflows. Too short scores slow-but-good PRs as not accepted. The window should be learned per workflow from its own outcome data.

Many workflows merge more than 90% of their PRs, which leaves little room for accepted to separate variants. zero_touch is the stricter metric for those workflows.

Proposed semantics

"Outputs" means safe outputs that produce an outcome status (see rule 3). "Resolved" means accepted, rejected, or still pending after the window (see rule 2).

1. Maturity window (learned). Only runs older than the window W are measured. Younger runs are reported as immature and don't count toward min_samples. By default (outcome_window: auto), W is learned per workflow at analyze time:
   • Estimate time to decision from the experiment's own outcome-bearing outputs, with all variants pooled so the window doesn't favor one variant.
   • Treat outputs that are still pending as censored, using a Kaplan–Meier estimate, rather than dropping them. Dropping them biases W short, because the slowest outputs are exactly the ones still open.
   • Set W to the time by which an estimated 95% of outputs are decided, clamped to between 24 h and 30 days.
   • If there are fewer than 20 decided outputs, fall back to 72 h. An explicit duration such as outcome_window: 7d overrides learning.
   • Report W, its source (learned, configured, or default), and the sample it came from in the analyze output.
2. Stale pending. An output still pending after the window counts as not accepted (0) and appears in the coverage report.
3. Exclusions. unknown and error are excluded and reported as unavailable, the same way eval UNKNOWN answers are. lifecycle, lifecycle_close, ignored, and skipped don't produce an outcome status.
4. No-op runs. Runs with no outputs that produce an outcome status are excluded and reported per variant. This prevents a variant from winning by doing less.
5. Zero-touch evidence. Today ZeroTouch is false both when a human touched the PR and when the supporting evidence was incomplete (outcome_eval_pr.go#L128). This metric must exclude incomplete evidence instead of scoring it 0.
6. CLI-only. Outcomes are evaluated when experiments analyze runs. This matches ADR-70205, which uses no scheduled collector.

Implementation plan

Please implement the following changes:

1. Compiler (pkg/workflow/compiler_experiments.go, ExperimentConfig, pkg/parser/schemas/main_workflow_schema.json):
   • Add ParseExperimentMetricOutcomeReference. It should accept outcome:accepted and outcome:zero_touch in metric, secondary_metrics, and guardrail_metrics.
   • Reject unknown outcome kinds using the error style "[what's wrong]. [what's expected]. [example]". Example: unknown outcome metric "outcome:merged". Expected outcome:accepted or outcome:zero_touch. Example: metric: outcome:accepted.
   • Add an optional outcome_window: auto (default) or a duration.
2. Observations (new pkg/cli/experiments_outcome_observations.go, modeled on experiments_eval_observations.go):
   • For each assigned run in the assignment history, call collectRunOutcomes.
   • Add estimateOutcomeWindow, a pure function that implements rule 1: Kaplan–Meier over the pooled decided and censored times, a 95% target, clamping, and the cold-start fallback. Then apply rules 1–5.
   • Produce one {run_id, variant, value} observation per run, using the existing observation-set type.
   • Merge the results in loadMetricObservationSetsForAnalysis, for both the primary metric and guardrails.
3. Zero-touch evidence (pkg/cli/outcome_eval_pr.go, OutcomeReport in pkg/cli/outcome_eval.go): record whether the zero-touch evidence was complete, so rule 5 can tell incomplete evidence apart from a PR a human touched.
4. Analysis and reporting:
   • Reuse the existing statistics. When analysis_type is absent, apply the R-STAT-004 defaults: the two-proportion test when every per-run value is 0 or 1, and Welch's t-test otherwise.
   • Report the window used (W, its source, and the sample size) and per-variant coverage (immature, stale_pending, unavailable, no_op) in both text and --json output.
5. Proposer (.github/workflows/daily-harness-experiment-proposer.md): allow outcome:accepted and outcome:zero_touch as primary and guardrail metrics.
6. Tests:
   • Fixtures for mature and immature runs, merged, closed, and stale-pending PRs, no-op runs, runs that create several PRs, and zero-touch with incomplete evidence.
   • estimateOutcomeWindow tests: censored pending outputs, the cold-start fallback, clamping at both bounds, and an explicit override.
   • Compiler validation tests for valid and invalid outcome: references and for outcome_window (auto, a valid duration, and an invalid value).
7. Docs: update docs/src/content/docs/experimental/experiments.md and experiments-specification.md.
8. Follow guidelines: run make agent-finish.

Out of scope

• Post-merge reverts and escaped defects for create_pull_request. That can be a separate follow-up.
• Scheduled or background outcome collection.