feat: add durable plan sharding

[joshblack]

Follow up to #203.

This PR adds a durable execution plan that can be created once, shared across jobs, and replayed without rebuilding trial order. Plans store stable trial IDs plus the scenario, treatment, model, and benchmark capability references needed to resolve each trial from the current configuration.

The benchmark and experiment workflows now use the same four-shard flow: create plan.json, run each shard from that plan into output-{shard}.json, then combine those manifests into the standard portable output.json. The merge keeps the existing per-trial file references instead of reading and rewriting every trial result, which keeps the shard handoff small and works with the split output layout from #212.

This is the fifth entry in stack 214. #199 adds the benchmark scenarios last.

Changelog

New

• Add --plan [path] and --from-plan [path] CLI modes.
• Add --shard <order/total> for deterministic plan sharding.
• Add --merge-results with --output-dir for combining output-*.json manifests.
• Export durable plan schemas, types, serialization, resolution, and selection APIs.

Changed

• Update shard runs to write output-{shard}.json manifests that reference split trial result files.
• Update shard merging to validate and combine manifests without reading or rewriting per-trial JSON files.
• Update the benchmark and experiment runner scripts with matching plan, shard, and merge modes.
• Update both Actions workflows to create one plan, run four shards, and retain one merged result bundle.

Removed

• None.

[Copilot]

Copilot review overview

🟡 Changes recommended

The experiment workflow’s merge job is missing a dependency install step (and inconsistent pnpm setup), which is likely to break CI execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1

New issues introduced by this change (1)

Severity	Finding
	.github/​workflows/​experiment.yml — The experiment workflow merge job runs pnpm run build without installing dependencies, and it…

[Copilot]

Copilot review overview

🔵 Needs a closer look

The experiment workflow’s merge job is missing a dependency install step and is inconsistent with the pnpm setup used in the other jobs, which is likely to break CI.

Review tier: Lite
Findings: 1

Pre-existing issues (1)

Severity	Finding
	.github/​workflows/​experiment.yml — The experiment workflow merge job runs pnpm run build without installing dependencies, and it… View comment

Suppressed comments (1)

.github/workflows/experiment.yml:129

• The merge job builds without installing dependencies, and it uses pnpm/setup (v2) while the other jobs use pnpm/action-setup. This will likely fail at pnpm run build and makes the workflow inconsistent across jobs; align the merge job with the plan/run jobs by using pnpm/action-setup, enabling pnpm caching on setup-node, and running pnpm install --frozen-lockfile before the build.

      - name: set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version-file: '.nvmrc'
      - name: build project

[Copilot]

Copilot review overview

🔵 Needs a closer look

The experiment workflow’s merge job uses a different pnpm setup path than the plan/run jobs, which risks inconsistent pnpm behavior between shards and merge.

Review tier: Lite
Findings: 1

Pre-existing issues (1)

Severity	Finding
	.github/​workflows/​experiment.yml — The experiment workflow merge job runs pnpm run build without installing dependencies, and it… View comment

Suppressed comments (1)

.github/workflows/experiment.yml:124

• The merge job uses pnpm/setup while the plan/run jobs use pnpm/action-setup + setup-node pnpm caching. Mixing pnpm setup actions in the same workflow can lead to different pnpm versions/install behavior between shards and the merge job, which risks non-reproducible merges or unexpected failures when the action defaults change. Consider aligning the merge job to the same pnpm+Node setup pattern as the other jobs (including explicit pnpm install --frozen-lockfile).

      - name: set up pnpm
        uses: pnpm/setup@703c52620218391530e48b9e8870d5c0082e1b9b # v2.1.0
        with:
          cache: true
          require-lockfile: true

[Copilot]

Copilot review overview

🟡 Changes recommended

The updated experiment workflow merge job is likely to fail on clean runners (missing dependency installation and inconsistent pnpm setup), and shard-merge should validate trial reference paths before producing a merged manifest.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 · 1

New issues introduced by this change (1)

Severity	Finding
	packages/​agent-eval/​src/​plan.ts — mergeTrialReferences currently accepts trial file references verbatim. A shard output that…

Pre-existing issues (1)

Severity	Finding
	.github/​workflows/​experiment.yml — The experiment workflow merge job runs pnpm run build without installing dependencies, and it… View comment

Suppressed comments (1)

.github/workflows/experiment.yml:130

• The experiment workflow merge job runs pnpm run build without installing dependencies, and it uses pnpm/setup while the other jobs use pnpm/action-setup + setup-node pnpm cache. This is likely to fail in a clean runner because node_modules will be missing.

      - name: set up pnpm
        uses: pnpm/setup@703c52620218391530e48b9e8870d5c0082e1b9b # v2.1.0
        with:
          cache: true
          require-lockfile: true
      - name: set up Node.js
        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
        with:
          node-version-file: '.nvmrc'
      - name: build project
        run: pnpm run build