feat(harness): pi.dev challenger harness (Phase A) - #38
Merged
Conversation
Add a `pi` harness (@earendil-works/pi-coding-agent) behind the switchboard as the challenger to the `anthropic` control, so the eval can A/B them (SEAM_WIZARD_HARNESS=pi). For a fair comparison it mirrors the anthropic harness: the Seam inference proxy is registered as an anthropic-messages provider (scoped wizard token as Bearer, no new secret); read/search/write tools only (no bash); and the seam-docs MCP is bridged in via pi-mcp-adapter using the SAME `npx mcp-remote https://mcp.seam.co` stdio command, reusing mcp-remote's OAuth cache. Cost is estimated from pi's token stats (the proxy meters real cost server-side). Heavy deps load lazily so the anthropic path never pulls them. HarnessRunStepArgs gains `inference`. Not runtime-verified from CI (a pi run needs a real agent invocation + MCP OAuth); validate with `SEAM_WIZARD_HARNESS=pi SEAM_API_KEY=… npm run eval`. The one thing to check on that first run is tool-name parity between pi's MCP adapter and the shared prompt's mcp__seam-docs__* references.
itelo
force-pushed
the
itelo/pi-harness
branch
from
August 20, 2026 20:28
438fc27 to
abba4c9
Compare
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase A — the
pichallenger harness, completing the control-vs-challenger eval setup. Adds apiharness (@earendil-works/pi-coding-agent) behind the switchboard alongside theanthropiccontrol, selectable viaSEAM_WIZARD_HARNESS=piand runnable through the eval.For a fair A/B it mirrors the anthropic harness:
anthropic-messagesprovider (scoped wizard token as the Bearer the proxy authenticates — no new secret).create{Read,Edit,Write,Ls,Find,Grep}ToolDefinition,noTools: 'builtin') — no bash, matching the control.pi-mcp-adapterusing the samenpx mcp-remote https://mcp.seam.co/mcpstdio command the anthropic harness uses, so it reuses mcp-remote's OAuth token cache.pi-coding-agent,pi-mcp-adapter,jiti) load lazily via dynamic import, so theanthropicpath never pulls them at runtime.HarnessRunStepArgsgainsinference(the anthropic harness usesagentEnv; pi usesinferenceto register the provider).SEAM_WIZARD_HARNESS=pi SEAM_API_KEY=… npm run evaltoolPrefix: 'seam-docs') and the shared system prompt'smcp__seam-docs__*references — may need a prompt/prefix tweak. The proxy discovery tool is left enabled as a hedge.Note
pi-coding-agentis a heavy dependency (pulls pi-ai/pi-agent-core/pi-tui). It's the cost of shipping the challenger; the lazy import keeps it off the default runtime path.Security & Compliance