Skip to content

feat(harness): give pi adaptive thinking for a fair A/B #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
itelo merged 1 commit into from
Aug 21, 2026
Merged

feat(harness): give pi adaptive thinking for a fair A/B #44

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions src/lib/steps/harness/pi.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,14 +59,15 @@ export const piHarness: Harness = {
name: modelId,
api: 'anthropic-messages',
baseUrl: inference.base_url,
// Must be false: pi-ai emits the deprecated `thinking:{type:'enabled',
// budget_tokens}` for reasoning models, which Opus 4.8 / Sonnet 5
// reject with a 400 (they only accept `thinking:{type:'adaptive'}`).
// With reasoning on, every request errored server-side → 0 tool calls,
// empty diff. Off means no extended thinking on the pi path, but the
// requests succeed. (The anthropic control uses the SDK's adaptive form;
// thinking parity needs a newer pi-ai.)
reasoning: false,
reasoning: true,
// forceAdaptiveThinking: send `thinking:{type:'adaptive'}` +
// output_config.effort (what Opus 4.8 / Sonnet 5 require) instead of
// pi-ai's default deprecated `thinking:{type:'enabled',budget_tokens}`,
// which those models reject with a 400 — matching the anthropic
// control's adaptive thinking. supportsTemperature:false because Opus
// 4.7+ rejects a non-default temperature. Both mirror how pi-ai's own
// built-in anthropic-messages models are configured.
compat: { forceAdaptiveThinking: true, supportsTemperature: false },
input: ['text'],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 1_000_000,
Expand Down Expand Up @@ -133,6 +134,8 @@ export const piHarness: Harness = {
const { session } = await createAgentSession({
model,
modelRegistry: registry,
// Adaptive-thinking effort, matching the anthropic control's medium effort.
thinkingLevel: 'medium',
cwd,
sessionManager: SessionManager.inMemory(cwd),
resourceLoader,
Expand Down
Loading