Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions assets/CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,9 @@ Once the reset passes or the login is restored, the item has cleared: whoever fi

The PR body's `## Models` section records the PR's size and each implementing session's harness, model, and effort.
A quest PR's size is its quest's; a skill that opens its own PR, such as `/quest-plan`, sizes it as a quest is sized.
List them as `Size: M` and one `Implementer: Codex, gpt-6.1-sol, high effort` line per session, which each session adds when it first pushes.
Read your model and effort rather than guess: Claude Code knows its model and exports `CLAUDE_EFFORT`; Codex records `model` and `effort` in the last `turn_context` of `~/.codex/sessions/*/*/*/rollout-*-$CODEX_THREAD_ID.jsonl`; Grok records `model_id` and `reasoning_effort` in `~/.grok/sessions/*/$GROK_SESSION_ID/chat_history.jsonl`.
Write `unknown` for any you cannot find.
The PR body's `## Review` section records the latest round: the reviewer harness, model, and effort, the reviewed head SHA, the round's score out of 10 with its rationale, and each finding's disposition (fixed, or refuted with the reason), or the skip and its reason.
Never block on a score: the findings and their refutations decide when the review is done.
Rewrite it after each round.
Expand Down
9 changes: 7 additions & 2 deletions cloudflare/src/github/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ import {
} from "quest/core";
import { checksState, latestContexts } from "../../../src/github/checks";
import { cleanAutoMergeRefusal, immediateMerge, queueHead } from "../../../src/github/merge";
import { implementers, withModels } from "../../../src/github/models";
import { BODY_LIMIT } from "../jobs/trail";
import { osiApproved } from "../licenses";
import { CI_PERMISSIONS, type CIPermission, ciProblems } from "./ci";
import { OBSERVED, type Observed, observedGap, REVIEW_CHECK, unreviewed } from "./review";
Expand Down Expand Up @@ -2204,10 +2206,13 @@ export class GitHubService {
if (pull?.state !== "open") throw error;
}
}
if (pull.title !== input.title || pull.body !== input.body)
// A later session's body keeps the sessions that delivered before it in its `## Models` record.
const body = implementers(pull.body ?? "").reduce(withModels, input.body);
if (body.length > BODY_LIMIT) throw new GitHubError("The pull request body is too long");
if (pull.title !== input.title || pull.body !== body)
pull = await this.#api<PullHead>(token, `${pulls}/${number(pull.number)}`, "PATCH", {
title: input.title,
body: input.body,
body,
});
// Finished work leaves draft; a maintainer's own choice to mark it ready is never reverted.
if (pull.draft && !input.draft) {
Expand Down
20 changes: 20 additions & 0 deletions cloudflare/test/seed.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -465,6 +465,26 @@ it("publishes jobs' work end to end, through the upstream branch to the pull req
// Finished work marks it ready for review.
await publish({ job: "seed-start-offline", head: first.head, pull: { ...pull, draft: false } });
expect(rocket.pulls.at(-1)!.draft).toBe(false);
// A later session's body keeps the sessions that delivered before it in its `## Models` record.
const models = (lines: string) => ({
...pull,
draft: false,
body: `Done.\n\n## Models\n\n${lines}\n\n## Review\n\nNone`,
});
await publish({
job: "seed-start-offline",
head: first.head,
pull: models("- Implementer: claude, first, high effort"),
});
await publish({
job: "seed-start-offline",
head: first.head,
pull: models("- Size: S\n- Implementer: opencode, second, high effort"),
});
expect(rocket.pulls.at(-1)!.body).toBe(
models("- Size: S\n- Implementer: opencode, second, high effort\n- Implementer: claude, first, high effort")
.body,
);

const continued = await latest("quest/m0/import.md");
expect(continued).toMatchObject({ id: retry.id, state: "running", head: rocket.branches["quest/m0/import"] });
Expand Down
15 changes: 9 additions & 6 deletions quest/m0/model-routing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ can run from a ranked list `quest` builds in for the quest's size, the same
for every repository unless its `AGENTS.md` overrides it: Grok for XS, Opus
from S up, with Sol as the fallback through M, and a usage limit falling
through to the next entry. Sol reviews, at an effort set by size, and Opus
reviews whatever Sol wrote, so no model reviews its own code. Every commit
and PR records the models and efforts behind it, and fixes link the PR that
reviews whatever Sol wrote, so no model reviews its own code. Every PR
records the models and efforts behind it, and fixes link the PR that
caused them, so the rankings can be audited and retuned. This epic's own
work is one live spawn in moq-dev/quest from Claude Code that starts an XS
quest on Grok and an S or larger on Opus, each recorded and reviewed by Sol.

Non-goals: routing kix.quest's runner jobs, which pick their own accounts
and models (their commits still record their models); billing agents to API
and models (their PRs still record their models); billing agents to API
keys, which this epic removes.

## Plan
Expand Down Expand Up @@ -67,8 +67,12 @@ are mostly judgment.
Sol xhigh. L and XL: Opus xhigh, no fallback. Reviews: Sol low, medium,
high, xhigh, xhigh for XS to XL, never the model that wrote the code;
Pullfrog and CodeRabbit add free reviews on top.
- Every commit and PR records its models and efforts, and every fix links
the PR that caused it, so the rankings can be audited (user).
- Every PR records its models and efforts, and every fix links the PR that
caused it, so the rankings can be audited (user). The PR record alone,
without commit trailers (user, 2026-11-04 design checkpoint of
[moq-dev/quest#494](https://github.com/moq-dev/quest/pull/494), revising
that day's /quest-plan pick of `Agent-*` trailers): squash merges drop
branch trailers from `main`, so audits read the PR record anyway.
- Every `/quest-review` scores the change out of 10 (user, while planning
[the council](/quest/m2/council.md), which joins m2).
- Which milestone? ✅ m0 (rejected: m1). It ranks first among m0's work
Expand All @@ -78,6 +82,5 @@ are mostly judgment.

- [Grok agents](/quest/m0/model-routing/agent-harnesses.md) - `quest agent` starts Grok as well as Claude Code and Codex
- [Route by size](/quest/m0/model-routing/models.md) - `/quest-spawn` starts each quest on the first entry of its size's ranked list that can run
- [Commits and PRs record their models](/quest/m0/model-routing/implementer.md) - commit trailers and PR records name each harness, model, and effort, and the PR's size
- [kix records its follow-up sessions](/quest/m0/model-routing/kix-sessions.md) - kix `iterate` and `merge` jobs that change an existing PR record their `Implementer:` line in a PR comment
- [moq-dev pins](/quest/m0/model-routing/pins.md) - moq-dev/moq and moq-dev/moq.pro bump their `quest` pin to pick up the routing
39 changes: 0 additions & 39 deletions quest/m0/model-routing/implementer.md

This file was deleted.

4 changes: 0 additions & 4 deletions quest/m0/model-routing/kix-sessions.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,3 @@ Implementation guidance:
comment too, and `quest context`'s `## Models` rule and its
`--exclude-model` rule ("each implementing model the PR body records")
to name it.

## Required

- [Commits and PRs record their models](/quest/m0/model-routing/implementer.md) - this builds on its `## Models` record and the runner's reporting
1 change: 0 additions & 1 deletion quest/m0/model-routing/pins.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,3 @@ repository uses the same settings, so nothing in them changes but the pin.
## Required

- [Route by size](/quest/m0/model-routing/models.md) - `/quest-spawn` reads the built-in size table
- [PRs record their implementer](/quest/m0/model-routing/implementer.md) - the record that shows the routing working
6 changes: 3 additions & 3 deletions quest/m1/model-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ the size tables of [model routing](/quest/m0/model-routing/README.md):
([m0's run](/quest/m0/model-check.md)), then each milestone (rejected:
each milestone only)

Audit the routing from the PRs merged since the last run, with
[`quest usage`](/quest/m1/usage.md) once it lands, which record
their size, implementer, and reviewer beside `## Review`:
Audit the routing from the records of the PRs merged since the last run,
with [`quest usage`](/quest/m1/usage.md) once it lands: each body's
`## Models` size and implementers and its `## Review` reviewer:

- Tally by size and by implementing harness, model, and effort: review
scores, review rounds, findings fixed versus refuted, design checkpoints reached, PRs
Expand Down
4 changes: 2 additions & 2 deletions quest/m1/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ Agent's choices, for the implementer to revisit:
- A session belongs to a quest or PR by its working directory's checkout
and branch: a quest branch names its quest, and a PR's head branch names
the PR. Sessions on no quest or PR branch roll up as unattributed.
- Accuracy comes from the records: commit trailers and the PR body's
size, implementers, and each `## Review` round's reviewer, score, and findings
- Accuracy comes from the PR record: the body's `## Models` size and
implementers, and each `## Review` round's reviewer, score, and findings
with their dispositions; follow-up fixes are later PRs that list it under
`## Caused by` (`quest context`'s Pull requests section).
- Prices are a pinned table beside the model lists, by model and token
Expand Down
29 changes: 24 additions & 5 deletions src/cli/land-fix.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,14 @@ export const signature = (failure: string) => failure.replace(/[`'"]/g, "").repl
* a commit on the base that skips the test and adds its follow-up quest, which the daemon opens as a PR
* of its own.
*/
export type FixResult = { agent?: string; model?: string; blame?: boolean; comment?: string; failure?: string } & (
export type FixResult = {
agent?: string;
model?: string;
effort?: string;
blame?: boolean;
comment?: string;
failure?: string;
} & (
| { number: number; action: "requeue"; reason: string; cause?: Cause; disable?: string }
| { number: number; action: "fixed"; head: string; reason: string }
| { number: number; action: "handback"; diagnosis: string; recommendation: string; log?: [number, number] }
Expand All @@ -97,7 +104,7 @@ export type FixResult = { agent?: string; model?: string; blame?: boolean; comme
* The fixer's verdict on the base alone, when no PR is supplied: a mechanical fix committed on its
* `base` branch, which the daemon opens as a PR, or a handback, which halts the queue with an issue.
*/
export type BaseFix = { agent?: string; model?: string } & (
export type BaseFix = { agent?: string; model?: string; effort?: string } & (
| { action: "fixed"; head: string; reason: string }
| { action: "handback"; diagnosis: string; recommendation: string }
);
Expand All @@ -122,7 +129,7 @@ function validate(item: any, pr: boolean): void {
: item.action === "handback"
? [...named, "action", "diagnosis", "recommendation"]
: [];
const metadata = ["agent", "model"].filter((key) => key in item);
const metadata = ["agent", "model", "effort"].filter((key) => key in item);
if (metadata.some((key) => typeof item[key] !== "string" || !/^[a-zA-Z0-9 ._/-]{1,100}$/.test(item[key])))
throw new Error("invalid fixer identity");
if (pr && "blame" in item) {
Expand Down Expand Up @@ -295,13 +302,18 @@ export async function fix(
...(model ? ["--model", model] : []),
"-",
];
// Codex names its model in the header it prints first, whether or not the command line does.
let header = "";
await new Promise<void>((done, reject) => {
// Relay the agent's output through pipes of its own rather than letting it inherit ours. Once Bun writes
// to a pipe it marks that pipe non-blocking, and an inheriting child shares the flag: Codex then panicked
// (exit 101) with EAGAIN as soon as one command's output outran the host's reader.
const child = spawn(program, argv, { cwd: root, env, stdio: ["pipe", "pipe", "pipe"] });
child.stdout.pipe(process.stdout, { end: false });
child.stderr.pipe(process.stderr, { end: false });
child.stderr.on("data", (chunk: Buffer) => {
if (header.length < 4096) header += chunk.toString("utf8");
});
child.once("error", reject);
child.stdin.on("error", () => {});
child.stdin.end(
Expand All @@ -312,8 +324,15 @@ export async function fix(
const parse = (text: string) => (numbers.length ? parseFixResult(text, numbers) : parseBaseFix(text));
const results = parse(readFixResult(root));
const index = argv.findIndex((arg) => arg === "-m" || arg === "--model");
const ranModel = index >= 0 ? argv[index + 1] : argv.find((arg) => arg.startsWith("--model="))?.slice(8);
const identity = { agent: basename(agent), model: ranModel ?? "Codex" };
const ranModel =
(index >= 0 ? argv[index + 1] : argv.find((arg) => arg.startsWith("--model="))?.slice(8)) ??
/^model: ([a-zA-Z0-9._/-]{1,100})$/m.exec(header)?.[1];
const identity = {
agent: basename(agent),
model: ranModel ?? "unknown",
// Replaced arguments may run the agent at any effort.
effort: args ? "unknown" : effort,
};
const result = JSON.stringify(
Array.isArray(results) ? results.map((item) => ({ ...item, ...identity })) : { ...results, ...identity },
);
Expand Down
9 changes: 5 additions & 4 deletions src/cli/land-fixer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -711,14 +711,15 @@ export function fixerText(text: string): string {
.replace(/[\r\n\u0000-\u001f\u007f]/g, " ");
return `\`${value}\``;
}
export function author(result?: Pick<FixResult, "agent" | "model">): string {
/** The fixer's harness, model, and effort, as a PR's `## Models` record names a session. */
export function author(result?: Pick<FixResult, "agent" | "model" | "effort">): string {
const agent = result?.agent ?? "Codex";
const model = result?.model;
return (
[agent, model && model !== agent ? model : undefined]
[agent, model && model !== agent ? model : undefined, result?.effort && `${result.effort} effort`]
.filter(Boolean)
.join(" / ")
.replace(/[^a-zA-Z0-9 ._/-]/g, " ")
.join(", ")
.replace(/[^a-zA-Z0-9 ,._/-]/g, " ")
.slice(0, 100) || "Codex"
);
}
Expand Down
33 changes: 33 additions & 0 deletions src/github/models.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
// A pull request body's `## Models` record, which names the PR's size and each session that implemented it.

const SECTION = /^## Models[ \t]*\r?$/m;

/** Where `body`'s `## Models` section runs, after its heading, or null without one. */
function section(body: string): { start: number; end: number } | null {
const heading = SECTION.exec(body);
if (!heading) return null;
const start = heading.index + heading[0].length;
const next = /^## /m.exec(body.slice(start));
return { start, end: next ? start + next.index : body.length };
}

/** The `Implementer:` items of `body`'s `## Models` section, each as its line without the bullet. */
export function implementers(body: string): string[] {
const found = section(body);
if (!found) return [];
return body
.slice(found.start, found.end)
.split(/\r?\n/)
.flatMap((line) => /^- (Implementer: .+?)\s*$/.exec(line)?.[1] ?? []);
}

/** `body` with `line` as an item of its `## Models` section, which it gains when it has none. */
export function withModels(body: string, line: string): string {
const item = `- ${line}`;
if (body.split(/\r?\n/).some((each) => each.trim() === item)) return body;
const found = section(body);
if (!found) return `${body.trimEnd()}\n\n## Models\n\n${item}\n`;
const { start, end } = found;
const gap = body.slice(start, end).trim() ? "\n" : "\n\n";
return `${body.slice(0, end).trimEnd()}${gap}${item}\n${end < body.length ? "\n" : ""}${body.slice(end)}`;
}
2 changes: 2 additions & 0 deletions src/runner/adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ export interface Running {
done: Promise<Exit>;
/** Ask the CLI to stop, killing it after a grace period. */
stop(): void;
/** The model and effort the session runs on, each `unknown` where the CLI does not show it. */
ran(): Promise<{ model: string; effort: string }>;
}

export interface SignedIn {
Expand Down
Loading
Loading