diff --git a/AGENTS.md b/AGENTS.md index 2fcf00b..651f895 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,7 +50,7 @@ changing credentials, claims, Sandboxes, publishing, or GitHub writes. - Validate every external or cross-process input with Zod at its boundary. - Build agent output objects with `z.strictObject`. Bound strings and arrays. - Keep task inputs and results small and JSON-serializable. -- Open Sandboxes through `withSandbox`. In claiming tasks, call +- Open Sandboxes through `withSandbox`. In GitHub tasks, call `claim.recordSandbox()` immediately after opening one. - Use `ExpectedFailure` only for expected messages safe to publish on GitHub. Keep outcome names in snake_case. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 506c85d..48f071f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -83,7 +83,7 @@ npx vitest run tests/gateway.test.ts Follow [docs/runbooks/extend.md](docs/runbooks/extend.md) to: - add a typed agent and prompt; -- add a claiming or maintenance task; +- add a GitHub or maintenance task; - route a GitHub event, label, or command; - update GitHub App requirements; - test authority and user-visible behavior; and diff --git a/README.md b/README.md index a150a61..b42566f 100644 --- a/README.md +++ b/README.md @@ -5,28 +5,43 @@ workflows on Render. It includes a complete issue-to-PR workflow that you can run as-is, change one agent at a time, or use as the foundation for different automations. -The template provides the infrastructure common to agent workflows: +Start by [forking this repository](https://github.com/render-examples/software-factory/fork). +Deploy from your fork: its `factory.config.ts` defines the repositories the +factory may access. -- a signed-webhook gateway; -- durable Render Workflows tasks with cancellation and run ownership; -- short-lived Render Sandboxes for repository code and model-directed tools; -- typed agent inputs and outputs; -- GitHub App authentication and artifact markers; and -- validated publishing that keeps write credentials outside Sandboxes. +The recommended setup is agent-guided: -## Included workflow +1. Check the prerequisites in the [Quickstart](docs/quickstart.md). +2. Ask a coding agent to follow the [Setup runbook](docs/runbooks/setup.md). +3. Complete the GitHub, secret, and paid-resource checkpoints when the agent + asks. -```text -issue --factory:triage--> specification -specification --factory:spec-approved--> draft pull request -pull request --factory:pr-review--> COMMENT review -``` +The runbook configures the fork before deployment, creates a GitHub App, +deploys the Render Blueprint, connects the webhook, verifies both services, +and runs one Triage smoke test. + +**What software-factory solves for you:** +1. Authenticating GitHub events and routing them to durable Render Workflows tasks. +2. Running agents and repository tools inside short-lived, isolated Render Sandboxes. +3. Handling overlapping runs, cancellation, cleanup, and idempotency. +4. Validating typed agent output and safely publishing proposed code to GitHub. +5. Providing deployment configuration, setup scripts, health checks, and structured telemetry. + +**What software-factory leaves for you:** +1. The agents, prompts, and workflow logic for your use case. +2. Repository-specific install and verification commands. +3. The GitHub triggers, approval gates, and permissions your workflow needs. +4. Model selection, budgets, and evaluation strategy. +5. Channel integrations. +6. Persistent data store. -- **Triage** reads the issue, discussion, and repository. It asks questions, +## Included workflow + +- **Triage:** reads the issue, discussion, and repository. It asks questions, writes a specification, or declines the request. -- **Spec-to-PR** implements the approved specification, runs your verification +- **Spec-to-PR:** implements the approved specification, runs your verification commands, and opens a draft pull request. -- **PR Reviewer** runs verification and reviews security, correctness, +- **PR Reviewer:** runs verification and reviews security, correctness, performance, and tests. It posts a non-approving GitHub COMMENT review. GitHub holds the discussion, labels, specifications, pull requests, and @@ -34,13 +49,19 @@ reviews. Render Workflows runs the tasks. Sandboxes contain the checkout, agent loop, and model-directed tools. The included workflow never approves or merges code. +The three labels above keep each transition human-gated. Apply +`factory:autopilot` to an issue to grant upfront approval for the same stages: +Triage drafts the spec, Spec-to-PR opens a draft pull request, and PR Reviewer +posts its review as child tasks in one Render Workflow. If Triage asks +questions, the workflow stops and continues when a person replies. + ## Extend the template The built-in agents and tasks are examples of the template's extension model: - Add or replace an agent with a Markdown prompt, a model tier, a bounded Zod output schema, and the minimum tools it needs. -- Add a claiming task for work attached to an issue or pull request. +- Define a GitHub task for work attached to an issue or pull request. - Add a plain Workflows task for maintenance or other work without a GitHub claim. - Route new GitHub events, labels, or commands through the authenticated @@ -53,29 +74,9 @@ publishing boundary are reusable building blocks. Start with the [extension runbook](docs/runbooks/extend.md), which covers the required files, tests, authority rules, deploys, and snapshot refreshes. -## Set up - -Start by -[forking this repository](https://github.com/render-examples/software-factory/fork). -Deploy from your fork: its `factory.config.ts` defines the repositories the -factory may access. - -The recommended setup is agent-guided: - -1. Check the prerequisites in the [Quickstart](docs/quickstart.md). -2. Ask a coding agent to follow the - [Setup runbook](docs/runbooks/setup.md). -3. Complete the GitHub, secret, and paid-resource checkpoints when the agent - asks. - -The runbook configures the fork before deployment, creates a GitHub App, -deploys the Render Blueprint, connects the webhook, verifies both services, -and runs one Triage smoke test. - ## Security -- Model-directed code and tools run inside Sandboxes, not on the Workflows - host. +- Model-directed code and tools run inside Sandboxes, not on the Workflow task. - GitHub write credentials never enter a Sandbox. - Clone and fetch credentials are read-only and removed before repository code or agents run. @@ -93,9 +94,7 @@ code or untrusted pull requests. ## Limits -- Main tasks do not chain. Labels, comments, commands, and base-branch pushes - start work through the gateway. -- Verification runs shell commands; browser testing is not built in. +- Verification runs shell commands. Browser testing is not included. - Turn limits and timeouts bound execution, but the factory has no hard dollar budget. - Anyone with `minimumPermission` can start model spend. diff --git a/app/agents/run.ts b/app/agents/run.ts index bcdf98f..801f7af 100644 --- a/app/agents/run.ts +++ b/app/agents/run.ts @@ -28,7 +28,7 @@ export class AgentError extends Error { export interface RunAgentOptions { /** Tools run only here, never in a Workflows task. */ sandbox: Sandbox; - /** Root task run of the claiming task; every event this run emits carries it. */ + /** Root task run of the GitHub task; every event this run emits carries it. */ traceId: string; runId?: string; repo?: string; diff --git a/app/claims/claim.ts b/app/claims/claim.ts index 35244dc..6944073 100644 --- a/app/claims/claim.ts +++ b/app/claims/claim.ts @@ -35,20 +35,23 @@ export interface Claim { } /** How the work ended. `summary` is shown on GitHub, so it must be safe to publish. */ -export interface Outcome { +export interface Outcome { outcome: string; summary: string; state?: "done" | "failed"; links?: StatusLink[]; + /** Typed data for a parent task. It is returned to Workflows, never rendered on GitHub. */ + data?: Data; } -/** What a claiming task returns to Render Workflows. */ -export interface TaskResult { +/** What a GitHub task returns to Render Workflows. */ +export interface TaskResult { state: Exclude; outcome: string; summary: string; traceId: string; runId: string; + data?: Data; } /** An expected failure: its message is shown on GitHub and the task succeeds. Anything else fails the task. */ @@ -70,6 +73,8 @@ export interface ClaimOptions { /** The task's name in the status comment, such as "PR Reviewer". */ title: string; actor: string; + /** Root tasks may replace another root; subtasks may only continue their own root. */ + mode?: "replace" | "continue"; gh?: GitHubClient; runs?: RunControl; } @@ -79,18 +84,18 @@ export interface ClaimOptions { * it. withClaim checks the actor's permission, stops the previous run and takes over the comment, keeps it * current, and records how the run ended. */ -export async function withClaim( +export async function withClaim( ctx: Pick, opts: ClaimOptions, - work: (claim: Claim) => Promise, -): Promise { + work: (claim: Claim) => Promise>, +): Promise> { const run = new ClaimedRun(ctx, opts); try { - if (!(await run.authorized())) return await run.reject(); + if (!(await run.authorized())) return await run.reject(); await run.acquire(); return await run.finish(await work(run)); } catch (error) { - return await run.fail(error); + return await run.fail(error); } } @@ -153,30 +158,41 @@ class ClaimedRun implements Claim { } /** A rejected trigger never held the claim; setStatus keeps its record from replacing a real run's. */ - async reject(): Promise { + async reject(): Promise> { const summary = `@${this.#opts.actor} needs \`${config.minimumPermission}\` permission on this repository to start it.`; - await finishStatus(this.#status("rejected", summary)); + await finishStatus(this.#status("rejected", summary), { rootFinished: this.#opts.mode !== "continue" }); return this.#record("rejected", "permission_denied", summary); } /** Stops the previous run and rewrites the status comment to name this one, then confirms no other run's write landed after it. */ async acquire(): Promise { const { task, title, runs } = this.#opts; - await takeOver({ gh: this.gh, repo: this.repo.fullName, subject: this.subject, task, title, runId: this.runId, runs }); + const acquired = await takeOver({ + gh: this.gh, + repo: this.repo.fullName, + subject: this.subject, + task, + title, + traceId: this.traceId, + runId: this.runId, + mode: this.#opts.mode, + runs, + }); + if (!acquired.acquired) throw new Superseded(); await this.checkpoint(); } - async finish(outcome: Outcome): Promise { + async finish(outcome: Outcome): Promise> { const state = outcome.state ?? "done"; if (!(await this.#publish(state, outcome.summary, outcome.links))) throw new Superseded(); - return this.#record(state, outcome.outcome, outcome.summary); + return this.#record(state, outcome.outcome, outcome.summary, {}, outcome.data); } /** * A superseded run and an expected failure end the task normally. Anything else is reported on GitHub without * its message, which can hold secrets or repository output, and rethrown so the task fails. */ - async fail(error: unknown): Promise { + async fail(error: unknown): Promise> { if (error instanceof Superseded) return this.#record("cancelled", "superseded", "a newer run took over."); if (error instanceof ExpectedFailure) { await this.#publish("failed", error.message); @@ -196,23 +212,41 @@ class ClaimedRun implements Claim { } async #holdsClaim(): Promise { - return (await readStatus(this.gh, this.subject.number))?.data.runId === this.runId; + const status = (await readStatus(this.gh, this.subject.number))?.data; + return status !== undefined && (status.traceId ?? status.runId) === this.traceId; } #status(state: S, summary: string, links?: StatusLink[]): StatusUpdate & { state: S } { const { task, title } = this.#opts; - return { gh: this.gh, number: this.subject.number, runId: this.runId, task, title, state, summary, links, sandboxId: this.#sandboxId }; + return { + gh: this.gh, + number: this.subject.number, + traceId: this.traceId, + runId: this.runId, + task, + title, + state, + summary, + links, + sandboxId: this.#sandboxId, + }; } /** Writes a final state while this run still holds the claim; false once another run's status is showing. */ async #publish(state: TaskResult["state"], summary: string, links?: StatusLink[]): Promise { if (!(await this.#holdsClaim())) return false; - return await finishStatus(this.#status(state, summary, links)); + return await finishStatus(this.#status(state, summary, links), { rootFinished: this.#opts.mode !== "continue" }); } - #record(state: TaskResult["state"], outcome: string, summary: string, extra: Record = {}): TaskResult { + #record( + state: TaskResult["state"], + outcome: string, + summary: string, + extra: Record = {}, + data?: Data, + ): TaskResult { const durationMs = Date.now() - this.#startedAt; emit({ ...this.#event, type: "run.finished", payload: { stage: this.#opts.task, state, outcome, durationMs, ...extra } }); - return { state, outcome, summary, traceId: this.traceId, runId: this.runId }; + return { state, outcome, summary, traceId: this.traceId, runId: this.runId, ...(data === undefined ? {} : { data }) }; } } diff --git a/app/claims/status.ts b/app/claims/status.ts index 7b9f804..9fa21c8 100644 --- a/app/claims/status.ts +++ b/app/claims/status.ts @@ -13,7 +13,7 @@ export interface StatusLink { url: string; } -/** On the issue or PR while a claiming task runs. */ +/** On the issue or PR while a GitHub task runs. */ export const WORKING_LABEL = "factory:working"; export type StatusGitHub = Pick< @@ -71,15 +71,27 @@ export interface StopContext { * was still active. */ export async function stopRun(runs: RunControl, status: StatusMarker, context: StopContext): Promise { - const active = await isRunActive(runs, status.runId); + const rootRunId = status.traceId ?? status.runId; + const active = await isRunActive(runs, rootRunId); if (active) { - await cancelActiveRun(runs, status.runId); + await cancelActiveRun(runs, rootRunId); const { repo, subject, by, outcome } = context; - // Claiming tasks are started by the gateway, so each is its own root task run and its run ID is its trace ID. - const payload = { stage: status.stage, state: "cancelled", outcome, stoppedBy: by }; - emit({ traceId: status.runId, runId: status.runId, repo, subject, type: "run.finished", payload }); + if (status.state === "running") { + const payload = { stage: status.stage, state: "cancelled", outcome, stoppedBy: by }; + emit({ traceId: rootRunId, runId: status.runId, repo, subject, type: "run.finished", payload }); + } + if (rootRunId !== status.runId) { + emit({ + traceId: rootRunId, + runId: rootRunId, + repo, + subject, + type: "run.finished", + payload: { stage: status.rootStage ?? "workflow", state: "cancelled", outcome, stoppedBy: by, currentRunId: status.runId }, + }); + } } - if (status.sandboxId) { + if (status.state === "running" && status.sandboxId) { const { sandboxId } = status; await runs.terminateSandbox(sandboxId).catch((error) => warn("could not terminate the stopped run's sandbox", error, { sandboxId })); } @@ -118,24 +130,37 @@ export async function takeOver(opts: { subject: Subject; task: string; title: string; + traceId: string; runId: string; + /** A subtask may continue only the root workflow that already owns the subject. */ + mode?: "replace" | "continue"; runs?: RunControl; -}): Promise<{ commentId: number; previousRunId: string | null; cancelledPrevious: boolean }> { - const { gh, repo, subject, task, title, runId } = opts; +}): Promise<{ acquired: boolean; commentId: number | null; previousRunId: string | null; cancelledPrevious: boolean }> { + const { gh, repo, subject, task, title, traceId, runId, mode = "replace" } = opts; const current = await readStatus(gh, subject.number); + const currentTraceId = current ? (current.data.traceId ?? current.data.runId) : null; + if (mode === "continue" && currentTraceId !== traceId) { + return { acquired: false, commentId: current?.comment.id ?? null, previousRunId: current?.data.runId ?? null, cancelledPrevious: false }; + } const previous = current && current.data.runId !== runId ? current.data : null; const stop = { repo, subject: subjectName(subject), by: runId, outcome: "superseded" } as const; - const cancelledPrevious = previous ? await stopRun(opts.runs ?? workflowRuns(), previous, stop) : false; - const body = statusBody({ task, title, runId, state: "running" }); + const previousTraceId = previous ? (previous.traceId ?? previous.runId) : null; + const cancelledPrevious = previous && previousTraceId !== traceId ? await stopRun(opts.runs ?? workflowRuns(), previous, stop) : false; + const rootStage = currentTraceId === traceId ? (current?.data.rootStage ?? current?.data.stage ?? task) : task; + const body = statusBody({ task, title, traceId, rootStage, runId, state: "running" }); const comment = current ? await gh.updateComment(current.comment.id, body) : await gh.createComment(subject.number, body); await addWorkingLabel(gh, subject.number); - return { commentId: comment.id, previousRunId: previous?.runId ?? null, cancelledPrevious }; + return { acquired: true, commentId: comment.id, previousRunId: previous?.runId ?? null, cancelledPrevious }; } export interface StatusUpdate { gh: StatusGitHub; number: number; runId: string; + /** Root Workflows run. Differs from runId when this task is a subtask. */ + traceId?: string; + /** Registered name of the root task. */ + rootStage?: string; /** The task's registered name. */ task: string; /** The task's name in the comment, such as "PR Reviewer". */ @@ -153,23 +178,28 @@ export interface StatusUpdate { */ export async function setStatus(update: StatusUpdate): Promise { const current = await readStatus(update.gh, update.number); - const body = statusBody(update); if (!current) { - await update.gh.createComment(update.number, body); + await update.gh.createComment(update.number, statusBody(update)); return true; } const ours = current.data.runId === update.runId; if (!ours && (current.data.state === "running" || (update.state === "rejected" && current.data.state !== "rejected"))) { return false; } - await update.gh.updateComment(current.comment.id, body); + const currentTraceId = current.data.traceId ?? current.data.runId; + const rootStage = update.rootStage ?? (currentTraceId === update.traceId ? current.data.rootStage : undefined); + await update.gh.updateComment(current.comment.id, statusBody({ ...update, rootStage })); return true; } /** Records a terminal state and removes the working label. */ -export async function finishStatus(update: StatusUpdate & { state: Exclude }): Promise { +export async function finishStatus( + update: StatusUpdate & { state: Exclude }, + opts: { rootFinished?: boolean } = {}, +): Promise { const wrote = await setStatus(update); - if (wrote) await removeWorkingLabel(update.gh, update.number); + const isRoot = !update.traceId || update.traceId === update.runId; + if (wrote && (isRoot || opts.rootFinished)) await removeWorkingLabel(update.gh, update.number); return wrote; } @@ -185,7 +215,16 @@ function statusBody(s: Omit): string { const links = (s.links ?? []).map((link) => `- [${link.label}](${link.url})`); const footer = s.state === "running" ? [] : [`run \`${s.runId}\``]; const sandboxId = s.state === "running" ? s.sandboxId : undefined; - const marker = renderMarker("status", { runId: s.runId, stage: s.task, title: s.title, state: s.state, at: new Date().toISOString(), sandboxId }); + const marker = renderMarker("status", { + runId: s.runId, + traceId: s.traceId, + rootStage: s.rootStage, + stage: s.task, + title: s.title, + state: s.state, + at: new Date().toISOString(), + sandboxId, + }); return [line, ...(links.length ? ["", ...links] : []), ...(footer.length ? ["", ...footer] : []), "", marker].join("\n"); } diff --git a/app/config/index.ts b/app/config/index.ts index 14eb61f..7bef46c 100644 --- a/app/config/index.ts +++ b/app/config/index.ts @@ -47,6 +47,7 @@ export const TASKS = { triage: "triage", specToPr: "spec-to-pr", prReview: "pr-review", + autopilot: "autopilot", cancel: "cancel", refreshSnapshot: "refresh-snapshot", } as const; diff --git a/app/gateway/routes.ts b/app/gateway/routes.ts index 250453d..8092f83 100644 --- a/app/gateway/routes.ts +++ b/app/gateway/routes.ts @@ -1,6 +1,6 @@ import { type ZodType, z } from "zod"; import { config, resolveRepo, TASKS, type TaskName } from "../config/index.js"; -import { CancelInput, PrReviewInput, RefreshSnapshotInput, SpecToPrInput, TriageInput } from "../schemas/index.js"; +import { AutopilotInput, CancelInput, PrReviewInput, RefreshSnapshotInput, SpecToPrInput, TriageInput } from "../schemas/index.js"; export type Route = | { @@ -94,7 +94,15 @@ export function route(event: string, payload: unknown, deliveryId: string): Rout } if (label?.name === config.labels.specApproved) { // GitHub bumps updated_at when the label is applied. - return start(TASKS.specToPr, SpecToPrInput, { issue: issue.number, actor, labeledAt: issue.updated_at }, subject); + return start( + TASKS.specToPr, + SpecToPrInput, + { issue: issue.number, actor, approval: { kind: "label", labeledAt: issue.updated_at } }, + subject, + ); + } + if (label?.name === config.labels.autopilot) { + return start(TASKS.autopilot, AutopilotInput, { issue: issue.number, actor }, subject); } return ignore("unhandled label"); } @@ -121,6 +129,9 @@ export function route(event: string, payload: unknown, deliveryId: string): Rout return start(TASKS.cancel, CancelInput, { number: issue.number, kind: isPr ? "pr" : "issue", actor }, subject); } if (isPr) return ignore("comment on pull request"); + if (issue.labels.some((l) => l.name === config.labels.autopilot)) { + return start(TASKS.autopilot, AutopilotInput, { issue: issue.number, actor, commentId: comment.id }, subject); + } if (!issue.labels.some((l) => l.name === config.labels.triage)) return ignore("issue not in triage"); return start(TASKS.triage, TriageInput, { issue: issue.number, actor, commentId: comment.id }, subject); } diff --git a/app/github/markers.ts b/app/github/markers.ts index 2ec8c3b..5e49497 100644 --- a/app/github/markers.ts +++ b/app/github/markers.ts @@ -12,6 +12,10 @@ export type RunState = (typeof RUN_STATES)[number]; const StatusMarker = z.object({ runId: z.string().min(1), + /** Root Workflows run. Differs from runId when this task is a subtask. */ + traceId: z.string().min(1).optional(), + /** Registered name of the root task, for cancellation telemetry. */ + rootStage: z.string().min(1).optional(), stage: z.string().min(1), /** The task's name as the comment shows it. Comments written before it existed lack it. */ title: z.string().min(1).optional(), diff --git a/app/schemas/index.ts b/app/schemas/index.ts index b93dc97..78c2cce 100644 --- a/app/schemas/index.ts +++ b/app/schemas/index.ts @@ -1,5 +1,5 @@ export { Implementation } from "./implementation.js"; export { Review, ReviewComment, SEVERITIES, Severity } from "./review.js"; export { Spec, SpecDraft, SpecNotes, type SpecValidation, validateSpec } from "./spec.js"; -export { CancelInput, PrReviewInput, RefreshSnapshotInput, SpecToPrInput, TriageInput } from "./tasks.js"; +export { AutopilotInput, CancelInput, PrReviewInput, RefreshSnapshotInput, SpecApproval, SpecToPrInput, TriageInput } from "./tasks.js"; export { TriageDecision } from "./triage.js"; diff --git a/app/schemas/tasks.ts b/app/schemas/tasks.ts index 67b8eda..b17fc42 100644 --- a/app/schemas/tasks.ts +++ b/app/schemas/tasks.ts @@ -15,17 +15,40 @@ export const TriageInput = RepoInput.extend({ commentId: z.number().int().positive().optional(), }); +export const AutopilotInput = RepoInput.extend({ + issue: z.number().int().positive(), + actor: ActorLogin, + /** Set when a human reply restarted an autopilot run that had asked questions. */ + commentId: z.number().int().positive().optional(), +}); + +export const SpecApproval = z.discriminatedUnion("kind", [ + z.object({ + kind: z.literal("label"), + /** When the approval label was applied; the spec must not have changed after it. */ + labeledAt: z.iso.datetime({ offset: true }), + }), + z.object({ + kind: z.literal("autopilot"), + /** The exact App-authored spec returned by the Triage child task. */ + commentId: z.number().int().positive(), + hash: z.string().regex(/^[0-9a-f]{64}$/), + version: z.number().int().positive(), + }), +]); + export const SpecToPrInput = RepoInput.extend({ issue: z.number().int().positive(), actor: ActorLogin, - /** When the approval label was applied (ISO 8601); the spec must not have changed after it. */ - labeledAt: z.iso.datetime({ offset: true }), + approval: SpecApproval, }); export const PrReviewInput = RepoInput.extend({ pr: z.number().int().positive(), headSha: z.string().regex(/^[0-9a-f]{40}$/), actor: ActorLogin, + /** Keeps an autopilot review under the issue-level root claim. */ + workflowIssue: z.number().int().positive().optional(), }); export const CancelInput = RepoInput.extend({ @@ -40,6 +63,8 @@ export const RefreshSnapshotInput = RepoInput.extend({ }); export type TriageInput = z.infer; +export type AutopilotInput = z.infer; +export type SpecApproval = z.infer; export type SpecToPrInput = z.infer; export type PrReviewInput = z.infer; export type CancelInput = z.infer; diff --git a/app/workflows/autopilot/index.ts b/app/workflows/autopilot/index.ts new file mode 100644 index 0000000..f26dd27 --- /dev/null +++ b/app/workflows/autopilot/index.ts @@ -0,0 +1,119 @@ +import type { TaskDefinition } from "@renderinc/sdk/workflows"; +import type { Claim, Outcome, TaskResult } from "../../claims/index.js"; +import { config, TASKS } from "../../config/index.js"; +import { AutopilotInput, type PrReviewInput, type SpecToPrInput, type TriageInput } from "../../schemas/index.js"; +import { prReviewWorkflow } from "../pr-review/index.js"; +import { type SpecToPrHandoff, specToPrWorkflow } from "../spec-to-pr/index.js"; +import { triageWorkflow } from "../triage/index.js"; +import type { TriageHandoff } from "../triage/spec.js"; +import { defineGitHubTask, type GitHubTaskContext, type GitHubTaskDeps } from "../github-task.js"; + +const TIMEOUT_SECONDS = 8 * 60 * 60; + +type StageName = "triage" | "spec-to-pr" | "pr-review"; +type StageTaskResult = TaskResult; + +export interface AutopilotStage { + task: StageName; + runId: string; + state: TaskResult["state"]; + outcome: string; + summary: string; +} + +export interface AutopilotData { + stages: AutopilotStage[]; +} + +export interface AutopilotDeps extends GitHubTaskDeps { + triage?: (input: TriageInput) => Promise>; + specToPr?: (input: SpecToPrInput) => Promise>; + prReview?: (input: PrReviewInput) => Promise; +} + +/** + * Runs the three built-in GitHub tasks as children of one issue-level root. + * Child results, not GitHub labels, move the workflow to its next stage. + */ +async function autopilot(claim: Claim, input: TriageInput, deps: AutopilotDeps, ctx: GitHubTaskContext): Promise> { + const gh = claim.gh; + const stages: AutopilotStage[] = []; + const record = (task: StageName, result: TaskResult): TaskResult => { + stages.push({ task, runId: result.runId, state: result.state, outcome: result.outcome, summary: result.summary }); + return result; + }; + const finish = (outcome: string, summary: string, state?: "done" | "failed"): Outcome => ({ + outcome, + summary, + state, + data: { stages }, + }); + const stop = (task: StageName, result: StageTaskResult): Outcome => { + const state = result.state === "done" ? "done" : "failed"; + return finish(`stopped_at_${task.replaceAll("-", "_")}`, result.summary, state); + }; + const child = ( + override: ((input: Input) => Promise>) | undefined, + workflow: { task: TaskDefinition<[raw: unknown], TaskResult> }, + ): ((input: Input) => Promise>) => { + if (override) return override; + const run = ctx.run; + if (!run) throw new Error("autopilot requires a Workflows task context"); + return (stageInput) => run(workflow.task, stageInput); + }; + + const runTriage = child(deps.triage, triageWorkflow); + const runSpecToPr = child(deps.specToPr, specToPrWorkflow); + const runPrReview = child(deps.prReview, prReviewWorkflow); + + const base = { repo: input.repo, installationId: input.installationId, actor: input.actor }; + // Trigger labels are consumed when a run starts. Autopilot adds its label + // back only while it is waiting for a human answer. + await gh.removeLabel(input.issue, config.labels.autopilot); + await claim.progress("running Triage"); + const triage = record("triage", await runTriage({ ...base, issue: input.issue, commentId: input.commentId })); + if (triage.state === "done" && triage.outcome === "asked") { + await claim.checkpoint(); + await gh.addLabels(input.issue, [config.labels.autopilot]); + return finish("waiting_for_answers", triage.summary); + } + if (triage.state !== "done" || triage.data?.kind !== "spec") return stop("triage", triage); + + await claim.progress("implementing the spec"); + const specToPr = record( + "spec-to-pr", + await runSpecToPr({ + ...base, + issue: input.issue, + approval: { + kind: "autopilot", + commentId: triage.data.commentId, + hash: triage.data.hash, + version: triage.data.version, + }, + }), + ); + if (specToPr.state !== "done" || specToPr.data?.kind !== "pull_request") return stop("spec-to-pr", specToPr); + + await claim.progress("reviewing the draft pull request"); + const review = record( + "pr-review", + await runPrReview({ ...base, pr: specToPr.data.pr, headSha: specToPr.data.headSha, workflowIssue: input.issue }), + ); + if (review.state !== "done" || !["reviewed", "already_reviewed"].includes(review.outcome)) return stop("pr-review", review); + + return finish("completed", `opened and reviewed draft PR #${specToPr.data.pr}.`); +} + +export const autopilotWorkflow = defineGitHubTask( + { + name: TASKS.autopilot, + title: "Autopilot", + input: AutopilotInput, + subject: (input) => ({ kind: "issue", number: input.issue }), + timeoutSeconds: TIMEOUT_SECONDS, + }, + autopilot, +); + +export const runAutopilot = autopilotWorkflow.run; diff --git a/app/workflows/cancel.ts b/app/workflows/cancel.ts index f0b9eb2..a3f8751 100644 --- a/app/workflows/cancel.ts +++ b/app/workflows/cancel.ts @@ -10,7 +10,10 @@ type CancelGitHub = StatusGitHub & Pick; interface CancelResult { cancelled: boolean; + /** Root workflow run that was cancelled. */ runId?: string; + /** Child task that was active when the root was cancelled. */ + currentRunId?: string; reason?: "permission_denied" | "nothing_running" | "already_stopped"; } @@ -45,10 +48,29 @@ export async function cancelRun( await gh.createComment(input.number, `@${input.actor} no factory run is in progress here.`); return { cancelled: false, reason: "nothing_running" }; } - const { runId: stopped, stage, title } = current.data; + const { runId: currentRunId, traceId: stoppedTrace, rootStage, stage, title } = current.data; + const rootRunId = stoppedTrace ?? currentRunId; + const childWasRunning = current.data.state === "running"; + const displayedRunId = childWasRunning ? currentRunId : rootRunId; + const displayedStage = childWasRunning ? stage : (rootStage ?? stage); + const displayedTitle = childWasRunning ? (title ?? stage) : (rootStage ?? title ?? stage); const summary = active ? `cancelled by @${input.actor}.` : `cleared by @${input.actor}; the run had already stopped without finishing.`; - await finishStatus({ gh, number: input.number, runId: stopped, task: stage, title: title ?? stage, state: "cancelled", summary }); - return active ? { cancelled: true, runId: stopped } : { cancelled: false, runId: stopped, reason: "already_stopped" }; + await finishStatus( + { + gh, + number: input.number, + runId: displayedRunId, + traceId: rootRunId, + rootStage, + task: displayedStage, + title: displayedTitle, + state: "cancelled", + summary, + }, + { rootFinished: true }, + ); + const child = childWasRunning && rootRunId !== currentRunId ? { currentRunId } : {}; + return active ? { cancelled: true, runId: rootRunId, ...child } : { cancelled: false, runId: rootRunId, ...child, reason: "already_stopped" }; })(); emit({ ...event, type: "run.finished", payload: { stage: TASKS.cancel, state: "done", ...result } }); diff --git a/app/workflows/claiming-task.ts b/app/workflows/claiming-task.ts deleted file mode 100644 index 8ebcf95..0000000 --- a/app/workflows/claiming-task.ts +++ /dev/null @@ -1,48 +0,0 @@ -import { type TaskContext, task } from "@renderinc/sdk/workflows"; -import type { z } from "zod"; -import { type Claim, type Outcome, type RunControl, type Subject, type TaskResult, withClaim } from "../claims/index.js"; -import { requireRepo } from "../config/index.js"; -import type { GitHubClient } from "../github/index.js"; - -/** Test seams every claiming task takes; each task adds its own. */ -export interface ClaimDeps { - gh?: GitHubClient; - runs?: RunControl; -} - -export interface ClaimingTaskSpec { - /** The task's name in Render Workflows, from TASKS; the gateway starts it by this name. */ - name: string; - /** How the status comment names the task, such as "PR Reviewer". */ - title: string; - /** Validates the task input the gateway sent. */ - input: z.ZodType; - /** The issue or PR the task claims. */ - subject(input: Input): Subject; - timeoutSeconds: number; -} - -/** The handler a claiming task registers, with injectable dependencies for tests. */ -export type ClaimingTaskHandler = (ctx: Pick, raw: unknown, deps?: Deps) => Promise; - -/** - * Registers a Workflows task that validates its input, resolves the repository, and runs `work` under a claim on - * the input's issue or PR (see withClaim). Returns the handler, so tests can call it with fakes. - */ -export function claimingTask( - spec: ClaimingTaskSpec, - work: (claim: Claim, input: Input, deps: Deps) => Promise, -): ClaimingTaskHandler { - const handler: ClaimingTaskHandler = (ctx, raw, deps = {} as Deps) => { - const input = spec.input.parse(raw); - const repo = requireRepo(input.repo, input.installationId); - const opts = { repo, subject: spec.subject(input), task: spec.name, title: spec.title, actor: input.actor, gh: deps.gh, runs: deps.runs }; - return withClaim(ctx, opts, (claim) => work(claim, input, deps)); - }; - // Never retried: a retry repeats model spend, and a relabel or a new comment starts a fresh run. - task( - { name: spec.name, timeoutSeconds: spec.timeoutSeconds, retry: { maxRetries: 0, waitDurationMs: 5_000, backoffScaling: 1 } }, - (ctx, raw: unknown) => handler(ctx, raw), - ); - return handler; -} diff --git a/app/workflows/github-task.ts b/app/workflows/github-task.ts new file mode 100644 index 0000000..6b83f84 --- /dev/null +++ b/app/workflows/github-task.ts @@ -0,0 +1,74 @@ +import { type TaskContext, type TaskDefinition, task as registerTask } from "@renderinc/sdk/workflows"; +import type { z } from "zod"; +import { type Claim, type Outcome, type RunControl, type Subject, type TaskResult, withClaim } from "../claims/index.js"; +import { requireRepo } from "../config/index.js"; +import type { GitHubClient } from "../github/index.js"; + +/** Test seams shared by tasks attached to a GitHub issue or pull request. */ +export interface GitHubTaskDeps { + gh?: GitHubClient; + runs?: RunControl; +} + +export interface GitHubTaskSpec { + /** The name registered in Render Workflows. */ + name: string; + /** The task name shown in the GitHub status comment. */ + title: string; + /** Validates input before any work starts. */ + input: z.ZodType; + /** Selects the issue or pull request this run owns. */ + subject(input: Input, ctx: GitHubTaskContext): Subject; + timeoutSeconds: number; +} + +export type GitHubTaskContext = Pick & Partial>; + +export type GitHubTaskRunner = (ctx: GitHubTaskContext, raw: unknown, deps?: Deps) => Promise>; + +/** + * A GitHub task has one production definition and one directly callable test seam. + * + * `task` runs as a Render Workflows task or subtask. `run` executes the same body + * in-process with injectable dependencies. + */ +export interface GitHubTask { + readonly task: TaskDefinition<[raw: unknown], TaskResult>; + readonly run: GitHubTaskRunner; +} + +/** + * Defines work attached to one GitHub issue or pull request. Every run validates + * input, resolves the configured repository, checks the actor's permission, and + * owns the subject's status comment until it finishes. + */ +export function defineGitHubTask( + spec: GitHubTaskSpec, + work: (claim: Claim, input: Input, deps: Deps, ctx: GitHubTaskContext) => Promise>, +): GitHubTask { + const run: GitHubTaskRunner = (ctx, raw, deps = {} as Deps) => { + const input = spec.input.parse(raw); + const repo = requireRepo(input.repo, input.installationId); + return withClaim( + ctx, + { + repo, + subject: spec.subject(input, ctx), + task: spec.name, + title: spec.title, + actor: input.actor, + mode: ctx.metadata.parentTaskRunId ? "continue" : "replace", + gh: deps.gh, + runs: deps.runs, + }, + (claim) => work(claim, input, deps, ctx), + ); + }; + // Replaying a run can repeat model spend and durable writes. A new GitHub + // trigger starts a new run instead. + const task = registerTask( + { name: spec.name, timeoutSeconds: spec.timeoutSeconds, retry: { maxRetries: 0, waitDurationMs: 5_000, backoffScaling: 1 } }, + (ctx, raw: unknown) => run(ctx, raw), + ); + return { run, task }; +} diff --git a/app/workflows/index.ts b/app/workflows/index.ts index e06875b..07ebf3e 100644 --- a/app/workflows/index.ts +++ b/app/workflows/index.ts @@ -1,4 +1,5 @@ // Importing a task module registers its tasks with the Render Workflows SDK. +import "./autopilot/index.js"; import "./cancel.js"; import "./pr-review/index.js"; import "./refresh-snapshot.js"; diff --git a/app/workflows/pr-review/index.ts b/app/workflows/pr-review/index.ts index 0163966..00ea25c 100644 --- a/app/workflows/pr-review/index.ts +++ b/app/workflows/pr-review/index.ts @@ -3,7 +3,7 @@ import type { Outcome } from "../../claims/index.js"; import { TASKS } from "../../config/index.js"; import { openSandbox } from "../../sandbox/index.js"; import { PrReviewInput } from "../../schemas/index.js"; -import { type ClaimDeps, claimingTask } from "../claiming-task.js"; +import { defineGitHubTask, type GitHubTaskDeps } from "../github-task.js"; import { runVerify } from "../run-verify.js"; import { headMoved, mergeBase, type OpenSandbox, withPullSandbox } from "./checkout.js"; import { commentable, pullDiffLines } from "./diff-lines.js"; @@ -11,7 +11,7 @@ import { reviewerMessage } from "./messages.js"; import { findReview, postReview } from "./post.js"; /** Test seams: everything that reaches GitHub, Sandboxes, or a model. */ -export interface PrReviewDeps extends ClaimDeps { +export interface PrReviewDeps extends GitHubTaskDeps { openSandbox?: OpenSandbox; runAgent?: typeof runAgent; } @@ -20,12 +20,15 @@ export interface PrReviewDeps extends ClaimDeps { * PR Reviewer: verify the PR head, review it in one sandbox, and post one COMMENT review. * It never approves or requests changes. */ -export const runPrReview = claimingTask( +export const prReviewWorkflow = defineGitHubTask( { name: TASKS.prReview, title: "PR Reviewer", input: PrReviewInput, - subject: (input) => ({ kind: "pr", number: input.pr }), + subject: (input, ctx) => { + if (input.workflowIssue && !ctx.metadata.parentTaskRunId) throw new Error("workflowIssue is valid only in a parent workflow"); + return input.workflowIssue ? { kind: "issue", number: input.workflowIssue } : { kind: "pr", number: input.pr }; + }, timeoutSeconds: 2 * 60 * 60, }, async (claim, input, deps: PrReviewDeps): Promise => { @@ -81,3 +84,5 @@ export const runPrReview = claimingTask( return { outcome: "reviewed", summary: `posted a COMMENT review: ${notes.join(", ")}.`, links: [{ label: "Review", url: posted.htmlUrl }] }; }, ); + +export const runPrReview = prReviewWorkflow.run; diff --git a/app/workflows/spec-to-pr/approval.ts b/app/workflows/spec-to-pr/approval.ts index cbc4fef..a50b8b0 100644 --- a/app/workflows/spec-to-pr/approval.ts +++ b/app/workflows/spec-to-pr/approval.ts @@ -1,5 +1,5 @@ import { config } from "../../config/index.js"; -import { Spec } from "../../schemas/index.js"; +import { Spec, type SpecApproval } from "../../schemas/index.js"; import { findMarkedComment, type GitHubClient, type IssueComment } from "../../github/index.js"; import { ExpectedFailure } from "../../claims/index.js"; @@ -11,10 +11,10 @@ interface ApprovedSpec { } /** - * The spec as it stood when the approval label was applied (invariant 9). The approval time is the newest - * `labeled` event for the label, or the time the webhook reported when the timeline doesn't have it yet. + * Loads the exact approved spec. Manual runs use the approval label's timestamp; + * autopilot names the App-authored spec comment and marker it received from Triage. */ -export async function loadApprovedSpec(gh: ApprovalGitHub, issue: number, labeledAt: string): Promise { +export async function loadApprovedSpec(gh: ApprovalGitHub, issue: number, approval: SpecApproval): Promise { const label = config.labels.specApproved; const found = findMarkedComment(await gh.listIssueComments(issue), "spec", await gh.botLogin()); if (!found) { @@ -24,10 +24,14 @@ export async function loadApprovedSpec(gh: ApprovalGitHub, issue: number, labele ); } - const events = await gh.getLabelEvents(issue, label); - const approvedAt = events.at(-1)?.createdAt ?? labeledAt; - if (Date.parse(found.comment.updatedAt) > Date.parse(approvedAt)) { - throw new ExpectedFailure(`the spec changed after it was approved. Review it and reapply the \`${label}\` label.`, "spec_changed"); + if (approval.kind === "label") { + const events = await gh.getLabelEvents(issue, label); + const approvedAt = events.at(-1)?.createdAt ?? approval.labeledAt; + if (Date.parse(found.comment.updatedAt) > Date.parse(approvedAt)) { + throw new ExpectedFailure(`the spec changed after it was approved. Review it and reapply the \`${label}\` label.`, "spec_changed"); + } + } else if (found.comment.id !== approval.commentId || found.data.hash !== approval.hash || found.data.version !== approval.version) { + throw new ExpectedFailure("the spec changed after Triage returned it to autopilot.", "spec_changed"); } const parsed = Spec.safeParse(found.data.spec); diff --git a/app/workflows/spec-to-pr/index.ts b/app/workflows/spec-to-pr/index.ts index 67f548b..fd65b2c 100644 --- a/app/workflows/spec-to-pr/index.ts +++ b/app/workflows/spec-to-pr/index.ts @@ -6,7 +6,7 @@ import { pullRequestBody, pullRequestTitle } from "../../format/index.js"; import { type GitDataClient, gitData } from "../../github/index.js"; import { type OpenedSandbox, openSandbox, type SandboxRef, withSandbox } from "../../sandbox/index.js"; import { type Implementation, SpecToPrInput } from "../../schemas/index.js"; -import { type ClaimDeps, claimingTask } from "../claiming-task.js"; +import { defineGitHubTask, type GitHubTaskDeps } from "../github-task.js"; import { runVerify, type VerifyRun } from "../run-verify.js"; import { loadApprovedSpec } from "./approval.js"; import { botIdentity, branchName, commitDate, commitMessage, ownedCommits, refusalSummary } from "./branch.js"; @@ -15,8 +15,15 @@ import { allPassed, failureSummary, hasChanges } from "./verify.js"; const IMPLEMENT_TIMEOUT_MS = 60 * 60 * 1000; +/** The draft pull request a parent task may pass to its next stage. */ +export interface SpecToPrHandoff { + kind: "pull_request"; + pr: number; + headSha: string; +} + /** Test seams; each defaults to the real thing. */ -export interface SpecToPrDeps extends ClaimDeps { +export interface SpecToPrDeps extends GitHubTaskDeps { gitData?: GitDataClient; openSandbox?: (repo: RepoContext, ref: SandboxRef) => Promise; runAgent?: typeof runAgent; @@ -28,7 +35,7 @@ export interface SpecToPrDeps extends ClaimDeps { * Approved spec to draft PR: implement in one sandbox, verify with up to `limits.repairRounds` repair runs, then * publish through the Git Data API and open a draft PR. A rerun (a relabel) replaces the earlier attempt's branch. */ -export const runSpecToPr = claimingTask( +export const specToPrWorkflow = defineGitHubTask( { name: TASKS.specToPr, title: "Spec-to-PR", @@ -36,11 +43,11 @@ export const runSpecToPr = claimingTask( subject: (input) => ({ kind: "issue", number: input.issue }), timeoutSeconds: 4 * 60 * 60, }, - async (claim, input, deps: SpecToPrDeps): Promise => { + async (claim, input, deps: SpecToPrDeps): Promise> => { const { gh, repo } = claim; // Fixed for the whole run, so publishing the same tree twice produces the same commit. const date = commitDate((deps.now ?? (() => new Date()))()); - const { spec, comment } = await loadApprovedSpec(gh, input.issue, input.labeledAt); + const { spec, comment } = await loadApprovedSpec(gh, input.issue, input.approval); const issue = await gh.getIssue(input.issue); if (issue.isPull) throw new ExpectedFailure("Spec-to-PR runs on issues, not pull requests.", "not_an_issue"); const author = await botIdentity(gh, date); @@ -103,12 +110,15 @@ export const runSpecToPr = claimingTask( outcome: "pr_opened", summary: `opened draft PR #${pull.number}. Review it before marking it ready.`, links: [{ label: `Draft PR #${pull.number}`, url: pull.htmlUrl }], + data: { kind: "pull_request", pr: pull.number, headSha: pull.headSha }, }; }); }, ); -const noChanges = (): Outcome => ({ +export const runSpecToPr = specToPrWorkflow.run; + +const noChanges = (): Outcome => ({ outcome: "no_changes", summary: "the implementation agent made no changes, so there is nothing to open a PR for.", }); diff --git a/app/workflows/triage/index.ts b/app/workflows/triage/index.ts index ee6478e..3138658 100644 --- a/app/workflows/triage/index.ts +++ b/app/workflows/triage/index.ts @@ -5,13 +5,13 @@ import { triageDeclineBody, triageQuestionsBody } from "../../format/index.js"; import { renderMarker } from "../../github/index.js"; import { openSandbox, type Sandbox, type SandboxRef, withSandbox } from "../../sandbox/index.js"; import { TriageInput } from "../../schemas/index.js"; -import { type ClaimDeps, claimingTask } from "../claiming-task.js"; +import { defineGitHubTask, type GitHubTaskDeps } from "../github-task.js"; import { type AgentRunner, triageAgents } from "./agents.js"; -import { draftSpec, publishSpec } from "./spec.js"; +import { draftSpec, publishSpec, type TriageHandoff } from "./spec.js"; import { readThread, specWriterMessage, triageMessage } from "./thread.js"; /** Test seams; production uses the real GitHub client, sandboxes, and runAgent. */ -export interface TriageDeps extends ClaimDeps { +export interface TriageDeps extends GitHubTaskDeps { openSandbox?: (repo: RepoContext, ref: SandboxRef) => Promise; runAgent?: AgentRunner; } @@ -20,9 +20,9 @@ export interface TriageDeps extends ClaimDeps { * Triage one issue: the triage agent reads the issue and the repository and decides to ask questions, draft * (or revise) the spec, or decline. Every GitHub write goes through the format helpers and carries a marker. */ -export const runTriage = claimingTask( +export const triageWorkflow = defineGitHubTask( { name: TASKS.triage, title: "Triage", input: TriageInput, subject: (input) => ({ kind: "issue", number: input.issue }), timeoutSeconds: 3600 }, - async (claim, input, deps: TriageDeps): Promise => { + async (claim, input, deps: TriageDeps): Promise> => { const { gh, repo } = claim; const [issue, comments, bot] = await Promise.all([gh.getIssue(input.issue), gh.listIssueComments(input.issue), gh.botLogin()]); if (input.commentId !== undefined) { @@ -63,3 +63,5 @@ export const runTriage = claimingTask( }); }, ); + +export const runTriage = triageWorkflow.run; diff --git a/app/workflows/triage/spec.ts b/app/workflows/triage/spec.ts index faba6a5..e4a74e6 100644 --- a/app/workflows/triage/spec.ts +++ b/app/workflows/triage/spec.ts @@ -7,6 +7,14 @@ import type { Sandbox } from "../../sandbox/index.js"; import type { TriageAgents } from "./agents.js"; import { type CurrentSpec, retryMessage } from "./thread.js"; +/** The spec a parent task may pass to its next stage. */ +export interface TriageHandoff { + kind: "spec"; + commentId: number; + hash: string; + version: number; +} + /** Drafts the spec, and redrafts once if validation rejects it. */ export async function draftSpec(agents: TriageAgents, sandbox: Sandbox, message: string): Promise<{ draft: SpecDraft; warnings: string[] }> { const existing = await trackedPaths(sandbox); @@ -38,13 +46,14 @@ export async function publishSpec( issue: Issue, current: CurrentSpec | null, { draft, warnings }: { draft: SpecDraft; warnings: string[] }, -): Promise { +): Promise> { const hash = contentHash(draft.spec); if (current && current.hash === hash) { return { outcome: "unchanged", summary: `the spec is unchanged (v${current.version}).`, links: [specLink(current.version, current.comment.htmlUrl)], + data: { kind: "spec", commentId: current.comment.id, hash, version: current.version }, }; } const version = (current?.version ?? 0) + 1; @@ -60,6 +69,7 @@ export async function publishSpec( outcome: current ? "revised" : "drafted", summary: `${current ? "revised the spec to" : "drafted spec"} v${version}${note}.`, links: [specLink(version, comment.htmlUrl)], + data: { kind: "spec", commentId: comment.id, hash, version }, }; } diff --git a/docs/architecture.md b/docs/architecture.md index 6b8fcfe..2dab38f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -53,6 +53,21 @@ PR Reviewer checks out the labeled pull request commit, runs the configured verification commands, and posts one COMMENT review. The review covers security, correctness, performance, and tests. +### Autopilot + +Applying `factory:autopilot` grants upfront approval for Triage, Spec-to-PR, and PR Reviewer. +The parent starts each GitHub task with `ctx.run`, so every stage has its own compute, claim, +status, and Sandbox. + +Autopilot does not apply the manual transition labels. Triage can still remove +a stale `factory:spec-approved` label when it revises a spec. Triage returns +the exact App-authored spec comment and content hash to Spec-to-PR, which +verifies that the current spec still matches. Spec-to-PR returns the pull +request number and head commit directly to PR Reviewer. If Triage asks +questions, the parent adds `factory:autopilot` back before it ends; a human +reply starts a new parent run. At other times the trigger label is absent, so +unrelated comments do not restart the workflow. + ## State Each system owns the state it already manages: @@ -80,12 +95,16 @@ at three boundaries: 3. Hidden markers identify App-authored specifications, pull requests, and reviews so writes can update or reuse them. -A new claiming run cancels the run named in the status comment, terminates +For a composed workflow, the status marker records both the root workflow run +and the current child task. Cancellation stops the root; the child ID still +identifies the current stage and Sandbox. + +A new GitHub task run cancels the run named in the status comment, terminates its recorded Sandbox, and writes its own run ID to the comment. The run whose write remains visible holds the claim. Other runs stop when they next check the comment. -Claiming tasks check ownership before progress updates and durable writes. +GitHub tasks check ownership before progress updates and durable writes. Publishing also uses a deterministic branch and refuses to overwrite human commits. @@ -95,7 +114,7 @@ database would create a second owner for the same claim. ## Sandboxes and snapshots A base-branch push builds a repository snapshot with the runner, a base -checkout, and installed dependencies. A claiming task then: +checkout, and installed dependencies. A GitHub task then: 1. restores the newest snapshot; 2. verifies runtime permissions and process hardening; @@ -106,7 +125,7 @@ checkout, and installed dependencies. A claiming task then: 7. closes the Sandbox in `finally`. Render terminates a cancelled task without running `finally`. For that case, -each claiming task records its Sandbox ID in the status comment. A replacement +each GitHub task records its Sandbox ID in the status comment. A replacement run or `/factory cancel` terminates the recorded Sandbox. The provider timeout is the final cleanup backstop. @@ -178,7 +197,7 @@ Changes must preserve these properties: 1. Verify a webhook signature before parsing its body. 2. Check the triggering actor's current repository permission for every - claiming run. + GitHub task run. 3. Let only code in the Workflows service write to GitHub. 4. Keep GitHub write credentials out of Sandboxes. 5. Run every model-directed tool inside a Sandbox. @@ -187,7 +206,7 @@ Changes must preserve these properties: 8. Stop automation at a draft pull request and a COMMENT review. 9. Implement the specification as it existed when approved. 10. Give every Sandbox a provider timeout and `finally` cleanup. Record a - claiming task's Sandbox so cancellation can terminate it. + GitHub task's Sandbox so cancellation can terminate it. Tests pin these invariants and the tool authority of every built-in agent. @@ -223,16 +242,18 @@ Start with: 1. [`app/gateway/routes.ts`](../app/gateway/routes.ts) — webhook-to-task routing. -2. [`app/workflows/claiming-task.ts`](../app/workflows/claiming-task.ts) — - claiming task declaration. +2. [`app/workflows/github-task.ts`](../app/workflows/github-task.ts) — + GitHub task declaration. 3. [`app/claims/claim.ts`](../app/claims/claim.ts) — permission, ownership, progress, and completion. 4. [`app/workflows/triage/index.ts`](../app/workflows/triage/index.ts) — one - complete claiming task. -5. [`app/agents/run.ts`](../app/agents/run.ts) — host-side agent execution. -6. [`app/runner/main.ts`](../app/runner/main.ts) — Sandbox runner entry point. -7. [`app/sandbox/sandbox.ts`](../app/sandbox/sandbox.ts) — Sandbox lifecycle. -8. [`app/changeset/index.ts`](../app/changeset/index.ts) — validated + complete GitHub task. +5. [`app/workflows/autopilot/index.ts`](../app/workflows/autopilot/index.ts) — + parent/subtask orchestration. +6. [`app/agents/run.ts`](../app/agents/run.ts) — host-side agent execution. +7. [`app/runner/main.ts`](../app/runner/main.ts) — Sandbox runner entry point. +8. [`app/sandbox/sandbox.ts`](../app/sandbox/sandbox.ts) — Sandbox lifecycle. +9. [`app/changeset/index.ts`](../app/changeset/index.ts) — validated publishing. Use the [extension runbook](runbooks/extend.md) to add agents, tasks, or diff --git a/docs/configuration.md b/docs/configuration.md index 87dac2e..f212f7f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -27,6 +27,7 @@ export default defineFactoryConfig({ triage: "factory:triage", specApproved: "factory:spec-approved", prReview: "factory:pr-review", + autopilot: "factory:autopilot", }, commandPrefix: "/factory", minimumPermission: "write", @@ -75,6 +76,7 @@ After adding a repository: | `labels.triage` | `factory:triage` | Starts or continues Triage | | `labels.specApproved` | `factory:spec-approved` | Starts Spec-to-PR | | `labels.prReview` | `factory:pr-review` | Starts PR Reviewer | +| `labels.autopilot` | `factory:autopilot` | Runs all three stages through a reviewed draft PR | | `commandPrefix` | `/factory` | Prefix for commands such as `/factory cancel` | | `minimumPermission` | `write` | Live repository permission required to start work | | `models.medium` | `anthropic/claude-sonnet-5` | Triage and PR Reviewer model | @@ -86,9 +88,9 @@ After adding a repository: `minimumPermission` accepts `triage`, `write`, `maintain`, or `admin`. The factory checks the triggering actor's current permission at the start of -every claiming task. +every GitHub task. -`setup:gibthub` also creates `factory:working`. This label records an active +`setup:github` also creates `factory:working`. This label records an active run; it is not a trigger. If you rename a trigger label, deploy both services and rerun @@ -165,7 +167,7 @@ Organization installations use the equivalent organization settings URL. ## Follow and control runs -While a claiming task runs, the issue or pull request has `factory:working`. +While a GitHub task runs, the issue or pull request has `factory:working`. The App-authored status comment shows progress, the run ID, and the final result. Its hidden marker records the current Sandbox for cancellation. @@ -198,6 +200,14 @@ the status comment and terminates its recorded Sandbox. It also clears a stale To restart work, remove the trigger label and apply it again. +`factory:autopilot` grants upfront approval to draft a spec, implement it, +open a draft pull request, and post a COMMENT review. It never approves or +merges the pull request, and it does not apply the three manual transition +labels. Triage may still clear a stale spec-approval label when it revises the +spec. The trigger label is removed when a run starts. If Triage asks questions, +the factory adds it back so a human reply can start the continuation. Use +`/factory cancel` to stop an active root workflow. + ## Check a deployment `doctor` is read-only, does not print secret values, and exits nonzero when a diff --git a/docs/runbooks/extend.md b/docs/runbooks/extend.md index 4169308..c46667a 100644 --- a/docs/runbooks/extend.md +++ b/docs/runbooks/extend.md @@ -122,11 +122,11 @@ list only if write tools are intentional. Add task-level tests for the user-visible result and meaningful failures. Use fakes; tests must not call GitHub, Render, or a model provider. -## Add a claiming task +## Add a GitHub task -Use a claiming task for work on one issue or pull request. The status comment -records the run that owns the subject and the Sandbox that cancellation must -terminate. +Use `defineGitHubTask()` for work attached to one issue or pull request. It +keeps run ownership, permission checks, status updates, and cancellation out +of the task body. ### 1. Define the task contract @@ -143,7 +143,7 @@ must pin. Do not accept these values from agent output. Create `app/workflows//index.ts`: ```typescript -export const runExample = claimingTask( +export const exampleWorkflow = defineGitHubTask( { name: TASKS.example, title: "Example", @@ -156,13 +156,18 @@ export const runExample = claimingTask( return { outcome: "completed", summary: "completed the example task." }; }, ); + +export const runExample = exampleWorkflow.run; ``` -`claimingTask()` validates input, resolves the configured repository, checks -the actor's permission, acquires the status-comment claim, and records the -final outcome. +`defineGitHubTask()` returns two explicit entry points: + +- `exampleWorkflow.task` is the registered Render task definition. Pass it to + `ctx.run()` from a parent workflow. +- `exampleWorkflow.run` calls the same body in-process with injectable + dependencies for tests. -Claiming tasks do not retry automatically. A retry can repeat model spend or +GitHub tasks do not retry automatically. A retry can repeat model spend or durable writes; a person starts a fresh run instead. ### 3. Use the claim correctly @@ -188,6 +193,43 @@ when the module loads. Keep task inputs and results small and JSON-serializable. +## Compose tasks + +Use a GitHub-task parent to run registered child tasks in order: + +```typescript +export const pipeline = defineGitHubTask( + { + name: "pipeline", + title: "Pipeline", + input: PipelineInput, + subject: input => ({ kind: "issue", number: input.issue }), + timeoutSeconds: 3600, + }, + async (_claim, input, _deps, ctx) => { + if (!ctx.run) throw new Error("pipeline requires a Workflows context"); + const first = await ctx.run(firstWorkflow.task, input); + if (first.state !== "done" || !first.data) { + return { state: "failed", outcome: "stopped", summary: first.summary }; + } + const second = await ctx.run(secondWorkflow.task, first.data); + return { outcome: "completed", summary: second.summary }; + }, +); +``` + +Each `ctx.run()` starts a child task on separate compute and waits for its +typed result. The parent owns the subject's root claim; its children continue +that claim, update the current stage and Sandbox, and run their own permission +checks. + +Keep parent retries disabled. Retrying a parent can create new child runs and +repeat model spend. Return immediately after a child is cancelled, rejected, +or fails. + +[`app/workflows/autopilot/index.ts`](../../app/workflows/autopilot/index.ts) +is the complete example. + ## Add a plain Workflows task Use the Render SDK's `task()` directly only when the work has no issue or pull @@ -246,7 +288,7 @@ Before handing off, confirm: - model output cannot select an external write target; - the gateway still imports no GitHub auth, Sandbox, agent, or workflow code; - `app/runner/` still imports nothing from the rest of `app/`; -- claiming tasks record their Sandbox before running an agent; +- GitHub tasks record their Sandbox before running an agent; - tests cover the user-visible success and important failures; and - documentation and setup requirements changed when the App contract changed. diff --git a/docs/runbooks/setup.md b/docs/runbooks/setup.md index 2fa38b3..11c4e93 100644 --- a/docs/runbooks/setup.md +++ b/docs/runbooks/setup.md @@ -213,7 +213,7 @@ The command: `https://.onrender.com/webhooks/github`; - preserves the generated webhook secret; and - creates `factory:triage`, `factory:spec-approved`, - `factory:pr-review`, and `factory:working`. + `factory:pr-review`, `factory:autopilot`, and `factory:working`. It is safe to rerun. Continue only when it reports no failed repository or webhook operation. diff --git a/factory.config.ts b/factory.config.ts index 9035017..5594ecc 100644 --- a/factory.config.ts +++ b/factory.config.ts @@ -12,7 +12,7 @@ export interface RepoConfig { export interface FactoryConfig { /** Repositories the factory acts on, keyed by "owner/name". Webhooks from any other repository are ignored. */ repos: Record; - labels: { triage: string; specApproved: string; prReview: string }; + labels: { triage: string; specApproved: string; prReview: string; autopilot: string }; /** Prefix for comment commands, such as `/factory cancel`. */ commandPrefix: string; /** GitHub permission the triggering actor needs on the repository. */ @@ -49,6 +49,7 @@ export default defineFactoryConfig({ triage: "factory:triage", specApproved: "factory:spec-approved", prReview: "factory:pr-review", + autopilot: "factory:autopilot", }, commandPrefix: "/factory", minimumPermission: "write", diff --git a/scripts/lib/requirements.ts b/scripts/lib/requirements.ts index ea2607d..401f8f2 100644 --- a/scripts/lib/requirements.ts +++ b/scripts/lib/requirements.ts @@ -49,6 +49,12 @@ export const LABELS: readonly LabelSpec[] = [ description: "software-factory: review this pull request", trigger: true, }, + { + name: config.labels.autopilot, + color: "c5def5", + description: "software-factory: run through a reviewed draft pull request", + trigger: true, + }, { name: WORKING_LABEL, color: "fbca04", description: "software-factory is working on this", trigger: false }, ]; diff --git a/tests/autopilot.test.ts b/tests/autopilot.test.ts new file mode 100644 index 0000000..9be28c2 --- /dev/null +++ b/tests/autopilot.test.ts @@ -0,0 +1,269 @@ +import type { TaskContext } from "@renderinc/sdk/workflows"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import type { TaskResult } from "../app/claims/index.js"; +import type { FactoryEvent } from "../app/events/index.js"; +import type { GitHubClient, IssueComment } from "../app/github/index.js"; +import { renderMarker } from "../app/github/index.js"; +import type { SpecToPrHandoff } from "../app/workflows/spec-to-pr/index.js"; +import type { TriageHandoff } from "../app/workflows/triage/spec.js"; + +const events: FactoryEvent[] = []; +vi.mock("../app/events/index.js", () => ({ + emit: (event: FactoryEvent) => { + events.push(event); + }, +})); + +const { config } = await import("../app/config/index.js"); +const { runAutopilot } = await import("../app/workflows/autopilot/index.js"); +const { triageWorkflow } = await import("../app/workflows/triage/index.js"); +const { specToPrWorkflow } = await import("../app/workflows/spec-to-pr/index.js"); +const { prReviewWorkflow } = await import("../app/workflows/pr-review/index.js"); + +const REPO = Object.keys(config.repos)[0]; +if (!REPO) throw new Error("factory.config.ts must configure at least one repository"); +const SPEC_HASH = "b".repeat(64); + +const input = { repo: REPO, installationId: 5, issue: 12, actor: "maintainer" }; +const ctx = { + metadata: { taskRunId: "trn-autopilot", rootTaskRunId: "trn-autopilot" }, + run: vi.fn(() => { + throw new Error("test did not inject a stage"); + }), +} as unknown as Pick; + +const result = (runId: string, outcome: string, overrides: Partial> = {}): TaskResult => ({ + state: "done", + outcome, + summary: outcome, + traceId: "trn-autopilot", + runId, + ...overrides, +}); + +class FakeGitHub { + readonly labels = new Map>([[12, new Set([config.labels.autopilot])]]); + readonly comments: IssueComment[] = []; + private nextId = 1; + + botLogin = async () => "software-factory[bot]"; + listIssueComments = async (_number: number) => this.comments.map((comment) => ({ ...comment })); + createComment = async (number: number, body: string) => { + const comment: IssueComment = { + id: this.nextId++, + body, + author: { login: "software-factory[bot]", type: "Bot" }, + createdAt: "2026-10-25T11:00:00Z", + updatedAt: "2026-10-25T11:00:00Z", + htmlUrl: `https://github.test/issues/${number}#issuecomment-${this.nextId}`, + }; + this.comments.push(comment); + return { ...comment }; + }; + updateComment = async (id: number, body: string) => { + const comment = this.comments.find((candidate) => candidate.id === id); + if (!comment) throw new Error(`missing comment ${id}`); + comment.body = body; + return { ...comment }; + }; + ensureLabel = async () => {}; + addLabels = async (number: number, labels: readonly string[]) => { + const current = this.labels.get(number) ?? new Set(); + this.labels.set(number, current); + for (const label of labels) current.add(label); + }; + getIssue = async (number: number) => ({ + number, + title: "Autopilot issue", + body: "Build the feature.", + state: "open", + labels: [...(this.labels.get(number) ?? [])], + author: { login: "reporter", type: "User" }, + isPull: false, + htmlUrl: `https://github.test/issues/${number}`, + }); + getPermission = async () => "write" as const; + removeLabel = async (number: number, label: string) => { + this.labels.get(number)?.delete(label); + }; + + replaceOwner(traceId: string) { + const comment = this.comments[0]; + if (!comment) throw new Error("status comment does not exist"); + comment.body = renderMarker("status", { runId: `${traceId}-child`, traceId, stage: "triage", title: "Triage", state: "running" }); + } +} + +const asClient = (gh: FakeGitHub) => gh as unknown as GitHubClient; + +beforeEach(() => { + events.length = 0; +}); + +describe("autopilot", () => { + it("runs the registered GitHub task definitions as Workflows children", async () => { + const gh = new FakeGitHub(); + const names: string[] = []; + const workflowCtx = { + metadata: ctx.metadata, + run: vi.fn(async (definition: { name: string }, stageInput: unknown) => { + names.push(definition.name); + if (definition.name === "triage") { + return result("trn-triage", "drafted", { + data: { kind: "spec", commentId: 100, hash: SPEC_HASH, version: 1 }, + }); + } + if (definition.name === "spec-to-pr") { + expect(stageInput).toMatchObject({ + approval: { kind: "autopilot", commentId: 100, hash: SPEC_HASH, version: 1 }, + }); + return result("trn-spec", "pr_opened", { + data: { kind: "pull_request", pr: 34, headSha: "a".repeat(40) }, + }); + } + expect(stageInput).toMatchObject({ pr: 34, workflowIssue: 12 }); + return result("trn-review", "reviewed"); + }), + } as unknown as Pick; + + const output = await runAutopilot(workflowCtx, input, { gh: asClient(gh) }); + expect(names).toEqual(["triage", "spec-to-pr", "pr-review"]); + expect(output.outcome).toBe("completed"); + }); + + it("runs Triage, Spec-to-PR, and PR Reviewer in order", async () => { + const gh = new FakeGitHub(); + const order: string[] = []; + const output = await runAutopilot(ctx, input, { + gh: asClient(gh), + triage: async () => { + order.push("triage"); + return result("trn-triage", "drafted", { + data: { kind: "spec", commentId: 100, hash: SPEC_HASH, version: 1 }, + }); + }, + specToPr: async (stageInput) => { + order.push("spec-to-pr"); + expect(stageInput.approval).toEqual({ kind: "autopilot", commentId: 100, hash: SPEC_HASH, version: 1 }); + return result("trn-spec", "pr_opened", { + data: { kind: "pull_request", pr: 34, headSha: "a".repeat(40) }, + }); + }, + prReview: async (stageInput) => { + order.push("pr-review"); + expect(stageInput).toMatchObject({ pr: 34, headSha: "a".repeat(40), workflowIssue: 12 }); + return result("trn-review", "reviewed"); + }, + }); + + expect(order).toEqual(["triage", "spec-to-pr", "pr-review"]); + expect(output).toMatchObject({ state: "done", outcome: "completed", summary: "opened and reviewed draft PR #34." }); + expect(output.data?.stages.map((stage) => [stage.task, stage.runId])).toEqual([ + ["triage", "trn-triage"], + ["spec-to-pr", "trn-spec"], + ["pr-review", "trn-review"], + ]); + expect(gh.labels.get(12)).not.toContain(config.labels.autopilot); + expect(events).toContainEqual( + expect.objectContaining({ type: "run.finished", payload: expect.objectContaining({ stage: "autopilot", outcome: "completed" }) }), + ); + }); + + it("keeps the autopilot label while Triage waits for answers", async () => { + const gh = new FakeGitHub(); + const specToPr = vi.fn(); + const output = await runAutopilot( + ctx, + { ...input, commentId: 99 }, + { + gh: asClient(gh), + triage: async (stageInput) => { + expect(stageInput.commentId).toBe(99); + return result("trn-triage", "asked", { summary: "asked one question." }); + }, + specToPr, + }, + ); + + expect(output).toMatchObject({ state: "done", outcome: "waiting_for_answers" }); + expect(gh.labels.get(12)).toContain(config.labels.autopilot); + expect(specToPr).not.toHaveBeenCalled(); + }); + + it("stops after a terminal stage result and removes the autopilot label", async () => { + const gh = new FakeGitHub(); + const prReview = vi.fn(); + const output = await runAutopilot(ctx, input, { + gh: asClient(gh), + triage: async () => + result("trn-triage", "unchanged", { + data: { kind: "spec", commentId: 100, hash: SPEC_HASH, version: 2 }, + }), + specToPr: async () => result("trn-spec", "verification_failed", { state: "failed", summary: "verification failed." }), + prReview, + }); + + expect(output).toMatchObject({ state: "failed", outcome: "stopped_at_spec_to_pr", summary: "verification failed." }); + expect(gh.labels.get(12)).not.toContain(config.labels.autopilot); + expect(prReview).not.toHaveBeenCalled(); + }); + + it("does not start another stage after the current one is superseded", async () => { + const gh = new FakeGitHub(); + const specToPr = vi.fn(); + const output = await runAutopilot(ctx, input, { + gh: asClient(gh), + triage: async () => { + gh.replaceOwner("trn-new-autopilot"); + return result("trn-triage", "superseded", { state: "cancelled", summary: "a newer run took over." }); + }, + specToPr, + }); + + expect(output).toMatchObject({ state: "cancelled", outcome: "superseded" }); + expect(gh.labels.get(12)).not.toContain(config.labels.autopilot); + expect(specToPr).not.toHaveBeenCalled(); + }); + + it("stops when a child rejects the actor's permission", async () => { + const gh = new FakeGitHub(); + const prReview = vi.fn(); + const output = await runAutopilot(ctx, input, { + gh: asClient(gh), + triage: async () => + result("trn-triage", "drafted", { + data: { kind: "spec", commentId: 100, hash: SPEC_HASH, version: 1 }, + }), + specToPr: async () => result("trn-spec", "permission_denied", { state: "rejected" }), + prReview, + }); + + expect(output).toMatchObject({ state: "failed", outcome: "stopped_at_spec_to_pr" }); + expect(gh.labels.get(12)).not.toContain(config.labels.autopilot); + expect(gh.labels.get(12)).not.toContain(config.labels.specApproved); + expect(prReview).not.toHaveBeenCalled(); + }); + + it("removes the autopilot label and rethrows an unexpected stage error", async () => { + const gh = new FakeGitHub(); + await expect( + runAutopilot(ctx, input, { + gh: asClient(gh), + triage: async () => { + throw new Error("task transport failed"); + }, + }), + ).rejects.toThrow("task transport failed"); + expect(gh.labels.get(12)).not.toContain(config.labels.autopilot); + expect(events).toContainEqual(expect.objectContaining({ type: "run.finished", payload: expect.objectContaining({ outcome: "error" }) })); + }); +}); + +describe("GitHub task authoring", () => { + it("exposes a direct runner and a registered subtask definition", () => { + expect(triageWorkflow.task.name).toBe("triage"); + expect(specToPrWorkflow.task.name).toBe("spec-to-pr"); + expect(prReviewWorkflow.task.name).toBe("pr-review"); + expect(typeof triageWorkflow.run).toBe("function"); + }); +}); diff --git a/tests/cancel.test.ts b/tests/cancel.test.ts index 8d0cb62..b21902c 100644 --- a/tests/cancel.test.ts +++ b/tests/cancel.test.ts @@ -72,12 +72,20 @@ beforeEach(() => { describe("cancel task", () => { it("cancels the active run recorded in the status comment, terminates its sandbox, and says who cancelled it", async () => { const { gh, comments, labels, add } = fakeGitHub("write"); - const marker = renderMarker("status", { runId: "trn-1", stage: "spec-to-pr", title: "Spec-to-PR", state: "running", sandboxId: "sbx-1" }); + const marker = renderMarker("status", { + runId: "trn-1", + traceId: "trn-root", + rootStage: "autopilot", + stage: "spec-to-pr", + title: "Spec-to-PR", + state: "running", + sandboxId: "sbx-1", + }); add(BOT, `running\n\n${marker}`); - const runs = fakeRuns({ "trn-1": "running" }); + const runs = fakeRuns({ "trn-root": "running" }); const result = await cancelRun(ctx, { repo: REPO, installationId: 5, number: 12, kind: "issue", actor: "alice" }, { gh, runs }); - expect(result).toEqual({ cancelled: true, runId: "trn-1" }); - expect(runs.cancelTaskRun).toHaveBeenCalledWith("trn-1"); + expect(result).toEqual({ cancelled: true, runId: "trn-root", currentRunId: "trn-1" }); + expect(runs.cancelTaskRun).toHaveBeenCalledWith("trn-root"); expect(runs.terminateSandbox).toHaveBeenCalledWith("sbx-1"); expect(comments).toHaveLength(1); expect(comments[0]?.body).toContain("**Spec-to-PR** was cancelled: cancelled by @alice."); @@ -85,10 +93,17 @@ describe("cancel task", () => { expect(labels.has(WORKING_LABEL)).toBe(false); expect(events.map((e) => [e.type, e.runId])).toEqual([ ["run.finished", "trn-1"], + ["run.finished", "trn-root"], ["run.finished", "trn-cancel"], ]); expect(events[0]?.payload).toMatchObject({ state: "cancelled", outcome: "cancelled", stoppedBy: "trn-cancel" }); - expect(events[1]).toMatchObject({ traceId: "trn-cancel", runId: "trn-cancel", repo: REPO }); + expect(events[0]).toMatchObject({ traceId: "trn-root", runId: "trn-1" }); + expect(events[1]).toMatchObject({ + traceId: "trn-root", + runId: "trn-root", + payload: { stage: "autopilot", state: "cancelled", outcome: "cancelled", stoppedBy: "trn-cancel", currentRunId: "trn-1" }, + }); + expect(events[2]).toMatchObject({ traceId: "trn-cancel", runId: "trn-cancel", repo: REPO }); }); it("clears a status left running by a run that already stopped, and terminates its leftover sandbox", async () => { @@ -105,6 +120,32 @@ describe("cancel task", () => { expect(labels.has(WORKING_LABEL)).toBe(false); }); + it("cancels the root without reclassifying a child that already finished", async () => { + const { gh, comments, labels, add } = fakeGitHub("write"); + const marker = renderMarker("status", { + runId: "trn-child", + traceId: "trn-root", + rootStage: "autopilot", + stage: "triage", + title: "Triage", + state: "done", + }); + add(BOT, `done\n\n${marker}`); + const runs = fakeRuns({ "trn-root": "running" }); + const result = await cancelRun(ctx, { repo: REPO, installationId: 5, number: 12, kind: "issue", actor: "alice" }, { gh, runs }); + + expect(result).toEqual({ cancelled: true, runId: "trn-root" }); + expect(runs.cancelTaskRun).toHaveBeenCalledWith("trn-root"); + expect(parseMarker(comments[0]?.body ?? "", "status")).toMatchObject({ + runId: "trn-root", + traceId: "trn-root", + stage: "autopilot", + state: "cancelled", + }); + expect(labels.has(WORKING_LABEL)).toBe(false); + expect(events.map((event) => event.runId)).toEqual(["trn-root", "trn-cancel"]); + }); + it("refuses an actor without permission and leaves the run alone", async () => { const { gh, comments, add } = fakeGitHub("read"); add(BOT, renderMarker("status", { runId: "trn-1", stage: "triage", state: "running" })); diff --git a/tests/claims.test.ts b/tests/claims.test.ts index 86296b7..346f670 100644 --- a/tests/claims.test.ts +++ b/tests/claims.test.ts @@ -58,8 +58,14 @@ class FakeGitHub { return { ...comment }; } - addStatus(runId: string, state: "running" | "done" | "rejected" = "running", sandboxId?: string): IssueComment { - return this.add(BOT, `status\n\n${renderMarker("status", { runId, stage: "triage", state, sandboxId })}`); + addStatus( + runId: string, + state: "running" | "done" | "rejected" = "running", + sandboxId?: string, + traceId?: string, + rootStage?: string, + ): IssueComment { + return this.add(BOT, `status\n\n${renderMarker("status", { runId, traceId, rootStage, stage: "triage", state, sandboxId })}`); } asClient(): GitHubClient { @@ -96,7 +102,16 @@ describe("takeOver", () => { const gh = new FakeGitHub(); const existing = gh.addStatus("trn-old", "running", "sbx-old"); const runs = new FakeRuns({ "trn-old": "running" }); - const result = await takeOver({ gh, repo: REPO.fullName, subject, task: "triage", title: "Triage", runId: "trn-new", runs }); + const result = await takeOver({ + gh, + repo: REPO.fullName, + subject, + task: "triage", + title: "Triage", + traceId: "trn-new", + runId: "trn-new", + runs, + }); expect(runs.cancelled).toEqual(["trn-old"]); expect(runs.terminated).toEqual(["sbx-old"]); // The stopped run is killed before it can record how it ended, so the takeover records it. @@ -108,12 +123,79 @@ describe("takeOver", () => { type: "run.finished", payload: { stage: "triage", state: "cancelled", outcome: "superseded", stoppedBy: "trn-new" }, }); - expect(result).toEqual({ commentId: existing.id, previousRunId: "trn-old", cancelledPrevious: true }); + expect(result).toEqual({ acquired: true, commentId: existing.id, previousRunId: "trn-old", cancelledPrevious: true }); expect(gh.comments).toHaveLength(1); expect(parseMarker(gh.comments[0]?.body ?? "", "status")).toMatchObject({ runId: "trn-new", state: "running", stage: "triage" }); expect(gh.comments[0]?.body).toContain("is running (run `trn-new`)"); expect(gh.labels.has(WORKING_LABEL)).toBe(true); }); + + it("cancels the previous root when the recorded task was its child", async () => { + const gh = new FakeGitHub(); + gh.addStatus("trn-child-old", "running", "sbx-old", "trn-root-old"); + const runs = new FakeRuns({ "trn-root-old": "running" }); + await takeOver({ + gh, + repo: REPO.fullName, + subject, + task: "triage", + title: "Triage", + traceId: "trn-root-new", + runId: "trn-child-new", + runs, + }); + + expect(runs.cancelled).toEqual(["trn-root-old"]); + expect(events).toContainEqual( + expect.objectContaining({ traceId: "trn-root-old", runId: "trn-child-old", payload: expect.objectContaining({ outcome: "superseded" }) }), + ); + }); + + it("moves to another child of the same root without cancelling the parent", async () => { + const gh = new FakeGitHub(); + gh.addStatus("trn-triage", "done", undefined, "trn-root", "autopilot"); + const runs = new FakeRuns({ "trn-root": "running" }); + await takeOver({ + gh, + repo: REPO.fullName, + subject, + task: "spec-to-pr", + title: "Spec-to-PR", + traceId: "trn-root", + runId: "trn-spec", + runs, + }); + + expect(runs.getTaskRun).not.toHaveBeenCalled(); + expect(runs.cancelled).toEqual([]); + expect(parseMarker(gh.comments[0]?.body ?? "", "status")).toMatchObject({ + traceId: "trn-root", + rootStage: "autopilot", + runId: "trn-spec", + stage: "spec-to-pr", + }); + }); + + it("refuses a late child instead of taking over from a newer root", async () => { + const gh = new FakeGitHub(); + const existing = gh.addStatus("trn-new-child", "running", undefined, "trn-new-root"); + const runs = new FakeRuns({ "trn-new-root": "running" }); + const result = await takeOver({ + gh, + repo: REPO.fullName, + subject, + task: "spec-to-pr", + title: "Spec-to-PR", + traceId: "trn-old-root", + runId: "trn-late-child", + mode: "continue", + runs, + }); + + expect(result).toEqual({ acquired: false, commentId: existing.id, previousRunId: "trn-new-child", cancelledPrevious: false }); + expect(runs.getTaskRun).not.toHaveBeenCalled(); + expect(parseMarker(gh.comments[0]?.body ?? "", "status")).toMatchObject({ traceId: "trn-new-root", runId: "trn-new-child" }); + }); }); describe("setStatus and finishStatus", () => { it("never overwrites a newer run's status", async () => { @@ -124,6 +206,36 @@ describe("setStatus and finishStatus", () => { expect(parseMarker(gh.comments[0]?.body ?? "", "status")?.runId).toBe("trn-new"); expect(gh.labels.has(WORKING_LABEL)).toBe(true); }); + + it("keeps the working label between child stages and removes it when the root finishes", async () => { + const gh = new FakeGitHub(); + gh.addStatus("trn-child", "running", undefined, "trn-root"); + gh.labels.add(WORKING_LABEL); + + await finishStatus({ + gh, + number: 12, + traceId: "trn-root", + runId: "trn-child", + task: "triage", + title: "Triage", + state: "done", + summary: "done", + }); + expect(gh.labels.has(WORKING_LABEL)).toBe(true); + + await finishStatus({ + gh, + number: 12, + traceId: "trn-root", + runId: "trn-root", + task: "autopilot", + title: "Autopilot", + state: "done", + summary: "complete", + }); + expect(gh.labels.has(WORKING_LABEL)).toBe(false); + }); }); describe("withClaim", () => { diff --git a/tests/pr-review.test.ts b/tests/pr-review.test.ts index 7dccfc8..70708c4 100644 --- a/tests/pr-review.test.ts +++ b/tests/pr-review.test.ts @@ -33,6 +33,7 @@ if (!BUILD_COMMAND || !TEST_COMMAND) throw new Error("test fixture requires two const HEAD = "a".repeat(40); const MERGE_BASE = "b".repeat(40); const ctx = { metadata: { taskRunId: "trn-run", rootTaskRunId: "trn-root" } }; +const childCtx = { metadata: { ...ctx.metadata, parentTaskRunId: "trn-parent" } }; const input = { repo: REPO, installationId: 5, @@ -64,6 +65,7 @@ class FakeGitHub { comments: ReviewComment[]; }> = []; createReviewErrors: Error[] = []; + statusNumbers: number[] = []; pull: PullRequest = { number: 34, state: "open", @@ -81,7 +83,8 @@ class FakeGitHub { botLogin = async () => BOT; getPermission = async (login: string) => (login === "alice" ? "write" : "none"); listIssueComments = async () => this.comments.map((c) => ({ ...c })); - createComment = async (_n: number, body: string) => { + createComment = async (number: number, body: string) => { + this.statusNumbers.push(number); const comment = { id: this.nextId++, body, @@ -381,6 +384,29 @@ describe("runPrReview", () => { expect(gh.status()).toContain("pullrequestreview-9"); }); + it("keeps an autopilot review's status and sandbox on the workflow issue", async () => { + await gh.createComment( + 12, + renderMarker("status", { + runId: "trn-root", + traceId: "trn-root", + rootStage: "autopilot", + stage: "autopilot", + title: "Autopilot", + state: "running", + }), + ); + const result = await runPrReview(childCtx, { ...input, workflowIssue: 12 }, deps()); + expect(result).toMatchObject({ state: "done", outcome: "reviewed" }); + expect(gh.statusNumbers).toEqual([12]); + }); + + it("rejects workflowIssue on a directly started PR Reviewer task", async () => { + expect(() => runPrReview(ctx, { ...input, workflowIssue: 12 }, deps())).toThrow("valid only in a parent workflow"); + expect(gh.comments).toEqual([]); + expect(opened).toEqual([]); + }); + it("fails cleanly when the reviewer stops", async () => { agents.failures["pr-reviewer"] = new AgentError("pr-reviewer", "timeout", "the runner timed out after reading /secret/path"); const result = await runPrReview(ctx, input, deps()); diff --git a/tests/routes.test.ts b/tests/routes.test.ts index c861b80..4e03102 100644 --- a/tests/routes.test.ts +++ b/tests/routes.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; import { config } from "../app/config/index.js"; -import { PrReviewInput, SpecToPrInput, TriageInput } from "../app/schemas/tasks.js"; +import { AutopilotInput, PrReviewInput, SpecToPrInput, TriageInput } from "../app/schemas/tasks.js"; import { route } from "../app/gateway/routes.js"; const REPO = Object.keys(config.repos)[0] as string; @@ -47,17 +47,50 @@ describe("route: starts", () => { if (r.kind === "start") expect(TriageInput.safeParse(r.input).success).toBe(true); }); - it("spec-approved label starts spec-to-pr with labeledAt from updated_at", () => { + it("spec-approved label starts spec-to-pr with a label approval", () => { const r = route("issues", issueLabeled(config.labels.specApproved), DELIVERY); expect(r).toMatchObject({ kind: "start", task: "spec-to-pr", - input: { repo: REPO, installationId: 7, issue: 12, actor: "maintainer", labeledAt: "2026-10-18T10:00:00Z" }, + input: { + repo: REPO, + installationId: 7, + issue: 12, + actor: "maintainer", + approval: { kind: "label", labeledAt: "2026-10-18T10:00:00Z" }, + }, idempotencyKey: DELIVERY, }); if (r.kind === "start") expect(SpecToPrInput.safeParse(r.input).success).toBe(true); }); + it("autopilot label on an issue starts autopilot", () => { + const r = route("issues", issueLabeled(config.labels.autopilot), DELIVERY); + expect(r).toEqual({ + kind: "start", + task: "autopilot", + input: { repo: REPO, installationId: 7, issue: 12, actor: "maintainer" }, + idempotencyKey: DELIVERY, + subject: "issue#12", + }); + if (r.kind === "start") expect(AutopilotInput.safeParse(r.input).success).toBe(true); + }); + + it("a human reply continues autopilot while its label remains", () => { + const r = route("issue_comment", comment("Here are the details.", { labels: [{ name: config.labels.autopilot }] }), DELIVERY); + expect(r).toMatchObject({ + kind: "start", + task: "autopilot", + input: { repo: REPO, installationId: 7, issue: 12, actor: "maintainer", commentId: 99 }, + subject: "issue#12", + }); + }); + + it("does not restart autopilot for comments after its trigger label was consumed", () => { + const r = route("issue_comment", comment("Unrelated note.", { labels: [] }), DELIVERY); + expect(r).toEqual({ kind: "ignore", reason: "issue not in triage" }); + }); + it("pr-review label on an open PR starts pr-review", () => { const r = route("pull_request", prLabeled(config.labels.prReview), DELIVERY); expect(r).toEqual({ diff --git a/tests/schemas.test.ts b/tests/schemas.test.ts index 4860a58..d1e542d 100644 --- a/tests/schemas.test.ts +++ b/tests/schemas.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { Review, Spec, SpecDraft, TriageDecision, validateSpec } from "../app/schemas/index.js"; +import { AutopilotInput, Review, Spec, SpecDraft, SpecToPrInput, TriageDecision, validateSpec } from "../app/schemas/index.js"; const spec = (overrides: Partial = {}): Spec => ({ title: "Add a dark mode toggle", @@ -49,6 +49,25 @@ describe("TriageDecision", () => { }); }); +describe("task inputs", () => { + it("accepts an autopilot start and an optional human reply", () => { + const base = { repo: "acme/web", installationId: 7, issue: 12, actor: "maintainer" }; + expect(AutopilotInput.safeParse(base).success).toBe(true); + expect(AutopilotInput.safeParse({ ...base, commentId: 99 }).success).toBe(true); + }); + + it("distinguishes manual and autopilot spec approval", () => { + const base = { repo: "acme/web", installationId: 7, issue: 12, actor: "maintainer" }; + expect(SpecToPrInput.safeParse({ ...base, approval: { kind: "label", labeledAt: "2026-10-25T12:00:00Z" } }).success).toBe(true); + expect( + SpecToPrInput.safeParse({ + ...base, + approval: { kind: "autopilot", commentId: 100, hash: "a".repeat(64), version: 1 }, + }).success, + ).toBe(true); + }); +}); + describe("Spec", () => { it("accepts a full spec and a draft with or without a change summary", () => { expect(Spec.parse(spec())).toEqual(spec()); diff --git a/tests/scripts-doctor.test.ts b/tests/scripts-doctor.test.ts index d8f58d1..7d3a6ae 100644 --- a/tests/scripts-doctor.test.ts +++ b/tests/scripts-doctor.test.ts @@ -34,8 +34,8 @@ const APP = { permissions: { contents: "write", issues: "write", pull_requests: "write", metadata: "read" }, events: ["issues", "issue_comment", "pull_request", "push"], }; -const LABEL_NAMES = ["factory:triage", "factory:spec-approved", "factory:pr-review", "factory:working"]; -const TASK_NAMES = ["triage", "spec-to-pr", "pr-review", "cancel", "refresh-snapshot"]; +const LABEL_NAMES = ["factory:triage", "factory:spec-approved", "factory:pr-review", "factory:autopilot", "factory:working"]; +const TASK_NAMES = ["triage", "spec-to-pr", "pr-review", "autopilot", "cancel", "refresh-snapshot"]; interface Call { method: string; diff --git a/tests/spec-to-pr.test.ts b/tests/spec-to-pr.test.ts index 4c425f5..e020b43 100644 --- a/tests/spec-to-pr.test.ts +++ b/tests/spec-to-pr.test.ts @@ -20,6 +20,7 @@ type GitDataClient = import("../app/github/index.js").GitDataClient; type GitCommit = import("../app/github/index.js").GitCommit; type ExecOptions = import("../app/sandbox/index.js").ExecOptions; type ExecResult = import("../app/sandbox/index.js").ExecResult; +type SpecToPrInput = import("../app/schemas/index.js").SpecToPrInput; type SpecToPrDeps = import("../app/workflows/spec-to-pr/index.js").SpecToPrDeps; type PublishResult = import("../app/changeset/index.js").PublishResult; @@ -30,12 +31,19 @@ if (!REPO) throw new Error("factory.config.ts must configure at least one reposi const [BUILD_COMMAND, TEST_COMMAND] = config.repos[REPO].verify; if (!BUILD_COMMAND || !TEST_COMMAND) throw new Error("test fixture requires two verify commands"); const BASE = "b".repeat(40); +const PR_HEAD = "c".repeat(40); const ISSUE = 12; const BRANCH = `factory/issue-${ISSUE}`; const ctx = { metadata: { taskRunId: "trn-spec" } }; const NOW = new Date("2026-10-18T23:37:05.123Z"); const LABELED_AT = "2026-10-18T20:00:00Z"; -const input = { repo: REPO, installationId: 5, issue: ISSUE, actor: "alice", labeledAt: LABELED_AT }; +const input = { + repo: REPO, + installationId: 5, + issue: ISSUE, + actor: "alice", + approval: { kind: "label" as const, labeledAt: LABELED_AT }, +}; const SPEC: Spec = { title: "Add a health endpoint", @@ -105,7 +113,7 @@ function fakeGitHub(opts: { spec?: unknown; specUpdatedAt?: string; labelEvents? getLabelEvents: vi.fn(async (_n: number, label: string) => (opts.labelEvents ?? []).map((createdAt) => ({ label, createdAt, actor: null }))), http: { json: vi.fn(async (_method: string, path: string) => (path === `/users/${encodeURIComponent(BOT)}` ? { id: BOT_ID } : undefined)) }, findOrCreatePull: vi.fn(async (_args: { head: string; base: string; title: string; body: string; draft?: boolean }) => ({ - pull: { number: 34, htmlUrl: `https://github.com/${REPO}/pull/34` }, + pull: { number: 34, headSha: PR_HEAD, htmlUrl: `https://github.com/${REPO}/pull/34` }, created: true, })), }; @@ -175,6 +183,7 @@ function setup( agents?: Parameters[0]; branch?: Parameters[0]; publish?: PublishResult | Error; + input?: SpecToPrInput; } = {}, ) { const github = fakeGitHub(opts.gh); @@ -195,7 +204,7 @@ function setup( publish: publish as unknown as SpecToPrDeps["publish"], now: () => NOW, }; - const run = () => runSpecToPr(ctx, input, deps); + const run = () => runSpecToPr(ctx, opts.input ?? input, deps); return { ...github, ...box, ...agents, git, openSandbox, publish, run }; } @@ -222,6 +231,30 @@ describe("approval", () => { expect(t.status().body).toContain("the spec changed after it was approved"); expect(t.openSandbox).not.toHaveBeenCalled(); }); + + it("accepts the exact spec handed off by an autopilot Triage child", async () => { + const t = setup({ + input: { + ...input, + approval: { kind: "autopilot", commentId: 1, hash: contentHash(SPEC), version: 1 }, + }, + }); + const outcome = await t.run(); + expect(outcome).toMatchObject({ state: "done", outcome: "pr_opened" }); + expect(t.gh.getLabelEvents).not.toHaveBeenCalled(); + }); + + it("refuses an autopilot handoff when the current spec does not match it", async () => { + const t = setup({ + input: { + ...input, + approval: { kind: "autopilot", commentId: 1, hash: "f".repeat(64), version: 1 }, + }, + }); + const outcome = await t.run(); + expect(outcome).toMatchObject({ state: "failed", outcome: "spec_changed" }); + expect(t.openSandbox).not.toHaveBeenCalled(); + }); }); describe("implement, verify, and publish", () => { @@ -229,7 +262,11 @@ describe("implement, verify, and publish", () => { const t = setup(); const outcome = await t.run(); - expect(outcome).toMatchObject({ state: "done", outcome: "pr_opened" }); + expect(outcome).toMatchObject({ + state: "done", + outcome: "pr_opened", + data: { kind: "pull_request", pr: 34, headSha: PR_HEAD }, + }); expect(t.openSandbox).toHaveBeenCalledWith(expect.objectContaining({ fullName: REPO }), { kind: "branch", name: "main" }); expect(t.calls.map((c) => c.agent)).toEqual(["implementer"]); const [impl] = t.calls; diff --git a/tests/triage.test.ts b/tests/triage.test.ts index edf0cec..f0a5162 100644 --- a/tests/triage.test.ts +++ b/tests/triage.test.ts @@ -266,7 +266,12 @@ describe("triage task", () => { const outcome = await runTriage(ctx, input, deps); - expect(outcome).toMatchObject({ state: "done", outcome: "drafted", summary: "drafted spec v1." }); + expect(outcome).toMatchObject({ + state: "done", + outcome: "drafted", + summary: "drafted spec v1.", + data: { kind: "spec", commentId: expect.any(Number), hash: expect.any(String), version: 1 }, + }); expect(calls.map((c) => c.agent)).toEqual(["triage", "spec-writer"]); expect(calls[1]).toMatchObject({ sandbox }); expect(calls[1]?.message).toContain(`in ${REPO}`);