diff --git a/AGENTS.md b/AGENTS.md index a630611..2fcf00b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,144 +1,103 @@ # Repository guide for agents -This file applies to the whole repository. Read -[docs/architecture.md](docs/architecture.md) for how the system works and -[docs/design-notes.md](docs/design-notes.md) for why it is built this way. - -## Task routing - -- **Set up or deploy the factory:** follow - [docs/runbooks/setup.md](docs/runbooks/setup.md) in order. Do not skip its - human checkpoints or claim completion before both deployed `doctor` checks - and the Triage smoke test pass. -- **Change repositories, labels, models, or limits:** use - [docs/configuration.md](docs/configuration.md). -- **Diagnose a live deployment:** use - [docs/operations.md](docs/operations.md). -- **Add an agent, claiming task, or trigger:** use - [CONTRIBUTING.md](CONTRIBUTING.md). - -## What this is - -software-factory turns a labeled GitHub issue into a spec, a draft PR, and a COMMENT review. Three claiming tasks (Triage, Spec-to-PR, PR Reviewer) run on Render Workflows, each holding a claim on its issue or PR while it works. Agents run in Render Sandboxes restored from a per-repository snapshot. The status comment on the issue or PR names the run that holds the claim. There is no database: GitHub and Workflows hold all state, and run telemetry is logged as JSON lines. - -Two processes deploy: - -- `app/entrypoints/gateway.ts`: the gateway web service. Verifies webhooks, routes each to one task, and calls `startTask` with the delivery ID as the idempotency key. -- `app/entrypoints/workflows.ts`: the Workflows service. Importing `app/workflows/index.ts` registers every task. - - -## Module map - -```text -factory.config.ts Repositories (base branch, install, verify), labels, permission, model tiers, publish and repair limits, sandbox timeout -app/ - entrypoints/ Gateway and Workflows process entry points - config/ Loads factory.config.ts; RepoContext and resolveRepo; TASKS names; modelFor; requireEnv - schemas/ Zod schemas: task inputs (tasks.ts) and agent outputs (triage, spec, implementation, review) - gateway/ Hono app: signature check, body cap, the routing table (routes.ts), which validates task inputs - workflows/ The tasks: triage/, spec-to-pr/, pr-review/, cancel.ts, refresh-snapshot.ts; claiming-task.ts and run-verify.ts are shared - claims/ withClaim lifecycle: permission, the status-comment claim, takeover and sandbox cleanup, checkpoints, expected failures - agents/ One directory with index.ts and prompt.md per agent; shared defineAgent/runAgent runtime - runner/ The agent runner: one Pi agent loop, tools, read-only shell policy, submit. Bundled into the snapshot - sandbox/ Sandbox API client with pacing and retries; snapshot build, openSandbox, withSandbox, verification - changeset/ Collects, validates, and publishes sandbox changes through the Git Data API - github/ REST client, App JWT and installation tokens, Git Data API, hidden comment markers - format/ Renders specs, questions, PR bodies, and reviews as GitHub markdown; sanitizes model text - events/ emit(): run.finished and agent.finished as JSON lines on stdout -sandbox/ Shell scripts copied into the snapshot (run-agent, as-runner, verify, collect-changes, ...) -scripts/ setup-app, setup-github, doctor -tests/ Vitest unit tests; fakes in tests/helpers/ -``` - -Dependencies point one way, from entry points down to leaves: - -```text -entrypoints → gateway, workflows -workflows → claims, changeset, agents, sandbox, github, format, events, schemas, config -claims → agents, sandbox, github, events, config -agents → sandbox, runner, events, schemas, config -changeset → sandbox, github, config -sandbox → runner (to bundle it), github (read tokens), config -format → schemas, config -gateway → schemas, events, config -github → config -runner, events, schemas, config → nothing in app/ -``` - -Two edges must never appear: - -- **Nothing in `runner/` imports from the rest of `app/`.** esbuild bundles `runner/main.ts` into a self-contained file that runs in the sandbox with only Node. -- **`gateway/` never imports `github/`, `sandbox/`, `agents/`, or `workflows/`.** The gateway holds no GitHub credential and runs no model. - -Import another module through its `index.ts` barrel where one exists. A barrel exports what other modules use; keep module-internal helpers out of it. - -## Conventions - -- Strict TypeScript, ESM, NodeNext. Relative imports end in `.js`. No path aliases. -- Biome is configured for tabs and a line width of 150 (`biome.json`); CI runs `biome lint`, which must pass with no warnings. -- Validate every external or cross-process input with Zod at the boundary: webhook payloads, task inputs, agent output, markers, sandbox output, environment. -- Task inputs and results stay small and JSON-serializable. Each task opens - one sandbox with `withSandbox`, which closes it within the same task. -- Throw `ExpectedFailure` for an expected failure whose message is safe to show on GitHub. `withClaim` also treats an agent - that stops on its own (turn limit, timeout, no or invalid submission) as expected; every other error fails the task. - Outcome names are snake_case. -- Quote every shell argument you build with `shq` (`app/sandbox/shell.ts`). Anything long or security-sensitive that runs in a sandbox belongs in a script in `sandbox/`, which CI checks with shellcheck. -- Model text that reaches GitHub goes through the helpers in `app/format/`. -- Task implementations take GitHub, sandboxes, `runAgent`, and run control as injectable dependencies. Tests never call live Render, GitHub, or model APIs. -- Keep the suite focused; it currently has 100+ tests. Test durable behavior at process and trust - boundaries: authenticated inputs, user-visible outcomes, authority, - cleanup, and security invariants. Prefer one end-to-end happy path and the - meaningful failure modes over helper permutations. Do not assert private - helper calls, incidental command sequences, task registration, barrel - exports, or exact intermediate formatting. Assert ordering only when the - ordering is itself a security contract, such as scrubbing a credential - before untrusted code runs. -- Add or update a test when externally observable behavior or a security - invariant changes; refactors with unchanged behavior should not require - test changes. -- Write comments only for constraints the code can't show (platform limits, security reasons, ordering requirements), not to narrate the next line. - -## Invariants - -Don't weaken these without an explicit change to the security model -([docs/architecture.md](docs/architecture.md#trust-boundaries) says where -each is enforced): - -1. The gateway verifies the webhook signature before parsing the body. -2. Every run checks the triggering actor's live repository permission before doing anything. -3. Only workflow code writes to GitHub, and every write finds its marker before creating anything. -4. No write credential enters a sandbox. A read-only token is present only to clone or fetch. -5. Tools run only in sandboxes, never in a Workflows task. -6. Models never choose sandbox IDs, branches, repositories, or GitHub targets. -7. A PR opens only after GitHub's tree SHA equals the verified sandbox tree. -8. Automation ends at a draft PR and a COMMENT review. It never approves, requests changes, marks a PR ready, or merges. -9. Spec-to-PR implements the spec as it stood when the approval label was applied. -10. Every sandbox is created with a timeout and closed by `withSandbox` (in `finally`). A claiming task records its sandbox in the status comment, so the run or `/factory cancel` that stops it can terminate the sandbox: Render kills a cancelled task without running `finally`. - -`tests/agents-catalog.test.ts` protects unique agent identities and tool authority. Change it only on purpose. - -## Common changes - -- **An agent or a claiming task:** follow - [CONTRIBUTING.md](CONTRIBUTING.md#customize-an-agent) or - [CONTRIBUTING.md](CONTRIBUTING.md#add-a-claiming-task). -- **A configuration key:** add it to `FactoryConfig` in `factory.config.ts` - with a doc comment, and to - [docs/configuration.md](docs/configuration.md#behavior-and-limits). -- **An environment variable:** add it to `render.yaml`, `.env.example`, - `SERVICE_ENV` in `scripts/lib/requirements.ts` (so `doctor` checks it), and - [docs/configuration.md](docs/configuration.md#environment-variables). -- **A webhook event, label, or App permission:** update `scripts/lib/requirements.ts`, which `setup:app`, `setup:github`, and `doctor` all read. -- **The runner or `sandbox/`:** these are baked into each snapshot, so they reach sandboxes only after a rebuild (the next push to each base branch, or a `refresh-snapshot` run). +This file applies to the whole repository. + +## Use the right guide + +- First deployment: + [docs/runbooks/setup.md](docs/runbooks/setup.md) +- Repositories, labels, models, permissions, limits, or snapshots: + [docs/configuration.md](docs/configuration.md) +- Agents, Workflows tasks, or GitHub triggers: + [docs/runbooks/extend.md](docs/runbooks/extend.md) +- State, coordination, publishing, or security: + [docs/architecture.md](docs/architecture.md) +- Local development: + [CONTRIBUTING.md](CONTRIBUTING.md) + +Follow runbooks in order. Stop at every human checkpoint. Do not claim setup +is complete until both `doctor` checks and the Triage smoke test pass. + +## System boundaries + +software-factory deploys two processes: + +- `app/entrypoints/gateway.ts` verifies webhooks and starts tasks. +- `app/entrypoints/workflows.ts` registers and runs tasks. + +Agents run inside Render Sandboxes. Only code in the Workflows service writes +to GitHub. GitHub and Render Workflows hold application state; there is no +database. + +Two dependency rules are mandatory: + +1. `app/runner/` imports nothing from the rest of `app/`. It is bundled into + a self-contained Sandbox program. +2. `app/gateway/` does not import GitHub authentication, Sandboxes, agents, or + workflows. The gateway holds no GitHub credential and runs no model. + +Read the canonical +[security invariants](docs/architecture.md#security-invariants) before +changing credentials, claims, Sandboxes, publishing, or GitHub writes. + +## Code conventions + +- Use strict TypeScript, ESM, and NodeNext. Relative imports end in `.js`. + There are no path aliases. +- Biome uses tabs and a line width of 150. +- Import public module APIs through `index.ts` barrels. Do not export + module-internal helpers. +- Validate every external or cross-process input with Zod at its boundary. +- Build agent output objects with `z.strictObject`. Bound strings and arrays. +- Keep task inputs and results small and JSON-serializable. +- Open Sandboxes through `withSandbox`. In claiming tasks, call + `claim.recordSandbox()` immediately after opening one. +- Use `ExpectedFailure` only for expected messages safe to publish on GitHub. + Keep outcome names in snake_case. +- Quote generated shell arguments with `shq`. Put long or + security-sensitive Sandbox commands in `sandbox/` scripts. +- Pass model text through `app/format/` before writing it to GitHub. +- Inject GitHub, Sandbox, agent, and run-control dependencies into task + implementations. Tests use fakes, never live services. +- Comment security reasons, platform constraints, and required ordering. Do + not narrate obvious code. + +## Tests + +Test behavior at process and trust boundaries: + +- authenticated input; +- user-visible output; +- authority and target selection; +- cleanup and cancellation; and +- security invariants. + +Do not test private helper calls, incidental command sequences, task +registration, barrel exports, or formatting that users cannot observe. Assert +ordering only when it is part of a security contract. + +Update tests when observable behavior or an invariant changes. A refactor +with unchanged behavior should not require new assertions. + +`tests/agents-catalog.test.ts` protects unique agent IDs, tool authority, and +strict output schemas. + +## Cross-cutting changes + +- Configuration key: update `FactoryConfig` in `factory.config.ts` and + `docs/configuration.md`. +- Environment variable: update `render.yaml`, `.env.example`, `SERVICE_ENV` + in `scripts/lib/requirements.ts`, and `docs/configuration.md`. +- App permission, event, or label: update + `scripts/lib/requirements.ts`, setup behavior, and doctor tests. +- `app/runner/` or `sandbox/`: refresh every repository snapshot after + deployment. ## Verify ```bash -npm run check # Biome lint, tsc --noEmit, all Vitest tests -shellcheck sandbox/*.sh sandbox/build/*.sh -render blueprints validate render.yaml # when render.yaml changes -npm audit --omit=dev --audit-level=high +npm run check ``` -For one test file: `npx vitest run tests/gateway.test.ts`. `npm run doctor` is read-only; see -[Operations](docs/operations.md#check-a-deployment) for local and deployed usage. +See [Contributing](CONTRIBUTING.md#verify-changes) for targeted and +change-specific checks. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9fde781..506c85d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,20 +1,24 @@ # Contributing -Read [Architecture](docs/architecture.md) before changing run coordination, -Sandbox boundaries, credentials, or publishing. +Read [Architecture](docs/architecture.md) before changing claims, credentials, +Sandboxes, publishing, or GitHub authority. Use the +[extension runbook](docs/runbooks/extend.md) for agents, Workflows tasks, and +GitHub triggers. ## Local development -Local development requires Render CLI 2.29 or later. It still uses -real Render Sandboxes, a real GitHub App, and a real model. Use a separate -test repository and preferably a separate App. +Local development requires Render CLI 2.29 or later. It still uses real +Render Sandboxes, a real GitHub App, and a real model. Use a test repository +and, when possible, a separate App. + +Install dependencies and create a local environment file: ```bash npm ci cp .env.example .env ``` -Fill `.env`, then start each process in its own terminal: +Fill `.env` without committing it. Start each process in its own terminal: ```bash # Terminal 1 @@ -24,20 +28,23 @@ npm run dev:workflows RENDER_USE_LOCAL_DEV=true npm run dev:gateway ``` -Expose port 3000 through a public HTTPS tunnel and point the App at it: +Expose port 3000 over public HTTPS: ```bash cloudflared tunnel --url http://localhost:3000 npm run setup:github -- --gateway-url https:// ``` -A GitHub App has one webhook URL. Repointing a production App interrupts its -production deliveries. +A GitHub App has one webhook URL. Pointing a production App at local +development interrupts production deliveries. + +### Start a task directly -You can also invoke a local task directly: +List and invoke local tasks with the Render CLI: ```bash render workflows tasks list --local + render workflows tasks runs start --local \ --task \ --input='[{...}]' @@ -48,14 +55,16 @@ Task input schemas are in ## Verify changes +Run the standard check before handing off: + ```bash npm run check ``` -This runs Biome, TypeScript, and Vitest. Tests fake GitHub, Render, and model +It runs Biome, TypeScript, and Vitest. Tests fake GitHub, Render, and model calls. -When relevant, also run: +Run additional checks when relevant: ```bash shellcheck sandbox/*.sh sandbox/build/*.sh @@ -63,161 +72,53 @@ render blueprints validate render.yaml npm audit --omit=dev --audit-level=high ``` -For one test: +Run one test file with: ```bash npx vitest run tests/gateway.test.ts ``` -## Customize an agent +## Extend the factory -An agent consists of a Markdown prompt and typed definition: +Follow [docs/runbooks/extend.md](docs/runbooks/extend.md) to: -```text -app/agents// - index.ts - prompt.md -``` - -```typescript -export const implementer = defineAgent({ - id: "implementer", - prompt: "implementer/prompt.md", - model: "large", - tools: ["read", "ls", "bash", "edit", "write"], - output: Implementation, - maxTurns: 150, -}); -``` +- add a typed agent and prompt; +- add a claiming or maintenance task; +- route a GitHub event, label, or command; +- update GitHub App requirements; +- test authority and user-visible behavior; and +- identify required deploys and snapshot refreshes. -The definition fixes: - -- a unique ID; -- a prompt relative to `app/agents/`; -- a configured model tier; -- tools selected from `read`, `ls`, `bash`, `edit`, and `write`; -- a bounded Zod output schema; -- a maximum number of model calls; and -- an optional thinking level. - -Put repository- and run-specific context in the typed message passed to -`runAgent`, not in the system prompt. Tell the model to call `submit` exactly -once; the tool supplies the output schema. - -Only an implementer should normally receive `edit` or `write`. The read-only -shell policy is a behavioral guard, not a security boundary. - -To add an agent: - -1. add its prompt and definition under `app/agents//`; -2. export it from `app/agents/index.ts`; -3. add or reuse a bounded schema in `app/schemas/`; -4. call it from a task with that task's Sandbox and `claim.agentOptions(sandbox)`; -5. add it to `tests/agents-catalog.test.ts`; and -6. run `npm run check`. - -Prompt changes require a Workflows deploy. Changes under `app/runner/` or -`sandbox/` also require repository snapshot refreshes. - -## Add a claiming task - -A claiming task acts on one issue or PR while it holds a claim on it. -Triage, Spec-to-PR, and PR Reviewer are claiming tasks. The claim is the -status comment: the run it names holds it. Declare a claiming task with -`claimingTask()` (`app/workflows/claiming-task.ts`). It registers the -Workflows task, validates the input, resolves the repository, and runs your -work inside `withClaim` (`app/claims/claim.ts`). `withClaim` checks the -actor's permission, stops the previous run and takes over the status comment, -keeps the comment current, and reports how the run ended. - -1. Add the task name to `TASKS` in `app/config/index.ts`. -2. Add a bounded input schema in `app/schemas/tasks.ts`. -3. Declare the task under `app/workflows//` with - `claimingTask({ name, title, input, subject, timeoutSeconds }, work)`. - `title` is how the status comment names the task. -4. Open Sandboxes through `withSandbox`, and call - `claim.recordSandbox(sandbox)` first thing, so a run that replaces yours - can terminate its Sandbox. -5. Call `claim.progress()` before major steps and `claim.checkpoint()` before - durable writes. -6. Throw `ExpectedFailure` only for expected messages safe to show on GitHub. -7. Import the task module from `app/workflows/index.ts`. -8. Route an authenticated webhook to it in `app/gateway/routes.ts`. -9. Update setup requirements and add boundary-focused tests. - -Claiming tasks never retry automatically, because a retry can repeat model -spend and durable work. A person reapplies a label to start a fresh run. - -## Add or change a GitHub trigger - -The gateway routing table is a pure mapping from authenticated event payload -to one validated task input. - -1. Update `route()` in `app/gateway/routes.ts`. -2. Update `REQUIRED_EVENTS`, `REQUIRED_PERMISSIONS`, or `LABELS` in - `scripts/lib/requirements.ts`. -3. Update the existing App when permissions or subscriptions changed. -4. Rerun `npm run setup:github`. -5. Add route and setup/doctor tests. - -Use the GitHub delivery ID as the task idempotency key. Derive repositories, -installations, actors, subjects, and commits from the authenticated payload. -Never accept write destinations from model output. - -Every GitHub write should update or reuse an App-authored artifact marker. -Workflows idempotency alone has a finite scope and is not sufficient. +The runbook is the canonical procedure. Keep implementation details in code +and architectural reasons in [Architecture](docs/architecture.md). ## Add persistent state -The factory keeps no database: GitHub and Render Workflows hold the state it -needs, and telemetry goes to logs. When a feature needs its own data, such as -a spec library, budgets, or run history you query: - -1. Add a Render Postgres database to `render.yaml` and wire `DATABASE_URL` - into the services that need it with `fromDatabase`. -2. Run migrations from the gateway's `preDeployCommand`, so each deploy - migrates before new code starts. -3. Use the internal connection string on Render. The external one is for - local tools and requires TLS. -4. Keep run coordination in the status comment. A database that also decides - who owns an issue becomes a second source of truth to reconcile. - -## Code structure - -```text -app/ - entrypoints/ Gateway and Workflows process entry points - gateway/ Webhook authentication and routing - workflows/ Triage, Spec-to-PR, PR Reviewer, cancel, snapshots - claims/ withClaim: permission, status-comment claim, takeover, failure reporting - agents/ Agent definitions and host-side execution - runner/ Self-contained agent runtime bundled into snapshots - sandbox/ Sandbox and snapshot lifecycle - changeset/ Candidate-tree validation and publishing - github/ App authentication, REST, Git Data, and markers - events/ Run and agent telemetry as JSON log lines - schemas/ Cross-process Zod contracts -sandbox/ Runtime scripts copied into snapshots -scripts/ Setup and doctor -tests/ Unit and boundary tests with fakes -``` +The default template has no database. GitHub owns collaboration artifacts, +Render Workflows owns task execution, and telemetry goes to logs. -Two dependency boundaries are mandatory: +Add a database only for new data with its own lifecycle, such as a searchable +spec library, budgets, or retained run history: -- `app/runner/` must not import from the rest of `app/`; it is bundled into a - self-contained Sandbox program. -- `app/gateway/` must not import GitHub authentication, Sandboxes, agents, or - workflows; the gateway holds no GitHub credential and runs no model. +1. Add a Render Postgres database to the `production` environment's + `databases` list in `render.yaml`. +2. Wire `DATABASE_URL` into only the services that need it. +3. Run migrations from a service `preDeployCommand`. +4. Use the internal connection string on Render. Local tools use the external + connection with TLS. -## Documentation rules +Do not move claim ownership into the database. The status comment must remain +the single source of truth for the run that owns an issue or pull request. -Keep each fact in its canonical document: +## Documentation ownership -- agent deployment protocol: `docs/runbooks/setup.md`; -- short human setup guide: `docs/quickstart.md`; -- supported settings: `docs/configuration.md`; -- runtime procedures and failures: `docs/operations.md`; -- state, boundaries, and invariants: `docs/architecture.md`; and -- contributor workflows: this file. +- `README.md`: product entry point. +- `docs/quickstart.md`: human setup handoff. +- `docs/runbooks/setup.md`: first-deployment agent protocol. +- `docs/configuration.md`: settings, run control, deployment checks, and + snapshot procedures. +- `docs/architecture.md`: state, boundaries, invariants, and accepted risks. +- `docs/runbooks/extend.md`: extension agent protocol. +- `CONTRIBUTING.md`: local development and verification. -Link to the canonical explanation instead of copying it. +Link to the canonical document instead of copying its content. diff --git a/README.md b/README.md index 2b26878..fd0be5f 100644 --- a/README.md +++ b/README.md @@ -1,95 +1,93 @@ -# Software Factory +# software-factory -Software Factory turns a labeled GitHub issue into an approved specification, -a verified draft pull request, and an automated code review. People approve -every transition; the factory never approves or merges changes. +software-factory turns GitHub issues into specifications and verified draft +pull requests. It also reviews pull requests. Labels start the three main +tasks; comments and base-branch pushes handle cancellation, follow-up Triage, +and snapshot refresh. The factory never approves or merges code. -It runs in your Render workspace using one web service, one Workflows -service, and Render Sandboxes. +It runs in your Render workspace as a web service, a Workflows service, and +short-lived Render Sandboxes. -**Start by [forking this repository](https://github.com/render-examples/software-factory/fork).** -The factory must deploy from your fork because `factory.config.ts` defines -which repositories it may access. - -## Workflow +## How it works ```text issue --factory:triage--> specification -specification --factory:spec-approved--> verified draft PR +specification --factory:spec-approved--> draft pull request pull request --factory:pr-review--> COMMENT review ``` -- **Triage** reads the issue, discussion, and code, then asks focused - questions, writes a specification, or declines. -- **Spec-to-PR** implements the exact approved specification, runs configured - verification, repairs bounded failures, and opens a draft pull request. +- **Triage** reads the issue, discussion, and repository. It asks questions, + writes a specification, or declines the request. +- **Spec-to-PR** implements the approved specification, runs your verification + commands, and opens a draft pull request. - **PR Reviewer** runs verification and reviews security, correctness, - performance, and tests. - -GitHub remains where people discuss and approve work. Render Workflows owns -durable execution. Sandboxes contain model-directed tools and repository -code. - -## Deploy - -You need a Render workspace with Sandboxes access in Oregon, a Render API -key, a dedicated spend-capped model API key, GitHub repository admin access, -and the Node.js version in `.node-version`. - -For the fastest path, ask a coding agent to follow -[`docs/runbooks/setup.md`](docs/runbooks/setup.md). It separates work the -agent can perform from GitHub, secret, and paid-resource checkpoints that -require you. - -1. Fork the repository and configure one target in `factory.config.ts`. -2. Run `npm ci` and `npm run setup:app -- --write-env`. -3. Install the generated GitHub App on the target repository. -4. Create a Render Blueprint from your fork and provide its secret values. -5. Run `npm run setup:github -- --gateway-url https://.onrender.com`. -6. Activate the webhook in GitHub App settings. -7. Run `npm run doctor`, then apply `factory:triage` to an issue. - -See the [Quickstart](docs/quickstart.md) for the short human guide or the -[Setup runbook](docs/runbooks/setup.md) for the complete agent protocol. - -## Security model - -- Agents and model-directed tools run only inside short-lived Sandboxes. -- No GitHub write credentials enter the Sandbox. -- Clone and fetch use temporary read-only credentials that are removed before - repository code or agents run. -- Models cannot choose repositories, branches, issue numbers, pull requests, - or GitHub write targets. -- Trusted Workflows code validates candidate changes and requires GitHub to - reproduce their tree SHA before publishing. -- Publishing rejects unsafe paths, workflow files, symlinks, submodules, - oversized changes, and branches containing human commits. - -Sandboxes use allow-all egress, and a model key exists in runner memory during -a call. Treat repository and pull request content as untrusted, and use a -dedicated Render workspace and provider-capped key when isolation matters. -Read the [trust boundaries and residual risks](docs/architecture.md#trust-boundaries) -before deploying against private code. + performance, and tests. It posts a non-approving GitHub COMMENT review. + +GitHub holds the discussion, labels, specifications, pull requests, and +reviews. Render Workflows runs the tasks. Sandboxes contain the checkout, +agent loop, and model-directed tools. + +## Set up + +Start by +[forking this repository](https://github.com/render-examples/software-factory/fork). +Deploy from your fork: its `factory.config.ts` defines the repositories the +factory may access. + +The recommended setup is agent-guided: + +1. Check the prerequisites in the [Quickstart](docs/quickstart.md). +2. Ask a coding agent to follow the + [Setup runbook](docs/runbooks/setup.md). +3. Complete the GitHub, secret, and paid-resource checkpoints when the agent + asks. + +The runbook configures the fork before deployment, creates a GitHub App, +deploys the Render Blueprint, connects the webhook, verifies both services, +and runs one Triage smoke test. + +## Security + +- Model-directed code and tools run inside Sandboxes, not on the Workflows + host. +- GitHub write credentials never enter a Sandbox. +- Clone and fetch credentials are read-only and removed before repository code + or agents run. +- Models propose typed results and file trees. Code in the Workflows service + chooses every repository, ref, branch, issue, pull request, and write target. +- Publishing rejects unsafe paths and modes, workflow files, symlinks, + submodules, oversized changes, and branches with human commits. +- GitHub must reproduce the candidate tree SHA before the factory publishes + it. + +Sandboxes use allow-all egress, and the model key exists in runner memory +during a call. Read the [trust boundaries](docs/architecture.md#trust-boundaries) +and [residual risks](docs/architecture.md#residual-risks) before using private +code or untrusted pull requests. ## Limits -- Tasks never chain automatically. -- Verification is command-based; browser testing is not built in. -- Turn limits and timeouts bound agents, but the factory has no hard dollar +- Main tasks do not chain. Labels, comments, commands, and base-branch pushes + start work through the gateway. +- Verification runs shell commands; browser testing is not built in. +- Turn limits and timeouts bound execution, but the factory has no hard dollar budget. -- Anyone with the configured GitHub permission can start model spend. -- PR Reviewer executes configured verification against the labeled PR code. -- Sandbox API calls share the Render workspace's rate limit. +- Anyone with `minimumPermission` can start model spend. +- PR Reviewer runs configured commands against the labeled pull request. +- Sandbox API calls count against the Render workspace's request limit. ## Documentation -- [Setup runbook](docs/runbooks/setup.md) — agent-guided deployment with verified checkpoints. -- [Quickstart](docs/quickstart.md) — deploy and complete the first run. -- [Configuration](docs/configuration.md) — repositories, labels, models, and limits. -- [Operations](docs/operations.md) — logs, cancellation, snapshots, and failures. -- [Architecture](docs/architecture.md) — state ownership and trust boundaries. -- [Contributing](CONTRIBUTING.md) — local development, agents, tasks, and triggers. -- [Design notes](docs/design-notes.md) — reusable patterns behind the system. +- [Quickstart](docs/quickstart.md) — what you need and how to start setup. +- [Setup runbook](docs/runbooks/setup.md) — agent protocol for the first + deployment. +- [Configuration](docs/configuration.md) — settings, logs, run control, + deployment checks, and snapshots. +- [Architecture](docs/architecture.md) — execution, state, coordination, and + trust boundaries. +- [Extension runbook](docs/runbooks/extend.md) — add agents, Workflows tasks, + and GitHub triggers. +- [Contributing](CONTRIBUTING.md) — local development and verification. ## License diff --git a/app/workflows/cancel.ts b/app/workflows/cancel.ts index b5cd8e1..f0b9eb2 100644 --- a/app/workflows/cancel.ts +++ b/app/workflows/cancel.ts @@ -11,10 +11,14 @@ type CancelGitHub = StatusGitHub & Pick; interface CancelResult { cancelled: boolean; runId?: string; - reason?: "permission_denied" | "nothing_running"; + reason?: "permission_denied" | "nothing_running" | "already_stopped"; } -/** `/factory cancel`: stops the run the status comment names, if it's still active, and terminates its sandbox. */ +/** + * `/factory cancel`: stops the run the status comment names, if it's still active, and terminates its sandbox. A + * comment that still says "running" for a run that already stopped (it was killed before it could finish) is + * marked cancelled too, so the stale status and working label don't linger. + */ export async function cancelRun( ctx: Pick, raw: unknown, @@ -36,14 +40,15 @@ export async function cancelRun( } const current = await readStatus(gh, input.number); const stop = { repo: repo.fullName, subject: event.subject, by: runId, outcome: "cancelled" } as const; - if (!current || !(await stopRun(deps.runs ?? workflowRuns(), current.data, stop))) { + const active = current ? await stopRun(deps.runs ?? workflowRuns(), current.data, stop) : false; + if (!current || (!active && current.data.state !== "running")) { await gh.createComment(input.number, `@${input.actor} no factory run is in progress here.`); return { cancelled: false, reason: "nothing_running" }; } - const { runId: cancelled, stage, title } = current.data; - const summary = `cancelled by @${input.actor}.`; - await finishStatus({ gh, number: input.number, runId: cancelled, task: stage, title: title ?? stage, state: "cancelled", summary }); - return { cancelled: true, runId: cancelled }; + const { runId: stopped, stage, title } = current.data; + const summary = active ? `cancelled by @${input.actor}.` : `cleared by @${input.actor}; the run had already stopped without finishing.`; + await finishStatus({ gh, number: input.number, runId: stopped, task: stage, title: title ?? stage, state: "cancelled", summary }); + return active ? { cancelled: true, runId: stopped } : { cancelled: false, runId: stopped, reason: "already_stopped" }; })(); emit({ ...event, type: "run.finished", payload: { stage: TASKS.cancel, state: "done", ...result } }); diff --git a/docs/architecture.md b/docs/architecture.md index 1a0d3fc..6b8fcfe 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,8 +1,8 @@ # Architecture -Software Factory separates model-directed work from authority. Agents inspect -and modify code inside short-lived Render Sandboxes; trusted Workflows code -decides what may be written to GitHub. +Agents inspect and modify repositories inside short-lived Render Sandboxes. +Code in the Workflows service validates their results and performs every +GitHub write. ## System @@ -10,197 +10,230 @@ decides what may be written to GitHub. flowchart LR GitHub -->|signed webhook| Gateway Gateway -->|start task| Workflows - Workflows -->|restore and execute| Sandbox - Sandbox -->|proposed result| Workflows - Workflows -->|validated writes| GitHub + Workflows -->|restore| Sandbox + Sandbox -->|typed result or candidate tree| Workflows + Workflows -->|validated write| GitHub Workflows -->|build| Snapshot Snapshot -.->|restore| Sandbox ``` -The Blueprint creates two services: +The Blueprint deploys two services: -- **Gateway** verifies GitHub webhooks and maps each delivery to one task. -- **Workflows** runs the claiming tasks, agents, cancellation, snapshot refresh, and - GitHub publishing. +- **Gateway** verifies webhooks and maps each delivery to one task. +- **Workflows** runs tasks, manages Sandboxes, and writes to GitHub. -Each claiming task receives a fresh Sandbox restored from a repository snapshot. -Render Workflows owns task state and timeouts. GitHub owns the human -conversation and artifacts. The application has no custom queue, dispatcher, -heartbeat loop, or maintenance cron. +The application does not implement its own queue, dispatcher, heartbeat +process, maintenance loop, or database. -## Human-gated flow +## Workflow ```text issue --factory:triage--> specification -specification --factory:spec-approved--> verified draft PR +specification --factory:spec-approved--> draft pull request pull request --factory:pr-review--> COMMENT review ``` -The tasks never chain automatically. The factory cannot approve, request -changes, mark ready, or merge. - ### Triage -Triage reads the issue discussion and base branch, then asks up to five -questions, writes a bounded specification, or declines. Questions and specs -are GitHub comments with hidden structured markers. +Triage reads the issue discussion and the base branch. The triage agent asks +up to five questions, declines the request, or sends it to the spec-writing +agent. The spec-writing agent produces a bounded specification. Questions and +specifications are GitHub comments with hidden markers. ### Spec-to-PR -Spec-to-PR verifies that the specification has not changed since approval, -implements it in a Sandbox, runs configured checks, permits bounded repair -attempts, and opens a draft pull request on -`factory/issue-`. +Spec-to-PR confirms that the approved specification has not changed. It +implements that specification in a Sandbox, runs the configured verification +commands, allows a bounded number of repairs, and opens a draft pull request +on `factory/issue-`. ### PR Reviewer -PR Reviewer checks out the labeled head commit, runs configured verification, -and posts one COMMENT review covering security, correctness, performance, and -tests. +PR Reviewer checks out the labeled pull request commit, runs the configured +verification commands, and posts one COMMENT review. The review covers +security, correctness, performance, and tests. + +## State -## State ownership +Each system owns the state it already manages: | State | Owner | |---|---| | Task status, timing, input, and result | Render Workflows | -| Issues, labels, specifications, PRs, and reviews | GitHub | -| Current run for an issue or PR, and its Sandbox | App-authored status comment marker | -| User-visible run status | The same status comment, and the working label | +| Issues, labels, specifications, pull requests, and reviews | GitHub | +| Current run and Sandbox for an issue or pull request | App-authored status comment | | Prepared repository filesystem | Render Sandbox snapshot | -| Run and agent outcomes, usage, and cost | JSON lines in the service logs | +| Run and agent outcomes, usage, and cost | JSON lines in Workflows logs | + +Telemetry is not application state. Nothing reads log events to decide what a +task may do. -Telemetry is write-only: nothing reads it back to decide what a task does. +## Coordination -## Idempotency and competing runs +Webhook delivery is not exactly once. GitHub can redeliver events, and people +can apply labels or post comments at the same time. The factory handles this +at three boundaries: -The system handles retries and overlap at three levels: +1. The GitHub delivery ID becomes the Workflows idempotency key. +2. The status comment names the run that holds the claim on an issue or pull + request. +3. Hidden markers identify App-authored specifications, pull requests, and + reviews so writes can update or reuse them. -1. GitHub's delivery ID is the Workflows idempotency key. -2. The status comment is the claim: a claiming task works only while the - comment names its run. -3. App-authored markers let writes update or reuse existing artifacts. +A new claiming run cancels the run named in the status comment, terminates +its recorded Sandbox, and writes its own run ID to the comment. The run whose +write remains visible holds the claim. Other runs stop when they next check +the comment. -A new trigger takes over by cancelling the run the status comment names, -terminating that run's Sandbox, and rewriting the comment to name itself. -Render kills a cancelled task run's process within about a second, so a -replaced run can't keep writing. Two triggers that arrive together both take -over; the run whose status write lands last keeps the claim, and the other -stops at its next checkpoint. Every claiming task checks the claim before -progress updates and durable writes. Publishing also uses a deterministic -branch and refuses to overwrite human commits. +Claiming tasks check ownership before progress updates and durable writes. +Publishing also uses a deterministic branch and refuses to overwrite human +commits. -## Sandbox lifecycle +This design keeps coordination beside the GitHub object it controls. Adding a +database would create a second owner for the same claim. -A base-branch push builds a repository snapshot containing runtime tools, a -base checkout, and installed dependencies. Every claiming task: +## Sandboxes and snapshots + +A base-branch push builds a repository snapshot with the runner, a base +checkout, and installed dependencies. A claiming task then: 1. restores the newest snapshot; -2. verifies its permissions and process hardening; +2. verifies runtime permissions and process hardening; 3. fetches the exact requested ref with a temporary read-only token; -4. removes and verifies removal of that token; -5. reinstalls dependencies if lockfiles or the install command changed; +4. removes the token and verifies its removal; +5. reinstalls dependencies when the lockfiles or install command changed; 6. runs agents and repository commands as an unprivileged user; and 7. closes the Sandbox in `finally`. -A cancelled task run is killed without reaching `finally`, so each claiming -task records its Sandbox in the status comment, and the run or `/factory cancel` -that stops it terminates the Sandbox. The provider-side timeout is the backstop -for anything else that dies before cleanup. +Render terminates a cancelled task without running `finally`. For that case, +each claiming task records its Sandbox ID in the status comment. A replacement +run or `/factory cancel` terminates the recorded Sandbox. The provider timeout +is the final cleanup backstop. + +Snapshots avoid reinstalling dependencies on every run. Restored Sandboxes +still fetch the requested commit and verify their current state; a snapshot is +prepared state, not trusted current state. -## Agent contract +## Agents -An agent definition fixes its prompt, model tier, tools, output schema, turn -limit, and optional thinking level. The complete model loop and every -model-directed tool run inside the Sandbox. +An agent definition fixes: -Agents finish through a typed `submit` tool. The Sandbox validates the -submission against JSON Schema, and the host validates it again with the -authoritative Zod schema. Repositories, refs, issue numbers, pull requests, -and write destinations come from authenticated task input—not model output. +- its prompt; +- model tier and thinking level; +- available tools; +- output schema; +- turn limit; and +- stable ID. -## Publishing boundary +The model loop and every model-directed tool run in the Sandbox beside the +checkout. Agent tools never run on the Workflows host. -The Sandbox has no GitHub write credential. It produces a candidate tree, -which trusted Workflows code validates before publishing: +Every agent finishes through a typed `submit` tool. The runner validates the +submission against JSON Schema. The host validates it again with the agent's +Zod schema. Repositories, refs, issue numbers, pull requests, branches, and +write targets come from authenticated task input, never model output. -1. collect changes through a throwaway Git directory; -2. validate paths, modes, file counts, sizes, and blob hashes; -3. reject workflow files, symlinks, and submodules; -4. ask GitHub to build the same tree independently; -5. require GitHub's tree SHA to match the Sandbox tree SHA; and -6. move the deterministic branch only when ownership is safe. +## Publishing -This makes the model a source of proposed bytes, not authority over GitHub. +The Sandbox has no GitHub write credential. It returns a candidate tree to +the Workflows host, which: + +1. collects changes through a throwaway Git directory; +2. validates paths, modes, file counts, sizes, and blob hashes; +3. rejects workflow files, symlinks, and submodules; +4. asks GitHub to build the same tree; +5. compares GitHub's tree SHA with the Sandbox tree SHA; and +6. moves the deterministic branch only when the factory still owns it. ## Trust boundaries -The factory treats issue text, comments, source files, dependency scripts, -and pull request diffs as untrusted. +Issue text, comments, source files, dependency scripts, and pull request diffs +are untrusted. ### Gateway -The gateway verifies the webhook signature before parsing JSON. It holds no -GitHub credential and runs no model or repository code. +The gateway verifies the webhook signature before parsing the body. It holds +no GitHub credential and runs no model or repository code. ### Workflows host -The Workflows service holds service credentials. Trusted code selects -destinations, enforces run ownership, validates Sandbox output, and performs -GitHub writes. No agent tool runs in this process. +The Workflows service holds GitHub and Render credentials. It selects every +destination, checks claim ownership, validates Sandbox output, and performs +GitHub writes. -### Sandboxes +### Sandbox -Clone and fetch use temporary read-only credentials that are removed before -repository code or agents run. Runtime files are root-owned and read-only. -Agents and repository commands run unprivileged. +Clone and fetch credentials are read-only and temporary. Runtime files are +root-owned and read-only. Agents and repository commands run as an +unprivileged user. -The model key enters the Sandbox for an agent call. It is delivered through a -root-only path and stdin, removed from the environment before tools run, and -protected by a non-dumpable runner. These controls reduce exposure but do not -eliminate it. +The model key enters the Sandbox during an agent call. The runner receives it +through a root-only path and stdin, removes provider keys from tool +environments, and runs under a non-dumpable Node binary. These controls reduce +exposure; they do not make untrusted code safe. ## Security invariants Changes must preserve these properties: -1. Verify webhook signatures before parsing. -2. Check the triggering actor's live permission for every run. -3. Keep GitHub write credentials out of Sandboxes. -4. Run all model-directed tools inside Sandboxes. -5. Reconstruct repositories, refs, branches, and write targets from trusted input. -6. Publish only after GitHub reproduces the validated tree SHA. -7. End automation at a draft PR and COMMENT review. -8. Implement the exact specification that was approved. -9. Put a timeout and `finally` cleanup around every Sandbox. - -Tests pin these invariants and each built-in agent's tool authority. +1. Verify a webhook signature before parsing its body. +2. Check the triggering actor's current repository permission for every + claiming run. +3. Let only code in the Workflows service write to GitHub. +4. Keep GitHub write credentials out of Sandboxes. +5. Run every model-directed tool inside a Sandbox. +6. Build repositories, refs, branches, and write targets from trusted input. +7. Publish only after GitHub reproduces the validated tree SHA. +8. Stop automation at a draft pull request and a COMMENT review. +9. Implement the specification as it existed when approved. +10. Give every Sandbox a provider timeout and `finally` cleanup. Record a + claiming task's Sandbox so cancellation can terminate it. + +Tests pin these invariants and the tool authority of every built-in agent. ## Residual risks -- Sandboxes use allow-all egress, leaving network and DNS exfiltration paths. +- Sandboxes use allow-all egress. - The model key exists in runner memory during a call. -- Any collaborator with `minimumPermission` can initiate model spend. -- PR Reviewer executes configured verification against the labeled PR code. -- Prompt injection can influence specs, code, and reviews. +- Prompt injection can influence specifications, code, and reviews. +- Anyone with `minimumPermission` can start model spend. +- PR Reviewer runs repository-defined commands from the labeled pull request. - The Render API key may have workspace-wide authority. -- The claim relies on reading the status comment back right after writing it. - Two triggers within a second of each other can both do work before one stops. -- Workflows inputs and results are visible to users with access to the service. - -Use a dedicated Render workspace and spend-capped model key when stronger -isolation is required. +- Near-simultaneous triggers can both work until one loses the status-comment + claim. +- Workflows inputs and results are visible to people with service access. +- Telemetry can include repository names, issue subjects, and errors that + quote repository output. +- The filesystem of each deployed Render service is ephemeral. Durable state + belongs in GitHub, Workflows, or Sandbox snapshots. +- A cancelled snapshot build can leave its builder Sandbox running until the + one-hour provider timeout because builders are not recorded in a status + comment. + +Use a dedicated Render workspace and a spend-capped model key when stronger +isolation matters. + +Expected work failures produce a useful status-comment result. Unexpected +platform or programming failures fail the Workflows task and require +operator investigation. ## Code map -Start with these files: - -1. [`app/gateway/routes.ts`](../app/gateway/routes.ts) — event routing. -2. [`app/claims/claim.ts`](../app/claims/claim.ts) — the claim lifecycle every claiming task shares. -3. [`app/workflows/triage/index.ts`](../app/workflows/triage/index.ts) — one complete claiming task. -4. [`app/agents/run.ts`](../app/agents/run.ts) — Sandbox agent execution. -5. [`app/sandbox/sandbox.ts`](../app/sandbox/sandbox.ts) — Sandbox lifecycle. -6. [`app/workflows/spec-to-pr/index.ts`](../app/workflows/spec-to-pr/index.ts) — implementation and verification. -7. [`app/changeset/index.ts`](../app/changeset/index.ts) — validated publishing. - -See [Contributing](../CONTRIBUTING.md) for local development and extension -recipes. +Start with: + +1. [`app/gateway/routes.ts`](../app/gateway/routes.ts) — webhook-to-task + routing. +2. [`app/workflows/claiming-task.ts`](../app/workflows/claiming-task.ts) — + claiming task declaration. +3. [`app/claims/claim.ts`](../app/claims/claim.ts) — permission, ownership, + progress, and completion. +4. [`app/workflows/triage/index.ts`](../app/workflows/triage/index.ts) — one + complete claiming task. +5. [`app/agents/run.ts`](../app/agents/run.ts) — host-side agent execution. +6. [`app/runner/main.ts`](../app/runner/main.ts) — Sandbox runner entry point. +7. [`app/sandbox/sandbox.ts`](../app/sandbox/sandbox.ts) — Sandbox lifecycle. +8. [`app/changeset/index.ts`](../app/changeset/index.ts) — validated + publishing. + +Use the [extension runbook](runbooks/extend.md) to add agents, tasks, or +triggers. diff --git a/docs/configuration.md b/docs/configuration.md index 2046b12..87dac2e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,9 +1,13 @@ # Configuration -[`factory.config.ts`](../factory.config.ts) is the factory's committed -application configuration. Both deployed services import it. +[`factory.config.ts`](../factory.config.ts) is committed application +configuration. The gateway and Workflows service both import it. -## Complete example +Edit it before the first deployment and whenever you change repositories, +labels, model assignments, permissions, publishing limits, or Sandbox +lifetimes. + +## Example ```typescript export default defineFactoryConfig({ @@ -43,73 +47,191 @@ export default defineFactoryConfig({ ## Repositories -`repos` is keyed by exact `owner/name`. Webhooks from any other repository -are ignored, even when the App is installed there. +`repos` is keyed by exact GitHub `owner/name`. The factory ignores webhooks +from any other repository, even if the App is installed there. -| Field | Required | Purpose | +| Field | Required | Meaning | |---|---|---| | `baseBranch` | Yes | Branch used for Triage, implementation, and snapshots | -| `install` | No | Dependency command run after repository credentials are removed | -| `verify` | Yes | Commands required before publishing and run during PR review | +| `install` | No | Dependency command run after clone credentials are removed | +| `verify` | Yes | Commands required before publishing and during PR review | Use deterministic install commands such as `npm ci`. Snapshot creation fails -if installation changes a tracked file. An empty `verify` list explicitly -disables command-based verification. +if installation changes a tracked file. Use `verify: []` only when you +intentionally want no command-based verification. After adding a repository: -1. commit and deploy `factory.config.ts`; +1. commit and deploy the configuration; 2. install the GitHub App on the repository; 3. run `npm run setup:github`; and -4. apply a factory label or push to the base branch to build its first snapshot. +4. push to the base branch, start a factory task, or + [refresh the snapshot directly](#refresh-a-snapshot). -## Behavior and limits +## Labels, permission, and limits -| Setting | Default | Purpose | +| Setting | Default | Meaning | |---|---|---| | `labels.triage` | `factory:triage` | Starts or continues Triage | | `labels.specApproved` | `factory:spec-approved` | Starts Spec-to-PR | | `labels.prReview` | `factory:pr-review` | Starts PR Reviewer | | `commandPrefix` | `/factory` | Prefix for commands such as `/factory cancel` | -| `minimumPermission` | `write` | Required live GitHub permission | +| `minimumPermission` | `write` | Live repository permission required to start work | | `models.medium` | `anthropic/claude-sonnet-5` | Triage and PR Reviewer model | | `models.large` | `anthropic/claude-opus-5` | Spec writer and implementer model | | `limits.repairRounds` | `2` | Repair attempts after failed verification | | `limits.maxChangedFiles` | `200` | Maximum files in a published change | | `limits.maxFileBytes` | `1000000` | Maximum size of one changed file | -| `sandbox.timeoutSeconds` | `14400` | Provider-side Sandbox lifetime | +| `sandbox.timeoutSeconds` | `14400` | Provider-side Sandbox lifetime in seconds | + +`minimumPermission` accepts `triage`, `write`, `maintain`, or `admin`. The +factory checks the triggering actor's current permission at the start of +every claiming task. -`setup:github` also creates `factory:working`, which marks an active run. It -is operational state rather than a configurable trigger label. +`setup:gibthub` also creates `factory:working`. This label records an active +run; it is not a trigger. -After renaming a label, deploy both services and rerun -`npm run setup:github`. Existing labels are not removed automatically. +If you rename a trigger label, deploy both services and rerun +`npm run setup:github`. The command creates required labels but does not +remove old ones. -Keep the Sandbox timeout at least as long as the longest Workflows task. +Keep `sandbox.timeoutSeconds` at least as long as the longest Workflows task. ## Models -Each agent selects `medium` or `large`; configuration maps the tier to a -`provider/model` in Pi's model registry. +Agent definitions select the `medium` or `large` tier. +`factory.config.ts` maps each tier to a `provider/model` supported by +`@earendil-works/pi-ai`, the model client used by the Sandbox runner. -To change Anthropic models, update the mappings and redeploy the Workflows -service. To use another supported provider: +Changing a model mapping requires a Workflows redeploy. To add another +provider: -1. choose its registered `provider/model`; -2. find its key variable in [`app/runner/keys.ts`](../app/runner/keys.ts); -3. add the variable to the Workflows service in [`render.yaml`](../render.yaml); -4. add it to [`.env.example`](../.env.example) and setup requirements; +1. choose a registered `provider/model`; +2. find its key name in [`app/runner/keys.ts`](../app/runner/keys.ts); +3. add the key to `factory-workflows` in [`render.yaml`](../render.yaml); +4. add it to [`.env.example`](../.env.example) and + `SERVICE_ENV` in `scripts/lib/requirements.ts`; 5. set the key in Render; and 6. run `npm run doctor -- --service workflows`. -Use a key dedicated to this factory with a provider-enforced spend limit. -Turn limits and timeouts bound execution, but the factory has no hard dollar -budget. +Use a key dedicated to this deployment with a provider-enforced spend limit. +The factory has no hard dollar budget. + +## What must be redeployed + +| Change | Required action | +|---|---| +| Repository, label, permission, or publishing limit | Deploy both services | +| Model mapping | Deploy `factory-workflows` | +| Prompt or Workflows service code | Deploy `factory-workflows` | +| Gateway routing or validation | Deploy `factory-gateway` | +| Label, App permission, or subscribed event | Run `npm run setup:github`; update the App when prompted | +| `app/runner/` or `sandbox/` | Deploy Workflows, then refresh each repository snapshot | +| Repository `install` command or lockfiles | Refresh the snapshot, or let the next task reinstall | + +## Snapshots + +Each configured repository has a dependency-installed Sandbox snapshot. A +push to its base branch refreshes it. If no snapshot exists, the first task +builds one. + +A failed refresh does not delete the previous usable snapshot. + +### Refresh a snapshot + +Start the `refresh-snapshot` task with the repository, its GitHub App +installation ID, and the base branch's current commit: + +```bash +git ls-remote https://github.com//.git refs/heads/ + +cat > input.json <<'JSON' +[{"repo":"/","installationId":,"sha":""}] +JSON + +render workflows tasks runs start \ + /refresh-snapshot \ + --input-file input.json +``` + +The installation ID is the number at the end of the installation settings +URL: + +```text +https://github.com/settings/installations/ +``` + +Organization installations use the equivalent organization settings URL. + +## Follow and control runs + +While a claiming task runs, the issue or pull request has `factory:working`. +The App-authored status comment shows progress, the run ID, and the final +result. Its hidden marker records the current Sandbox for cancellation. + +Open that run under `factory-workflows` in the Render Dashboard to inspect its +input, logs, and result. + +The service also writes structured events: + +- `run.finished` records the task, state, outcome, and duration. +- `agent.finished` records the agent, model, turns, token usage, reported + cost, and stop reason. + +Search them with the Render CLI: + +```bash +render logs -r --text run.finished +render logs -r --text agent.finished +``` + +To cancel a run, post this as the first line of an issue or pull request +comment: + +```text +/factory cancel +``` + +The commenter needs `minimumPermission`. Cancellation stops the run named in +the status comment and terminates its recorded Sandbox. It also clears a stale +`running` status when the task has already stopped. + +To restart work, remove the trigger label and apply it again. + +## Check a deployment + +`doctor` is read-only, does not print secret values, and exits nonzero when a +required check fails. + +From a local checkout, this checks the services and credentials represented +by local `.env`: + +```bash +npm run doctor +``` + +It cannot inspect values stored only in Render. Check the deployed gateway +from its Render Shell: + +```bash +npm run doctor -- --service gateway +``` + +Workflows services have no Shell. Put the deployed Workflows values and exact +`RENDER_WORKFLOW_SLUG` in local `.env`, then run: + +```bash +npm run doctor -- --service workflows +``` + +The [setup runbook](runbooks/setup.md#phase-9-run-doctor) lists the required +values and common failures. ## Environment variables -Every local variable appears in [`.env.example`](../.env.example). The -Blueprint wires the gateway's reference to the Workflows service. +Local variables are listed in [`.env.example`](../.env.example). The +Blueprint supplies `RENDER_WORKFLOW_SLUG` to the gateway from the Workflows +service. ### Gateway @@ -131,9 +253,8 @@ Blueprint wires the gateway's reference to the Workflows service. | `ANTHROPIC_API_KEY` | Call the default models | | `GITHUB_API_URL` | Optional GitHub API base URL | -`GITHUB_API_URL` is not declared by the Blueprint. GitHub Enterprise Server -users must set it in local `.env` for setup commands and manually on -`factory-workflows` in the Render Dashboard. +`GITHUB_API_URL` is not in the Blueprint. GitHub Enterprise Server users must +set it in local `.env` for setup commands and on `factory-workflows` in the +Render Dashboard. -The gateway must bind to `0.0.0.0:$PORT` when deployed. Never commit `.env` -or provider keys. +Never commit `.env` or provider keys. diff --git a/docs/design-notes.md b/docs/design-notes.md deleted file mode 100644 index 10717a1..0000000 --- a/docs/design-notes.md +++ /dev/null @@ -1,61 +0,0 @@ -# Design notes - -These are the reusable ideas behind Software Factory. They are explanatory, -not required for deployment or operation. - -## Let existing systems own their state - -Render Workflows owns task execution. GitHub owns collaboration artifacts, -including the status comment that says which run currently works on an issue -or pull request. There is no database; telemetry goes to logs. Every -additional source of truth creates reconciliation work. - -## Make approval an explicit transition - -Each label authorizes one bounded task. Triage cannot start implementation, -and implementation cannot start review. Native GitHub actions make these -gates visible and auditable without a separate control plane. - -## Design for overlapping execution - -Webhook redelivery, relabels, and near-simultaneous comments make “exactly -once” execution unrealistic. The factory combines task-start idempotency, a -claim held in the status comment, and artifact markers that make effects -repeatable. - -A new run stops the old one (Render kills a cancelled task run) and terminates -its Sandbox, then checks that it still holds the claim immediately before each -durable effect. - -## Prepare expensive state, verify current state - -Snapshots avoid reinstalling tools and dependencies for every task. A -restored Sandbox still fetches the requested ref, checks its manifest, and -reinstalls when lockfiles or configuration changed. - -## Put tools next to code, away from authority - -The agent loop runs in the Sandbox beside the checkout. The security boundary -is isolated tool placement, an unprivileged user, temporary read-only -credentials, bounded lifetime, and host-side acceptance—not a shell-command -filter. - -## Separate proposal from authority - -Agents return bounded typed values and candidate file trees. Trusted host code -reconstructs destinations, validates content against independent ground -truth, and performs writes. - -## Bound work and classify failures - -Turn limits, call timeouts, repair limits, task timeouts, and -provider-enforced spend caps create practical bounds. Expected work failures -produce useful GitHub messages; platform failures fail the Workflows task for -an operator. - -## State residual risk plainly - -Layered controls reduce authority; they do not make arbitrary code and model -execution perfectly safe. See -[Architecture](architecture.md#residual-risks) for this project's accepted -risks. diff --git a/docs/operations.md b/docs/operations.md deleted file mode 100644 index 3ffbd33..0000000 --- a/docs/operations.md +++ /dev/null @@ -1,140 +0,0 @@ -# Operations - -Use this guide after completing the [Quickstart](quickstart.md). - -## Follow a run - -While a task is running, its issue or pull request has `factory:working`. The -factory's status comment shows progress, the current run ID, and final links. - -Find that run under `factory-workflows` in the Render Dashboard to inspect -logs, inputs, and its result. - -Every task run ends with a `run.finished` JSON line, and every agent run with -an `agent.finished` line, in the `factory-workflows` logs. Search for them, or -send them to a log stream: - -```bash -render logs -r --text agent.finished -``` - -`run.finished` carries the task, state, outcome, and duration. `agent.finished` -carries the agent, model, turns, token usage, reported cost, and, when an agent -is stopped, why. A run stopped by a newer run or by `/factory cancel` gets its -`run.finished` line from whatever stopped it, because Render kills a cancelled -run before it can write one. Task state remains in Render Workflows; GitHub remains the -source of truth for issues, specifications, pull requests, and reviews. - -## Cancel or restart - -To cancel, post this as the first line of an issue or pull request comment: - -```text -/factory cancel -``` - -The commenter must have `minimumPermission`. Cancellation stops the run the -status comment names and terminates its Sandbox. Render kills a cancelled run -within about a second. - -To restart a task, remove its trigger label and apply it again. - -## Check a deployment - -From a configured local checkout, this checks the credentials and services -named by your local `.env`: - -```bash -npm run doctor -``` - -It does not inspect environment variables stored only on Render. To verify -each deployed service's actual environment, run `doctor` from its Render -Shell: - -```bash -# factory-gateway -npm run doctor -- --service gateway - -# factory-workflows -npm run doctor -- --service workflows -``` - -`doctor` is read-only, does not print secret values, and exits nonzero when a -required check fails. - -## Understand snapshots - -Each configured repository has a dependency-installed Sandbox snapshot. A -push to its base branch refreshes that snapshot. If no snapshot exists, the -first task to run builds one inline. - -Configuration and code changes take effect at different times: - -- `factory.config.ts`, application code, and prompt changes require a deploy. -- `app/runner/` and `sandbox/` changes require a snapshot refresh. -- A changed `install` command triggers reinstallation when a Sandbox opens; - the next base-branch push bakes it into a new snapshot. -- Added repositories and renamed labels require `npm run setup:github`. -- GitHub permission or event changes require updating the App and rerunning - `doctor`. - -Snapshot refresh keeps the previous usable version when a new build fails. - -## Troubleshoot - -### No status comment - -Inspect the GitHub App's recent webhook deliveries. Confirm: - -- the webhook is active; -- its URL ends in `/webhooks/github`; -- the repository appears in `factory.config.ts`; and -- the triggering actor has `minimumPermission`. - -### Webhook delivery returns 401 - -`GITHUB_WEBHOOK_SECRET` differs between GitHub and `factory-gateway`. - -### Webhook delivery returns 502 - -The gateway could not start the task. Check `RENDER_API_KEY`, -`RENDER_WORKFLOW_SLUG`, and the Workflows service. - -### Snapshot build fails - -Inspect the `refresh-snapshot` run or the first task run. Common causes are: - -- an invalid install command; -- installation changing tracked files; -- a missing App installation; or -- a Sandbox verification failure. - -### Agent cannot start - -Check the selected provider key in `factory-workflows`, then run: - -```bash -npm run doctor -- --service workflows -``` - -### Publishing refuses a change - -Inspect task logs for rejected workflow files, unsafe paths, symlinks, -submodules, file-count or size limits, branch ownership, or a tree-SHA -mismatch. - -## Capacity, cost, and risk - -- Sandbox API calls share the workspace's request limit. Expect a few - concurrent tasks rather than dozens. -- The factory has no dollar budget. Use a dedicated, provider-capped model - key. -- Anyone with `minimumPermission` can start model spend. -- PR Reviewer executes the labeled PR's configured verification commands. -- Sandboxes use allow-all egress. -- Local filesystem writes on Render services are ephemeral; durable state - belongs in GitHub, Workflows, or Sandbox snapshots. - -Review [Architecture](architecture.md#residual-risks) before using private -code or untrusted pull requests. diff --git a/docs/quickstart.md b/docs/quickstart.md index 16dc83f..cdd5f04 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,64 +1,64 @@ # Quickstart -The fastest path is to ask a coding agent to follow the -[Setup runbook](runbooks/setup.md). It gives the agent exact commands, -verification criteria, recovery steps, and explicit checkpoints for actions -only you should take. +Use a coding agent for the first deployment. The +[Setup runbook](runbooks/setup.md) gives it the commands, checks, recovery +steps, and points where you must take over. -## What you need +## Before you start + +Prepare: - a fork of this repository; - a Render workspace with Sandboxes access in Oregon; - a Render API key; -- a dedicated, spend-capped Anthropic API key; +- a dedicated Anthropic API key with a provider-enforced spend limit; - admin access to each target GitHub repository; and - the Node.js version in [`.node-version`](../.node-version). -The Blueprint creates paid Render resources: a Starter web service and a -Workflows service. Sandbox and model usage are charged separately. +The Blueprint creates paid resources: a Starter web service and a Workflows +service. Sandbox and model usage are billed separately. -## What the setup agent will do +## Start the setup -The agent will: +Open your fork in a coding agent and use this prompt: -1. inspect the repository and run its checks; -2. configure target repositories in `factory.config.ts`; -3. guide creation and installation of a GitHub App; -4. guide deployment of the Render Blueprint; -5. connect the webhook and create labels; -6. verify both deployed services with `doctor`; and -7. run one approved Triage smoke test. +```text +Set up software-factory by following docs/runbooks/setup.md. +Configure with base branch . +Stop at every human checkpoint. Do not print or expose secrets. +``` -The agent must stop for your confirmation before: +The agent will configure the fork, create a GitHub App, deploy the Blueprint, +connect the webhook, verify the deployment, and guide one Triage smoke test. -- committing or pushing configuration unless already authorized; -- creating the GitHub App; -- installing it on repositories; -- creating paid Render resources; -- entering secrets; -- activating the webhook; or -- creating and labeling a test issue. +You must take over for: -Never paste API keys, private keys, or webhook secrets into chat. Enter them -directly in `.env` or the Render Dashboard. +- approval to commit or push; +- GitHub App creation and installation; +- paid Render resource creation; +- secret entry; +- webhook activation; and +- creation or labeling of the smoke-test issue. -## Start +Enter secrets directly in `.env`, GitHub, or the Render Dashboard. Never put +them in chat, issues, commits, or command output. -Open the fork in your coding agent and ask: +## Definition of done -```text -Set up Software Factory by following docs/runbooks/setup.md. -Configure using base branch . -Stop at every human checkpoint and do not expose secrets. -``` +Setup is complete when: -Setup is complete only when both deployed `doctor` checks pass and applying -`factory:triage` produces an App-authored status comment. +- `factory-gateway` and `factory-workflows` are live; +- the GitHub App is installed and its webhook is active; +- gateway `doctor` passes in its Render Shell and Workflows `doctor` passes + locally with the deployed service's values; and +- applying `factory:triage` produces an App-authored status comment and a + useful outcome. -## Continue +## Next steps -- [Setup runbook](runbooks/setup.md): complete agent execution protocol. -- [Configuration](configuration.md): repositories, models, labels, and limits. -- [Operations](operations.md): logs, cancellation, snapshots, and troubleshooting. -- [Architecture](architecture.md): state ownership and trust boundaries. -- [Contributing](../CONTRIBUTING.md): local development and implementation changes. +- [Configuration](configuration.md) — change settings, follow or cancel runs, + check a deployment, and manage snapshots. +- [Architecture](architecture.md) — understand state, coordination, and + security. +- [Extension runbook](runbooks/extend.md) — add agents, Workflows tasks, or + GitHub triggers. diff --git a/docs/runbooks/extend.md b/docs/runbooks/extend.md new file mode 100644 index 0000000..4169308 --- /dev/null +++ b/docs/runbooks/extend.md @@ -0,0 +1,261 @@ +# Extension runbook + +Use this runbook when adding an agent, a Render Workflows task, or a GitHub +trigger. It is written for coding agents working in a fork of +software-factory. + +Read [Architecture](../architecture.md) first. Do not change the authority +boundary as part of an extension: agents propose typed values or file changes; +code in the Workflows service selects targets, validates results, and writes +to GitHub. + +## Before editing + +Inspect: + +- `factory.config.ts`; +- `app/agents/`; +- `app/workflows/`; +- `app/schemas/`; +- `app/gateway/routes.ts`; +- `app/config/index.ts`; +- `scripts/lib/requirements.ts`; and +- the closest existing test. + +Run `npm run check` before changing code. Record any existing failure and +preserve unrelated work. + +Decide which change you need: + +- **Agent only:** a new prompt, tool set, and typed result used by an existing + task. +- **Claiming task:** work attached to one issue or pull request that must stop + when another run takes over. +- **Plain task:** maintenance work without an issue or pull request claim, + such as snapshot refresh. +- **GitHub trigger:** a new authenticated event, label, or command that starts + a task. + +Do not add a task when an existing task can call the new agent and own the +result. + +## Add an agent + +### 1. Define the output + +Add or reuse a schema in `app/schemas/`. Build every output object with +`z.strictObject`: + +```typescript +export const Finding = z.strictObject({ + summary: z.string().min(1).max(4000), + paths: z.array(z.string().min(1)).max(100), +}); + +export type Finding = z.infer; +``` + +Bound strings and arrays. The model sees this schema through the `submit` +tool, and the Workflows host validates the result again. + +Export the schema from `app/schemas/index.ts`. + +### 2. Add the prompt and definition + +Create: + +```text +app/agents// + index.ts + prompt.md +``` + +Define the agent: + +```typescript +export const finder = defineAgent({ + id: "finder", + prompt: "finder/prompt.md", + model: "medium", + tools: ["read", "ls", "bash"], + output: Finding, + maxTurns: 40, +}); +``` + +The definition fixes the agent's authority: + +- `id` must be unique and stable; +- `prompt` is relative to `app/agents/`; +- `model` is `medium` or `large`; +- `tools` may include `read`, `ls`, `bash`, `edit`, and `write`; +- `output` is a bounded strict Zod object; +- `maxTurns` is a positive model-call limit; and +- `thinking` is optional. + +Grant `edit` or `write` only when the agent must change the checkout. The +read-only shell policy in `app/runner/tools.ts` guides read-only agents; it is +not a security boundary. + +The prompt should explain the job, evidence requirements, limits, and when to +call `submit`. Put repository-, issue-, and run-specific data in the message +passed to `runAgent`, not in the prompt. + +Export the agent from `app/agents/index.ts`. + +### 3. Call the agent + +Call the agent inside a Sandbox owned by the task: + +```typescript +const result = await runAgent(finder, message, claim.agentOptions(sandbox)); +``` + +Use the typed `result.output`. Do not parse assistant text or accept a target +chosen by the model. + +### 4. Test the authority + +Add the agent to `tests/agents-catalog.test.ts`. Update the expected writer +list only if write tools are intentional. + +Add task-level tests for the user-visible result and meaningful failures. Use +fakes; tests must not call GitHub, Render, or a model provider. + +## Add a claiming task + +Use a claiming task for work on one issue or pull request. The status comment +records the run that owns the subject and the Sandbox that cancellation must +terminate. + +### 1. Define the task contract + +1. Add a name to `TASKS` in `app/config/index.ts`. +2. Add a bounded input schema to `app/schemas/tasks.ts`. +3. Export the schema from `app/schemas/index.ts`. + +Task input must carry trusted routing data from the authenticated webhook: +repository, installation, actor, subject number, and any commit SHA the task +must pin. Do not accept these values from agent output. + +### 2. Declare the task + +Create `app/workflows//index.ts`: + +```typescript +export const runExample = claimingTask( + { + name: TASKS.example, + title: "Example", + input: ExampleInput, + subject: (input) => ({ kind: "issue", number: input.issue }), + timeoutSeconds: 3600, + }, + async (claim, input, deps): Promise => { + // Code in the Workflows service owns all external writes. + return { outcome: "completed", summary: "completed the example task." }; + }, +); +``` + +`claimingTask()` validates input, resolves the configured repository, checks +the actor's permission, acquires the status-comment claim, and records the +final outcome. + +Claiming tasks do not retry automatically. A retry can repeat model spend or +durable writes; a person starts a fresh run instead. + +### 3. Use the claim correctly + +- Call `claim.progress()` before a major step. +- Open Sandboxes only through `withSandbox`. +- Call `claim.recordSandbox(sandbox)` immediately after opening one. +- Pass `claim.agentOptions(sandbox)` to `runAgent`. +- Call `claim.checkpoint()` before a durable effect that is not already + guarded by the claim helper. +- Throw `ExpectedFailure` only for an expected message safe to show on + GitHub. +- Return a short snake_case `outcome`, user-facing `summary`, and relevant + links. + +Only code in the Workflows service may call GitHub write methods. Format model +text through `app/format/` before publishing it. + +### 4. Register the task + +Import the task module from `app/workflows/index.ts`. Render registers tasks +when the module loads. + +Keep task inputs and results small and JSON-serializable. + +## Add a plain Workflows task + +Use the Render SDK's `task()` directly only when the work has no issue or pull +request claim. `app/workflows/refresh-snapshot.ts` is the reference. + +A plain task must still: + +- validate raw input at entry with Zod; +- resolve repositories through trusted configuration; +- set an explicit timeout and intentional retry policy; +- bound external work; +- return a small JSON-serializable result; and +- clean up every Sandbox it creates. + +Do not use a plain task to bypass permission checks or status-comment +ownership for issue or pull request work. + +## Add a GitHub trigger + +The gateway route is a pure mapping from an authenticated delivery to a +validated task input. + +1. Add the task name and input schema first. +2. Update `route()` in `app/gateway/routes.ts`. +3. Derive repository, installation, actor, issue or pull request, and commit + from the authenticated payload. +4. Use the GitHub delivery ID as the task idempotency key. For content-addressed + maintenance work, use a deterministic key such as `repo@sha`. +5. Update `REQUIRED_EVENTS`, `REQUIRED_PERMISSIONS`, or `LABELS` in + `scripts/lib/requirements.ts`. +6. Add route tests for the start case, ignored cases, and invalid input. +7. Rerun `npm run setup:github` after deployment. + +Never put GitHub credentials or model calls in `app/gateway/`. The gateway +authenticates and routes; it does not perform the task. + +Every GitHub write must create, update, or reuse an App-authored artifact +marker. Workflows idempotency alone expires and cannot protect a permanent +GitHub effect. + +## Verify + +Run: + +```bash +npm run check +``` + +Use the additional checks in +[Contributing](../../CONTRIBUTING.md#verify-changes) when relevant. + +Before handing off, confirm: + +- all external and cross-process inputs are validated; +- the new agent has the minimum tools it needs; +- model output cannot select an external write target; +- the gateway still imports no GitHub auth, Sandbox, agent, or workflow code; +- `app/runner/` still imports nothing from the rest of `app/`; +- claiming tasks record their Sandbox before running an agent; +- tests cover the user-visible success and important failures; and +- documentation and setup requirements changed when the App contract changed. + +## Deploy the extension + +- Prompt, agent, schema, or Workflows changes require a Workflows deploy. +- Gateway route changes require a gateway deploy. +- App permission, event, or label changes require `npm run setup:github`. +- Changes under `app/runner/` or `sandbox/` also require a snapshot refresh for + every configured repository. + +State the required deploy and refresh actions in the handoff. diff --git a/docs/runbooks/setup.md b/docs/runbooks/setup.md index 27634f7..2fa38b3 100644 --- a/docs/runbooks/setup.md +++ b/docs/runbooks/setup.md @@ -1,70 +1,54 @@ # Setup runbook -Use this runbook when helping someone deploy Software Factory for the first -time. Follow it in order. Do not claim setup is complete until both deployed -services pass `doctor` and a labeled issue produces a factory status comment. +Use this runbook to deploy software-factory from a fork and connect its first +repository. Follow the phases in order. Stop at every **Human checkpoint**. -## Goal +Setup is complete only after gateway `doctor` passes in its Render Shell, +Workflows `doctor` passes locally with the deployed values, and a Triage smoke +test reaches an App-authored status comment. -Deploy Software Factory from the user's fork, connect it to one GitHub -repository, and complete one Triage run. +## Inputs -## Success criteria - -Setup is complete only when: - -- `factory-gateway` and `factory-workflows` are live; -- the GitHub App is installed on every configured repository; -- the App webhook is active and points to the gateway; -- the factory labels exist; -- deployed `doctor` checks pass for the gateway and Workflows services; and -- applying `factory:triage` produces an App-authored status comment. - -## Required inputs - -Collect or infer these before editing: +Collect: - fork owner and repository; - target repository in `owner/name` form; - target base branch; - deterministic dependency install command, if needed; - verification commands; -- GitHub App owner: the user or an organization; +- GitHub App owner: a user or organization; - Render workspace with Sandboxes access in Oregon; and -- whether the default Anthropic models are acceptable. +- confirmation that the default Anthropic models are acceptable. -The user must provide these secrets directly to local environment files or -Render—not in chat, issues, commits, or command output: +The user must enter these secrets directly in local files, GitHub, or Render: - Render API key; -- model provider API key; and -- generated GitHub App private key and webhook secret. +- Anthropic API key; +- GitHub App private key; and +- webhook secret. + +Never ask the user to paste a secret into chat. Never print `.env`. -## Non-negotiable constraints +## Rules - Deploy from the user's fork, not `render-examples/software-factory`. -- Use the Node.js version pinned in `.node-version`. -- Never commit `.env` or print secret values. -- Do not weaken GitHub App permissions or security invariants to make setup - pass. -- Do not change existing GitHub App permissions, Render resources, or paid - plans without explaining the effect and receiving approval. -- Do not create a test issue or apply a label without the user's approval. -- Stop at every **Human checkpoint** until the user confirms completion. +- Use the Node.js version in `.node-version`. - Preserve unrelated working-tree changes. +- Do not guess repository install or verification commands. +- Do not weaken App permissions or security checks to make setup pass. +- Ask before committing, pushing, creating paid resources, changing external + settings, creating issues, or applying labels. -## Phase 1: Inspect and preflight +## Phase 1: Preflight -### Agent action - -Confirm the current directory is the Software Factory root and inspect: +Confirm that the current directory is the software-factory root. Inspect: - `.node-version`; - `package.json`; - `factory.config.ts`; - `render.yaml`; - `.env.example`; and -- the current Git status. +- Git status. Run: @@ -74,27 +58,13 @@ npm ci npm run check ``` -### Pass condition - -- Node satisfies the pinned project version. -- Dependencies install successfully. -- Lint, type checking, and tests pass. -- Existing unrelated changes have been identified and left untouched. - -### Recovery - -- Wrong Node version: install or select the version in `.node-version`, then - rerun the phase. -- `npm ci` failure: report the failing command and relevant error; do not - replace it with a lockfile-changing install. -- Existing test failure: distinguish it from changes made during setup before - continuing. - -## Phase 2: Configure one repository +Continue only when the pinned Node version is active, dependencies install, +checks pass, and unrelated changes are accounted for. Do not replace a failed +`npm ci` with a lockfile-changing install. -### Agent action +## Phase 2: Configure repositories -Edit `factory.config.ts` so `repos` contains the intended targets: +Replace the example `repos` entry in `factory.config.ts`: ```typescript repos: { @@ -106,16 +76,17 @@ repos: { }, ``` -Rules: +Requirements: -- The key must be the exact GitHub `owner/name`. -- `baseBranch` must already exist. -- `install` must be deterministic and must not change tracked files. -- `verify` must contain the commands required before publishing. Use an empty - list only when skipping command-based verification is intentional. +- use the exact GitHub `owner/name`; +- name an existing base branch; +- use an install command that is deterministic and leaves tracked files + unchanged; and +- list every command required before publishing. -Keep the default labels, permission, limits, and Sandbox timeout unless the -user requested a change. +Use `verify: []` only when the user explicitly wants no command-based +verification. Keep the default labels, permission, limits, and Sandbox timeout +unless the user requested changes. Run: @@ -124,46 +95,28 @@ npm run typecheck git diff --check ``` -### Pass condition +Use [Configuration](../configuration.md) for field semantics. -- Configuration parses and type-checks. -- The diff contains only the intended repository and explicitly requested - behavior changes. +## Phase 3: Publish the configured fork -### Recovery +Confirm that the Git remote points to the user's fork. Show the configuration +diff and check results. -Use [Configuration](../configuration.md) for field semantics. If install or -verification commands are unknown, stop and ask the user; do not guess -project-specific commands when the target repository does not establish -them. +### Human checkpoint: commit and push -## Phase 3: Save the fork +Ask the user to confirm: -### Agent action +- the destination fork and branch; +- the configured repositories and commands; and +- permission to commit and push, if not already granted. -Confirm the repository remote points to the user's fork. Present the -configuration diff and verification result. - -Commit and push only when the user has asked the agent to do so. The Render -Blueprint must be able to read the committed configuration from the fork. - -### Human checkpoint: fork ownership and push - -The user must confirm: - -- the destination fork is correct; -- the configuration is safe to deploy; and -- the configured commit is available on the fork's deployment branch. - -### Pass condition - -The fork contains the intended `factory.config.ts` and `render.yaml`. +Do not continue until the deployment branch on the fork contains the intended +`factory.config.ts` and `render.yaml`. The Blueprint reads these files from +that branch. ## Phase 4: Create the GitHub App -### Agent action - -Install local dependencies if needed, then start the manifest flow: +Start the GitHub App manifest flow: ```bash npm run setup:app -- --write-env @@ -175,17 +128,12 @@ For an organization-owned App: npm run setup:app -- --org --write-env ``` -The command starts a local callback server and prints a browser URL. Keep it -running while the user completes the checkpoint. +Keep the command running while the user completes the browser flow. ### Human checkpoint: create the App -Ask the user to: - -1. open the printed local URL; -2. review the App name, repositories, permissions, and events; -3. select **Create on GitHub**; and -4. return when the command reports success. +Ask the user to open the printed URL, review the manifest, and select +**Create on GitHub**. The manifest requests: @@ -193,17 +141,15 @@ The manifest requests: - Metadata: read; and - Issues, Issue comments, Pull requests, and Pushes events. -It does not request Actions, Workflows, or Administration permission. +It does not request GitHub Actions, GitHub Workflows, or Administration +permission. -### Agent verification +After the callback completes, confirm that the command reports the App slug +and ID, prints an installation URL, and writes these values to `.env`: -Confirm the command: - -- reports the App slug and ID; -- prints an installation URL; -- writes `GITHUB_APP_ID`, `GITHUB_APP_PRIVATE_KEY`, and - `GITHUB_WEBHOOK_SECRET` to `.env`; and -- exits successfully. +- `GITHUB_APP_ID`; +- `GITHUB_APP_PRIVATE_KEY`; and +- `GITHUB_WEBHOOK_SECRET`. Verify that Git ignores the file: @@ -211,212 +157,240 @@ Verify that Git ignores the file: git check-ignore .env ``` -Do not display `.env`. +## Phase 5: Install the App -### Pass condition +### Human checkpoint: repository access -The GitHub App exists, `.env` is ignored, and the three generated values are -available locally without being exposed. +Ask the user to open the installation URL and install the App on every +repository in `factory.config.ts`. If an organization requires approval, wait +for an owner to approve the installation. -## Phase 5: Install the GitHub App +Continue only after the user confirms access to every configured repository. -### Human checkpoint: repository installation +## Phase 6: Deploy the Blueprint -Ask the user to open the installation URL printed by `setup:app` and install -the App on every repository in `factory.config.ts`. +[`render.yaml`](../../render.yaml) creates two paid resources in the +`production` environment of a `software-factory` project: -When organization approval is required, stop until an organization owner has -approved the installation. +- `factory-gateway`, a Starter web service in Oregon; and +- `factory-workflows`, a Workflows service in Oregon. -### Pass condition +### Human checkpoint: resources and secrets -The user confirms installation on every exact configured repository. +Explain the resources and expected charges. Ask the user to create the +Blueprint from the configured fork and branch. -## Phase 6: Deploy the Render Blueprint +Enter these values for `factory-gateway`: -### Agent action +- `GITHUB_WEBHOOK_SECRET`; +- `RENDER_API_KEY`. -Explain that [`render.yaml`](../../render.yaml) creates paid resources in -Oregon: +Enter these values for `factory-workflows`: -- `factory-gateway`, a Starter web service; and -- `factory-workflows`, a Workflows service. +- `RENDER_API_KEY`; +- `RENDER_WORKSPACE_ID`; +- `GITHUB_APP_ID`; +- `GITHUB_APP_PRIVATE_KEY`; +- `ANTHROPIC_API_KEY`. -The user should create the Blueprint from the configured fork and deployment -branch. +The workspace ID starts with `tea-`. Copy GitHub values from local `.env` +without showing them in chat or command output. -### Human checkpoint: paid resources and secrets +Continue when both services have deployed and the gateway is live. -Before creation, ask the user to confirm the resources and expected charges. -Then ask them to enter every `sync: false` value in the Render Dashboard. +## Phase 7: Configure GitHub -For `factory-gateway`: +Ask for the public gateway URL. It is not a secret. Run: -- `GITHUB_WEBHOOK_SECRET` -- `RENDER_API_KEY` +```bash +npm run setup:github -- \ + --gateway-url https://.onrender.com +``` -For `factory-workflows`: +The command: -- `RENDER_API_KEY` -- `RENDER_WORKSPACE_ID` -- `GITHUB_APP_ID` -- `GITHUB_APP_PRIVATE_KEY` -- `ANTHROPIC_API_KEY` +- sets the App webhook URL to + `https://.onrender.com/webhooks/github`; +- preserves the generated webhook secret; and +- creates `factory:triage`, `factory:spec-approved`, + `factory:pr-review`, and `factory:working`. -The user should copy GitHub values from the local `.env` without pasting them -into chat. The Render workspace ID starts with `tea-`. +It is safe to rerun. Continue only when it reports no failed repository or +webhook operation. -### Pass condition +## Phase 8: Activate the webhook -Both services are created, the gateway reports live, and the Workflows -service has deployed successfully. +`setup:github` cannot activate an App webhook through GitHub's API. -### Recovery +### Human checkpoint: webhook activation -- Blank `sync: false` field: set the missing value and redeploy. -- Build failure: inspect the failing service's build log before changing - configuration. -- Port or health failure: confirm the gateway binds to `0.0.0.0:$PORT` and - serves `/ready`. +Ask the user to open the App's **Webhook** settings, confirm that the URL ends +in `/webhooks/github`, enable **Active**, and save without changing the secret. -Use [Operations](../operations.md#troubleshoot) for service-specific failures. +## Phase 9: Run `doctor` -## Phase 7: Connect the webhook and create labels +`doctor` checks the environment in which it runs. It is read-only and does +not print secret values. -### Agent action +### Gateway -Ask the user for the public gateway URL; this URL is not a secret. Run: +Run this in the `factory-gateway` Render Shell: ```bash -npm run setup:github -- \ - --gateway-url https://.onrender.com +npm run doctor -- --service gateway ``` -The command reads GitHub credentials from `.env`. It is idempotent and should: - -- set the App webhook URL to - `https://.onrender.com/webhooks/github`; -- preserve the generated webhook secret; and -- create `factory:triage`, `factory:spec-approved`, - `factory:pr-review`, and `factory:working` in each configured repository. +This checks the gateway's deployed environment. -### Pass condition +### Workflows -The command exits successfully and reports no failed repository or webhook -operation. +Workflows services have no Shell. Run `doctor` locally with the values used by +`factory-workflows`. -### Recovery +Local `.env` already contains the GitHub App values. Ask the user to add: -- Missing credentials: confirm `setup:app --write-env` completed and `.env` - exists without printing it. -- Repository failure: confirm exact `owner/name`, App installation, and the - installing user's repository administration access. -- Wrong gateway URL: use the public HTTPS URL for `factory-gateway`, without - adding `/webhooks/github`; the command adds the path. - -## Phase 8: Activate the webhook +- `RENDER_API_KEY`; +- `RENDER_WORKSPACE_ID`; +- `ANTHROPIC_API_KEY`; and +- `RENDER_WORKFLOW_SLUG`. -GitHub's API does not allow `setup:github` to activate an App webhook. +Use the deployed workflow slug, including any suffix Render added. Copy the +exact value from `factory-gateway`'s deployed `RENDER_WORKFLOW_SLUG` into +local `.env`. -### Human checkpoint: webhook activation +Run: -Ask the user to: +```bash +npm run doctor -- --service workflows +``` -1. open the GitHub App settings; -2. open **Webhook**; -3. verify the URL ends in `/webhooks/github`; -4. enable **Active**; and -5. save without changing the URL or secret. +The warning `no webhook deliveries yet, so the hook may be inactive` is +expected before the smoke test. Do not continue while either command has a +failed check. -### Pass condition +## Phase 10: Smoke-test Triage -The user confirms the webhook is active. +### Human checkpoint: issue and label -## Phase 9: Verify the deployed services +Ask before creating an issue or changing labels. Use a disposable issue in a +configured test repository: -Run `doctor` inside each service's Render Shell so it reads that service's -deployed environment. +```text +Title: software-factory setup smoke test -### Human checkpoint or agent action with Render access +Confirm the repository can be inspected and return a minimal specification. +Do not request a code change. +``` -In `factory-gateway`: +After approval, create and label it in GitHub or with `gh`: ```bash -npm run doctor -- --service gateway +gh issue create \ + --repo \ + --title "software-factory setup smoke test" \ + --body "Confirm the repository can be inspected and return a minimal specification. Do not request a code change." + +gh issue edit \ + --repo \ + --add-label factory:triage ``` -In `factory-workflows`: +The first run may spend several minutes building the repository snapshot. + +The smoke test passes when: + +- the issue receives an App-authored status comment; +- `factory:working` appears while the task runs; +- Triage ends with questions, a specification, or an intentional decline; + and +- the status comment shows the final outcome or a useful failure. + +Optionally confirm that the Workflows logs contain `run.finished`: ```bash -npm run doctor -- --service workflows +render logs -r --text run.finished ``` -`doctor` is read-only and does not print secret values. +## Troubleshooting setup -### Pass condition +### A service does not deploy -Both commands exit successfully with every required check passing. +- For a missing `sync: false` value, set it and redeploy. +- For a build failure, inspect the failing service's build log before changing + configuration. +- For a gateway health failure, confirm that it binds to `0.0.0.0:$PORT` and + serves `/ready`. -### Recovery +### `doctor` fails -- Gateway failure: check webhook secret, Render API key, and workflow slug. -- Workflows failure: check Render workspace ID, Render API key, GitHub App - credentials, provider key, and registered tasks. -- App failure: compare required permissions and events with Phase 4. -- Label or installation failure: rerun `npm run setup:github` after fixing - App access. +- Gateway: check the webhook secret, Render API key, and workflow slug. +- Workflows: check the workspace ID, Render API key, App credentials, + provider key, and registered tasks. +- `No tasks found`: use the deployed workflow slug, not an assumed service + name. +- App or label failure: fix App access and rerun `npm run setup:github`. -Do not proceed to the smoke test while either service fails `doctor`. +### No status comment appears -## Phase 10: Smoke-test Triage +Open the App's **Advanced** settings and inspect **Recent Deliveries**. -### Human checkpoint: external write +- No delivery: activate the webhook and verify its URL. +- `401`: GitHub and `factory-gateway` have different webhook secrets. +- `422`: inspect the gateway log entry + `authenticated delivery produced an invalid task input`. +- `502`: inspect `could not start task`; then check `RENDER_API_KEY`, + `RENDER_WORKFLOW_SLUG`, and the Workflows deployment. +- `200` with `{"ignored":""}`: the event did not start a task. Common + reasons are `unconfigured repo`, `unhandled label`, `issue not in triage`, + `bot sender`, `pull request not open`, and `not base branch`. +- `202`: the task started; inspect it under `factory-workflows`. -Ask permission before creating an issue or modifying labels. Use a small, -disposable issue in a configured test repository. +If a task starts but the comment says it did not run, the triggering actor +lacks `minimumPermission`. -Suggested issue: +### A snapshot build fails -```text -Title: Software Factory setup smoke test +Inspect the `refresh-snapshot` run or the first task run. Check the install +command, tracked-file changes after install, App installation, and Sandbox +verification. -Confirm the repository can be inspected and return a minimal specification. -Do not request a code change. -``` +### An agent submits an invalid result -Create the issue and apply `factory:triage`. This can be done in GitHub or, -when the user has approved CLI writes: +The runner accepts three attempts. Search Workflows logs for the last schema +errors and submitted field names: ```bash -gh issue create \ - --repo \ - --title "Software Factory setup smoke test" \ - --body "Confirm the repository can be inspected and return a minimal specification. Do not request a code change." +render logs -r --text invalid_submission +``` -gh issue edit \ - --repo \ - --add-label factory:triage +Repeated errors for one agent usually require a prompt or schema change. + +### An agent cannot start + +Check the selected provider key on `factory-workflows`, then run: + +```bash +npm run doctor -- --service workflows ``` -The first run may take several minutes while it builds the repository -snapshot. +### Publishing refuses a change -### Pass condition +Inspect the task logs. Publishing rejects workflow files, unsafe paths, +symlinks, submodules, file-count and file-size limits, branches with human +commits, and candidate trees GitHub cannot reproduce by SHA. -- The issue receives an App-authored status comment. -- `factory:working` appears while the task is running. -- Triage ends with questions, a specification, or an intentional decline. -- The status comment includes the final outcome or a useful failure. +### The status comment remains on `running` -If the status comment never appears, use -[Operations](../operations.md#no-status-comment). +Post `/factory cancel` as the first line of a comment. This clears stale +status and terminates the recorded Sandbox. Reapplying the trigger label +starts a new run. -## Phase 11: Handoff +## Handoff -Report setup using this format without including secrets: +Report: ```text -Software Factory setup: complete +software-factory setup: complete Fork: Configured repositories: @@ -430,17 +404,8 @@ Local uncommitted changes: Follow-up actions: ``` -If any success criterion is missing, report setup as **incomplete** and name -the exact checkpoint, evidence, and next action. - -## Ready to extend - -After setup: +If any required check is missing, report setup as **incomplete**. Name the +failed phase, evidence, and next action. -- add or change repositories, labels, models, and limits through - [Configuration](../configuration.md); -- diagnose live behavior through [Operations](../operations.md); -- read [Architecture](../architecture.md) before changing authority, - credentials, coordination, or publishing; and -- use [Contributing](../../CONTRIBUTING.md) to add agents, tasks, and GitHub - triggers. +For later changes, use [Configuration](../configuration.md) or the +[extension runbook](extend.md). diff --git a/render.yaml b/render.yaml index 5e2cd09..cae47ad 100644 --- a/render.yaml +++ b/render.yaml @@ -1,43 +1,47 @@ -# Software Factory: two services. +# software-factory: two services in a software-factory project. # - factory-gateway: verifies GitHub webhooks and starts the matching Workflows task. # - factory-workflows: runs every task (agents, snapshot refresh, cancel). It has no `repo` field, so it # deploys from the same repository as this Blueprint, including forks. # Run and agent telemetry is logged as JSON lines; find it in each service's logs or a log stream. -services: - - type: web - name: factory-gateway - region: oregon - runtime: node - plan: starter - buildCommand: npm ci - startCommand: npm run start:gateway - healthCheckPath: /ready - envVars: - - key: GITHUB_WEBHOOK_SECRET - sync: false - - key: RENDER_API_KEY - sync: false - - key: RENDER_WORKFLOW_SLUG - fromService: - type: workflow - name: factory-workflows - property: slug +projects: + - name: software-factory + environments: + - name: production + services: + - type: web + name: factory-gateway + region: oregon + runtime: node + plan: starter + buildCommand: npm ci + startCommand: npm run start:gateway + healthCheckPath: /ready + envVars: + - key: GITHUB_WEBHOOK_SECRET + sync: false + - key: RENDER_API_KEY + sync: false + - key: RENDER_WORKFLOW_SLUG + fromService: + type: workflow + name: factory-workflows + property: slug - - type: workflow - name: factory-workflows - region: oregon - runtime: node - buildCommand: npm ci - startCommand: npm run start:workflows - envVars: - - key: RENDER_API_KEY - sync: false - - key: RENDER_WORKSPACE_ID - sync: false - - key: GITHUB_APP_ID - sync: false - - key: GITHUB_APP_PRIVATE_KEY - sync: false - - key: ANTHROPIC_API_KEY - sync: false + - type: workflow + name: factory-workflows + region: oregon + runtime: node + buildCommand: npm ci + startCommand: npm run start:workflows + envVars: + - key: RENDER_API_KEY + sync: false + - key: RENDER_WORKSPACE_ID + sync: false + - key: GITHUB_APP_ID + sync: false + - key: GITHUB_APP_PRIVATE_KEY + sync: false + - key: ANTHROPIC_API_KEY + sync: false diff --git a/scripts/lib/manifest.ts b/scripts/lib/manifest.ts index d19b908..a9c9fca 100644 --- a/scripts/lib/manifest.ts +++ b/scripts/lib/manifest.ts @@ -27,7 +27,7 @@ export function buildManifest(opts: { name: string; redirectUrl: string }): Mani return { name: opts.name, url: HOMEPAGE, - description: "Software Factory: labeled issue to spec to draft PR to review, on Render.", + description: "software-factory: labeled issue to spec to draft PR to review, on Render.", hook_attributes: { url: PLACEHOLDER_WEBHOOK_URL, active: false }, redirect_url: opts.redirectUrl, public: false, diff --git a/tests/agents-catalog.test.ts b/tests/agents-catalog.test.ts index a5a3882..b3f8989 100644 --- a/tests/agents-catalog.test.ts +++ b/tests/agents-catalog.test.ts @@ -15,4 +15,14 @@ describe("agent authority", () => { expect(writers).toEqual(["implementer"]); expect(agents.every((agent) => agent.tools.length > 0)).toBe(true); }); + + it("rejects unknown fields in every object an agent can submit, so a misspelled field is named in the error", () => { + const looseObjects = (schema: unknown, path: string): string[] => { + if (typeof schema !== "object" || schema === null) return []; + const node = schema as Record; + const own = node.type === "object" && node.additionalProperties !== false ? [path || "(root)"] : []; + return [...own, ...Object.entries(node).flatMap(([key, value]) => looseObjects(value, `${path}/${key}`))]; + }; + for (const agent of agents) expect(looseObjects(agent.outputJsonSchema, ""), agent.id).toEqual([]); + }); }); diff --git a/tests/cancel.test.ts b/tests/cancel.test.ts index 9286c5b..8d0cb62 100644 --- a/tests/cancel.test.ts +++ b/tests/cancel.test.ts @@ -91,6 +91,20 @@ describe("cancel task", () => { expect(events[1]).toMatchObject({ traceId: "trn-cancel", runId: "trn-cancel", repo: REPO }); }); + it("clears a status left running by a run that already stopped, and terminates its leftover sandbox", async () => { + const { gh, comments, labels, add } = fakeGitHub("write"); + const marker = renderMarker("status", { runId: "trn-dead", stage: "triage", title: "Triage", state: "running", sandboxId: "sbx-dead" }); + add(BOT, `running\n\n${marker}`); + const runs = fakeRuns({}); + const result = await cancelRun(ctx, { repo: REPO, installationId: 5, number: 12, kind: "issue", actor: "alice" }, { gh, runs }); + expect(result).toEqual({ cancelled: false, runId: "trn-dead", reason: "already_stopped" }); + expect(runs.cancelTaskRun).not.toHaveBeenCalled(); + expect(runs.terminateSandbox).toHaveBeenCalledWith("sbx-dead"); + expect(comments).toHaveLength(1); + expect(comments[0]?.body).toContain("**Triage** was cancelled: cleared by @alice; the run had already stopped without finishing."); + expect(labels.has(WORKING_LABEL)).toBe(false); + }); + it("refuses an actor without permission and leaves the run alone", async () => { const { gh, comments, add } = fakeGitHub("read"); add(BOT, renderMarker("status", { runId: "trn-1", stage: "triage", state: "running" }));