Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/prompt-contracts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,9 @@ jobs:
run: cargo test --test prompt_contract_tests --test prompt_eval_contract_tests

- name: Install gh-aw
uses: github/gh-aw-actions/setup-cli@ba6380cc6e5be5d21677bebe04d52fb48e3abec7 # v0.81.6
uses: github/gh-aw-actions/setup-cli@6aab9e5b5c91c615506061f09bedd81a23babe3c # v0.86.2
with:
version: v0.81.6
version: v0.86.2

- name: Compile prompt evaluator strictly
run: |
Expand Down
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ fail-closed and only pauses when the agent actually proposed a reviewed output.
│ │ │ └── integration_tests.rs # Import resolution + merge integration tests
│ │ ├── extensions/ # CompilerExtension trait and infrastructure extensions
│ │ │ ├── mod.rs # Trait, Extension enum, collect_extensions(), re-exports
│ │ │ ├── container_runtime.rs # Typed Docker runtime config shared by MCPG stdio servers (Mount/Network/Tmpfs/AddHost/ContainerUser, ContainerRuntimeConfig)
│ │ │ ├── ado_aw_marker.rs # Always-on metadata marker extension (emits # ado-aw-metadata JSON)
│ │ │ ├── github.rs # Always-on GitHub MCP extension
│ │ │ ├── safe_outputs.rs # Always-on SafeOutputs MCP extension
Expand Down Expand Up @@ -623,7 +624,7 @@ the directive and it's inert at runtime.
To review the generated shell as ordinary files:

```bash
cargo run -- export-bash-scripts --out /tmp/ado-aw-shell
cargo run -- export-bash-scripts --output /tmp/ado-aw-shell
```

### Markdown-only smoke suite
Expand Down
25 changes: 20 additions & 5 deletions docs/ado-script.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ pipeline** as runtime helpers. Today it produces the following shipped bundles:
the target tip at depth 1. Cross-org repositories are partitioned into a
separate trusted credential scope and passed with validated
organization/project/repository coordinates. The checkout remote must match
those coordinates exactly before the Bearer reaches REST or git; mismatch or
those coordinates exactly before attempting REST or git; mismatch or
preparation failure stops the trusted task before Agent/executor execution.
Same-org per-dir failures remain isolated warnings. The bearer remains
shell-local or in masked `SYSTEM_ACCESSTOKEN` env and spawned-git
Expand Down Expand Up @@ -207,7 +207,7 @@ Resolution is single-pass: nested markers inside an inlined body are not
re-expanded.

The bundle lives at `import.js` and ships in the same
`ado-script.zip` release asset as `gate.js` and the ten
`ado-script.zip` release asset as `gate.js` and the nine
`exec-context-*.js` bundles listed in the workspace layout, so
pipelines download it through the same Agent-job asset flow.
`import.js` uses only the Node standard library, so the ncc bundle is
Expand Down Expand Up @@ -654,6 +654,18 @@ scripts/ado-script/
│ ├── prepare-pr-base/ # prepare-pr-base.js entry point + create-pull-request base-ref fetch/deepen
│ │ ├── index.ts # main(): fetch/deepen target branch + set origin/HEAD so mcp.rs finds a diff base
│ │ └── __tests__/ # unit tests for fetch/deepen + origin/HEAD + benign-failure paths
│ ├── ado-proxy/ # ado-proxy.js entry point + credential-isolated ADO policy proxy
│ │ ├── index.ts # main(): starts the trusted HTTP proxy server
│ │ ├── server.ts # HTTP server, request routing
│ │ ├── policy.ts # scope/capability policy evaluation
│ │ ├── scope.ts # organization-relative current/additional scope index
│ │ ├── catalog.ts # versioned deny-by-default read-operation catalog
│ │ ├── catalog.gen.json # generated by `cargo run -- export-ado-proxy-catalog`
│ │ ├── route.ts / upstream.ts / response.ts / headers.ts / token.ts / config.ts / api-version.ts / ca.ts / log.ts
│ │ └── *.test.ts # per-module unit tests + `proxy.e2e.test.ts`
│ ├── azure-wif-refresh/ # azure-wif-refresh.js entry point + renewable WIF assertion sidecar
│ │ ├── index.ts # main(): rotate a private token file for user-defined stdio MCP servers
│ │ └── __tests__/ # unit tests for rotation and isolation behaviour
│ ├── trigger-e2e/ # test-only: FACT_META gate-spec table + trigger-evaluation E2E scenarios (not a bundle)
│ │ ├── gate-spec.ts # FACT_META mirror of Rust Fact::ALL; drift-guarded by export-fact-catalog + fact-catalog.gen.json
│ │ ├── fact-catalog.gen.json # generated by `cargo run -- export-fact-catalog`; deep-compared by gate-spec.test.ts
Expand All @@ -675,7 +687,9 @@ scripts/ado-script/
├── conclusion.js # ncc bundle output (gitignored)
├── approval-summary.js # ncc bundle output (gitignored)
├── github-app-token.js # ncc bundle output (gitignored)
└── prepare-pr-base.js # ncc bundle output (gitignored)
├── prepare-pr-base.js # ncc bundle output (gitignored)
├── ado-proxy.js # ncc bundle output (gitignored)
└── azure-wif-refresh.js # ncc bundle output (gitignored)
```

The release workflow (`.github/workflows/release.yml`) runs
Expand All @@ -685,8 +699,9 @@ captures every bundle, including `gate.js`, `import.js`,
`exec-context-manual.js`, `exec-context-pipeline.js`,
`exec-context-ci-push.js`, `exec-context-workitem.js`,
`exec-context-schedule.js`, `exec-context-pr-checks.js`,
`exec-context-repo.js`, `conclusion.js`, `approval-summary.js`, and
`github-app-token.js` — into the
`exec-context-repo.js`, `conclusion.js`, `approval-summary.js`,
`github-app-token.js`, `prepare-pr-base.js`, `ado-proxy.js`, and
`azure-wif-refresh.js` — into the
`ado-script.zip` release asset. Pipelines download that asset at
runtime by URL pinned to the compiler's `CARGO_PKG_VERSION`, verify
its SHA-256 against the `checksums.txt` asset, then extract.
Expand Down
14 changes: 14 additions & 0 deletions docs/conclusion.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,20 @@ Conclusion reports deduplicate by rendered work-item title. The job
searches for an existing open work item with the same title; if it finds
one, it appends a comment. Otherwise it creates a new work item.

## Testing

Unit coverage lives in
`scripts/ado-script/src/conclusion/__tests__/index.test.ts` (manifest parsing,
signal rendering, per-tool config).

End-to-end coverage lives in the deterministic executor suite
([`tests/executor-e2e/`](../tests/executor-e2e/README.md)): the `conclusion-*`
scenarios run `ado-aw execute` for a `noop` / `missing-tool` / `missing-data`
signal, then run the compiled `conclusion.js` over the resulting
`safe-outputs-executed.ndjson`, and assert the filed Azure DevOps work item
(title, type, tags, body), the append-on-duplicate-title path, and the
`report-as-work-item: false` opt-out.

## Relationship to gh-aw

This mirrors gh-aw's conclusion-job pattern: a single always-running
Expand Down
4 changes: 2 additions & 2 deletions docs/extending.md
Original file line number Diff line number Diff line change
Expand Up @@ -457,8 +457,8 @@ ShellScript::new(&START_CONTAINER)
### Reviewing the scripts as files

```bash
cargo run -- export-bash-scripts --out /tmp/ado-aw-shell
cargo run -- export-bash-scripts --out /tmp/ado-aw-shell --format json
cargo run -- export-bash-scripts --output /tmp/ado-aw-shell
cargo run -- export-bash-scripts --output /tmp/ado-aw-shell --format json
```

Writes one `.sh` per registered script with a provenance header naming the
Expand Down
10 changes: 7 additions & 3 deletions docs/ir.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ _Part of the [ado-aw documentation](../AGENTS.md)._

ado-aw no longer compiles pipelines by substituting strings into YAML template files. Every production target builds a typed Azure DevOps pipeline IR, resolves graph-level facts, lowers that IR to `serde_yaml::Value`, and serializes once with `serde_yaml::to_string`.

The implementation lives under `src/compile/ir/`. The canonical agentic-pipeline shape (Setup → Agent → Detection → SafeOutputsTeardown, plus an optional always-running Conclusion job when `conclusion:` is configured) lives in `src/compile/agentic_pipeline.rs` and is shared by every target. Per-target wrappers handle only the envelope:
The implementation lives under `src/compile/ir/`. The canonical agentic-pipeline shape (Setup → Agent → Detection → (ManualReview?)Custom_\<tool\>* → SafeOutputs(+SafeOutputs_Reviewed?) → Teardown → Conclusion) lives in `src/compile/agentic_pipeline.rs` and is shared by every target. `ManualReview` is inserted only when a safe output is configured with `require-approval`; the `SafeOutputs`/`SafeOutputs_Reviewed` split occurs only when both gated and non-gated outputs are configured; Conclusion is emitted whenever `safe-outputs:` is configured (there is no separate `conclusion:` front-matter field). Per-target wrappers handle only the envelope:

- `src/compile/standalone_ir.rs`
- `src/compile/onees_ir.rs`
Expand Down Expand Up @@ -224,7 +224,11 @@ The extension trait lives in `src/compile/extensions/mod.rs` and now has exactly
pub trait CompilerExtension {
fn name(&self) -> &str;
fn phase(&self) -> ExtensionPhase;
fn declarations(&self, ctx: &CompileContext) -> Result<Declarations>;
/// Default returns `Ok(Declarations::default())` — override when the
/// extension contributes steps, hosts, tools, or other signals.
fn declarations(&self, ctx: &CompileContext) -> Result<Declarations> {
Ok(Declarations::default())
}
}
```

Expand Down Expand Up @@ -287,7 +291,7 @@ The production target wrappers are:
- `job_ir.rs` — wraps the canonical shape as a target-job template with external `dependsOn` / `condition` template parameters.
- `stage_ir.rs` — wraps the canonical shape as a target-stage template with the stage-level external-parameter wrapper.

The canonical Setup → Agent → Detection → SafeOutputsTeardown shape, plus the optional Conclusion job, lives in `agentic_pipeline.rs` and is reused unchanged by every wrapper above; extensions plug into it via `Declarations` (steps, env, hosts, MCPG entries, and Agent-job condition clauses — see `Declarations::agent_conditions`).
The canonical Setup → Agent → Detection → (ManualReview?)Custom_\<tool\>* → SafeOutputs(+SafeOutputs_Reviewed?) → Teardown → Conclusion shape lives in `agentic_pipeline.rs` and is reused unchanged by every wrapper above; extensions plug into it via `Declarations` (steps, env, hosts, MCPG entries, and Agent-job condition clauses — see `Declarations::agent_conditions`).

When adding a target, follow the same pattern: parse and validate front matter, collect extension `Declarations`, build typed jobs/stages/steps, set the correct `PipelineShape`, and call the shared emit path.

Expand Down
5 changes: 3 additions & 2 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ _Part of the [ado-aw documentation](../AGENTS.md)._

Network isolation is provided by AWF (Agentic Workflow Firewall), which provides L7 (HTTP/HTTPS) egress control using Squid proxy and Docker containers. AWF restricts network access to an allowlist of approved domains.

Generated pipelines run AWF v0.27.32+ in **strict topology mode**: both the Agent and Detection jobs invoke AWF rootlessly with an explicit `--network-isolation` flag — there is no `sudo`, `--enable-host-access`, or `--legacy-security` fallback, and no author-facing knob to opt back into the legacy topology. The Agent additionally passes `--topology-attach awmg-mcpg` so the trusted MCPG container is attached to AWF's internal `awf-net`, and appends that hostname to `NO_PROXY`/`no_proxy` so MCP traffic bypasses Squid; Detection has no MCPG attachment. See [`docs/mcpg.md`](mcpg.md) for the MCPG topology and [`docs/mcp.md`](mcp.md) for MCP server configuration.
Generated pipelines run AWF v0.27.32+ in **strict topology mode**: both the Agent and Detection jobs invoke AWF rootlessly with an explicit `--network-isolation` flag — there is no `sudo`, `--enable-host-access`, or `--legacy-security` fallback, and no author-facing knob to opt back into the legacy topology. The Agent additionally passes `--topology-attach awmg-mcpg` so the trusted MCPG container is attached to AWF's internal `awf-net`, and appends that hostname to `NO_PROXY`/`no_proxy` so MCP traffic bypasses Squid; when the credential-isolated `ado-proxy` sidecar is enabled (`permissions.read` is configured), the Agent passes a second `--topology-attach awmg-ado-proxy` and adds that hostname to `NO_PROXY`/`no_proxy` too. Detection has no MCPG/ado-proxy attachment. See [`docs/mcpg.md`](mcpg.md) for the MCPG topology, [`docs/mcp.md`](mcp.md) for MCP server configuration, and [`docs/ado-proxy-design.md`](ado-proxy-design.md) for the ado-proxy sidecar.

The `ado-aw` compiler binary is distributed via [GitHub Releases](https://github.com/githubnext/ado-aw/releases) with SHA256 checksum verification. The AWF binary is distributed via [GitHub Releases](https://github.com/github/gh-aw-firewall/releases) with SHA256 checksum verification. Docker is sourced via the `DockerInstaller@0` ADO task.

Expand Down Expand Up @@ -205,14 +205,15 @@ Available ecosystem identifiers include:
| `swift` | Swift.org, CocoaPods |
| `terraform` | HashiCorp releases, Terraform registry |
| `threat-detection` | Copilot API and telemetry domains used by the Detection stage |
| `copilot-vendor` | Copilot vendor API/telemetry domains (`api.business.githubcopilot.com`, `api.enterprise.githubcopilot.com`, `api.individual.githubcopilot.com`, `telemetry.enterprise.githubcopilot.com`) |

**Compound identifier** (expands to a union of component identifiers):

| Identifier | Expands to |
|------------|------------|
| `default-safe-outputs` | `defaults` + `dev-tools` + `github` + `local` — the standard set of domains needed for most safe-output execution scenarios |

Additional ecosystems: `bazel`, `chrome`, `clojure`, `dart`, `deno`, `elixir`, `fonts`, `github-actions`, `haskell`, `julia`, `kotlin`, `latex`, `lean`, `lua`, `node-cdns`, `ocaml`, `perl`, `php`, `playwright`, `powershell`, `python-native`, `r`, `scala`, `zig`.
Additional ecosystems: `bazel`, `chrome`, `clojure`, `copilot-vendor`, `dart`, `deno`, `elixir`, `fonts`, `github-actions`, `haskell`, `julia`, `kotlin`, `latex`, `lean`, `lua`, `node-cdns`, `ocaml`, `perl`, `php`, `playwright`, `powershell`, `python-native`, `r`, `scala`, `zig`.

The full domain lists for direct identifiers are defined in `src/data/ecosystem_domains.json`. Compound identifiers are defined in `src/ecosystem_domains.rs`.

Expand Down
8 changes: 4 additions & 4 deletions docs/tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,10 +161,10 @@ subcommands. When configured, use `tools.azure-devops` for authenticated ADO
reads. Do not run `az login` or inject Azure credentials into the Agent
sandbox; use SafeOutputs or request a supported tool instead.

A daily smoke pipeline at
[`tests/safe-outputs/azure-cli.md`](../tests/safe-outputs/azure-cli.md)
exercises binary/subcommand availability without claiming authenticated direct
ADO access.
Detection, mounting, and prompt-advisory gating are covered by unit tests in
`src/compile/extensions/azure_cli.rs` and `tests/compiler_tests.rs`; there is
no dedicated agentic smoke pipeline exercising binary/subcommand availability
end-to-end.

### GitHub CLI (`gh`)

Expand Down
2 changes: 1 addition & 1 deletion prompts/create-ado-agentic-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ If interactive, ask only missing essentials first.
### 2. Build Front Matter
Use only required keys plus task-required options:
- `name`, `description`
- optional: `target`, `engine`, `workspace`, `pool`, `repos`, `imports`, `tools`, `runtimes`, `mcp-servers`, `safe-outputs`, `on`, `steps`, `post-steps`, `setup`, `teardown`, `permissions`, `parameters`, `env`, `variable-groups`, `network`, `execution-context`, `inlined-imports`, `supply-chain`
- optional: `target`, `engine`, `workspace`, `pool`, `repos`, `imports`, `tools`, `runtimes`, `mcp-servers`, `safe-outputs`, `on`, `steps`, `post-steps`, `setup`, `teardown`, `permissions`, `permissions-required`, `parameters`, `env`, `variable-groups`, `network`, `execution-context`, `inlined-imports`, `supply-chain`

Rules:
- Omit fields that equal defaults.
Expand Down
30 changes: 24 additions & 6 deletions scripts/ado-script/src/conclusion/__tests__/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -329,38 +329,56 @@ describe("conclusion/main", () => {
});

it("files a missing-tool work item when the manifest contains missing_tool", async () => {
setManifestEntries([{ name: "missing_tool", tool_name: "gh", context: "tool_name: gh" }]);
setManifestEntries([
{
name: "missing_tool",
status: "succeeded",
result: { tool_name: "gh", context: "needed for repository inspection" },
},
]);

await main();

expect(fileOrAppendWorkItem).toHaveBeenCalledTimes(1);
const body = (fileOrAppendWorkItem as ReturnType<typeof vi.fn>).mock
.calls[0]?.[3] as string;
expect(fileOrAppendWorkItem).toHaveBeenCalledWith(
"MyProject",
expect.objectContaining({ enabled: true }),
"[ado-aw] Agent encountered missing tool: feature reporter",
expect.stringContaining("- gh"),
body,
);
expect(body).toContain("- gh");
expect(body).toContain("- needed for repository inspection");
});

it("files a missing-data work item when the manifest contains missing_data", async () => {
setManifestEntries([
{
name: "missing_data",
data_type: "pull_request",
reason: "PR metadata not available",
context: "data_type: pull_request",
status: "succeeded",
result: {
data_type: "pull_request",
reason: "PR metadata not available",
context: "needed for review",
},
},
]);

await main();

expect(fileOrAppendWorkItem).toHaveBeenCalledTimes(1);
const body = (fileOrAppendWorkItem as ReturnType<typeof vi.fn>).mock
.calls[0]?.[3] as string;
expect(fileOrAppendWorkItem).toHaveBeenCalledWith(
"MyProject",
expect.objectContaining({ enabled: true }),
"[ado-aw] Agent reported missing data: feature reporter",
expect.stringContaining("PR metadata not available"),
body,
);
expect(body).toContain("- pull_request");
expect(body).toContain("- PR metadata not available");
expect(body).toContain("- needed for review");
});

it("appends a comment to an existing work item instead of creating a duplicate", async () => {
Expand Down
18 changes: 13 additions & 5 deletions scripts/ado-script/src/conclusion/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ function extractNamedValue(
fieldName: "tool_name" | "data_type",
pattern: RegExp,
): string | undefined {
const direct = entry[fieldName];
const direct = entry[fieldName] ?? entry.result?.[fieldName];
if (typeof direct === "string" && direct.trim().length > 0) {
return direct.trim();
}
Expand Down Expand Up @@ -319,7 +319,9 @@ function buildPipelineFailureReport(config: RuntimeConfig): SignalReport | null
function buildNoopReport(config: RuntimeConfig, entries: readonly ManifestEntry[]): SignalReport | null {
if (entries.length === 0) return null;

const contexts = unique(entries.map((entry) => entry.context ?? undefined));
const contexts = unique(
entries.map((entry) => entry.context ?? toOptionalString(entry.result?.context)),
);
const lines = [
"The conclusion job detected one or more `noop` diagnostic signals.",
"",
Expand Down Expand Up @@ -347,7 +349,9 @@ function buildMissingToolReport(
extractNamedValue(entry, "tool_name", /tool[_ -]?name\s*:\s*([^\r\n,;]+)/i)
),
);
const contexts = unique(entries.map((entry) => entry.context ?? undefined));
const contexts = unique(
entries.map((entry) => entry.context ?? toOptionalString(entry.result?.context)),
);
const lines = [
"The conclusion job detected one or more `missing_tool` diagnostic signals.",
"",
Expand Down Expand Up @@ -378,8 +382,12 @@ function buildMissingDataReport(
extractNamedValue(entry, "data_type", /data[_ -]?type\s*:\s*([^\r\n,;]+)/i)
),
);
const contexts = unique(entries.map((entry) => entry.context ?? undefined));
const reasons = unique(entries.map((entry) => entry.reason ?? undefined));
const contexts = unique(
entries.map((entry) => entry.context ?? toOptionalString(entry.result?.context)),
);
const reasons = unique(
entries.map((entry) => entry.reason ?? toOptionalString(entry.result?.reason)),
);
const lines = [
"The conclusion job detected one or more `missing_data` diagnostic signals.",
"",
Expand Down
Loading
Loading