Skip to content

refactor: consolidate reviewed bot improvements - #2189

Open
jamesadevine wants to merge 5 commits into
mainfrom
chore/consolidate-bot-prs-2026-09-17
Open

jamesadevine wants to merge 5 commits into
mainfrom
chore/consolidate-bot-prs-2026-09-17

Conversation

@jamesadevine

@jamesadevine jamesadevine commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Consolidates the strongest correct changes from 45 open bot-authored PRs reviewed
against 559b9188537520d94b442813825699d44f27bdaf.

  • Integrates 12 PRs wholly and the unique/correct portions of 10 PRs.
  • Deduplicates 18 overlapping PRs.
  • Excludes 3 obsolete stale branches and 1 incorrect documentation change.
  • Leaves feat(safe-outputs): allow model-selected PR reviewers #2099 open because its head advanced after the frozen review with
    substantial new follow-up changes that are not superseded by this aggregate.
  • Leaves all 14 dependency/version bump PRs untouched.

The aggregate includes documentation corrections, audit and safe-output
complexity refactors, CLI/compiler regression coverage, and Conclusion
executor-E2E scenarios.

Validation

  • cargo test --all-targets
  • cargo clippy --all-targets -- -D warnings
  • cargo test --test audit_it --test compiler_tests
  • npm test -- --run src/executor-e2e/__tests__/conclusion-cli.test.ts src/executor-e2e/__tests__/runner.test.ts
  • npm run typecheck
  • npm run build:executor-e2e
  • cargo run --quiet -- export-bash-scripts --help confirms --output
  • git diff --check
  • Azure executor E2E build 641382

The Astro site build fails identically on unchanged main because the locally
cached Starlight version rejects the repository's existing sidebar
autogenerate configuration. The reviewed #2075 site change was independently
rejected and is not included.

Dispositions

PR Disposition Result
#2184 Accepted Canonical container_runtime.rs architecture entry.
#2181 Accepted partially One-file create_github_issue complexity refactor.
#2179 Duplicate Superseded by #2184.
#2172 Accepted set_github_issue_field refactor.
#2167 Accepted partially add_report_findings helper extraction.
#2163 Accepted Canonical ADO proxy findings decomposition.
#2161 Accepted trace --step CLI coverage.
#2160 Duplicate Superseded by #2184.
#2154 Accepted Correct permissions-required prompt field list.
#2151 Accepted ADO script documentation corrections.
#2150 Accepted Canonical Azure CLI test references.
#2148 Accepted Correct export-bash-scripts --output documentation.
#2147 Accepted Correct IR pipeline and Conclusion semantics.
#2145 Accepted partially Detailed copilot-vendor ecosystem entry.
#2144 Accepted partially ADO proxy topology and ecosystem-list corrections.
#2136 Accepted partially Artifact-download helper extraction.
#2125 Accepted partially Milestone safe-output refactor.
#2123 Duplicate Superseded by #2184.
#2119 Duplicate Superseded by #2163.
#2116 Accepted partially GitHub sub-issue refactor.
#2115 Accepted partially CompilerExtension::declarations documentation only.
#2114 Accepted MCP warning-propagation coverage.
#2113 Duplicate Superseded by #2184.
#2110 Accepted Conclusion executor-E2E coverage.
#2109 Duplicate Superseded by #2184.
#2106 Duplicate Superseded by #2184.
#2104 Duplicate Superseded by #2184.
#2102 Accepted Canonical ADO proxy catalog decomposition.
#2101 Duplicate Superseded by #2184 and #2150.
#2099 Changed after review Frozen head 4dbcee44 was duplicate of existing functionality; current head 220878c4 adds substantial new integration fixes and remains open for separate review.
#2096 Duplicate Superseded by #2184.
#2094 Accepted partially Canonical build-attachment refactor.
#2093 Duplicate Superseded by #2184 and #2144/#2145.
#2089 Duplicate Superseded by #2184.
#2081 Duplicate Superseded by #2102.
#2080 Duplicate Superseded by #2184.
#2078 Duplicate Superseded by #2184.
#2075 Rejected Legacy repositories: example omits checkout and is functionally incorrect.
#2074 Duplicate Superseded by #2184.
#2025 Duplicate Superseded by #2102.
#2006 Accepted partially Isolated create_work_item refactor commit only.
#1946 Obsolete Stale conflicting create_pull_request rewrite.
#1909 Duplicate Superseded by #2094.
#1838 Obsolete Valuable typed-install idea, but unsafe stale implementation.
#1831 Obsolete Valuable package-feed idea, but unsafe old-architecture branch.

Review safeguards

  • Every disposition was made against a frozen head SHA.
  • All 45 head SHAs and origin/main were rechecked before commit; none changed.
  • The closure-time SHA guard caught feat(safe-outputs): allow model-selected PR reviewers #2099 advancing after review; it was not
    closed or represented as superseded.
  • Duplicate families use one canonical implementation.
  • Partial integrations omit unrelated cumulative commits and duplicate edits.

Combine canonical documentation fixes, audit and safe-output complexity
refactors, regression coverage, and Conclusion executor E2E scenarios.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: aba7a965-412b-4f58-8d80-cf67bde7b0ce
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions

This comment has been minimized.

Use the same gh-aw v0.86.2 compiler that generated the committed prompt
evaluator lock file.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: aba7a965-412b-4f58-8d80-cf67bde7b0ce
@github-actions

This comment has been minimized.

Project the bearer token kind alongside compiler-owned ADO tokens and mirror
that contract in the Conclusion executor E2E harness.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: aba7a965-412b-4f58-8d80-cf67bde7b0ce
@github-actions

This comment has been minimized.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: aba7a965-412b-4f58-8d80-cf67bde7b0ce
@github-actions

This comment has been minimized.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: aba7a965-412b-4f58-8d80-cf67bde7b0ce
@github-actions

Copy link
Copy Markdown
Contributor

Prompt evaluation

Note

This is an advisory static review. Only Prompt Contracts is merge-blocking.

Only prompts/create-ado-agentic-workflow.md changed; the update and debug
prompts and the shared contract are byte-identical between base and head, so
only the create suite is in scope. The only textual change is adding
permissions-required to the "optional" front-matter key list in step 2
(Build Front Matter). The field is real (src/compile/types.rs,
docs/front-matter.md) but per the doc comment is "usually set by an imported
component rather than authored directly," and none of the three create cases
involve imports, so it has no material effect on any case's expected output.

Prompt Cases Improved Unchanged Regressed Inconclusive
create 3 0 3 0 0

Potential regressions

None identified.

Per-case scores
Case Prompt Base Candidate Result
create-minimal-manual create 16/16 16/16 unchanged
create-scheduled-workitem-report create 16/16 16/16 unchanged
create-needs-clarification create 16/16 16/16 unchanged

Scores are the sum of 4 common criteria (task_completion, grounding,
safety_and_consent, clarity_and_done_criteria) + 4 create criteria
(create_workflow_coherence, create_trigger_scope,
create_tools_outputs_permissions, create_no_action), each 0-2, against the
prompt text (this is a static instruction review, not an executed run).

  • create-minimal-manual: Both prompt versions instruct a manual trigger,
    minimal fields, no safe-outputs, and an explicit "No Action" section mapped
    to summarizing zero matches — matches ground truth (trigger: manual,
    safe_outputs_required: false). The added permissions-required key is
    irrelevant here (no imports).
  • create-scheduled-workitem-report: Both versions require a compact front
    matter with tools, permissions, and safe-outputs, and an explicit
    "No Action" section mapped to noop — matches ground truth (schedule,
    comment-on-work-item max 3, target). permissions-required is not implicated
    since the reference config uses concrete permissions.read/write, not
    imports.
  • create-needs-clarification: Both versions' step 1 ("Gather Required
    Inputs" / "ask only missing essentials first") and Done Criteria ("When
    essential inputs are missing, concise clarification is returned instead of
    an invented workflow") are unchanged and drive the expected clarification
    outcome identically.

Generated by Prompt Evaluator for #2189 · auto · 28.8 AIC · ⌖ 1.6 AIC · ⊞ 8.3K ·

This was referenced Sep 18, 2026
This was referenced Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant