You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(ci): stage catalog sync and auto-fix with review skill - #825
Expose seven dependent jobs in the Actions graph: scan -> metadata -> grouping -> Details -> write/validate -> Draft PR -> AI review. A reusable stage workflow shares setup for the three generation jobs.
Pass state and candidate catalog through run/attempt-scoped artifacts; pin every checkout to the workflow SHA. Publish and review create separate short-lived App tokens; no credentials cross job outputs or artifacts.
Automatically apply bounded prose corrections: at most two repair passes and one fresh final verification; unresolved findings fail the workflow while preserving the Draft PR. No manual corrections are required on the successful path, and approval/merge remain human-controlled.
Run pinned Copilot CLI 1.0.88 in a read-only, network-isolated container with only view/grep/glob/skill tools. The host retains GitHub/Azure credentials and exposes only a bounded inference proxy.
Validate field scope, pinned-source line references, catalog invariants, and trusted tests before appending a commit parented to the captured PR head with a non-force ref update. Refuse reruns that would overwrite an existing run branch.
Give the reviewer explicit mounted source paths and line markers. Trusted code resolves path/startLine/endLine references to exact original text; copied quotations are no longer required.
Persist raw answers, parsed patches, resolved evidence and validation errors before deciding whether to proceed. Rejected patches leave the candidate unchanged and receive specific feedback within the same three-attempt limit. Corrections require a fresh independent clean pass; service/time/budget failures do not start extra recovery calls.
Use the same review-sample-catalog skill for CI and maintainer review. The catalog snapshot itself is unchanged.
Supply only affected cards and all their member templates to the model; trusted code still validates the full catalog. Stop forwarding retries after reported token exhaustion.
Pin all 19 external action usages across both catalog workflows to verified immutable commit SHAs.
Protect templateSelection, dimension identities/labels/placeholders and retained option metadata/order; allow only options corresponding to actual template values.
Validation
Latest head 1a2185e: 162/162 local regression tests passed; actionlint and diagnostics passed. Structural comparison confirms the workflow behavior equals pre-Environment 1da714f except a clarified checkout label.
Added coverage rejects renaming/removing a card with surviving members, merging it into another card, or moving a surviving template; complete deletion of all a card's templates remains allowed.
Earlier implementation verification below is historical coverage, not a substitute for the current-head results above.
Deterministic recovery tests cover invalid line references and malformed JSON -> persisted failure -> unchanged candidate -> specific feedback -> correction -> independent final verification. Tests also cover protected-field rejection, unresolved findings, three-attempt exhaustion, service errors, source ownership, CRLF/Markdown preservation and mounted-path/line-count consistency.
Intermediate run https://github.com/microsoft/foundry-dev-tools/actions/runs/36388503609 retained all three responses and extracted real source-line evidence, but remained blocked by unresolved source-access findings in the final pass. Nothing was pushed. Explicit mounted read paths were added afterward; the latest run above passed.
Execution and source provenance are not semantic approval. Human content review remains required; no hosted sample deployment/runtime validation or automatic merge was performed.
Review Disposition
All five review threads have been answered and resolved. Action mutability and picker metadata were fixed in 1da714f; surviving card identity/ownership was fixed in 2076a55 with four additional regression cases.
The dispatch-ref comment was re-evaluated with the owner against the pre-existing maintainer trust model: manual dispatch requires write access, and the base workflow already ran code from the maintainer-selected ref with the same Repository secrets. Writers authorized to edit/dispatch workflows are trusted; unreviewed external refs must not be dispatched.
Commit 1a2185e removes the added Environment dependency, restores explicit model-secret forwarding, and documents assumptions in .github/workflows/README.md. github.sha pins execution for reproducibility; it does NOT isolate credentials from malicious write-authorized maintainers. A stronger repository-wide policy remains a separate hardening decision, not a security guarantee claimed by this PR.
Existing shared Repository secrets were not modified or deleted. The experimental catalog-sync Environment and its three Azure secrets remain configured but are not referenced by these workflows; no migration is required.
Resolving the discussion records this scoped owner-confirmed disposition, not completion of the abandoned migration. Human approval is still required; no review approval or merge was automated.
Guardrails
No agent shell, edits, GitHub tools, source execution, or raw write credentials.
Maximum three agent passes, 40 model requests per pass and a 300k reported-token stop threshold (one in-flight response may cross it; subsequent retries are rejected), 16k output tokens/request, 12 minutes/pass.
Only affected card prose and new template name/description may change; existing identities, membership, Patterns and unaffected metadata/Details are protected.
Evidence-reference validation is deterministic, but semantic judgment remains model-based; blocked reports must never be treated as approval.
Add an on-demand repository skill for the agreed CI-generated Draft PR followed by human-led AI review. Define snapshot and structural checks, per-implementation semantic standards, minimal authorized data fixes, validation and release-promotion boundaries. Reference current code rather than hard-coded counts or versions; do not change the sync workflow.
Split incremental generation into resumable stages and append a sandboxed Copilot skill review to the same Draft PR workflow. Keep repository and model credentials outside the agent, constrain prose patches and pinned-source evidence, and publish via non-force Git ref updates after trusted validation. Add offline container smoke coverage and 123 passing regression tests.
Yimin-Jin
changed the title
docs(skills): define source-grounded sample catalog review
feat(ci): stage catalog sync and auto-fix with review skill
Sep 24, 2026
commitSha is not constrained to template-path changes
.github/scripts/review_catalog_pr.mjs:36
The review scope never constrains commitSha to the template-path diff. A candidate with unchanged templates can replace it with any valid SHA and still pass; collectSources will then fetch evidence from that different revision, violating the catalog contract that the source revision advances only when templates are added or removed. Compare the base and candidate template-path sets and require the SHA to change iff that set changes.
Review coverage omits existing card members
.github/scripts/review_catalog_pr.mjs:95
The review contract asks the agent to review every current member of each affected card, but this guard only requires the newly added templates in scope.templates; reviewedTemplates may omit all existing members and still pass with an empty patch and no findings. That allows a clean review to be accepted without coverage of the variants whose Details are being relied on. Require the reviewed-template set to equal the supplied member set (or otherwise validate coverage for every member of each affected card).
Recovery validation can clear unresolved findings
.github/scripts/review_catalog_pr.mjs:181
When a valid pass reports unresolved factual findings and the next recovery response is rejected, this assignment replaces the feedback with only the validation error and drops the earlier unresolved list. The final pass can then return an empty report without seeing those blockers, and report.unresolved is overwritten with the empty list. Preserve any prior unresolved findings when adding validation feedback so recovery cannot silently clear them.
Review report artifact name is not unique per attempt
.github/workflows/sync-sample-catalog.yml:432
Unlike the state and catalog artifacts above, this artifact name is not scoped by github.run_id and github.run_attempt. A rerun can reuse the existing catalog-review name, and immutable v4 artifact uploads then fail in the always() reporting step, losing the report for that attempt. Scope the name consistently so each review attempt can publish its own report.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Actions
Failure
Summary
Expose seven dependent jobs in the Actions graph: scan -> metadata -> grouping -> Details -> write/validate -> Draft PR -> AI review. A reusable stage workflow shares setup for the three generation jobs.
Pass state and candidate catalog through run/attempt-scoped artifacts; pin every checkout to the workflow SHA. Publish and review create separate short-lived App tokens; no credentials cross job outputs or artifacts.
Automatically apply bounded prose corrections: at most two repair passes and one fresh final verification; unresolved findings fail the workflow while preserving the Draft PR. No manual corrections are required on the successful path, and approval/merge remain human-controlled.
Run pinned Copilot CLI 1.0.88 in a read-only, network-isolated container with only view/grep/glob/skill tools. The host retains GitHub/Azure credentials and exposes only a bounded inference proxy.
Validate field scope, pinned-source line references, catalog invariants, and trusted tests before appending a commit parented to the captured PR head with a non-force ref update. Refuse reruns that would overwrite an existing run branch.
Give the reviewer explicit mounted source paths and line markers. Trusted code resolves path/startLine/endLine references to exact original text; copied quotations are no longer required.
Persist raw answers, parsed patches, resolved evidence and validation errors before deciding whether to proceed. Rejected patches leave the candidate unchanged and receive specific feedback within the same three-attempt limit. Corrections require a fresh independent clean pass; service/time/budget failures do not start extra recovery calls.
Use the same review-sample-catalog skill for CI and maintainer review. The catalog snapshot itself is unchanged.
Supply only affected cards and all their member templates to the model; trusted code still validates the full catalog. Stop forwarding retries after reported token exhaustion.
Pin all 19 external action usages across both catalog workflows to verified immutable commit SHAs.
Protect templateSelection, dimension identities/labels/placeholders and retained option metadata/order; allow only options corresponding to actual template values.
Validation
Review Disposition
Guardrails