Skip to content

ci: sync sample catalog from foundry-samples (yimin/catalog-review-correction-e2e-20260928) - #836

Closed
foundry-toolkit-sync-bot[bot] wants to merge 2 commits into
yimin/catalog-review-correction-e2e-20260928from
ci/sync-sample-catalog-yimin/catalog-review-correction-e2e-20260928-36383059780
Closed

foundry-toolkit-sync-bot[bot] wants to merge 2 commits into
yimin/catalog-review-correction-e2e-20260928from
ci/sync-sample-catalog-yimin/catalog-review-correction-e2e-20260928-36383059780

Conversation

@foundry-toolkit-sync-bot

Copy link
Copy Markdown
Contributor

Summary

This automated draft PR incrementally updates samples/hosted-agent/sample-catalog.json
from a pinned scan of microsoft-foundry/foundry-samples.
The single snapshot keeps template facts in templates and grouping, Patterns,
and Details in cards/patterns. Clients read one Raw file without a REST revision lookup.

What changed

  • Surviving template metadata, card IDs, titles, Patterns, and relative ordering are preserved.
  • Cards with unchanged membership keep their Details verbatim and require no review call.
  • Deleted templates and their card references are removed together; empty cards and unused dimension options are removed.
  • New template metadata is generated by the existing AI service from its README; dimensions come from source metadata.
  • AI prefers existing cards with the same core task, title and Pattern. Occupied language/framework/protocol tuples cannot be reused within a card.
  • Proposed new cards are independently compared with every tuple-compatible candidate in the same Pattern, including cards created earlier in the run. A same-task result reuses the candidate; SDK or protocol differences alone do not justify a new card.
  • After grouping, membership-changed cards receive an explicit keep/change audit of all eight Details fields against the final members' pinned READMEs. Audit actions must match the sparse patch. Both reviews select supplied README excerpt IDs; code resolves them to original text instead of asking AI to copy quotes. Reviews and changed fields are logged.
  • Valid Requirements are preserved unless factually inaccurate. Commas do not create extra values, and requiresModel=false does not imply model access is unnecessary. Variant-specific approval/recovery claims must be qualified; generated output describes one selected implementation.
  • Deletion-only membership changes also require Details review for surviving cards; removing a whole card does not.
  • New cards are created only when no compatible candidate fits; their Details and one primary Pattern are AI-generated for review.
  • The catalog commitSha advances only when templates are added or removed. It also pins the implementations of surviving templates to that revision.
  • Failed AI generation/review, incomplete audits, unknown README references, incomplete scans, or invalid grouping stop the job before PR creation. Verified source references do not replace human review of semantic accuracy.
  • A final bounded agent step reads the repository review skill and fixed-source evidence, then proposes prose corrections to this Draft PR. Trusted code validates scope, evidence references and tests before appending a commit. Failed or unresolved review leaves the Draft PR open and the workflow blocked; no approval or merge is automated.

Reviewer Checks

  • Confirm the catalog is self-contained and card-only edits do not change template facts.
  • Confirm new templates have correct language, framework, and protocol values.
  • Review the auto-generated description for any new templates.
  • Confirm no PM-edited display fields were accidentally lost.
  • Confirm every template belongs to one card and each card's selection combinations are unique.
  • Review card Details and Pattern assignments when their template membership changes.
  • Confirm existing-card reuse preserves the core task and Details edits are minimal and accurate for every variant; uniqueness is enforced by code, not left to AI.

@foundry-toolkit-sync-bot

Copy link
Copy Markdown
Contributor Author

Automated Catalog Review: passed

Input: 8572518

Output: 9045400

Skill SHA-256: 103ad8f96cc30df1b99754f75c88b733908235cf3c647274006d8c228f4664f6

Phase: complete

Completed passes: 2

Proposed corrections passed scope checks, structural validation and regression tests.

The PR remains draft. This is automated evidence-assisted review, not human approval or runtime deployment validation.

@Yimin-Jin

Copy link
Copy Markdown
Collaborator

Closing this isolated correction-path fixture without merging. The final test passed at https://github.com/microsoft/foundry-dev-tools/actions/runs/36383059780 with automated correction commit 9045400: description 193 -> 81 characters, all other catalog values unchanged; 137 regression tests passed. The implementation remains in #825. This verifies execution and protected edit scope, not human approval of AI prose; the generated description still repeats language/protocol wording. Test branches are being removed; this PR and CI report retain the evidence.

@Yimin-Jin Yimin-Jin closed this Sep 28, 2026
@Yimin-Jin
Yimin-Jin deleted the ci/sync-sample-catalog-yimin/catalog-review-correction-e2e-20260928-36383059780 branch September 28, 2026 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant