You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a deterministic verifier for the downloaded preset manifest, release tag, README install references, issue metadata, and resulting catalog and documentation files. It reads archive contents without executing them and supports preset-scoped manifests in monorepos.
Defer validation-passed and the draft catalog PR request until the generated files pass consistency and ordering checks. Distinguish confirmed submission mismatches from blocked checks and agent-generated files that need repair.
Regenerate the compiled workflow lock file and add positive and negative regression cases, including a stale --from URL alongside a valid --dev command, archive unavailability, and preserved created_at on updates.
gh aw compile add-community-preset --no-check-update
Live submission workflow run (requires an issue labeled for maintainer automation)
AI Disclosure
I did not use AI assistance for this contribution
I did use AI assistance
AI disclosure: GitHub Copilot App, GPT-6 Sol (runtime-default reasoning effort), autonomously generated and tested the verifier, workflow updates, and regression tests at the contributors request. The contributor requested the commit and PR; no human line-by-line review or live workflow validation is claimed.
Compare published manifests and README release references with issue fields, then verify generated catalog and documentation before success labeling. Regenerate the workflow lock file and cover submission, blocked, and repair outcomes.
Refs github#4746
Assisted-by: GitHub Copilot (model: GPT-6 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Record the submission UTC date and require generated catalog timestamps to match it; keep update creation dates intact. Detect stale README release links whose scoped tag prefix matches the submitted release, including ZIP archive URLs.
Refs github#4746
Assisted-by: GitHub Copilot (model: GPT-6 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review-round update (commit 6acc33db): the preset verifier now records the submission UTC date and checks generated catalog timestamps against it while preserving existing creation dates on updates. README release checks also recognize the submitted scoped tag prefix, so stale URLs cannot be masked by another accepted install command. Added regressions for both archive URL forms, stale new/update timestamps, and unrelated monorepo releases; the focused suite passed (143 tests), and the workflow lock was regenerated. The two review threads are left open for reviewer verification.
On behalf of @mnriem: GitHub Copilot App (GPT-6 Sol, runtime-default reasoning effort, autonomous) authored these code, workflow, and test changes and this review-round summary. No human line-by-line review or live workflow run is claimed.
Include homepage in the validated snapshot. Step 4 requires this field for new catalog entries, and every current community preset has it, but generated() only compares keys present in expected; consequently a generated entry with a missing or stale homepage still passes. Since the issue form has no separate homepage field, the deterministic value here is the submitted repository URL.
Avoid treating unrelated unscoped monorepo release URLs as stale while retaining checks for matching preset scopes and release assets. Require generated catalog homepage to equal the submitted repository URL for new and updated presets.
Refs github#4746
Assisted-by: GitHub Copilot (model: GPT-6 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review-round update (commit 0007401e): unscoped monorepo README releases are now compared only when the submitted preset is identifiable by the same release asset (or by an unscoped archive tag from an unscoped submission); unrelated releases no longer fail validation. Generated catalog entries must also include homepage equal to the submitted repository URL, including updates. Added success and failure regressions for both cases, reran the focused suite (150 passed), and regenerated the workflow lock file. The review thread remains open for reviewer verification.
On behalf of @mnriem: GitHub Copilot App (GPT-6 Sol, runtime-default reasoning effort, autonomous) authored the verifier, workflow, and test changes in this round and this summary. No human line-by-line review or live workflow run is claimed.
Use the downloaded archive manifest count to apply bare-tag stale URL checks only when the archive contains one preset. Keep scoped tags and matching release assets checked, and document the monorepo exception with regression coverage.
Refs github#4746
Assisted-by: GitHub Copilot (model: GPT-6 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review-round update (commit 33815e53): bare archive tags are now treated as belonging to the submitted preset only when the downloaded archive contains a single preset.yml. A multi-preset archive can therefore have another preset鈥檚 unscoped archive URL in its README without being marked stale; scoped tags and matching release assets remain checked. Added before/after regression coverage for the monorepo case and matching/stale single-preset archives, regenerated the workflow lock file, and ran the focused suite (153 passed). The review thread is left open for reviewer verification.
On behalf of @mnriem: GitHub Copilot App (GPT-6 Sol, runtime-default reasoning effort, autonomous) authored this verifier, workflow, and test update and this review-round summary. No human line-by-line review or live workflow run is claimed.
Escaped pipes in a valid human-readable preset name are still treated as table delimiters here. documentation_row() deliberately renders Data | Governance as Data \| Governance, but this split parses the name as Data \, so the generated phase can never pass for that submission. Split on unescaped delimiters and unescape the cell value; please add a regression case with a pipe in the preset name.
Addressed the review feedback in commit 6cbc4851. The verifier now preflights preset.yml entries with a 100-manifest limit and a 10 MiB cumulative uncompressed budget before YAML parsing, and generated documentation parsing now handles escaped pipes in preset names. Added regressions for manifest count, aggregate size, and Data | Governance; tests/test_community_preset_validation.py plus tests/test_github_workflows.py pass (156 tests).
Posted on behalf of @mnriem by GitHub Copilot (GPT-5.6 Sol, autonomous mode). AI assistance implemented the fixes, added the regression coverage, and ran the reported validation.
This strips . from every argument, so the valid current-directory forms specify preset add --dev . and --dev .. become empty and a README containing only that accepted --dev <path> form is rejected. Preserve periods for development paths and strip sentence punctuation only when parsing a --from URL; add a regression case for --dev ..
Any nonempty token is accepted as a development path, including another option. For example, specify preset add --dev --priority 20 is missing the required --dev value and the CLI rejects it, but this verifier marks the README valid. Exclude option-looking values so only an actual path satisfies this form, and cover the rejection in a negative test.
Verify documentation row replacement using prior metadata
preset_name is a human-readable display value, not a unique identifier鈥攖he submission form only guarantees uniqueness for preset IDs. This check therefore blocks two different presets with the same display name, while a renamed update that leaves the old row and adds a new row can still pass because each name occurs once. Preserve enough prior entry metadata in the snapshot to verify that the target's old documentation row was replaced, rather than imposing global name uniqueness.
Addressed all three previously missed review items in commit db8baccb. README validation now preserves ./.. development paths while rejecting option-like missing values, and documentation verification now snapshots original row counts so duplicate display names remain valid while stale rows from renamed presets are rejected. Added positive and negative regressions for each path; the related verifier and workflow suites pass (161 tests).
Posted on behalf of @mnriem by GitHub Copilot (GPT-5.6 Sol, autonomous mode). AI assistance implemented the fixes, added the regression coverage, and ran the reported validation.
Addressed the stale-row finding in commit ca9df554. Update verification now identifies and snapshots the literal pre-edit documentation row using the prior display name and repository link, rather than reconstructing it from catalog prose. The mismatch regression fails on the reviewed commit and passes with this fix; all 40 current catalog entries resolve to exactly one existing row, and the related verifier/workflow suites pass (161 tests).
Posted on behalf of @mnriem by GitHub Copilot (GPT-5.6 Sol, autonomous mode). AI assistance implemented the fix, updated regression coverage, and ran the reported validation.
\s+ can cross line boundaries, so an invalid bare --dev at the end of a line can consume the first word of the next paragraph as its path and be accepted. Restrict command separators to horizontal whitespace, and add a negative regression for a line-ending --dev.
Detect stale URLs when versioned asset filenames differ
The stale-URL check only recognizes an unscoped release when the asset filename is byte-for-byte identical. A common versioned asset therefore escapes detection: with submitted .../v1.2.3/sample-1.2.3.zip, README .../v1.2.2/sample-1.2.2.zip plus a valid --dev command passes, even though the stale URL is clearly for this preset. This misses the PR's stale---from acceptance criterion; compare version-normalized asset identity (or another preset-aware identity) and cover this case.
Addressed both previously missed README parsing findings in commit 516d2432. Command matching now uses horizontal whitespace so a bare line-ending --dev cannot consume the next paragraph, and stale unscoped release URLs are detected by comparing semantic-version-normalized asset filenames. Added regressions for both cases; the related verifier/workflow suites pass (163 tests, with 50 focused verifier tests collected).
Posted on behalf of @mnriem by GitHub Copilot (GPT-5.6 Sol, autonomous mode). AI assistance implemented the fixes, added the regression coverage, and ran the reported validation.
cell() escapes pipes without first escaping existing backslashes. For a submitted name or description containing \|, this emits \\|; Markdown treats the pipe as a column delimiter, while the verifier compares against that same malformed raw row and can still pass the generated phase. Escape backslashes before pipes and add a regression case for this input.
Addressed the previously missed Markdown escaping finding in commit f9eea075. Documentation cells now escape existing backslashes before pipes, and table parsing decodes escaped backslashes and pipes symmetrically, so valid \| input remains one cell and round-trips correctly. Added a regression for this input; the related verifier/workflow suites pass (164 tests, with 51 focused verifier tests collected), and all 40 current documentation rows still resolve uniquely.
Posted on behalf of @mnriem by GitHub Copilot (GPT-5.6 Sol, autonomous mode). AI assistance implemented the fix, added the regression coverage, and ran the reported validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
triage-can-waitVerdict: valid and in-scope but deprioritized; held behind the evidence gate
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mnriem
added
the
triage-can-wait
Description
Closes #4746.
validation-passedand the draft catalog PR request until the generated files pass consistency and ordering checks. Distinguish confirmed submission mismatches from blocked checks and agent-generated files that need repair.--fromURL alongside a valid--devcommand, archive unavailability, and preservedcreated_aton updates.Testing
.venv/bin/python -m pytest -q tests/test_community_preset_validation.py tests/test_github_workflows.py --tb=short(137 passed)gh aw compile add-community-preset --no-check-updateAI Disclosure
AI disclosure: GitHub Copilot App, GPT-6 Sol (runtime-default reasoning effort), autonomously generated and tested the verifier, workflow updates, and regression tests at the contributors request. The contributor requested the commit and PR; no human line-by-line review or live workflow validation is claimed.