Skip to content

Add blind-test skill 🤖🤖🤖 - #6232

Closed
arez-xd wants to merge 2 commits into
github:mainfrom
arez-xd:add-blind-test
Closed

arez-xd wants to merge 2 commits into
github:mainfrom
arez-xd:add-blind-test

Conversation

@arez-xd

@arez-xd arez-xd commented Nov 3, 2026

Copy link
Copy Markdown

Pull Request Checklist

  • I have read and followed the CONTRIBUTING.md guidelines.
  • I have read and followed the Guidance for submissions involving paid services.
  • My contribution adds a new instruction, prompt, agent, skill, workflow, or canvas extension file in the correct directory.
  • The file follows the required naming convention.
  • The content is clearly structured and follows the example format.
  • I have tested my instructions, prompt, agent, skill, workflow, or canvas extension with GitHub Copilot.
  • I have run npm start and verified that README.md is up to date.
  • I am targeting the main branch for this pull request.

Description

Checks whether a skill, an instructions file, a prompt or an MCP server changes what the agent does. It builds a fictional target with planted problems and traps (legitimate decisions that look like mistakes), writes the answer key before any run, runs the same request with and without the customization in isolated agents, grades the outputs without knowing which arm wrote them and reports what the numbers can honestly claim.

It has two levels: a quick check by default, and the full test with calibration and normalization when the numbers go public. Useful to anyone writing skills or instructions for Copilot before they publish them. The repository includes a test of the skill with its own method.

No paid service or account needed. Source, MIT: https://github.com/arez-xd/blind-test


Type of Contribution

  • New instruction file.
  • New prompt file.
  • New agent file.
  • New plugin.
  • New skill file.
  • New agentic workflow.
  • New canvas extension.
  • Update to existing instruction, prompt, agent, plugin, skill, workflow, or canvas extension.
  • Other (please specify):

Additional Notes

Bundled scripts, Node 18+, no dependencies, network calls or installs:

  • score.mjs, refcheck.mjs, blind-labels.mjs work on local files: scoring grades against the key, checking that a rewrite kept every file and line, copying results under random labels.
  • target-server.mjs serves the test folder on 127.0.0.1; any request other than GET or HEAD gets 405 and is logged.

agents/claude.yaml and agents/openai.yaml are display metadata for agents that read them. npm run skill:validate and npx @microsoft/vally-cli lint pass.


By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.

🤖 Generated with Claude Code

arez-xd and others added 2 commits November 3, 2026 22:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@arez-xd
arez-xd requested review from a team as code owners November 3, 2026 17:27
@github-actions github-actions Bot added new-submission PR adds at least one new contribution skills PR touches skills labels Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔒 PR Risk Scan Results

Scanned 17 changed file(s).

Severity Count
🔴 High 0
🟠 Medium 0
ℹ️ Info 4
Severity Rule File Line Match
ℹ️ skill-script-touched skills/blind-test/scripts/blind-labels.mjs 1 skills/blind-test/scripts/blind-labels.mjs
ℹ️ skill-script-touched skills/blind-test/scripts/refcheck.mjs 1 skills/blind-test/scripts/refcheck.mjs
ℹ️ skill-script-touched skills/blind-test/scripts/score.mjs 1 skills/blind-test/scripts/score.mjs
ℹ️ skill-script-touched skills/blind-test/scripts/target-server.mjs 1 skills/blind-test/scripts/target-server.mjs

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

🚦 Submission status: 👀 Ready for review

Risk tier: merge-risk:high — Privileged execution, automation, or review-policy change
Required to merge: passing submission-gate checks plus 1 approval from reviewers with write access, including a maintainer with admin or maintain permission.

Why this tier
  • skills/blind-test/scripts/blind-labels.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
  • skills/blind-test/scripts/refcheck.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
  • skills/blind-test/scripts/score.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
  • skills/blind-test/scripts/target-server.mjs is a high-risk path (automation, scripts, MCP config, hooks, or review policy)

Automated checks

Check Status Details
Line endings ✅ Passed Passed · logs
Spelling ✅ Passed Passed · logs
Generated README consistency ✅ Passed Passed · logs
Skill validation ✅ Passed Passed · logs
Skill lint (vally) ✅ Passed Passed · logs
Risk scan ✅ Passed Passed · logs
Contributor reputation ✅ Passed Passed · logs
Duplicate resource scan ✅ Passed Passed · logs
PR quality signal ⏭️ Skipped Skipped by its workflow · logs

Review

  • Approvals: 0/1
  • Assigned reviewer: not assigned yet — comment /request-review to ask for one
  • Review target date: not set
  • Still needed: 1 more approval(s); an approval from a maintainer with admin or maintain permission
  • The core-maintainers pool is not staffed yet; an approver with admin or maintain permission is required instead.

Commands

Command Who What it does
/rerun-checks PR author, maintainers Re-runs failed or incomplete checks and re-evaluates this gate
/request-review PR author, maintainers Asks the review rotation to assign a reviewer (adds needs-reviewer)

Updated for 552dd12 · gate run · This comment is maintained automatically — see submission gate docs.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🔍 Vally Lint Results

✅ All checks passed

Scope Checked
Skills 1
Agents 0
Total 1
Severity Count
❌ Errors 0
⚠️ Warnings 0
ℹ️ Advisories 0

Summary

Level Finding
ℹ️ ✅ blind-test (2/2 checks passed)
ℹ️ ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️ ✓ spec-compliance: All spec checks passed.
ℹ️ ✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️ ✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️ 1 skill(s) linted, 1 passed
Full linter output
### Linting skills/blind-test
✅ blind-test (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed awaiting-automation labels Nov 3, 2026
@aaronpowell aaronpowell closed this Nov 4, 2026
@aaronpowell

Copy link
Copy Markdown
Contributor

The upstream README for the linked source repository routes readers to Agensi referral links for Design Taste Pro and an Agensi edition of Blind Test. That creates the same free-skill-to-paid-companion promotion path identified in the reviews of #6230 and #6231, so we can’t accept this contribution in the catalog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk:high new-submission PR adds at least one new contribution ready-for-review Submission passed intake validation and is ready for maintainer review skills PR touches skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants