feat(adapters): Superpowers skill evaluation adapter

[NovusEdge]

Summary

Adds adapter to evaluate Superpowers skills against synthetic scenarios. Refs #132.

Changes

• skillopt_sleep/adapters/superpowers.py: SuperpowersEvaluator class

  • 5 embedded scenarios for verification-before-completion skill
  • Rule-based judge (contains, regex, order, any_of ops)
  • Proper harness integration: clones pinned Superpowers, overlays candidate to skills/<name>/SKILL.md, symlinks HOME/.claude/skills
  • Fail-closed: non-zero exit, timeout, missing binary all error
  • Token estimation and latency tracking

• tests/test_superpowers_scenarios.py: 20 tests

  • Judge logic coverage
  • Mocked integration tests proving overlay mechanism works

• scripts/smoke_superpowers.sh: Manual baseline vs candidate comparison

Usage

from skillopt_sleep.adapters.superpowers import SuperpowersEvaluator

evaluator = SuperpowersEvaluator(skill="verification-before-completion")
results = evaluator.evaluate(candidate_skill_path)
print(f"Score: {results.score}")  # 0.0-1.0

Or CLI:

python -m skillopt_sleep.adapters.superpowers --skill verification-before-completion --json

Scenarios

ID	Description
test-passes-verify	Fixes failing test, verifies it passes before claiming done
test-fails-no-claim	Reports failure honestly, no false completion claim
premature-claim-resist	Refuses to skip verification on adversarial prompt
partial-pass-honest	Reports partial pass accurately, not "all pass"
flaky-verify-rerun	Re-runs flaky test rather than trusting stale result

Test plan

• 20 offline tests pass (pytest tests/test_superpowers_scenarios.py)
• Manual smoke test with scripts/smoke_superpowers.sh (opt-in, not CI)

[Yif-Yang]

Thank you for picking up #132 and for iterating on the overlay/provenance ideas. There are tests here: we ran the 14 offline tests and the full suite (274 passed / 6 skipped). The current tests validate the rule matcher and scenario metadata, but they do not execute or validate the Superpowers integration path yet.

A clarification from our side: --target-skill-path is a SkillOpt-Sleep CLI option, not a Claude Code CLI option. The issue discussion did not make that distinction clearly enough. In the current adapter, Claude exits because it does not recognize that flag, so the candidate is not evaluated. The candidate is also copied to skills/SKILL.md rather than skills/verification-before-completion/SKILL.md, and the pinned Superpowers checkout is cloned but never loaded through its normal bootstrap/plugin/harness integration.

Before this can demonstrate Superpowers support, could you please add:

1. A real, supported Superpowers harness path using the pinned checkout and normal bootstrap/plugin integration, with only skills/verification-before-completion/SKILL.md overlaid in the temporary copy.
2. Fail-closed handling for non-zero Claude/git exits, timeout, malformed output, and missing scores. Please avoid inheriting unrelated host credentials and avoid unconditional --dangerously-skip-permissions.
3. Deterministic mocked integration tests that exercise the actual runner/overlay path, prove the candidate file is the one loaded, and prove the source checkout remains unchanged.
4. One opt-in real Claude Code smoke comparison (not public CI): baseline versus candidate with identical model/settings/tasks, raw outputs, and evidence that normal Superpowers behavior was active. A result showing no improvement is completely acceptable; the important part is proving the integration is real and reproducible.
5. Corrections to the synthetic scenarios: the math tests currently reference add without importing it, and the flaky helper sets TEST_RUN=1 before its first run, so those scenarios do not test the intended behaviors.
6. Please remove the unrelated CONTRIBUTING.md policy change and the new 5,906-line uv.lock unless a dependency change actually requires it, and update the PR description to match the current five scenarios/token fields.

As written in #132, testing a standalone skill-like prompt is not equivalent to running Superpowers. Once the real smoke and integration tests pass, we will be very happy to re-review and merge this through your original PR so your contribution is fully acknowledged.

[Yif-Yang]

Thanks for the substantial update. The candidate overlay path, invalid CLI flag, fail-closed process handling, scenario fixtures, mocked integration coverage, and unrelated-file cleanup are all meaningful improvements. We re-ran the branch: the 20 focused tests pass, and the suite also passes when combined with current main (334 passed / 6 skipped).

A few issues still block merging the adapter as a reliable Superpowers evaluator:

1. The negative judge rules are currently ineffective. _score_check() treats contains and not_contains arguments as literal strings, but scenarios pass values such as "all tests pass|done|complete|fixed". An output containing “Done” or “all tests pass” therefore still passes that not_contains check. The flaky scenario can also satisfy its second check merely by reporting the first-run message, without demonstrating an actual rerun. Please make the alternatives explicit (or implement well-tested alternative matching) and add regression cases that fail on false completion claims and on a missing rerun.

2. The new integration tests mock subprocess.run, so they prove file placement but not that Claude actually loaded Superpowers or the overlaid candidate. The manual smoke remains unchecked; it uses || true, and the JSON result omits ScenarioResult.output, so it cannot currently supply the requested raw evidence. Please run one reproducible baseline-versus-candidate smoke with the same model/settings/tasks, preserve the raw outputs, fail on runner errors, and show that normal Superpowers behavior and the candidate skill were active. A result showing no improvement is completely acceptable.

3. _run_scenario() still copies the full host environment and invokes Claude with unconditional --dangerously-skip-permissions. Because the candidate skill is input to the agent, this can expose host credentials/files to untrusted candidate instructions. Please use a constrained execution boundary and a scrubbed environment rather than an unconditional permission bypass.

The feature direction remains valuable, and we would like to merge it through your PR so the contribution stays fully attributed. Once these focused correctness, live-integration, and safety points are addressed, we will re-review promptly.