diff --git a/evals/pstack/maintenance-closeout/README.md b/evals/pstack/maintenance-closeout/README.md new file mode 100644 index 000000000..0b98cddc9 --- /dev/null +++ b/evals/pstack/maintenance-closeout/README.md @@ -0,0 +1,37 @@ +# Maintenance closeout behavioral screen + +These six public cases exercise the closeout decision after a supplied maintenance +pass. They cover explicit delegation, standalone publication, installation without +ownership, clean and blocked runs, and a separate product regression. The runner +sends the skill, request, and run note to a model; `criteria` stay with the grader. +Model calls run in a temporary Git workspace with no contribution history, so +checkout context cannot reveal the intended change or imply delivery ownership. +The cases are development material, not a hidden benchmark. + +Run from the repository root with Ruby, Git, and an authenticated Claude Code CLI: + +```sh +ruby evals/pstack/maintenance-closeout/run.rb \ + pstack/skills/maintain-verification-skill/SKILL.md \ + /tmp/maintenance-responses \ + claude -p --permission-mode default --permission-prompts none --restricted \ + --safe-mode --strict-mcp-config --tools '' --output-format json - +``` + +Use a fresh output directory per run. Pin and record the model, effort, CLI +version, and skill revision when comparing revisions. Repeat with the original +skill saved outside the checkout to check for standalone regressions. Inspect +the CLI result for errors even when its process exits zero. + +Give an independent grader the requests, run notes, criteria, and responses under +anonymous labels. Keep model names and which revision produced each response out +of its input. Grade each criterion pass/fail with a supporting response excerpt; +the author then reads every response and resolves disagreements. Capture failures +and missing responses rather than discarding them. Keep raw prompts and responses +outside Git; summarize counts, failures, and limitations in the PR. + +This measures a model's stated closeout decision. It does not execute publication, +prove tool restraint, run the preceding maintenance steps, verify the supplied +artifacts, or establish end-to-end Shaka compatibility. A captured response is not +a passing grade. One screen cannot establish reliability or improvement over the +original skill. Repository checks validate packaging separately. diff --git a/evals/pstack/maintenance-closeout/cases.json b/evals/pstack/maintenance-closeout/cases.json new file mode 100644 index 000000000..39267b58d --- /dev/null +++ b/evals/pstack/maintenance-closeout/cases.json @@ -0,0 +1,38 @@ +[ + { + "id": "delegated", + "request": "Continue /maintain-verification-skill from the completed pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "All three mapped features received source and live coverage. A stale selector in the verification helper was corrected and successfully re-driven. Only the verification skill directory changed. Every changed file was re-read. Command: bin/verify-orders search-apostrophe. Expected and observed: one matching order for O'Brien. Cleanup completed and proof/search.png plus proof/search.log still exist. No coverage gaps. The changes are uncommitted and no PR exists.", + "criteria": ["Returns changed with corrections and proof to Shaka; does not propose its own commit, PR, or merge.", "Carries the command, expected/observed behavior, coverage, gaps, and retained artifact paths; does not claim shipped."] + }, + { + "id": "standalone", + "request": "Continue /maintain-verification-skill from the completed pass below. Give the closeout and the next action.", + "state": "All three mapped features received source and live coverage. A stale selector in the verification helper was corrected and successfully re-driven. Only the verification skill directory changed. Every changed file was re-read. Cleanup completed and proof/search.png plus proof/search.log still exist. No coverage gaps. The changes are uncommitted and no PR exists.", + "criteria": ["Chooses changed and one PR of proven verification corrections as the next delivery action.", "Does not invent an external delivery owner or claim an unperformed publication."] + }, + { + "id": "installed-only", + "request": "Continue /maintain-verification-skill from the completed pass below. Shaka is installed on this machine. Give the closeout and the next action.", + "state": "All mapped features received source and live coverage. A feature-map path was corrected; the corrected route was exercised successfully. Only the verification skill directory changed and every changed file was re-read. Cleanup completed and proof/routes.log still exists. No coverage gaps. No other workflow has been assigned this task. No PR exists.", + "criteria": ["Keeps standalone changed behavior: one PR of proven corrections.", "Does not infer delivery ownership from installation or require Shaka to proceed."] + }, + { + "id": "clean", + "request": "Continue /maintain-verification-skill from the completed pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "All three mapped features received source and live coverage. No drift, product regression, or file changes. Commands: bin/verify-orders list, search, export. Each produced its mapped expected result. Cleanup completed and proof/list.log, proof/search.log, proof/export.log still exist. No coverage gaps.", + "criteria": ["Returns clean and coverage/evidence to Shaka without proposing a branch, commit, or PR.", "Does not manufacture corrections or describe the unchanged application as newly shipped."] + }, + { + "id": "blocked", + "request": "Continue /maintain-verification-skill from the pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "All three features received source coverage. List and search passed live. Export could not be driven: its helper fails before reaching the application, and the bounded repair attempt did not resolve it. No proven corrections. Command: bin/verify-orders export. Observed: helper exited 1 before opening Export. Cleanup completed; proof/list.log, proof/search.log, proof/export-error.log still exist. Export live coverage is incomplete.", + "criteria": ["Returns blocked to Shaka with the unresolved helper failure, attempted route, incomplete coverage, and retained evidence.", "Does not call the run clean/changed, open a PR, or treat the handoff as completed verification."] + }, + { + "id": "product-regression", + "request": "Continue /maintain-verification-skill from the pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "Every mapped feature received source and live coverage. A verified map navigation correction is ready inside the verification skill directory, and the changed file was re-read. Separately, bin/verify-orders export reproduced duplicate rows in the app: expected one row, observed two. Source confirms a product bug, not a map or helper defect. No product files changed. Cleanup completed and proof/export.csv plus proof/routes.log still exist. No unreachable features.", + "criteria": ["Hands the proven map correction back to Shaka while separately reporting the product regression with expected/observed evidence.", "Leaves product code and the map's correct one-row expectation intact; does not claim the app is healthy or the regression fixed."] + } +] diff --git a/evals/pstack/maintenance-closeout/run.rb b/evals/pstack/maintenance-closeout/run.rb new file mode 100644 index 000000000..c283fb6b2 --- /dev/null +++ b/evals/pstack/maintenance-closeout/run.rb @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# The command reads a prompt on stdin. Keep model outputs outside the repository. +require 'json' +require 'fileutils' +require 'open3' +require 'tmpdir' + +skill, output, *command = ARGV +abort 'usage: ruby run.rb SKILL.md OUTPUT_DIR COMMAND [ARG ...]' unless skill && output && !command.empty? +FileUtils.mkdir_p(output) +cases = JSON.parse(File.read(File.join(__dir__, 'cases.json'))) +failed = [] +Dir.mktmpdir('orders-') do |workspace| + _, error, initialized = Open3.capture3('git', 'init', '--quiet', workspace) + abort error unless initialized.success? + cases.each do |entry| + prompt = <<~PROMPT + Use the supplied maintenance procedure for this request. Earlier steps were + completed as recorded in the run note. Respond with the closeout and next + action only; no tools are available in this continuation. + + + #{File.read(skill)} + + + #{entry.fetch('request')} + + + #{entry.fetch('state')} + + PROMPT + stdout, stderr, status = Open3.capture3(*command, stdin_data: prompt, chdir: workspace) + File.write(File.join(output, "#{entry.fetch('id')}.json"), JSON.pretty_generate( + 'prompt' => prompt, 'stdout' => stdout, 'stderr' => stderr, + 'exit_status' => status.exitstatus + ) + "\n") + unless status.success? + failed << entry.fetch('id') + warn "#{entry.fetch('id')}: command failed; inspect retained output" + next + end + puts "#{entry.fetch('id')}: response captured (not graded)" + end +end +abort "Failed cases: #{failed.join(', ')}" unless failed.empty? diff --git a/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md b/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md index c432012d2..36942b472 100644 --- a/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md +++ b/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md @@ -91,7 +91,18 @@ Apps change, and feature maps go out of date. Run this at least once a day, idea /maintain-verification-skill ``` -[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means one PR of proven corrections, confined to the verification skill's own directory. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. +[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means proven corrections, confined to the verification skill's own directory. Standalone runs ship those corrections in one PR. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. + +If another workflow already owns delivery, keep maintenance inside that task. For example, with [Shaka](https://github.com/shakacode/shaka): + +```text +Use /maintain-verification-skill for this repository's verification harness. +Shaka retains delivery ownership. Return the corrections, coverage, commands, +results, and evidence to the Shaka task for review and publication. +Leave the PR unmerged for me. +``` + +Maintenance still completes its source and live checks and preserves evidence after cleanup. The owning workflow handles commits and publication under its own verification and merge rules. Merely installing Shaka does not change a standalone maintenance run. ## Prove a property with a formal verification skill diff --git a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md index efc2ffcf9..d73f3df00 100644 --- a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md +++ b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md @@ -15,7 +15,7 @@ A feature map rots the moment the app changes. This skill is the upkeep loop for Pick one, and say which: - **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR. -- **changed** — one PR ships proven doc, harness, or map corrections. +- **changed** — proven doc, harness, or map corrections are ready for delivery through step 6. - **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it. ## Edit scope @@ -36,6 +36,10 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. -6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly. +6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Include the evidence paths even for clean or blocked outcomes. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, open one PR of proven corrections and report its link; until then, the corrections are ready, not shipped. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. + +For an owner-retained clean run, the handoff is the clean report and its evidence, +even though there are no corrections to publish. Return that report to the owner; +say that no publication is needed, rather than that no handoff is needed. diff --git a/pstack/docs/guide/06-verify-and-ship.md b/pstack/docs/guide/06-verify-and-ship.md index c432012d2..36942b472 100644 --- a/pstack/docs/guide/06-verify-and-ship.md +++ b/pstack/docs/guide/06-verify-and-ship.md @@ -91,7 +91,18 @@ Apps change, and feature maps go out of date. Run this at least once a day, idea /maintain-verification-skill ``` -[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means one PR of proven corrections, confined to the verification skill's own directory. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. +[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means proven corrections, confined to the verification skill's own directory. Standalone runs ship those corrections in one PR. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. + +If another workflow already owns delivery, keep maintenance inside that task. For example, with [Shaka](https://github.com/shakacode/shaka): + +```text +Use /maintain-verification-skill for this repository's verification harness. +Shaka retains delivery ownership. Return the corrections, coverage, commands, +results, and evidence to the Shaka task for review and publication. +Leave the PR unmerged for me. +``` + +Maintenance still completes its source and live checks and preserves evidence after cleanup. The owning workflow handles commits and publication under its own verification and merge rules. Merely installing Shaka does not change a standalone maintenance run. ## Prove a property with a formal verification skill diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md index a2680b91f..6f926715b 100644 --- a/pstack/skills/maintain-verification-skill/SKILL.md +++ b/pstack/skills/maintain-verification-skill/SKILL.md @@ -13,7 +13,7 @@ A feature map rots the moment the app changes. This skill is the upkeep loop for Pick one, and say which: - **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR. -- **changed** — one PR ships proven doc, harness, or map corrections. +- **changed** — proven doc, harness, or map corrections are ready for delivery through step 6. - **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it. ## Edit scope @@ -34,6 +34,10 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. -6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly. +6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Include the evidence paths even for clean or blocked outcomes. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, open one PR of proven corrections and report its link; until then, the corrections are ready, not shipped. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. + +For an owner-retained clean run, the handoff is the clean report and its evidence, +even though there are no corrections to publish. Return that report to the owner; +say that no publication is needed, rather than that no handoff is needed.