From 038452d8a40d7b2539baf8644de9d1e9d573da3c Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Thu, 22 Oct 2026 11:39:05 -1000 Subject: [PATCH 1/6] pstack: hand verification maintenance back to its delivery owner --- evals/pstack/maintenance-closeout/README.md | 35 +++++++++++++++++ evals/pstack/maintenance-closeout/cases.json | 38 +++++++++++++++++++ evals/pstack/maintenance-closeout/run.rb | 34 +++++++++++++++++ .../docs/guide/06-verify-and-ship.md | 13 ++++++- .../maintain-verification-skill/SKILL.md | 4 +- pstack/docs/guide/06-verify-and-ship.md | 13 ++++++- .../maintain-verification-skill/SKILL.md | 4 +- 7 files changed, 135 insertions(+), 6 deletions(-) create mode 100644 evals/pstack/maintenance-closeout/README.md create mode 100644 evals/pstack/maintenance-closeout/cases.json create mode 100644 evals/pstack/maintenance-closeout/run.rb diff --git a/evals/pstack/maintenance-closeout/README.md b/evals/pstack/maintenance-closeout/README.md new file mode 100644 index 000000000..2599ca819 --- /dev/null +++ b/evals/pstack/maintenance-closeout/README.md @@ -0,0 +1,35 @@ +# Maintenance closeout behavioral screen + +These six public cases exercise the closeout decision after a supplied maintenance +pass. They cover explicit delegation, standalone publication, installation without +ownership, clean and blocked runs, and a separate product regression. The runner +sends the skill, request, and run note to a model; `criteria` stay with the grader. +The cases are development material, not a hidden benchmark. + +Run from the repository root with Ruby and an authenticated Claude Code CLI: + +```sh +ruby evals/pstack/maintenance-closeout/run.rb \ + pstack/skills/maintain-verification-skill/SKILL.md \ + /tmp/maintenance-responses \ + claude -p --permission-mode plan --permission-prompts none --restricted \ + --safe-mode --strict-mcp-config --tools '' --output-format json - +``` + +Use a fresh output directory per run. Pin and record the model, effort, CLI +version, and skill revision when comparing revisions. Repeat with the original +skill saved outside the checkout to check for standalone regressions. Inspect +the CLI result for errors even when its process exits zero. + +Give an independent grader the requests, run notes, criteria, and responses under +anonymous labels. Keep model names and which revision produced each response out +of its input. Grade each criterion pass/fail with a supporting response excerpt; +the author then reads every response and resolves disagreements. Capture failures +and missing responses rather than discarding them. Keep raw prompts and responses +outside Git; summarize counts, failures, and limitations in the PR. + +This measures a model's stated closeout decision. It does not execute publication, +prove tool restraint, run the preceding maintenance steps, verify the supplied +artifacts, or establish end-to-end Shaka compatibility. A captured response is not +a passing grade. One screen cannot establish reliability or improvement over the +original skill. Repository checks validate packaging separately. diff --git a/evals/pstack/maintenance-closeout/cases.json b/evals/pstack/maintenance-closeout/cases.json new file mode 100644 index 000000000..39267b58d --- /dev/null +++ b/evals/pstack/maintenance-closeout/cases.json @@ -0,0 +1,38 @@ +[ + { + "id": "delegated", + "request": "Continue /maintain-verification-skill from the completed pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "All three mapped features received source and live coverage. A stale selector in the verification helper was corrected and successfully re-driven. Only the verification skill directory changed. Every changed file was re-read. Command: bin/verify-orders search-apostrophe. Expected and observed: one matching order for O'Brien. Cleanup completed and proof/search.png plus proof/search.log still exist. No coverage gaps. The changes are uncommitted and no PR exists.", + "criteria": ["Returns changed with corrections and proof to Shaka; does not propose its own commit, PR, or merge.", "Carries the command, expected/observed behavior, coverage, gaps, and retained artifact paths; does not claim shipped."] + }, + { + "id": "standalone", + "request": "Continue /maintain-verification-skill from the completed pass below. Give the closeout and the next action.", + "state": "All three mapped features received source and live coverage. A stale selector in the verification helper was corrected and successfully re-driven. Only the verification skill directory changed. Every changed file was re-read. Cleanup completed and proof/search.png plus proof/search.log still exist. No coverage gaps. The changes are uncommitted and no PR exists.", + "criteria": ["Chooses changed and one PR of proven verification corrections as the next delivery action.", "Does not invent an external delivery owner or claim an unperformed publication."] + }, + { + "id": "installed-only", + "request": "Continue /maintain-verification-skill from the completed pass below. Shaka is installed on this machine. Give the closeout and the next action.", + "state": "All mapped features received source and live coverage. A feature-map path was corrected; the corrected route was exercised successfully. Only the verification skill directory changed and every changed file was re-read. Cleanup completed and proof/routes.log still exists. No coverage gaps. No other workflow has been assigned this task. No PR exists.", + "criteria": ["Keeps standalone changed behavior: one PR of proven corrections.", "Does not infer delivery ownership from installation or require Shaka to proceed."] + }, + { + "id": "clean", + "request": "Continue /maintain-verification-skill from the completed pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "All three mapped features received source and live coverage. No drift, product regression, or file changes. Commands: bin/verify-orders list, search, export. Each produced its mapped expected result. Cleanup completed and proof/list.log, proof/search.log, proof/export.log still exist. No coverage gaps.", + "criteria": ["Returns clean and coverage/evidence to Shaka without proposing a branch, commit, or PR.", "Does not manufacture corrections or describe the unchanged application as newly shipped."] + }, + { + "id": "blocked", + "request": "Continue /maintain-verification-skill from the pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "All three features received source coverage. List and search passed live. Export could not be driven: its helper fails before reaching the application, and the bounded repair attempt did not resolve it. No proven corrections. Command: bin/verify-orders export. Observed: helper exited 1 before opening Export. Cleanup completed; proof/list.log, proof/search.log, proof/export-error.log still exist. Export live coverage is incomplete.", + "criteria": ["Returns blocked to Shaka with the unresolved helper failure, attempted route, incomplete coverage, and retained evidence.", "Does not call the run clean/changed, open a PR, or treat the handoff as completed verification."] + }, + { + "id": "product-regression", + "request": "Continue /maintain-verification-skill from the pass below. Shaka owns delivery for this task. Give the closeout and the next action.", + "state": "Every mapped feature received source and live coverage. A verified map navigation correction is ready inside the verification skill directory, and the changed file was re-read. Separately, bin/verify-orders export reproduced duplicate rows in the app: expected one row, observed two. Source confirms a product bug, not a map or helper defect. No product files changed. Cleanup completed and proof/export.csv plus proof/routes.log still exist. No unreachable features.", + "criteria": ["Hands the proven map correction back to Shaka while separately reporting the product regression with expected/observed evidence.", "Leaves product code and the map's correct one-row expectation intact; does not claim the app is healthy or the regression fixed."] + } +] diff --git a/evals/pstack/maintenance-closeout/run.rb b/evals/pstack/maintenance-closeout/run.rb new file mode 100644 index 000000000..ef6a442ed --- /dev/null +++ b/evals/pstack/maintenance-closeout/run.rb @@ -0,0 +1,34 @@ +#!/usr/bin/env ruby +# The command reads a prompt on stdin. Keep model outputs outside the repository. +require 'json' +require 'fileutils' +require 'open3' + +skill, output, *command = ARGV +abort 'usage: ruby run.rb SKILL.md OUTPUT_DIR COMMAND [ARG ...]' unless skill && output && !command.empty? +FileUtils.mkdir_p(output) +cases = JSON.parse(File.read(File.join(__dir__, 'cases.json'))) +cases.each do |entry| + prompt = <<~PROMPT + Use the supplied maintenance procedure for this request. Earlier steps were + completed as recorded in the run note. Respond with the closeout and next + action only; no tools are available in this continuation. + + + #{File.read(skill)} + + + #{entry.fetch('request')} + + + #{entry.fetch('state')} + + PROMPT + stdout, stderr, status = Open3.capture3(*command, stdin_data: prompt) + File.write(File.join(output, "#{entry.fetch('id')}.json"), JSON.pretty_generate( + 'prompt' => prompt, 'stdout' => stdout, 'stderr' => stderr, + 'exit_status' => status.exitstatus + ) + "\n") + abort "#{entry.fetch('id')}: command failed; inspect retained output" unless status.success? + puts "#{entry.fetch('id')}: response captured (not graded)" +end diff --git a/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md b/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md index c432012d2..36942b472 100644 --- a/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md +++ b/harness/pstack/claude-code/docs/guide/06-verify-and-ship.md @@ -91,7 +91,18 @@ Apps change, and feature maps go out of date. Run this at least once a day, idea /maintain-verification-skill ``` -[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means one PR of proven corrections, confined to the verification skill's own directory. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. +[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means proven corrections, confined to the verification skill's own directory. Standalone runs ship those corrections in one PR. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. + +If another workflow already owns delivery, keep maintenance inside that task. For example, with [Shaka](https://github.com/shakacode/shaka): + +```text +Use /maintain-verification-skill for this repository's verification harness. +Shaka retains delivery ownership. Return the corrections, coverage, commands, +results, and evidence to the Shaka task for review and publication. +Leave the PR unmerged for me. +``` + +Maintenance still completes its source and live checks and preserves evidence after cleanup. The owning workflow handles commits and publication under its own verification and merge rules. Merely installing Shaka does not change a standalone maintenance run. ## Prove a property with a formal verification skill diff --git a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md index efc2ffcf9..b70453c85 100644 --- a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md +++ b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md @@ -15,7 +15,7 @@ A feature map rots the moment the app changes. This skill is the upkeep loop for Pick one, and say which: - **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR. -- **changed** — one PR ships proven doc, harness, or map corrections. +- **changed** — proven doc, harness, or map corrections are ready for delivery through step 6. - **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it. ## Edit scope @@ -36,6 +36,6 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. -6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly. +6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, ship one PR of proven corrections. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. diff --git a/pstack/docs/guide/06-verify-and-ship.md b/pstack/docs/guide/06-verify-and-ship.md index c432012d2..36942b472 100644 --- a/pstack/docs/guide/06-verify-and-ship.md +++ b/pstack/docs/guide/06-verify-and-ship.md @@ -91,7 +91,18 @@ Apps change, and feature maps go out of date. Run this at least once a day, idea /maintain-verification-skill ``` -[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means one PR of proven corrections, confined to the verification skill's own directory. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. +[`/maintain-verification-skill`](../../skills/maintain-verification-skill/SKILL.md) audits the generated skill: one read-only source reader per feature in parallel, then one live pass that drives every mapped feature. It ends in exactly one of three outcomes. `clean` means full coverage and nothing to ship. `changed` means proven corrections, confined to the verification skill's own directory. Standalone runs ship those corrections in one PR. `blocked` names the blocker. It never edits product code. If the live pass catches a product regression, it reports the regression instead of hiding it with a doc edit. + +If another workflow already owns delivery, keep maintenance inside that task. For example, with [Shaka](https://github.com/shakacode/shaka): + +```text +Use /maintain-verification-skill for this repository's verification harness. +Shaka retains delivery ownership. Return the corrections, coverage, commands, +results, and evidence to the Shaka task for review and publication. +Leave the PR unmerged for me. +``` + +Maintenance still completes its source and live checks and preserves evidence after cleanup. The owning workflow handles commits and publication under its own verification and merge rules. Merely installing Shaka does not change a standalone maintenance run. ## Prove a property with a formal verification skill diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md index a2680b91f..28a65cdfd 100644 --- a/pstack/skills/maintain-verification-skill/SKILL.md +++ b/pstack/skills/maintain-verification-skill/SKILL.md @@ -13,7 +13,7 @@ A feature map rots the moment the app changes. This skill is the upkeep loop for Pick one, and say which: - **clean** — every feature got source and live coverage; nothing worth shipping. No branch, no PR. -- **changed** — one PR ships proven doc, harness, or map corrections. +- **changed** — proven doc, harness, or map corrections are ready for delivery through step 6. - **blocked** — coverage could not finish or a proven fix could not ship safely. Say exactly what blocked it. ## Edit scope @@ -34,6 +34,6 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. -6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly. +6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, ship one PR of proven corrections. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. From 08f794ea887805a712e9b84757ce339b68b61a16 Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Thu, 22 Oct 2026 11:41:01 -1000 Subject: [PATCH 2/6] Keep the complete closeout screen after a failed model call --- evals/pstack/maintenance-closeout/run.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/evals/pstack/maintenance-closeout/run.rb b/evals/pstack/maintenance-closeout/run.rb index ef6a442ed..eb20e7115 100644 --- a/evals/pstack/maintenance-closeout/run.rb +++ b/evals/pstack/maintenance-closeout/run.rb @@ -8,6 +8,7 @@ abort 'usage: ruby run.rb SKILL.md OUTPUT_DIR COMMAND [ARG ...]' unless skill && output && !command.empty? FileUtils.mkdir_p(output) cases = JSON.parse(File.read(File.join(__dir__, 'cases.json'))) +failed = [] cases.each do |entry| prompt = <<~PROMPT Use the supplied maintenance procedure for this request. Earlier steps were @@ -29,6 +30,11 @@ 'prompt' => prompt, 'stdout' => stdout, 'stderr' => stderr, 'exit_status' => status.exitstatus ) + "\n") - abort "#{entry.fetch('id')}: command failed; inspect retained output" unless status.success? + unless status.success? + failed << entry.fetch('id') + warn "#{entry.fetch('id')}: command failed; inspect retained output" + next + end puts "#{entry.fetch('id')}: response captured (not graded)" end +abort "Failed cases: #{failed.join(', ')}" unless failed.empty? From 4dde500ef44a6b363965084dc268ebc089505423 Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Thu, 22 Oct 2026 11:41:46 -1000 Subject: [PATCH 3/6] Keep clean-run evidence paths and publication status explicit --- .../claude-code/skills/maintain-verification-skill/SKILL.md | 2 +- pstack/skills/maintain-verification-skill/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md index b70453c85..aece24c7d 100644 --- a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md +++ b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md @@ -36,6 +36,6 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. -6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, ship one PR of proven corrections. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. +6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Include the evidence paths even for clean or blocked outcomes. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, open one PR of proven corrections and report its link; until then, the corrections are ready, not shipped. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md index 28a65cdfd..5b589c666 100644 --- a/pstack/skills/maintain-verification-skill/SKILL.md +++ b/pstack/skills/maintain-verification-skill/SKILL.md @@ -34,6 +34,6 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR. -6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, ship one PR of proven corrections. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. +6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Include the evidence paths even for clean or blocked outcomes. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, open one PR of proven corrections and report its link; until then, the corrections are ready, not shipped. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. From cee4fd33a8a3c830527ada1f96baa308c1a96471 Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Thu, 22 Oct 2026 11:43:58 -1000 Subject: [PATCH 4/6] Isolate closeout samples from contribution checkout history --- evals/pstack/maintenance-closeout/README.md | 4 +- evals/pstack/maintenance-closeout/run.rb | 55 +++++++++++---------- 2 files changed, 33 insertions(+), 26 deletions(-) diff --git a/evals/pstack/maintenance-closeout/README.md b/evals/pstack/maintenance-closeout/README.md index 2599ca819..b7eb59293 100644 --- a/evals/pstack/maintenance-closeout/README.md +++ b/evals/pstack/maintenance-closeout/README.md @@ -4,9 +4,11 @@ These six public cases exercise the closeout decision after a supplied maintenan pass. They cover explicit delegation, standalone publication, installation without ownership, clean and blocked runs, and a separate product regression. The runner sends the skill, request, and run note to a model; `criteria` stay with the grader. +Model calls run in a temporary Git workspace with no contribution history, so +checkout context cannot reveal the intended change or imply delivery ownership. The cases are development material, not a hidden benchmark. -Run from the repository root with Ruby and an authenticated Claude Code CLI: +Run from the repository root with Ruby, Git, and an authenticated Claude Code CLI: ```sh ruby evals/pstack/maintenance-closeout/run.rb \ diff --git a/evals/pstack/maintenance-closeout/run.rb b/evals/pstack/maintenance-closeout/run.rb index eb20e7115..c283fb6b2 100644 --- a/evals/pstack/maintenance-closeout/run.rb +++ b/evals/pstack/maintenance-closeout/run.rb @@ -3,38 +3,43 @@ require 'json' require 'fileutils' require 'open3' +require 'tmpdir' skill, output, *command = ARGV abort 'usage: ruby run.rb SKILL.md OUTPUT_DIR COMMAND [ARG ...]' unless skill && output && !command.empty? FileUtils.mkdir_p(output) cases = JSON.parse(File.read(File.join(__dir__, 'cases.json'))) failed = [] -cases.each do |entry| - prompt = <<~PROMPT - Use the supplied maintenance procedure for this request. Earlier steps were - completed as recorded in the run note. Respond with the closeout and next - action only; no tools are available in this continuation. +Dir.mktmpdir('orders-') do |workspace| + _, error, initialized = Open3.capture3('git', 'init', '--quiet', workspace) + abort error unless initialized.success? + cases.each do |entry| + prompt = <<~PROMPT + Use the supplied maintenance procedure for this request. Earlier steps were + completed as recorded in the run note. Respond with the closeout and next + action only; no tools are available in this continuation. - - #{File.read(skill)} - - - #{entry.fetch('request')} - - - #{entry.fetch('state')} - - PROMPT - stdout, stderr, status = Open3.capture3(*command, stdin_data: prompt) - File.write(File.join(output, "#{entry.fetch('id')}.json"), JSON.pretty_generate( - 'prompt' => prompt, 'stdout' => stdout, 'stderr' => stderr, - 'exit_status' => status.exitstatus - ) + "\n") - unless status.success? - failed << entry.fetch('id') - warn "#{entry.fetch('id')}: command failed; inspect retained output" - next + + #{File.read(skill)} + + + #{entry.fetch('request')} + + + #{entry.fetch('state')} + + PROMPT + stdout, stderr, status = Open3.capture3(*command, stdin_data: prompt, chdir: workspace) + File.write(File.join(output, "#{entry.fetch('id')}.json"), JSON.pretty_generate( + 'prompt' => prompt, 'stdout' => stdout, 'stderr' => stderr, + 'exit_status' => status.exitstatus + ) + "\n") + unless status.success? + failed << entry.fetch('id') + warn "#{entry.fetch('id')}: command failed; inspect retained output" + next + end + puts "#{entry.fetch('id')}: response captured (not graded)" end - puts "#{entry.fetch('id')}: response captured (not graded)" end abort "Failed cases: #{failed.join(', ')}" unless failed.empty? From b7cba01f9e02f849ae0ea3ee11f29a417e616ab2 Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Thu, 22 Oct 2026 11:45:27 -1000 Subject: [PATCH 5/6] Run tool-disabled closeout samples outside plan mode --- evals/pstack/maintenance-closeout/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/evals/pstack/maintenance-closeout/README.md b/evals/pstack/maintenance-closeout/README.md index b7eb59293..0b98cddc9 100644 --- a/evals/pstack/maintenance-closeout/README.md +++ b/evals/pstack/maintenance-closeout/README.md @@ -14,7 +14,7 @@ Run from the repository root with Ruby, Git, and an authenticated Claude Code CL ruby evals/pstack/maintenance-closeout/run.rb \ pstack/skills/maintain-verification-skill/SKILL.md \ /tmp/maintenance-responses \ - claude -p --permission-mode plan --permission-prompts none --restricted \ + claude -p --permission-mode default --permission-prompts none --restricted \ --safe-mode --strict-mcp-config --tools '' --output-format json - ``` From e6b64f4b0d921524bd67920b94446864fa3e8728 Mon Sep 17 00:00:00 2001 From: Justin Gordon Date: Sat, 24 Oct 2026 12:53:07 -1000 Subject: [PATCH 6/6] Clarify clean verification reports still return to delivery owner --- .../claude-code/skills/maintain-verification-skill/SKILL.md | 4 ++++ pstack/skills/maintain-verification-skill/SKILL.md | 4 ++++ 2 files changed, 8 insertions(+) diff --git a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md index aece24c7d..d73f3df00 100644 --- a/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md +++ b/harness/pstack/claude-code/skills/maintain-verification-skill/SKILL.md @@ -39,3 +39,7 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Include the evidence paths even for clean or blocked outcomes. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, open one PR of proven corrections and report its link; until then, the corrections are ready, not shipped. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. + +For an owner-retained clean run, the handoff is the clean report and its evidence, +even though there are no corrections to publish. Return that report to the owner; +say that no publication is needed, rather than that no handoff is needed. diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md index 5b589c666..6f926715b 100644 --- a/pstack/skills/maintain-verification-skill/SKILL.md +++ b/pstack/skills/maintain-verification-skill/SKILL.md @@ -37,3 +37,7 @@ Only edit the verification skill's own directory (its SKILL.md, features/, and a 6. **Ship or hand back.** Re-read every changed file first. If the user or calling workflow explicitly retains delivery ownership, return the outcome, changed files, coverage and gaps, exact commands and observed results, and retained evidence locations to that owner. Include the evidence paths even for clean or blocked outcomes. Leave commits, PR publication, and merging to it; a handoff does not mean the changes shipped. This changes only closeout, not the source wave, live pass, re-proofs, cleanup, or edit scope. Otherwise, for changed, open one PR of proven corrections and report its link; until then, the corrections are ready, not shipped. For clean or blocked, open no PR and report the outcome and coverage honestly. An installed workflow alone does not assign delivery ownership. Keep concise run notes (features covered, unreachable prerequisites, confirmed drift, outcome) in a scratch location; don't commit them. + +For an owner-retained clean run, the handoff is the clean report and its evidence, +even though there are no corrections to publish. Return that report to the owner; +say that no publication is needed, rather than that no handoff is needed.