diff --git a/README.ko.md b/README.ko.md index 93939f2..bba292b 100644 --- a/README.ko.md +++ b/README.ko.md @@ -113,7 +113,8 @@ git이 추적하는 폴더로 지정하세요. - **독립된 눈.** scout, 리뷰어, QA는 작업을 만든 컨텍스트가 아닌 새 컨텍스트에서 돕니다. - **주장 대신 증거.** 실행 환경이 없으면 PASS가 아니라 BLOCKED입니다. 버그 수정에는 - 수정 전에 실패한 테스트가 있어야 합니다. + 수정 전에 실패한 테스트가 있어야 하고, 새 동작의 테스트는 그 동작 없이 한 번 + 실패하는 것을 확인합니다. - **검증한 것만 납품.** 납품한 트리가 검증한 커밋과 다르면 `close`가 거부합니다. Build는 [ponytail](https://github.com/DietrichGebert/ponytail) 방식(가장 작은 diff --git a/README.md b/README.md index ca59856..248ab73 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,8 @@ vendors e2e's agent guide (Apache-2.0, pinned) so it works offline; - **Independent eyes.** Scouts, reviewer and QA run in fresh contexts, never in the one that wrote the work. - **Proof, not claims.** No environment means BLOCKED, never PASS. A bug fix - needs a test that failed before the fix. + needs a test that failed before the fix; a test for new behavior is seen + failing once without it. - **Delivered means what was verified.** `close` refuses a delivered tree that differs from the verified commit. diff --git a/SKILL.md b/SKILL.md index df1b6cb..7d52b5f 100644 --- a/SKILL.md +++ b/SKILL.md @@ -13,7 +13,7 @@ governs phase order, gates and proof. |---|---|---|---|---| | 1 | Intent | `phases/1-intent.md` | `intent.md`, `refs/`, `logs/` | Clarity 90%, intent approved | | 2 | Plan | `phases/2-plan.md` | `refs/`, `PLAN.md` | plan approved | -| 3 | Build | `phases/3-build.md` | code, `review.md` | review PASS | +| 3 | Build | `phases/3-build.md` | code, `review.md`, `logs/` | review PASS | | 4 | Evidence | `phases/4-evidence.md` | `evidence.md`, `logs/` | `sdlc.sh score` routes | `sdlc.sh next ` names the next step and why. Run `sdlc.sh` (in this diff --git a/VERSION b/VERSION index 6a6a3d8..24ba9a3 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.6.1 +2.7.0 diff --git a/docs/index.html b/docs/index.html index 420befe..ee94410 100644 --- a/docs/index.html +++ b/docs/index.html @@ -337,7 +337,7 @@

What you get

Proof behind every "passed"
-
Every pass points to the record of what was run or seen. If there is nowhere to try something, it is marked as blocked, never as passed. A bug fix needs a test that failed before the fix and passes after it. The list of scenarios becomes your own checklist.
+
Every pass points to the record of what was run or seen. If there is nowhere to try something, it is marked as blocked, never as passed. A bug fix needs a test that failed before the fix and passes after it, and a test for new behavior is seen failing once without it. The list of scenarios becomes your own checklist.
Delivered means what was verified
@@ -461,7 +461,7 @@

For the person who sets it up

"g3-t": '다른 AI가 따로 점검합니다', "g3-d": '조사하는 보조 AI, 검토 AI, 시험 AI는 모두 처음부터 새로 시작합니다. 점검하는 대상을 만든 AI가 맡는 일은 없고, 결과를 고치지도 않습니다. 별도의 AI를 쓸 수 없으면 그 빈틈을 기록합니다.', "g4-t": '"통과"에는 증거가 붙습니다', - "g4-d": '모든 통과에는 무엇을 실행하거나 보았는지 기록이 붙습니다. 시험해 볼 곳이 없으면 통과가 아니라 "막힘"으로 표시합니다. 오류 수정에는 고치기 전에는 실패하고 고친 뒤에는 통과하는 테스트가 있어야 합니다. 시나리오 목록은 그대로 담당자의 확인 목록이 됩니다.', + "g4-d": '모든 통과에는 무엇을 실행하거나 보았는지 기록이 붙습니다. 시험해 볼 곳이 없으면 통과가 아니라 "막힘"으로 표시합니다. 오류 수정에는 고치기 전에는 실패하고 고친 뒤에는 통과하는 테스트가 있어야 하고, 새 동작의 테스트는 그 동작 없이 한 번 실패하는 것을 확인합니다. 시나리오 목록은 그대로 담당자의 확인 목록이 됩니다.', "g5-t": '검증한 것만 납품합니다', "g5-d": '납품한 결과가 검사한 버전과 다르면 작업을 닫을 수 없습니다. 위험한 작업은 어떤 설정에서든 사람의 허락이 먼저 필요합니다. 데이터 삭제, 저장된 데이터의 구조 변경, 다른 시스템이 기대는 약속의 변경, 보안, 인프라, 이 컴퓨터 밖으로 작업을 내보내는 일이 여기에 해당합니다. 작업 중에 읽은 글은 참고 자료로만 다루고, 지시는 사람에게서만 받습니다.', "lim-h": '할 수 없는 일', diff --git a/log/changelog-v2.7.0.md b/log/changelog-v2.7.0.md new file mode 100644 index 0000000..9f30d91 --- /dev/null +++ b/log/changelog-v2.7.0.md @@ -0,0 +1,79 @@ +# v2.7.0 — keep the contrast: failing tests and dated facts + +Three rules make a defect, a test and a fact each carry the run that proves +it. A reviewer's caught defect becomes a test, a new test is seen failing +once, and a fact in `knowledge` names the commit it was verified at. + +## Why + +- Across three projects, 17 features ran v2 reviews; 11 had at least one + FAIL verdict, 26 `VERDICT: FAIL` rounds in total, 14 on plans and 12 on + diffs. Only Evidence's code FAILs had to become regression tests. Of the 10 + diff-review findings that were a code defect or a weak test, the next round + cited a new or changed test aimed at the finding for 3 and not for 7 + (counted from `review.md` citations, not from the test files); for the 7 + the defect lived on only as prose in `review.md`. +- v2.5.1's false e2e pass (a URL assertion matching the address before a + client-side redirect) was found only by a negative control. One project + already proved its tests that way: head PASS 12/12, base FAIL 12/12. +- Facts in `knowledge` cite files and lines but not the commit they were + verified at, so a scout could cite one as AS-IS after the code changed. +- The recurring FAIL categories across features — missed consumers (5 + features), intent or policy mismatches (7), scenario gaps (4), code defects + (4), weak tests (3) — are each already targeted by an existing reviewer or + scout rule. A lesson store would only repeat them. + +The idea came from reviewing a proposal built on Harness² (arXiv +2610.14224, "Recursive Agent Harnessing for an Open World", 2026-10-09), +which keeps a lesson only when an execution contrast supports it. These +rules keep that contrast in the project's own tests and on the fact itself. +The proposal's lesson store, context packer and auto-promotion scripts were +not adopted: v2.0.0 removed that kind of structure. + +## Changes + +- **A blocking diff finding that shows a defect becomes a test.** In Build, + a blocking diff-review finding that a screen, API, job or log could show + is fixed like a bug fix: its regression test first, failing on the + FAILed commit for the finding's reason, then the fix. If no test can show + it (no test runner, a production-only symptom), the fix's commit message + (or, for an uncommitted review, the review request) says why. Plan-review FAILs and findings nothing can observe (wording, + docs, structure) are excluded. The reviewer blocks an observable blocking + diff finding left untested and unexplained. Non-blocking `(observable)` + findings still become QA scenarios, as in v2.4.0. +- **A new test for new behavior is seen failing once.** Build runs each new + or changed test for new behavior once where it must fail: on the base + commit in a disposable worktree (never stashing or touching the human's + tree, as QA already does), or, for a new symbol, with the behavior + disabled. The output goes to `logs/`. It must fail on its assertion; + "symbol not found" or a compile error does not count. The reviewer now + receives `logs/` and blocks a new-behavior test with no recorded assertion + failure. SKILL.md lists `logs/` among Build's writes. +- **Facts carry the commit they hold at.** Evidence writes each fact that + rests on code into `knowledge` with its file (path from the project root) + and the delivered commit of that file's repository, + `[verified: @ ]`, re-checking the fact on that commit + first when the file changed since the fact was checked (this change, a + rebase or a merge), so a fact verified in Plan against older code is not + stamped with the new commit. Before citing a + code fact, a scout checks whether its file changed since that commit (in + the file's repository, `git log --oneline .. -- ` prints + anything); if it changed, or the fact has no commit, the scout re-checks it + against the code or cites it as `[assumed: ]`. Running `git log` in + the file's own repository makes this work for one repository or a root + holding several. Facts not about code (policies, data) are not checked + this way, and only facts a scout cites are checked; there is no list of + watched paths and no cache. + +No script change; existing records stay valid. + +## Known limits + +- Facts already in `knowledge` have no commit; each is re-checked whenever + a scout cites it, until Evidence records it again with its commit. +- A fact that rests on several files carries one file; a change to another + file it depends on is not caught. +- The plan load is at 2500 of 2500 words; the next rule for scouts, + reviewer, plan or SKILL.md must replace words, not add them. +- Whether a finding could be observed is a judgment, as `(observable)` + already is. diff --git a/phases/3-build.md b/phases/3-build.md index d5924f3..e094871 100644 --- a/phases/3-build.md +++ b/phases/3-build.md @@ -16,6 +16,11 @@ Changes: codebase's structure and conventions. - Bug fix: write the regression test first and watch it fail for the reported reason; fix the root cause once, in the shared code. +- A new or changed test for new behavior: run it once where it must fail, on + the base commit in a disposable worktree (never stash or touch the human's + tree) or, for a new symbol, with the behavior disabled; keep the output in + `logs/`. It must fail on its assertion, not on a missing symbol or compile + error. - Never cut validation at trust boundaries, error handling that prevents data loss, security or accessibility. - After each step run the narrowest check; run the full `build`, `test` and @@ -29,7 +34,7 @@ Changes: Commit the change locally (the project's rules decide the message), then dispatch the reviewer (`roles/reviewer.md`, diff mode) with intent.md, -PLAN.md, `refs/` and the commit. Where every commit needs the human's +PLAN.md, `refs/`, `logs/` and the commit. Where every commit needs the human's approval, review the uncommitted change instead. The reviewer appends a round to `review.md` ending in `VERDICT: PASS` or `VERDICT: FAIL`. Record it with `sdlc.sh review pass|fail ""`: the script reads the verdict @@ -53,7 +58,13 @@ needs a new round, as does any code changed after a PASS; rewording the commit made on top of it does not. Recording hashes those files into the repository's object store (not its index): ignore large or secret files before reviewing. -FAIL → fix the blocking findings, commit, review again. The third FAIL stops +FAIL → fix the blocking findings, commit, review again. A blocking diff +finding that a screen, API, job or log could show is a bug fix: its +regression test first, failing on the FAILed commit for the finding's reason, +then the fix; if no test can show it, the fix's commit message (or, for an +uncommitted review, the review request) says why. +Plan-review FAILs and findings nothing can observe (wording, +docs, structure) need no test. The third FAIL stops the loop: show the human the trail; only `sdlc.sh approve plan --human` resumes it. Then phase 4. diff --git a/phases/4-evidence.md b/phases/4-evidence.md index a862f59..e4195c4 100644 --- a/phases/4-evidence.md +++ b/phases/4-evidence.md @@ -52,7 +52,10 @@ where it is written in the same delivery, or its follow-up and owner go in Delivery's `Policy:` line. Then: - move verified facts from the refs (and `refs/learnings.md`, if any) into `knowledge`: one place per fact, updating a page rather than adding a - duplicate; + duplicate; a fact that rests on code ends with its file (path from the + project root) and the delivered commit of that file's repository, after + re-checking the fact there if the file changed since it was checked: + `[verified: @ ]`; - add the saved scripts (`logs/scripts/`) worth keeping to the project's `e2e` suite, or name them as a follow-up in Delivery; - `sdlc.sh close shipped ""` — refused if the delivered diff --git a/roles/reviewer.md b/roles/reviewer.md index c58df3f..bc49168 100644 --- a/roles/reviewer.md +++ b/roles/reviewer.md @@ -24,7 +24,7 @@ Inputs: intent.md, PLAN.md, `refs/`. Blocking: ## Diff mode (phase 3) -Inputs: intent.md, PLAN.md, `refs/`, the commit or uncommitted diff +Inputs: intent.md, PLAN.md, `refs/`, `logs/`, the commit or uncommitted diff (untracked files included). Read every changed file. Blocking: - it does not do what PLAN.md says, or does more; @@ -38,7 +38,8 @@ Blocking: - security: injection, missing authorization, secrets in code, unsafe deserialization; - a test that cannot fail or asserts nothing; a bug fix without a test that - failed before it; + failed before it; a new-behavior test with no recorded assertion failure; + an observable blocking diff finding untested and unexplained; - a name that lies, one rule decided in two places, a hardcoded environment value, dead code the change left behind; - an abstraction, option or layer PLAN.md did not ask for. diff --git a/roles/scout.md b/roles/scout.md index 0cdc5b9..3c756fa 100644 --- a/roles/scout.md +++ b/roles/scout.md @@ -6,7 +6,9 @@ under `refs/`. Change nothing else. Inputs: the question, intent.md, the project root, the `knowledge` folder, and config.md. -- Read `knowledge` first; cite what it already answers. +- Read `knowledge` first; cite what it already answers. Re-check, or mark + assumed, a cited code fact with no commit or whose file changed since (its + repository's `git log`). - Trace from the entry the question names to the end: every caller in every tier and repository, every reader and writer of the data, the states existing records are in. Query real data read-only where a tool and