From fc265ceddef7c1c419a497dc5e393901e0ab5a6f Mon Sep 17 00:00:00 2001 From: Remi Antczak <140408771+remigiusz-antczak@users.noreply.github.com> Date: Fri, 30 Oct 2026 06:42:36 +0100 Subject: [PATCH] Wave 6: decision-council surface fixes, standing every-test-wired check, locale-stable gates, audit-pr footer markup Decisions (idea-critic): - veche.py --help: --option states the exact name rule (the TOKEN pattern, which the refusal for a bad name now quotes too); --stakes and --reversibility state the real gate (the council is skipped only when stakes are low or medium and reversibility is easy, and either flag alone convenes); --json-out says how the file is written (refs #2136, #2139). - --json-out writes through a temp file in the same directory and os.replace, at the early write and the final one and on every outcome, so a reader or a crash never sees half a record. A symlinked path keeps pointing at its target, an existing file keeps its mode, a device or FIFO target is written in place, and an unwritable path still exits 2 with the "--json-out: cannot write" message. The write needs a writable directory: a read-only file in one is replaced, while a read-only directory or a bind-mounted file exits 2, and --help says so (refs #2139). - A --json-out write that fails at the end of the run no longer turns a blocking verdict into exit 2: COULD_NOT_DECIDE keeps exit 3 and NOT_CONVENED keeps exit 4; only a verdict that would exit 0 becomes exit 2. - A decided record carries kill_criteria_by_lens ({lens: text}), each seat's own kill criterion as worded before near-duplicates collapse. It is additive: kill_criteria and validate_verdict.py are unchanged, and the evil twin's criterion stays in minority_report.twin (refs #2140). - The kill-criteria dedupe key maps the Unicode comparison signs to <= and >= first, so the same tripwire written both ways is kept once, while opposite or strict-versus-inclusive comparisons stay two. - council.md states the reply contract a custom --cmd must meet, as the parser really accepts it: a JSON object with a verdict and string-only claims items, prose and stray braces around it skipped, the first object with a valid verdict wins, and the scores map may sit under scores in that verdict object or in any other object (refs #2137). - New scripts/test_veche_doc_claims.py pins the --help text, the read-only file and directory behaviors and the council.md wording. Delivery checks: - scripts/ci-gates.sh tests-wired: a standing check that every scripts/test_*.py and scripts/test-*.sh is run by CI (a ci-tests.d suite, a workflow step, or a test those run names it on a non-comment line); otherwise it needs a namereason row in scripts/ci-test-allowlist.tsv. A row without a reason, for a missing file, or for a test CI already runs also fails, so the list only shrinks (refs #1592). - Its first run found 71 tests no CI step ran: 59 are now registered in four suites (one of them a background suite), and the 12 corpus re-scores of 30 s or more are allowlisted with their measured runtimes. - Four of the newly registered tests timed a guard on 100 KB lines against 0.5 s or 0.2 s (test_destructive_guard_w20.py and test_secret_extra_rules_w12/w13/w14.py); they now allow 2.0 s, so a loaded CI runner does not fail them, while a backtracking regression still takes far longer. - scripts/ci-gates.sh rules compares its sorted rows in the C locale, like the sort -u calls that feed them. Under en_US.UTF-8 its comm calls saw unsorted input, so a short baseline could report a baselined rule as new or a shrunk baseline as grown. The shipped baseline is empty, so nothing failed yet. The tests-wired comm calls use the same locale. Both locale tests print a SKIP line and still run the C-locale cases on a runner without en_US.UTF-8. - test_auto_update_legacy_marker.py skips docs/bench (about 90k files) and runs in about 4 s instead of about 326 s; it, test_install_canonical_global.py and test_peer_pains_5.py set DCR_NO_PROBE=1, so none of them calls github.com through the sandbox probe. Communication structure: - comms_lint.py audit-pr: the attribution footer may sit in bounded markup (, , , *, _), so a compliant body that ends with one no longer fails "Not fixed carries detail". A wrapped line that is not a Generated or Co-Authored-By line, a Generated line with no rule or robot line before it, and anything after a footer are still checked. artifact-templates.md states the rule. Review hub: - test-review-checks-scan-summary.sh unsets REVIEW_SEMGREP_REGISTRY, so a host that sets it to 0 still passes; a new suite runs it with the variable set. Tests: 41/41 affected CI suites pass, run one at a time through scrub_env.sh on da58c6b9e (these changes before the 1.622.0 stamp was merged under them). 39 passed in the batch run. secret-rule-waves.sh failed there because the worktree lacked the gitignored .banlist.local.txt that the privacy gate requires when the main checkout has one (it fails closed), and security-gate-waves.bg.sh failed once when the wall-clock case of test_destructive_guard_w21.py did not end within its 20 s wait at a 15-minute load average near 13. Both passed when re-run alone, with no code change. test-ci-gates.sh is left to CI. Co-Authored-By: Claude Opus 5.5 --- .../skills/communication-structure/INDEX.md | 2 +- .../references/artifact-templates.md | 4 +- .../scripts/comms_lint.py | 13 +- .claude/skills/idea-critic/INDEX.md | 2 +- .../skills/idea-critic/references/council.md | 10 +- .claude/skills/idea-critic/scripts/veche.py | 55 +++-- SHA256SUMS | 12 +- changelog.d/rules-gate-locale.md | 2 + changelog.d/tests-wired.md | 6 + changelog.d/veche-surface-leftovers.md | 9 + changelog.d/w6-review-fixes.md | 7 + changelog.d/w6a-review-blues.md | 7 + scripts/ci-gates.sh | 77 ++++++- scripts/ci-test-allowlist.tsv | 17 ++ scripts/ci-tests.d/audit-pr-footer-markup.sh | 2 + scripts/ci-tests.d/orphan-tests-misc.sh | 8 + scripts/ci-tests.d/orphan-tests.sh | 2 +- ...review-checks-scan-summary-registry-env.sh | 2 + scripts/ci-tests.d/rules-locale.sh | 2 + scripts/ci-tests.d/secret-rule-waves.sh | 16 ++ scripts/ci-tests.d/security-gate-waves.bg.sh | 40 ++++ scripts/ci-tests.d/tests-wired.sh | 3 + scripts/ci-tests.d/veche-doc-claims.sh | 2 + .../ci-tests.d/veche-json-out-exit-code.sh | 2 + scripts/ci-tests.d/veche-surface.sh | 2 + scripts/ci-tests.d/veche-uniq-unicode-fold.sh | 2 + scripts/size-budgets.tsv | 4 +- scripts/test-review-checks-scan-summary.sh | 1 + scripts/test-rules-locale.sh | 45 ++++ scripts/test-tests-wired.sh | 90 ++++++++ scripts/test_audit_pr_footer_markup.py | 86 +++++++ scripts/test_auto_update_legacy_marker.py | 6 +- scripts/test_destructive_guard_w20.py | 4 +- scripts/test_install_canonical_global.py | 2 +- scripts/test_peer_pains_5.py | 2 +- scripts/test_secret_extra_rules_w12.py | 2 +- scripts/test_secret_extra_rules_w13.py | 2 +- scripts/test_secret_extra_rules_w14.py | 2 +- scripts/test_veche_doc_claims.py | 73 ++++++ scripts/test_veche_json_out_exit_code.py | 64 ++++++ scripts/test_veche_surface.py | 211 ++++++++++++++++++ scripts/test_veche_uniq_unicode_fold.py | 32 +++ 42 files changed, 882 insertions(+), 50 deletions(-) create mode 100644 changelog.d/rules-gate-locale.md create mode 100644 changelog.d/tests-wired.md create mode 100644 changelog.d/veche-surface-leftovers.md create mode 100644 changelog.d/w6-review-fixes.md create mode 100644 changelog.d/w6a-review-blues.md create mode 100644 scripts/ci-test-allowlist.tsv create mode 100644 scripts/ci-tests.d/audit-pr-footer-markup.sh create mode 100644 scripts/ci-tests.d/orphan-tests-misc.sh create mode 100644 scripts/ci-tests.d/review-checks-scan-summary-registry-env.sh create mode 100644 scripts/ci-tests.d/rules-locale.sh create mode 100644 scripts/ci-tests.d/secret-rule-waves.sh create mode 100644 scripts/ci-tests.d/security-gate-waves.bg.sh create mode 100644 scripts/ci-tests.d/tests-wired.sh create mode 100644 scripts/ci-tests.d/veche-doc-claims.sh create mode 100644 scripts/ci-tests.d/veche-json-out-exit-code.sh create mode 100644 scripts/ci-tests.d/veche-surface.sh create mode 100644 scripts/ci-tests.d/veche-uniq-unicode-fold.sh create mode 100644 scripts/test-rules-locale.sh create mode 100644 scripts/test-tests-wired.sh create mode 100644 scripts/test_audit_pr_footer_markup.py create mode 100644 scripts/test_veche_doc_claims.py create mode 100644 scripts/test_veche_json_out_exit_code.py create mode 100644 scripts/test_veche_surface.py create mode 100644 scripts/test_veche_uniq_unicode_fold.py diff --git a/.claude/skills/communication-structure/INDEX.md b/.claude/skills/communication-structure/INDEX.md index 43c5e61dd..c26d00bab 100644 --- a/.claude/skills/communication-structure/INDEX.md +++ b/.claude/skills/communication-structure/INDEX.md @@ -6,7 +6,7 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing | File | Read this when... | Tok | |---|---|---| -| `references/artifact-templates.md` | you draft a status update, PR body, release note, decision memo or executive summary, incident summ… | 2292 | +| `references/artifact-templates.md` | you draft a status update, PR body, release note, decision memo or executive summary, incident summ… | 2308 | | `references/leadership-status-update.md` | you write any status update a non-technical leader will read: a tracker project update, an issue st… | 500 | ## Scripts diff --git a/.claude/skills/communication-structure/references/artifact-templates.md b/.claude/skills/communication-structure/references/artifact-templates.md index edc0ce7b9..2882dddef 100644 --- a/.claude/skills/communication-structure/references/artifact-templates.md +++ b/.claude/skills/communication-structure/references/artifact-templates.md @@ -41,8 +41,8 @@ verified and command. Fenced code does not count toward the budget. `audit-pr` is the body of an audit, security or hardening PR. `sections` fails, at any length, when a required heading (every heading of the template below except `Dependency upgrades`) is missing, `What changes for reviewers and users` is empty (items, or `None`), or `Not fixed` carries a file:line, route, URL or file path (counts only). A closing footer -(`---` or `🤖 Generated`, then only blank, `Generated with` and `Co-Authored-By:` lines) is not checked; other text after -a `---` is. With `--range BASE..HEAD` it also fails when a changed file is not named under `Changes by area`, by full path +(`---` or `🤖 Generated`, then only blank, `Generated with` and `Co-Authored-By:` lines, any of them in ``, ``, +``, `*` or `_` markup) is not checked; other text after a `---` is. With `--range BASE..HEAD` it also fails when a changed file is not named under `Changes by area`, by full path or by a containing directory with a trailing slash. The guard hook picks this kind when the `gh pr create|edit` title says audit, security or harden, and for `create` passes the diff against `--base` (default `origin/HEAD`). diff --git a/.claude/skills/communication-structure/scripts/comms_lint.py b/.claude/skills/communication-structure/scripts/comms_lint.py index 9c0a5fa0e..a3af9aac2 100644 --- a/.claude/skills/communication-structure/scripts/comms_lint.py +++ b/.claude/skills/communication-structure/scripts/comms_lint.py @@ -20,8 +20,9 @@ AUDIT_SECTIONS heading is present, "What changes for reviewers and users" is not empty (items, or "None"), "Not fixed" holds counts only (no file:line, route or URL), and, with --range BASE..HEAD, every file the range changes is named under "Changes by area" by its full path or by a containing directory with a trailing slash. A trailing attribution footer -(a `---` rule or `🤖 Generated` line, then only blank, `---`, `Generated with/by` and `Co-Authored-By:` lines) is not part -of the last section; any other text after a `---` is still checked. +(a `---` rule or `🤖 Generated` line, then only blank, `---`, `Generated with/by` and `Co-Authored-By:` lines; those lines +may sit in ``, ``, ``, `*` or `_` markup) is not part of the last section; any other text after a `---` is +still checked. Prints `FAIL L: . Fix: ` per failure and a score line; exit 1 when any non-advisory rule fails (below the bar), 0 otherwise, 2 on usage, unreadable input or a --range git cannot resolve (fail closed). @@ -239,8 +240,10 @@ def _meeting(res, prose): DETAIL = re.compile(r"\.[A-Za-z]\w{0,7}:\d+|#L\d+|\blines?\s+\d+|\b(?:GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS)\s+/|" r"(?-{3,}|🤖\s*Generated\b.*)|Generated (?:with|by)\b.*|Co-Authored-By:.*)?", re.I) +# `Co-Authored-By: ...` or nothing. The `🤖 Generated`, `Generated with/by` and `Co-Authored-By:` shapes may open in , , , `*` or `_` markup (the closer is part of the tail). +# ponytail: a line is judged by its prefix, so detail on a `Generated ...` or `Co-Authored-By:` line goes unchecked; tighten the shapes if that is abused +_MARKUP = r"(?:|[*_]){0,6}" # bounded: an unbounded repeat would be quadratic on a long run of `*` +FOOTER = re.compile(r"(?:(?P-{3,}|" + _MARKUP + r"🤖[\s*_]*Generated\b.*)|" + _MARKUP + r"Generated (?:with|by)\b.*|" + _MARKUP + r"Co-Authored-By:.*)?", re.I) def _name(text): @@ -265,7 +268,7 @@ def _sections(text): runs to the next heading of the same or a higher level (sub-headings stay inside it); the first of a repeated heading wins; HTML comments are dropped, so a template hint is not content. A trailing attribution footer is not content either: the last section ends at the first `---` rule or `🤖 Generated` line that is followed only by FOOTER-shaped lines (blank, - `---`, `Generated with/by ...`, `Co-Authored-By: ...`) to the end; any other text after a rule keeps the rule as content.""" + `---`, `Generated with/by ...`, `Co-Authored-By: ...`, in markup or not) to the end; any other text after a rule keeps the rule as content.""" lines = _blank_comments(text).splitlines() heads, fence, foot = [], False, len(lines) for i, ln in enumerate(lines): diff --git a/.claude/skills/idea-critic/INDEX.md b/.claude/skills/idea-critic/INDEX.md index 3af5eb107..4fea54eb2 100644 --- a/.claude/skills/idea-critic/INDEX.md +++ b/.claude/skills/idea-critic/INDEX.md @@ -7,7 +7,7 @@ Generated by `scripts/skill_index.py` (do not hand-edit; rerun it after changing | File | Read this when... | Tok | |---|---|---| | `references/anti-rationalization.md` | you are about to emit PASS_TO_USER, skip a hat, or accept a shortcut (see SKILL.md, Escalate). | 332 | -| `references/council.md` | a decision is high-stakes AND hard to reverse, and one inline pass is too weak: you want independen… | 2267 | +| `references/council.md` | a decision is high-stakes AND hard to reverse, and one inline pass is too weak: you want independen… | 2379 | | `references/decision-hygiene.md` | the owner faces a hard, high-stakes call of their *own* — a pivot, quitting or killing a line of wo… | 1137 | | `references/full-map.md` | the quick attack in SKILL.md is not enough: an owner-decision or high-blast claim, an agent-origina… | 3395 | diff --git a/.claude/skills/idea-critic/references/council.md b/.claude/skills/idea-critic/references/council.md index d7c49fc4b..2a3367f6a 100644 --- a/.claude/skills/idea-critic/references/council.md +++ b/.claude/skills/idea-critic/references/council.md @@ -23,7 +23,7 @@ neither and it convenes (legacy behavior), so declare them. | The evil twin argues the strongest dissent | `--twin`: after the decision, one call (`lenses/evil-twin.txt`) writes a pre-mortem and the strongest case against it; kept in `minority_report` with the dissenting seats' names (their answers stay verbatim in `dissent`) | | Pre-mortem | every seat answers `premortem`, `kill_criterion`, `leading_metric` (recorded in `premortems`), then the twin's | | Deterministic aggregate; the chair breaks ties | decision = highest weighted mean of seat scores (0-5 per option and criterion); a tie goes to `--chair`'s own top option, then to option order (`do-nothing` first). The verdict tally ties go to the chair, else `HOLD` > `REVISE` > `PASS_TO_USER` | -| Output: decision, kill criteria, metric, review date | record keys `decision`, `kill_criteria`, `leading_metric`, `review_date` (`--review-date`, else today + `--review-days`, default 30). A closer no seat supplied is `UNVERIFIED`, never invented | +| Output: decision, kill criteria, metric, review date | record keys `decision`, `kill_criteria`, `kill_criteria_by_lens` (`{lens: text}`), `leading_metric`, `review_date` (`--review-date`, else today + `--review-days`, default 30). A closer no seat supplied is `UNVERIFIED`, never invented | | Convene only when it matters | the gate above | ## Run @@ -36,7 +36,7 @@ python3 scripts/veche.py --cmd "claude -p --strict-mcp-config" \ --stakes high --reversibility hard ``` -- `--cmd` reads the prompt on stdin and prints the answer. Without `--lenses` it is solo-3: one cmd, the three hats. +- `--cmd` reads the prompt on stdin and prints the answer. A custom `--cmd` reply needs a JSON object with a `verdict` (`HOLD|REVISE|PASS_TO_USER`) and string-only `claims` items; prose and stray braces around it are skipped, the first object with a valid `verdict` wins, and with `--option` the `scores` map may sit under `scores` in that object, or in any other object (bare or under `scores`). Without `--lenses` it is solo-3: one cmd, the three hats. - `--option` names must be short tokens (letters, digits, `-`, `_`, starting with a letter or digit; at most 32): seats echo them as JSON keys. Others fail before any call. - The packet is shown with numbered lines and its path under `--root`, so a model with no file tools can still cite `path:line`; keep the packet under `--root`. @@ -56,7 +56,7 @@ python3 scripts/veche.py --cmd "claude -p --strict-mcp-config" \ checked before each call. Any cap hit, non-zero exit, timeout or unparseable answer yields `COULD_NOT_DECIDE` (exit 3), never a guessed verdict, with `dropped` reasons and `cost`. A timed-out command's grandchildren may outlive it, so use a cmd that exits cleanly. `--dry-run` prints planned calls. -- `--json-out PATH` also writes the record there (a bad path fails before any call). A gate skip prints `NOT_CONVENED` on stderr, noting an ignored `--twin`. +- `--json-out PATH` also writes the record there (a bad path fails before any call; a write that fails at the end exits 2 only when the verdict would exit 0, so 3 and 4 stay). A gate skip prints `NOT_CONVENED` on stderr, noting an ignored `--twin`. - Validate: `scripts/validate_verdict.py --file record.json` accepts the record (it has `tally`), `COULD_NOT_DECIDE` and `NOT_CONVENED` with a `reason`, and a record with a `decision` only when it also carries `kill_criteria`, `leading_metric` and an ISO `review_date`. @@ -80,8 +80,8 @@ which yields the `default_action` above. A malformed `--criteria` error states t When the critique round runs, each seat also ranks the other (anonymized, lens names masked) answers best first; valid rankings are Borda-aggregated into `peer_ranking` (lens to mean points, informational: the verdict still comes from the tally and evidenced attacks). There is no separate adapter: any other council -can be a seat or the whole panel through the existing `--cmd`, a script that reads the prompt on stdin and -prints the JSON contract. A seat that fails (cap, timeout, cmd exit) is dropped and lowers confidence; a +can be a seat or the whole panel through the existing `--cmd` (a script; its reply contract is under Run). +A seat that fails (cap, timeout, cmd exit) is dropped and lowers confidence; a failed critique call or twin keeps the round-1 verdict. Versus the fetched sources (`docs/standards-index.md`): | | llm-council | Du et al. debate | Mixture-of-Agents | Veche | diff --git a/.claude/skills/idea-critic/scripts/veche.py b/.claude/skills/idea-critic/scripts/veche.py index a34dd6623..473c9e153 100644 --- a/.claude/skills/idea-critic/scripts/veche.py +++ b/.claude/skills/idea-critic/scripts/veche.py @@ -20,18 +20,19 @@ dropped with their reasons and the call cost, and a stderr COUNCIL_BLOCKING_WORK alarm (fired while the council runs, once --alarm-s has passed), so a stuck council never stalls unrelated work. --cmd can wrap any external council (a script reading the prompt on stdin, printing the JSON contract). -Side effects: spawns the given commands, reads files under --root, writes --json-out (an unwritable path exits 2, also after the -record was printed). Exit 0 decided. +Side effects: spawns the given commands, reads files under --root, writes --json-out (a temp file beside it, then os.replace; an unwritable path exits 2, +also after the record was printed, unless the verdict already exits 3 or 4). Exit 0 decided. """ from __future__ import annotations -import argparse, concurrent.futures, datetime, threading, json, os, random, re, secrets, shlex, subprocess, sys, time +import argparse, concurrent.futures, contextlib, datetime, threading, json, os, random, re, secrets, shlex, subprocess, sys, time from pathlib import Path VERDICTS = ("HOLD", "REVISE", "PASS_TO_USER") # most cautious first: ties go left HATS = "skeptic,better-way,kill-criteria" NAME = re.compile(r"^[a-z][a-z-]{0,30}$") TOKEN = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{0,31}") # an --option name: every seat must echo it exactly as a JSON key +TOKEN_RULE = f"letters, digits, - or _, starting with a letter or digit, at most 32 chars: {TOKEN.pattern}" CAP = "cap reached before call" CITE = re.compile(r"^(?P

[^\s:][^:]*):(?P\d+)(?:-(?P\d+))?$") DO_NOTHING = "do-nothing" @@ -72,10 +73,11 @@ def norm(t: str) -> str: def uniq(items: list) -> list: """Drop near-duplicate strings, keeping the first wording and the order: case, spacing and punctuation are - ignored; digits and comparison operators are not (5% and 6% stay two items, and so do `< 200 ms` and `> 200 ms`).""" + ignored; digits and comparison operators are not (5% and 6% stay two items, and so do `< 200 ms` and `> 200 ms`); `≤` and `≥` + are the same operators as `<=` and `>=`, so the Unicode and ASCII spellings fold together.""" seen: dict = {} for t in items: - seen.setdefault(" ".join(re.findall(r"\w+|[<>≤≥]=?", t.lower())), t) + seen.setdefault(" ".join(re.findall(r"\w+|[<>]=?", t.lower().replace("≤", "<=").replace("≥", ">="))), t) return list(seen.values()) @@ -200,7 +202,7 @@ def run(a, st=None) -> dict: names = [o.strip() for o in (a.option or []) if o.strip() and o.strip() != DO_NOTHING] for o in names: if not TOKEN.fullmatch(o): - raise Stop(f"bad --option {o!r}: use a short token (letters, digits, - or _, at most 32 chars), e.g. ship; " + raise Stop(f"bad --option {o!r}: use a short token ({TOKEN_RULE}), e.g. ship; " "every seat must repeat it exactly as a JSON key") options = [DO_NOTHING] + names if len(options) > 8 or len(set(options)) != len(options): @@ -388,6 +390,7 @@ def others_of(i): ["UNVERIFIED: no seat wrote a pre-mortem"] kills = uniq([x["kill"] for x in answers if x["kill"]] + ([twin["kill_criterion"]] if isinstance(twin.get("kill_criterion"), str) and twin["kill_criterion"].strip() else [])) rec["kill_criteria"] = kills or ["UNVERIFIED: no seat stated a measurable tripwire"] + rec["kill_criteria_by_lens"] = {x["lens"]: x["kill"] for x in answers if x["kill"]} # each seat's own wording, before the dedupe metrics = {x["lens"]: x["metric"] for x in answers if x["metric"]} rec["leading_metric"] = metrics.get(a.chair) or next(iter(metrics.values()), "UNVERIFIED: no seat named a metric") rec["minority_report"] = {"twin": twin or None, "dissenting_seats": [x["lens"] for x in dissent]} # their answers are in `dissent` @@ -424,7 +427,7 @@ def stop(): def main(argv=None) -> int: """CLI entry. Prints the JSON record (and writes it to --json-out). Exit 0 decided, 2 --json-out unwritable (the record is still - printed), 3 COULD_NOT_DECIDE (fail closed), 4 NOT_CONVENED.""" + printed; a write that fails at the end changes only exit 0, never 3 or 4), 3 COULD_NOT_DECIDE (fail closed), 4 NOT_CONVENED.""" ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) ap.add_argument("--cmd", action="append", required=True) ap.add_argument("--claim", required=True) @@ -433,12 +436,15 @@ def main(argv=None) -> int: ap.add_argument("--origin", default="agent-originated", choices=("owner-request", "agent-originated")) ap.add_argument("--root", default=".") ap.add_argument("--lenses", default=HATS, help="comma list of lens files; seats: cpo,cto,cfo,ciso,user-advocate") - ap.add_argument("--option", action="append", help="an option to score: a short token such as ship (repeat; do-nothing is always added; include the claim)") + ap.add_argument("--option", action="append", help=f"an option to score, a short token ({TOKEN_RULE}) since every seat repeats it as a JSON key, e.g. ship (repeat the flag; do-nothing is always added; include the claim)") ap.add_argument("--criteria", help="name:weight,... fixed before scoring; required with --option") ap.add_argument("--chair", help="a lens name: breaks ties in the verdict tally and the option scores") ap.add_argument("--twin", action="store_true", help="add the evil twin: pre-mortem and strongest dissent") - ap.add_argument("--stakes", choices=("low", "medium", "high")) - ap.add_argument("--reversibility", choices=("easy", "hard")) + ap.add_argument("--stakes", choices=("low", "medium", "high"), + help="low or medium skips the council (NOT_CONVENED, exit 4, no seat called) only if --reversibility easy is also given; high, or no --reversibility, convenes") + ap.add_argument("--reversibility", choices=("easy", "hard"), + help="easy skips the council (NOT_CONVENED, exit 4, no seat called) only if --stakes low or medium is also given; hard, or no --stakes, convenes. " + "easy also makes the default_action of a COULD_NOT_DECIDE record proceed, else hold") ap.add_argument("--review-date", help="ISO date to re-check the decision (default: today + --review-days)") ap.add_argument("--review-days", type=int, default=30) ap.add_argument("--max-calls", type=int, default=12) @@ -447,15 +453,28 @@ def main(argv=None) -> int: ap.add_argument("--timeout", type=int, default=120) ap.add_argument("--alarm-s", type=int, default=300, help="print COUNCIL_BLOCKING_WORK to stderr once the council has run this long, while it runs") ap.add_argument("--progress-s", type=int, default=30, help="stderr heartbeat every N seconds while the council runs (0 = off)") - ap.add_argument("--json-out", help="also write the JSON record to this file (opened before any call, so a bad path fails early)") + ap.add_argument("--json-out", help="also write the JSON record to this file on every outcome, via a temp file in the same directory and os.replace; " + "an empty file is written first, before any call, so a bad path fails early. The directory must be writable: a read-only file in it is replaced, " + "while a read-only directory or a bind-mounted file (os.replace cannot replace a mount point) exits 2") ap.add_argument("--seed", type=int, default=0) ap.add_argument("--dry-run", action="store_true") a = ap.parse_args(argv) - def write_out(text): # an unwritable --json-out is a usage error (exit 2, no traceback), at the start and at the end + def write_out(text): # atomic: a reader sees the old file or the whole new one. ponytail: no fsync, so not durable across power loss + real = os.path.realpath(a.json_out) # a symlinked path keeps pointing at its target + tmp = f"{real}.{secrets.token_hex(4)}.tmp" try: - Path(a.json_out).write_text(text) - except OSError as e: + if os.path.exists(a.json_out) and not os.path.isfile(a.json_out): # a device, FIFO or directory: renaming over it would replace the node itself + Path(a.json_out).write_text(text) # (a directory fails here, as before) + else: + with open(tmp, "x", encoding="utf-8") as f: + f.write(text) + with contextlib.suppress(OSError): # an existing record keeps the mode its owner gave it + os.chmod(tmp, os.stat(real).st_mode & 0o7777) + os.replace(tmp, real) + except OSError as e: # an unwritable --json-out is a usage error (exit 2, no traceback), at the start and at the end + with contextlib.suppress(OSError): + os.unlink(tmp) ap.error(f"--json-out: cannot write {a.json_out}: {e.strerror}") if a.json_out: @@ -487,9 +506,13 @@ def write_out(text): # an unwritable --json-out is a usage error (exit 2, no tr f"default_action={rec.get('default_action', 'n/a')}", file=sys.stderr) out = json.dumps(rec, indent=2) print(out) + code = {"COULD_NOT_DECIDE": 3, "NOT_CONVENED": 4}.get(rec["verdict"], 0) if a.json_out: - write_out(out + "\n") # the record is already on stdout, so an unwritable path loses nothing - return {"COULD_NOT_DECIDE": 3, "NOT_CONVENED": 4}.get(rec["verdict"], 0) + try: + write_out(out + "\n") # the record is already on stdout, so an unwritable path loses nothing + except SystemExit as e: # the message is printed; a blocking verdict exit (3, 4) must not be turned into 2 + code = code or e.code + return code if __name__ == "__main__": diff --git a/SHA256SUMS b/SHA256SUMS index f637065aa..ae4fe2309 100644 --- a/SHA256SUMS +++ b/SHA256SUMS @@ -179,15 +179,15 @@ b8eaccc017ff43654798f4781f3e6c804ca3306d1365c2ab7870e9e1f855e0ba .claude/skills 6967542c52536ecea3dcaf1374a2ea78928e19d3b4595aacb09b025cc46bfe08 .claude/skills/business-ops/references/full-map.md 8be21023d9bc5b2eb1d10bf95c45de8526b212afbd46eb429f9856fc2620a86f .claude/skills/business-ops/references/regulated-domain-triage.md 66b3862368182ef01649d0db8f99913b518bdce78fce4a9282da247d77c7ea3a .claude/skills/business-ops/scripts/bizops_check.py -40261b90d9a303cfa2f3d1eb1cc6d35c73970afe85119b04ff4b0b0534e8f8d4 .claude/skills/communication-structure/INDEX.md +476d2b728eef9a96d144e78dc618f080e9533118efb26a6937dba696eb51bfb0 .claude/skills/communication-structure/INDEX.md 0e5afb32996c1f721bbfe709a78a21b6feaa1f347fdb705be1ed6a102b0db719 .claude/skills/communication-structure/SKILL.md 563a0b4eec3c77bbc34ac630d4fa5214f5c6726ae01468404fd704473110084e .claude/skills/communication-structure/VERSION c90009107b343eff57af6565f8a7bec56626688402eb113bc4315760f0fdbdb7 .claude/skills/communication-structure/evals/evals.json 7024c68d4964f2b247f5b9cc05d6827611c362664d055f137cb00a80f5d48e70 .claude/skills/communication-structure/evals/reader-checks.json 29020bf7db659b0e2019587f6857d34794d05dc1f693f909543b20dd7f0873b3 .claude/skills/communication-structure/evals/triggers.json -594df34b2266cf2882af7f9f6aa02168ab308f020bd22e4e2e44f67f3957d58d .claude/skills/communication-structure/references/artifact-templates.md +72564c2554208142c5d87122939a3771e8f2b7251cce36b60fc817c24eb7c6a5 .claude/skills/communication-structure/references/artifact-templates.md b9e8c4a71aa0102503226c0180379bc3bc25f13485ac35232521da88b88fc843 .claude/skills/communication-structure/references/leadership-status-update.md -a702bb2650756cbc89aaeaa4e7dc08eda7fce5e808ab0a4e2edbabd001454532 .claude/skills/communication-structure/scripts/comms_lint.py +ef7df98af7a1db3e2ac62dd18d9b858f11c5908b6a3a8e6eb258791e57ea996b .claude/skills/communication-structure/scripts/comms_lint.py aee1b377e135c8dcdb808cb02b29d7bd8ca9ab3038023ee4afd616f5ffc74d6f .claude/skills/communication-structure/scripts/reader_check.py 4195df9b36e2a1f73fd6ffa397fca22e95e86465064110fdfaea65c19f130d9d .claude/skills/communication-structure/scripts/status_update_lint.py 2524bf62b627d362d888535ec9ef6527642cd7eeccbfd518454ed68f30d21344 .claude/skills/communication-structure/scripts/status_vocab_lint.py @@ -418,14 +418,14 @@ b7df4ae54d1d7ec6038c6ae0d32f9da6a2359d5991c91367c528487c7f90e1f0 .claude/skills e951eaa63fd0d1870701d69c89c29ef4fe74dbad9b680ab6ef0a4e993fb8bc20 .claude/skills/growth-analytics/evals/triggers.json fd316c63bbf1cc0ea64322d7d923d66692eaf7b545c056367bfecca35146c862 .claude/skills/growth-analytics/references/full-map.md 2f1d7499e9ed7ae59f550b5c00043e465b2a4e885821ad153438e2e33fcccb45 .claude/skills/growth-analytics/scripts/readout_check.py -62e1a508a4a96cb0e318d9127b7613338228c233f41dd15f5ba7092515363fcd .claude/skills/idea-critic/INDEX.md +67e71ec40535ad0b3931c51f3decf323bce809ed4c9586c6d422b7ad5698129f .claude/skills/idea-critic/INDEX.md edd6c254cb4ceaf00b3171439eaacf225cde74b921e96c9e829cb892008aa092 .claude/skills/idea-critic/SKILL.md eea5482cd77684430199a61baf7daf8bd18e43ad0009df2732fb35c4e28f3d38 .claude/skills/idea-critic/VERSION 9f77d5d6d13c99fd16cd8c398795d8be226d74bae5a5bf65369ed94a59fa56b7 .claude/skills/idea-critic/evals/evals.json 0e1c0db300661a1ab1a427a7157c75e2ed67130ab24d5c6cb02ec3ff9e793f58 .claude/skills/idea-critic/evals/heldout.json d8098c5ced63e8b8608867dd144091049563a0553cf62d27ce05b9d52ac54266 .claude/skills/idea-critic/evals/triggers.json 37d13d879460b9e65f017d62303e30adbf2dcd590ff3a2156ba2b53e931e611d .claude/skills/idea-critic/references/anti-rationalization.md -48c52f686a86638c6dd7a6c32e59d29aed52c7904f38ff3cf0e91bee78573ae1 .claude/skills/idea-critic/references/council.md +420fc6ccbc762a123f47e7382f79a8f1a27629732be996464ca0ac0bf7c451d2 .claude/skills/idea-critic/references/council.md c5615e5c59bc2e452c2c49619be2ec5e456f3c167e772498316f8b5a45deb2c8 .claude/skills/idea-critic/references/decision-hygiene.md 7ffd569fd3a9aa4b2fc969c8f163b88373929ce77388aeebc8fb5737783754bb .claude/skills/idea-critic/references/full-map.md b9339c3a39f7af13fc3a3103b57d6a203207986b1d9912df50ac94b990a68c9d .claude/skills/idea-critic/scripts/council.py @@ -439,7 +439,7 @@ ffa9792bd3dc750725d85e32f91a8051d2f21932d035f3f4e47fcccb66186860 .claude/skills 0b3872c37c94e1db02a094ccdb04a39d1a245b1089f801729c2c1827a154ff6e .claude/skills/idea-critic/scripts/lenses/skeptic.txt e3c08eee63d6da353029bfcc78f0ba7638d75b3acb02fd0af045e3c399a857a6 .claude/skills/idea-critic/scripts/lenses/user-advocate.txt 5baa5b2f797a24208cac893de5ba34bda548590c0340ca41826bbcdc60dabf6c .claude/skills/idea-critic/scripts/validate_verdict.py -05ba052fb35a91056404c97dbc1b4695d346ad4d379e4fb6289b5529fbff09e6 .claude/skills/idea-critic/scripts/veche.py +79c1b4b00b8c18c5cd69a8d916300291c38074a0be4f4d15dfc8cdf00418ad28 .claude/skills/idea-critic/scripts/veche.py b812c50e074a71f342c5a9cf9d032f31be84768cd231840a0006c26c66536ccf .claude/skills/positioning/INDEX.md 595e209389ef87fded12b1f0b9a7f368449bc622e8b431f74a4f603438ed20b8 .claude/skills/positioning/SKILL.md f5e5c9534532840fcd45bb002e69ed85c61dba2753450e26b448b7df1481e24d .claude/skills/positioning/VERSION diff --git a/changelog.d/rules-gate-locale.md b/changelog.d/rules-gate-locale.md new file mode 100644 index 000000000..bc8e71853 --- /dev/null +++ b/changelog.d/rules-gate-locale.md @@ -0,0 +1,2 @@ +### Fixed +- `scripts/ci-gates.sh rules` now compares its sorted rows in the C locale, like the `sort -u` calls that feed them. Under a UTF-8 locale such as `en_US.UTF-8` its two `comm` calls saw unsorted input, so a baseline with a few rows could report a baselined rule as new, or a baseline that only shrank as grown and needing a `rule-baseline-raise:` line. The shipped baseline is empty, so nothing failed yet. Regression test: `scripts/test-rules-locale.sh` (registered from `scripts/ci-tests.d/rules-locale.sh`) runs the gate under C and `en_US.UTF-8`. diff --git a/changelog.d/tests-wired.md b/changelog.d/tests-wired.md new file mode 100644 index 000000000..7ff235fc0 --- /dev/null +++ b/changelog.d/tests-wired.md @@ -0,0 +1,6 @@ +### Added +- `scripts/ci-gates.sh tests-wired --allowlist `: a standing check that every `scripts/test_*.py` and `scripts/test-*.sh` is run by CI. A test counts as run when `scripts/ci-tests.d/*.sh`, a `.github/workflows/*.yml` step, or a test those run (for example `test-ci-gates.sh`) names it on a non-comment line; otherwise it needs a `namereason` row in `scripts/ci-test-allowlist.tsv`. A row without a reason, for a missing file, or for a test CI already runs also fails, so the list only shrinks. It runs from `scripts/ci-tests.d/tests-wired.sh`; its regression test is `scripts/test-tests-wired.sh` (a planted orphan fails, and it gives the same answers under `en_US.UTF-8` as under the C locale). +- The first run found 71 tests that no CI step ran: 59 are now registered (`secret-rule-waves.sh`, `security-gate-waves.bg.sh` as a background suite, `orphan-tests-misc.sh`, `orphan-tests.sh`) and the 12 corpus re-scores of 30 s or more are allowlisted with their measured runtimes. + +### Fixed +- `scripts/test_auto_update_legacy_marker.py` took about 326 s because it copied `docs/bench` (about 90k files) into a scratch tree; it now skips that directory and runs in about 4 s. Its `install.sh` run, `scripts/test_install_canonical_global.py` and `scripts/test_peer_pains_5.py` now set `DCR_NO_PROBE=1`, so none of them calls github.com through the sandbox probe. diff --git a/changelog.d/veche-surface-leftovers.md b/changelog.d/veche-surface-leftovers.md new file mode 100644 index 000000000..ac98011b2 --- /dev/null +++ b/changelog.d/veche-surface-leftovers.md @@ -0,0 +1,9 @@ +### Added +- `veche.py`: a decided record carries `kill_criteria_by_lens` (`{lens: text}`), each seat's own kill criterion as worded before near-duplicates collapse, so a reader can see which seat asked for which tripwire. It is additive: `kill_criteria` and `validate_verdict.py` are unchanged, a seat that gave none is left out of the map, and the evil twin's criterion stays in `minority_report.twin` (#2140). + +### Changed +- `veche.py --help`: `--option` states the exact name rule (the `TOKEN` pattern, which the refusal for a bad name now quotes too), `--stakes` and `--reversibility` state the real gate (the council is skipped only when stakes are low or medium and reversibility is easy, and either flag alone convenes), and `--json-out` says how the file is written (#2136, #2139). +- `council.md` states the reply contract a custom `--cmd` must meet, as the parser really accepts it: a JSON object with a `verdict` and string-only `claims` items, prose and stray braces around it skipped, the first object with a valid `verdict` wins, and the `scores` map may sit in the verdict object or in any other object (#2137). It also lists `kill_criteria_by_lens` in the output row. + +### Fixed +- `veche.py --json-out` writes through a temp file in the same directory and `os.replace`, at the early write and at the final one and on every outcome, so a reader or a crash never sees half a record. A symlinked path keeps pointing at its target, an existing file keeps its mode, a device or FIFO target is written in place, and an unwritable path still exits 2 with the `--json-out: cannot write` message (#2139). diff --git a/changelog.d/w6-review-fixes.md b/changelog.d/w6-review-fixes.md new file mode 100644 index 000000000..f84632d48 --- /dev/null +++ b/changelog.d/w6-review-fixes.md @@ -0,0 +1,7 @@ +### Fixed +- The wall-clock pins of `scripts/test_destructive_guard_w20.py`, `scripts/test_secret_extra_rules_w12.py`, `scripts/test_secret_extra_rules_w13.py` and `scripts/test_secret_extra_rules_w14.py` (0.2 s and 0.5 s on 100 KB lines) are now 2.0 s, so a loaded CI runner no longer fails them; a catastrophic-backtracking regression still takes far longer. +- `scripts/test-rules-locale.sh` and `scripts/test-tests-wired.sh` print `SKIP en_US.UTF-8 ...` and run their C-locale cases when the runner has no `en_US.UTF-8`, instead of failing on a locale that is not installed. + +### Changed +- `veche.py --help` states the `--json-out` directory rule: the record is written through a temp file beside the target, so a read-only file in a writable directory is replaced, while a read-only directory or a bind-mounted file exits 2. Test: `scripts/test_veche_doc_claims.py`. +- `council.md` says the `scores` map may sit under `scores` in the verdict object itself, or in any other object (bare or under `scores`), as `veche.py` reads it. diff --git a/changelog.d/w6a-review-blues.md b/changelog.d/w6a-review-blues.md new file mode 100644 index 000000000..a06e852fe --- /dev/null +++ b/changelog.d/w6a-review-blues.md @@ -0,0 +1,7 @@ +### Fixed +- `comms_lint.py` `audit-pr`: the attribution footer may sit in markup. A `🤖 Generated with ...` line (with or without a `---` rule before it), a `*Generated with ...*` or `_Generated by ..._` line after a rule or the robot line, and a `Co-Authored-By:` line in the same markup are footer-shaped, so a compliant body that ends with one no longer fails "Not fixed carries detail ('https://')". Markup alone does not make a footer: a wrapped line that is not a Generated or Co-Authored-By line, a Generated line with no rule or robot before it, and anything after a footer are still checked. The accepted markup is bounded (``, ``, ``, `*`, `_`, at most six), so a long run of `*` stays linear. `references/artifact-templates.md` states the rule. Test: `scripts/test_audit_pr_footer_markup.py`. +- `veche.py`: the kill-criteria dedupe key maps `≤` to `<=` and `≥` to `>=` first, so `rate ≤ 5%` and `rate <= 5%` are one tripwire (the first wording stays), while `≤` against `≥` and `<` against `≤` stay two. Test: `scripts/test_veche_uniq_unicode_fold.py`. +- `veche.py --json-out`: a write that fails at the end of the run no longer turns a blocking verdict into exit 2. `COULD_NOT_DECIDE` keeps exit 3 and `NOT_CONVENED` keeps exit 4; only a verdict that would exit 0 becomes exit 2. The stderr message is unchanged, and `council.md` says so in one clause. Test: `scripts/test_veche_json_out_exit_code.py` (a write that works once, then fails, for each of the three outcomes). +- `scripts/test-review-checks-scan-summary.sh` unsets `REVIEW_SEMGREP_REGISTRY`, so a host that sets it to `0` (which drops the `--config p/default` the test expects) still passes. Test: `scripts/ci-tests.d/review-checks-scan-summary-registry-env.sh` runs it with the variable set to `0`. +- size-budget-raise: .claude/skills/communication-structure/references/artifact-templates.md 9171→9234 the audit-pr footer sentence names the markup a footer line may sit in +- size-budget-raise: .claude/skills/idea-critic/references/council.md 9071→9516 kill_criteria_by_lens in the output row, one sentence stating the --cmd reply contract (the verdict object may carry the scores), and one clause: a --json-out write that fails at the end exits 2 only when the verdict would exit 0 diff --git a/scripts/ci-gates.sh b/scripts/ci-gates.sh index 2dc2831a8..2a8ebca24 100644 --- a/scripts/ci-gates.sh +++ b/scripts/ci-gates.sh @@ -23,6 +23,8 @@ # mustload --config every SKILL.md load-map archetype's MUST-LOAD token-est total is within its frozen mustload-budgets.tsv ceiling; the Phase 0-2 floor equals its pin exactly, and every " when " ref is a pinned phase-conditional row # rules --baseline [--base ] [--list] # every MUST/NEVER/ALWAYS (or bold "Never") rule line in a SKILL.md must sit in a paragraph that names a mechanism (scripts/, ci-gates, evals, hook) or says "advisory"; unenforced rules not in the baseline FAIL (ratchet); --base: baseline rows added vs need a `rule-baseline-raise:` commit marker +# tests-wired --allowlist +# every scripts/test_*.py and scripts/test-*.sh is run by CI (named in scripts/ci-tests.d/*.sh, .github/workflows/*.yml or a test they run) or listed in with a reason; a stale row FAILS # set -euo pipefail @@ -78,6 +80,7 @@ Usage: ci-gates.sh fragments [--config ]

ci-gates.sh binaries ci-gates.sh mustload --config + ci-gates.sh tests-wired --allowlist EOF } @@ -1561,7 +1564,8 @@ cmd_rules() { cur="$(rules_scan "$root")" if [ "$list" -eq 1 ]; then printf '%s\n' "$cur"; return 0; fi [ -n "$baseline" ] && [ -f "$baseline" ] || die "rules: --baseline is required and must exist (fail closed)" - new="$(comm -23 <(printf '%s\n' "$cur") <(LC_ALL=C sort -u "$baseline"))" + # comm in the C locale like the sorts that feed it: a UTF-8 collation orders these rows differently and comm then sees unsorted input + new="$(LC_ALL=C comm -23 <(printf '%s\n' "$cur") <(LC_ALL=C sort -u "$baseline"))" if [ -n "$new" ]; then printf 'RULE UNENFORCED (new): name a mechanism (scripts/, ci-gates, evals, hook) in the same paragraph or mark it advisory:\n%s\n' "$new" >&2 die "rules: new hard rule(s) without a named mechanism" @@ -1572,7 +1576,7 @@ cmd_rules() { local rel grown rel="$(cd "$(dirname "$baseline")" && pwd)/$(basename "$baseline")"; rel="${rel#"$(cd "$root" && pwd)"/}" if git -C "$root" cat-file -e "${base}:${rel}" 2>/dev/null; then - grown="$(comm -13 <(git -C "$root" show "${base}:${rel}" | LC_ALL=C sort -u) <(LC_ALL=C sort -u "$baseline"))" + grown="$(LC_ALL=C comm -13 <(git -C "$root" show "${base}:${rel}" | LC_ALL=C sort -u) <(LC_ALL=C sort -u "$baseline"))" range_log="$(git -C "$root" log --format=%B "${base}..HEAD")" # capture first: `git log | grep -q` SIGPIPEs under pipefail if [ -n "$grown" ] && ! grep -q '^rule-baseline-raise: .\+' <<<"$range_log"; then printf '%s\n' "$grown" >&2 @@ -1584,6 +1588,74 @@ cmd_rules() { "$(printf '%s\n' "$cur" | grep -c . || true)" } +# --------------------------------------------------------------------------- +# tests-wired — every scripts/test_*.py and scripts/test-*.sh is run by CI. +# +# A per-feature test that no CI step runs never fails a build (the #1592 audit found dozens). A test counts as run +# when its basename is named on a non-comment line of a scripts/ci-tests.d/*.sh registration or a +# .github/workflows/*.yml, or of a test that is itself run (scripts/test-ci-gates.sh runs many per-feature files one +# level down); or when has a `` row for it (reason required; `#` comments and blank +# lines ignored). A row naming no test, or a test CI already runs, FAILS too, so the list only shrinks. +# Limit: matches the basename; a name inside a string counts as run, and a path built by string concatenation is +# missed (a false orphan: name the file whole). The reachability walk is one awk program: the regression +# test is scripts/test-tests-wired.sh (planted orphan, stale row, missing reason). +# --------------------------------------------------------------------------- +cmd_tests_wired() { + local allow="" root="" all run allowed orphans stale gone total listed bad=0 name reason + while [ "$#" -gt 0 ]; do + case "$1" in + --allowlist) allow="${2:-}"; shift 2 ;; + -*) die "tests-wired: unknown option: $1" ;; + *) [ -z "$root" ] || die "tests-wired: unexpected extra argument: $1"; root="$1"; shift ;; + esac + done + [ -n "$allow" ] && [ -n "$root" ] || die "tests-wired: --allowlist and are required" + [ -f "$allow" ] || die "tests-wired: allowlist not found: $allow (fail closed)" + [ -d "$root/scripts/ci-tests.d" ] || die "tests-wired: $root/scripts/ci-tests.d not found (fail closed)" + all="$(cd "$root/scripts" && shopt -s nullglob && printf '%s\n' test_*.py test-*.sh | LC_ALL=C sort -u)" + [ -n "$all" ] || die "tests-wired: no scripts/test_*.py or scripts/test-*.sh under $root (fail closed)" + # Roots are the registration files and workflows; every other file read is a test, whose named tests join the run set. + # shellcheck disable=SC2016 # awk program, not shell + run="$(cd "$root" && shopt -s nullglob && awk ' + { sub(/(^|[ \t])#.*/, "") } + { n = split($0, w, "[^A-Za-z0-9_./-]+") + for (i = 1; i <= n; i++) { + t = w[i]; sub(/.*\//, "", t) + if (t !~ /^test[_-].*\.(py|sh)$/) continue + b = FILENAME; sub(/.*\//, "", b) + if (FILENAME ~ /^(scripts\/ci-tests\.d|\.github\/workflows)\//) b = "." + refs[b] = refs[b] " " t } } + END { n = split(refs["."], q, " ") + for (h = 1; h <= n; h++) { + t = q[h]; if (t in seen) continue + seen[t] = 1; print t + m = split(refs[t], r, " "); for (j = 1; j <= m; j++) q[++n] = r[j] } } + ' scripts/ci-tests.d/*.sh .github/workflows/*.yml scripts/test_*.py scripts/test-*.sh /dev/null | LC_ALL=C sort -u)" || die "tests-wired: cannot scan $root (fail closed)" + allowed="" + while IFS=$'\t' read -r name reason || [ -n "$name" ]; do + case "$name" in ''|'#'*) continue ;; esac + [ -n "$reason" ] || { printf 'tests-wired: FAIL %s: allowlist row has no reason (namereason)\n' "$name"; bad=1; } + allowed="$allowed$name"$'\n' + done <"$allow" + allowed="$(printf '%s' "$allowed" | LC_ALL=C sort -u)" + # comm in the C locale like the sorts above: a UTF-8 collation orders `_` and `-` differently and comm then sees unsorted input + orphans="$(LC_ALL=C comm -23 <(printf '%s\n' "$all") <(printf '%s\n%s\n' "$run" "$allowed" | LC_ALL=C sort -u))" + gone="$(LC_ALL=C comm -13 <(printf '%s\n' "$all") <(printf '%s\n' "$allowed"))" + stale="$(LC_ALL=C comm -12 <(printf '%s\n' "$run") <(printf '%s\n' "$allowed"))" + while IFS= read -r name; do + [ -z "$name" ] || { printf 'tests-wired: FAIL scripts/%s is run by no CI step: name it in scripts/ci-tests.d/.sh (one command line), or add a `namereason` row to %s\n' "$name" "$allow"; bad=1; } + done <<<"$orphans" + while IFS= read -r name; do + [ -z "$name" ] || { printf 'tests-wired: FAIL %s: allowlist row names no scripts/test_*.py or scripts/test-*.sh (delete the row)\n' "$name"; bad=1; } + done <<<"$gone" + while IFS= read -r name; do + [ -z "$name" ] || { printf 'tests-wired: FAIL %s: allowlisted but CI already runs it (delete the row)\n' "$name"; bad=1; } + done <<<"$stale" + [ "$bad" -eq 0 ] || exit 1 + total="$(printf '%s\n' "$all" | awk 'END { print NR }')"; listed="$(printf '%s' "$allowed" | awk 'NF { n++ } END { print n + 0 }')" + printf 'tests-wired: ok (%d tests: %d run by CI, %d allowlisted)\n' "$total" "$((total - listed))" "$listed" +} + [ "$#" -gt 0 ] || { usage; exit 2; } subcmd="$1"; shift case "$subcmd" in @@ -1598,6 +1670,7 @@ case "$subcmd" in binaries) cmd_binaries "$@" ;; mustload) cmd_mustload "$@" ;; rules) cmd_rules "$@" ;; + tests-wired) cmd_tests_wired "$@" ;; -h|--help) usage; exit 0 ;; *) printf 'ci-gates: unknown subcommand: %s\n' "$subcmd" >&2; usage; exit 2 ;; esac diff --git a/scripts/ci-test-allowlist.tsv b/scripts/ci-test-allowlist.tsv new file mode 100644 index 000000000..ced3b4d90 --- /dev/null +++ b/scripts/ci-test-allowlist.tsv @@ -0,0 +1,17 @@ +# scripts/ci-test-allowlist.tsv: reasoned exceptions to `ci-gates.sh tests-wired` (#1592). +# Every scripts/test_*.py and scripts/test-*.sh must be run by CI (named in scripts/ci-tests.d/*.sh or a workflow) or listed here. +# FORMAT: ``; the reason is required (a measured runtime, or why CI cannot run it). Blank lines and +# lines starting with '#' are ignored. A row for a test CI already runs, or for a file that no longer exists, fails the gate: +# register the test and delete its row. +test_secrets_heldout10_blind.py ~55 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_secrets_heldout11_blind.py ~54 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_secrets_heldout12_blind.py ~55 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_secrets_heldout4_blind.py ~55 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_secrets_heldout56_blind.py ~105 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_secrets_heldout7_blind.py ~50 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_secrets_heldout8_blind.py ~57 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_secrets_heldout9_blind.py ~48 s locally; re-scores the held-out secrets corpus through the privacy gate; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_security_gates_heldout20_blind.py ~31 s locally; re-scores the held-out security-gates corpus, one guard process per command; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_security_gates_heldout21_blind.py ~32 s locally; re-scores the held-out security-gates corpus, one guard process per command; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_security_gates_heldout22_blind.py ~32 s locally; re-scores the held-out security-gates corpus, one guard process per command; run it by hand after a rule or guard edit, register as .bg.sh once sped up +test_security_gates_heldout24_blind.py ~39 s locally; re-scores the held-out security-gates corpus, one guard process per command; run it by hand after a rule or guard edit, register as .bg.sh once sped up diff --git a/scripts/ci-tests.d/audit-pr-footer-markup.sh b/scripts/ci-tests.d/audit-pr-footer-markup.sh new file mode 100644 index 000000000..6f1c63efa --- /dev/null +++ b/scripts/ci-tests.d/audit-pr-footer-markup.sh @@ -0,0 +1,2 @@ +# Registers scripts/test_audit_pr_footer_markup.py with CI (audit-pr footer: , * and _ markup around the attribution line still ends the last section, and hides nothing). +python3 scripts/test_audit_pr_footer_markup.py diff --git a/scripts/ci-tests.d/orphan-tests-misc.sh b/scripts/ci-tests.d/orphan-tests-misc.sh new file mode 100644 index 000000000..e634c9acd --- /dev/null +++ b/scripts/ci-tests.d/orphan-tests-misc.sh @@ -0,0 +1,8 @@ +# Registers other per-feature tests that no CI step ran (#1592 audit). Offline. +python3 scripts/test_bizops_keys.py +python3 scripts/test_contribution_keys.py +python3 scripts/test_hook_latency.py +python3 scripts/test_idea_critic_keys.py +python3 scripts/test_install_canonical_global.py +python3 scripts/test_outline.py +python3 scripts/test_token_orchestration_actions.py diff --git a/scripts/ci-tests.d/orphan-tests.sh b/scripts/ci-tests.d/orphan-tests.sh index e56326248..6c407d57f 100644 --- a/scripts/ci-tests.d/orphan-tests.sh +++ b/scripts/ci-tests.d/orphan-tests.sh @@ -1,5 +1,5 @@ # Registers per-feature tests that existed but no CI step ran (see the "Per-PR tests" step in ci.yml; #1592 audit). -# test_auto_update_legacy_marker.py is left out: ~326 s locally; register it once its runtime is cut. +python3 scripts/test_auto_update_legacy_marker.py python3 scripts/test_auto_update_move.py python3 scripts/test_auto_update_stale_lock.py python3 scripts/test_bench_corpus_90.py diff --git a/scripts/ci-tests.d/review-checks-scan-summary-registry-env.sh b/scripts/ci-tests.d/review-checks-scan-summary-registry-env.sh new file mode 100644 index 000000000..c3d83db44 --- /dev/null +++ b/scripts/ci-tests.d/review-checks-scan-summary-registry-env.sh @@ -0,0 +1,2 @@ +# The review_checks.sh scan-summary test passes on a host that sets REVIEW_SEMGREP_REGISTRY=0 (it unsets it before expecting `--config p/default`). +REVIEW_SEMGREP_REGISTRY=0 bash scripts/test-review-checks-scan-summary.sh diff --git a/scripts/ci-tests.d/rules-locale.sh b/scripts/ci-tests.d/rules-locale.sh new file mode 100644 index 000000000..34b3eb484 --- /dev/null +++ b/scripts/ci-tests.d/rules-locale.sh @@ -0,0 +1,2 @@ +# Registers scripts/test-rules-locale.sh with CI: `ci-gates.sh rules` answers the same under en_US.UTF-8 as under the C locale (see the "Per-PR tests" step in ci.yml). +bash scripts/test-rules-locale.sh diff --git a/scripts/ci-tests.d/secret-rule-waves.sh b/scripts/ci-tests.d/secret-rule-waves.sh new file mode 100644 index 000000000..f6ccd4751 --- /dev/null +++ b/scripts/ci-tests.d/secret-rule-waves.sh @@ -0,0 +1,16 @@ +# Registers the per-wave secret-rule pins and the placeholder-precision pass that no CI step ran (#1592 audit). The 30 s+ test_secrets_heldout*_blind.py +# corpus re-scores are listed with their runtimes in scripts/ci-test-allowlist.tsv instead. Offline. +python3 scripts/test_secret_extra_rules.py +python3 scripts/test_secret_extra_rules_w10.py +python3 scripts/test_secret_extra_rules_w11.py +python3 scripts/test_secret_extra_rules_w12.py +python3 scripts/test_secret_extra_rules_w13.py +python3 scripts/test_secret_extra_rules_w14.py +python3 scripts/test_secret_extra_rules_w3.py +python3 scripts/test_secret_extra_rules_w4.py +python3 scripts/test_secret_extra_rules_w5.py +python3 scripts/test_secret_extra_rules_w6.py +python3 scripts/test_secret_extra_rules_w7.py +python3 scripts/test_secret_extra_rules_w8.py +python3 scripts/test_secret_extra_rules_w9.py +python3 scripts/test_secret_precision.py diff --git a/scripts/ci-tests.d/security-gate-waves.bg.sh b/scripts/ci-tests.d/security-gate-waves.bg.sh new file mode 100644 index 000000000..fe5698579 --- /dev/null +++ b/scripts/ci-tests.d/security-gate-waves.bg.sh @@ -0,0 +1,40 @@ +# Registers the per-wave guard pins and held-out security-gate pins that no CI step ran (#1592 audit). The 30 s+ corpus +# re-scores are listed with their runtimes in scripts/ci-test-allowlist.tsv instead. Offline. About 50 s and it leaves the +# working tree unchanged, so it runs as a background suite (.bg.sh) alongside the gate steps and adds no serial time. +python3 scripts/test_destructive_guard_stub.py +python3 scripts/test_destructive_guard_stub_crash.py +python3 scripts/test_destructive_guard_w20.py +python3 scripts/test_destructive_guard_w21.py +python3 scripts/test_security_gates_guard_w10.py +python3 scripts/test_security_gates_guard_w11.py +python3 scripts/test_security_gates_guard_w3.py +python3 scripts/test_security_gates_guard_w4.py +python3 scripts/test_security_gates_guard_w5.py +python3 scripts/test_security_gates_guard_w5b.py +python3 scripts/test_security_gates_guard_w6.py +python3 scripts/test_security_gates_guard_w7.py +python3 scripts/test_security_gates_guard_w8.py +python3 scripts/test_security_gates_guard_w9.py +python3 scripts/test_security_gates_heldout10.py +python3 scripts/test_security_gates_heldout10_blind.py +python3 scripts/test_security_gates_heldout11_blind.py +python3 scripts/test_security_gates_heldout12_blind.py +python3 scripts/test_security_gates_heldout13_blind.py +python3 scripts/test_security_gates_heldout14_blind.py +python3 scripts/test_security_gates_heldout15_blind.py +python3 scripts/test_security_gates_heldout17_blind.py +python3 scripts/test_security_gates_heldout18_blind.py +python3 scripts/test_security_gates_heldout19_blind.py +python3 scripts/test_security_gates_heldout4.py +python3 scripts/test_security_gates_heldout5.py +python3 scripts/test_security_gates_heldout5_blind.py +python3 scripts/test_security_gates_heldout6.py +python3 scripts/test_security_gates_heldout6_blind.py +python3 scripts/test_security_gates_heldout7.py +python3 scripts/test_security_gates_heldout7_blind.py +python3 scripts/test_security_gates_heldout8.py +python3 scripts/test_security_gates_heldout8_blind.py +python3 scripts/test_security_gates_heldout9.py +python3 scripts/test_security_gates_heldout9_blind.py +python3 scripts/test_security_gates_w18_insample.py +python3 scripts/test_security_gates_w19_insample.py diff --git a/scripts/ci-tests.d/tests-wired.sh b/scripts/ci-tests.d/tests-wired.sh new file mode 100644 index 000000000..979b7ea14 --- /dev/null +++ b/scripts/ci-tests.d/tests-wired.sh @@ -0,0 +1,3 @@ +# Standing check (#1592): every scripts/test_*.py and scripts/test-*.sh is run by CI or listed with a reason. Offline. +bash scripts/test-tests-wired.sh +bash scripts/ci-gates.sh tests-wired --allowlist scripts/ci-test-allowlist.tsv . diff --git a/scripts/ci-tests.d/veche-doc-claims.sh b/scripts/ci-tests.d/veche-doc-claims.sh new file mode 100644 index 000000000..9bc60a21b --- /dev/null +++ b/scripts/ci-tests.d/veche-doc-claims.sh @@ -0,0 +1,2 @@ +# Registers scripts/test_veche_doc_claims.py with CI (veche --json-out help states the directory rule and the read-only file and directory behavior, council.md says the verdict object may carry the scores). +python3 scripts/test_veche_doc_claims.py diff --git a/scripts/ci-tests.d/veche-json-out-exit-code.sh b/scripts/ci-tests.d/veche-json-out-exit-code.sh new file mode 100644 index 000000000..3eab613fc --- /dev/null +++ b/scripts/ci-tests.d/veche-json-out-exit-code.sh @@ -0,0 +1,2 @@ +# Registers scripts/test_veche_json_out_exit_code.py with CI (veche --json-out failing at the end keeps exit 3 and 4 and turns only exit 0 into 2). +python3 scripts/test_veche_json_out_exit_code.py diff --git a/scripts/ci-tests.d/veche-surface.sh b/scripts/ci-tests.d/veche-surface.sh new file mode 100644 index 000000000..96c9b72d8 --- /dev/null +++ b/scripts/ci-tests.d/veche-surface.sh @@ -0,0 +1,2 @@ +# Registers scripts/test_veche_surface.py with CI (veche.py help rules for --option, --stakes and --reversibility, atomic --json-out, kill_criteria_by_lens, and the --cmd reply contract in council.md). +python3 scripts/test_veche_surface.py diff --git a/scripts/ci-tests.d/veche-uniq-unicode-fold.sh b/scripts/ci-tests.d/veche-uniq-unicode-fold.sh new file mode 100644 index 000000000..a1de242e2 --- /dev/null +++ b/scripts/ci-tests.d/veche-uniq-unicode-fold.sh @@ -0,0 +1,2 @@ +# Registers scripts/test_veche_uniq_unicode_fold.py with CI (veche kill-criteria dedupe folds the Unicode and ASCII spellings of <= and >=). +python3 scripts/test_veche_uniq_unicode_fold.py diff --git a/scripts/size-budgets.tsv b/scripts/size-budgets.tsv index a0543b15f..eacc5c84b 100644 --- a/scripts/size-budgets.tsv +++ b/scripts/size-budgets.tsv @@ -202,7 +202,7 @@ README.md 16118 .claude/skills/agentic-delivery/references/worktree-isolation.md 16482 .claude/skills/communication-structure/references/leadership-status-update.md 2003 .claude/skills/idea-critic/references/anti-rationalization.md 1330 -.claude/skills/idea-critic/references/council.md 9071 +.claude/skills/idea-critic/references/council.md 9516 .claude/skills/idea-critic/references/full-map.md 13580 .claude/skills/product-output-safety/references/full-map.md 13348 .claude/skills/product-discovery/references/full-map.md 12384 @@ -215,7 +215,7 @@ README.md 16118 .claude/skills/deep-code-review/INDEX-scripts.md 6045 .claude/skills/agentic-delivery/references/pipeline-optimizer.md 19857 .claude/skills/agentic-delivery/references/focus-kit.md 5235 -.claude/skills/communication-structure/references/artifact-templates.md 9171 +.claude/skills/communication-structure/references/artifact-templates.md 9234 .claude/skills/deep-code-review/references/code-structure.md 6449 .claude/skills/deep-code-review/references/compatibility-portability.md 5771 .claude/skills/deep-code-review/references/delivery-speed.md 7666 diff --git a/scripts/test-review-checks-scan-summary.sh b/scripts/test-review-checks-scan-summary.sh index 5b077785c..b01918540 100644 --- a/scripts/test-review-checks-scan-summary.sh +++ b/scripts/test-review-checks-scan-summary.sh @@ -36,6 +36,7 @@ printf 'x\n' >app.txt git add . && git -c user.email=t@example.com -c user.name=t commit -q -m files export PERUN_HEAVY_DIR="$T/heavy0" # never touch the real lease dir export DCR_CACHE_ROOT="$T/cache" # empty: a warm semgrep snapshot on the host would swap `--config p/default` for the local file +unset REVIEW_SEMGREP_REGISTRY # a host that sets it to 0 would drop the `--config p/default` this test expects run() { PATH="$T/stub:$PATH" bash "$RC" --base base "$@" 2>"$T/err" >"$T/out"; } run diff --git a/scripts/test-rules-locale.sh b/scripts/test-rules-locale.sh new file mode 100644 index 000000000..63ffc9af9 --- /dev/null +++ b/scripts/test-rules-locale.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# Tests that `ci-gates.sh rules` answers the same under en_US.UTF-8 as under the C locale (#1592). Its rows are C-sorted, so its +# comm calls must run in the C locale too: `Zeta` sorts before `alpha` in C but after it in en_US.UTF-8, which made comm see +# unsorted input and report false FAILs. Hermetic: an own git fixture under $TMPDIR, no network. +# Prints PASS/FAIL per case and a final "Tests: P/N"; exits non-zero on any failure. +set -uo pipefail +export GIT_CONFIG_NOSYSTEM=1 GIT_CONFIG_GLOBAL=/dev/null # the fixture commits must not depend on the host's git config +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/dcr-test-rules-locale.XXXXXX")" || exit 1 +trap 'rm -rf "$WORK"' EXIT +GATE="$ROOT/scripts/ci-gates.sh" +pass=0 total=0 +ok() { total=$((total + 1)); if [ "$1" -eq 0 ]; then pass=$((pass + 1)); echo "PASS $2"; else echo "FAIL $2"; fi; } + +# Two unenforced hard rules in one SKILL.md. The baseline also holds a stale row (Zoo) that sits between them in C order. +R="$WORK/r"; SK="$R/.claude/skills/demo/SKILL.md"; RB="$R/scripts/rule-enforcement-baseline.tsv" +mkdir -p "${SK%/*}" "${RB%/*}" +git -C "$R" init -q; git -C "$R" config user.email test@example.com; git -C "$R" config user.name Test +SKILL_MD='# Demo\n\nZeta: you MUST do this.\n\nalpha: you MUST do that.\n' +printf '%b' "$SKILL_MD" >"$SK" +row() { printf '.claude/skills/demo/SKILL.md\t%s\n' "$1"; } +{ row 'Zeta: you MUST do this.'; row 'Zoo: you MUST do a thing since removed.'; row 'alpha: you MUST do that.'; } >"$RB" +git -C "$R" add -A >/dev/null && git -C "$R" commit -qm base && BASE="$(git -C "$R" rev-parse HEAD)" + +# en_US.UTF-8 is the locale that showed the bug. A runner without it (a minimal container) prints a SKIP and still runs the C cases. +LOCALES=(C) +if [ "$(LC_ALL=en_US.UTF-8 locale charmap 2>/dev/null)" = UTF-8 ]; then LOCALES+=(en_US.UTF-8); else echo "SKIP en_US.UTF-8 cases: that locale is not installed on this runner"; fi + +for L in "${LOCALES[@]}"; do + gate() { out="$(LC_ALL=$L bash "$GATE" rules "$@" 2>&1)"; rc=$?; } + gate --baseline "$RB" "$R" + [ "$rc" -eq 0 ] && grep -qF 'rules: ok (2 ' <<<"$out"; ok $? "$L: baselined rules pass, with a stale baseline row between them in C order" + printf '\nYak: you MUST do the third thing.\n' >>"$SK"; gate --baseline "$RB" "$R" + [ "$rc" -eq 1 ] && grep -qF 'Yak: you MUST' <<<"$out" && ! grep -qF 'alpha: you MUST' <<<"$out"; ok $? "$L: a new rule is named and the baselined ones are not" + printf '%b' "$SKILL_MD" >"$SK" +done + +{ row 'Zeta: you MUST do this.'; row 'alpha: you MUST do that.'; } >"$RB"; git -C "$R" commit -qam shrink +for L in "${LOCALES[@]}"; do + out="$(LC_ALL=$L bash "$GATE" rules --baseline "$RB" --base "$BASE" "$R" 2>&1)"; rc=$? + [ "$rc" -eq 0 ] && grep -qF 'rules: ok (2 ' <<<"$out"; ok $? "$L: a baseline that only shrank vs base is not growth" +done + +echo "Tests: $pass/$total" +[ "$pass" -eq "$total" ] diff --git a/scripts/test-tests-wired.sh b/scripts/test-tests-wired.sh new file mode 100644 index 000000000..1afb8781d --- /dev/null +++ b/scripts/test-tests-wired.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# Tests for `ci-gates.sh tests-wired` (#1592): a test that no CI step runs fails the gate, and the allowlist row that excuses one +# needs a reason. Hermetic: each case builds a throwaway fixture root under $TMPDIR (no repo state, no network). +# Prints PASS/FAIL per case and a final "Tests: P/N"; exits non-zero on any failure. +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +WORK="$(mktemp -d "${TMPDIR:-/tmp}/dcr-test-wired.XXXXXX")" || exit 1 +trap 'rm -rf "$WORK"' EXIT +GATE="$ROOT/scripts/ci-gates.sh" +pass=0 total=0 +ok() { total=$((total + 1)); if [ "$1" -eq 0 ]; then pass=$((pass + 1)); echo "PASS $2"; else echo "FAIL $2"; fi; } + +# mk : a fixture root where test_a.py is run by a ci-tests.d registration and test-b.sh by a workflow step +mk() { + R="$WORK/$1"; mkdir -p "$R/scripts/ci-tests.d" "$R/.github/workflows" + : >"$R/scripts/test_a.py"; : >"$R/scripts/test-b.sh"; : >"$R/allow.tsv" + echo 'python3 scripts/test_a.py' >"$R/scripts/ci-tests.d/a.sh" + printf ' - run: bash scripts/test-b.sh\n' >"$R/.github/workflows/ci.yml" +} +LOC="" # extra `env` words for the gate runs that follow; the collation cases at the end set it +# shellcheck disable=SC2086 # $LOC is meant to split into env assignments +gate() { out="$(env $LOC bash "$GATE" tests-wired --allowlist "${1:-$R/allow.tsv}" "$R" 2>&1)"; rc=$?; } +passes() { [ "$rc" -eq 0 ] && grep -qF -- "$1" <<<"$out"; ok $? "$2"; } #