new(security): check-staged-secrets.sh, an opt-in staged-blob secret check (TK-025)

CHANGELOG.md

@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
 
 ## Unreleased
 
+- new(security): **`project-bin/check-staged-secrets.sh` — compare the project's real secret values (`.env`, `.mxcli/run-local.json` bootConfig) against the staged blobs (or `--tree`, every tracked file at HEAD) and print only the file and the key name, never a value (TK-025).** Plus a section in `learned-constants-and-secrets.md`: the secret-bearing files and commands an agent never prints (`run-local.json`, the `run --local` command line, run state, `describe constant`, `.env`, rendered files that interpolate a secret), read-by-testing habits, and "a leaked value is rotated, not deleted". Field evidence: a five-app lab run with six leak events into transcripts and a demo password in git (mxcli v0.26.0). Opt-in, not wired into the generated pre-commit hook. Test with made-up values: `tests/wave2/test-check-staged-secrets.sh` (15 checks). Not adopted: the report's `exec-secret.sh` is a project-local script that does not ship here, and masking `--constant` values in `app-restart.sh` is TK-003. Values under 8 characters, well-known defaults such as `mendix` (the mxcli database password, present in thousands of tracked files) and values equal to `.env.example` are not sources, and `.docker/.env` is read. It runs one `git grep -F -f` per value, not a process per file: 230 staged files in a 6,000-file synthetic repo took 1.0 s (the whole tree 2.6 s). — five-app lab report, TK-025
 - fix(wire-agents): **a sync re-wire no longer wipes `.claude/settings.json`.** `mxcli init` overwrites that file; wire-agents now snapshots it and merges after init: the project's entries win and keep their order (permission profile, user entries, the `mxcli exec` deny rule, hooks), and entries that only mxcli's new file has are appended. Merged rather than added to PRESERVE because PRESERVE restores old bytes and would also drop anything a newer mxcli adds; an unparsable snapshot is restored byte for byte. Test: `tests/wave2/test-wire-agents-settings.sh` (4 of 6 checks fail on master). — review of #414 (#419)
 - fix(sync): **the exec-approval item-3 rewrite now works on BSD awk (stock macOS) and `sync-project.sh` no longer prints "Rewrote" for a rewrite that failed.** `_ea_rewrite_item3` passed the multi-line replacement through `awk -v`, which BSD awk rejects ("newline in string", exit 2); the file stayed unchanged and sync still reported the rewrite on every run. The replacement is now streamed from a temp file (as the Q9 repair does), the helper returns 2 when awk fails, and the caller prints a `WARN:` instead of "Rewrote" and does not count a change. `test-exec-approval.sh` gains 3 assertions (failing-awk shim: no "Rewrote", a WARN, file unchanged). No other multi-line `awk -v` call site exists (the rest are single-line values) ([#426](https://github.com/MendixMau/mxcli-project-toolkit/issues/426)) — MendixMau
 - process(testing): **unit tests are now off by default and `affected-tests.sh` enforces it; each project picks its test strategy with the developer at CAC-5.** Fast-tracked by the maintainer on 2026-10-16 because users report tests firing on every change (the baseline-text A/B is waived; E3 follows as a check). Moments: every change = gate plus what it touched (unit only for affected modules with unit on); module done = that module's journeys (+ its unit tests if on); build complete and before every later deploy = everything once; "run everything" any time. The CAC-5 Offer records it in PROJECT.md `## Test strategy` (Status `ASSUMED` in auto mode). `affected-tests.sh` reads `Unit tests default:` (`no` or absent means unit off, one `unit off` line, never a `.test.mdl` run row); turn unit on per project with `Unit tests default: yes`, per module with `Unit modules: Shop, Admin` in PROJECT.md or `Unit tests: yes` in the module brief; the strategy row's Unit cell is a scope only and never turns unit on. A grant on a microflow or page the same script changes is part of that subject and no longer selects the owning module's tests. `module-review.md` PROVE now runs `--moment module --module <M>`. `testing-shape.md`, `checkpoint-build.md`, `iterative-build-loop.md`, `conversion-runbook.md` and the exec.sh logic lane text updated. `tests/wave2/test-test-strategy-docs.sh` and the extended `test-affected-tests.sh` pin it. Baseline words stay at or below master (75,219 vs 75,225). Split out of #409; found by the field test of the release candidate — the maintainer's decision, 2026-10-16

bin/lib/install-manifest.sh

@@ -115,7 +115,7 @@ MXTK_AGENTS_STAGE_BUILD="mdl-agent.md gate-agent.md test-agent.md review-agent.m
 # may run itself (stops only the pid run --local recorded for THIS project), and `mxcli test` on a
 # scratch copy under the test-port lock. exec.sh names app-restart.sh in its watch advisory, so
 # it has to be installed beside it.
-MXTK_PROJECT_BIN="app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py mxpatch.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh affected-tests.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh"
+MXTK_PROJECT_BIN="app-restart.sh test-scratch.sh assemble-prototype.js prototype-route.js check-prototype-links.js wf-add-path-terminators.py wf-set-call-captions.py mxunit_bson.py mxpatch.py _common.sh app-facts.sh _claims.sh snapshot-mpr.sh restore-mpr.sh exec.sh affected-tests.sh look-ledger.sh save-sp.sh restart-sp.sh check-sp-health.sh verify-module.sh test-stack-up.sh fixture-manifest.sh check-root-clean.sh lint-gate.sh lint-trend.sh metrics-ledger.sh metrics-trend.sh close-task.sh conformance-check.sh coverage-preflight.sh graph-sweep.sh review-module.sh coherence-cadence.sh build-plan-status.sh done-drift-check.sh page-scope.sh render-improvement-register.sh check-design-portability.sh check-design-reaches-app.sh check-page-shell.sh page-fidelity.js model-stamp.sh verify-model.sh install-project-hooks.sh mx-metadata-note.py session-check.sh constants-audit.sh context-pack.sh check-staged-secrets.sh"
 
 # Files in project-bin/ that are deliberately NOT installed into projects. The reverse check
 # below flags anything named by NEITHER list, so a new file in project-bin/ has to be either

project-bin/check-staged-secrets.sh

@@ -0,0 +1,182 @@
+#!/usr/bin/env bash
+# check-staged-secrets.sh — fail when a secret VALUE from this project's own secret-bearing files
+# is about to be committed (staged blobs) or is already in the tracked tree. Prints the file and
+# the NAME of the key that matched. It never prints a value.
+#
+# WHY THIS EXISTS (field run, five apps, mxcli v0.26.0). A demo password reached git through a
+# rendered login page, and several lane transcripts through `.mxcli/run-local.json`, whose
+# bootConfig holds database and API passwords in clear. `.gitignore` does not help once a file is
+# tracked, and `git commit -- <path>` re-adds an ignored tracked file at commit time. The one
+# check that sees what git will actually store is a scan of the blobs, against the real values.
+#
+# Where the values come from (read here, never echoed):
+#   .env  .env.*               at the project root and in .docker/, docker/, .devcontainer/ (the
+#                              `mxcli docker` layout keeps M2EE_ADMIN_PASS and DB_PASSWORD in
+#                              .docker/.env). KEY=VALUE where KEY looks like PASSWORD / SECRET /
+#                              TOKEN / KEY / PAT. CRLF, `KEY = value`, quotes and an unquoted
+#                              trailing ` # comment` are normalised.
+#   .mxcli/run-local.json      every string in bootConfig whose key looks like one of those
+#                              Exception: a bootConfig.MicroflowConstants value that is already in
+#                              the committed model (HEAD, *.mpr or mprcontents/). `run --local` writes
+#                              EVERY constant's model default there, so a placeholder default such as
+#                              `your-api-key-here` is the model's own text, not a leak, and unstaging
+#                              cannot fix it. A real credential in a model default is already
+#                              committed: that is bin/constants-audit.sh's job, not this script's. A
+#                              constant passed with a value that is NOT in the model is still a source.
+#   $MXTK_SECRET_VALUES_FILE   optional: one literal value per line (name shown as "extra");
+#                              the filters below do NOT apply, you named it a secret
+# What is NOT treated as a secret, and why (this is a leak detector, not a password-strength
+# check; a weak demo password is not protected by it):
+#   - values under 8 characters: too short to scan for without matching ordinary text;
+#   - well-known development defaults (mendix, password, admin, postgres, changeme, ...): every
+#     mxcli project carries `mendix` as the database password, so it sits in thousands of tracked
+#     files (model blobs, java sources, docs) and would fail every commit. A check that is red on
+#     every commit gets switched off, which is worse than not guarding a published default;
+#   - a value equal to the same key's value in a sibling .env.example/.sample/.template file:
+#     that value is already published on purpose.
+# Example files are never sources.
+#
+# Speed: one `git grep -l -F -f <600 temp file>` per value, over the staged paths (--cached) or
+# HEAD, instead of a process per file per value. The value goes through a mode 600 file, never
+# argv, so `ps` cannot show it.
+#
+# Usage:
+#   bin/check-staged-secrets.sh            # staged blobs (what the next commit stores)
+#   bin/check-staged-secrets.sh --tree     # every tracked file at HEAD (the "already pushed?" check)
+# Hook it yourself (it is NOT wired into the generated pre-commit):
+#   "$(git rev-parse --show-toplevel)/bin/check-staged-secrets.sh" || exit 1
+#
+# Exit: 0 clean (or no sources found) · 1 a secret value is in the scanned blobs · 2 not a git repo
+set -u
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="${PROJECT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
+cd "$PROJECT_ROOT" || { echo "check-staged-secrets: cannot cd to $PROJECT_ROOT" >&2; exit 2; }
+git rev-parse --show-toplevel >/dev/null 2>&1 || { echo "check-staged-secrets: not a git repository" >&2; exit 2; }
+MODE=staged; [ "${1:-}" = "--tree" ] && MODE=tree
+# Non-fatal interpreter probe: without Python the named-key scan is skipped, the rest still runs.
+# resolve_py is _common.sh's (portable.sh is toolkit-side and not copied into projects).
+[ -f "$SCRIPT_DIR/_common.sh" ] && . "$SCRIPT_DIR/_common.sh"
+if ! type resolve_py >/dev/null 2>&1; then
+  resolve_py() {
+    _c=""
+    for _c in "${PYTHON:-}" python3 python py; do  # portability-ok: this IS the interpreter probe
+      [ -n "$_c" ] || continue
+      case "$(command -v "$_c" 2>/dev/null)" in *[Ww]indows[Aa]pps*) continue ;; esac
+      if "$_c" -c 'import sys; sys.exit(0 if sys.version_info[0] == 3 else 1)' >/dev/null 2>&1; then
+        echo "$_c"; return 0
+      fi
+    done
+    return 1
+  }
+fi
+PY="$(resolve_py || true)"
+
+NAMEPAT='(PASS|SECRET|TOKEN|KEY|PAT$|CREDENTIAL)'
+MINLEN=8
+# Well-known development defaults, lowercase. Compared case-insensitively.
+DEFAULTS=" mendix password passw0rd admin administrator postgres postgresql changeme change-me change_me secret root test testing demo example default "
+SRC="$(mktemp "${TMPDIR:-/tmp}/mxtk-secrets.XXXXXX")"
+VF="$(mktemp "${TMPDIR:-/tmp}/mxtk-secval.XXXXXX")"
+trap 'rm -f "$SRC" "$VF"' EXIT
+chmod 600 "$SRC" "$VF"
+
+# name<TAB>value, one per line, before filtering.
+for f in .env .env.* .docker/.env .docker/.env.* docker/.env docker/.env.* .devcontainer/.env .devcontainer/.env.*; do
+  [ -f "$f" ] || continue
+  case "$f" in *.example|*.sample|*.template) continue ;; esac
+  d="$(dirname "$f")"; ex=""
+  for e in "$d/.env.example" "$d/.env.sample" "$d/.env.template"; do [ -f "$e" ] && ex="$ex $e"; done
+  # shellcheck disable=SC2086
+  awk -v NAMEPAT="$NAMEPAT" -v SRCF="$f" '
+    function trim(x) { sub(/^[ \t\r]+/, "", x); sub(/[ \t\r]+$/, "", x); return x }
+    function parse(line, kv,   k, v, i) {
+      sub(/\r$/, "", line); line = trim(line)
+      if (line == "" || substr(line, 1, 1) == "#") return 0
+      sub(/^export[ \t]+/, "", line)
+      i = index(line, "="); if (i == 0) return 0
+      k = trim(substr(line, 1, i - 1)); v = trim(substr(line, i + 1))
+      if (v ~ /^"/) { sub(/^"/, "", v); sub(/".*$/, "", v) }
+      else if (v ~ /^\x27/) { sub(/^\x27/, "", v); sub(/\x27.*$/, "", v) }
+      else { sub(/[ \t]+#.*$/, "", v); v = trim(v) }
+      kv[1] = k; kv[2] = v; return 1
+    }
+    FNR == 1 { isex = (FILENAME != SRCF) }
+    { split("", kv); if (!parse($0, kv)) next
+      if (isex) { exv[kv[1]] = kv[2]; next }
+      if (toupper(kv[1]) !~ NAMEPAT) next
+      if ((kv[1] in exv) && exv[kv[1]] == kv[2]) next
+      printf "%s:%s\t%s\n", SRCF, kv[1], kv[2] }
+  ' $ex "$f" >> "$SRC"
+done
+if [ -f .mxcli/run-local.json ] && [ -n "$PY" ]; then
+  "$PY" -I - "$NAMEPAT" >> "$SRC" 2>/dev/null <<'PYEOF'
+import json, re, sys
+pat = re.compile(sys.argv[1], re.I)
+try:
+    data = json.load(open(".mxcli/run-local.json"))
+except Exception:
+    sys.exit(0)
+def walk(o, path=""):
+    if isinstance(o, dict):
+        for k, v in o.items():
+            yield from walk(v, path + "." + k if path else k)
+    elif isinstance(o, str) and "bootConfig" in path:
+        yield path, o
+for p, v in walk(data):
+    if pat.search(p.rsplit(".", 1)[-1]) and "\t" not in v and "\n" not in v:
+        print(".mxcli/run-local.json:" + p.split("bootConfig.", 1)[-1] + "\t" + v)
+PYEOF
+fi
+
+# Filter: length and known defaults (lowercased compare). Then the explicit extra values.
+FILT="$(mktemp "${TMPDIR:-/tmp}/mxtk-secfilt.XXXXXX")"; chmod 600 "$FILT"; trap 'rm -f "$SRC" "$VF" "$FILT"' EXIT
+skipped=0
+while IFS="$(printf '\t')" read -r name value; do
+  [ -n "$name" ] || continue
+  lv="$(printf '%s' "$value" | tr 'A-Z' 'a-z')"
+  case "$DEFAULTS" in *" $lv "*) skipped=$((skipped + 1)); continue ;; esac
+  [ "${#value}" -ge "$MINLEN" ] || { skipped=$((skipped + 1)); continue; }
+  case "$name" in
+    .mxcli/run-local.json:MicroflowConstants.*)
+      printf '%s\n' "$value" > "$VF"
+      if git grep -q -F -f "$VF" HEAD -- '*.mpr' 'mprcontents/' 2>/dev/null; then skipped=$((skipped + 1)); continue; fi ;;
+  esac
+  printf '%s\t%s\n' "$name" "$value" >> "$FILT"
+done < "$SRC"
+if [ -n "${MXTK_SECRET_VALUES_FILE:-}" ] && [ -f "$MXTK_SECRET_VALUES_FILE" ]; then
+  while IFS= read -r v; do v="${v%$(printf '\r')}"; [ -n "$v" ] && printf 'extra\t%s\n' "$v"; done < "$MXTK_SECRET_VALUES_FILE" >> "$FILT"
+fi
+sort -u "$FILT" -o "$FILT"
+n="$(wc -l < "$FILT" | tr -d ' ')"
+if [ "$n" -eq 0 ]; then
+  echo "check-staged-secrets: no secret values to scan for (none found, or $skipped skipped as known defaults / under $MINLEN chars) — nothing checked."
+  exit 0
+fi
+
+if [ "$MODE" = staged ]; then
+  STAGED="$(git diff --cached --name-only -z --diff-filter=ACMR | tr '\0' '\n' | head -c 1)"
+  if [ -z "$STAGED" ]; then echo "✓ check-staged-secrets: nothing staged ($n value(s) would be checked)."; exit 0; fi
+fi
+hits=0
+while IFS="$(printf '\t')" read -r name value; do
+  printf '%s\n' "$value" > "$VF"
+  if [ "$MODE" = tree ]; then
+    files="$(git grep -l -F -f "$VF" HEAD 2>/dev/null | sed 's/^HEAD://')"
+  else
+    files="$(git diff --cached --name-only -z --diff-filter=ACMR | xargs -0 git --literal-pathspecs grep --cached -l -F -f "$VF" -- 2>/dev/null)"
+  fi
+  [ -n "$files" ] || continue
+  while IFS= read -r f; do
+    echo "  ✗ $f holds the value of $name"
+    hits=$((hits + 1))
+  done <<EOF2
+$files
+EOF2
+done < "$FILT"
+if [ "$hits" -gt 0 ]; then
+  echo "✗ check-staged-secrets: $hits secret value(s) in $MODE blobs. Values are not shown."
+  echo "  → Unstage the file (git rm --cached <file>), ignore it, keep the template without the value."
+  echo "  → A value that reached a remote or a transcript is spent: rotate it. Removing the commit does not."
+  exit 1
+fi
+echo "✓ check-staged-secrets: no secret value in the $MODE blobs ($n value(s) checked, $skipped skipped as defaults / under $MINLEN chars)."