fix(exec): pause the project watcher during the gate and retry once on an infra exit

CHANGELOG.md

@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
 
 ## Unreleased
 
+- fix(exec): **the gate pauses this project's `mxcli run --local --watch` while mxbuild runs and retries once on an infra exit.** The gate's mxbuild and a live watcher both write `deployment/`; in a field migration project (12 modules) 17 of 27 failed gates were "gate could not run" (mxbuild exit 1, no CE code), against 0 of 22 in watcher-free copies. The watcher is stopped (SIGSTOP) for the gate and resumed (SIGCONT, also in the EXIT trap); an exit 1 or 3 with no CE code and no errors file is retried once and the BUILD-LOG row says `retry after infra exit`; a final failure still takes the #424 rollback. The match is exact (the watcher's project dir must equal this project's root, so a sibling `-copy` or a nested worktree watcher is never paused), and the paused pids are recorded in `.mpr-snapshots/.watcher-paused` so the next exec.sh resumes a watcher left stopped by a SIGKILLed one and says so. Test: `tests/wave2/test-exec-gate-retry.sh` (Refs #441, #444, #171, #285, #424) — field migration project
 - learn(load-test): **trap 9, an unlicensed local runtime caps concurrent sessions at 6 ("Maximum number of sessions exceeded!") and stops itself after a random 2 to 4 hours, both reading as the app failing under load; more than 6 users needs a licensed runtime or a cloud sandbox.** Text only, ondemand skill. Source: a personal licence-budget gate written after parallel journey runs hit both — MendixMau
 - fix(progress-report): **the full plan tables (one table per phase plus views A and B) now sit in one closed `<details>` card below the phase cards, so a large plan no longer pushes the rest of the report off screen; the phase cards above stay the summary, and nothing is written when the project has no plan fragment.** Test: P6 in `test-progress-report-polish.sh` — MendixMau
 - fix(build-plan-status): **every State the build-plan skill lists (built, partial, not built, blocked) and every spelling the phase card already counts (complete, verified, in progress, not needed and the like) is now read the same way by the plan header and the phase badge, so card, header and badge give one count; blocked counts as not built and is named in the header, genuinely unreadable values are still reported as unreadable; the later-stage placeholder FAIL is relabelled PENDING only while that stage is pending in PROJECT.md, with tests that a real FAIL stays FAIL and that FAIL stays when the current stage is unknown** — MendixMau

project-bin/_common.sh

@@ -569,6 +569,138 @@ find_java_exe() {
   esac
 }
 
+# ---------------------------------------------------------------------------
+# Watcher pause and infra retry around the exec gate's mxbuild.
+#
+# Field (a migration project, 12 modules, 2026-10-16/17): the gate's mxbuild and a live
+# `mxcli run --local --watch` on the SAME project both write deployment/. One of the two dies
+# part-way ("Directory not empty: .../deployment/web", mxbuild exit 1 or 3, no errors file):
+# 17 of 27 failed gates, against 0 of 22 gates run in watcher-free copies of the same project.
+#
+# Chosen fix: pause the watcher for the length of the gate (SIGSTOP, then SIGCONT). mxbuild
+# has no option that moves the deploy target's output out of <project>/deployment, so a
+# separate output dir is not available; a pause needs no mxbuild support and no change to the
+# watcher. Only the watcher process itself is stopped. Its runtime keeps serving, and a rebuild
+# the watcher already started before the pause can still collide, which is what the retry below
+# is for. exec.sh resumes the watcher in its EXIT trap too, so an interrupted run does not leave
+# it stopped. A SIGKILL runs no trap, so the paused pids are also written to
+# .mpr-snapshots/.watcher-paused; the next exec.sh resumes them once it holds the lock. The file
+# is a dot FILE: the `*/` snapshot globs in restore-mpr.sh and snapshot-mpr.sh never match it.
+# ---------------------------------------------------------------------------
+MXTK_PAUSED_PIDS=""
+MXTK_PAUSED_FILE=""
+MXTK_GATE_RETRY_NOTE=""
+
+# mxtk_watcher_pids <project-root> - pids of `mxcli run --local --watch` that belong to this
+# project. Exact match only: the watcher's project dir (the directory of its -p argument,
+# resolved against its cwd, or its cwd when it has no -p) must equal the root after pwd -P. A
+# substring test also hit a sibling `<App>-copy` and a nested `.claude/worktrees/x` watcher.
+mxtk_watcher_pids() {
+  local _root _pid _cmd _cwd _p _dir
+  _root="$(cd "$1" 2>/dev/null && pwd -P)" || return 0
+  for _pid in $(pgrep -f 'run --local' 2>/dev/null || true); do  # portability-ok: pgrep -f ships with Linux procps and macOS; Windows has no watcher to pause here
+    [ "$_pid" = "$$" ] && continue
+    _cmd="$(ps -o command= -p "$_pid" 2>/dev/null)" || continue
+    case "$_cmd" in *mxcli*"run --local"*--watch*) ;; *) continue ;; esac
+    _cwd="$(readlink "/proc/$_pid/cwd" 2>/dev/null || lsof -a -p "$_pid" -d cwd -Fn 2>/dev/null | sed -n 's/^n//p')"
+    [ -n "$_cwd" ] && _cwd="$(cd "$_cwd" 2>/dev/null && pwd -P)"
+    _p=""
+    case "$_cmd" in
+      *" -p "*) _p="${_cmd#* -p }" ;;
+      *" --project "*) _p="${_cmd#* --project }" ;;
+    esac
+    if [ -n "$_p" ]; then
+      case "$_p" in /*) ;; *) [ -n "$_cwd" ] || continue; _p="$_cwd/$_p" ;; esac
+      # ps drops the quotes, so "-p /x/App copy/App.mpr" must not read as /x/App: keep the
+      # longest run of words that exists on disk (it also drops any flags after the path).
+      while [ ! -e "$_p" ]; do case "$_p" in *" "*) _p="${_p% *}" ;; *) continue 2 ;; esac; done
+      if [ -d "$_p" ]; then _dir="$_p"; else _dir="$(dirname "$_p")"; fi
+      _dir="$(cd "$_dir" 2>/dev/null && pwd -P)" || continue
+    else
+      _dir="$_cwd"
+    fi
+    [ -n "$_dir" ] && [ "$_dir" = "$_root" ] && echo "$_pid"
+  done
+  return 0
+}
+
+# mxtk_pause_watchers <project-root> - SIGSTOP every watcher of this project, remember the pids
+# (in memory, and in <root>/.mpr-snapshots/.watcher-paused for a SIGKILL of this process).
+mxtk_pause_watchers() {
+  local _pid _cands
+  MXTK_PAUSED_PIDS=""
+  MXTK_PAUSED_FILE="$1/.mpr-snapshots/.watcher-paused"
+  _cands="$(mxtk_watcher_pids "$1" | tr '\n' ' ')"
+  [ -n "$_cands" ] || return 0
+  # Record before stopping: a SIGKILL between kill -STOP and the record would leave a stopped
+  # watcher nobody resumes. Resuming a pid that was never stopped is harmless.
+  { mkdir -p "$1/.mpr-snapshots" && echo "$$ $_cands" > "$MXTK_PAUSED_FILE"; } 2>/dev/null || true
+  for _pid in $_cands; do
+    if kill -STOP "$_pid" 2>/dev/null; then
+      MXTK_PAUSED_PIDS="$MXTK_PAUSED_PIDS $_pid"
+      echo "  → paused the run --local --watch watcher (pid $_pid) for the gate; it resumes right after" >&2
+    fi
+  done
+  if [ -n "$MXTK_PAUSED_PIDS" ]; then
+    { mkdir -p "$1/.mpr-snapshots" && echo "$$$MXTK_PAUSED_PIDS" > "$MXTK_PAUSED_FILE"; } 2>/dev/null || true
+  fi
+  return 0
+}
+
+# mxtk_resume_watchers - SIGCONT whatever mxtk_pause_watchers stopped. Safe to call twice.
+mxtk_resume_watchers() {
+  local _pid
+  for _pid in $MXTK_PAUSED_PIDS; do kill -CONT "$_pid" 2>/dev/null || true; done
+  [ -n "$MXTK_PAUSED_FILE" ] && rm -f "$MXTK_PAUSED_FILE"
+  MXTK_PAUSED_PIDS=""
+  return 0
+}
+
+# mxtk_resume_leftover <project-root> - call once the exec lock is held. A watcher left stopped
+# by an exec.sh that was SIGKILLed (its record is still on disk) is resumed when its pid still
+# exists and is still a watcher of this project; the record is then cleared.
+mxtk_resume_leftover() {
+  local _f="$1/.mpr-snapshots/.watcher-paused" _rec _pid _live _n=0
+  [ -f "$_f" ] || return 0
+  _rec="$(head -1 "$_f" 2>/dev/null || true)"
+  _live="$(mxtk_watcher_pids "$1")"
+  for _pid in ${_rec#* }; do
+    case "$_pid" in ''|*[!0-9]*) continue ;; esac
+    if grep -Fxq -- "$_pid" <<<"$_live" && kill -CONT "$_pid" 2>/dev/null; then _n=$((_n+1)); fi
+  done
+  rm -f "$_f"
+  [ "$_n" -gt 0 ] && echo "  → resumed $_n watcher(s) left stopped by an exec.sh that was killed during the gate"
+  return 0
+}
+
+# mxtk_gate_mxbuild <mpr> <timeout-seconds> <errors-file> - the exec gate's way to run
+# mxbuild: mxtk_mxbuild_error_count with this project's watcher paused around it, and ONE
+# retry when mxbuild exits 1 or 3 and left no sign of a model problem (no CE code in its
+# output, nothing in the errors file). That shape is an infrastructure failure, not a verdict
+# on the model; a real model error always names a CE code. An errors file with content (for
+# example a version mismatch in errors[]) is deterministic and is not retried.
+# Prints nothing on stdout (callers discard the count); sets the same MXTK_MXBUILD_* globals as
+# mxtk_mxbuild_error_count, plus MXTK_GATE_RETRY_NOTE ("retry after infra exit" or empty).
+# Returns mxtk_mxbuild_error_count's code for the final attempt.
+mxtk_gate_mxbuild() {
+  local _mpr="$1" _timeout="${2:-0}" _ef="$3" _rc=0
+  MXTK_GATE_RETRY_NOTE=""
+  mxtk_pause_watchers "${PROJECT_ROOT:-$(dirname "$_mpr")}"
+  mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$?
+  case "${MXTK_MXBUILD_EXIT:-}" in
+    1|3)
+      if [ ! -s "$_ef" ] && ! grep -Eq -- 'CE[0-9]{4}' <<<"${MXTK_MXBUILD_OUT:-}"; then
+        echo "  ⚠ retry after infra exit: mxbuild exited $MXTK_MXBUILD_EXIT with no CE code and no errors file; building once more" >&2
+        MXTK_GATE_RETRY_NOTE="retry after infra exit"
+        : > "$_ef"
+        _rc=0
+        mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$?
+      fi ;;
+  esac
+  mxtk_resume_watchers
+  return "$_rc"
+}
+
 # ---------------------------------------------------------------------------
 # mxtk_mxbuild_error_count — run mxbuild once, print its Error-severity count.
 #

project-bin/exec.sh

@@ -437,6 +437,7 @@ BUILD_LOG="$PROJECT_ROOT/docs/BUILD-LOG.md"
 GATE_STATE="not-run"   # not-run | skipped | unverified | pass | fail
 GATE_ERRORS="?"        # mxbuild error count behind a pass (0 = clean); recorded in the stamp
 MXB_WHY=""             # mxbuild's errors[] reason when it exited without checking the model
+GATE_NOTE=""           # "retry after infra exit" when a gate mxbuild was run twice
 
 # ── Exec approval (auto records itself) ──────────────────────────────────────
 # bin/exec-approval.sh decides ask vs auto; the ASKING happens before exec.sh ever runs (in
@@ -490,7 +491,7 @@ HDR2
   # The gate cell is read from GATE_STATE rather than passed in, so no caller
   # can omit it — a blank cell in this table reads as "fine", which is the exact
   # false-green the gate exists to prevent.
-  _detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}"
+  _detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${GATE_NOTE:+ · $GATE_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}"
   printf '| %s | `%s` | %s | %s | %s |\n' \
     "$(date '+%Y-%m-%dT%H:%M:%S%z')" "$LOG_NAME" \
     "${GATE_STATE:-not-run}" "$1" "$_detail" >> "$BUILD_LOG"
@@ -708,7 +709,8 @@ if [ "$LOCK_TAKEN" != 1 ]; then
   echo "  → Wait for the other exec to finish, then re-run. Nothing was written."
   exit 1
 fi
-trap 'exec_lock_release' EXIT
+trap 'mxtk_resume_watchers; exec_lock_release' EXIT
+mxtk_resume_leftover "$PROJECT_ROOT"   # a watcher an earlier, SIGKILLed exec.sh left stopped
 
 # 3. A raw `mxcli exec` elsewhere on the machine is NOT this project's business. The old guard
 #    here (`pgrep -fl "mxcli exec"`) matched every exec on the machine, so a test run on a
@@ -1353,7 +1355,8 @@ if [ "${SKIP_BASELINE:-0}" != "1" ] && [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ];
   # can drift apart. Its own printed count is discarded here: this call site
   # already has err_set/err_count/err_codes on hand and needs BASELINE_SET
   # (the message set, not just a count) for the delta-gate comparison below.
-  mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true
+  mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true
+  [ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE"
   # Studio Pro 11's mxbuild (its --help says so) writes --write-errors "only if the
   # project has errors": on a clean model the mktemp'd file stays EMPTY, json.load
   # fails, and this read "Model ALREADY has ? error(s) [?]" on every clean run
@@ -1633,7 +1636,8 @@ if [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ]; then
   # discarded here: this call site keeps its own file-presence/exit-code
   # branching below unchanged, computing CE_COUNT via err_count once it knows
   # which branch applies.
-  mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true
+  mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true
+  [ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE"
   MXBUILD_EXIT="$MXTK_MXBUILD_EXIT"
   MXBUILD_OUT="$MXTK_MXBUILD_OUT"
   MXB_WHY="${MXTK_MXBUILD_WHY:-}"
@@ -1751,7 +1755,7 @@ EOF
           BASE_ERRS=$(mktemp /tmp/mxbuild-base.XXXXXX)
           # Same shared run+capture as the pre-flight baseline and the main gate
           # above (mxtk_mxbuild_error_count, _common.sh) — one implementation.
-          mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true
+          mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true
           # Same empty-file rule as the pre-flight baseline: SP 11 mxbuild writes no
           # errors file on a clean build, and that plus exit 0 is a verified 0.
           if [ ! -s "$BASE_ERRS" ] && [ "$MXTK_MXBUILD_EXIT" -eq 0 ]; then