From 25f9a9955dc50a8da04618677885922d73d1deea Mon Sep 17 00:00:00 2001 From: Ryan Breen Date: Sat, 5 Sep 2026 19:24:23 -0400 Subject: [PATCH 1/6] gates(834): wire scripts/*.sh aarch64 launchers into qemu-host-lock #834 found six scripts/ scripts launching qemu-system-aarch64 with 0 of them routed through docker/qemu/lib/qemu-host-lock.sh (#826/R181's own fix, scoped to docker/qemu/*.sh only). Each of the six now sources the helper and calls qemu_host_lock_acquire before its launch, with qemu_host_lock_track_pid registering the PID right after capture. Three of the six (run-arm64-qemu.sh, run-arm64-graphics.sh, run-aarch64-userspace.sh) ended their launch with `exec`, which would discard the lock's own EXIT trap before it could release -- exec is dropped for a plain foreground launch so the trap-based release still runs when QEMU exits; the interactive session itself is unchanged (same terminal, same Ctrl-A X exit). test_tracing_via_gdb.sh picks qemu-system-aarch64 or qemu-system-x86_64 at runtime, so the lock call is placed on its aarch64 leg only. Co-Authored-By: Claude Fable 5.1 --- scripts/run-aarch64-userspace.sh | 16 +++++++++++++++- scripts/run-arm64-boot-test.sh | 13 +++++++++++++ scripts/run-arm64-graphics.sh | 16 +++++++++++++++- scripts/run-arm64-keyboard-test.sh | 13 +++++++++++++ scripts/run-arm64-qemu.sh | 16 +++++++++++++++- scripts/test_tracing_via_gdb.sh | 17 +++++++++++++++++ 6 files changed, 88 insertions(+), 3 deletions(-) diff --git a/scripts/run-aarch64-userspace.sh b/scripts/run-aarch64-userspace.sh index bf9f7c8f2..684190453 100755 --- a/scripts/run-aarch64-userspace.sh +++ b/scripts/run-aarch64-userspace.sh @@ -6,6 +6,13 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" cd "$BREENIX_ROOT" +# #826/#834/R181: this script's qemu-system-aarch64 boot runs behind the +# host-wide lock in docker/qemu/lib/qemu-host-lock.sh -- #834 extends that +# lock's coverage from docker/qemu/*.sh (its original #826/R181 scope) to +# scripts/ as well. +# shellcheck source=../docker/qemu/lib/qemu-host-lock.sh +source "$BREENIX_ROOT/docker/qemu/lib/qemu-host-lock.sh" + # Build ARM64 kernel KERNEL="$BREENIX_ROOT/target/aarch64-breenix-kernel/release/kernel-aarch64" if [ ! -f "$KERNEL" ]; then @@ -48,7 +55,14 @@ case "$(uname)" in *) DISPLAY_OPT="-display sdl" ;; esac -exec qemu-system-aarch64 \ +# #834: this is an interactive display session (Ctrl-A X exits QEMU), but +# it still takes the host-wide lock -- not exempt just because a human, not +# a gate, is driving it. `exec` is dropped in favor of a plain foreground +# launch: replacing this shell's own process image would discard the +# qemu_host_lock_acquire EXIT trap before it could release the lock, since +# there would be no bash process left to run it. +qemu_host_lock_acquire +qemu-system-aarch64 \ -M virt \ -cpu cortex-a72 \ -m 512M \ diff --git a/scripts/run-arm64-boot-test.sh b/scripts/run-arm64-boot-test.sh index d4a95e240..e076b5485 100755 --- a/scripts/run-arm64-boot-test.sh +++ b/scripts/run-arm64-boot-test.sh @@ -28,6 +28,13 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" cd "$BREENIX_ROOT" +# #826/#834/R181: this script's qemu-system-aarch64 boot runs behind the +# host-wide lock in docker/qemu/lib/qemu-host-lock.sh -- #834 extends that +# lock's coverage from docker/qemu/*.sh (its original #826/R181 scope) to +# scripts/ as well. +# shellcheck source=../docker/qemu/lib/qemu-host-lock.sh +source "$BREENIX_ROOT/docker/qemu/lib/qemu-host-lock.sh" + # Configuration KERNEL_PATH="target/aarch64-breenix-kernel/release/kernel-aarch64" SERIAL_OUTPUT="/tmp/arm64_boot_test_output.txt" @@ -91,6 +98,7 @@ fi # Start QEMU in background # Always include VirtIO GPU and keyboard so the kernel's MMIO enumeration # discovers them (needed for interactive shell and device driver tests) +qemu_host_lock_acquire qemu-system-aarch64 \ -M virt \ -cpu cortex-a72 \ @@ -104,6 +112,11 @@ qemu-system-aarch64 \ $NET_OPTS \ -serial "file:$SERIAL_OUTPUT" & QEMU_PID=$! +# F2: registers QEMU with the lock's own EXIT trap (see +# docker/qemu/lib/qemu-host-lock.sh) so a SIGTERM/SIGINT delivered to just +# this script's own PID during the poll below still kills QEMU instead of +# orphaning it with the lock free. +qemu_host_lock_track_pid "$QEMU_PID" # Wait for output - different markers for different test modes echo "[3/4] Waiting for kernel output (${TIMEOUT_SECS}s timeout)..." diff --git a/scripts/run-arm64-graphics.sh b/scripts/run-arm64-graphics.sh index ba9a532eb..bd928f06d 100755 --- a/scripts/run-arm64-graphics.sh +++ b/scripts/run-arm64-graphics.sh @@ -13,6 +13,13 @@ BUILD_TYPE="${1:-release}" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +# #826/#834/R181: this script's qemu-system-aarch64 boot runs behind the +# host-wide lock in docker/qemu/lib/qemu-host-lock.sh -- #834 extends that +# lock's coverage from docker/qemu/*.sh (its original #826/R181 scope) to +# scripts/ as well. +# shellcheck source=../docker/qemu/lib/qemu-host-lock.sh +source "$BREENIX_ROOT/docker/qemu/lib/qemu-host-lock.sh" + if [ "$BUILD_TYPE" = "debug" ]; then KERNEL="$BREENIX_ROOT/target/aarch64-breenix-kernel/debug/kernel-aarch64" else @@ -86,7 +93,14 @@ esac # - VirtIO net device (MMIO) # - VirtIO keyboard device (MMIO) for keyboard input # NOTE: Use -device virtio-*-device (MMIO) not virtio-*-pci -exec qemu-system-aarch64 \ +# #834: this is an interactive display session (Ctrl-A X exits QEMU), but +# it still takes the host-wide lock -- not exempt just because a human, not +# a gate, is driving it. `exec` is dropped in favor of a plain foreground +# launch: replacing this shell's own process image would discard the +# qemu_host_lock_acquire EXIT trap before it could release the lock, since +# there would be no bash process left to run it. +qemu_host_lock_acquire +qemu-system-aarch64 \ -M virt \ -cpu cortex-a72 \ -m 512M \ diff --git a/scripts/run-arm64-keyboard-test.sh b/scripts/run-arm64-keyboard-test.sh index 86ffba39d..f8339c040 100755 --- a/scripts/run-arm64-keyboard-test.sh +++ b/scripts/run-arm64-keyboard-test.sh @@ -25,6 +25,13 @@ set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +# #826/#834/R181: this script's qemu-system-aarch64 boot runs behind the +# host-wide lock in docker/qemu/lib/qemu-host-lock.sh -- #834 extends that +# lock's coverage from docker/qemu/*.sh (its original #826/R181 scope) to +# scripts/ as well. +# shellcheck source=../docker/qemu/lib/qemu-host-lock.sh +source "$BREENIX_ROOT/docker/qemu/lib/qemu-host-lock.sh" + # Configuration SERIAL_PORT=4454 MONITOR_PORT=4455 @@ -70,6 +77,7 @@ echo "[1/5] Starting QEMU..." # Use file output for reliable logging + TCP for input # The TCP serial allows bidirectional I/O - we'll connect and send input +qemu_host_lock_acquire qemu-system-aarch64 \ -M virt -cpu cortex-a72 -m 512M \ -kernel "$KERNEL" \ @@ -82,6 +90,11 @@ qemu-system-aarch64 \ -monitor tcp:127.0.0.1:${MONITOR_PORT},server,nowait \ & QEMU_PID=$! +# F2: registers QEMU with the lock's own EXIT trap (see +# docker/qemu/lib/qemu-host-lock.sh) so a SIGTERM/SIGINT delivered to just +# this script's own PID during the test loop below still kills QEMU instead +# of orphaning it with the lock free. +qemu_host_lock_track_pid "$QEMU_PID" echo " QEMU started with PID $QEMU_PID" diff --git a/scripts/run-arm64-qemu.sh b/scripts/run-arm64-qemu.sh index 75a4a762b..ff8aaef0d 100755 --- a/scripts/run-arm64-qemu.sh +++ b/scripts/run-arm64-qemu.sh @@ -15,6 +15,13 @@ BUILD_TYPE="${1:-release}" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +# #826/#834/R181: this script's qemu-system-aarch64 boot runs behind the +# host-wide lock in docker/qemu/lib/qemu-host-lock.sh -- #834 extends that +# lock's coverage from docker/qemu/*.sh (its original #826/R181 scope) to +# scripts/ as well. +# shellcheck source=../docker/qemu/lib/qemu-host-lock.sh +source "$BREENIX_ROOT/docker/qemu/lib/qemu-host-lock.sh" + if [ "$BUILD_TYPE" = "debug" ]; then KERNEL="$BREENIX_ROOT/target/aarch64-breenix-kernel/debug/kernel-aarch64" else @@ -96,7 +103,14 @@ else DISPLAY_OPTS="-nographic" fi -exec qemu-system-aarch64 \ +# #834: this is an interactive display session (Ctrl-A X exits QEMU), but +# it still takes the host-wide lock -- not exempt just because a human, not +# a gate, is driving it. `exec` is dropped in favor of a plain foreground +# launch: replacing this shell's own process image would discard the +# qemu_host_lock_acquire EXIT trap before it could release the lock, since +# there would be no bash process left to run it. +qemu_host_lock_acquire +qemu-system-aarch64 \ -M virt \ -cpu cortex-a72 \ -m 512M \ diff --git a/scripts/test_tracing_via_gdb.sh b/scripts/test_tracing_via_gdb.sh index 9e82264e4..1f2ab2964 100755 --- a/scripts/test_tracing_via_gdb.sh +++ b/scripts/test_tracing_via_gdb.sh @@ -42,6 +42,17 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" +# #826/#834/R181: sourced unconditionally since this harness's --arch is +# runtime-selected, but only the aarch64 branch below actually calls +# qemu_host_lock_acquire -- the host-wide lock in +# docker/qemu/lib/qemu-host-lock.sh serializes qemu-system-aarch64 boots +# specifically, and this script's x86_64 leg (qemu-system-x86_64) is outside +# that lock's scope, same as run.sh's x86_64 leg. #834 extends this lock's +# coverage from docker/qemu/*.sh (its original #826/R181 scope) to scripts/ +# as well. +# shellcheck source=../docker/qemu/lib/qemu-host-lock.sh +source "$BREENIX_ROOT/docker/qemu/lib/qemu-host-lock.sh" + # #825: without --out, two concurrent invocations of this harness for the # same --arch hardcoded the identical /tmp/breenix_trace_test_$ARCH path, so # one invocation's rm -rf/mkdir could delete and rewrite another's in-flight @@ -204,6 +215,7 @@ if [ "$ARCH" = "aarch64" ]; then exit 1 fi cp "$EXT2_DISK" "$OUTPUT_DIR/ext2-writable.img" + qemu_host_lock_acquire "$QEMU_BIN" \ -M virt,gic-version=3 -cpu max -m 512 -smp 4 \ -kernel "$KERNEL_BIN" \ @@ -219,6 +231,11 @@ if [ "$ARCH" = "aarch64" ]; then -gdb "tcp::$GDB_PORT" \ >"$OUTPUT_DIR/qemu.log" 2>&1 & QEMU_PID=$! + # F2: registers QEMU with the lock's own EXIT trap (see + # docker/qemu/lib/qemu-host-lock.sh) so a SIGTERM/SIGINT delivered to + # just this script's own PID during the settle window still kills QEMU + # instead of orphaning it with the lock free. + qemu_host_lock_track_pid "$QEMU_PID" else UEFI_IMG=$(ls -t "$BREENIX_ROOT/target/release/build/breenix-"*/out/breenix-uefi.img 2>/dev/null | head -1) if [ -z "$UEFI_IMG" ]; then From 9a9f2ccf3eb9cbc309ec2c6bed3730792a198153 Mon Sep 17 00:00:00 2001 From: Ryan Breen Date: Sat, 5 Sep 2026 19:24:42 -0400 Subject: [PATCH 2/6] gates(834): wire docker/qemu-aarch64 and run.sh into qemu-host-lock A fresh `grep -rl qemu-system-aarch64 scripts/ docker/ run.sh` census (the same command #834's own issue body names) found two more real aarch64 launch sites beyond #834's own six named scripts: - docker/qemu-aarch64/run-arm64-boot.sh sits in a directory sibling to docker/qemu/, unreachable by the #826/R181 ratchet's own walk rooted at docker/qemu regardless of how its detection predicates are written. Wired the same way as docker/qemu/run-aarch64-test.sh's own Docker-wrapped launch (lock held around the host-side `docker run`). - run.sh's native (non-Parallels, non-VMware) arm64 launch. The Parallels and VMware code paths boot a VM, not a native QEMU process, and exit before reaching this launch line, so they are outside the lock's scope by construction. run.sh picks arm64 or x86_64 at runtime, so the lock call and PID tracking are guarded by the arm64 branch only. Co-Authored-By: Claude Fable 5.1 --- docker/qemu-aarch64/run-arm64-boot.sh | 17 +++++++++++++++++ run.sh | 27 +++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/docker/qemu-aarch64/run-arm64-boot.sh b/docker/qemu-aarch64/run-arm64-boot.sh index 7e5104c30..d121e6120 100755 --- a/docker/qemu-aarch64/run-arm64-boot.sh +++ b/docker/qemu-aarch64/run-arm64-boot.sh @@ -7,6 +7,17 @@ set -e SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +# #826/#834/R181: this script's qemu-system-aarch64 boot runs behind the +# host-wide lock in docker/qemu/lib/qemu-host-lock.sh. This script is +# Docker-wrapped -- see docker/qemu/run-aarch64-test.sh's identical comment +# for why the host-side lock still applies (the `docker run` CLI blocks on +# the host with the qemu-system-aarch64 token in its own argv). #834 +# extends this lock's coverage from docker/qemu/*.sh (its original +# #826/R181 scope, which this sibling directory falls outside of) to the +# rest of the tree that launches qemu-system-aarch64. +# shellcheck source=../qemu/lib/qemu-host-lock.sh +source "$SCRIPT_DIR/../qemu/lib/qemu-host-lock.sh" + # Find the ARM64 kernel binary KERNEL_BIN="$BREENIX_ROOT/target/aarch64-breenix-kernel/release/kernel-aarch64" if [ ! -f "$KERNEL_BIN" ]; then @@ -32,6 +43,7 @@ docker build -q -t breenix-qemu-aarch64 "$SCRIPT_DIR" > /dev/null # - 1 CPU # - PL011 UART for serial output # - No graphics +qemu_host_lock_acquire docker run --rm \ -v "$KERNEL_BIN:/breenix/kernel.elf:ro" \ -v "$OUTPUT_DIR:/output" \ @@ -48,6 +60,11 @@ docker run --rm \ -no-reboot \ & QEMU_PID=$! +# F2: registers the docker run client with the lock's own EXIT trap (see +# docker/qemu/lib/qemu-host-lock.sh) so a SIGTERM/SIGINT delivered to just +# this process during the poll below still stops the container instead of +# orphaning it with the lock free. +qemu_host_lock_track_pid "$QEMU_PID" # Wait for output or timeout echo "Waiting for kernel output (30s timeout)..." diff --git a/run.sh b/run.sh index b50ad8613..df7f9d178 100755 --- a/run.sh +++ b/run.sh @@ -33,6 +33,19 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" BREENIX_ROOT="$SCRIPT_DIR" cd "$BREENIX_ROOT" +# #826/#834/R181: this script's native (non-Parallels, non-VMware) +# qemu-system-aarch64 boot runs behind the host-wide lock in +# docker/qemu/lib/qemu-host-lock.sh -- sourced unconditionally since arch is +# runtime-selected, but only the arm64 leg below actually calls +# qemu_host_lock_acquire (the lock serializes qemu-system-aarch64 boots +# specifically; the x86_64 leg's qemu-system-x86_64 is outside its scope). +# The Parallels and VMware code paths above this point boot a VM, not a +# native qemu-system-aarch64 process, and exit before reaching here. #834 +# extends this lock's coverage from docker/qemu/*.sh (its original +# #826/R181 scope) to run.sh as well. +# shellcheck source=docker/qemu/lib/qemu-host-lock.sh +source "$BREENIX_ROOT/docker/qemu/lib/qemu-host-lock.sh" + # Defaults: ARM64 with graphics ARCH="arm64" HEADLESS=false @@ -1062,6 +1075,13 @@ if [ "$ARCH" = "arm64" ]; then # MMIO addresses in reverse command-line order. The kernel discovers devices # by scanning MMIO addresses low-to-high, so the LAST device here gets the # lowest address and becomes device 0 (the system/root disk). + # + # #834: this is an interactive display session (Ctrl+C stops QEMU below), + # but it still takes the host-wide lock -- not exempt just because a + # human, not a gate, is driving it. qemu_host_lock_acquire prints its own + # one-line notice (host count, then a wait message if contended) before + # blocking. + qemu_host_lock_acquire qemu-system-aarch64 \ -M virt,gic-version=3 -cpu max -smp 4 \ -m 512M \ @@ -1117,6 +1137,13 @@ else fi QEMU_PID=$! +if [ "$ARCH" = "arm64" ]; then + # F2: registers QEMU with the lock's own EXIT trap (see + # docker/qemu/lib/qemu-host-lock.sh) so a SIGTERM/SIGINT delivered to + # just this script's own PID still kills QEMU instead of orphaning it + # with the lock free. + qemu_host_lock_track_pid "$QEMU_PID" +fi echo "Paste: echo 'code' | ./scripts/paste.sh" echo "Monitor: tcp://127.0.0.1:4444" From c034dc3c4c00ecf24e18b7e59c3d84416fcc9805 Mon Sep 17 00:00:00 2001 From: Ryan Breen Date: Sat, 5 Sep 2026 19:24:54 -0400 Subject: [PATCH 3/6] gates(834): widen the qemu-host-lock structural ratchet's census tests/qemu_host_lock_structure.rs walked docker/qemu only (#826/R181's own scope). Widens the walk to docker (recursive, reaching the docker/qemu-aarch64 sibling directory a docker/qemu-rooted walk cannot see) plus scripts (recursive) plus run.sh as a single named file. The anti-vacuity floor moves from 20 to 28, matching the exact count this branch's own two prior commits wired (20 pre-existing + 8 new). Adds a second mutation leg to qemu_host_lock_predicates_are_not_vacuous proving the widened scripts/ scope specifically: strips the real qemu_host_lock_acquire call from scripts/run-arm64-boot-test.sh's in-memory text (parallel to the pre-existing docker/qemu/ leg) and asserts the source line survives. Adds a reach check confirming the widened census lists docker/qemu-aarch64/run-arm64-boot.sh and run.sh by name. The detection predicate's own logic is unchanged; it already covers scripts/test_tracing_via_gdb.sh's indirect "$QEMU_BIN" launch via the QEMU_BIN=qemu-system-aarch64 assignment line that feeds it. Co-Authored-By: Claude Fable 5.1 --- tests/qemu_host_lock_structure.rs | 185 +++++++++++++++++++++++------- 1 file changed, 141 insertions(+), 44 deletions(-) diff --git a/tests/qemu_host_lock_structure.rs b/tests/qemu_host_lock_structure.rs index ad24e2535..448e08a84 100644 --- a/tests/qemu_host_lock_structure.rs +++ b/tests/qemu_host_lock_structure.rs @@ -1,29 +1,51 @@ -//! #826/R181 host-wide aarch64 QEMU lock structural ratchet. +//! #826/#834/R181 host-wide aarch64 QEMU lock structural ratchet. //! //! R181's own measurement: 4-6 concurrent qemu-system-aarch64 processes on //! this Mac ran the guest clock at 37-53% of wall-clock, which then falsely //! reds the strict gate's ~18s poll ceiling (#826) even on a healthy guest. //! `docker/qemu/lib/qemu-host-lock.sh` is the fix: a shared helper each -//! aarch64-launching gate script sources and routes its -//! `qemu-system-aarch64` invocation(s) through, so "at most one aarch64 -//! QEMU boot alive on this host at a time" holds mechanically. +//! aarch64-launching script sources and routes its `qemu-system-aarch64` +//! invocation(s) through, so "at most one aarch64 QEMU boot alive on this +//! host at a time" holds mechanically. //! -//! This file's one property: each `.sh` script under `docker/qemu/` -//! (recursive -- stays inside that tree, matching R181's own -//! scope; a `scripts/`-directory gap this scan also found is disclosed in -//! `docs/planning/green-program/gates/HOST-QEMU-LOCK-2026-09-05.md`, not -//! fixed here) that contains a real `qemu-system-aarch64` LAUNCH line -- -//! not merely a mention of the token, which the lock helper's own -//! `pgrep -f 'qemu-system-aarch64'` line and this file's own doc comments -//! both are -- also sources the helper and calls -//! `qemu_host_lock_acquire` somewhere in its text. +//! This file's one property: each `.sh` script under `docker/` (recursive) +//! or `scripts/` (recursive), plus `run.sh` itself, that contains a real +//! `qemu-system-aarch64` LAUNCH line -- not merely a mention of the token, +//! which the lock helper's own `pgrep -f 'qemu-system-aarch64'` line and +//! this file's own doc comments both are -- also sources the helper and +//! calls `qemu_host_lock_acquire` somewhere in its text. +//! +//! #826/R181's own ratchet scoped this census to `docker/qemu/*.sh` only, +//! by its own wording ("a shared helper ... sourced by each aarch64 gate +//! script"). #834 found six more real launchers under `scripts/`, 0 of them +//! wired, plus a seventh the original scan could not reach at all: +//! `docker/qemu-aarch64/run-arm64-boot.sh` sits in a directory sibling to +//! `docker/qemu/`, so `shell_scripts_below("docker/qemu")` (a recursive walk +//! *rooted* at that one directory) cannot see it regardless of how the +//! `launches`/`sources`/`acquires` predicates below are written -- widening +//! the walk's root to `docker` closes that reach gap the same motion that +//! adds `scripts/` closes the `scripts/`-directory gap. `run.sh`'s own +//! native (non-Parallels, non-VMware) aarch64 launch is the third and last +//! site #834's own grep found outside the original scope; it is checked as +//! a single named file since it is not itself a directory root. +//! +//! Deliberately NOT covered by this file, disclosed rather than silently +//! excluded (see `docs/planning/green-program/gates/ +//! HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md` for the reasoning): the three +//! `.py` launchers under `scripts/`/`docker/qemu/` (no bash to `source`) and +//! `docker/qemu/run-aarch64-test.exp` (a Tcl/expect script, unreferenced by +//! any caller in this repo). `shell_scripts_below`'s `.sh`-extension filter +//! excludes each of these four by construction, so this ratchet's `>= 28` floor below +//! cannot regress by silently absorbing one of them into "covered." //! //! Census-shaped, not a closed file list (the #549/#551/#527-r1 lesson this //! campaign has learned three times already): `launches_qemu_aarch64` //! re-derives "this script starts a real qemu-system-aarch64 process" from -//! each script's own text on each run, so a new aarch64 gate script that -//! forgets to route through the lock is caught automatically, not only the -//! 20 this branch itself wired up. +//! each script's own text on each run, so a new aarch64 launcher anywhere +//! under `docker/`, `scripts/`, or `run.sh` that forgets to route through +//! the lock is caught automatically, not only the 28 total (20 from #826/ +//! R181 plus #834's 8: 6 under `scripts/`, `docker/qemu-aarch64/ +//! run-arm64-boot.sh`, and `run.sh`) wired as of this file's own last edit. use std::fs; use std::path::{Path, PathBuf}; @@ -64,17 +86,31 @@ fn shell_scripts_below(relative: &str) -> Vec<(String, String)> { scripts } +/// The full launch-site census this ratchet polices: each `.sh` file under +/// `docker/` (recursive -- reaches both `docker/qemu/` and the sibling +/// `docker/qemu-aarch64/`) and `scripts/` (recursive), plus `run.sh` itself +/// checked as a single named file since it is not a directory. +fn all_shell_scripts() -> Vec<(String, String)> { + let mut scripts = shell_scripts_below("docker"); + scripts.extend(shell_scripts_below("scripts")); + scripts.push(("run.sh".to_string(), repo_text("run.sh"))); + scripts.sort_by(|left, right| left.0.cmp(&right.0)); + scripts +} + /// A real `qemu-system-aarch64` launch: a non-comment line that, once any /// trailing `\` line-continuation and the whitespace around it are /// stripped, ends with the bare token. Each real invocation in this tree /// (`timeout N qemu-system-aarch64 \`, `nice -n 19 qemu-system-aarch64 \`, -/// a bare `qemu-system-aarch64 \`, or the `qemu-system-aarch64 \` line -/// that follows a `docker run ... \` chain) is shaped exactly this -/// way. This is deliberately narrower than "the file contains the -/// substring": the lock helper's own `pgrep -f 'qemu-system-aarch64'` -/// line contains the token too but is a *search*, not a launch, and does -/// not end with a continuation -- the predicate below excludes it without -/// needing a path-based exemption. +/// a bare `qemu-system-aarch64 \`, the `qemu-system-aarch64 \` line that +/// follows a `docker run ... \` chain, or a plain +/// `QEMU_BIN=qemu-system-aarch64` assignment feeding an indirect `"$QEMU_BIN" +/// \` invocation further down the same file) is shaped exactly this way. +/// This is deliberately narrower than "the file contains the substring": +/// the lock helper's own `pgrep -f 'qemu-system-aarch64'` line contains the +/// token too but is a *search*, not a launch, and does not end with a +/// continuation -- the predicate below excludes it without needing a +/// path-based exemption. fn is_qemu_aarch64_launch_line(line: &str) -> bool { let trimmed_start = line.trim_start(); if trimmed_start.starts_with('#') { @@ -94,10 +130,10 @@ fn sources_qemu_host_lock(script: &str) -> bool { } /// A real call: a non-comment line whose trimmed text is exactly the bare -/// function name (each call site this branch added is ` qemu_host_lock_acquire` -/// alone on its line, at some indentation). This excludes the helper's own -/// `qemu_host_lock_acquire() {` definition line and any comment mentioning -/// the function by name. +/// function name (each call site this campaign added is +/// ` qemu_host_lock_acquire` alone on its line, at some indentation). +/// This excludes the helper's own `qemu_host_lock_acquire() {` definition +/// line and any comment mentioning the function by name. fn calls_qemu_host_lock_acquire(script: &str) -> bool { script.lines().any(|line| { let trimmed = line.trim(); @@ -107,20 +143,22 @@ fn calls_qemu_host_lock_acquire(script: &str) -> bool { #[test] fn every_aarch64_qemu_launch_script_sources_and_acquires_the_host_lock() { - let scripts = shell_scripts_below("docker/qemu"); + let scripts = all_shell_scripts(); let launching: Vec<&(String, String)> = scripts .iter() .filter(|(_, text)| launches_qemu_aarch64(text)) .collect(); - // Anti-vacuity floor: measured at 20 on this branch (the census that - // found them is reproduced in the ratchet's own sibling test below). - // Not a closed list -- a future aarch64 gate script only needs to + // Anti-vacuity floor: measured at 28 on this branch (20 from #826/R181's + // original docker/qemu/ census, plus #834's 8: six under scripts/, the + // docker/qemu-aarch64/ sibling-directory script, and run.sh). The + // census this counts is reproduced in the ratchet's own sibling test + // below. Not a closed list -- a future aarch64 launcher only needs to // raise this count, not edit it down. assert!( - launching.len() >= 20, - "only {} script(s) under docker/qemu/ launch qemu-system-aarch64; expected at least 20", + launching.len() >= 28, + "only {} script(s) under docker/, scripts/, or run.sh launch qemu-system-aarch64; expected at least 28", launching.len() ); @@ -135,7 +173,7 @@ fn every_aarch64_qemu_launch_script_sources_and_acquires_the_host_lock() { (true, false) => "calls qemu_host_lock_acquire", (true, true) => unreachable!(), }; - violations.push(format!("{path}: launches qemu-system-aarch64 but does not {missing} (#826/R181)")); + violations.push(format!("{path}: launches qemu-system-aarch64 but does not {missing} (#826/#834/R181)")); } } assert!( @@ -148,16 +186,20 @@ fn every_aarch64_qemu_launch_script_sources_and_acquires_the_host_lock() { /// ANTI-VACUITY: the launch/source/acquire predicates must fire on the /// real shapes they claim to, in both directions, and the whole-suite rule /// above must actually redden on a real script with the lock call removed -/// -- not just on a synthetic string. +/// -- not just on a synthetic string. Covers both halves of #834's widened +/// census: a `docker/qemu/` script (the original #826/R181 scope) and a +/// `scripts/` script (the gap #834 closes), so the widening is proven, not +/// merely asserted by the doc comment above. #[test] fn qemu_host_lock_predicates_are_not_vacuous() { - // Positive: each real continuation shape this branch used is detected. + // Positive: each real continuation shape this campaign used is detected. for line in [ " timeout 20 qemu-system-aarch64 \\", "timeout \"$BOOT_SECONDS\" qemu-system-aarch64 \\", " nice -n 19 qemu-system-aarch64 \\", "qemu-system-aarch64 \\", " qemu-system-aarch64 \\", + " QEMU_BIN=qemu-system-aarch64", ] { assert!( is_qemu_aarch64_launch_line(line), @@ -184,9 +226,10 @@ fn qemu_host_lock_predicates_are_not_vacuous() { for a call to it" ); - // ANTI-VACUITY mutation: strip the real qemu_host_lock_acquire call this - // branch added to run-aarch64-boot-test-strict.sh (the gate #826's own - // health battery ran) and confirm the whole-suite rule reddens by name. + // ANTI-VACUITY mutation, leg 1 (docker/qemu/, #826/R181's original + // scope): strip the real qemu_host_lock_acquire call from + // run-aarch64-boot-test-strict.sh (the gate #826's own health battery + // ran) and confirm the whole-suite rule reddens by name. let real_strict = repo_text("docker/qemu/run-aarch64-boot-test-strict.sh"); assert!( launches_qemu_aarch64(&real_strict), @@ -201,19 +244,73 @@ fn qemu_host_lock_predicates_are_not_vacuous() { real_strict.contains(acquire_line), "the reconstructed acquire line must match the real file, or this mutation proves nothing" ); - let mutated = real_strict.replacen(acquire_line, "", 1); - assert_ne!(mutated, real_strict, "mutation must apply"); + let mutated_strict = real_strict.replacen(acquire_line, "", 1); + assert_ne!(mutated_strict, real_strict, "mutation must apply"); + assert!( + sources_qemu_host_lock(&mutated_strict), + "the mutation must leave the source line intact -- this proves the acquire check \ + specifically, not the source check" + ); + assert!( + !calls_qemu_host_lock_acquire(&mutated_strict), + "reddening: the mutated text must no longer show an acquire call" + ); + assert!( + launches_qemu_aarch64(&mutated_strict), + "the mutation must not have touched the launch line itself" + ); + + // ANTI-VACUITY mutation, leg 2 (scripts/, #834's widened scope): the + // same proof against a real script under scripts/ -- a bypass here is + // exactly the shape #834 disclosed (six scripts/ launchers, 0 wired) + // and the whole-suite rule above must catch it the same way it catches + // a docker/qemu/ bypass, not merely by extending the doc comment. + let real_boot_test = repo_text("scripts/run-arm64-boot-test.sh"); + assert!( + launches_qemu_aarch64(&real_boot_test), + "sanity: scripts/run-arm64-boot-test.sh must still launch qemu-system-aarch64" + ); + assert!( + sources_qemu_host_lock(&real_boot_test) && calls_qemu_host_lock_acquire(&real_boot_test), + "sanity: scripts/run-arm64-boot-test.sh must be clean before mutation" + ); + let scripts_acquire_line = "qemu_host_lock_acquire\n"; assert!( - sources_qemu_host_lock(&mutated), + real_boot_test.contains(scripts_acquire_line), + "the reconstructed acquire line must match the real scripts/ file, or this mutation \ + proves nothing" + ); + let mutated_boot_test = real_boot_test.replacen(scripts_acquire_line, "", 1); + assert_ne!(mutated_boot_test, real_boot_test, "mutation must apply"); + assert!( + sources_qemu_host_lock(&mutated_boot_test), "the mutation must leave the source line intact -- this proves the acquire check \ specifically, not the source check" ); assert!( - !calls_qemu_host_lock_acquire(&mutated), + !calls_qemu_host_lock_acquire(&mutated_boot_test), "reddening: the mutated text must no longer show an acquire call" ); assert!( - launches_qemu_aarch64(&mutated), + launches_qemu_aarch64(&mutated_boot_test), "the mutation must not have touched the launch line itself" ); + + // ANTI-VACUITY reach check: docker/qemu-aarch64/run-arm64-boot.sh sits + // in a directory sibling to docker/qemu/, unreachable by a walk rooted + // at "docker/qemu" no matter how the predicates above are written. + // Confirms the root widened to "docker" actually reaches it, rather + // than the file merely existing on disk unchecked. + let census = all_shell_scripts(); + assert!( + census + .iter() + .any(|(path, _)| path == "docker/qemu-aarch64/run-arm64-boot.sh"), + "the docker-rooted walk must reach the docker/qemu-aarch64/ sibling directory, \ + not only docker/qemu/ -- a docker/qemu-only root would silently exempt it" + ); + assert!( + census.iter().any(|(path, _)| path == "run.sh"), + "the census must include run.sh itself, not only files under docker/ or scripts/" + ); } From 834e641e910da7eaf1e5f54c2ef25973734f24cb Mon Sep 17 00:00:00 2001 From: Ryan Breen Date: Sat, 5 Sep 2026 19:25:07 -0400 Subject: [PATCH 4/6] docs(834): record the qemu-host-lock scripts/ widening round Round doc under docs/planning/green-program/gates/ recording what #834 disclosed, the fresh census that found two more launch sites beyond its six named scripts, the 8 scripts wired, why three lost their exec, the ratchet's widened census and its two mutation records (in-suite and a live on-disk mutation against the tracked scripts/run-arm64-boot-test.sh, reverted after observing the reddened suite), boot proofs (a quick-mode run of the cheapest newly-wired script and one strict-gate run), the 33-suite/593-test structural result, and what remains disclosed but not fixed (three .py launchers, one unreferenced .exp launcher, two pre-existing kill-by-pattern hazards). Co-Authored-By: Claude Fable 5.1 --- .../HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md | 322 ++++++++++++++++++ 1 file changed, 322 insertions(+) create mode 100644 docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md diff --git a/docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md b/docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md new file mode 100644 index 000000000..dea99b00f --- /dev/null +++ b/docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md @@ -0,0 +1,322 @@ +# Widening the host-wide aarch64 QEMU lock past docker/qemu/ (#834/R181) + +## Why + +#826/R181's own fix (`docker/qemu/lib/qemu-host-lock.sh`, landed via #835) was +scoped, by its own wording ("a shared helper ... sourced by each aarch64 gate +script"), to `docker/qemu/*.sh`, and its own structural ratchet +(`tests/qemu_host_lock_structure.rs`) policed only that one tree. #834 +disclosed the gap that fix left: a `grep -rl qemu-system-aarch64 scripts/ +docker/ run.sh` census found six more `.sh` scripts under `scripts/` with a +real, unwired `qemu-system-aarch64` launch line, 0 of them routed through the +lock. This branch closes that gap. + +A fresh census run at this branch's own start (the same `grep -rl +qemu-system-aarch64 scripts/ docker/ run.sh` command #834's own issue body +names) found two more real launch sites #834's own disclosure did not name: + +- `docker/qemu-aarch64/run-arm64-boot.sh` -- a Docker-wrapped launcher in a + directory *sibling* to `docker/qemu/`. #826/R181's ratchet walked + `docker/qemu` as its recursive root; a walk rooted at one directory does + not descend into a sibling directory regardless of how its + `launches`/`sources`/`acquires` predicates are written -- only widening the + root itself reaches it. This script was invisible to the original ratchet, + not merely unwired by it. +- `run.sh` itself -- the project's own primary interactive dev-loop launcher, + named in the issue's suggested-shape section as a candidate but not counted + among its six named scripts. + +Total closed by this branch: 8 (6 named by #834, plus these 2 the fresh +census found), for a new total of 28 wired launch sites (20 from #826/R181 +plus these 8). + +## What changed: 8 scripts wired to the lock + +Each script below now `source`s `docker/qemu/lib/qemu-host-lock.sh` and calls +`qemu_host_lock_acquire` before its `qemu-system-aarch64` launch, with +`qemu_host_lock_track_pid` registering the launched PID immediately after +capture (per #826/R181's own fix-round: the tracked PID lets the lock's +chained `EXIT` trap kill the child even on a script with no cleanup trap of +its own, or on a SIGTERM/SIGINT delivered to just the script's own PID). + +| Script | Shape | Change beyond wiring | +|---|---|---| +| `scripts/run-arm64-keyboard-test.sh` | bare `qemu-system-aarch64 \`, backgrounded, existing `trap cleanup EXIT` | wiring only -- chains onto the existing trap | +| `scripts/run-arm64-boot-test.sh` | bare `qemu-system-aarch64 \`, backgrounded, no existing trap | wiring only -- the lock installs its own `EXIT` trap | +| `scripts/run-arm64-qemu.sh` | `exec qemu-system-aarch64 \` (interactive, Ctrl-A X exits) | `exec` dropped for a plain foreground launch (see below) | +| `scripts/run-arm64-graphics.sh` | `exec qemu-system-aarch64 \` (interactive) | same `exec`-drop | +| `scripts/run-aarch64-userspace.sh` | `exec qemu-system-aarch64 \` (interactive) | same `exec`-drop | +| `scripts/test_tracing_via_gdb.sh` | `"$QEMU_BIN" \` (aarch64 leg only; the x86_64 leg launches `qemu-system-x86_64` and is untouched) | lock call placed inside the `if [ "$ARCH" = "aarch64" ]` branch only | +| `docker/qemu-aarch64/run-arm64-boot.sh` | `timeout 30 qemu-system-aarch64 \`, Docker-wrapped, backgrounded | wiring only -- same Docker-wrapped shape as `docker/qemu/run-aarch64-test.sh` | +| `run.sh` | bare `qemu-system-aarch64 \` (arm64 leg only; the x86_64 leg launches `qemu-system-x86_64`), backgrounded, interactive (`wait $QEMU_PID`, Ctrl+C to stop) | lock call and `track_pid` guarded by `[ "$ARCH" = "arm64" ]` | + +### Why three scripts lost their `exec` + +`scripts/run-arm64-qemu.sh`, `run-arm64-graphics.sh`, and +`run-aarch64-userspace.sh` each ended in `exec qemu-system-aarch64 \ ...` +-- replacing the shell's own process image with QEMU so the script's PID +becomes QEMU's PID (a bash-scripting convenience for an interactive session +with no further work to do after boot). This is incompatible with the +`mkdir`-based lock's release mechanism: `qemu_host_lock_acquire` installs an +`EXIT` trap that releases the lock when the *shell process* exits, and `exec` +replaces that process outright -- there is no bash process left to run the +trap when QEMU itself later exits, so the lock directory would be left +behind for the next acquirer's stale-PID reclaim path on each interactive +run, not only on a crash. + +Each of the three now runs QEMU as a plain foreground command (no `exec`, +still no `&`) after `qemu_host_lock_acquire`. This is an interactive display +session precisely the same as before -- Ctrl-A X still exits QEMU exactly as +it did with `exec`, the same terminal remains attached to the same monitor, +and no output or behavior difference is visible to whoever runs the script. +The only difference is internal: on QEMU's exit, control returns to the +shell for one instruction (falling off the end of the script) instead of the +process image being replaced, so the lock's `EXIT` trap can run and release +it. `scripts/run-arm64-qemu.sh` was checked by hand for a trailing action +after its old `exec` line that this change might now skip -- there is no +such action; the `exec` was already the file's last line. + +### `test_tracing_via_gdb.sh` and `run.sh`: arch-conditional locking + +Both scripts pick `qemu-system-aarch64` or `qemu-system-x86_64` at runtime +(`--arch`/host `uname -m` for the former, `--x86` for the latter). The +host-wide lock in `docker/qemu/lib/qemu-host-lock.sh` serializes +`qemu-system-aarch64` boots specifically (its own `qemu_host_lock_count()` +counts aarch64 processes only); routing an x86_64 launch through it would +serialize x86_64 boots against aarch64 boots for no reason the lock's own +design intends. Both scripts `source` the helper unconditionally (harmless -- +sourcing has no runtime cost) but only call `qemu_host_lock_acquire` and +`qemu_host_lock_track_pid` on the aarch64 leg; the x86_64 leg is untouched in +both files. + +## What is disclosed, not fixed, in this branch + +Consistent with #826/R181's own precedent of disclosing rather than silently +absorbing a gap it did not close (`docs/planning/green-program/gates/ +HOST-QEMU-LOCK-2026-09-05.md`'s own "scripts/ is not covered" section is what +produced #834 in the first place): + +- **Three `.py` launchers are not wired**: `scripts/debug_smp_deadlock.py`, + `scripts/debug_elr0_crash.py`, and `docker/qemu/run-aarch64-test-runner.py` + each build a `subprocess` argument list containing `qemu-system-aarch64`. + `docker/qemu/lib/qemu-host-lock.sh` is a bash library; there is no `source` + equivalent from Python, and porting its `mkdir`/stale-PID protocol to + Python is a materially different task from wiring an existing bash helper + into a bash caller. These three are one-off local debugging tools invoked by + a human running one GDB session at a time, not scripts a gate or CI runs + concurrently with anything else -- the same category + `docs/planning/green-program/gates/GATE-TMP-BASEDIR-2026-09-05.md`'s own + review already placed two of these three in ("one-off local debugging + sessions, not the shared container") for an unrelated `/tmp` collision + finding. Not fixed here; a Python-native lock implementation, if wanted, + is its own follow-up. +- **`docker/qemu/run-aarch64-test.exp` is not wired**: a Tcl/expect script + that `spawn`s `qemu-system-aarch64` directly. `grep -rln + "run-aarch64-test\.exp" .` across the whole repository returns only the + file itself -- 0 callers anywhere in this tree. Same bash/Tcl boundary as + the `.py` scripts above, compounded by being unreferenced; not fixed here. +- **Two pre-existing kill-by-pattern hazards are unchanged**: + `scripts/run-arm64-boot-test.sh`'s `pkill -9 -f + "qemu-system-aarch64.*kernel-aarch64"` and `docker/qemu-aarch64/ + run-arm64-boot.sh`'s `docker kill $(docker ps -q --filter + ancestor=breenix-qemu-aarch64)` each terminate by pattern/image match + rather than a specific PID or container ID this script itself launched -- + the same shape #834's own issue body named for the first of these two + ("a second, distinct issue in its own right... same shape as the + already-filed #829") and asked to be looked at "regardless of which fix + this issue gets," not as a precondition of this fix. Wiring the host-wide + lock does not touch either kill line; both are left exactly as found. +- **`shell_scripts_below`'s `.sh`-extension filter is what keeps the three + `.py` files and the one `.exp` file out of this branch's own `>= 28` + ratchet floor**, not a path-based exemption list -- so a future `.sh` + launcher anywhere under `docker/`, `scripts/`, or `run.sh` is still caught + automatically by the widened census below, and the floor cannot be + satisfied by miscounting one of these four non-`.sh` files as "covered." + +## The ratchet: `tests/qemu_host_lock_structure.rs`, widened + +`shell_scripts_below("docker/qemu")` (the original #826/R181 walk root) +becomes two calls -- `shell_scripts_below("docker")` and +`shell_scripts_below("scripts")` -- plus `run.sh` added as a single named +file (it is not itself a directory, so the recursive walk helper does not +apply to it). Widening the root from `docker/qemu` to `docker` is what +reaches `docker/qemu-aarch64/run-arm64-boot.sh`: a walk rooted at one +directory does not descend into a sibling directory regardless of how the +`launches`/`sources`/`acquires` predicates are written, so no predicate +change alone could have closed that reach gap -- only widening the root +could. The anti-vacuity floor moves from 20 to 28, matching the exact count +this branch wired (20 pre-existing + 8 new). + +The `launches_qemu_aarch64`/`is_qemu_aarch64_launch_line` predicate itself is +unchanged in its detection logic; it incidentally already covers +`scripts/test_tracing_via_gdb.sh`'s indirect `"$QEMU_BIN" \` invocation shape +via the `QEMU_BIN=qemu-system-aarch64` assignment line that feeds it (that +assignment line itself ends with the bare token, which is exactly what the +predicate's "ends with `qemu-system-aarch64` after stripping a trailing `\`" +rule matches) -- no special-casing was needed for that file's indirection. + +### Mutation records + +Two independent proofs of the widened census, both against real files in the +working tree (not synthetic strings): + +**In-suite** (`qemu_host_lock_predicates_are_not_vacuous`'s second mutation +leg, added by this branch): the real `qemu_host_lock_acquire` call is +stripped from `scripts/run-arm64-boot-test.sh`'s in-memory text (parallel to +the pre-existing first leg's proof against `docker/qemu/ +run-aarch64-boot-test-strict.sh`), and the source line is asserted to remain +-- proving the acquire check specifically, not the source check. A further +reach-check in the same test asserts the widened census actually lists +`docker/qemu-aarch64/run-arm64-boot.sh` and `run.sh` by name. + +**On-disk, live**: the real `qemu_host_lock_acquire` line was removed from +the tracked `scripts/run-arm64-boot-test.sh` on disk (not a scratch copy), +the whole-suite test re-run, then the file was restored from a +pre-mutation backup and re-verified byte-identical (`grep -c +"^qemu_host_lock_acquire$" scripts/run-arm64-boot-test.sh` read `1` again +after restore). + +``` +cmd: scripts/run-arm64-boot-test.sh's `qemu_host_lock_acquire` line (exact + text "qemu_host_lock_acquire\n", zero-indented) removed from the real + tracked file, then: + cargo test --test qemu_host_lock_structure + every_aarch64_qemu_launch_script_sources_and_acquires_the_host_lock +exit: 101 (test binary FAILED; "test result: FAILED. 0 passed; 1 failed") +assertion: "aarch64 QEMU launch(es) bypass the host-wide lock: + scripts/run-arm64-boot-test.sh: launches qemu-system-aarch64 but + does not calls qemu_host_lock_acquire (#826/#834/R181)" +``` + +After restore, `cargo test --test qemu_host_lock_structure` returned to +`test result: ok. 2 passed; 0 failed`. + +## Boot proofs (2026-09-05) + +Kernel built at this branch's own head, in a fresh worktree needing the same +one-time setup #835's own doc recorded (`rust-fork` symlink, prebuilt +`userspace/programs/aarch64/*.elf` files, and `target/ext2-aarch64.img` -- +these are not git-tracked and this branch's diff does not touch them): + +``` +touch kernel/src/main_aarch64.rs +cargo build --release --target aarch64-breenix-kernel.json \ + -Z build-std=core,alloc -Z build-std-features=compiler-builtins-mem \ + -p kernel --bin kernel-aarch64 +-> Finished `release` profile [optimized] target(s) + +scripts/check-kernel-no-neon.sh target/aarch64-breenix-kernel/release/kernel-aarch64 +-> PASS: 0 FP/SIMD load/store instructions in kernel .text (allowlisted & suppressed: 0) +``` + +Host aarch64 QEMU count was 0 before each run below (`pgrep -x +qemu-system-aarch64 | wc -l`), and 0 after each run. + +### (a) `scripts/run-arm64-boot-test.sh quick` -- the cheapest newly-wired script + +``` +./scripts/run-arm64-boot-test.sh quick +``` + +Output (verbatim, the lock's own notice line included): + +``` +[2/4] Starting QEMU... +QEMU HOST LOCK: host aarch64 QEMU count before acquire: 0 +[3/4] Waiting for kernel output (30s timeout)... +... +[4/4] Quick Boot Check: +======================================== +PASS: 'Hello from ARM64' found +``` + +### (b) One strict-gate run, unaffected by the widening + +Rebuilt with `--features boot_tests` (the strict gate's own requirement), +plus a fresh `target/ext2-aarch64.img` (`scripts/create_ext2_disk.sh --arch +aarch64`, default 256MB size -- not git-tracked, needed once per fresh +worktree same as the ELFs above): + +``` +BREENIX_GATE_TMP= ./docker/qemu/run-aarch64-boot-test-strict.sh 1 +``` + +``` +QEMU HOST LOCK: host aarch64 QEMU count before acquire: 0 +qemu-system-aarch64: terminating on signal 15 from pid 91619 () + [OK] Boot 1: SUCCESS + [GATE_BOOT_FACTS:boot=1:host_ms=1788650287847-1788650298996:qemu_at_start=0:load_at_start=5.05:qemu_at_end=1:load_at_end=5.66:qemu_cpu_s=20.24:guest_uptime_ms=10581:ended_by=scored_pass] +========================================= +PASS: 1/1 boots succeeded +========================================= +``` + +This confirms the widened ratchet (root moved from `docker/qemu` to +`docker`) did not disturb `docker/qemu/run-aarch64-boot-test-strict.sh`'s +own existing wiring from #826/R181 -- the strict gate is untouched by this +branch's diff and still passes. + +## Structural suites and claim-lint + +``` +cargo test --test for each of the 33 tests/*_structure.rs files +-> 33/33 green, 593 test cases total (summed from each suite's own "N passed" + line; qemu_host_lock_structure.rs contributed 2 of the 593) + +python3 scripts/claim-lint.py +-> "claim-lint: clean (9 file(s) checked, changed hunks vs b40fbee49aea)." + "claim-lint: 25 pre-existing finding(s) outside this branch's changed + hunks not reported (--whole-file shows them)." exit 0 +``` + +The no-argument run first found 3 findings, 3 of 3 in this branch's own new +comment text in `tests/qemu_host_lock_structure.rs` -- each an unquantified +absolute word (a negation-based one and a totality-based one in two +doc-comment paragraphs, and a third totality word opening a third paragraph) +-- the same "over-broad phrasing in a repeated comment block" shape #825's +and #826/R181's own docs record finding in their own changed hunks. Each was +reworded to bounded phrasing carrying the identical meaning (see the diff on +`tests/qemu_host_lock_structure.rs`'s doc comments), and the no-argument run +then reported 0 findings in this branch's changed hunks. + +``` +claim-lint: scripts/claim-lint.py -> exit 0 +claim-lint: scripts/claim-lint.py --files -> exit 0 +claim-lint: scripts/claim-lint.py --commit-msg -> exit 0 (one per commit) +``` + +## What is NOT claimed + +- **The three `.py` launchers and the one `.exp` launcher are unwired**, for + the reasons given above (no bash `source` mechanism, one-off local + debugging tools per existing precedent, and -- for the `.exp` file -- + 0 callers anywhere in this repository). If a developer runs one of these + four alongside a wired script, or two of the four together, the contention + #826/R181 measured is not prevented. +- **The two pre-existing kill-by-pattern hazards + (`scripts/run-arm64-boot-test.sh`'s `pkill -9 -f`, `docker/qemu-aarch64/ + run-arm64-boot.sh`'s `docker kill $(docker ps -q --filter ancestor=...)`) + are unchanged.** Both can still terminate a different, unrelated process + matching the same pattern/image; this branch's own issue disclosed the + first of these two explicitly as a separate concern, not a precondition + of the lock-wiring fix. +- **`run.sh --parallels` and `run.sh --vmware` are unaffected.** Both exit + before reaching the native `qemu-system-aarch64` launch line this branch + wires; they boot a VM via `prlctl`/VMware tooling, not a native QEMU + process, and are outside this lock's scope by construction (there is no + `qemu-system-aarch64` process on the host to serialize against in either + path). +- **This branch is not a soak.** The strict-gate run above is 1 boot, and + the quick-mode run is 1 invocation; neither is a many-iteration statistical + claim about contention frequency the way #826/R181's own 200-boot health + battery was for the original lock. +- **#827's gate-side instrumentation gap remains untouched**, as it was in + #826/R181's own branch -- this work adds lock coverage, not the per-boot + host-fact fields #827 asks for. +- **This branch adds no new x86-only script and touches no x86-only launch + line.** `scripts/test_tracing_via_gdb.sh`'s and `run.sh`'s own x86_64 legs + are read but not edited; both still launch `qemu-system-x86_64` + unconditionally, exactly as before this branch. + +claim-lint: scripts/claim-lint.py -> exit 0 +claim-lint: scripts/claim-lint.py --files docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md -> exit 0 From 92cf77889722b68dc69c8de7314dc69aebf4411b Mon Sep 17 00:00:00 2001 From: Ryan Breen Date: Sat, 5 Sep 2026 20:11:57 -0400 Subject: [PATCH 5/6] gates(834,fix-round): close F1-F4 from the qemu-host-lock scripts review F1 (blocking): the three interactive scripts (run-arm64-qemu.sh, run-arm64-graphics.sh, run-aarch64-userspace.sh) ran qemu-system-aarch64 as a plain foreground command after dropping `exec`. A direct-PID SIGTERM/SIGINT does not propagate to a foreground child, so it released the host lock while orphaning a running QEMU -- reproduced live against the real lock file, then fixed by backgrounding QEMU with an explicit 0<&0 (preserving the serial console's stdin) and qemu_host_lock_track_pid, matching the five already-correct wired scripts. Re-verified live with the real scripts/run-arm64-qemu.sh and a real kernel boot: a direct-PID SIGTERM now kills the tracked QEMU process and releases the lock. F2 (major): scripts/run-arm64-boot-test.sh's cleanup pkill ran before its own qemu_host_lock_acquire, so it could kill a different lock-holding script's in-progress boot; moved the acquire ahead of it. docker/qemu-aarch64/run-arm64-boot.sh's cleanup docker-killed by ancestor image instead of its own container; named the container per invocation and kill that name. A related, unfixed occurrence in docker/qemu/run-aarch64-interactive.sh (untouched by this branch) is reported on #829. F3 (major): parameterized OUTPUT_DIR (run-arm64-boot.sh) and SERIAL_OUTPUT (run-arm64-boot-test.sh) behind BREENIX_GATE_TMP (default /tmp), matching the #825 fix already applied to the sibling docker/qemu/run-aarch64-test.sh and run-aarch64-userspace.sh. F4 (minor): the original round's claim-lint receipt quoted an intermediate commit's file count; this round's own doc section re-quotes claim-lint's output at this round's own final state instead. Evidence: qemu_host_lock_structure.rs's 2 tests still pass; full 33-suite/ 593-case structural battery still green; live scripts/run-arm64-boot-test.sh quick and docker/qemu-aarch64/run-arm64-boot.sh both PASS with BREENIX_GATE_TMP pointed at a private scratch dir, 0 qemu-system-aarch64 processes and 0 breenix-arm64-boot-* containers left running after either. --- docker/qemu-aarch64/run-arm64-boot.sh | 29 ++- .../HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md | 190 +++++++++++++++--- scripts/run-aarch64-userspace.sh | 25 ++- scripts/run-arm64-boot-test.sh | 31 ++- scripts/run-arm64-graphics.sh | 25 ++- scripts/run-arm64-qemu.sh | 25 ++- 6 files changed, 274 insertions(+), 51 deletions(-) diff --git a/docker/qemu-aarch64/run-arm64-boot.sh b/docker/qemu-aarch64/run-arm64-boot.sh index d121e6120..e6051856f 100755 --- a/docker/qemu-aarch64/run-arm64-boot.sh +++ b/docker/qemu-aarch64/run-arm64-boot.sh @@ -18,6 +18,19 @@ BREENIX_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" # shellcheck source=../qemu/lib/qemu-host-lock.sh source "$SCRIPT_DIR/../qemu/lib/qemu-host-lock.sh" +# #834 fix-round F3 (2026-09-05): OUTPUT_DIR is a HOST path this script +# rm -rf's, mkdir's, and bind-mounts into the container, so it collides with +# any other concurrent invocation the same way #825 already reports for +# docker/qemu/run-aarch64-test.sh's OUTPUT_DIR (see that file's own #825 +# comment for the identical BREENIX_GATE_TMP shape). Defaulting to /tmp +# keeps a caller that leaves it unset byte-identical; a concurrent-lane +# launcher sets this to a per-worktree directory instead. +BREENIX_GATE_TMP="${BREENIX_GATE_TMP:-/tmp}" +case "$BREENIX_GATE_TMP" in + /*) ;; + *) echo "FAIL: BREENIX_GATE_TMP must be an absolute path, got: $BREENIX_GATE_TMP"; exit 1 ;; +esac + # Find the ARM64 kernel binary KERNEL_BIN="$BREENIX_ROOT/target/aarch64-breenix-kernel/release/kernel-aarch64" if [ ! -f "$KERNEL_BIN" ]; then @@ -30,7 +43,7 @@ echo "Running ARM64 boot test in Docker..." echo "Kernel: $KERNEL_BIN" # Create output directory -OUTPUT_DIR="/tmp/breenix_arm64_boot" +OUTPUT_DIR="$BREENIX_GATE_TMP/breenix_arm64_boot" rm -rf "$OUTPUT_DIR" mkdir -p "$OUTPUT_DIR" @@ -43,8 +56,17 @@ docker build -q -t breenix-qemu-aarch64 "$SCRIPT_DIR" > /dev/null # - 1 CPU # - PL011 UART for serial output # - No graphics +# #834 fix-round F2 (2026-09-05): CONTAINER_NAME is unique to this +# invocation (this script's own PID), so the cleanup `docker kill` below +# can target the exact container this run started instead of matching by +# ancestor image -- an ancestor-image filter can kill a DIFFERENT running +# container from that image, including one a concurrent invocation of this +# same script (or of another script sharing the image) legitimately owns. +CONTAINER_NAME="breenix-arm64-boot-$$" +docker rm -f "$CONTAINER_NAME" >/dev/null 2>&1 || true qemu_host_lock_acquire docker run --rm \ + --name "$CONTAINER_NAME" \ -v "$KERNEL_BIN:/breenix/kernel.elf:ro" \ -v "$OUTPUT_DIR:/output" \ breenix-qemu-aarch64 \ @@ -81,8 +103,9 @@ for i in $(seq 1 30); do sleep 1 done -# Cleanup -docker kill $(docker ps -q --filter ancestor=breenix-qemu-aarch64) 2>/dev/null || true +# Cleanup: targets this invocation's own container by name (see +# CONTAINER_NAME above), not the whole image's running containers. +docker kill "$CONTAINER_NAME" 2>/dev/null || true # Check results echo "" diff --git a/docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md b/docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md index dea99b00f..71108d462 100644 --- a/docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md +++ b/docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md @@ -43,7 +43,7 @@ its own, or on a SIGTERM/SIGINT delivered to just the script's own PID). |---|---|---| | `scripts/run-arm64-keyboard-test.sh` | bare `qemu-system-aarch64 \`, backgrounded, existing `trap cleanup EXIT` | wiring only -- chains onto the existing trap | | `scripts/run-arm64-boot-test.sh` | bare `qemu-system-aarch64 \`, backgrounded, no existing trap | wiring only -- the lock installs its own `EXIT` trap | -| `scripts/run-arm64-qemu.sh` | `exec qemu-system-aarch64 \` (interactive, Ctrl-A X exits) | `exec` dropped for a plain foreground launch (see below) | +| `scripts/run-arm64-qemu.sh` | `exec qemu-system-aarch64 \` (interactive, Ctrl-A X exits) | `exec` dropped for a backgrounded launch + `track_pid` (see below, revised by the 2026-09-05 fix round) | | `scripts/run-arm64-graphics.sh` | `exec qemu-system-aarch64 \` (interactive) | same `exec`-drop | | `scripts/run-aarch64-userspace.sh` | `exec qemu-system-aarch64 \` (interactive) | same `exec`-drop | | `scripts/test_tracing_via_gdb.sh` | `"$QEMU_BIN" \` (aarch64 leg only; the x86_64 leg launches `qemu-system-x86_64` and is untouched) | lock call placed inside the `if [ "$ARCH" = "aarch64" ]` branch only | @@ -64,17 +64,38 @@ trap when QEMU itself later exits, so the lock directory would be left behind for the next acquirer's stale-PID reclaim path on each interactive run, not only on a crash. -Each of the three now runs QEMU as a plain foreground command (no `exec`, -still no `&`) after `qemu_host_lock_acquire`. This is an interactive display -session precisely the same as before -- Ctrl-A X still exits QEMU exactly as -it did with `exec`, the same terminal remains attached to the same monitor, -and no output or behavior difference is visible to whoever runs the script. -The only difference is internal: on QEMU's exit, control returns to the -shell for one instruction (falling off the end of the script) instead of the -process image being replaced, so the lock's `EXIT` trap can run and release -it. `scripts/run-arm64-qemu.sh` was checked by hand for a trailing action -after its old `exec` line that this change might now skip -- there is no -such action; the `exec` was already the file's last line. +**Revised by the 2026-09-05 fix round (F1, blocking).** The branch's first +cut ran QEMU as a plain foreground command (no `exec`, still no `&`) after +`qemu_host_lock_acquire`, on the reasoning that a foreground child is +equivalent to the old `exec`'d process for signal purposes. That reasoning +was wrong: a `SIGTERM`/`SIGINT` delivered to just the script's own PID +during the interactive session -- e.g. `kill -TERM ` from +another terminal, distinct from the terminal-generated Ctrl-C the running +session itself swallows into the guest -- does not propagate to a +foreground child on its own. Reproduced live against the real +`qemu-host-lock.sh` with a stand-in `qemu-system-aarch64` on `PATH`: a +direct-PID `SIGTERM` fired the chained `EXIT` trap (releasing the lock) +while the foreground stand-in kept running, orphaned and untracked -- the +opposite of "no behavior difference," and worse than the pre-branch `exec` +shape for this exact signal, since `exec` at least made the script's PID +*be* QEMU's PID. + +Each of the three now backgrounds QEMU instead, registers the PID with +`qemu_host_lock_track_pid` (the same mechanism the other five wired +interactive/gate scripts already use), then `wait`s on it -- so the lock's +own `EXIT` trap kills QEMU before releasing the lock on the direct-PID-signal +path, exactly like those five. The backgrounded launch adds an explicit +`0<&0` redirect: bash redirects a backgrounded command's stdin from +`/dev/null` unless the command carries its own explicit stdin redirection, +and these sessions' `-serial mon:stdio` (or `-nographic`) consoles need the +script's own stdin attached for Ctrl-A X and any serial-console typing to +keep working. Reproduced live (piped stdin into the backgrounded shape with +`0<&0` reaches the stand-in unchanged) and re-verified the SIGTERM case +against the corrected shape: the stand-in is killed and the lock is released, +matching the five already-correct scripts. `scripts/run-arm64-qemu.sh` was +checked by hand for a trailing action after its old `exec` line that either +revision might skip -- there is no such action; the `exec` was already the +file's last line. ### `test_tracing_via_gdb.sh` and `run.sh`: arch-conditional locking @@ -115,17 +136,32 @@ produced #834 in the first place): "run-aarch64-test\.exp" .` across the whole repository returns only the file itself -- 0 callers anywhere in this tree. Same bash/Tcl boundary as the `.py` scripts above, compounded by being unreferenced; not fixed here. -- **Two pre-existing kill-by-pattern hazards are unchanged**: +- **Two pre-existing kill-by-pattern hazards, addressed by the 2026-09-05 + fix round (F2, major) -- narrowed, not eliminated as a mechanism**: `scripts/run-arm64-boot-test.sh`'s `pkill -9 -f - "qemu-system-aarch64.*kernel-aarch64"` and `docker/qemu-aarch64/ - run-arm64-boot.sh`'s `docker kill $(docker ps -q --filter - ancestor=breenix-qemu-aarch64)` each terminate by pattern/image match - rather than a specific PID or container ID this script itself launched -- - the same shape #834's own issue body named for the first of these two - ("a second, distinct issue in its own right... same shape as the - already-filed #829") and asked to be looked at "regardless of which fix - this issue gets," not as a precondition of this fix. Wiring the host-wide - lock does not touch either kill line; both are left exactly as found. + "qemu-system-aarch64.*kernel-aarch64"` ran *before* this script's own + `qemu_host_lock_acquire`, so it could hit a different, lock-holding + script's in-progress boot the moment this script started -- undermining + the "wired into the lock" property for exactly the file this branch + claimed to have closed the gap for. Fixed by moving + `qemu_host_lock_acquire` ahead of the pkill: by the time this script's own + cleanup runs, any lock-cooperating peer must already have released the + lock, so the pkill can no longer reach a peer's active, lock-protected + boot. `docker/qemu-aarch64/run-arm64-boot.sh`'s `docker kill $(docker ps + -q --filter ancestor=breenix-qemu-aarch64)` matched by image, not by the + container this invocation started; fixed by naming the container + (`--name breenix-arm64-boot-$$`) and killing that name specifically, + matching the fix shape #829 itself proposes ("capture this invocation's + own container id ... and docker kill that one id"). Neither fix makes the + underlying primitive PID/container-scoped in general -- a genuinely + unwired caller of `qemu-system-aarch64` (the `.py`/`.exp` scripts below, + or `docker/qemu/run-aarch64-interactive.sh`'s own *pre-acquire* `docker + kill $EXISTING` cleanup, found while reviewing this bullet and reported + on #829 rather than fixed here since that file is untouched by this + branch's diff) can still reach a lock-cooperating script's process from + outside the lock's own serialization. What is closed is the specific + claim this branch made about these two files: their own cleanup can no + longer defeat the mutual-exclusion property this branch wires them into. - **`shell_scripts_below`'s `.sh`-extension filter is what keeps the three `.py` files and the one `.exp` file out of this branch's own `>= 28` ratchet floor**, not a path-based exemption list -- so a future `.sh` @@ -293,13 +329,20 @@ claim-lint: scripts/claim-lint.py --commit-msg -> exit 0 (one per commit 0 callers anywhere in this repository). If a developer runs one of these four alongside a wired script, or two of the four together, the contention #826/R181 measured is not prevented. -- **The two pre-existing kill-by-pattern hazards - (`scripts/run-arm64-boot-test.sh`'s `pkill -9 -f`, `docker/qemu-aarch64/ - run-arm64-boot.sh`'s `docker kill $(docker ps -q --filter ancestor=...)`) - are unchanged.** Both can still terminate a different, unrelated process - matching the same pattern/image; this branch's own issue disclosed the - first of these two explicitly as a separate concern, not a precondition - of the lock-wiring fix. +- **The two pre-existing kill-by-pattern hazards in + `scripts/run-arm64-boot-test.sh` and `docker/qemu-aarch64/ + run-arm64-boot.sh` are narrowed by the 2026-09-05 fix round, not + eliminated as a mechanism.** `run-arm64-boot-test.sh`'s cleanup pkill now + runs after this script's own `qemu_host_lock_acquire`, so it cannot reach + a lock-cooperating peer's active boot; `run-arm64-boot.sh`'s cleanup + `docker kill` now targets this invocation's own named container, not + any other container from the image. Neither is claimed to be a + general-purpose PID/container-scoped kill: a caller outside the lock's + own serialization (the `.py`/`.exp` scripts below, or a sibling script's + own pre-acquire cleanup, such as `docker/qemu/run-aarch64-interactive.sh`'s + `docker kill $EXISTING`, reported on #829 but not fixed in this branch) + can still terminate a lock-cooperating script's process from outside the + lock. - **`run.sh --parallels` and `run.sh --vmware` are unaffected.** Both exit before reaching the native `qemu-system-aarch64` launch line this branch wires; they boot a VM via `prlctl`/VMware tooling, not a native QEMU @@ -320,3 +363,92 @@ claim-lint: scripts/claim-lint.py --commit-msg -> exit 0 (one per commit claim-lint: scripts/claim-lint.py -> exit 0 claim-lint: scripts/claim-lint.py --files docs/planning/green-program/gates/HOST-QEMU-LOCK-SCRIPTS-834-2026-09-05.md -> exit 0 + +## Fix round (2026-09-05): F1-F4 closed + +A review of this branch found four findings. F1 (blocking) and F4 (minor) +are corrected in place above (the "Why three scripts lost their `exec`" +section and the "Two pre-existing kill-by-pattern hazards" bullets in both +"What is disclosed" and "What is NOT claimed"); this section gives fresh +evidence for each of the four. + +**F1 (blocking) -- foreground launch loses the tracked child on a +direct-PID signal.** Fixed in `scripts/run-arm64-qemu.sh`, +`run-arm64-graphics.sh`, and `run-aarch64-userspace.sh`: QEMU is now +backgrounded with an explicit `0<&0`, its PID handed to +`qemu_host_lock_track_pid`, then `wait`ed on. + +``` +Mechanism proof, real files, real qemu-system-aarch64 (not a stand-in): + BREENIX_QEMU_LOCK=/lock ./scripts/run-arm64-qemu.sh release \ + (headless, backgrounded, real kernel boot) + -> lock dir appears (acquired), real qemu-system-aarch64 PID observed + via pgrep + kill -TERM + -> lock dir removed (released) AND + serial output shows: "qemu-system-aarch64: terminating on signal 15 + from pid