nix: homa-vm-matrix (pair plan phase 3); init_on_free in test VMs

homa-vm-matrix runs every per-kernel smoke test and every cross-kernel
pair in parallel, each in its own directory (microvm.nix puts the QMP
socket in the cwd), and prints a summary table (verdict, echo, GRO
calls, cp_node rates) to the terminal and summary.md.

Its first run caught a use-after-free in homa_route_free on netns exit
and rmmod (STATUS.md F8; fix as fork draft #22). The guests now boot
with init_on_free=1, which turns that race into an oops on every run.

STATUS.md also records John's answer on PlatformLab#120: main will target an LTS
of 7.3 or later, so F1 (6.18 backports) is closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Author: randomizedcoder

flake.nix

@@ -24,6 +24,7 @@
 #   nix run   .#homa-pair-test             # two VMs: echo + cp_node (Homa, TCP) over a link
 #   nix run   .#homa-pair-test-6_12-7_3    # ... server kernel, client kernel (every pairing)
 #   nix run   .#homa-pair-exp              # latency sweep (sysctls x workloads, metric deltas)
+#   nix run   .#homa-vm-matrix             # every VM test + pair in parallel, summary table
 #
 #   nix build .#homa-image                 # OCI userspace image  (docker load < result)
 #   nix build .#homa-image-debug

nix/README.md

@@ -206,6 +206,7 @@ nix run   .#homa-microvm[-<s>]       # interactive guest
 nix run   .#homa-pair-test           # two VMs: echo + cp_node (Homa, TCP) across a link
 nix run   .#homa-pair-test-<s>-<c>   # ... server kernel <s>, client kernel <c>
 nix run   .#homa-pair-exp            # latency sweep: cp_node per sysctl setting + metric deltas
+nix run   .#homa-vm-matrix           # every VM test and pair in parallel -> summary table
 nix run   .#homa-pair-exp-socket     # ... over QEMU's TCP socket link (STATUS.md F2)
 
 nix build .#homa-image[-debug]       # OCI user-space image (docker load < result)

nix/STATUS.md

@@ -12,6 +12,7 @@ Related docs:
 | | |
 |---|---|
 | **Last update** | 2026-10-28 |
+| 2026-10-29 | John: `main` will target an LTS of 7.3+ (F1 closed). Phase 3 `homa-vm-matrix`; guests boot with `init_on_free=1`; found F8 (UAF in `homa_route_free`), fix as fork #22 |
 | **Branch** | `nix-flake` on randomizedcoder/HomaModule, rebased on upstream `main` |
 | **nixpkgs** | `nixos-unstable` |
 | **Default kernel** | `linuxPackages_latest` (7.3); unit harness and fuzz-mock use 6.18.57 headers (F5) |
@@ -24,14 +25,15 @@ Related docs:
 
 | Area | Command | Result on `main` |
 |---|---|---|
-| Module build | `nix build .#homa-build-matrix` | ✅ 6.12, 7.0, 7.1, 7.2, 7.3. ❌ 6.18.57 (F1). ❌ 6.1/6.6 (no `net/rps.h`, below Homa's range) |
+| Module build | `nix build .#homa-build-matrix` | ✅ 6.12, 7.0, 7.1, 7.2, 7.3. ❌ 6.18.57 (F1, not a target). ❌ 6.1/6.6 (no `net/rps.h`, below Homa's range) |
 | Stripped build | `make __STRIP__=1` (7.3) | ✅ clean |
 | Unit tests | `nix build .#homa-tests` | ✅ 884/884, in IPv6 and `--ipv4` modes |
 | Fuzzing, real parsers | `nix build .#fuzz-mock` | ✅ all 10 harnesses run clean; the known-findings list is empty |
 | Fuzzing, wire decoder | `fuzz-run`, `fuzz-selftest` | ✅ selftest canary PASS |
 | Single VM | `homa-microvm-test-{6_12,7_3}` | ✅ insmod, loopback echo, netns/veth with GRO+RPS (about 1.7k `homa_gro_receive` and 460 `homa_set_softirq_cpu` calls), rmmod, clean dmesg |
 | Two VMs | `homa-pair-test-<server>-<client>` | ✅ all four 6.12/7.3 pairings: ping, echo 100 B–1 MB, cp_node w4 over Homa and over TCP, clean dmesg on both. 8 VMs in parallel finish in 50 s |
 | Latency sweep | `homa-pair-exp`, `homa-pair-exp-socket` | ✅ F2 explained: QEMU's TCP `socket` link stalls packets; over the UDP `dgram` link (now the default) Homa's P50 matches TCP's |
+| VM matrix | `homa-vm-matrix` | ❌ on `main`: all 6 oops in `homa_route_free` (F8). ✅ 6/6 with the fix (fork #22), 46 s |
 | KASAN VM | `homa-microvm-test-debug` | ⏳ not run yet (F7) |
 | Static analysis | `analysis-deep` | Four rounds triaged; see the bugs below |
 
@@ -59,7 +61,7 @@ that traffic flows, not how fast Homa is.
 | 6.1.190 | nixpkgs | ❌ no `net/rps.h` (below Homa's supported range) |
 | 6.6.159 | nixpkgs | ❌ no `net/rps.h` (below Homa's supported range) |
 | 6.12.113 | nixpkgs | ✅ |
-| 6.18.57 | nixpkgs | ❌ 6.18-stable backports (F1) |
+| 6.18.57 | nixpkgs | ❌ 6.18-stable backports (F1; not a target, LTS will be 7.3+) |
 | 7.0.14 | kernel.org | ✅ compile |
 | 7.1.13 | kernel.org | ✅ compile |
 | 7.2.12 | kernel.org | ✅ compile |
@@ -88,17 +90,20 @@ that traffic flows, not how fast Homa is.
 
 ## Findings
 
-### F1: 6.18-stable backports break the build on 6.18.40 and later
+### F1: 6.18-stable backports break the build on 6.18.40 and later (won't fix)
 
 6.18 stable backported two 7.x socket API changes: the recvmsg `addr_len` removal in 6.18.40,
 and `sockaddr_unsized` for bind in 6.18.57. Homa's gates (`< KERNEL_VERSION(7, 1, 0)` and
 `(7, 0, 0)`) therefore pick the old signatures on 6.18.57. That is a hard error with gcc 15+,
 and a `-Wincompatible-pointer-types` warning in the unit build.
 
 - **Status:** raised with John on
-  [#120](https://github.com/PlatformLab/HomaModule/pull/120#issuecomment-6486061071). He plans
-  to move `main` to the next LTS. If that is 7.3 or later, the problem goes away.
-- **Follow-up:** wait for his answer. If 6.18 is the target, send a small gate fix.
+  [#120](https://github.com/PlatformLab/HomaModule/pull/120#issuecomment-6486061071).
+  [His answer](https://github.com/PlatformLab/HomaModule/pull/120#issuecomment-6500983772)
+  (2026-10-29): the first LTS `main` targets will be 7.3 or later, so Homa doesn't need to
+  build on 6.18. No gate fix.
+- **Follow-up:** none upstream. 6.18 stays in `homa-build-matrix` as a known failure, so a
+  change in the gates shows up there.
 
 ### F2: Homa's 5 ms median over the VM link was QEMU's `socket` backend (resolved)
 
@@ -191,8 +196,9 @@ The unit harness and fuzz-mock build against 6.18.57 headers (`unitKernelPackage
 `test/mock.c` is maintained for John's development kernel, not for 7.3. That is fine for
 logic tests, but kernel-API drift on 7.x is only caught by the module builds and the VMs.
 
-- **Follow-up:** try `test/mock.c` against 7.3 headers. If the gaps are small, move the unit
-  KDIR to follow the default kernel, or to the LTS John picks (F1).
+- **Follow-up:** `main`'s LTS will be 7.3 or later (F1), so 6.18 headers will soon be the
+  odd one out. Try `test/mock.c` against 7.3 headers now, and move the unit KDIR to 7.3 when
+  `main` switches.
 
 ### F6: Small items noted while reading the code
 
@@ -210,20 +216,44 @@ logic tests, but kernel-API drift on 7.x is only caught by the module builds and
   IPv6 between VMs, and more than two nodes.
 - **7.0–7.2:** compile-checked only; there is no bootable kernel for them in the flake yet.
 
+### F8: Use-after-free in `homa_route_free` on netns exit and rmmod (fix: fork #22)
+
+`homa_peer_free_net()` (the pernet `.exit` path) releases each of the namespace's routes
+with `call_rcu(homa_route_free)`. The callback dereferences `route->key.hnet->homa->peertab`.
+But `hnet` is pernet data (`.size = sizeof(struct homa_net)`), which the netns core frees
+straight after `.exit` and before its own `rcu_barrier()`. It came in with upstream
+`8db9f5e` (routes, 2026-10-14).
+
+- **Found by:** the first `homa-vm-matrix` run. One 7.3 smoke VM hit a general protection
+  fault in `homa_route_free` (from `rcu_do_batch`) after `ip netns del`. Unloaded VMs rarely
+  show it (3 of 8 under load, 0 of 16 alone), because the freed `hnet` usually still holds
+  valid pointers.
+- **Made deterministic:** the test guests now boot with `init_on_free=1`. Unpatched, every
+  smoke run oopses on 6.12 and 7.3 (8/8), and so does every pair client on `rmmod`
+  (`unregister_pernet_subsys` frees the init_net `hnet` the same way).
+- **Fix:** `rcu_barrier()` at the end of `homa_peer_free_net()`. Unit test
+  `homa_peer_free_net__wait_for_rcu_callbacks` is red then green, with 885/885 unit tests
+  passing in IPv6 and `--ipv4` modes. The 7.3 build is clean normal and stripped, checkpatch
+  is clean, and the VMs go from 8/8 oops to 8/8 pass and the matrix from 0/6 to 6/6.
+  Draft:
+  [randomizedcoder/HomaModule#22](https://github.com/randomizedcoder/HomaModule/pull/22).
+- **Until it lands,** `homa-vm-matrix` and the VM tests fail on `main`, as they should.
+
 ---
 
 ## Follow-up actions
 
 | # | Action | Why | Effort | Who |
 |---|---|---|---|---|
 | 1 | ~~F2 experiments~~ ✅ done: QEMU `socket` backend stall, pair tests moved to `dgram` | — | — | — |
-| 2 | PLAN-pairs phase 3: `homa-vm-matrix` summary runner | One-command demo; would also have caught F2 at a glance | small | us |
+| 2 | ~~PLAN-pairs phase 3: `homa-vm-matrix`~~ ✅ done; its first run found F8 | — | — | — |
+| 13 | Send F8 upstream once the user has reviewed fork #22 | Use-after-free on every netns exit and rmmod | — | us → upstream |
 | 3 | PLAN-pairs phase 4: netem loss and delay | Test RESEND and timeout paths across a real link (F3) | small | us |
 | 4 | PLAN-pairs phase 5: Homa qdisc on virtio | Should reproduce round-4 candidate 5 in a real kernel; exercises the pacer | medium | us |
 | 5 | Run `homa-microvm-test-debug`, then a KASAN pair test | Memory-safety coverage of the real data path | medium (kernel build) | us |
 | 6 | IPv6 pair test (`--ipv6` for cp_node, v6 addresses on eth1) | #137/#138 were IPv6 bugs; nothing tests v6 between nodes | small | us |
 | 7 | Round-4 candidates 5–8, test-first | More upstream fixes in John's style | medium | us → upstream |
-| 8 | Which LTS will `main` target? | Decides F1 and the unit KDIR (F5) | — | ask John (#120) |
+| 8 | ~~Which LTS will `main` target?~~ ✅ 7.3 or later (John, #120, 2026-10-29) | Closes F1; unit KDIR moves to 7.3 (F5) | — | — |
 | 9 | Restore the `homa_drop_packet` call, or drop its sysctls? | F3 | — | ask John |
 | 10 | ~~Post F2 upstream~~ not needed: VM link, not Homa | — | — | — |
 | 11 | Bootable 7.0–7.2 kernels (PLAN-pairs phase 6) | Full VM coverage of the 7.x series | large (kernel builds) | optional |

nix/microvms/PLAN-pairs.md

@@ -60,7 +60,7 @@ Two guests built by the existing `mkVm.nix`, each with a second NIC:
 |---|---|---|---|
 | 1 ✅ | Pair harness, same kernel | `homa-pair-test` (default kernel) | **Done:** echo + cp_node Homa/TCP pass on 7.3 ↔ 7.3, about 35 s cached; a client with a dead link fails at `PING`. The interactive pair (`homa-pair`) moves to phase 3 |
 | 2 ✅ | Cross-kernel pairs | `homa-pair-test-<A>-<B>` for supported kernels (6.12 ↔ 7.3 both ways) | **Done:** all four pairings pass; each has its own link port, and all four run in parallel in 50 s. `vmSeries` in `kernel.nix` limits VM targets to 6.12 and 7.3 |
-| 3 | One command for everything | `homa-vm-matrix`: every single-VM test plus every pair, run in parallel, printing a table (kernel(s), verdict, RPC counts, cp_node rates) | `nix run .#homa-vm-matrix` is the demo |
+| 3 ✅ | One command for everything | `homa-vm-matrix` (`matrix.nix`): every single-VM test plus every pair, run in parallel, printing a table (kernel(s), verdict, echo, GRO calls, cp_node rates) | **Done:** 6 tests in 46 s. Its first run found a use-after-free in `homa_route_free` (STATUS.md F8); guests now boot with `init_on_free=1` |
 | 4 | Lossy and slow links | `homa-pair-test-netem`: `tc netem loss 1% delay 1ms` on eth1 | Echo still completes, and the RESEND/retransmit counters in `/proc/net/homa_metrics` move |
 | 5 | Homa qdisc on virtio | the pair with `tc qdisc replace dev eth1 root homa` | Exercises the pacer across VMs. virtio-net reports `SPEED_UNKNOWN`, so this should reproduce round-4 candidate 5 (link speed −1): a red test in a VM, then green with the fix |
 | 6 (stretch) | Boot 7.0–7.2 | full kernels built from the kernel-src.nix tarballs (`buildLinux`, about 30 min each, cached) | The VM matrix covers every 7.x series, not just compile-only |

nix/microvms/default.nix

@@ -7,6 +7,10 @@
 #                                  RPCs, rmmod), PASS/FAIL from the console log
 #   homa-microvm-debug             default kernel with KASAN/KMEMLEAK
 #   homa-microvm-test-debug        ... the same test under sanitizers
+#   homa-pair-test[-<s>-<c>]       two VMs (lib-pair.nix): Homa between server <s>, client <c>
+#   homa-pair-exp[-socket]         latency sweep over the pair link (STATUS.md F2)
+#   homa-vm-matrix                 every homa-microvm-test-<series> and pair, in parallel,
+#                                  with a summary table (matrix.nix)
 # <series> is every kernel.nix matrix entry (6_1 ... 7_3); the unsuffixed names use the
 # default kernel. Tests share nothing on the host, so they can run in parallel.
 #
@@ -108,6 +112,26 @@ let
 
   defaultKp = kernelConfig.kernelPackages;
 
+  # homa-vm-matrix: every per-kernel smoke test and every pair, in parallel (phase 3).
+  matrix = import ./matrix.nix {
+    inherit pkgs lib;
+    tests =
+      lib.mapAttrsToList
+        (s: _: {
+          name = "homa-microvm-test-${s}";
+          kernels = s;
+          exe = perKernel."homa-microvm-test-${s}";
+        })
+        kernelConfig.vmMatrix
+      ++ map
+        (p: {
+          name = "homa-pair-test-${p.server}-${p.client}";
+          kernels = "${p.server} → ${p.client}";
+          exe = perPair."homa-pair-test-${p.server}-${p.client}";
+        })
+        pairList;
+  };
+
   perKernel = lib.concatMapAttrs
     (s: kp: {
       "homa-microvm-${s}" = mkGuest { kernelPackages = kp; };
@@ -148,5 +172,6 @@ in
       experiments = latencyExperiments;
       link = "socket";
     };
+    homa-vm-matrix = matrix;
   } // perKernel // perPair;
 }

nix/microvms/matrix.nix

@@ -0,0 +1,90 @@
+# nix/microvms/matrix.nix
+#
+# One command for every VM test (PLAN-pairs.md phase 3): runs each per-kernel single-VM
+# smoke test and each (server, client) pair test in parallel, then prints a summary table
+# (also written to $logdir/summary.md):
+#
+#   | Test | Kernel(s) | Verdict | Time (s) | Echo | GRO calls | Homa Kops/s | Homa P50 / P99 (us) | TCP ... |
+#
+# Each test keeps its own console logs and output under $HOMA_VM_LOGDIR/<test>/. The tests
+# share nothing on the host (each pair has its own link port), so they can all run at once;
+# HOMA_VM_JOBS caps how many run together (default: all). Exits non-zero if any test fails.
+
+{ pkgs
+, lib
+, tests   # list of { name, kernels, exe }: exe is a homa-microvm-test-* or homa-pair-test-* package
+}:
+
+pkgs.writeShellApplication {
+  name = "homa-vm-matrix";
+  runtimeInputs = with pkgs; [ coreutils gnugrep gawk ];
+  text = ''
+    logdir="''${HOMA_VM_LOGDIR:-$(mktemp -d)}"
+    mkdir -p "$logdir"
+    jobs="''${HOMA_VM_JOBS:-${toString (builtins.length tests)}}"
+    echo "VM matrix: ${toString (builtins.length tests)} tests, up to $jobs at once; logs in $logdir"
+
+    # run <name> <exe>: one test, its output in <name>.out, exit code and seconds in <name>.rc
+    run() {
+      local start rc=0
+      start=$(date +%s)
+      mkdir -p "$logdir/$1"
+      # Each test runs in its own directory: microvm.nix puts the QMP socket
+      # (<hostname>.sock) in the current directory, and every pair's server is node0.
+      (cd "$logdir/$1" && HOMA_VM_LOGDIR="$logdir/$1" "$2") > "$logdir/$1.out" 2>&1 || rc=$?
+      echo "$rc $(( $(date +%s) - start ))" > "$logdir/$1.rc"
+      echo "  done: $1 (exit $rc)"
+    }
+
+    start=$(date +%s)
+    ${lib.concatMapStrings (t: ''
+      while [ "$(jobs -rp | wc -l)" -ge "$jobs" ]; do wait -n || true; done
+      run ${t.name} ${lib.getExe t.exe} &
+    '') tests}
+    wait
+
+    # field <line> <word>: the word after <word> in a cp_node stats line ("clients:", "P50",
+    # "P99"), or a dash when there is none (single-VM tests have no cp_node runs).
+    field() {
+      echo "$1" | awk -v w="$2" '
+        { for (i = 1; i < NF; i++) if ($i == w) { v = $(i + 1); exit } }
+        END { print (v == "" ? "—" : v) }'
+    }
+
+    summary="$logdir/summary.md"
+    {
+      echo "| Test | Kernel(s) | Verdict | Time (s) | Echo | GRO calls | Homa Kops/s | Homa P50 / P99 (us) | TCP Kops/s | TCP P50 / P99 (us) |"
+      echo "|---|---|---|---|---|---|---|---|---|---|"
+    } > "$summary"
+    failed=0
+    row() {
+      local name=$1 kernels=$2 out="$logdir/$1.out" rc secs verdict echo_ gro homa tcp
+      read -r rc secs < "$logdir/$name.rc"
+      verdict=FAIL
+      grep -q '^VERDICT: PASS' "$out" && verdict=PASS
+      [ "$verdict" = PASS ] || failed=$((failed + 1))
+      # Any of these may be missing (a failed run, or a single-VM test without cp_node).
+      echo_=$(grep -aoE 'HOMA_(SMOKE|PAIR)_ECHO responses=[0-9]+' "$out" | head -n 1 | sed 's/.*=//' || true)
+      gro=$(grep -aoE 'homa_gro_receive calls=[0-9]+' "$out" | head -n 1 | sed 's/.*=//' || true)
+      homa=$(grep -a 'HOMA_PAIR_CPNODE_homa ' "$out" | tail -n 1 || true)
+      tcp=$(grep -a 'HOMA_PAIR_CPNODE_tcp ' "$out" | tail -n 1 || true)
+      printf '| %s | %s | %s | %s | %s | %s | %s | %s | %s | %s |\n' \
+        "$name" "$kernels" "$verdict" "$secs" "''${echo_:-0}/6" "''${gro:-—}" \
+        "$(field "$homa" clients:)" "$(field "$homa" P50) / $(field "$homa" P99)" \
+        "$(field "$tcp" clients:)" "$(field "$tcp" P50) / $(field "$tcp" P99)" >> "$summary"
+    }
+    ${lib.concatMapStrings (t: ''
+      row ${t.name} ${lib.escapeShellArg t.kernels}
+    '') tests}
+
+    echo
+    cat "$summary"
+    echo
+    echo "summary: $summary ($(( $(date +%s) - start )) s)"
+    if [ "$failed" -gt 0 ]; then
+      echo "VERDICT: FAIL ($failed of ${toString (builtins.length tests)})"
+      exit 1
+    fi
+    echo "VERDICT: PASS (${toString (builtins.length tests)} of ${toString (builtins.length tests)})"
+  '';
+}

nix/microvms/mkVm.nix

@@ -81,6 +81,9 @@ let
         boot.kernelParams = [
           "console=ttyS0,115200"
           "systemd.show_status=true"
+          # Zero freed memory, so a use-after-free in Homa reads NULLs and oopses at once
+          # instead of usually reading stale but valid pointers (STATUS.md F8).
+          "init_on_free=1"
         ] ++ lib.optionals isPair [
           # Stable names: the user NIC is eth0 and the pair link eth1.
           "net.ifnames=0"