Skip to content

Wait for route RCU callbacks in homa_peer_free_net - #22

Draft
randomizedcoder wants to merge 1 commit into
mainfrom
fix/peer-free-net-rcu-barrier
Draft

randomizedcoder wants to merge 1 commit into
mainfrom
fix/peer-free-net-rcu-barrier

Conversation

@randomizedcoder

@randomizedcoder randomizedcoder commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Bug: homa_peer_free_net releases routes with call_rcu(homa_route_free), and the callback dereferences route->key.hnet. hnet is pernet data, which cleanup_net frees right after .exit returns and before its own rcu_barrier. That is a use-after-free when a namespace that carried Homa traffic is deleted, and also on rmmod (unregister_pernet_subsys frees the init_net hnet the same way). It came in with 8db9f5e (routes).

Fix: call rcu_barrier() at the end of homa_peer_free_net, after dropping the lock.

Found: by the nix microVM smoke test (ip netns del after GRO traffic). On 7.3 under load: GPF in homa_route_free from rcu_do_batch in 3 of 8 runs.

Verified:

unpatched patched
VM, init_on_free=1, 6.12 + 7.3 8/8 oops in homa_route_free 8/8 pass
VM matrix (2 smoke + 4 two-VM pairs, rmmod after traffic) 6/6 oops in homa_route_free 6/6 pass
unit homa_peer_free_net__wait_for_rcu_callbacks fail pass (885/885, plus --ipv4)

The 7.3 build is clean both normal and __STRIP__=1, and checkpatch is clean.

🤖 Generated with Claude Code

homa_peer_free_net releases each route in the namespace with
call_rcu(homa_route_free), and homa_route_free dereferences
route->key.hnet to find the peertab. But hnet is the pernet data for
the namespace, which cleanup_net frees right after the exit methods
return (its rcu_barrier comes later). A callback that runs after that
reads freed memory.

Found by deleting network namespaces that had carried Homa traffic, in
a VM: with init_on_free=1 every run oopsed in homa_route_free (6.12 and
7.3); without it, 3 of 8 runs on 7.3 hit a general protection fault
there under load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
randomizedcoder added a commit that referenced this pull request Oct 29, 2026
homa-vm-matrix runs every per-kernel smoke test and every cross-kernel
pair in parallel, each in its own directory (microvm.nix puts the QMP
socket in the cwd), and prints a summary table (verdict, echo, GRO
calls, cp_node rates) to the terminal and summary.md.

Its first run caught a use-after-free in homa_route_free on netns exit
and rmmod (STATUS.md F8; fix as fork draft #22). The guests now boot
with init_on_free=1, which turns that race into an oops on every run.

STATUS.md also records John's answer on PlatformLab#120: main will target an LTS
of 7.3 or later, so F1 (6.18 backports) is closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
randomizedcoder added a commit that referenced this pull request Oct 30, 2026
Following the pcp lifecycle tests: the guest gets two consoles on host TCP ports. ttyS0
stays the only kernel console and QEMU logs it to serial.log from the first byte; hvc0 is
a virtio console with a root shell that the host drives:

  1 start, 2 serial, 2b virtio, 3 smoke (homa-smoke over hvc0), 4 module cycles
  (insmod / netns add+del / rmmod x5), 5 dmesg, 6 poweroff, 7 qemu exit, 8 serial log

Each phase has its own timeout and time; phases that need the shell are skipped once it
is gone, and phase 8 still catches oopses after that (late RCU callbacks, F8). Reading
dmesg over hvc0 takes about 200 ms instead of seconds at 115200 baud.

On main, phase 3 crashes in homa_route_free (F8). With the fork #22 fix all 9 phases
pass in about 22 s, and homa-vm-matrix (now with homa-lifecycle-<series>) is 8/8 in 51 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant