Skip to content

Support vhost-user-vsock with the QEMU VMM - #3198

Open
dimorinny wants to merge 1 commit into
google:mainfrom
dimorinny:qemu-vhost-user-vsock
Open

dimorinny wants to merge 1 commit into
google:mainfrom
dimorinny:qemu-vhost-user-vsock

Conversation

@dimorinny

Copy link
Copy Markdown
Contributor

The qemu_cli VMM only wired up the kernel's vhost-vsock device, so run_cvd's vsock servers fail in containers without AF_VSOCK. Emit a vhost-user-vsock-pci-non-transitional device backed by vhost_device_vsock instead. auto now keys off the guest target_arch and covers arm64 and riscv64, so x86_64 hosts running arm64 guests under qemu_cli switch to vhost-user vsock.

QEMU connects to the chardev once at startup instead of retrying like crosvm, so wait for vhost.socket to listen first. That wait reads /proc/net/unix instead of lsof, and must not connect: the backend would consume the probe as its one allowed frontend.

Bug: b/561671404
Test: cvd start --vm_manager=qemu_cli --vhost_user_vsock=true, for an
x86_64 guest on an x86_64 host and an arm64 guest on an x86_64
host. Both reach sys.boot_completed=1 with adb working over vsock.

@Databean

Databean commented Sep 19, 2026 •

Copy link
Copy Markdown
Member

and must not connect: the backend would consume the probe as its one allowed frontend.

That was the original motivation for using lsof, see aosp/3047934. Is there an issue using it now?

@Databean Databean left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are presubmit errors:

 FAIL: //cuttlefish/common/libs/utils:wait_for_unix_socket_clang_tidy (Exit 1) (see /home/runner/.cache/bazel/_bazel_runner/a1ea073c0d21ddc5ae89652639f74223/execroot/_main/bazel-out/k8-fastbuild/testlogs/cuttlefish/common/libs/utils/wait_for_unix_socket_clang_tidy/test.log)
  ==================== Test output for //cuttlefish/common/libs/utils:wait_for_unix_socket_clang_tidy:
  37 warnings generated.
  INFO: From Testing //cuttlefish/common/libs/utils:wait_for_unix_socket_clang_tidy:
  warning: redefining builtin macro [clang-diagnostic-builtin-macro-redefined]
  /home/runner/.cache/bazel/_bazel_runner/a1ea073c0d21ddc5ae89652639f74223/sandbox/processwrapper-sandbox/10928/execroot/_main/cuttlefish/common/libs/utils/wait_for_unix_socket.cpp:19:1: error: included header sched.h is not used directly [misc-include-cleaner,-warnings-as-errors]
     19 | #include <sched.h>
        | ^~~~~~~~~~~~~~~~~~
     20 | 
  /home/runner/.cache/bazel/_bazel_runner/a1ea073c0d21ddc5ae89652639f74223/sandbox/processwrapper-sandbox/10928/execroot/_main/cuttlefish/common/libs/utils/wait_for_unix_socket.cpp:25:1: error: included header regex is not used directly [misc-include-cleaner,-warnings-as-errors]
     25 | #include <regex>
        | ^~~~~~~~~~~~~~~~
     26 | #include <sstream>
  /home/runner/.cache/bazel/_bazel_runner/a1ea073c0d21ddc5ae89652639f74223/sandbox/processwrapper-sandbox/10928/execroot/_main/cuttlefish/common/libs/utils/wait_for_unix_socket.cpp:29:1: error: included header utility is not used directly [misc-include-cleaner,-warnings-as-errors]
     29 | #include <utility>
        | ^~~~~~~~~~~~~~~~~~
     30 | 
  /home/runner/.cache/bazel/_bazel_runner/a1ea073c0d21ddc5ae89652639f74223/sandbox/processwrapper-sandbox/10928/execroot/_main/cuttlefish/common/libs/utils/wait_for_unix_socket.cpp:31:1: error: included header log.h is not used directly [misc-include-cleaner,-warnings-as-errors]
     31 | #include "absl/log/log.h"
        | ^~~~~~~~~~~~~~~~~~~~~~~~~
     32 | 
  /home/runner/.cache/bazel/_bazel_runner/a1ea073c0d21ddc5ae89652639f74223/sandbox/processwrapper-sandbox/10928/execroot/_main/cuttlefish/common/libs/utils/wait_for_unix_socket.cpp:37:1: error: included header command.h is not used directly [misc-include-cleaner,-warnings-as-errors]
     37 | #include "cuttlefish/process/command.h"
        | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     38 | #include "cuttlefish/process/managed_stdio.h"
  /home/runner/.cache/bazel/_bazel_runner/a1ea073c0d21ddc5ae89652639f74223/sandbox/processwrapper-sandbox/10928/execroot/_main/cuttlefish/common/libs/utils/wait_for_unix_socket.cpp:38:1: error: included header managed_stdio.h is not used directly [misc-include-cleaner,-warnings-as-errors]
     38 | #include "cuttlefish/process/managed_stdio.h"
        | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     39 | #include "cuttlefish/result/result.h"
  Suppressed 28 warnings (28 in non-user code).
  Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
  6 warnings treated as errors
  ================================================================================

The qemu_cli VMM only wired up the kernel's vhost-vsock device, so
run_cvd's vsock servers fail in containers without AF_VSOCK. Emit a
vhost-user-vsock-pci-non-transitional device backed by vhost_device_vsock
instead, and let `auto` enable it for qemu_cli and riscv64 guests.

QEMU connects to the chardev once instead of retrying like crosvm, so
for QEMU wait for vhost.socket to listen first. The wait reads
/proc/net/unix rather than lsof, which isn't a dependency and doesn't
report unix socket state before 4.94.0. Both wait loops now sleep 20 ms
instead of spinning.

Bug: b/561671404
Test: cvd start --vm_manager=qemu_cli --vhost_user_vsock=true, for an
      x86_64 guest on an x86_64 host and an arm64 guest on an x86_64
      host. Both reach sys.boot_completed=1 with adb working over vsock.
@dimorinny
dimorinny force-pushed the qemu-vhost-user-vsock branch from 30c8f24 to f4120d2 Compare September 25, 2026 03:04
@dimorinny

Copy link
Copy Markdown
Contributor Author

That was the original motivation for using lsof, see aosp/3047934. Is there an issue using it now?

lsof itself does the right check, but it's a problem in practice:

  1. It isn't a declared dependency: cuttlefish-base doesn't depend on it, and container/image/Containerfile (debian:13, --no-install-recommends) doesn't pull it in, so the nightly cuttlefish-orchestration image has no /usr/bin/lsof. A missing lsof fails the boot (RunAndCaptureStdout rejects nonzero exits). Today that's hidden because the only callers are behind vhost_user_block (default off); this PR starts using the wait for vhost-user vsock, on by default for arm64/riscv64 guests under qemu_cli, so I expect lsof to start hitting problems.
  2. lsof reports unix socket state only since 4.94.0 (Nov 2020). Ubuntu 22.04 and bullseye ship 4.93.2, where lsof -F TST prints no state line at all, so the wait times out after 30 s (verified on Ubuntu 22.04). That likely means vhost_user_block is already broken on those hosts.
  3. Cost: ~0.7 s per call, since it scans every process (why [assemble_cvd] Replace slow lsof for file cleanup #2489 dropped it).

lsof 4.94+ reads the same thing anyway: /proc/net/unix Flags & __SO_ACCEPTCON. The column layout is documented in proc(5), and the kernel has printed that bit there unchanged since 2.6.12. The new code does that check in process, still without connecting, so the aosp/3047934 constraint holds.

Pushed:

  • The vhost.socket wait is now QEMU-only, since crosvm already retries via --vhost-user-connect-timeout-ms.
  • clang-tidy include fix: <sched.h> dropped, lsof-fallback-only headers under #ifndef linux.
  • The flag check uses __SO_ACCEPTCON from <linux/net.h> instead of a literal.

@dimorinny
dimorinny requested a review from Databean September 25, 2026 03:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants