feat(vmm): run virtio-net on vhost-net with configurable queue pairs

[kvinwang]

Problem

A CVM's virtio-net NIC cannot exceed one core's worth of packet processing, no
matter how many vCPUs it has. The VMM builds -netdev bridge,... with no
vhost=, and a device line hardcoded to virtio-net-pci,netdev=net0,mac=...
with no mq=on. Every packet is drained by QEMU's single main-loop thread:

ceiling ≈ 1 core ÷ per-packet main-loop cost

Sampled on a production CVM (phala-tdx-prod7, pid 3543413, 16 vCPU / 32 GiB,
-netdev bridge,id=net0,br=dstack-br0):

31,822 pps in, 0 drops in the sample window, qemu main thread 91%, total 721%
→ 0.91 core / 31,822 pps = 28.6 µs/packet → ceiling ≈ 35k pps

That matches the reported symptom exactly — clean at 31k pps, 36% loss at 40k.
The three busy TAPs on that host had lifetime drop rates of 26.7% / 19.3% /
17.8%, hundreds of millions of packets each. The drops are at the TAP, so
guest-side rx_dropped and softnet_stat stay at zero and the cliff looks
like a network fault rather than a host CPU limit. Outbound guest traffic uses
the same thread, so a chatty guest pays it twice.

docs/bridge-networking.md claimed this was unavoidable: "vhost-net ... is
not enabled for bridge mode. TDX encrypts guest memory, which prevents the
host kernel from performing DMA-based packet offload." That is wrong. A TDX
guest's virtio rings and buffers live in shared, unencrypted memory precisely
so a host-side backend can reach them — which is why vhost-vsock-pci, which
dstack has always used, works.

Fix

Two settings on [cvm.networking], each overridable per VM through the
deployment RPC. vhost is off by default: an upgraded node keeps building
byte-for-byte the device its VMs always had (userspace virtio, one queue pair)
until the operator opts in:

[cvm]
max_net_queues = 16         # ceiling for both the default and a request

[cvm.networking]
vhost = true                # opt-in; the shipped default is false

Queue pairs are not a node setting: with vhost on they default to the VM's vCPU
count, capped at max_net_queues, because the useful number follows the VM
rather than the host; without vhost the default stays one queue pair. A
deployment overrides that per VM.

vmm-cli.py deploy ... --net bridge --net-queues 4
vmm-cli.py deploy ... --net bridge --net-no-vhost

vectors is derived, never configured: 2N + 2. A single queue pair emits no
mq=on or vectors= at all, so a 1-vCPU VM and any VM deployed with
--net-queues 1 keep the historical command line byte for byte.

Each backend needed different plumbing:

Mode	netdev	notes
user	unchanged	no vhost backend, no multiqueue; a node-wide default is ignored here rather than failing the launch
bridge	tap,br=,helper=,vhost=on	QEMU's bridge netdev accepts neither vhost= nor queues=; the same setuid qemu-bridge-helper works behind a tap netdev, so the VMM still needs no CAP_NET_ADMIN
bridge, >1 queue	tap,ifname=,queues=N	the helper returns one descriptor, so netd creates a persistent multi_queue TAP — now also for unfiltered nodes
bridge + libvirt filter	tap,ifname=,vhost=on	the hardcoded vhost=off is now the configured value
macvtap	tap,fds=a:b,vhost=on	the launcher opens /dev/tapN once per queue; netd creates the link with matching numtxqueues/numrxqueues
custom	verbatim	the operator owns the netdev string; queues still drives the device line, which is what makes a hand-written netdev usable with multiqueue

Because macvtap now consumes several descriptors per NIC, the fd layout moved
into one macvtap_fd_layout() that both the launcher's open list and the
-netdev arguments derive from, replacing the 3 + index convention the two
sites previously duplicated.

QEMU aborts when vhost=on cannot open /dev/vhost-net, and it does so from
inside the launcher where the reason is easy to miss — verified on the QEMU
dstack resolves (8.2.2): there is no userspace fallback at netdev init, and
8.2 dies on qemu#1486's
assert(nc) rather than a clean error. The VMM warns when the device node is
missing and when its own open of the device is denied, but never refuses the
launch over it: QEMU need not share the VMM's credentials, so the VMM's own
access does not decide what the host can run.

Neither field is part of the measured VM configuration — that covers the OS
image, the vCPU and memory counts, the QEMU layout flags, the number of
NICs, and mr_config_id (compose hash and instance info) — so retuning a NIC
does not change app identity.

Verification

All five configurations deployed as real TDX CVMs on phala-tdx-lab (kernel
6.8, QEMU 8.2.2+tdx1.1, guest image dstack-0.6.0), against an isolated bridge
and a dedicated VMM instance.

Generated command lines, read back from /proc/<qemu>/cmdline:

A default        tap,id=net0,br=dstack-perf0,helper=/usr/lib/qemu/qemu-bridge-helper,vhost=on
                 virtio-net-pci,netdev=net0,mac=c6:29:aa:27:f7:05
B --net-no-vhost bridge,id=net0,br=dstack-perf0
                 virtio-net-pci,netdev=net0,mac=ba:6e:f8:29:d5:56
C --net-queues 4 tap,id=net0,ifname=dt4a44fb6e472b,script=no,downscript=no,vhost=on,queues=4
                 virtio-net-pci,netdev=net0,mac=4a:ea:3d:87:02:f1,mq=on,vectors=10
D macvtap, 2q    tap,id=net0,fds=3:4,vhost=on
                 virtio-net-pci,netdev=net0,mac=f6:c6:a1:17:e4:2c,mq=on,vectors=6
E libvirt filter tap,id=net0,ifname=dt851b876fcf4a,script=no,downscript=no,vhost=on

Host and guest state:

	A (default)	B (no vhost)	C (4 queues)	D (macvtap 2q)
guest ethtool -l eth0 combined	1	1	4	n/a (see below)
vhost worker threads in the QEMU thread group	2	1	5	3
DHCP lease + ssh into guest	yes	yes	yes	—

The worker count is net queue pairs + 1 for vhost-vsock in every case, which
is what confirms the guest actually negotiated that many queue pairs.

netd created what was asked for, and cleaned up on VM removal:

dt4a44fb6e472b: tun type tap ... multi_queue numqueues 4 persist on user kvin
dt7472999c6242@dstack-perf0: macvtap mode bridge ... numtxqueues 2 numrxqueues 2
/proc/212991/fd/3 -> /dev/tap1759
/proc/212991/fd/4 -> /dev/tap1759      # two opens, two queues

After removing all five VMs: no dt* interfaces and no nwfilter bindings left.

Filtering still works with vhost on. docs/libvirt-network-filter.md said
flipping this bit needed "equivalent filter integration tests", so from inside
VM E, with clean-traffic bound to the TAP and vhost=on:

ping -c 3 10.90.0.1                       → 3 received, 0% loss
ip addr add 10.90.0.222/24 dev eth0
ping -c 3 -I 10.90.0.222 10.90.0.1        → 0 received, 100% loss

The nwfilter binding lives on the host TAP interface, so packets traverse it
whether QEMU or a vhost worker wrote them.

Node policy, over the RPC:

--net-queues 9         → networking queues must not exceed 8 on this node
--net user --net-queues 4 → user-mode networking does not support multiple queues

Throughput, 64-byte UDP host→guest via kernel pktgen, same 8 vCPU / 8 GiB CVM
shape on the same host (the lab host was running other tenants' CVMs, so treat
the absolute rates as noisy and the CPU columns as the signal):

target	A: helper + vhost, 1q	B: legacy netdev, no vhost	C: netd + vhost, 4q
200k	0% loss, main 0%	0% loss, main 90%	0% loss, main 0%
500k	17.8% loss, main 0%	0.3% loss, main 93%	0% loss, main 0%
1M	26.6% loss, main 0%	5.2% loss, main 91%	0% loss, main 0%

The main thread goes from saturated to idle — that is the wall coming down, and
a whole core returned to the tenant. Note that vhost alone relocates the
ceiling rather than removing it: with one queue the guest's single receive queue
becomes the limit and drops reappear at a higher rate (the guest was at 99% CPU
in the 500k row). Multiqueue is what removes them. Hence the two travel
together: enabling vhost also enables the vCPU-scaled queue default.

A/B on the customer's own host (phala-tdx-prod7, production short-connection
workload, 4 alternating rounds to suppress noise) measured earlier in this
investigation: 10,922 → 13,933 conn/s (+27.6%), with the main thread going
94% → 0%.

Review follow-ups

A self-review pass found seven issues, all fixed in this branch:

• netd fail-open. Making filter optional so netd can build unfiltered
  multiqueue TAPs meant mode = "libvirt" with an explicit filter = "" no
  longer failed — netd skipped nwfilter-binding-create and every VM booted
  onto an unbound TAP, where it previously refused to start. Config load now
  rejects an empty filter in libvirt mode.
• One-shot mode. one_shot.rs still gated its "cannot manage TAP lifecycle"
  error on network_filter.mode == Libvirt, so queues > 1 on a bridge emitted
  -netdev tap,ifname=dt…,queues=4 for a TAP nobody created. It now uses the
  same needs_netd_interface predicate as the server path.
• Tuning forced a mode. --net-queues 4 alone was rejected despite the
  flag's "default: use global config" help, and on a node whose default backend
  is not in allowed_network_modes, restating the mode tripped node policy — so
  per-VM tuning was impossible there at all. A tuning-only request now keeps the
  node's backend; policy still governs backends a caller chooses.
• A missing bridge helper broke the node. bridge_helper() hard-failed the
  launch when none of three hardcoded paths existed on the VMM's filesystem.
  With vhost defaulting on, a working bridge node whose helper lives elsewhere
  would have lost every VM on upgrade — the same mistake the /dev/vhost-net
  probe was already corrected for. It now falls back to the non-vhost bridge
  netdev with a warning.
• Warning on every launch. The "user mode supports neither vhost nor
  multiqueue" warning tested the resolved value, and the shipped vmm.toml
  sets mode = "user" with vhost = true, so it fired on every launch of a
  stock node. Inheritance being ignored there is the documented design, not a
  silent skip; the warning is gone.
• Node tuning was pinned into the manifest. resolve_requested_networks
  persisted the merged values, so setting vhost = false node-wide to roll back
  reached VMs deployed with no override but not those deployed with
  --net bridge. Only what a deployment explicitly asks for is recorded now;
  identity-bearing fields are still pinned as before.
• max_net_queues was unvalidated. 0 rejected even queues = 1, and
  128 let a request past the node cap only to be rejected against the
  different MAX_NET_QUEUES = 64 bound.

Re-verified on phala-tdx-lab after the fixes:

--net-queues 4 with no --net   → tap,id=net0,ifname=dt…,vhost=on,queues=4
                                 virtio-net-pci,…,mq=on,vectors=10
manifest networks              → {"mode":"bridge",…,"queues":4}   # no "vhost" key
node vhost = false, restart    → runtime-networks {"vhost":false,"queues":4}
fresh VM, vhost=off + 4 queues → tap,…,vhost=off,queues=4 ; guest combined = 4
                                 1 vhost thread (vsock only) ; multi_queue numqueues 4
libvirt mode, empty filter     → cvm.network_filter.filter must not be empty…
max_net_queues = 0             → cvm.max_net_queues must be between 1 and 64

One unrelated observation from that run: these lab VMs exit after a stop/start
cycle, with dstack-prepare.service failing on the second boot. A control VM on
plain user networking — whose netdev string this PR leaves byte-for-byte
unchanged — reproduces it identically, so it is a property of the test app
(key_provider: none), not of this change.

Second review pass

Four more, all fixed:

• netd could not tear down an unfiltered TAP without libvirt. remove_interface
  decided whether to delete an nwfilter binding by checking that /usr/bin/virsh
  exists. On a mode = "none" node that has virsh installed but no reachable
  libvirtd, delete_binding fails on anything but the literal "binding not
  found", and prepare_bridge starts by calling remove_interface — so every
  multiqueue bridge VM would fail to start and leak its TAP. Remove/Check now
  carry whether the interface was created with a binding, defaulting to true on
  the wire so an older VMM's removals still clean up.
• Check reported healthy unfiltered TAPs as broken, since it ran
  nwfilter-binding-dumpxml for every non-macvtap interface. Same flag.
• netd version skew was invisible. netd is a separately deployed root
  service, and queues is #[serde(default)], so an older netd silently built a
  single-queue TAP while the VMM emitted queues=N — the exact IFF_MULTI_QUEUE
  mismatch the adjacent comment warns about, surfacing only as a QEMU failure
  inside the launcher. netd now echoes the queue count it created and the VMM
  refuses to launch on a mismatch.
• Tuning-only requests pinned a backend they were not allowed to choose. The
  first-pass fix let {"queues": 2} inherit the node's backend and skip
  allowed_network_modes, but resolve_requested_networks then pinned that
  mode, parent, and bridge into the manifest — so tuning was strictly more
  powerful than naming the backend, and a later node change no longer reached
  the VM. The rule is now uniform: pin what the caller named, inherit the rest.

Re-verified on phala-tdx-lab with virsh installed and libvirt_uri pointed at
a dead socket, which is the configuration that used to fail:

virsh --connect qemu+unix:///system?socket=/nonexistent  → failed to connect to the hypervisor

deploy --net bridge --net-queues 4
  → tap,id=net0,ifname=dte333c1530de5,…,vhost=on,queues=4 + mq=on,vectors=10
  → tun type tap … multi_queue numqueues 4 persist on user kvin
  → netd: prepared TAP … filter= queues=4
remove
  → netd: removed managed network interface … ; no leftover dt* interfaces

And the filtered path is unchanged, against a real libvirt:

deploy --net bridge (mode = "libvirt")
  → nwfilter-binding-list: dt1f60d8a14a79   clean-traffic
remove
  → binding deleted ; no leftover interfaces

Final regression on phala-tdx-lab

47 assertions against the final binary on a real TDX host (kernel 6.8, QEMU
8.2.2+tdx1.1, guest dstack-0.6.0), on a dedicated bridge and VMM instance —
config validation, RPC policy, generated command lines, host interface state,
guest-visible state, node rollback, and teardown. All pass.

A  config validation      max_net_queues 0 / 65 rejected; node queues 65 rejected;
                          libvirt mode with an empty filter rejected
B  RPC policy             queues 9 > cap rejected; user+multiqueue rejected;
                          queues 8 at the cap accepted -> 8 net vhost threads + vsock
C1 bridge default         tap,br=…,helper=/usr/lib/qemu/qemu-bridge-helper,vhost=on
                          device line unchanged, no mq=on; 1 net vhost thread
C2 --net-no-vhost         bridge,id=net0,br=dstack-perf0 ; vsock vhost thread only
C3 --net-queues 4         tap,ifname=dt…,vhost=on,queues=4 + mq=on,vectors=10
                          netd tap: multi_queue numqueues 4 ; 4 net vhost threads
C4 macvtap --net-queues 2 tap,fds=3:4,vhost=on + mq=on,vectors=6
                          fd 3 and fd 4 both -> /dev/tap1783 ; numtxqueues 2 numrxqueues 2
C5 --net-queues 2 alone   inherits the node backend: tap,ifname=dt…,vhost=on,queues=2
                          manifest: {"mode":"bridge","bridge":"","queues":2}  <- nothing pinned
G  guest ethtool -l       c1=1  c2=1  c3=4  c5=2 channels; all four got DHCP leases
D  node vhost=false       a VM deployed with --net-queues 4 comes back as
                          vhost=off,queues=4 with no net vhost threads
E  libvirt unreachable    unfiltered multiqueue VM deploys, runs and tears down;
                          netd never invokes virsh; no unowned interface left

Throughput on the final binary, 64-byte UDP host→guest via kernel pktgen, same
CVM shape on the same host:

offered	c1: helper + vhost, 1q	c2: legacy netdev, no vhost	b3: netd + vhost, multiqueue
200k	0.16% loss, main 0%	0.19% loss, main 88%	0% loss, main 0%
500k	8.7% loss, main 0%	1.4% loss, main 94%	0% loss, main 0%
1M	—	—	0% loss, main 0%

The main thread going 88–94% → 0% is the wall coming down. The c1 column is the
nuance stated above: with one queue vhost relocates the ceiling into the guest's
receive queue rather than removing it (guest CPU 121% vs 99% in the 500k row),
which is what the multiqueue column fixes.

One observation worth recording: that multiqueue VM had 2 vCPUs, and although
QEMU was given queues=8,vectors=18 the guest reported Combined: 2. The
virtio-net driver uses at most one queue pair per vCPU. Over-provisioning is
therefore inert rather than harmful, and it is not rejected at deployment
because resize can raise the vCPU count later; this is now documented.

Default queue count

With vhost on, queue pairs default to min(vcpu, 16). Since the default
flip (tenth pass, below) both are opt-ins rather than upgrade behaviour
changes; two consequences of opting in are still worth stating plainly:

• Every VM with two or more vCPUs gets a different virtio-net device on its
  next boot. It gains mq=on,vectors=2N+2 and N queue pairs where it
  previously had one. The device is not part of the measured VM configuration
  — that covers the OS image, vCPU/memory, QEMU layout flags, NIC count, and
  mr_config_id — so app identity is unaffected, but the guest does see a
  different NIC.
• Bridge nodes need netd to get it. qemu-bridge-helper returns a single
  descriptor and cannot create a multi_queue TAP. A node that has never
  deployed netd does not fail: bridge NICs fall back to one queue pair with a
  warning. A deployment that asked for a queue count explicitly still fails, so
  the caller learns their request was not met rather than silently getting less.

The measured trade-off, from an 8-vCPU TDX CVM with only the guest's channel
count changed (ethtool -L), is real in both directions:

queue pairs	short-connection throughput
1	22.3k conn/s
2	~20k conn/s
4	15–21k conn/s
8	6.2–7.7k conn/s

The same CVM moved 3.0 Mpps of 64-byte UDP with no loss at 8 queues against
roughly 600k at one. Bandwidth-bound workloads want the default; a VM serving
many short connections should set --net-queues 1. Cross-vCPU wakeups cost an
IPI and a VM exit under TDX, which is why the scaling is capped at 16 rather
than following large vCPU counts.

Verified on phala-tdx-lab, 23 assertions:

[cvm.networking] queues = 4   -> rejected: "has been removed"
 1 vCPU  -> helper netdev, no mq=on                    guest combined = 1
 4 vCPU  -> queues=4,  mq=on,vectors=10, 4+1 threads   guest combined = 4
20 vCPU  -> queues=16, mq=on,vectors=34, 16+1 threads  guest combined = 16
 8 vCPU --net-queues 1 -> helper netdev, no mq=on      guest combined = 1
 8 vCPU --net user     -> user netdev, no mq=on
no netd, default        -> helper netdev, one queue, warning logged
no netd, --net-queues 4 -> deploy fails naming netd

Surfaces and interactions

• vhost off turns the multiqueue default off too. Without vhost the QEMU main
  loop drains every queue on one thread, so extra queues buy little while still
  costing a netd interface, more MSI-X vectors, and a changed guest device.
  Anyone disabling vhost wants the old data plane, so they get the old shape.
  An explicit queue count is still honoured without vhost, since that
  combination is a deliberate request rather than a default.
• Raising the request ceiling does not raise the default. max_net_queues
  bounds what a deployment may ask for; the default's own cap is a fixed 16, so
  a larger VM never silently acquires a worse default. Hard ceiling from any
  source is 64.
• UpdateVm changes both fields, applying from the VM's next boot.
  vmm-cli.py update grew --net, --net-vhost / --net-no-vhost and
  --net-queues to reach it; the update path previously had no networking
  options at all.
• The web UI exposes both per NIC in the deploy and update dialogs, next to
  the mode selector. Leaving a control on its default emits no field, so the
  node keeps owning that value.

Verified on phala-tdx-lab (22 assertions):

8 vCPU --net-no-vhost           -> bridge,id=net0,br=…   no queues=, no mq=on, vsock vhost thread only
8 vCPU --net-no-vhost --net-queues 4 -> vhost=off,queues=4 + mq=on,vectors=10
24 queues, max_net_queues=16    -> rejected "must not exceed 16"
   after raising it to 32       -> accepted; queues=24, vectors=50
24 vCPU default, max=32         -> queues=16   (the default's cap did not move)
UpdateVm --net-queues 2 --net-no-vhost, restart -> vhost=off,queues=2, vectors=6
GetMeta.networking.max_queues   -> present for the UI to bound its input
UI payload {vhost:false,queues:2} -> accepted, applied
UI payload with both left unset   -> vCPU default, and neither field written to the manifest

Third review pass

Eight more, all fixed. Three would have stopped a working node from launching
VMs, which is the failure mode a defaulted-on feature has to be judged by:

• cvm.max_net_queues never bounded the default. Lowering it to 2 still
  handed a 16-vCPU VM sixteen queue pairs, contradicting both the vmm.toml
  comment and the docs. Raising it above 16 still only widens what a caller may
  request; lowering it below 16 now lowers the default too, because a node that
  refuses a request for four should not hand out sixteen by itself.
• One-shot dstack-vmm run broke on every bridge VM. It never applied the
  netd fallback, so with the default queue count a ≥2-vCPU bridge NIC hard-failed
  with "does not manage netd interface lifecycle" on a node that worked before.
• A dead netd made every bridge VM fail to start. Availability was probed
  with socket.exists(), but netd does not unlink its socket on shutdown, so a
  stale file read as "netd is here" and the launch then failed to connect
  instead of falling back. It now connects, the same way netd's own
  bind-time staleness check does.
• Two failure paths leaked a TAP. The new queue-mismatch bail! and the
  pre-existing response.device ? returned without the rollback loop, so a
  version-skewed netd left an interface on the bridge with nothing recorded to
  clean it up. Both now unwind through one shared rollback.
• Legacy VMs could not be stopped. The two "infer missing runtime networks"
  paths still called resolved_networks() rather than the clamped
  runtime_networks(), giving an already-running VM an unclamped queue count
  and failing stop_vm with a netd connect error.
• GetInfo reported a data plane the NIC did not get. A bridge NIC that
  fell back to the non-vhost bridge netdev for want of qemu-bridge-helper
  still reported vhost: true. The QEMU arguments and the reported status now
  read the same effective_vhost.
• The UI sent values it had hidden. Setting a queue count and then switching
  the NIC to user mode still submitted it — the deploy failed with no visible
  control to fix. queues/vhost are now scoped to non-user modes, like
  bridge_name.
• vmm-cli update --net-* replaced the whole NIC list, dropping extra
  interfaces and un-pinning a bridge. It now merges into the VM's existing NIC
  and refuses outright on a multi-NIC VM rather than guessing.

Fixing that last one surfaced a round-trip bug worth calling out on its own:
GetInfo reported parent and macvtap_mode on every interface, including
bridge NICs that had merely inherited them from [cvm.networking] — and the
deployment RPC rejects parent outside macvtap mode. Reported configuration
could be read but not sent back. Both fields are now scoped to macvtap the way
bridge_name is scoped to bridge, with a test that a reported interface
satisfies the RPC's own validation.

Verified on phala-tdx-lab (11 assertions):

16 vCPU, max_net_queues=2      -> queues=2         (the default follows the ceiling down)
stale /run/.../netd.sock       -> deploy succeeds, helper netdev, one queue, warning logged
update --net-queues 2 --net-no-vhost on a bridge VM
                               -> manifest keeps bridge=dstack-perf0, gains vhost:false queues:2
                               -> restart: tap,…,vhost=off,queues=2
GetInfo interfaces             -> {"vhost": false, "queues": 2}   matching the running QEMU
GetInfo configuration          -> {"mode":"bridge","bridge_name":"dstack-perf0","parent":""}
                                  i.e. round-trippable back into UpdateVm

Fourth and fifth review passes

Fourteen more findings, all fixed, and the shape of them says something about
the first three passes: nine came from the fixes those passes made.

Two root causes accounted for most of it, and both were the same mistake --
recomputing, from configuration an operator can edit at any time, a fact that
was only true at the moment something was built:

• Teardown re-derived whether netd owned an interface. Turning
  network_filter.mode off while VMs existed orphaned TAPs and leaked their
  nwfilter bindings. Interface names are a deterministic hash of the VM
  identity, so the same VM comes back on the same name and inherits the
  leftover ebtables rules -- silently filtered traffic on a NIC the operator
  believes is unfiltered. Reproduced on a TDX host against the pre-fix build:
  bindings after removal: dt77a0317f8bca clean-traffic. NICs now record what
  netd built for them.
• A tuning-only deployment pinned the node's backend into its manifest. It
  then stopped following the node, including into a mode node policy never let
  that caller choose. Networking now carries inherit_mode, and an inherited
  entry reports an empty mode, which is what it was deployed with.

GetInfo output could not be sent back. Its configuration is the input
UpdateVm takes, and both vmm-cli and the web UI read it, change one field,
and resend the rest. The node's own bridge was refused by the allowed_bridges
gate, and macvtap_mode was refused outright -- so vmm-cli update --net-*
failed on every bridge VM on a default-configured node and on every macvtap VM.
There is now one test that asserts the property over every mode and tuning
combination, rather than the one example each earlier pass fixed:

BRIDGE  ROUNDTRIP = Err("bridge_name 'br-node' is not allowed by node policy")
MACVTAP ROUNDTRIP = Err("macvtap_mode is node-controlled ...")

The rest: a stopped VM reported the NICs of a finished boot rather than the
ones its next launch would build; delete_binding was the one netd helper that
spawned virsh unbounded, and best-effort cleanup had just put it on every
prepare, where one unreachable libvirt could stall netd's serialized loop for
every other VM; status polling probed netd once per stopped VM, from inside the
global state lock, costing netd two warnings per probe; a queue count was
refused for a backend the caller inherited, which left the VM uneditable once
its node moved; the web UI could silently delete a NIC and renumber the rest,
changing their MAC addresses; and netd stopped answering requests it could
not parse, which is exactly what a VMM newer than its netd needs to read.

Verified on a TDX host: 21 assertions across the deployment, teardown,
inheritance and reporting paths, each one first shown to fail against the
pre-fix binary.

Sixth and seventh review passes

Twenty more, and the character of them finally changed: no data-plane or
teardown correctness left, but a long tail in the surfaces an operator actually
touches, and a lot of documentation that had drifted behind the code.

The knobs were unobservable. vmm-cli.py info printed nothing about
networking at all, and the web UI's interface panel was never extended, so the
two status fields this PR adds — the effective vhost state and queue count —
could only be read as raw JSON. That matters because both silently degrade: a
node without netd drops every defaulted bridge NIC to one queue pair, and a
missing qemu-bridge-helper drops it off vhost. Both now appear in info and
in the UI, with a note when the VM is not running and the numbers are a
prediction.

vmm-cli.py update --net <mode> could not change a mode. It merged the
NIC's current fields and overwrote only mode, so switching a bridge VM to user
sent a bridge_name along with it and was refused for a field the caller never
typed and had no flag to clear. Mode-owned fields are now dropped on a switch.
The same shape had a second cause: a VM pins its bridge for life, and once the
operator moved the node's own default, the VM's reported configuration stopped
being accepted back. An update may now restate what its own VM already holds.

The knobs could be set but not unset — --net-queues auto and
--net-vhost-default now exist, --net-queues 0 is refused with a sentence
instead of being silently discarded, and --net-queues -1 no longer surfaces a
serde error naming a column offset.

Two more instances of the pattern this PR keeps hitting — deciding something
at launch and then recomputing it later from configuration that has moved:
GetInfo recomputed each interface's vhost state, so an edit to
cvm.qemu_bridge_helper changed what a running VM was said to be using. The
data plane is now settled once at launch and written down, and VmInfo::to_pb
no longer takes a CvmConfig at all. And the single-queue fallback fired for
libvirt-filtered bridges, where it cannot help — filtering needs netd whatever
the queue count — reporting a shape no launch could produce.

Also: GetMeta advertised modes node policy forbids, so the deploy dialog
offered choices whose only outcome was "not allowed by node policy"; a queue
count or vhost request is now refused for a backend the caller chose and
accepted-but-dormant for one they inherited, since only the first is theirs to
correct; and a failed multiqueue prepare says a separately deployed netd may
predate multiqueue support, which the version-skew message previously missed in
the default unfiltered configuration.

Fourteen documentation claims were false or misleading, checked line by line
against the code: the queue-refusal rule was stated backwards after the code
changed under it, "byte for byte identical" applied only to the guest device
line and not the netdev, queues was described as a per-node setting it has
never been, the netd TAP form was missing from the bridge row that produces it
most often, and the pre-6.4 cgroup claim was wrong in the direction that
matters. Several code comments had drifted the same way.

Eighth review pass

Four independent lenses, one of them told nothing about the previous seven:
the Web UI's contract with the server, the CLI and RPC surface, a blind
correctness sweep of the Rust, and a line-by-line fact-check of every doc claim
against the code. Twenty-eight findings, all fixed and verified.

The two that mattered most were both about the privileged side of the boundary,
and neither was in the data plane.

netd stopped enforcing its own invariant. Unfiltered multiqueue needs a
bridge TAP with no nwfilter binding, so this PR taught netd to build one when
the caller asks. That moved the "every bridge TAP is filtered" invariant out of
the privileged daemon and into its unprivileged caller — and anything that can
reach the socket is a caller. On a node configured with
network_filter.mode = "libvirt", one JSON message now bought a root-created
TAP on a shared host bridge with no ebtables rules on it. netd holds the
policy itself again, via [netd] require_filter, derived from
cvm.network_filter.mode when the two share one vmm.toml so they cannot
drift.

Upgrading the VMM bricked every bridge VM on the node. netd is a separate
long-lived root service, so replacing the binary and restarting dstack-vmm
without restarting netd is the ordinary upgrade path. The liveness probe was
a bare connect, which cannot tell "netd is old" from "netd is fine" — so a
defaulted multiqueue request went to a netd that refuses it, and every bridge VM
failed to start on a node that had worked with no netd at all. There is now a
capabilities operation; a netd that predates it answers "unknown variant",
which is itself the answer. A defaulted NIC degrades to one queue pair the way
it does with no netd; an explicit request still fails loudly.

The type that was two things. Networking was simultaneously
[cvm.networking], what a VM pins, and what a launch hands QEMU. Resolution
therefore copied the node's whole value into every VM, so a bridge NIC's
manifest entry carried whatever macvtap parent the node happened to have
configured — and held_networking_config, which widens deployment policy from
what a VM already holds, widened from that. Repoint the node's parent and a
bridge VM could move itself to a macvtap parent policy forbids. NicNetworking
is now its own type and Networking flattens it, so the node's half is
unreachable from a VM's record by construction; the flatten keeps the
serialized shape byte-identical, and a test pins that so existing manifests and
snapshots keep loading.

Also fixed: GetInfo recomputed a running VM's data plane from current node
configuration when its snapshot predated the feature, and persisted the guess —
a build old enough to leave no snapshot had no vhost at all, so it now records
what that VMM actually launched. update --net user carried an inherited
vhost/queue pin into a mode that has neither and was rejected for a flag the
operator never typed. --net could pin a mode but nothing could un-pin it;
there is now --net default, alongside --net-queues auto and
--net-vhost-default. Lowering cvm.max_net_queues blocked every networking
update on VMs already above it while doing nothing about their actual queue
count. An explicit queues: 0 was silently reread as "unset". custom mode
reported vhost: off, queues: 1 over a netdev string the VMM never parses —
both fields are optional on the wire now and absent there. The "not running"
label was derived from the status string, which says something else: a VM whose
QEMU is up while it boots was labelled a prediction. cvm.qemu_bridge_helper
went unvalidated into a QEMU option list that splits on , and =.

The Web UI got its own pass: the update dialog has no <form>, so its
min/max never ran and 0 was silently dropped; queue counts are parsed
strictly in shared code now, so both dialogs behave the same. Macvtap NICs had
no parent field at all and rendered identically to each other, which made
"Remove" on a multi-NIC VM a coin flip that also changed the surviving NIC's
MAC. Networking rows survived between dialog openings, so a cancelled deploy or
a clone silently pinned the next VM's backend. Fourteen documentation claims
were wrong, including two acceptance criteria this PR itself contradicts, and
--net passt, which has never been a valid flag value.

Verification. 180 unit tests, clippy and fmt clean, tsc --noEmit clean,
and 41 assertions on real TDX hardware: the 26 from earlier rounds re-run after
the type split, plus 15 new ones covering the capability probe, netd's filter
policy, the un-pin paths, the queue cap, and the running-state label. Host left
with no leftover TAPs and no leftover nwfilter bindings.

Ninth review pass

Three lenses: an adversarial attack on the netd protocol, a completeness audit
of the type split, and a blind sweep by a reviewer told nothing about the
previous eight. Also a scoping correction from the reviewer that reshaped the
result: backward compatibility only has to hold against v0.5.11, the last
release. v0.5.11 has no netd.rs, no runtime-networks.json, no macvtap, no
cvm.network_filter, and its per-VM networking record stores mode and
nothing else.

The worst finding was one this PR introduced two rounds ago. The strict
queue-count parser added in the eighth pass assumed a string. Vue's v-model
casts for <input type="number">, so the field is a string only until the
operator types into it — and then .trim() throws a TypeError out of every
deploy or update that set a queue count. The headline feature was unreachable
from the web UI. Reproduced under jsdom against the exact Vue build the UI
loads, fixed, and re-reproduced green; the form field's type is now
string | number, so reintroducing the assumption fails tsc.

netd's new policy was bypassable in one request. The eighth pass gave netd
its own require_filter so the "every bridge TAP is filtered" invariant lived
in the privileged daemon again. It was checked in prepare_bridge only — and a
macvtap parent may be a bridge, while nwfilter does not apply to macvtap. So
prepare_macvtap with parent: "br0" returned a root-created /dev/tapN with
full unfiltered L2 access to the exact bridge the policy protects. Refused now,
along with a parent enslaved to one.

And the policy only checked that a filter was named, not that it filtered.
The caller supplied the nwfilter name and its parameters, so allow-arp — which
contains no drop rule at all — satisfied it, as did clean-traffic pinned
through its parameters to the gateway's MAC and IP. The request no longer
carries a filter name or parameters at all: it says whether to bind, and
[netd.network_filter] says what. A malformed policy section is now a startup
error rather than a silent fall back to "filter nothing".

The capability negotiation is gone. Under the v0.5.11 rule there is no
released netd, so the capabilities operation and its Legacy state were
compatibility machinery for a peer that does not exist — and they introduced a
failure mode the bare connect never had: netd serves connections serially, so a
netd busy preparing another VM's interface answered nothing within the probe
timeout and read as missing, silently dropping the VM to one queue pair. The
liveness check is a connect again, which is the one signal that does not depend
on what netd is doing right now.

Also fixed: manifest_networks still wrote the node's other mode's identity
field into a VM's record — the exact shape whose consequence, one round earlier,
was a policy bypass — so it is scoped once at the write point instead of by two
hand-written guards downstream. A request of {queues: 0} and nothing else
returned "no override at all" before anything read the count, answering a
request for no queues with up to sixteen. resolve_networking cleared
netd_interface but not device, under a comment claiming both. one_shot
never settled vhost, so it warned about a /dev/vhost-net the netdev it then
built does not open. The effective macvtap forwarding mode was reported nowhere
after the type split, though the proto still documented it. warn_if_vhost_net_missing
only stated the device — but the node is root:root 0600 on several
distributions, where turning vhost on node-wide stops every bridge VM from
restarting with the reason buried in a per-VM launcher, so it now also reports
an inaccessible device. The web UI reimplemented a queue cap the server
deliberately widens by what a VM already holds, making such a VM's networking
uneditable; and a VM pinned to a mode node policy no longer offers rendered a
blank <select>. cvm.networking.queues was rejected with "has been removed",
for a setting that never shipped, quoting the wrong cap. The pinning
documentation claimed a VM keeps its MAC prefix, subnet and DHCP start for life;
it keeps its mode and the bridge or parent that names it, and nothing else.

Verification. 181 unit tests, clippy and fmt clean, tsc --noEmit clean,
and 48 assertions on real TDX hardware — every earlier round's suite re-run
plus 10 new ones covering both spellings of the filter bypass, the filter that
netd itself names, the startup refusal on a malformed policy, and the bare
queues: 0. The generated QEMU command line was read off a live VM
(tap,...,vhost=on,queues=4 against virtio-net-pci,...,mq=on,vectors=10, TAP
multi_queue and enslaved to the bridge), and the host was left with no
leftover TAPs and no leftover nwfilter bindings.

Tenth pass: default flipped to opt-in, review comments verified on hardware

The human review on this PR asked for an upgrade story and questioned two
failure modes. Resolution:

• vhost now defaults to false (code default and shipped vmm.toml).
  The vCPU-scaled queue default only applies once vhost is on, so an upgraded
  node builds exactly the device its VMs always had until the operator opts
  in. docs/network-data-plane.md gained an "Enabling vhost on a node"
  checklist (verify /dev/vhost-net access for QEMU's account, restart netd
  with or before the VMM, roll back with vhost = false); 11 tests were
  re-pointed at explicit opt-in and a new one pins the no-op upgrade: a toml
  that predates the key and the shipped default both resolve to userspace
  virtio, one queue pair, no netd dependency.
• Startup vs runtime vhost failure, verified on phala-tdx-lab (QEMU
  8.2.2 — the binary dstack resolves — and 10.2): an unopenable
  /dev/vhost-net kills QEMU at netdev init on every version, single- and
  multi-queue alike, with no userspace fallback; 8.2 SIGABRTs via qemu#1486's
  assert(nc), newer versions exit cleanly. The falling back on userspace virtio path exists but is gated on a successfully opened vhost fd, so an
  access failure never reaches it. Both failure points are now documented,
  including that the runtime one is the single case where GetInfo can
  overstate the data plane.
• swiotlb, tested adversarially (dstack-0.6.0 guest, 16 vCPU x 16 queue
  pairs, offloads off, ~177 Gbit/s plus concurrent direct block I/O): the
  guest kernel clamps the bounce pool to a 64 MB floor regardless of RAM,
  while demand is ring-bounded at ~2 MB per queue pair; a deliberately
  undersized 32 MB pool produced zero swiotlb buffer is full events. The
  only failure producible at all was the guest page allocator at 1 GB RAM x
  16 vCPU — a RAM:vCPU ratio no deployment uses; 2 GB at the same shape ran
  clean — recorded in the docs as a searchable symptom, not a knob.
• CodeQL's two cleartext-logging alerts on netd.rs were dismissed as
  false positives: owner_uuid is a deterministic UUID derived from public
  identity fields, and the flagged lines are test assertion messages.

Operator notes

• vhost is off by default; enable it per node with [cvm.networking] vhost = true after verifying the account running QEMU can open
  /dev/vhost-net (root:kvm 0660 — add it to kvm). On prod7 this was
  already true and the module autoloaded on first open; no root, no
  pre-created TAPs, no CAP_NET_ADMIN. If it is not, QEMU dies at startup
  (8.2: net_client_init1: Assertion 'nc' failed) and the VM does not boot.
• On host kernels older than 6.4 the vhost worker is a free-standing kernel
  thread whose CPU time escapes the VM's cgroup. Since 6.4 it is a vhost_task
  inside the QEMU thread group — verified on the 6.8 lab host, where the worker
  tids appear under /proc/<qemu>/task/ with the same cgroup — so cpu.max
  and cgroup accounting still attribute it to the CVM.
• Bridge nodes that want more than one queue pair must run netd, including
  when network_filter.mode = "none".
• --net-no-vhost restores the previous -netdev bridge,... command line
  exactly, so the old behaviour is one flag away.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Leechael]

Reviewed against next and read the netd/VMM paths end to end. Design and the on-host verification look right; the points below are the ones that would bite us on rollout.

1. max_net_queues does not bound the default.
Networking::default_queue_pairs() (config.rs:1015) is vcpu.clamp(1, DEFAULT_QUEUE_SCALING_CAP) and never reads cvm.max_net_queues; validate_resolved_network only checks the hard 64. With max_net_queues = 4, --net-queues 5 is rejected while a 16-vCPU VM deployed without the flag gets 16 queue pairs. vmm.toml:57-58 ("default to the VM's vCPU count, capped at this value") and the description say the opposite. Since cvm.networking.queues is rejected at config load, an operator has no node-level way to bound the default short of vhost = false. Suggest vcpu.clamp(1, DEFAULT_QUEUE_SCALING_CAP.min(cfg.max_net_queues)), which keeps the "raising the ceiling does not raise the default" intent.

2. Queue-echo mismatch leaks the prepared interface; upgrade order is undocumented.
In prepare_filtered_networks the rollback loop only runs on the netd::request error arm. The two post-prepare exits — missing macvtap device (app.rs:638) and the queues echo check (app.rs:644) — bail after netd has already created the interface, without Remove and before set_runtime_networks, so stop/remove never see it. Each rejected start leaves one TAP per NIC behind (plus the earlier NICs already in prepared).
The trigger is the skew you describe in the follow-ups: dstack-netd.service is a separate long-running process, so upgrading and restarting dstack-vmm alone leaves the old netd answering. On such a node every bridge (libvirt mode) or macvtap VM with more than one vCPU is refused at start until netd is restarted. Fail-closed is the right call; please route these two exits through the same rollback, and add an upgrade note (restart netd before or together with the VMM) to docs/network-data-plane.md — none of the docs mention it today.

3. Existing nodes change their QEMU command line on upgrade with no config change.
vhost is Option<bool> with unwrap_or(true), so a vmm.toml that predates this PR turns vhost on. On a stock bridge node (network_filter.mode = none, no netd socket, helper at /usr/lib/qemu/qemu-bridge-helper) the netdev goes from bridge,id=…,br=X to tap,id=…,br=X,helper=…,vhost=on; QEMU then exits if the QEMU user cannot open /dev/vhost-net, and the VMM only checks that the node exists. Nodes running netd additionally move every multi-vCPU libvirt-bridge/macvtap VM to multiqueue on restart. I understand this is intentional; the description should state it as an upgrade step (verify /dev/vhost-net access as the QEMU user, or set vhost = false first), and I'd argue the safer default for a release is vhost = false with the on-by-default flip as a separate change.

4. update_networking replaces networks wholesale.
A request that changes mode without queues/vhost clears the previous per-VM tuning; a request with only tuning stores the node's current backend in the manifest. The UI round-trips the stored values so it is unaffected, but an RPC caller has to read-modify-write. Not a defect, but worth one sentence in the proto comments ("unset means cleared, not kept").

Context that may be useful in the docs:

• Runtime vhost_net_start failure does not exit: QEMU logs falling back on userspace virtio and continues on the userspace backend, and vhostforce does not change that. Worth listing next to the vhost thread-count check as something to alert on.
• The guest kernel has no swiotlb= and CONFIG_SWIOTLB_DYNAMIC off, so the bounce pool is a fixed clamp(6% RAM, 64 MB, 1 GB). The default queue count follows vCPUs while the pool follows RAM; a 16 vCPU / 4 GB shape gets 246 MB. Your 8 vCPU / 8 GiB lab shape does not exercise it.

Separately, port_map on a non-user NIC is still accepted and silently dropped (only hostfwd_index consumes it). Not this PR's scope; I'll follow up.

[Leechael]

Reference notes on how clouds pick a queue count — background for later tuning. dstack operators set --net-queues on the node or the VM; Phala Cloud will fill the same field from an instance type.

16 is a tuning convention, not a spec limit

Layer	Cap	Source
virtio / Linux uapi	VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX = 0x8000	virtio_net.h
Linux TAP (4.x+)	MAX_TAP_QUEUES = 256	if_tap.h
TAP on 3.x	8	Nova #1847367
RHEL 8/9 advice	one pair per vCPU, up to 16	RHEL 9, RHEL 8
RHEL 7	1–256, no 16 cap	RHEL 7
QEMU cmdline	queues=N, vectors=2N+2	linux-kvm Multiqueue

RHEL's "up to 16" is host-thread budget (one vhost-net worker per pair) plus diminishing returns. Same reason a default cap of 16 can sit below max_net_queues.

How clouds encode it on the instance type

They publish default per NIC, max per NIC, and often an instance-wide quota. The SKU fills the hypervisor knob.

AWS ENA — per SKU three columns. General-purpose default plateaus at 8; network-optimized default goes to 16/32. Formula: min(MAX_NUM_QUEUES_PER_ENI, vCPUs), where MAX_NUM_QUEUES_PER_ENI is 8 for most types and 32 for network-accelerated. Single ENI cannot exceed the instance vCPU count. 2025: queues are a pool allocatable across ENIs.

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ena-queues.html
• https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/ENA_Linux_Best_Practices.rst
• https://aws.amazon.com/about-aws/whats-new/2025/05/aws-ec2-instances-ena-queue-allocation-network-interfaces/

GCP — formula from machine type + NIC driver, override at create.

• VirtIO: min(vCPU / vNIC, 32)

• gVNIC + Titanium: min(vCPU / vNIC, 16) (C4N 64; A2/G2 / some Tier_1 32)

• older gVNIC: min(vCPU / vNIC / 2, 16)

• total usually ≤ vCPU; Tier_1 can oversubscribe

• https://docs.cloud.google.com/compute/docs/network-bandwidth

• https://cloud.google.com/blog/products/compute/boost-your-vm-with-virtual-network-interfaces

Aliyun ECS — closest to a SKU column. Bind-to-type applies the default; API exposes PrimaryEniQueueNumber, MaximumQueueNumberPerEni, TotalEniQueueQuantity. U-instance is roughly min(vCPU, 16). Stopped ENI can change within those two quotas; guest ethtool -L can lower further (lost on reboot).

• https://help.aliyun.com/zh/ecs/user-guide/nic-multi-queue
• https://help.aliyun.com/zh/ecs/user-guide/general-work-force

OpenStack Nova — min(vcpus, max_queues). max_queues is operator config (legacy 1/8/256 by kernel major); flavor extra spec only enables/disables MQ.

• https://docs.openstack.org/nova/latest/admin/networking.html
• https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html

Azure — SKU capability is Accelerated Networking (SR-IOV), not virtio queue count. Different dataplane.

• https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview

Practices that show up across those sources

Lined up with the numbers already in this PR (short connections 22k→6k conn/s from 1q to 8q; 64B UDP ~600k→3.0 Mpps). TDX makes the left-hand side worse than plain KVM: a cross-vCPU wakeup is an IPI plus a VM exit.

1. One pair per vCPU is the useful maximum, not the useful default. Extra pairs above vCPU are inert (guest virtio_net uses min(vcpu, advertised)). Too few pairs on a PPS-bound VM leave a single guest RX queue as the ceiling.
2. Default and max are different columns. AWS general-purpose defaults plateau at 8 on a 32-vCPU SKU; the max is 32. Aliyun's U-instance uses min(vCPU, 16) for both. Raising the request ceiling is not the same as retuning every VM's default.
3. Workload picks the number, not vCPU count.
   • many short TCP / RPC / conntrack churn → keep 1 (or a small N)
   • many concurrent flows / high PPS / large-UDP → raise toward min(vcpu, 8) and measure; 16 only if the host can afford the vhost threads
   • bandwidth with few large TSO flows saturates with a handful of pairs; more queues do not buy Gbps
4. Compute-shaped SKUs stay conservative; network-shaped SKUs raise the default. AWS states this directly (ENA best practices: more queues for network-intensive apps, fewer for CPU-intensive).
5. Instance-wide quota once there is more than one NIC. AWS / GCP / Aliyun cap Σ queues, not only per NIC.
6. virtio + vhost-net costs more on the host than ENA / gVNIC / Azure VF. Each pair is a vhost-net worker. A request ceiling of 32 can exist without the SKU default being 32.
7. Who sets the knob.
   • dstack operator: --net-queues / max_net_queues after measuring the VM
   • Phala Cloud: default_queues (and later max_queues) on the instance type; CreateVm sends that value. A plausible fill is min(vcpu, family_default_cap) — general/compute ≈ 8, a later network family ≈ 16

[kvinwang]

Thanks for the review — every point landed. Status, in your order:

1. max_net_queues not bounding the default — fixed as specified: default_queue_pairs is now vcpu.clamp(1, DEFAULT_QUEUE_SCALING_CAP.min(max_net_queues).max(1)), with a test pinning that lowering the cap lowers the default while raising it only widens what a caller may request.

2. Prepared-interface leak on the queue-echo mismatch — fixed; both post-prepare exits now route through the same rollback as a failed prepare.

3. Upgrade path — resolved more conservatively than an upgrade note: the default is now vhost = false. An upgraded node builds byte-for-byte the same QEMU device it always did (userspace virtio, one queue pair) until the operator opts in; the vCPU-scaled queue default only applies once vhost is on. docs/network-data-plane.md gained an "Enabling vhost on a node" checklist (verify /dev/vhost-net access for the account QEMU runs under → restart netd with/before the VMM → rollback path), and the VMM now warns at startup when its own open of /dev/vhost-net is denied, not just when the node is missing.

4. update_networking proto comment — states it now: "Empty + update_networking=true resets to node default."

Runtime fallback footnote — verified on hardware (QEMU 8.2.2, the version dstack resolves on the test node, plus 10.2): an unopenable /dev/vhost-net kills QEMU at netdev init on every version — 8.2 actually SIGABRTs via qemu#1486's assert(nc) rather than exiting cleanly — with no userspace fallback. The falling back on userspace virtio path you pointed at is real but gated on get_vhost_net(), i.e. reachable only after the vhost fd opened successfully at launch, so an access problem never lands there. Both failure points are documented separately now, including that the runtime one is the single case where GetInfo can overstate the data plane.

swiotlb footnote — tested adversarially (dstack-0.6.0 guest, 16 vCPU × 16 queue pairs, offloads off, ~177 Gbit/s plus concurrent direct block I/O): not a risk at any realistic shape. The guest kernel clamps the pool to a 64 MB floor regardless of RAM — the small-RAM tail where the concern would live doesn't exist — and demand is ring-bounded at ~2 MB per queue pair; a deliberately undersized 32 MB pool produced zero swiotlb buffer is full events under full load. The only failure we could produce at all was the guest page allocator at 1 GB RAM × 16 vCPU — a RAM:vCPU ratio no deployment uses, and 2 GB at the same shape ran clean — recorded in the docs as a searchable symptom, not a knob. No swiotlb= parameter is warranted; it would only shrink the allocator that actually fails.

On your provider survey: with vhost now opt-in, no VM acquires the min(vcpu, 16) default silently on upgrade. Whether 16 should come down toward the 8-at-32-vCPU shape your data shows is still open — happy to lower DEFAULT_QUEUE_SCALING_CAP if you think the AWS column is the right anchor.