feat(marketplace): the node configures its own data plane - #366
Conversation
A node had a tunnel allocated to it and no way to use one. This gives it the document that says what its network should be, and the code that makes the machine match. `GET /api/v1/node/dataplane` returns the whole thing at once: the tunnel, the bridge guests are placed on, the gateway addresses the node must answer for, and the guests assigned to it. One call rather than three because the node applies them together or not at all — a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list the firewall increment will enforce. **The daemon configures the machine itself**, with `ip` and `wg`, rather than writing files for something else to read. A marketplace node runs on hardware LNVPS does not own: a data plane that depends on the operator having wired it up correctly is one whose mistakes surface as a customer's VM having no network. Everything is stated declaratively — `ip addr replace`, `ip route replace`, `wg set` — so a node that is already right is not disturbed and one that has drifted is corrected without being torn down. Three things the shape of the network forced: - **A guest's gateway belongs to its range, not to this node.** The guest is configured with the range's gateway and believes it is on-link, so the node holds that address on the bridge as a *host* address and answers for it with proxy ARP/NDP. Holding the whole range instead would make the node believe every other node's guests were local, and their traffic would vanish into the bridge rather than going up the tunnel. - **The bridge takes the tunnel's MTU.** A guest sending 1500 bytes into a 1420-byte tunnel gets a connection that opens and then hangs on the first large transfer, which is the worst failure shape available. - **The data plane is applied before the listener binds.** The control API binds an address of the tunnel interface, and on a fresh machine that interface does not exist until this has run. A failure to fetch is a warning rather than fatal: a node already up from a previous run must keep serving through an LNVPS outage, or one API blip takes the whole fleet dark. Two hygiene points that are not incidental. A peer that is not the route server is removed from `wg0` — most likely a stale key from a re-key, otherwise still able to send traffic the node treats as LNVPS's. And routes for departed guests are swept, because a released address goes straight back in the pool and may already be somebody else's; the bridge's own gateways are excluded from that sweep so tidying up after a guest cannot take the bridge's addressing with it. The node's WireGuard key is generated in-process rather than by shelling out to `wg genkey`, so a missing `wg` fails when the interface is configured, with that error. It is written `0600` with the mode set *before* the key exists, and reaches `wg` as a path, never an argument: arguments are visible in `ps` to every user on the machine, and these machines usually have more than one login. `/api/v1/status` now reports the observed data plane, queried on demand rather than cached — a cached answer says the tunnel was up once, which is exactly what the health gate must not accept. A tunnel that has never handshaken is reported as configured but not working, because `wg0` comes up perfectly happily with a peer that never answers. `lnvps-node dataplane show|apply|observe` exposes the same thing to an operator; `observe` deliberately needs no credential, because "what does this machine actually have?" is the question asked when something is already broken.
… is proved Three changes to the node's data plane, and the test that made them necessary. **Netlink instead of `ip`.** The daemon now talks to the kernel directly: `rtnetlink` for links, addresses and routes, WireGuard's own netlink interface for the tunnel, and `/proc/sys` for the forwarding knobs. `ip` is a program that formats netlink messages and formats the replies back into text for us to parse — going direct drops a dependency on iproute2's presence and version, drops the output parsing that changes between releases, and turns "a line of English on stderr" into kernel error codes. **The data plane lives in its own network namespace.** The first cut configured the *machine's* network: it replaced the operator's default route and turned on forwarding machine-wide, on hardware that is frequently not only an LNVPS node. Now `wg0` and `br-lnvps` live in an `lnvps` namespace: - their default route stays theirs, and the forwarding and proxy-ARP knobs are ours alone; - guests cannot reach the operator's network — not because a rule forbids it, which can be mis-ordered or flushed, but because no interface leads there; - a tunnel that is down means no path at all, instead of customer traffic leaking out the operator's uplink sourced from LNVPS addresses, which looks like spoofing to their upstream and can get *their* connection null-routed. `wg0` is created in the machine's namespace and then moved, because a WireGuard interface keeps its UDP socket where it was created: the encrypted outer traffic still leaves by the operator's uplink while everything carried over the tunnel is isolated. **The bridge name is no longer sent to the node.** Both sides hold it as a constant. The daemon needs the name before it has ever spoken to LNVPS — `dataplane observe` takes no credential — so a document that could name a different bridge would leave the node holding two answers to one question. **And a harness that sends real packets.** `lnvps_e2e/tests/tunnel_netns.rs` builds both ends out of network namespaces: the route server configured through its real code path, the node through its real netlink calls, a guest on the bridge behind it. It pings the node over the tunnel and then the guest through it, which is the path a customer's traffic takes. It earned its place immediately by finding four things no unit test could: - the namespace was pinned from `/proc/self/ns/net`, which in a multi-threaded process is the *process's* namespace — so every "isolated" interface was silently landing in the operator's own network; - WireGuard's netlink calls ran outside the namespace the interface had been moved into, reporting "no such device" about an interface that plainly existed; - giving an interface an address makes the kernel write entries in the *local* routing table, which the node then tried to delete as strays; - **the route server never routed its pool's own block.** An address on a point-to-point interface does not route the rest of its prefix, so a route server holding `10.66.0.1/16` answered "network is unreachable" for every node in the pool. That is merged 4b code doing exactly what it was written to do. The route server's command transport is now injectable so the harness can run its real commands in a namespace rather than over SSH; the orchestration on the node sits behind a `NetOps` trait so what the node *decides* is testable without root, with the kernel implementation proved by the harness.
|
Second commit, covering three of your points at once — and the test that made them necessary. An end-to-end harness that sends real packets
It found four bugs on its first run, none of which a unit test could have:
Netlink instead of
|
Increment 4c1 of the marketplace work. 4b configured LNVPS's end of the tunnel; this gives the node the document describing what its network should be, and the code that makes the machine match.
4c as planned was XL, so
work/marketplace.mdnow splits it: 4c1 node data plane (this PR) → 4c2 anti-spoof/anti-LAN firewall (nftandiptables, detected at runtime) → 4c3 health gate (probe guest, pinged from the route server, before the host is enabled).API
GET /api/v1/node/dataplane(node token) — tunnel, bridge, the gateway addresses the node must answer for, and the guests assigned to it (address, gateway, MAC). One call rather than three because the node applies them together or not at all: a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list 4c2 will enforce.Node
The daemon configures the machine itself with
ip/wg, idempotently, rather than writing files for something else to read — a node is hardware LNVPS does not own, and a data plane depending on the operator having wired it up correctly is one whose mistakes surface as a customer's VM having no network.Three things the shape of the network forced:
Hygiene that is not incidental: a peer that is not the route server is removed from
wg0(most likely a stale key from a re-key, still able to send traffic the node would treat as LNVPS's), and routes for departed guests are swept — a released address goes back in the pool and may already be somebody else's — with the bridge's own gateways excluded so the sweep cannot take the bridge's addressing with it.Key handling: generated in-process (a missing
wgshould fail when the interface is configured, with that error), written0600with the mode set before the key exists, and handed towgas a path — an argument is visible inpsto every login on the machine./api/v1/statusnow reports the observed data plane, queried on demand rather than cached: a cached answer says the tunnel was up once, which is what the gate must not accept. A tunnel that has never handshaken reports as configured but not working, sincewg0comes up happily with a peer that never answers.lnvps-node dataplane show|apply|observe;observeneeds no credential, because that is the question asked when something is already broken.Tests
net.rsruns commands through aCommandRunner, faked to answerip/wgqueries from a script and record every change — these run as root on somebody else's hardware, so the exact command is the thing worth asserting. Covered: a bare machine getting the whole data plane, an existing interface not being recreated, the private key never reaching a command line, stale peer removal, departed-guest sweep leaving the gateway alone, single-stack not producing a bogus default route, failures naming the command, and observation of a healthy, never-handshaken, and unconfigured machine. The API client is tested against a real server on a real socket.Workspace suite green.
net.rsandcontrol.rsat 100% function coverage; the residue inwgkey.rs/api.rsis error-context closures, matching the crate's existing pattern.