Skip to content

feat(marketplace): the node configures its own data plane - #366

Open
v0l wants to merge 2 commits into
masterfrom
feat/marketplace-node-dataplane
Open

feat(marketplace): the node configures its own data plane#366
v0l wants to merge 2 commits into
masterfrom
feat/marketplace-node-dataplane

Conversation

@v0l

@v0l v0l commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Increment 4c1 of the marketplace work. 4b configured LNVPS's end of the tunnel; this gives the node the document describing what its network should be, and the code that makes the machine match.

4c as planned was XL, so work/marketplace.md now splits it: 4c1 node data plane (this PR) → 4c2 anti-spoof/anti-LAN firewall (nft and iptables, detected at runtime) → 4c3 health gate (probe guest, pinged from the route server, before the host is enabled).

API

GET /api/v1/node/dataplane (node token) — tunnel, bridge, the gateway addresses the node must answer for, and the guests assigned to it (address, gateway, MAC). One call rather than three because the node applies them together or not at all: a bridge with no tunnel carries nothing, a tunnel with no guest routes carries nothing back, and a document that can be half-fetched is a data plane that can be half-applied. The guest list is also the anti-spoof list 4c2 will enforce.

Node

The daemon configures the machine itself with ip/wg, idempotently, rather than writing files for something else to read — a node is hardware LNVPS does not own, and a data plane depending on the operator having wired it up correctly is one whose mistakes surface as a customer's VM having no network.

Three things the shape of the network forced:

  • A guest's gateway belongs to its range, not to this node. The guest believes it is on-link, so the node holds it as a host address on the bridge with proxy ARP/NDP. Holding the whole range would make the node believe every other node's guests were local, and their traffic would vanish into the bridge instead of going up the tunnel.
  • The bridge takes the tunnel's MTU. 1500 bytes into a 1420-byte tunnel opens a connection and then hangs on the first large transfer — the worst failure shape available.
  • The data plane is applied before the listener binds, because the control API binds an address of the tunnel interface that does not exist until this runs. A failed fetch is a warning, not fatal: a node already up must survive an LNVPS outage, or one API blip darkens the fleet.

Hygiene that is not incidental: a peer that is not the route server is removed from wg0 (most likely a stale key from a re-key, still able to send traffic the node would treat as LNVPS's), and routes for departed guests are swept — a released address goes back in the pool and may already be somebody else's — with the bridge's own gateways excluded so the sweep cannot take the bridge's addressing with it.

Key handling: generated in-process (a missing wg should fail when the interface is configured, with that error), written 0600 with the mode set before the key exists, and handed to wg as a path — an argument is visible in ps to every login on the machine.

/api/v1/status now reports the observed data plane, queried on demand rather than cached: a cached answer says the tunnel was up once, which is what the gate must not accept. A tunnel that has never handshaken reports as configured but not working, since wg0 comes up happily with a peer that never answers.

lnvps-node dataplane show|apply|observe; observe needs no credential, because that is the question asked when something is already broken.

Tests

net.rs runs commands through a CommandRunner, faked to answer ip/wg queries from a script and record every change — these run as root on somebody else's hardware, so the exact command is the thing worth asserting. Covered: a bare machine getting the whole data plane, an existing interface not being recreated, the private key never reaching a command line, stale peer removal, departed-guest sweep leaving the gateway alone, single-stack not producing a bogus default route, failures naming the command, and observation of a healthy, never-handshaken, and unconfigured machine. The API client is tested against a real server on a real socket.

Workspace suite green. net.rs and control.rs at 100% function coverage; the residue in wgkey.rs/api.rs is error-context closures, matching the crate's existing pattern.

v0l added 2 commits August 6, 2026 17:11
A node had a tunnel allocated to it and no way to use one. This gives it the
document that says what its network should be, and the code that makes the
machine match.

`GET /api/v1/node/dataplane` returns the whole thing at once: the tunnel, the
bridge guests are placed on, the gateway addresses the node must answer for, and
the guests assigned to it. One call rather than three because the node applies
them together or not at all — a bridge with no tunnel carries nothing, a tunnel
with no guest routes carries nothing back, and a document that can be
half-fetched is a data plane that can be half-applied. The guest list is also
the anti-spoof list the firewall increment will enforce.

**The daemon configures the machine itself**, with `ip` and `wg`, rather than
writing files for something else to read. A marketplace node runs on hardware
LNVPS does not own: a data plane that depends on the operator having wired it up
correctly is one whose mistakes surface as a customer's VM having no network.
Everything is stated declaratively — `ip addr replace`, `ip route replace`, `wg
set` — so a node that is already right is not disturbed and one that has drifted
is corrected without being torn down.

Three things the shape of the network forced:

- **A guest's gateway belongs to its range, not to this node.** The guest is
  configured with the range's gateway and believes it is on-link, so the node
  holds that address on the bridge as a *host* address and answers for it with
  proxy ARP/NDP. Holding the whole range instead would make the node believe
  every other node's guests were local, and their traffic would vanish into the
  bridge rather than going up the tunnel.
- **The bridge takes the tunnel's MTU.** A guest sending 1500 bytes into a
  1420-byte tunnel gets a connection that opens and then hangs on the first
  large transfer, which is the worst failure shape available.
- **The data plane is applied before the listener binds.** The control API binds
  an address of the tunnel interface, and on a fresh machine that interface does
  not exist until this has run. A failure to fetch is a warning rather than
  fatal: a node already up from a previous run must keep serving through an
  LNVPS outage, or one API blip takes the whole fleet dark.

Two hygiene points that are not incidental. A peer that is not the route server
is removed from `wg0` — most likely a stale key from a re-key, otherwise still
able to send traffic the node treats as LNVPS's. And routes for departed guests
are swept, because a released address goes straight back in the pool and may
already be somebody else's; the bridge's own gateways are excluded from that
sweep so tidying up after a guest cannot take the bridge's addressing with it.

The node's WireGuard key is generated in-process rather than by shelling out to
`wg genkey`, so a missing `wg` fails when the interface is configured, with that
error. It is written `0600` with the mode set *before* the key exists, and
reaches `wg` as a path, never an argument: arguments are visible in `ps` to every
user on the machine, and these machines usually have more than one login.

`/api/v1/status` now reports the observed data plane, queried on demand rather
than cached — a cached answer says the tunnel was up once, which is exactly what
the health gate must not accept. A tunnel that has never handshaken is reported
as configured but not working, because `wg0` comes up perfectly happily with a
peer that never answers.

`lnvps-node dataplane show|apply|observe` exposes the same thing to an operator;
`observe` deliberately needs no credential, because "what does this machine
actually have?" is the question asked when something is already broken.
… is proved

Three changes to the node's data plane, and the test that made them necessary.

**Netlink instead of `ip`.** The daemon now talks to the kernel directly:
`rtnetlink` for links, addresses and routes, WireGuard's own netlink interface
for the tunnel, and `/proc/sys` for the forwarding knobs. `ip` is a program that
formats netlink messages and formats the replies back into text for us to parse
— going direct drops a dependency on iproute2's presence and version, drops the
output parsing that changes between releases, and turns "a line of English on
stderr" into kernel error codes.

**The data plane lives in its own network namespace.** The first cut configured
the *machine's* network: it replaced the operator's default route and turned on
forwarding machine-wide, on hardware that is frequently not only an LNVPS node.
Now `wg0` and `br-lnvps` live in an `lnvps` namespace:

- their default route stays theirs, and the forwarding and proxy-ARP knobs are
  ours alone;
- guests cannot reach the operator's network — not because a rule forbids it,
  which can be mis-ordered or flushed, but because no interface leads there;
- a tunnel that is down means no path at all, instead of customer traffic
  leaking out the operator's uplink sourced from LNVPS addresses, which looks
  like spoofing to their upstream and can get *their* connection null-routed.

`wg0` is created in the machine's namespace and then moved, because a WireGuard
interface keeps its UDP socket where it was created: the encrypted outer traffic
still leaves by the operator's uplink while everything carried over the tunnel
is isolated.

**The bridge name is no longer sent to the node.** Both sides hold it as a
constant. The daemon needs the name before it has ever spoken to LNVPS —
`dataplane observe` takes no credential — so a document that could name a
different bridge would leave the node holding two answers to one question.

**And a harness that sends real packets.** `lnvps_e2e/tests/tunnel_netns.rs`
builds both ends out of network namespaces: the route server configured through
its real code path, the node through its real netlink calls, a guest on the
bridge behind it. It pings the node over the tunnel and then the guest through
it, which is the path a customer's traffic takes.

It earned its place immediately by finding four things no unit test could:

- the namespace was pinned from `/proc/self/ns/net`, which in a multi-threaded
  process is the *process's* namespace — so every "isolated" interface was
  silently landing in the operator's own network;
- WireGuard's netlink calls ran outside the namespace the interface had been
  moved into, reporting "no such device" about an interface that plainly existed;
- giving an interface an address makes the kernel write entries in the *local*
  routing table, which the node then tried to delete as strays;
- **the route server never routed its pool's own block.** An address on a
  point-to-point interface does not route the rest of its prefix, so a route
  server holding `10.66.0.1/16` answered "network is unreachable" for every node
  in the pool. That is merged 4b code doing exactly what it was written to do.

The route server's command transport is now injectable so the harness can run
its real commands in a namespace rather than over SSH; the orchestration on the
node sits behind a `NetOps` trait so what the node *decides* is testable without
root, with the kernel implementation proved by the harness.
@v0l

v0l commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Second commit, covering three of your points at once — and the test that made them necessary.

An end-to-end harness that sends real packets

lnvps_e2e/tests/tunnel_netns.rs builds both ends in network namespaces — route server, node, and a guest on the bridge behind it — and pings across the tunnel, then through it to the guest. Both ends run production code: the route server through LinuxSshRouter with its command transport pointed at ip netns exec instead of SSH, the node through its real netlink calls. sudo ./scripts/tunnel-e2e.sh.

It found four bugs on its first run, none of which a unit test could have:

  1. The namespace was pinned from /proc/self/ns/net — in a multi-threaded process that is the process's namespace, not the calling thread's, so every "isolated" interface was silently landing in the operator's own network.
  2. WireGuard's netlink calls ran outside the namespace the interface had been moved into, reporting "no such device" about an interface that plainly existed.
  3. Giving an interface an address makes the kernel write entries in the local routing table; the node treated them as strays and tried to delete them.
  4. The route server never routed its pool's own block — merged 4b code. An address on a point-to-point interface does not route the rest of its prefix, so a route server holding 10.66.0.1/16 answers "network is unreachable" for every node in that pool. Peers configured perfectly, and unreachable.

Netlink instead of ip

rtnetlink for links/addresses/routes, WireGuard's netlink interface for the tunnel, /proc/sys for the forwarding knobs. No dependency on iproute2's presence or version, no output parsing that shifts between releases, kernel error codes instead of English on stderr.

A network namespace for the data plane

You were right that this is worth doing, and the previous cut was worse than "not isolated": it replaced the operator's default route and turned on forwarding machine-wide, on hardware that is often not only an LNVPS node. Now wg0 and br-lnvps live in an lnvps namespace:

  • their default route stays theirs; forwarding and proxy-ARP are ours alone;
  • guests cannot reach the operator's network, because no interface leads there — not because a rule says so (4c2 shrinks accordingly);
  • tunnel down means no path at all, rather than customer traffic leaking out their uplink sourced from LNVPS addresses, which looks like spoofing to their upstream.

wg0 is created in the machine's namespace and moved, since a WireGuard interface keeps its UDP socket where it was created — that is what keeps the encrypted outer traffic on the operator's uplink.

Worth flagging for increment 5: a guest's tap now has to be created in (or moved into) that namespace, i.e. the VM backend runs with NetworkNamespacePath=/run/netns/lnvps.

The bridge name is no longer sent

Both sides hold it as a constant. The daemon needs the name before it has ever spoken to LNVPS (dataplane observe takes no credential), so a document that could name a different one leaves the node holding two answers. The harness asserts the two constants agree.

Workspace suite green; harness green (2 passed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant