[multicast] DDM multicast exchange: V4 protocol, MRIB sync by zeeshanlakhani · Pull Request #696 · oxidecomputer/maghemite

zeeshanlakhani · 2026-04-02T11:42:34Z

Adds multicast group subscription distribution to the DDM exchange protocol with a V4 version.

Key changes/additions:

V4 exchange protocol with multicast support, negotiated via an MCAST_CAPABLE discovery flag (RFC 5492 style). The wire version byte stays at the V2 floor, so V2/V3 peers are unaffected and mixed-version topologies interoperate cleanly.
Two validated newtypes in the recently merged client-common sub-crate (the cycle-free leaf crate, so the API-types crates Omicron consumes avoid an omicron_common cycle):
- OverlayMulticast (announced group is multicast)
- UnderlayMulticastIpv6 (admin-local mapping within ff04::/64)
  Both validate at construction and on deserialization, rejecting invalid origins at every ingress.
Membership half/lower: ddmd programs DPD replication-group members in-process (ddm::mcast) from imported routes plus the peer table via a triggered sweep with periodic reconciliation. This matches how ddmd installs imported unicast underlay routes (sys::add_underlay_routes).
Periodic exchange resync in the peer state machine to repair drift from missed pulls
Learned multicast state is exposed via DDM admin API for Omicron consumption
update_imported_mcast is atomic within the Db (single lock for import/delete/diff, which differs from the tunnel work)
Collapsed send_update dispatch, with exchange responses now checked for HTTP success (require_success, unicast included)
Shared pull handler functions (collect_underlay_tunnel, collect_multicast), and a shared tfport name parser in mg-common (port_link_from_ifname), fixing link derivations
Admin API binds synchronously and consistently on ::, with an --api-only flag (admin API without state machines, analogous to mgd --no-bgp-dispatcher) for test fixtures and the Linux build.
More serde round-trip and validation tests, a version negotiation contract test pinning mixed-version behavior, plus trio coverage of multicast advertise, router restart, and withdraw.
CI: softnpu/sidecar-lite/dendrite pins bumped to multicast-capable builds and the trio/quartet jobs have been moved to helios-3.0 to match dendrite's build target. The test harness now fails fast if dendrite or tfport lands in SMF maintenance instead of timing out on peering.

References

Stacked on zl/mrib (MRIB: Multicast RIB implementation, #675).

The tfport datalink naming logic, parsing names like tfportrear0_0 and tfportqsfp0_0 into a kind, port, and link, was duplicated in ddm's sys module and mg-lower's dendrite module. Hoist it into a single mg_common::tfport module so both consumers share one parser and one set of error types. No behavior change.

…rite mod The call sites that resolve a tfport interface name to a DPD (PortId, LinkId) tuple repeated the same inline parse-and-construct sequence. This work factors it into a single `port_link_from_ifname` helper function so that the resolution lives in one place. Note: no behavior change.

…ted routes This work splits DDM underlay multicast (RFD 488) into two directional halves. Multicast origination advertises locally originated MRIB groups outward (MRIB to the DDM admin API to the underlay). Membership, in turn, consumes the routes other sleds originate and programs the local switch's replication members in DPD directly. This commit implements the membership half and refactors origination to match the unicast lower-half's shape and approach. Membership updates run in `ddmd` because both of its inputs (the set of DDM-imported multicast routes and the peer table) are already owned in-process by the DDM state machine. A dedicated sweep thread (`ddm::mcast::run`) reconciles each affected underlay group's DPD members. This mechanism is trigger-driven over an mpsc channel from the import, withdraw, peer-expiry, and peer-link-resolution paths, with a 10s backstop and a startup seed from DPD's member-bearing groups so orphans left by a withdraw during downtime are properly drained. Each DPD member GET/PUT is bounded at 3s (DPD_REQUEST_TIMEOUT, kept so a group's fetch-write pair stays under the chosen backstop interval) and stalls are classified as typed `TimedOut` outcomes. Therefore, an unresponsive DPD retains the group for the next pass rather than dropping it and leaking any replication state. mg-lower's mrib module is now origination-only, watch-driven with a 1s periodic resync backstop, mirroring the unicast lower-half loop. The DPD and DDM clients take an optional address so integration harnesses can target a listener on a dynamically assigned port, exposed as the --dendrite-addr and --ddm-addr mgd flags respectively. Supporting changes attached to this work: - db: factor the imported-route tree scan into scan_origin_tree and return removed unicast and multicast next hops together via `RemovedNexthopRoutes`. - ddmd: wire the sweep's notify channel and start it unconditionally, including --api-only, where it owns the receiver with a known empty peer set. - admin: collect get_peers via filter_map rather than a manual loop.

Merge main and land DDM underlay multicast support, where DPD is updated directly from maghemite. Here, discovery gains a capability-negotiated V4, as nodes still advertise the V2 floor in the version byte, and (now) signal support with a new `MCAST_CAPABLE` flag, so deployed V2/V3 peers ignore the unknown bit (RFC 5492) while two capable nodes negotiate up to V4. The multicast wire types land in a frozen `ddm-protocol::v4` module post-merge, with validated counterparts in API version 3 (MULTICAST_SUPPORT). The correct-by-construction newtypes are consolidated into the cycle-free `client-common` leaf, including `Vni`, `UnderlayMulticastIpv6`, and `OverlayMulticast`.

Reword the peer-context and --api-only doc comments across the admin handler context, the multicast sweep, the discovery module, and ddmd startup to state that the peer set is empty when no state machines run. Start the admin Dropshot server synchronously so the API is bound before handler() returns, and log the bound address, which reflects the assigned port when 0 is requested.

…lict mapping, port-file admin binding

…e resync The membership sweep moves from a dedicated thread with a std mpsc channel to a task on the daemon's runtime with a bounded tokio channel. Triggers are wake hints only, so a try_send dropped on a full channel costs at most one reconcile interval, and the sweep selects between a trigger and a periodic tick that defers rather than bursts after a long pass. Exchange state gains a periodic resync pull. The initial pull is one-shot, so routes a neighbor originates after we pull it, late multicast group memberships in particular, would otherwise never be imported absent a push from that neighbor. Periodic pulls import without redistributing, and each router runs its own resync, so a transit re-flooding its peers on every periodic pull would churn in steady state for no benefit. The smf manifest default for admin_host moves from ::1 to :: so the admin API is reachable over the underlay rather than only from the local host, matching mgd's existing default.

DDM V4 multicast state is not bounded by the fabric's prefix and tunnel counts, so a complete exchange set cannot be assumed to fit into a single HTTP body. Multicast pushes to V4 peers are now sized by serialized bytes and split into batches. V4 pulls are served with keyset pagination ordered by multicast group identity, keeping each group's path vectors together. Underlay and tunnel sections ride on the first page only, and multicast withdrawal reconciliation runs after the final page, since a partial snapshot would read every unvisited group as withdrawn. The V4 `PullResponse` gains a continuation token, minted and parsed with Dropshot's token codec while keeping our own response envelope. Peer tokens are validated before URI construction, responses are read under a 10 MiB bound, and a pull stops after 64 pages. The same bound replaces Dropshot's 1 KiB `default_request_body_max_bytes`, which batched V4 pushes would otherwise exceed. Pulls run on a spawned thread rather than the state-machine loop, with at most one active pull per peer. Deterministic size errors leave healthy sessions alive, while transport and protocol failures retain peer expiry behavior. V2 and V3 exchange remains whole-snapshot. Neither carries multicast, so neither batches nor pages.

…st types

zeeshanlakhani force-pushed the zl/ddm-mcast branch 3 times, most recently from 339f250 to 1b5996d Compare April 2, 2026 11:49

zeeshanlakhani changed the title ~~ddm-mcast~~ [multicast] DDM multicast exchange: V4 protocol, MRIB sync, M2P hooks Apr 2, 2026

zeeshanlakhani force-pushed the zl/ddm-mcast branch from 1b5996d to 0671b1f Compare April 2, 2026 15:38

zeeshanlakhani requested review from rcgoodfellow and taspelund April 2, 2026 15:43

zeeshanlakhani marked this pull request as ready for review April 2, 2026 15:43