When port_settings_apply is run, it grabs a copy of dpd's internal state for each port/link, diffs it against the state requested by the API client, and then reconciles the difference.
The problem we have here is that omicron is the main (practically only) consumer of port_settings_apply, and it doesn't include any link-local IPv6 addresses in its request... since these are handled by tfportd and illumos.
tfportd learns about IPv6 link-local addresses in illumos from its reconciler, and synchronizes them into dpd via link_ipv6_create.
Both port_settings_apply and link_ipv6_create modify the same internal state: Link.ipv6
#[derive(Debug)]
pub struct Link {
pub port_id: PortId,
pub link_id: LinkId,
pub port_hdl: PortHdl,
pub asic_port_id: AsicId,
pub updated: i64,
pub presence: bool,
pub ipv6_enabled: bool,
pub tx_eq: Option<TxEq>,
pub fsm_state: asic::PortFsmState,
pub link_state: LinkState,
pub media: PortMedia,
/// A list of IPv4 addresses assigned to this link.
pub ipv4: BTreeSet<Ipv4Entry>,
/// A list of IPv6 addresses assigned to this link.
pub ipv6: BTreeSet<Ipv6Entry>, <<<<<<<<<<<
pub linkup_tracker: LinkUpTracker,
pub autoneg_tracker: AutonegTracker,
/// Configuration requested by the user/sled-agent.
pub config: LinkConfig,
/// State that actually exists in the ASIC layer.
pub(crate) plumbed: LinkPlumbed,
}
When port_settings_apply is run, the internal dpd state includes all items in Link.ipv6 (including the link-locals added by tfportd) and thus the diff always indicates that the link-local is old state that should be removed.
This results in the omicron reconciler (via port_settings_apply) and the tfport reconciler (via link_ipv6_create) battling each other and the IPv6 link-local addresses being removed from the ASIC anytime omicron's reconciler runs after the tfport reconciler... which means that periodically, the ASIC will stop punting traffic destined for that link-local address. This would affect NDP, BGP unnumbered, etc. and could potentially cause service disruptions if the blackholing were to last long enough.
When
port_settings_applyis run, it grabs a copy of dpd's internal state for each port/link, diffs it against the state requested by the API client, and then reconciles the difference.The problem we have here is that omicron is the main (practically only) consumer of
port_settings_apply, and it doesn't include any link-local IPv6 addresses in its request... since these are handled by tfportd and illumos.tfportd learns about IPv6 link-local addresses in illumos from its reconciler, and synchronizes them into dpd via
link_ipv6_create.Both
port_settings_applyandlink_ipv6_createmodify the same internal state:Link.ipv6When
port_settings_applyis run, the internal dpd state includes all items inLink.ipv6(including the link-locals added by tfportd) and thus the diff always indicates that the link-local is old state that should be removed.This results in the omicron reconciler (via
port_settings_apply) and the tfport reconciler (vialink_ipv6_create) battling each other and the IPv6 link-local addresses being removed from the ASIC anytime omicron's reconciler runs after the tfport reconciler... which means that periodically, the ASIC will stop punting traffic destined for that link-local address. This would affect NDP, BGP unnumbered, etc. and could potentially cause service disruptions if the blackholing were to last long enough.