Skip to content

OPNET-772: Add NAT64/DNS64 support for IPv6-only clusters on IPv4-only hosts #1952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
emy wants to merge 3 commits into
base: master
Choose a base branch
from
Open

OPNET-772: Add NAT64/DNS64 support for IPv6-only clusters on IPv4-only hosts #1952

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ac8bd15
Add NAT64/DNS64 support for IPv6-only clusters on IPv4-only hosts
emy Aug 19, 2026
28a095e
NAT64: make sushy BMC cert valid for the IPv6 baremetal address
emy Aug 19, 2026
29cd165
apply PR review fixes, systemd TAYGA, unbound DNS64, sudo-safe sushy …
emy Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions 01_install_requirements.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,15 @@ case $DISTRO in
;;
"rhel10"|"centos10")
sudo dnf -y install python3-pip
if sudo subscription-manager identity > /dev/null 2>&1; then
sudo subscription-manager repos --enable "codeready-builder-for-rhel-10-$(arch)-rpms" || true
if [[ $DISTRO == "centos10" ]]; then
sudo dnf config-manager --set-enabled crb
sudo dnf -y install epel-release
elif [[ $DISTRO == "rhel10" ]]; then
if sudo subscription-manager identity > /dev/null 2>&1; then
sudo subscription-manager repos --enable "codeready-builder-for-rhel-10-$(arch)-rpms" || true
fi
# EPEL provides tayga (needed for NAT64); mirror the EL9 rhel handling.
sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm
fi
sudo ln -s /usr/bin/python3 /usr/bin/python || true
PYTHON_DEVEL="python3-devel"
Expand Down Expand Up @@ -207,6 +214,19 @@ if [[ "${NODES_PLATFORM:-}" == "baremetal" ]] ; then
sudo dnf -y install ipmitool
fi

# Install NAT64 dependencies if enabled
if [[ "${ENABLE_NAT64:-false}" == "true" ]]; then
echo "Installing NAT64 dependencies (TAYGA, unbound)..."
# TAYGA provides the NAT64 translation; unbound provides DNS64 synthesis.
# unbound is used rather than CoreDNS because its built-in dns64 module
# reliably synthesizes AAAA records (the CoreDNS dns64 plugin build did not).
sudo dnf -y install tayga unbound
Comment on lines +218 to +223

@mkowalski mkowalski Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: this fails on a clean EL10 host — tayga ships only in EPEL, which the EL10 branch never enables

centos10/rhel10 are an explicitly supported branch (line 111; the catch-all at 119-121 exits 1 for anything else), but only the EL9 branch installs EPEL (line 97 epel-release, line 106 the EPEL-9 RPM). The EL10 branch (111-118) only enables CRB via subscription-manager ... || true. This install sits after the case, so it runs on every branch. Under set -euxo pipefail (line 2) the failure aborts the whole script.

Nothing else enables EPEL first: the only other reference is agent/01_agent_requirements.sh:58 (different script, gated on AGENT_E2E_TEST_BOOT_MODE), and the pinned metal3-dev-env packages_installation role has no epel/tayga references.

Fix: enable EPEL 10 in the EL10 branch — dnf -y install epel-release (CentOS Stream) or the EPEL-10 release RPM (RHEL) — or explicitly reject NAT64 on EL10. unbound is fine; it is in BaseOS/AppStream on both.

Reproducer

Steps: Queried remote repodata with dnf repoquery --repofrompath against mirror.stream.centos.org and dl.fedoraproject.org, then reproduced end-to-end in a clean quay.io/centos/centos:stream10 container using the EL10 branch semantics (CRB on, no EPEL).

Expected: dnf -y install tayga unbound succeeds, as it does on EL9.

Actual:

Error: Unable to find a match: tayga
DNF_EXIT_CODE=1
tayga: NOT FOUND in enabled repos
unbound: FOUND
repo tayga unbound
cs10-BaseOS / AppStream / CRB ABSENT present
epel10 tayga-0.9.6-...el10_2 absent
cs9-BaseOS / AppStream / CRB ABSENT present
epel9 tayga-0.9.2-17.el9 absent

Correction to one panel claim: EPEL 10 does ship tayga, and epel-release + dnf -y install tayga unbound on EL10 returns RC=0. So enabling EPEL is a sufficient fix.

# We run our own DNS64 unbound instance on a dedicated port; make sure the
# stock unbound.service does not also grab port 53 and clash with the host
# NetworkManager resolver.
sudo systemctl disable --now unbound.service 2>/dev/null || true
fi

retry_with_timeout 5 60 "curl -L $OPENSHIFT_CLIENT_TOOLS_URL | sudo tar -U -C /usr/local/bin -xzf -"
sudo chmod +x /usr/local/bin/oc
oc version --client -o json
28 changes: 28 additions & 0 deletions 02_configure_host.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ set -euxo pipefail
source logging.sh
source common.sh
source network.sh
source nat64.sh
source utils.sh
source validation.sh
source oc_mirror.sh
Expand Down Expand Up @@ -293,6 +294,13 @@ if [ "${NUM_EXTRA_WORKERS}" -ne 0 ] || [ "${NUM_ARM_WORKERS}" -ne 0 ]; then
fi
fi

# For NAT64 (IPv6-only cluster on an IPv4-only host) the BMC emulator must be
# reached over IPv6 by the in-cluster Ironic pods; rewrite the node BMC addresses
# to the host's IPv6 baremetal address.
if [[ "${ENABLE_NAT64}" == "true" ]]; then
nat64_fixup_bmc_addresses
fi

# shellcheck disable=SC2034
ZONE="\nZONE=libvirt"

Expand Down Expand Up @@ -474,6 +482,16 @@ if [ "$EXT_IF" ]; then
sudo $IPTABLES -A FORWARD --in-interface "${BAREMETAL_NETWORK_NAME}" -j ACCEPT
fi

# When NAT64 is enabled, ensure IPv6 forwarding rules are set for the bridge
if [[ "${ENABLE_NAT64}" == "true" ]]; then
# Check-then-add so re-running does not accumulate duplicate rules; cleanup_nat64
# removes these with the matching -D commands.
sudo ip6tables -C FORWARD --in-interface "${BAREMETAL_NETWORK_NAME}" -j ACCEPT 2>/dev/null || \
sudo ip6tables -A FORWARD --in-interface "${BAREMETAL_NETWORK_NAME}" -j ACCEPT
sudo ip6tables -C FORWARD --out-interface "${BAREMETAL_NETWORK_NAME}" -m state --state RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || \
sudo ip6tables -A FORWARD --out-interface "${BAREMETAL_NETWORK_NAME}" -m state --state RELATED,ESTABLISHED -j ACCEPT
fi
Comment on lines +485 to +493

@mkowalski mkowalski Aug 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: these rules are broader than needed, duplicate on every run, and are never removed

Three separate issues, in descending order of importance:

  1. Never cleaned up. cleanup_nat64 (nat64.sh:279-326) contains zero ip6tables lines — its only netfilter teardown is the IPv4 MASQUERADE at nat64.sh:319. These rules survive make clean.
  2. Not idempotent. Unconditional -A, unlike the adjacent NAT64 rule at nat64.sh:64-65 which correctly does -C ... || -A .... I measured 3 copies of each after 3 runs; combined with (1) they accumulate without bound.
  3. Over-broad. They accept all forwarded IPv6 on the bridge rather than cluster-subnet ↔ NAT64-prefix via the TUN.

I want to be precise about severity, because this was initially filed as a security bypass and that framing did not hold up. I tested it four ways: the ip6 FORWARD policy is already accept on a stock host; -A cannot override an earlier DROP; firewalld filters in a separate nft base chain at priority filter + 10 so an accept here does not bypass it; and on a real dev-scripts host these rules sit behind LIBVIRT_FWO — which already ends with -s <bm v6 subnet> -i ostestbm -j ACCEPT — and matched 0 packets. So this is code hygiene, not a reachability change. Also note lines 481-482 already do the same thing for IPv4, so the breadth follows existing repo convention.

Fix: ip6tables -C ... || ip6tables -A ... scoped to -i ${BAREMETAL_NETWORK_NAME} -o ${NAT64_TUN_INTERFACE} -s ${EXTERNAL_SUBNET_V6} -d ${NAT64_PREFIX} plus the reverse established rule, with matching -D calls in cleanup_nat64. Dropping 2>/dev/null || true would also stop hiding real failures.


# Switch NetworkManager to internal DNS
if [ "$MANAGE_BR_BRIDGE" == "y" ]; then
switch_to_internal_dns
Expand Down Expand Up @@ -536,6 +554,16 @@ fi
sudo virsh net-list | grep "${PROVISIONING_NETWORK_NAME}" || sudo virsh net-start "${PROVISIONING_NETWORK_NAME}"
sudo virsh net-list | grep "${BAREMETAL_NETWORK_NAME}" || sudo virsh net-start "${BAREMETAL_NETWORK_NAME}"

# Configure NAT64/DNS64 if enabled
if [[ "${ENABLE_NAT64}" == "true" ]]; then
configure_nat64_bridge_ipv6
configure_tayga
configure_dns64
# Make the sushy BMC cert valid for the IPv6 baremetal address so the
# IPv6-only in-cluster Ironic can reach the BMC. Must run before step 05
# embeds this cert into the install-config trust bundle.
nat64_fixup_sushy_cert
fi

# Setup a single nfs export for image registry
if [ "${PERSISTENT_IMAGEREG}" == true ] ; then
Expand Down
27 changes: 27 additions & 0 deletions common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -632,6 +632,33 @@ if [[ ! -z ${AGENT_E2E_TEST_SCENARIO} ]]; then
fi
fi

# Validate NAT64 configuration. This must run after the agent scenario parsing
# above, which derives IP_STACK from AGENT_E2E_TEST_SCENARIO; validating earlier
# would only ever see the default IP_STACK and silently pass for v4/dual-stack
# agent scenarios.
if [[ "${ENABLE_NAT64:-false}" == "true" ]]; then
if [[ "${IP_STACK:-v6}" != "v6" ]]; then
error "ENABLE_NAT64=true requires IP_STACK=v6 (got IP_STACK=${IP_STACK:-v6})"
exit 1
fi
if [[ "${HOST_IP_STACK:-${IP_STACK:-v6}}" != "v4" ]]; then
error "ENABLE_NAT64=true requires HOST_IP_STACK=v4 (got HOST_IP_STACK=${HOST_IP_STACK:-${IP_STACK:-v6}})"
exit 1
fi
# The IPv6-only in-cluster Ironic can only reach the BMC emulator over IPv6,
# which only sushy/redfish listens on; vbmc (ipmi) binds IPv4 only. Reject the
# ipmi and mixed drivers rather than silently leaving those BMCs unreachable.
if [[ "${BMC_DRIVER}" != "redfish" && "${BMC_DRIVER}" != "redfish-virtualmedia" ]]; then
error "ENABLE_NAT64=true requires a redfish-based BMC_DRIVER (redfish or redfish-virtualmedia), got BMC_DRIVER=${BMC_DRIVER}"
exit 1
fi
# NAT64 provides external registry access, so image mirroring is skipped and no
# local registry is created. Default MIRROR_IMAGES to false here (before the
# registry-override decision later in this file) so the installer is not pointed
# at a registry that was never stood up.
export MIRROR_IMAGES=${MIRROR_IMAGES:-false}
fi

if [[ ! -z ${AGENT_E2E_TEST_BOOT_MODE} ]]; then
case "$AGENT_E2E_TEST_BOOT_MODE" in
"ISO" | "PXE" | "DISKIMAGE" | "ISCSI"| "ISO_NO_REGISTRY")
Expand Down
51 changes: 51 additions & 0 deletions config_example.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,6 +441,57 @@ set -x
#export EXTERNAL_SUBNET_V4="192.168.111.0/24"
#export EXTERNAL_SUBNET_V6="fd2e:6f44:5dd8:c956::/120"

# ENABLE_NAT64 -
# Enable NAT64/DNS64 to allow IPv6-only clusters (IP_STACK=v6) to run on
# IPv4-only hosts (HOST_IP_STACK=v4). Uses TAYGA for NAT64 translation and
# unbound for DNS64 synthesis, enabling cluster VMs to reach external IPv4
# resources (e.g. container registries) via synthesized IPv6 addresses.
# Requires: IP_STACK=v6, HOST_IP_STACK=v4, and a redfish-based BMC_DRIVER
# (redfish or redfish-virtualmedia) since the IPv6-only in-cluster Ironic can
# only reach the BMC emulator over IPv6 (vbmc/ipmi listens on IPv4 only).
# Default: false
#
#export ENABLE_NAT64=true

# NAT64_PREFIX -
# IPv6 prefix used for NAT64 address translation. Packets sent to addresses
# in this prefix are translated to IPv4 by TAYGA.
# Default: "64:ff9b::/96"
# Note: The well-known prefix 64:ff9b::/96 cannot reach RFC1918 addresses.
# Use a ULA prefix (e.g. "fd00:64::/96") if you need to reach private IPv4.
#
#export NAT64_PREFIX="64:ff9b::/96"

# NAT64_V4_POOL -
# IPv4 address pool used by TAYGA for dynamic NAT64 mappings.
# Default: "192.168.255.0/24"
#
#export NAT64_V4_POOL="192.168.255.0/24"

# NAT64_V4_ADDR -
# TAYGA's own IPv4 address on the NAT64 TUN interface.
# Default: "192.168.255.1"
#
#export NAT64_V4_ADDR="192.168.255.1"

# NAT64_V6_ADDR -
# TAYGA's own IPv6 address (source of the ICMPv6 errors it generates, e.g.
# for PMTUD). Defaults to the 3rd address of EXTERNAL_SUBNET_V6, which is
# on-link on the baremetal bridge. If you carve TAYGA its own subnet, set this
# to an address outside any locally-attached subnet and route it to the NAT64
# TUN device so error replies are delivered symmetrically.
# Default: nth_ip(EXTERNAL_SUBNET_V6, 3)
#
#export NAT64_V6_ADDR="fd2e:6f44:5dd8:c956::3"

# NAT64_DNS64_UPSTREAM -
# Space-separated upstream resolver(s) the DNS64 unbound instance forwards to.
# By default these are auto-discovered from the host's resolver config; set this
# explicitly when the host has no usable upstream in resolv.conf (otherwise DNS64
# falls back to 8.8.8.8, which is usually unreachable on a firewalled NAT64 host).
#
#export NAT64_DNS64_UPSTREAM="10.0.0.53"

# ENABLE_BOOTSTRAP_STATIC_IP -
# Configure a static IP for the bootstrap VM external NIC
# (Currently this just expects a non-empty value, the IP is fixed to .9)
Expand Down
8 changes: 8 additions & 0 deletions host_cleanup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ set -x

source logging.sh
source common.sh
source network.sh
source nat64.sh
source utils.sh
source validation.sh

Expand Down Expand Up @@ -32,6 +34,12 @@ ansible-playbook \
-i "${VM_SETUP_PATH}/inventory.ini" \
-b -vvv "${VM_SETUP_PATH}/teardown-playbook.yml"

# Always clean up NAT64/DNS64. cleanup_nat64 is fully idempotent (every step is a
# no-op when nothing is present), so run it unconditionally rather than gating on
# the current ENABLE_NAT64 value: otherwise unsetting ENABLE_NAT64 before teardown
# would strand the unbound service, dnsmasq drop-in, TUN device, routes and rules.
cleanup_nat64

sudo rm -rf "/etc/NetworkManager/dnsmasq.d/openshift-${CLUSTER_NAME}.conf" /etc/yum.repos.d/delorean*
sudo rm -rf /etc/NetworkManager/conf.d/dnsmasq.conf
sudo rm -rf /etc/NetworkManager/dnsmasq.d/upstream.conf
Expand Down
Loading