Kazoo reuses stale DNS results across connection attempts instead of re-resolving hostnames

[kosa-as]

Expected Behavior

When a connection attempt to an IP address fails, Kazoo should re-resolve the corresponding hostname before starting the next connection attempt.

This follows the behavior proposed in ZOOKEEPER-2184: ZooKeeper Client should re-resolve hosts when connection attempts fail

For example, if the initial DNS result is:

zk.example.com -> [10.0.0.1, 10.0.0.2]

and DNS changes to:

zk.example.com -> [10.0.1.1]

after the connection to 10.0.0.1 fails, Kazoo should re-resolve zk.example.com and discover 10.0.1.1 before attempting another connection.

Actual Behavior

Kazoo resolves all configured hostnames once at the beginning of _connect_loop() and stores the results in host_ports:

host_ports = self._expand_client_hosts()

It then attempts every address from this fixed snapshot:

for host, hostip, port in host_ports:
    status = self._connect_attempt(host, hostip, port, retry)

The hostname is resolved again only after all addresses in host_ports have failed and _connect_loop() is retried.

Relevant code:

https://github.com/python-zk/kazoo/blob/master/kazoo/protocol/connection.py

Therefore, Kazoo does eventually re-resolve DNS, but it does not follow the connection-failure re-resolution semantics proposed by ZOOKEEPER-2184. It first exhausts every address from the previous DNS snapshot.

In cloud and container environments, service IPs may change frequently. Continuing to attempt stale addresses can delay reconnection, consume the ZooKeeper session timeout, cause session expiration, or create connection-drift risk if an old IP is reassigned to another instance or workload.

Snippet to Reproduce the Problem

Configure zk-dns-test.example.com to initially resolve to multiple unreachable addresses:

zk-dns-test.example.com -> [192.0.2.1, 192.0.2.2]

Run the following client with DEBUG logging enabled:

import logging

from kazoo.client import KazooClient


logging.basicConfig(level=logging.DEBUG)

zk = KazooClient(
    hosts="zk-dns-test.example.com:2181",
    timeout=20.0,
)

try:
    zk.start(timeout=60)
finally:
    zk.stop()
    zk.close()

After Kazoo starts connecting to 192.0.2.1, update the DNS record so that the hostname resolves to a reachable ZooKeeper server:

zk-dns-test.example.com -> [<reachable-zookeeper-ip>]

Kazoo will still attempt 192.0.2.2 from the old DNS snapshot before resolving the hostname again.

Specifications

• Kazoo version:2.11.0