Slow/hung DoH requests after a WAN flap until the process is restarted — proposing in-process recovery (netlink self-heal and/or SIGUSR1)

[stangri]

@aarond10 @karl82 @baranyaib90 -- I'd appreciate your feedback on below.

Summary

On routers (I maintain the OpenWrt https-dns-proxy package), when the WAN link goes down and comes back up, https_dns_proxy keeps trying to reuse the TLS/TCP connections it had cached before the flap. Those connections are now dead (stale route / changed source address), so subsequent DoH queries stall for a very long time — effectively until the daemon is restarted. Today the OpenWrt package works around this by fully restarting the daemon on every WAN change, which is heavier than it should be.

The reset logic to recover from this already exists in the codebase (https_client_reset()); it just isn't triggered promptly or reliably when the network changes. I'd like to fix that in-process and would appreciate maintainer feedback on which of the two approaches below (or both) you'd prefer before I open a PR.

Environment

• Platform: Linux (OpenWrt), also relevant to any Linux host behind a flapping WAN
• Trigger: WAN interface down→up, DHCP lease change, PPPoE reconnect, or source-IP change

Root cause

The daemon relies on connection reuse / HTTP‑2 multiplexing (as it should — that's the point of a long‑lived DoH proxy). After a WAN flap, libcurl's connection pool still holds the pre‑flap connections. They're half‑open: bound to a route/source address that no longer works. On the next query curl reuses one and the write sits in kernel TCP retransmit backoff (tcp_retries2, minutes) before erroring out. That's the "requests take forever" symptom.

The c‑ares bootstrap resolver uses fixed IPs, so it mostly keeps working — the hang is essentially all in the curl connection pool.

The recovery already exists, but only fires reactively

https_client_reset() rebuilds the whole curl multi handle, which empties the connection pool — exactly what's needed after a flap. Its own doc comment (https_client.h#L58-L60) already describes this scenario:

// https_client.h
// Used to reset state of libcurl because streaming connections + IP changes
// seem to cause curl to flip out.
void https_client_reset(https_client_t *c);

Today it's only reached via a reactive timeout path. A request must first hit CURLE_OPERATION_TIMEDOUT, which arms reset_timer for conn_loss_time (default 15s) — https_client.c#L357-L362:

// https_client.c — https_fetch_ctx_process_response()
case CURLE_OPERATION_TIMEDOUT:
  if (!ev_is_active(&client->reset_timer)) {
    ILOG_REQ("Client reset timer started");
    ev_timer_start(client->loop, &client->reset_timer);
  }
  __attribute__((fallthrough));

So the best case is roughly CURLOPT_TIMEOUT (5s while connections are active, https_client.c#L313) + conn_loss_time (15s) ≈ 20s of dead queries before recovery — and it's fragile, because the reset timer is cancelled as soon as the connection count drops to zero, https_client.c#L133-L136:

// https_client.c — closesocket_callback()
if (client->connections <= 0 && ev_is_active(&client->reset_timer)) {
  ILOG("Client reset timer cancelled, since all connection closed");
  ev_timer_stop(client->loop, &client->reset_timer);
}

In practice, on a real WAN flap this often doesn't converge and users have to restart the daemon.

Proposal

Trigger the existing https_client_reset() proactively, the moment the network changes, instead of waiting for timeouts. Both approaches below reuse the same reset path plus a small helper to force an immediate re‑resolution of the resolver hostname; neither reimplements the reset itself.

Helper to kick the poller (new function in dns_poller.c / .h):

void dns_poller_trigger(dns_poller_t *d) {
  ares_cancel(d->ares);            // drop zombie queries, recreate ares sockets
  d->request_ongoing = 0;
  ev_timer_stop(d->loop, &d->timer);
  ev_timer_set(&d->timer, 0, 0);   // fire ASAP
  ev_timer_start(d->loop, &d->timer);
}

Approach A — netlink self-heal (Linux), no external trigger

Open an AF_NETLINK/NETLINK_ROUTE socket subscribed to address/route/link changes, wrap it in an ev_io on the existing loop, debounce the flap burst, and call the reset. This makes the daemon recover on its own on any Linux host — no init-system or package glue required. Guarded with #ifdef __linux__ so non‑Linux builds are unaffected and keep today's behavior.

// sketch
int fd = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE);
struct sockaddr_nl sa = { .nl_family = AF_NETLINK,
    .nl_groups = RTMGRP_IPV4_IFADDR | RTMGRP_IPV6_IFADDR |
                 RTMGRP_IPV4_ROUTE  | RTMGRP_IPV6_ROUTE | RTMGRP_LINK };
bind(fd, (struct sockaddr *)&sa, sizeof(sa));
// ev_io on fd -> on relevant RTM_* message, debounce ~1s, then:
//   https_client_reset(hc);
//   dns_poller_trigger(dp);

Pros: self-contained, fixes every Linux user with zero configuration, catches changes that don't produce a clean ifdown/ifup (source-IP change, route flip). Cons: ~80–100 lines of rtnetlink parsing; Linux-only (needs the #ifdef fence); no BSD/macOS equivalent here (would be a PF_ROUTE socket).

Approach B — SIGUSR1 handler (tiny, externally triggered)

Add an ev_signal for SIGUSR1 that performs the same reset. (SIGUSR2 is already used for the flight-recorder dump; SIGUSR1 is free.) An external supervisor — e.g. the OpenWrt hotplug/procd trigger I already maintain — sends kill -USR1 on WAN up, replacing the current full restart. This slots in next to the existing signal handlers in main.c.

typedef struct { https_client_t *hc; dns_poller_t *dp; uint8_t poll; } reset_ctx_t;

static void signal_reset_cb(struct ev_loop __attribute__((unused)) *loop,
                            ev_signal *w, int __attribute__((unused)) revents) {
  reset_ctx_t *c = (reset_ctx_t *)w->data;
  ILOG("SIGUSR1: network change, resetting HTTPS client");
  https_client_reset(c->hc);          // empties curl connection pool
  if (c->poll) dns_poller_trigger(c->dp);
}

// in main(), after dns_poller_init:
reset_ctx_t reset_ctx = { &https_client, &dns_poller, using_dns_poller };
ev_signal sigusr1;
ev_signal_init(&sigusr1, signal_reset_cb, SIGUSR1);
sigusr1.data = &reset_ctx;
ev_signal_start(loop, &sigusr1);

Pros: ~25 lines, portable, compiles everywhere, and doubles as a manual/test trigger. Cons: only half a fix — something external must send the signal, so on its own it doesn't help users without that glue.

Optional, complementary

Enable kernel TCP keepalive in https_fetch_ctx_init() so dead connections are detected faster even if a reset is ever missed:

// https_fetch_ctx_init()
ASSERT_CURL_EASY_SETOPT(ctx, CURLOPT_TCP_KEEPALIVE, 1L);
ASSERT_CURL_EASY_SETOPT(ctx, CURLOPT_TCP_KEEPIDLE, 30L);
ASSERT_CURL_EASY_SETOPT(ctx, CURLOPT_TCP_KEEPINTVL, 10L);

(Deliberately not proposing CURLOPT_FORBID_REUSE/CURLOPT_FRESH_CONNECT — that forces a full TLS handshake per query and defeats the connection reuse this daemon depends on.)

Questions for maintainers

1. Would you prefer A (netlink self-heal) as the general fix, B (SIGUSR1) as a lightweight trigger, or both (netlink as the universal self-heal, SIGUSR1 as a manual/test hook)?
2. For netlink, is an #ifdef __linux__ fence acceptable, or would you rather see a small portable abstraction (e.g. PF_ROUTE on BSD)?
3. Any preference on debounce duration and on whether the reset should also revisit conn_loss_time / CURLOPT_MAXAGE_CONN handling?

Happy to open a PR for whichever direction you prefer — I have working sketches for both.