tls_mgm: match SNI / SIP domain filters case-insensitively

[kvwake]

Summary

Make the matching of the TLS SNI (Server Name Indication) hostname against the configured SIP DOMAIN FILTERS case-insensitive, as required by RFC 6066 for DNS hostnames.

Details

This is a bug fix.

When tls_mgm selects a server TLS domain based on the SNI hostname from the TLS ClientHello, it matches that hostname against the configured match_sip_domain filters in tls_find_domain_by_filters()
(modules/tls_mgm/tls_domain.c). The match was done with:

fnmatch(dom_array->arr[i].hostname->s.s, fnm_s, 0)

fnmatch() is case-sensitive unless FNM_CASEFOLD is passed, and the SNI hostname is taken verbatim from the ClientHello (via SSL_get_servername() in tls_openssl, forwarded unchanged through tls_sni_cb()), with no
case normalization on either side.

RFC 6066 §3 defines server_name as a DNS hostname, and DNS hostname comparison is case-insensitive (RFC 4343). The consequence of the current behavior is that a client presenting SNI Example.COM fails to match a
filter configured as example.com, making it effectively impossible to reliably match TLS certificates with SIP domain filters whenever the casing differs. This affects both the tls_openssl and tls_wolfssl backends,
since the matching code is shared in tls_mgm.

Solution

Pass FNM_CASEFOLD to the fnmatch() call so the SNI hostname is matched case-insensitively. This matches the existing in-tree precedent in the identity (identity.c:1877) and sipmsgops (sipmsgops.c:435) modules,
which already use FNM_CASEFOLD for hostname/header matching.

FNM_CASEFOLD is a GNU extension only exposed by glibc when _GNU_SOURCE is defined, so _GNU_SOURCE is defined at the top of tls_domain.c before any include (this is more correct than identity.c, which defines it
after <fnmatch.h>). On BSD/macOS the flag is available by default.

The match_sip_domain documentation is updated to note the case-insensitive behavior.

I deliberately left the config-load de-duplication comparison (str_strcmp at tls_domain.c:802) case-sensitive: making it case-insensitive would silently merge filter entries that look distinct in the configuration
(e.g. foo.com and FOO.com), which is a separate behavioral decision and not required to fix this bug.

Verified by building the full tree (all modules except the un-vendored tls_wolfssl submodule) under -Werror with gcc 13 on Ubuntu 24.04, and running the unit test suite — both pass. A standalone check confirms
fnmatch("example.com", "Example.COM", FNM_CASEFOLD) matches while the old flag 0 does not.

Compatibility

No configuration migration is required. The change only makes matching more permissive (case-insensitive), so all filters that matched before continue to match; additionally, SNI hostnames that differ only in case now
match as intended. No API or database schema changes.

Closing issues

closes #3982