Firebird Client Deadlock #9214

Title: Client (4.0+) calls LoadLibrary for missing key-holder plugin “NoDefault” on every attach while holding a global mutex, which can deadlock with the Windows loader lock

Environment

Client: fbclient.dll 5.0.2.1613, 32-bit (also present in 5.0.4.1812 and current master source)
Server: Firebird 5.0.4.1812 SuperServer, same host
OS: Windows Server 2016 (10.0.14393), 32-bit application under WOW64
fbclient.dll loaded from C:\Windows\SysWOW64, with no firebird.conf or plugins folder beside it
The same application with fbclient.dll 3.0.2.32703 has no problem: 13+ days of uptime under the same workload pattern.

Summary

A multi-threaded application deadlocked permanently inside fbclient.dll. Two full-memory dumps taken 78 seconds apart show identical state. Two threads are involved:

Thread A is attaching to a database. It is inside PluginSet::next(), holding loadModuleMutex, and blocked in LoadLibraryExA("C:\Windows\SYSTEM32\plugins\NoDefault") waiting for the OS loader lock:
ntdll!NtWaitForSingleObject
ntdll!LdrpDrainWorkQueue
ntdll!LdrpLoadDllInternal
ntdll!LdrpLoadDll
ntdll!LdrLoadDll
KERNELBASE!LoadLibraryExW
KERNELBASE!LoadLibraryExA <- "C:\Windows\SYSTEM32\plugins\NoDefault"
FBCLIENT!...
Thread B holds the loader lock because it is inside LoadLibraryW of an application DLL whose initialization calls isc_attach_database. It is blocked on a critical section in fbclient’s data section that Thread A owns. !locks reports OwningThread = Thread A.
ntdll!RtlEnterCriticalSection <- CS in FBCLIENT data section, owned by Thread A
FBCLIENT!...
FBCLIENT!isc_attach_database

ntdll!LdrpCallInitRoutine
ntdll!LdrpInitializeNode
...
ntdll!LdrLoadDll
KERNELBASE!LoadLibraryW

We accept that attaching from DLL initialization, under the loader lock, is wrong on our side, and we are removing it. We’re reporting this because the client behavior turns a latent mistake into an intermittent hard hang, and because the same lock-plus-LoadLibrary pattern could also interact badly with other loader activity in the host process.

Analysis (from source, v5.0.2)

src/remote/remote.h, ClntAuthBlock::ClientCrypt constructs GetPlugins(TYPE_KEY_HOLDER, "NoDefault"). This came in with CORE-6220 (commit 0453bbe, first released in 4.0). As a result, every remote attach looks up a key-holder plugin named NoDefault.
No such plugin is registered. PluginSet::next() (src/yvalve/PluginManager.cpp) therefore calls loadModule(), which tries ModuleLoader::loadModule() on /plugins/NoDefault and its extension variants.
A failed load isn’t cached, so this LoadLibrary attempt repeats on every attach.
Since commit 791d46b (first in 4.0), PluginSet::next() holds a static loadModuleMutex across loadModule(). Every attach therefore does LoadLibrary, which needs the loader lock, while holding a process-wide client mutex. Any thread that holds the loader lock and then enters the client deadlocks against it.

Neither change is in 3.0.x, which matches our experience that 3.0.2 doesn’t show the problem. Both are unchanged in 5.0.4 and master.

Suggestions

Skip the key-holder lookup when no KeyHolderPlugin is configured, or treat the NoDefault placeholder specially so it never reaches ModuleLoader.
Cache negative results for modules that failed to load, so a missing plugin isn’t retried with LoadLibrary on every attach.
Consider not holding loadModuleMutex across the actual LoadLibrary call, for example with a per-module “loading” marker and a condition wait.

Reproduction

We haven’t built a minimal standalone repro. The deadlock is timing-dependent; it appeared after about 18 hours in production. The ingredients are:

One thread repeatedly doing remote attaches through a 4.0+ client.
Another thread calling LoadLibrary on a DLL whose initialization calls isc_attach_database.