P0: guarantee at most one Modbus TCP connection per device endpoint, and detect session-eviction wars

[frahlg]

Incident (2026-08-29, live site)

A Sungrow hybrid on direct Ethernet accepts exactly one Modbus TCP session. Each new connect makes the inverter FIN+RST the previous session ~2 ms after the new SYN-ACK. Two clients polling the same inverter therefore evict each other on every request — an eviction war.

On the live site FTW logged modbus reconnected ~20–30/min (~17,000 lines in 14 h). tcpdump showed the eviction pattern above. Root cause was environmental — a forgotten experiment process on the same host had been holding/redialing the inverter's only session since May — but the investigation exposed that FTW itself can also open competing connections to a device it is already polling. Most inverters tolerate multiple sessions badly; this class of bug must be impossible by construction.

FTW-internal surfaces that can open a second connection to the same endpoint

1. Two configured drivers with the same host:port — Registry.Add calls ModbusFactory per driver (go/internal/drivers/registry.go:520), one modbus.Capability (one TCP client) each. Typical case: a Modbus gateway fronting several unit IDs, one logical driver per unit.
2. Driver test — handleDriverTest builds a throwaway registry with the same factory (go/internal/api/api_drivers_debug.go:280) and dials a second connection to a device the live driver is polling. On a single-session inverter, pressing Test in Settings can knock the production battery session out mid-dispatch.
3. Fingerprint — s.deps.DriverModbusFactory("__fingerprint", ...) (go/internal/api/api_drivers_fingerprint.go:181), same problem as (2).

The reprobe/reload path added in #982/#984 is safe — it reuses the driver's existing Env and capability.

Hardening

1. One shared connection per endpoint, process-wide. A package-level registry in go/internal/modbus keyed by host:port. Dial returns a Capability backed by the shared connection; every capability for the same endpoint serializes requests through the same mutex and socket, setting its own unit ID per request. Test/fingerprint then borrow the live driver's session instead of evicting it. Refcounted close: a capability Close releases its reference; the socket closes when the last user is gone.
2. Detect external eviction wars. Count reconnects per endpoint in a sliding window. Above a threshold (e.g. >6/min sustained), log a rate-limited WARN naming the likely cause ("another Modbus client keeps taking this device's only session") and surface it as a support-report Finding. Yesterday's war was invisible until someone read raw logs; it should be a named Finding.
3. Stop the log flood. modbus reconnected at INFO on every dial wrote 17k lines in 14 h into the log ring. Log the first reconnect per endpoint at INFO, then rate-limit repeats (e.g. once/min with a suppressed-count), independent of the eviction WARN.

Non-goals

FTW cannot prevent other processes from dialing the inverter (that is what happened here). Detection (2) is the answer for that case.

🤖 Generated with Claude Code