Repository navigation
fix(backend): service data-collection bugs across all 13 integrations - #145
Merged
Merged
Conversation
…eachable Most checkHealth implementations resolve ok() with reachable:false when the service can't be reached (withHostPing), so POST /config/services/:id/test returned ok:true for a dead service. It now returns ok:false with error.code UNREACHABLE and keeps the snapshot for diagnosis. Docs for the endpoint described a request/response shape the code never had; rewritten.
… torrent detail; remember slow-lane failures qBittorrent: getStats fanned out 4 parallel requests that each logged in on a cold start or expired session, so a wrong password made ~5 failed logins per poll and tripped qBittorrent's default IP ban (5 failures), renewed every poll. Logins are now single-flight and a rejected login is not retried for 5 minutes. Response bodies on login and 403 are consumed. Wrong credentials (200 "Fails.") now say so. activeTorrents spread every maindata field (magnet_uri, save_path, ...) into each snapshot; it now keeps the 11 documented fields. ttlMemo: a failing slow-lane fetch was retried on every poll. A failure is now remembered for min(ttl, 60s), serving the stale value or the same error.
…dentials Every non-2xx was UNAVAILABLE, so a wrong password showed as an outage. Docs no longer claim a protection toggle (Watchman is read-only).
…C health Unparsed responses (401 wallet calls once the JWT expires, every legacy probe response, non-2xx info) were never read, so undici held their sockets until GC. NWC health said "returned no response" for timeouts; it now carries the error message.
…slow lane A full repo/stat walks the whole blockstore to count NumObjects, which on a large repo exceeded the timeout every poll while Kubo kept working.
Coverage Report
File Coverage
|
||||||||||||||||||||||||||||||||||||||
- Per-call batch errors were dropped, so a node warming up (-28 "Loading block index…") showed only "data incomplete". The node's message now reaches the health message and the stats error. - HTTP 403 (rpcwhitelist rejecting a method) maps to UNAUTHORIZED. - A failed ZMQ connect left stats looking like a working setup with no blocks. New zmqConnected/zmqError metrics say why. A connect that finishes after onStop is closed, and the zeromq socket is closed when connect/subscribe throws.
- sseClient reported transport errors but not a clean end, so after the bridge closed the stream the service kept serving its light map without ever re-fetching, missing every change made during the reconnect gap. A clean end now goes to onError. - Health probed /resource/light (every light, every 10 s) and never read the body. It now probes /resource/bridge and reads the body; non-2xx bodies are read everywhere. - Health latency was measured before the API call (≈0 with ping off). It is now the API round trip, falling back to ping.
…erval Error state was one set per service id for both lanes, and any health update counted as recovery, including ok(reachable:false). A down service (or one whose stats keep failing) produced an error toast every stats poll and a false "recovered" toast on the next health tick, forever. Error state is now per id+scope, and an unreachable health snapshot is not a recovery.
… newer state Before a service's first poll, GET /services and /services/:kind/health probe live on the request signal with no limit, so one unreachable host made the whole dashboard request time out (frontend gives up at 10 s), e.g. after every profile switch. The probe now has its own 5 s limit, and its result is cached only if it was not cut short, the instance is still the registered one, and no poll published meanwhile.
…utMs Every poll and cached stats read was cut at a fixed 10 s, so raising timeoutMs above 10 s (slow NAS, Bitcoin node) never helped: every poll failed with "aborted after 10000ms" and tripped the breaker. The limit is now max(10 s, 2 × timeoutMs), unchanged at the 5 s default.
…ted signals net-snmp emits 'error' on a session when an incoming datagram fails to parse. With no listener that is an uncaught exception that exits the backend (Synology and router both use this getter). get and walk now share one settle path that also removes the abort listener and rejects at once for an already-aborted signal.
105 means the account lacks privilege for the API. The client treated it as session expiry, so a non-admin account logged in again on every poll and left a new DSM session each time. It now fails with an UnauthorizedError naming the API. Only 106/107/119 re-login, and the retry uses the sid its own login returned.
…m partial walks - DSM-only mode (no SNMPv3 credentials) always reported the NAS offline, because health only probed SNMP. Health now probes SYNO.DSM.Info getinfo. - DSM-only getStats where every DSM call failed returned ok() with zero CPU, memory and disk. It now returns the first DSM error. - A failed ifDescr or counter walk was summed as 0 and stored as the rate baseline, so the next poll reported the whole counter as one interval's traffic (~1e12 B / 30 s). Failed walks now give no sample, the baseline is kept, and rates are only computed between samples of the same counter kind (HC vs 32-bit).
A failed ifDescr walk left no active interfaces, so octets were summed as 0 and stored as the rate baseline; the next poll reported the whole counter as one interval's traffic. A failed HC walk also fell back to the 32-bit sum, mixing counter families. Now a failed walk gives no octet sample and keeps the baseline, and rates are only taken between samples of the same counter kind.
…lies and close controlClient and eventSubscription called removeAllListeners() once connected. That left the control socket without an 'error' listener (a reset between commands was an uncaught exception that kills the backend) and also stripped net.Socket's internal 'end' handler, so a peer FIN never closed the socket and reads waited for the abort signal. Only the connect listeners are removed now; the control handle keeps permanent 'error'/'close' listeners and fails reads fast once the socket is gone. The BW event subscription waited for replies with no limit, so close() (and TorService.onStop) hung on a half-open socket until TCP gave up. Each reply is now awaited for at most timeoutMs, after which the socket is dropped.
…b on SETEVENTS failure - Onionoo mode searched onionoo.torproject.org on every health (10 s) and stats (30 s) poll. Health and stats now share one lookup per instance for 10 minutes; failures are not cached. - trafficDelta* went hugely negative after a Tor restart reset the counters. That poll now reports null and rebases. - subscribe() leaked the authenticated socket when SETEVENTS failed.
…led run ping sends one echo per second, so pingCount 8 with a 5 s timeout ran ~7 s, was killed, and a responsive host was reported unreachable. The count is now capped at timeoutMs/1000, and a run killed after printing a reply line counts as reachable (average taken from the per-reply times).
…rotocol reference The cited configure flag does not exist; document the real cookie locations (<DataDirectory>/control_auth_cookie, Debian /run/tor/control.authcookie), CookieAuthFileGroupReadable, and that cookie auth needs Watchman on the Tor host. trafficDeltaRead/Written directions were swapped. Note the Onionoo cache, the pingCount cap and the bounded subscription close.
…io connections; split throttled flags - PiStatsCollector: a pigpiod connect failure no longer fails stats when direct SSH or the relay works (pigpiod doesn't run on the Pi 5 family). hwRevision/pigpioVersion stay null and pigpioError says why; stats fail only when every source failed. - pigpioClientImpl: end() the client when the connect times out or errors (it kept retrying in the background), and keep an 'error' listener for the life of the connection so a later socket error can't go unhandled. - sharedPigpioClient: bound each command by timeoutMs (a powered-off Pi never answers, which hung the health check) and end a connection that finishes after close(). - New throttledNow (bits 0-3) and throttledSinceBoot (bits 16-19) metrics; the sticky since-boot bits made any past brownout look active. - Docs: compound exec, reachability, relay fallback, pigpiod optional, throttled bits, SSH pool behaviour.
A hung remote command kept its channel open on the pooled connection; after sshd's MaxSessions (default 10) every later exec on that host failed. The channel is now closed when the exec times out or is aborted, including one that opens after the request already settled.
Each compound segment runs in its own subshell, so the Homebrew PATH set by the temperature segment never reached smartctl, and SMART metrics were always null on Homebrew installs. Disk fields were 0 (shown as "Disk 0%") when df produced nothing; they are now null like the other unmeasured metrics. Docs: health is ping only, Mac prerequisites, memTotalBytes.
node-roon-api writes its state to config.json in the process cwd unless given persisted-state hooks: an untracked secret in a dev checkout, or lost on a read-only cwd so every restart asks to enable the extension again. The token now lives in DATA_DIR/roon-state.json (0600), keyed per Core address. The old config.json is read as a fallback so existing pairings survive.
EraPartner
changed the title
Replaces the old 105 retry test (the behaviour it pinned was the bug) and uses 106 for the retry-also-fails case.
EraPartner
marked this pull request as ready for review
October 28, 2026 08:25
EraPartner
enabled auto-merge (squash)
October 28, 2026 08:25
EraPartner
added a commit
that referenced
this pull request
Oct 28, 2026
…144) * docs: add per-service setup guides with least-privilege credentials Add docs/guides/service-setup/ with an index and one guide for each of the 13 service kinds: what to enable on the service, which read-only or least-privilege credentials to create, what to enter in Watchman, and troubleshooting keyed to the error messages in the code. Link the guides from the integration pages and indexes, and fix doc drift found while checking against the code: AdGuard no longer claims toggle control, the Homebridge health probe path, Bitcoin's Tor claim, and the unused zmqRawtxEndpoint. * docs: drop Test connection caveat, fix useServiceHealth hook doc #146 made Test connection report an unreachable service as Failed, so the setup guides no longer need the known-issue callout. docs/components/use-service-health.md described a protection-toggle mutation hook and a cache-clear hook that don't exist; rewrite it from apps/frontend/src/hooks/useServiceHealth.ts (read-only, ADR-032). * docs: update setup guides for the service bug fixes on main Rebased onto #145/#147/#148: document SSH host-key pinning (Mac Mini, Pi), the DSM certificate fingerprint and DSM-only health, the now-used Bitcoin rawtx ZMQ endpoint, Roon pairing state in DATA_DIR, and the new error messages and login backoffs. --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by EraPartner · project thread
This PR fixes bugs in how the backend collects data from monitored services, and in the polling and alert pipeline. All 13 integrations, plus the poller, transport and infra layers, were reviewed against the code and the 2026-10-21 audit notes. Items the audits had already fixed, or had deliberately left alone, were skipped. Every fix below has a regression test except where marked. Each commit message has the details.
Fixes
POST /config/services/:id/test)ok:truefor an unreachable service, because probes resolveok(reachable:false).ok:falsewitherror.code: UNREACHABLEand keeps the snapshot. OpenAPI andgenerated.tsdescriptions are updated, anddocs/api/config.mdis rewritten (it described a shape the code never had).activeTorrentsspread every maindata field, includingmagnet_uriandsave_path, into history and the WebSocket.ttlMemo(slow lanes: AdGuard, IPFS, Homebridge, Hue, qBittorrent)UNAVAILABLE.UNAUTHORIZED, so the tile asks for credentials.repo/statwalks the whole blockstore on every 30 s poll.size-only=trueper poll, and gets the object count on the 10-minute slow lane.rpcwhitelistwas reported asUNAVAILABLE.UNAUTHORIZED.onStopwas never closed.zmqConnectedandzmqErrormetrics. The socket is closed in both cases./resource/bridgeand reads the body. Latency is now the API round trip.reachable:falseno longer counts as a recovery.GET /servicesprobed with no time limit, so one unreachable host failed the dashboard request (e.g. after every profile switch). A probe cut short could overwrite newer state.timeoutMspast 10 s never worked.timeoutMs), which is unchanged at the default.'error'had no listener, so a malformed packet crashed the backend. Abort listeners leaked.UnavailableError, and listeners are cleaned up.SYNO.DSM.Info, and a total failure now returns an error. Walk failures no longer move the rate baseline. 105 now givesUNAUTHORIZEDwith no re-login.close()also leaked. Health could hang forever.throttledstayed set for past events.pigpioErroris reported). Commands are time-bounded and the leaks are fixed. NewthrottledNowandthrottledSinceBootmetrics.MaxSessionswas exhausted.smartctlwas not found on Homebrew installs, because PATH was set in another subshell. Disk showed 0 whendffailed.'error'listener (crash on reset). Event-subscriptionclose()could hang for about 15 minutes. Traffic deltas went negative after a Tor restart. Onionoo was queried on every poll.pingCountwas killed mid-run, so a host that was up was reported down.timeoutMs, and replies printed before the kill still count../config.jsonin the cwd.DATA_DIR/roon-state.json(0600), one entry per Core. The old file is still read as a fallback.Docs updated:
docs/api/config.md,docs/api/services-health.md,docs/features/real-time-updates.md,docs/features/service-monitoring.md, anddocs/integrations/{adguard,bitcoin,ipfs,macmini,philips-hue,qbittorrent,raspberry-pi,roon,router,synology,tor}.md.Decisions (standing rule: pick the recommended option)
timeoutMs). Two, because probes chain at most two bounded requests in series.-iflag; the count is clamped instead. Short intervals need root on some platforms.host:port. This keeps two Cores from overwriting each other.Not changed (needs a decision or a live device)
https://…:5001with DSM's self-signed certificate: needs a newallowSelfSignedorcertHashfield plus UI. Today DSM over HTTPS fails silently.PROTOCOLINFOauto-auth, a persistent Tor control connection, Macroute get defaultfor the interface, routerifNamematching, reusing the SNMPv3 session, KuboAPI.Authorizations, and an Alby expired-JWT warning.throttledNow. The Synology chart should handlenetworkRx/networkTxbeing absent.Read-only rule
No new calls to monitored services. Removed calls: Hue
/resource/lighton health checks, the full IPFSrepo/staton every poll, Onionoo on every poll, and DSM re-logins on code 105.Checks
Nothing ran in the cloud (no npm access). CI on
a574e39: all checks green (backend tests, frontend tests, typecheck, lint, generated-artifact check, CodeQL, Trivy). The first CI runs (019b6c3,2ad418b) failed on twodsmClienttests that still expected a re-login on code 105;a574e39updated them. CI re-runs after the main merge.🤖 Generated with Claude Code
https://claude.ai/code/session_018QJtaL6RxEK3gXc6YDZdPD