🐛 fix: stop waking the scale-to-zero federated peer (and stop it self-warming afterwards)

[flupkede]

What

A mounted federated cloud peer (Azure Container Apps, minReplicas: 0) was waking up roughly every two hours and staying warm for about an hour, with nobody querying it. This branch stops the wake and stops the self-warming that followed it.

Background polling of local repos is unchanged and unaffected — the local/federated split is a deliberate design constraint, and it was previously honoured in only one direction.

Ground truth

From Azure Log Analytics on the deployed peer, over a window with zero federated searches:

Observation	Value
Interval between wakes	120, 121, 120 minutes
Warm period per wake	~67 min (1h idle window + 5min KEDA cooldown)
Resulting duty cycle	≈13.4h warm/day, ~56% — at zero searches

The 120-minute spacing is the tell: that is the local host's 2h DEFAULT_IDLE_SUSPEND_SECS, not any value configured on the peer. Each wake additionally paid an azcopy sync of the docs blob and a git pull of the KB.

Root cause — two defects

1. The trigger. spawn_remote_discovery in the serve TUI used state.idle_suspend_secs() as a baseline poll interval and ran a JoinSet /status fan-out to every configured peer. The poll itself was the ingress traffic that woke the replica.

2. The amplifier. The cloud keep-warm loop computed most_recent_tool_call().unwrap_or(start). /status and /healthz have their own handlers and never call record_tool_call, so a replica woken by anything other than a genuine tool call fell back to the process start time and self-pinged every 120s for the entire idle window — ~67 min instead of the ~6 min a bare wake costs (~11x amplification). That fallback was unreachable in the case it was written for: a real tool call always records itself, so it could only ever fire when the wake was not real work.

Neither defect alone produces the observed duty cycle; together they do.

Why this is the third attempt

PR #181 introduced the cadence on the reasoning that polling no faster than the host's own suspend term is harmless. PR #184 named "waking the scale-to-zero cloud peer for no real reason" as the defect and then explicitly sanctioned that same cadence.

The flaw: not keeping a peer awake past its suspend term is strictly weaker than not waking it — and the two windows were unrelated values anyway (the cadence read the local host's 2h default, not the peer's ~1h). Both PRs correctly said "local repos unaffected", confirming the split was real but honoured one-way.

The rejected reasoning is now recorded in AGENTS.md, the diagnosis doc and the code comments, specifically so it is not re-litigated a fourth time.

Changes

• 6f1d1c5 — no timer poll of federated peers. The discovery tick is now config-only (REMOTE_ROW_REFRESH_SECS = 5s, zero HTTP): it rebuilds mounted-remote rows from the remote_mounts allowlist so mount/unmount edits and l reloads still surface. A peer is contacted only by an activity poke (a real federated tool call just hit it, so it is demonstrably awake — single peer, never a fan-out) or the explicit i info-overlay keypress. Removes ServeState::idle_suspend_secs, which existed solely to feed the cadence and became write-only; --idle-suspend-secs still works because keep-warm resolves flag > env > default itself. Also fixes a pre-existing bug where removing the last peer left its rows rendered forever.
• 12edcf2 — keep-warm requires a real recorded tool call. With none, it does not ping and lets the host suspend the replica. Also fixes the startup "target isn't self" warning from 55fa36b, which false-positived on the only deployment where keep-warm is correct: Azure binds 0.0.0.0 while the target is the ingress FQDN, so it fired on every cold start. A wildcard bind means our external host is unknown, so the check now stays silent — a warning that cries wolf on the correct configuration just trains operators to ignore it. Rule extracted into a testable keep_warm_foreign_target helper.
• 3bcf153 / 08276de — docs. Diagnosis rewritten, AGENTS.md + CHANGELOG.md corrected, work log added.

Behaviour explicitly preserved

A peer staying warm for its full idle window after real use is correct and is unchanged. Verified: an inbound federated search forces project=<alias> and reaches record_tool_call, and last_tool_call is insert-only (no remove/clear/retain, untouched by repo idle-eviction), so once one real query lands the previous behaviour holds for the process lifetime.

The earlier diagnosis was wrong

The pre-existing diagnosis blamed a misconfigured local CODESEARCH_KEEP_WARM_URL. Disproven on four independent grounds, now recorded under What was ruled out: the env var is set nowhere locally (process env, HKCU, HKLM, every shell profile); the one-time 🔥 keep-warm enabled line appears in zero logs from 2026-04-26 on; that absence is meaningful because init_serve_logger is always file-only in serve mode and those logs do carry other INFO lines; and no local process held a :443 connection.

Validation

cargo fmt --check, cargo check --all-targets, cargo clippy --all-targets -- -D warnings all clean. 1134 passed, 42 ignored (up from 1124 — five new tests covering wildcard binds, a genuine foreign host, a matching host, loopback targets and an unparseable URL).

Every commit was code-reviewed; the final full-branch review returned PASS with no findings.

Known residual (not exploitable today)

The MCP status tool, when project-scoped, does record a tool call (allow_unscoped=true reduces the guard to !is_multi), so an automated poller of that tool would still buy a warm window. No such poller exists — Watch-CodesearchServeReplicas.ps1 and FederationClient::list_repos both use the HTTP /status endpoint, which does not record. Documented as the first place to look if the symptom recurs.

Not yet verified in production

Validated by tests and review, not by observing the deployed peer stay asleep. Worth re-running the Log Analytics query once this reaches the cloud replica.

🤖 Generated with Claude Code