Coroot alerts can outlive their application report #5594

🤖 Generated by the Agentic Engineer

Expected Behavior

Alert state should distinguish a check that no longer applies to an application from a check whose telemetry is temporarily unavailable. Retiring an inapplicable report should not leave an unexplained firing alert, and loss of monitoring must remain visible.

Actual Behavior

At Coroot source 15137ef2f43e01d0a87245e44b937b25515e0afe, the evaluation loop skips an application when the selected check has no report. The retirement pass retains the existing alert while its rule still matches and the application exists. A fresh application view can therefore have no corresponding report while its stored alert remains firing. Current normal-SSO views reproduce that inconsistency; detailed operator evidence remains private.

This is a source-traced lifecycle case, not a completed native regression or proof of the original metric-attribution cause. The UI does not expose the stored alert's last evaluation timestamp. A missing availability check within an UNKNOWN database report can mean lost instrumentation and must not be treated as recovery.

Steps to Replicate

1. Use one matching enabled availability rule and a firing alert for an application, with no pending or keep-firing delay.
2. Audit a later world where the same live application no longer has the associated report, and run the normal alert evaluation and retirement passes.
3. Compare that retained alert with a control whose check is explicitly healthy, and with a known database whose telemetry is missing.

Acceptance Criteria

• Reproduce the lifecycle with the existing native database and watcher APIs, including the explicit-healthy control.
• Define and test a positive boundary between retired applicability and missing instrumentation, including UNKNOWN reports, empty instances and collector outages.
• Repair the owning source, validate its complete required checks and review, and adopt a verified publication through the protected platform delivery path.
• Observe the obsolete alert retire naturally while genuine database failures and monitoring gaps remain visible, without dismissal, suppression or a changed observation window.

Planning

Bug; medium priority, high end-to-end effort, Standard service class. The maintainer selected stabilization work on 3 November 2026 and explicitly waived the active WIP ceiling for this work. No active competing claim or implementation was found before selection.

The first bounded delivery is a native database/watcher regression and repair for treating an explicit UNKNOWN check as recovery. Source inspection shows that loss of telemetry can produce UNKNOWN for an existing check; this must preserve a firing or suppressed alert and restart any pending continuous-breach timer. Genuine OK recovery and explicit rule/app retirement must retain their existing behavior. Separately, absent PostgreSQL reports still need a positive applicability boundary; this first repair does not claim to retire those alerts.

First started: 2026-11-03. Acceptance, original scope and deployment verification remain open.

The alert lifecycle belongs to Coroot. An exclusion or dismissal inside the platform cannot correct that lifecycle and would weaken the observation instead. A minimal upstream report or fix is justified after a synthetic native reproduction, using only public source and synthetic data. Coroot is a public platform dependency unrelated to employment; its current contribution policy contains no AI-assistance prohibition. The UNKNOWN-state correction is now proposed in coroot/coroot#1099, offered for upstream review after local validation and passing hosted Go/UI checks. The absent-report applicability boundary and runtime acceptance remain open.

Part of #2043.