Allow modules to provision alert rules

[Amygos]

Problem

NS8 modules can publish metrics and Grafana resources, but they cannot publish their own Prometheus alert rules through a supported cross-module contract.

As a result, application-specific rules must be maintained by the metrics module or placed in generated state. This breaks module ownership and makes rule lifecycle management unreliable. The existing custom_alerts mechanism is metrics-local and is not suitable as a provider integration API.

Proposed solution

Allow module instances to publish alert rules through Redis and notify the metrics module when they change.

The metrics module will provision these rules, scope them to the publishing instance, and load them into Prometheus. Legacy metrics-local custom alerts will remain on their existing implementation.

Challenges

• Rules from one module instance must not evaluate metrics belonging to another instance.
• Invalid provider rules must not disrupt Prometheus or independent providers.
• Rule additions, updates, and removals must be applied without leaving stale state.
• Existing built-in rules, custom alerts, and alert delivery must remain compatible.

How the challenges are addressed

• The publisher identity is derived from the Redis key and enforced across metrics, rules, and alerts.
• Each provider rule set is validated and managed independently, retaining its previous valid version after an invalid update.
• Provider rules use a dedicated generated namespace, atomic updates, scoped cleanup, and verified Prometheus reloads.
• Built-in rules and legacy custom_alerts remain outside the new pipeline, while provider alerts continue through the existing Alertmanager and alert-proxy flow.