Provide single-writer failover for security progress reporting #5695

🤖 Generated by the Agentic Engineer

Outcome and evidence

Security progress reporting should survive routine Pod replacement while retaining one authoritative snapshot writer and remote-write stream. The current source deliberately uses one replica and Recreate to prevent overlapping writers. A read-only deployment check and Coroot agree on that availability risk; increasing the replica count alone would violate the existing ingestion contract.

Acceptance

• Establish and test a bounded single-writer failover design, including overlapping candidates, lost acknowledgements, replacement identities and an unavailable coordination service.
• Preserve atomic complete snapshots, explicit unknowns on failed collection, source freshness, project binding and the current metric meaning. Duplicate or out-of-order samples must not be hidden by dropping data or treating unknown observations as healthy.
• Keep permissions narrowly scoped to the actual ownership mechanism. Do not add a broadly privileged controller or another monitoring product.
• Validate native ingestion and failure controls, then deliver through protected review/deployment. Verify distinct healthy Pod identities, one active writer, continued fresh reporting and truthful liveness during failover.
• Observe the availability risk retire naturally without dismissal or a changed inspection threshold.

Planning

Feature; Medium priority, High end-to-end effort, Standard service class. Capture in Backlog pending a reviewed ownership design and downstream capacity. This is not an urgent security deadline, a cost pass or permission for an additional draft.

Part of #4981.