Add composite MySQL safety checks and stable recovery

[pradeepkintali]

Summary

• compose replica lag with required primary-load and ProxySQL-capacity safety metrics
• fail closed when configured probes have no eligible hosts or required runtime metrics are unavailable
• keep throttling active until recovery thresholds and continuous cooling periods are satisfied
• require a complete healthy recovery window after process or leader startup
• include the blocking metric name in composite check responses and publish recovery-latch metrics
• support explicit MASTER/PRIMARY Vitess tablet discovery while preserving REPLICA as the default
• validate required-cluster references, dependency cycles, and Vitess tablet types at configuration load time
• document the configuration model and scope boundaries

Configuration model

A normal replica-lag cluster can list primary-load, connection-capacity, or other scalar safety metrics in RequiredClusters. Each required metric is sampled centrally by the Freno leader and may opt into:

• FailOnNoHosts
• RecoveryThreshold
• RecoveryDurationMillis
• an explicit Vitess tablet type where applicable

All settings are opt-in, preserving existing behavior for current clusters.

Validation

• go test -count=1 ./...
• go vet ./pkg/base ./pkg/config ./pkg/throttle ./pkg/http ./pkg/mysql ./pkg/proxysql ./pkg/vitess

Full-repository go vet ./... still reports pre-existing testing.T.Fatalf calls from non-test goroutines in internal/raft tests.

Scope boundaries

This PR provides reusable admission checks and recovery behavior. Deployment-specific metrics, thresholds, host discovery, operational alerting, and application concurrency or fallback controls remain the responsibility of each deployment.

[Copilot]

Copilot review overview

🟡 Changes recommended

Recovery semantics and direct-memcache behavior can bypass or inconsistently apply the new safety gates.

Review effort: Balanced
Findings: 1 · 3

Open (4)

• Removing IsValidReplica breaks source compatibility · New
• Direct client path bypasses required cluster checks · New
• Recovery state is not reset after leadership regain · New
• Recovery ignores the effective read threshold · New

What changed in this PR

Adds composite MySQL safety checks, recovery hysteresis, and configurable Vitess tablet discovery.

Changes:

• Composes replica checks with required primary and ProxySQL metrics.
• Adds fail-closed behavior, recovery thresholds, cooling periods, and metrics.
• Supports validated REPLICA, MASTER, and PRIMARY Vitess tablet configuration.

File	Description
pkg/​base/​throttle_metric.go	Adds recovery-state metric errors.
pkg/​config/​config_test.go	Tests new configuration validation.
pkg/​config/​mysql_config.go	Adds composite and recovery settings.
pkg/​config/​vitess_config.go	Validates tablet types.
pkg/​throttle/​check.go	Evaluates required clusters.
pkg/​throttle/​check_result.go	Reports the blocking metric.
pkg/​throttle/​throttler.go	Implements recovery latching.
pkg/​throttle/​throttler_test.go	Tests composition and recovery.
pkg/​vitess/​api_client.go	Filters configured tablet types.
pkg/​vitess/​api_client_test.go	Tests primary discovery.
doc/​mysql.md	Documents configuration and rollout scope.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[gh-sre-bot]

The incident-repair label has been added to this issue.

Why was this labeled incident-repair?

• Repair items are identified based on if they have been included in an availability review's list of action items.
• This label is applied to all repair items so they can be tracked and given priority.

This isn't a repair item. How do I stop this from being tracked?

If this issue is not a repair item, you can stop the bot from auto-labeling by removing it from the list of action items in the associated availability review. Then the incident-repair label can be removed from this issue.

Service Catalog Tracking

This repair item is being tracked under the Service Catalog service freno.

Related Links 🔗

• Repair Item SLA Documentation in The Hub
• Source Code for Labeling

Please reach out to the incident-lifecycle team if you have any questions.

🤖 This comment was created automatically by the SRE bot. Any bugs or feedback please open a bug report issue in github/sre-availability-bot or come chat in the #incident-lifecycle Slack channel.