feat(tool): add tool-level circuit breaker to stop offering repeatedly failing tools

[pure11011]

AgentScope-Java Version

2.0.3-SNAPSHOT (current main, ea511ec)

Description

Implements #2983.

Problem. A ReAct loop is reason → call tool → read result → reason again. When a tool returns an error the model usually calls it again, because one failure looks incidental from where it stands, and nothing in the framework remembers that a tool is broken. Against a dependency that is genuinely down each retry costs a model call, an outbound request and seconds of latency, and still fails; ten consecutive failures are ten wasted reasoning rounds. The existing controls address a different problem — applyTimeout bounds a single call, applyRetry recovers a transient blip within a single call (and defaults to maxAttempts=1 for tools), and maxIters counts iterations rather than failures — so none of them can express "this tool has failed N times in a row, leave it alone for a while".

Change. An opt-in, per-tool breaker with CLOSED → OPEN → HALF_OPEN transitions and exponential backoff, added as new classes only. Nothing existing changes behaviour.

New in io.agentscope.core.tool.circuitbreaker:

• ToolCircuitBreaker — the state machine and backoff policy. A plain object with no reactive or agent dependencies, so it is unit-testable against an injected Clock.
• ToolCircuitBreakerConfig — supervised/excluded tool sets, threshold, initial cooldown, multiplier, ceiling, with validation in build().
• ToolCircuitBreakerStore — persistence SPI, plus InMemoryToolCircuitBreakerStore as the default.
• ToolCircuitSnapshot / ToolCircuitState — the persisted state and the derived lifecycle state.
• ToolCircuitBreakerMiddleware — the adapter onto MiddlewareBase.

New in agentscope-extensions-redis: RedisToolCircuitBreakerStore, so replicas share one view of a broken tool instead of each rediscovering the outage and sending N times the failing traffic.

Three design points that reviewers will want to weigh:

1. The breaker withholds the tool rather than rejecting the call. A classic breaker sits between caller and dependency and fails fast once open. Here there is a better option: drop the tool from the schema list the model receives. A tool the model cannot see is a tool it cannot ask for, so the retry loop disappears at the source instead of being absorbed, and no prompt has to argue the model out of it.

onReasoning filters ReasoningInput.tools() into a per-turn copy; the Toolkit is never touched. Mutating ToolGroup membership is the other way to hide a tool, and I deliberately avoided it: group membership is shared mutable state, so a circuit tripped for one session would remove the tool from every concurrent session and would overwrite the registrations the application declared. Per-turn filtering also means recovery needs no repair step — once the breaker stops withholding, the unfiltered list is already correct — which removes a whole class of state-thrashing bugs.

2. Failures are classified by typed state, not by matching error text. onActing reads the ToolResultState already carried on ToolResultEndEvent. Only ERROR counts. DENIED is a permission refusal and INTERRUPTED a cancellation; neither is evidence about the dependency, and counting DENIED would let a user who declines a confirmation prompt trip the circuit. RUNNING marks a suspended call whose outcome is not known yet.

3. Supervision is opt-in, and the default configuration is inert. Only tools named via monitorTools(...) are watched, unless monitorAllTools(true) is set; excludeTools(...) overrides both. Withholding a flaky weather API degrades an agent gracefully, but withholding the database or filesystem tool cripples it, and opt-out supervision would also cover framework-supplied per-call tools such as the structured-output tool — losing that one stops the agent completing at all. Requiring the set to be named means a breaker cannot take down a tool nobody considered. The monitorAllTools javadoc spells this out.

Cooldown is min(initialCooldown * multiplier^(generation-1), maxCooldown), derived from the stored generation rather than persisted, so changing the policy takes effect immediately and never needs migrating. Defaults of 60s / x2 / 600s give 60s, 120s, 240s, 480s, 600s… cooldownFor clamps non-finite and over-ceiling values, so a large generation cannot isolate a tool indefinitely. State is derived from the stored snapshot plus the current time, so a cooldown that elapsed while the agent was idle is picked up on the next read: no scheduler, no timer, no background thread, and nothing to leak.

Both store implementations make the two read-modify-write operations atomic — ConcurrentHashMap.compute/computeIfAbsent in memory, single-key Lua scripts in Redis. The Redis keys are addressed one at a time, so no script needs its arguments to share a hash slot and the store works unchanged on Redis Cluster.

Checklist

Please check the following items before code is ready to be reviewed.

• Code has been formatted with mvn spotless:apply
• All tests are passing (mvn test) — agentscope-core 2360 tests and agentscope-extensions-redis 12 tests pass locally; full reactor build runs in CI
• Javadoc comments are complete and follow project conventions — mvn -pl agentscope-core clean verify builds the javadoc jar without warnings
• Related documentation has been updated (e.g. links, examples, etc.) — no docs page added yet; the API is documented in class-level javadoc with a usage example on ToolCircuitBreakerMiddleware. Happy to add a page under docs/ if maintainers want one, once the API shape is agreed.
• Code is ready for review

[codecov]

Codecov Report

❌ Patch coverage is 85.02024% with 37 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../tool/circuitbreaker/ToolCircuitBreakerConfig.java	80.00%	7 Missing and 6 partials ⚠️
...l/circuitbreaker/ToolCircuitBreakerMiddleware.java	74.35%	5 Missing and 5 partials ⚠️
...e/core/tool/circuitbreaker/ToolCircuitBreaker.java	89.47%	6 Missing and 2 partials ⚠️
...s/circuitbreaker/RedisToolCircuitBreakerStore.java	86.66%	3 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

[CryoThrust]

One concurrency edge case to consider before merge: the implementation documents HALF_OPEN as a single recovery probe, but state() only derives HALF_OPEN from the elapsed timestamp and does not claim a probe. Once the cooldown elapses, concurrent reasoning turns can all observe HALF_OPEN, advertise the tool, and invoke it before any result is recorded. Each successful call will close the circuit, while concurrent failures can call open() repeatedly and advance the generation/backoff more than once. The same race exists across replicas with the Redis store.

Could the store/middleware expose an atomic half-open permit (for example, compare-and-set OPEN -> HALF_OPEN with a lease/token) so only one in-flight probe is allowed, while other turns continue withholding the tool until that probe succeeds or its lease expires? If the intended policy is to allow concurrent probes, the state-machine/Javadoc should say so and the tests should cover the behavior, since it differs from the stated “single probe” semantics.

[CryoThrust]

A second concurrency issue affects the “consecutive failures” contract, independently of the HALF_OPEN probe race above. recordFailure() performs recordFailure() -> checks the returned count -> open() -> resetFailures() as separate store operations, while recordSuccess() can call resetFailures() between those steps. A possible interleaving is: (1) failure A increments the closed counter to the threshold; (2) a successful call B observes CLOSED and clears the counter; (3) failure A still calls open() using its stale threshold result and trips the circuit even though the failures were not consecutive. The same window exists with the Redis store because the increment and open scripts are separate commands. Please make the threshold transition conditional/atomic (for example, a store operation that records the outcome and opens only if the same counter version still reaches the threshold, or a CAS/version token), or document and test a weaker policy. A deterministic concurrent test with a barrier between the threshold check and open() would expose this regression.