fix(harness): detach memory flush and maintenance from agent call response

[birdie7761]

Fixes #2225
Fixes #2276
Also addresses #2774 and #2821 (same root cause).

Problem

MemoryFlushMiddleware / MemoryMaintenanceMiddleware appended their LLM-backed work via concatWith onto the returned Flux. Because ReActAgent.callInternal ends with takeLast(1) — which can't emit until upstream signals onComplete — callers consuming the response to completion (blockLast(), takeLast(1), WebFlux controllers awaiting Mono<Msg>) waited for the full memory-flush LLM round-trip (19–27s per call) and consolidation (~44s first run).

This is a regression: the original implementation was fire-and-forget (doOnComplete); PR #1802 (RC4) swapped it for concatWith.

Fix

Both middlewares subscribe their work independently via doOnComplete().subscribe(), restoring the historical detached behavior, with production hardening:

• Lifecycle: both middlewares are AutoCloseable; pending Disposables are tracked under synchronized(pending) with pre/post-subscribe closed checks. HarnessAgent.close() drains them (5s bound) before workspace teardown — no writes racing temp-dir deletion.
• Bounded backlog (new): at most 4 detached runs in flight per middleware; beyond that, runs are skipped and logged instead of accumulating without bound. Skipping is safe — a flush extracts from the full conversation context, so the next successful flush covers the same messages.
• Durability opt-out (new): MemoryConfig.asyncFlush(false) chains the per-call memory work back onto the response stream for callers relying on "response completed == memory persisted" — bounded by the same 5-minute timeout, failures logged and never fail the response. Default is true (the restored behavior).
• Hung-model protection: flush/consolidation LLM calls capped at 5 minutes (measured after subscribeOn, excluding queue wait).
• Snapshot isolation: FlushRequest copies messages on the completion thread, immune to later context mutation.

Relationship to #2777 and #2833

Same root cause cluster (#2225/#2276/#2774/#2821). Differences:

• fix(harness): make memory flush fire-and-forget to unblock conversation completion #2777 makes flush fire-and-forget without lifecycle draining or backlog bounds.
• fix(harness): support asynchronous memory flush #2833 adds an opt-in asyncFlush flag that defaults to synchronous — the regression stays on the default path, maintenance is not covered, and its static single-worker scheduler has no timeout (one hung flush stalls the whole pipeline) and is shared across all agents in the JVM.
• This PR fixes the default for everyone, covers flush and maintenance, drains on close, bounds the backlog, and now also provides the asyncFlush(false) opt-out and docs — a superset of both.

Test evidence

17 middleware tests (TDD, red on main before the fix): completes-before-slow-flush/consolidation, errors-don't-propagate, close-prevents-schedules/drains/disposes-hung, sync-mode waits (×2), backlog cap skips beyond limit, config default + opt-out. Full harness suite 861/0 failures.

Checklist

• Code has been formatted with mvn spotless:apply
• All tests are passing (mvn test)
• Javadoc comments are complete and follow project conventions
• Related documentation has been updated (en/zh memory guides)
• Code is ready for review

[codecov]

Codecov Report

❌ Patch coverage is 82.99320% with 25 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...arness/agent/middleware/MemoryFlushMiddleware.java	79.72%	13 Missing and 2 partials ⚠️
.../agent/middleware/MemoryMaintenanceMiddleware.java	85.29%	9 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[AgentScopeJavaBot]

🤖 AI Review

This PR fixes a real and impactful latency bug: MemoryFlushMiddleware and MemoryMaintenanceMiddleware previously used concatWith to append their LLM-backed work onto the returned Flux, forcing callers (e.g. ReActAgent.callInternal with takeLast(1)) to block until the memory flush/maintenance LLM call completed. The fix correctly switches to doOnComplete(() -> subscribe()) for genuine fire-and-forget behavior, adds AutoCloseable lifecycle management with bounded drain on shutdown, and includes thorough test coverage for the async semantics. The concurrency design (synchronized pending-set pattern, double-check on closed flag, timeout caps) is well thought-out. Two areas warrant attention: a subtle race in the subscribe-then-add-to-pending pattern, and missing stack traces in close-failure logs.

(inline comments could not be attached — line numbers fell outside PR hunks. See archived report.)

[AgentScopeJavaBot]

🤖 AI Review

This PR fixes a real and impactful latency bug: MemoryFlushMiddleware and MemoryMaintenanceMiddleware previously used concatWith to append their LLM-backed work onto the returned Flux, forcing callers (e.g. ReActAgent.callInternal with takeLast(1)) to block until the memory flush/maintenance LLM call completed. The fix correctly switches to doOnComplete(() -> subscribe()) for genuine fire-and-forget behavior, adds AutoCloseable lifecycle management with bounded drain on shutdown, and includes thorough test coverage for the async semantics. The concurrency design (synchronized pending-set pattern, double-check on closed flag, timeout caps) is well thought-out. Two areas warrant attention: a subtle race in the subscribe-then-add-to-pending pattern, and missing stack traces in close-failure logs.

(inline comments could not be attached — line numbers fell outside PR hunks. See archived report.)

[birdie7761]

Thank you for the thorough review — both points were valid and are fixed in the updated commit.

1. subscribe-then-add race: Confirmed. The failure mode: a fast-completing task (e.g. maintenance rejected by the throttle gate returns in microseconds) could run doFinally and read holder[0] while it was still null (the assignment happens after subscribe() returns), skip the removal, and then leak a terminated Disposable into pending forever — stalling close()'s drain loop for its full 5s timeout.

The fix restructures both scheduleFlush and scheduleMaintenance: subscribe() and pending.add(d) now happen inside a single synchronized(pending) section (one closed check, replacing the previous pre/post double-check), and doFinally reads holder[0] and removes it under the same lock — so both the ordering and the visibility of holder[0] = d are guaranteed (the write happens inside the lock before release; the read happens after acquiring it). subscribeOn(boundedElastic) ensures no callback ever runs on the scheduling thread, so the lock cannot self-deadlock. Both critical sections carry inline comments explaining this.

2. Missing stack traces: All log.warn("...: {}", e.getMessage()) calls in the touched files now pass the throwable as the last argument (log.warn("...", e)) — covering flush/maintenance scheduling failures, the LLM-call failures, consolidation failures, and HarnessAgent.close()'s middleware-close failure.

Additionally, while re-verifying the fix I hardened two adjacent paths in the same commit:

• The maintenance RuntimeContext snapshot is now taken at completion time (inside doOnComplete) rather than eagerly at onAgent assembly, so attributes added during the call are visible — matching the original rc-at-completion semantics and the flush side's capture timing.
• The whole schedule* body is wrapped in try/catch so no synchronous exception (operator assembly, subscribe()) can escape into doOnComplete and turn a completed Flux into an error.

New regression tests: rapidFastCompletingFlushes_leaveNoTerminatedEntriesInPending and rapidFastCompletingMaintenance_leavesNoTerminatedEntriesInPending — each schedules 50 instantly-completing tasks and asserts pending settles with no leaked terminated entries. This exercises the old race aggressively (it reproduced readily under the previous code) and is deterministic under the single-lock pattern.

Full harness suite: 797 tests, 0 failures (JDK 17), spotless clean.

[doctormacky]

Hi guys, when this PR will be merged ? any deadline ?

[birdie7761]

Closing in favor of #2777, which landed the core detach (flush + maintenance) with per-conversation coalescing and close-time quiescence — congratulations @chcodex, nice work.

I plan to follow up with a small hardening PR building on MemoryBackgroundTasks: (1) a bounded timeout on the flush/consolidation LLM calls so a hung model can't pin a conversation's coalescing slot forever, (2) disposing hung tasks at close (awaitQuiescence currently waits-then-gives-up, so a hung flush keeps writing after teardown — the race #2935 patched at the test level), and (3) a MemoryConfig.asyncFlush(false) opt-out for callers needing completion-order persistence. Thanks all for the reviews here — the subscribe-then-add discussion materially improved that follow-up.