fix(harness): prevent missed compaction for multilingual contexts

[guslegend0510]

AgentScope-Java Version

2.0.3-SNAPSHOT

Description

Fixes #2985

Chinese-heavy conversations could exceed the model's context window before compaction triggered. The estimator used the same character ratio for every language, while the trigger excluded system prompts and tool definitions.

This change:

• Counts non-ASCII text more conservatively while preserving the existing ASCII estimate, and includes thinking and hint text.
• Includes system prompts, tool definitions and response-format schemas in the trigger budget.
• Uses the primary reasoning model's context window for dynamic thresholds, even when a separate summary model is configured.
• Applies argument truncation and tool-result pruning even when summarization is skipped.
• Preserves original history when summarization fails or returns empty text.
• Retains existing public method signatures and updates English and Chinese documentation.

Token counts remain estimates; exact Qwen tokenizer accuracy and live deployment behavior have not been validated.

Validation

• 40 focused compaction tests passed.
• Full Core suite: 2,318 tests, 0 failures/errors, 9 skipped.
• Full Harness suite: 942 tests, 0 failures/errors, 5 skipped.
• The 15 new budget regression cases produced 14 failures on unchanged main and all passed with this fix.
• Spotless and git diff --check passed.

Commands:

• mvn -pl agentscope-harness -am -Dtest=TokenCounterUtilTest,CompactionBudgetTest,ConversationCompactorTest,CompactionMiddlewareTest -Dsurefire.failIfNoSpecifiedTests=false test
• mvn -pl agentscope-harness -am test
• mvn -pl agentscope-harness -am spotless:check -DskipTests

Related links

• Issue: [Bug]:上下文压缩失效导致agent运行失败 #2985
• Commit: guslegend0510@408e064
• Changes: main...guslegend0510:codex/fix-2985-compaction-token-budget

Checklist

• Code has been formatted with mvn spotless:apply
• Full Core/Harness reactor tests pass; other modules were not run
• Javadoc comments have been updated
• Related English and Chinese documentation has been updated
• Code is ready for review

[codecov]

Codecov Report

❌ Patch coverage is 94.33962% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...agent/memory/compaction/ConversationCompactor.java	87.50%	1 Missing and 1 partial ⚠️
...harness/agent/middleware/CompactionMiddleware.java	88.88%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[CryoThrust]

The new request-overhead accounting is used to decide whether compaction starts, but it is not applied when choosing the preserved tail. compactIfNeeded(..., requestOverheadTokens) calls determineCutoffIndex(messages, totalTokens, config) with conversation-only tokens, and findTokenBasedCutoff still keeps up to the full config.getKeepTokens(). The rebuilt request then adds the same system/tool/response-schema overhead back on top of that tail.\n\nFor example, with keepTokens=800, a preserved 800-token conversation suffix and 400 fixed system/tool/schema tokens, compaction can report success but send roughly 1200 input tokens before the summary itself is counted. This is especially likely for large tool schemas, which is one of the cases this PR explicitly adds to the trigger budget.\n\nCould the cutoff budget be max(0, keepTokens - requestOverheadTokens) (with a documented behavior when the fixed overhead alone exceeds the budget), and could there be a regression test asserting that the post-compaction request estimate, including system/tools/response format, stays within the resolved budget? Otherwise the change detects the overflow earlier but cannot reliably make the final request fit.