feat(dashscope): add dashscope_image_to_image (image editing) tool to DashScopeMultiModalTool - #2995
Open
tanzzj wants to merge 8 commits into
Open
feat(dashscope): add dashscope_image_to_image (image editing) tool to DashScopeMultiModalTool#2995tanzzj wants to merge 8 commits into
tanzzj wants to merge 8 commits into
Conversation
…r dashscope_image_to_image - default the edit model to qwen-image-edit instead of a text-to-image model - reject models that cannot consume an input image on the multimodal-generation endpoint locally, with an actionable hint, so the agent does not silently fall back to text-to-image - drop the size parameter for models that do not support an explicit output resolution (plus / max / 2.0 series only) - inline an existing local file as a Base64 data URL, since this endpoint rejects file:// - keep the request to a single user message, which the endpoint requires, and surface the model-side text refusal as the error message
… and result blocks Adds 11 unit tests plus a nested ImageEditGuardTests block that exercises the private model predicates by reflection. The request-shape tests assert the built MultiModalConversationParam (model fallback for a blank model, single user message, image then text content order), all four input forms (http URL, local path, oss URL, base64 data URL), the output-size capability gate, and the refusal / empty-response / error paths. The unsupported-model case asserts no conversation object is constructed at all. Also adds two E2E cases: a real edit call and a zero-cost guard check that returns before the request is built.
…nerations The edit family changed naming with its second generation: qwen-image-2.0 and -pro drop the -edit token, so the guards listed that version as a literal. A later generation therefore matched neither branch and was refused as text-to-image only - a hard outage for a working model, attributed to a reason the message states incorrectly. Detect the generation by its leading digit instead, so an unreleased model needs no code change, and keep refusing the versionless text-to-image members and the wan family, whose image editing runs on a different task API. The output-size gate stays an explicit list on purpose, in the opposite direction: an unexpected size fails the whole request, while a dropped one only degrades the resolution to that of the input image. Also move the model guard ahead of resolveOutputSize, which logged a warning about dropping a size for a request that was about to be rejected without being sent.
|
|
tanzzj
changed the title
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
The additive tool shape and local model guards look well-scoped. One security/privacy concern: the new DEBUG log records |
|
/check |
…e debug log image_url is an advertised input form as a Base64 data URL, so the previous line wrote the entire encoded image into the log - megabytes per call, and in the clear for anything that ships logs elsewhere. The prompt is user content and has no better claim to being logged; a signed object URL also carries credentials in its query string. The call site now renders only the reference kind, the reference length and the prompt length, with no prefix of either value, and the method documents that as its logging contract. Covered by two tests. Note the summarizer is asserted directly rather than through a captured LoggingEvent: the module test classpath has slf4j-api with no binding, so the logger is a NOP and nothing can be captured without adding a logging backend as a test dependency.
…example This example is the only place in the repository that names each multimodal tool and pairs it with a copy-paste prompt, and it had drifted behind the class it documents. The doc tree has no equivalent per-tool surface to extend: docs/v1 enumerates capabilities per tool class and its import path no longer resolves after the 2.x layout, while the v2 DashScope page covers model wiring only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AgentScope-Java Version
2.0.3-SNAPSHOT(branch baseorigin/main@c32de522; local<revision>in the rootpom.xml)Description
Background
DashScopeMultiModalToolcovers text-to-image, image-to-text, text-to-video,image-to-video, first-last-frame-to-video, TTS/ASR and video-to-text, but has no
image-to-image (image editing) tool. An agent asked to edit a user-provided image
("colorize this sketch", "replace the background") therefore has no tool that takes
an input image plus a prompt, and falls back to
dashscope_text_to_image, whichredraws from a textual description of the image. The original composition is lost and
the call still reports success, so the degradation is silent.
Changes made
One file, additive:
agentscope-extensions-model-dashscope/.../dashscope/tool/DashScopeMultiModalTool.java@ToolmethoddashscopeImageToImage(...)registered asdashscope_image_to_image,parameters
image_url,prompt,model(optional),size(optional),use_base64(optional).MultiModalConversation(themultimodal-generationendpoint) with content[{image: ref}, {text: prompt}]and defaults toqwen-image-edit; returns anImageBlockwith
URLSource, orBase64Sourcewhenuse_base64=true— the same result shape asdashscope_text_to_image, so formatters and downstream consumers need no change.rejectsImageInput(model): thewan*family and the text-to-image members of theqwen-imagefamily cannot consume an image on this endpoint, so they are rejectedlocally with an actionable message instead of an opaque service-side parameter error
(without this the model retries with a text-to-image model and the silent degradation
comes back).
user; a leadingsystemmessage ora second message is rejected by the endpoint.
resolveOutputSize/supportsOutputSize: onlyqwen-image-edit-plus,qwen-image-edit-maxand theqwen-image-2.0series accept an explicit resolution, sosizeis dropped with alog.warnfor other models rather than failing the whole call(output resolution then follows the input image).
toEditableImageRef: an existing local file is inlined as a Base64 data URL, because thisendpoint rejects the
file://protocol that the vision tools use viaMediaUtils.urlToProtocolUrl().textfield of the same content map is surfaced as theerror message (that is where a model-side refusal lands).
descriptionupdated to state the input formats and tosteer back to
dashscope_text_to_imagewhen there is no input image.How to test
Static / unit (no key required):
mvn spotless:apply -pl agentscope-extensions/agentscope-extensions-model/agentscope-extensions-model-dashscope mvn test -pl agentscope-extensions/agentscope-extensions-model/agentscope-extensions-model-dashscopeManual, against the real service:
export DASHSCOPE_API_KEY=sk-xxxthen register the tool on an agent and ask "edit this image: , make it
night-time". Expected: one
ImageBlock(URLSource)in the tool result and a visibly editedimage. Negative cases worth checking:
model=wanx2.1-t2i-turbo-> local rejection with thehint message;
model=qwen-image-edit+size=1024*1024->sizedropped with a warning anda successful call;
image_urlpointing at a local file -> inlined as Base64 and accepted.Also verified in a production deployment (an application that consumes this module via
agentscope-extensions-model-dashscope), where the tool replaced silent text-to-imagefallbacks for attachment-based edits.
``
Not included
The tool tables under
docs/v1/en/**/tool.mdanddocs/v1/zh/**/tool.mddonot list
dashscope_image_to_imageyet. Happy to add them here if you'dprefer that over a follow-up PR.
Checklist
Please check the following items before code is ready to be reviewed.
mvn spotless:applymvn test)