Repository navigation
Copilot image-view tool succeeds, but PNG content appears missing from outgoing inference requests #74972
Description
Summary
In a Copilot-based agentic workflow, the built-in view tool successfully opens a PNG and records an image in binaryResultsForLlm. However, the subsequent logged model request contains only the text Viewed image file successfully. No structured image content appears in the request, and the model reports that it cannot see the image.
I am reporting this here because the failure occurs in the gh-aw-managed Copilot inference path. I have not established whether the defect belongs to gh-aw, Copilot CLI, or the provider integration.
Environment
| Component | Version / setting |
|---|---|
| gh-aw compiler | v1.0.1 |
| Copilot CLI | 1.0.103 |
| AWF | v0.29.0 |
| Model | gpt-5.3-codex |
| Trigger | workflow_dispatch |
The workflow uses the Copilot engine with the firewall enabled.
Reproduction
This minimal workflow has been run and reproduces the failure.
- Workflow: https://gist.github.com/johndowns/04e3ddbd8a5f1f3f1de82cae41a5a7e3#file-png-vision-diagnostic-m
- Test image: https://learn.microsoft.com/en-us/azure/reliability/media/reliability-storage/geo-redundant-storage.png
- Save the workflow as
.github/workflows/png-vision-diagnostic.mdin a repository configured to run Copilot-based agentic workflows. - Save the test image as
vision-test.pngat the repository root. It is a 731 x 299 pixel PNG, 21,942 bytes. - Compile with
gh aw compile png-vision-diagnosticusing gh-aw v1.0.1. - Commit and push the Markdown, generated lock, and image to the default branch.
- Manually run PNG vision diagnostic.
The workflow asks the model to open the PNG with the built-in image-viewing tool and transcribe its labels, numbers, positions, and relationships. It prohibits reading other files or using OCR commands to infer the answer.
Expected behavior
The image content returned by view should reach the vision-capable model in the provider's supported image-input format.
If this inference path does not support image inputs, the workflow should surface an explicit unsupported-capability error rather than providing only a success message.
Actual behavior and evidence
1. The tool records an image result
The tool-completion event reports success and includes the following metadata. The asset ID and repository path are omitted here.
{
"content": "Viewed image file successfully.",
"binaryResultsForLlm": [
{
"type": "image",
"mimeType": "image/png",
"byteLength": 21942
}
]
}2. The subsequent logged model request contains only text
The corresponding Responses API input item is:
{
"type": "function_call_output",
"call_id": "<view-call-id>",
"output": "Viewed image file successfully."
}Across all three logged model requests in this run, there are no input_image items or other structured image payloads. These are client-side logs; I cannot independently inspect what the upstream inference service received.
3. The model reports missing visual content
The model calls report_missing_data with data type image visual content and this reason:
I need visible pixel content from vision-test.png to transcribe labels and numbers, but the image-view tool returned only a success status without any renderable image data.
The agent job itself succeeds; the workflow's conclusion job then marks the run failed. No label transcription is produced.
Questions
- Is forwarding built-in
viewimage results supported through the gh-aw-managed Copilot inference path? - Is a capability setting or configuration required to preserve
binaryResultsForLlmimage content? - If this is a Copilot CLI or provider-integration defect, which component should receive this report?
Activity
muhammadkhan0009 commentedon Nov 5, 2026
I reproduce this too, with the Copilot engine and
claude-opus-4.8on two gh-aw versions, which adds the following to your report:claude-opus-4.8claude-opus-4.8Setup. A minimal
workflow_dispatchworkflow with the Copilot engine and the firewall on. Apre-agent-stepsstep copies two 390x500 PNGs into/tmp/gh-aw/agent/. Each PNG shows a coloured header, a title and a unique secret word, so only a model that receives the pixels can answer. We gave the images two ways:@/tmp/gh-aw/agent/probe-at.pngappended to/tmp/gh-aw/aw-prompts/prompt.txt(the CLI runs with--prompt-file)/tmp/gh-aw/agent/probe-view.pngwith its file toolResult. Neither image reached the model on either version.
[Image: Image file at path /tmp/gh-aw/agent/probe-at.png [image/png] 390x500], but the model gets onlyViewed image file successfully.viewcall logs1 line read.@pathattachment in the prompt file had no effect either.The silent failure leads to confabulation. On v0.93.6 the model first called
noopwith a full, confident and wrong description:The real images were purple and green headers, with "Profile > Uploads" / "Settings > Notifications", MARIGOLD-58 / ORCHID-24, and orange circles. Only a "never guess" instruction in the prompt made it send a correction (
not visible) afterwards. In a real workflow, a screenshot-based agent can act on a screen it never saw.Possible cause. The compiled lock file runs Copilot CLI with
COPILOT_API_KEY="$COPILOT_DUMMY_BYOK"through the gh-aw API proxy. That looks like the same path as github/copilot-cli#4600, where images in tool results are silently dropped under a BYOK provider while user-turn attachments still work.Asks.
lpcox commentedon Nov 7, 2026
🔗 AWF tracking issue: #aw_track1
lpcox commentedon Nov 8, 2026
🔗 AWF tracking issue: #aw_track1
lpcox commentedon Nov 8, 2026
🔗 AWF tracking issue: #aw_track1