feat(fetch-url): support fetching images as multimodal content#2011
feat(fetch-url): support fetching images as multimodal content#2011bj456736 wants to merge 1 commit into
Conversation
When a URL points to an image (content-type: image/*), the fetcher now
reads the binary body, base64-encodes it, and returns it as an
UrlFetchResult with kind='image'. FetchURLTool detects this and emits
a multimodal ContentPart[] (text + image_url) so the model can see the
image directly instead of receiving an opaque binary blob description.
Changes:
- UrlFetchKind: add 'image'
- UrlFetchResult: add optional imageData { mimeType, base64 }
- LocalFetchURLProvider: detect image/* responses, read arrayBuffer,
return base64-encoded image data
- FetchURLTool: when kind === 'image', return ContentPart[] with
image_url instead of plain text output
- fetch-url.md: update description to mention image fetching
- Tests: add coverage for image response handling in both provider
and tool layers
Fixes: Fetch 工具支持把图片也拉下来
|
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ebb3ae1ff2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| // Image responses: read binary body and return as base64 for direct | ||
| // model consumption. | ||
| if (contentType.startsWith('image/')) { |
There was a problem hiding this comment.
When the fetched URL returns any image/* such as image/svg+xml, image/avif, image/heic, or even the alias image/jpg, this branch marks it as an image and FetchURLTool later persists a data image_url with that MIME. The repo's shared image policy says only PNG/JPEG/GIF/WebP are allowed and that unsupported inline images make later provider requests fail (packages/agent-core/src/tools/support/image-format-policy.ts:4-8 and :169-175), while MCP results already refuse these at packages/agent-core/src/mcp/output.ts:146. Please run the same MIME normalization/sniffing gate, or return a text notice, before returning imageData.
Useful? React with 👍 / 👎.
| // Image responses: read binary body and return as base64 for direct | ||
| // model consumption. | ||
| if (contentType.startsWith('image/')) { | ||
| const arrayBuffer = await response.arrayBuffer(); |
There was a problem hiding this comment.
For image/* responses with Content-Length greater than maxBytes, this line buffers the entire body with arrayBuffer() before the size check runs. That bypasses the pre-buffer rejection that still exists for text responses below, so a large image can consume memory/bandwidth or OOM the process before being rejected; move the existing content-length guard above the image branch or duplicate it here.
Useful? React with 👍 / 👎.
| if (kind === 'image' && imageData !== undefined) { | ||
| return { | ||
| output: [ | ||
| { type: 'text', text: `Fetched image from ${args.url}` }, | ||
| { | ||
| type: 'image_url', | ||
| imageUrl: { url: `data:${imageData.mimeType};base64,${imageData.base64}` }, | ||
| }, |
There was a problem hiding this comment.
Implementing this only in packages/agent-core misses the shipped kap-server path: I checked packages/kap-server/package.json:26 (depends on @moonshot-ai/agent-core-v2) and packages/agent-core-v2/src/app/web/tools/fetch-url-types.ts:13-18 / fetch-url.ts:61-78, where UrlFetchKind is still only passthrough | extracted and the tool still renders text. As a result, kimi web/kap-server users fetching an image URL still get the old HTML/text extraction behavior, so this feature won't work in the default CLI/server environment unless the v2 FetchURL provider/tool is updated too.
Useful? React with 👍 / 👎.
1 participant
Problem
The FetchURL tool only handles text-based responses (HTML, plain text, markdown). When a URL points to an image, the tool falls through to the HTML extraction path, producing meaningless garbage output instead of the actual image.
This addresses the feature request: Fetch 工具支持把图片也拉下来.
Solution
Extend the fetch pipeline to detect image responses () and return them as multimodal content that the model can view directly.
Changes
Type system ():
Provider ():
Tool ():
Documentation ():
Tests:
Why this approach