-
Notifications
You must be signed in to change notification settings - Fork 929
Description
User Feedback
Update the Copilot SDK to accept image inputs as blobs to replicate the direct image-copy paste experience of CLI.
The image attachment feature is supported on the Copilot CLI with "Alt + V" shortcut (github/copilot-cli#476). It would be great if the GitHub Copilot SDK had a provision to accept attachments/images into context for a lot of development use cases given the models already have multimodal support natively built into it.
Current State
The SDK supports image attachments today, but only via file paths:
await session.send({
prompt: "What's in this image?",
attachments: [{ type: "file", path: "/path/to/image.png" }],
});This means SDK consumers who have image data in memory (clipboard paste, HTTP response, canvas capture, programmatic generation) must write to a temp file first — a poor developer experience compared to the CLI's direct paste support.
Proposed Solution
Add a new "image" attachment type that accepts inline base64-encoded data across all 4 SDKs:
TypeScript:
await session.send({
prompt: "What's in this image?",
attachments: [{
type: "image",
data: base64EncodedString,
mimeType: "image/png",
displayName: "screenshot.png",
}],
});Python:
await session.send({
"prompt": "What's in this image?",
"attachments": [{
"type": "image",
"data": base64_encoded_string,
"mime_type": "image/png",
"display_name": "screenshot.png",
}],
})Go:
session.Send(copilot.MessageOptions{
Prompt: "What's in this image?",
Attachments: []copilot.Attachment{{
Type: copilot.Image,
Data: base64EncodedString,
MimeType: "image/png",
DisplayName: "screenshot.png",
}},
})C#:
await session.SendAsync(new MessageOptions {
Prompt = "What's in this image?",
Attachments = [new ImageAttachment {
Data = base64EncodedString,
MimeType = "image/png",
DisplayName = "screenshot.png",
}],
});Dependencies
This requires a runtime change first:
- Runtime issue: github/copilot-agent-runtime#4231 — adds
"image"attachment type to the schema and attachment processing pipeline
Once the runtime supports inline image data, the SDK changes are:
- Add
ImageAttachmenttype to attachment union in all 4 languages - Update
MessageOptionstype definitions - Add documentation and examples to READMEs
- Add E2E test with inline image attachment
Supported Formats
Should match the runtime's existing image support: PNG, JPG, JPEG, GIF, WebP, BMP, TIFF, ICO, HEIC, AVIF.