Skip to content

Support inline image/blob attachments for direct paste experience #683

@patniko

Description

@patniko

User Feedback

Update the Copilot SDK to accept image inputs as blobs to replicate the direct image-copy paste experience of CLI.

The image attachment feature is supported on the Copilot CLI with "Alt + V" shortcut (github/copilot-cli#476). It would be great if the GitHub Copilot SDK had a provision to accept attachments/images into context for a lot of development use cases given the models already have multimodal support natively built into it.

Current State

The SDK supports image attachments today, but only via file paths:

await session.send({
  prompt: "What's in this image?",
  attachments: [{ type: "file", path: "/path/to/image.png" }],
});

This means SDK consumers who have image data in memory (clipboard paste, HTTP response, canvas capture, programmatic generation) must write to a temp file first — a poor developer experience compared to the CLI's direct paste support.

Proposed Solution

Add a new "image" attachment type that accepts inline base64-encoded data across all 4 SDKs:

TypeScript:

await session.send({
  prompt: "What's in this image?",
  attachments: [{
    type: "image",
    data: base64EncodedString,
    mimeType: "image/png",
    displayName: "screenshot.png",
  }],
});

Python:

await session.send({
    "prompt": "What's in this image?",
    "attachments": [{
        "type": "image",
        "data": base64_encoded_string,
        "mime_type": "image/png",
        "display_name": "screenshot.png",
    }],
})

Go:

session.Send(copilot.MessageOptions{
    Prompt: "What's in this image?",
    Attachments: []copilot.Attachment{{
        Type:        copilot.Image,
        Data:        base64EncodedString,
        MimeType:    "image/png",
        DisplayName: "screenshot.png",
    }},
})

C#:

await session.SendAsync(new MessageOptions {
    Prompt = "What's in this image?",
    Attachments = [new ImageAttachment {
        Data = base64EncodedString,
        MimeType = "image/png",
        DisplayName = "screenshot.png",
    }],
});

Dependencies

This requires a runtime change first:

  • Runtime issue: github/copilot-agent-runtime#4231 — adds "image" attachment type to the schema and attachment processing pipeline

Once the runtime supports inline image data, the SDK changes are:

  1. Add ImageAttachment type to attachment union in all 4 languages
  2. Update MessageOptions type definitions
  3. Add documentation and examples to READMEs
  4. Add E2E test with inline image attachment

Supported Formats

Should match the runtime's existing image support: PNG, JPG, JPEG, GIF, WebP, BMP, TIFF, ICO, HEIC, AVIF.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions