feat: use [Image #X] placeholder for clipboard paste and drag and drop

[jackwotherspoon]

Summary

• Replaces @path/to/image.png syntax with user-friendly [Image #1] placeholders for pasted/dropped images
• Images are inlined as base64 data when the message is sent, rather than referenced by path
  • This allows drag and dropped images to work out of the box and not have to be located within current project
• Adds validation for image size (20MB limit) and supported formats (PNG, JPG, JPEG, WebP, HEIC, HEIF)
• Users can delete [Image #N] placeholders from their message to exclude images before sending

drag.and.drop.images.mp4

Details

The previous @path approach for clipboard images had limitations:

1. Long paths cluttered the input and were hard to read
2. Users couldn't easily distinguish between file references and image attachments
3. No validation occurred until send time

The new [Image #N] placeholder system provides immediate visual feedback, cleaner input, and upfront validation.

• useClipboardImages hook: Manages image registry with sequential IDs, handles validation, and converts images to base64 parts for the API
• Race condition prevention: Uses synchronized ref/state pattern for rapid multi-image registration (e.g., drag-drop of multiple files)
• Syntax highlighting: [Image #N] placeholders are highlighted in accent color alongside commands and file references
• Image validation: Checks MIME type support and 20MB size limit before registration, with user-visible warnings

Related Issues

Newer version of #14706
Related to #5316

How to Validate

• Paste image from clipboard → shows [Image #1] placeholder
• Paste multiple images → shows sequential [Image #1], [Image #2], etc.
• Drag and drop image files → converts to placeholders
• Drag and drop mixed files → images become placeholders, non-images become @path
• Delete placeholder from input → image excluded from message
• Paste image > 20MB → shows warning, no placeholder inserted
• Paste unsupported format (e.g., .gif) → shows warning
• Unit tests pass for useClipboardImages, clipboardUtils, and highlight

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Added/updated tests (if needed)
• Noted breaking changes (if any)
• Validated on required platforms/methods:
  • MacOS
    • npm run
    • npx
    • Docker
    • Podman
    • Seatbelt
  • Windows
    • npm run
    • npx
    • Docker
  • Linux
    • npm run
    • npx
    • Docker

[gemini-code-assist]

Summary of Changes

Hello @jackwotherspoon, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly overhauls how images are handled within the CLI, moving from simple file path references to a more robust and user-friendly system. The core change introduces [Image #N] placeholders for pasted or dropped images, which are then inlined as base64 data upon message submission. This enhancement includes immediate validation for image size and format, better visual feedback through syntax highlighting, and improved user control over attached images, addressing previous limitations and streamlining the image submission workflow.

Highlights

• Enhanced Image Handling: Replaces the previous @path/to/image.png syntax with user-friendly [Image #N] placeholders for images pasted or drag-and-dropped into the input.
• Inline Image Data: Images are now inlined as base64 data when the message is sent, rather than being referenced by file path, improving reliability and ensuring the image content is part of the message.
• Pre-send Validation: Implements validation for image size (20MB limit) and supported formats (PNG, JPG, JPEG, WebP, HEIC, HEIF) before the message is sent, providing immediate user feedback on invalid images.
• User Control: Users can easily remove images from their message by simply deleting the [Image #N] placeholders from the input text.
• New useClipboardImages Hook: Introduces a dedicated React hook to manage the lifecycle of clipboard images, including registration, validation, and conversion to API-compatible parts, designed to prevent race conditions during rapid multi-image input.
• Improved Drag-and-Drop: The system now intelligently categorizes dragged/dropped items, converting images to [Image #N] placeholders and non-images to @path references.
• Visual Feedback: [Image #N] placeholders are syntax-highlighted in the input, and warnings are displayed for invalid images or during processing.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Size Change: +12.5 kB (+0.06%)

Total Size: 22 MB

Filename	Size	Change
./bundle/gemini.js	22 MB	+12.5 kB (+0.06%)

ℹ️ View Unchanged

Filename	Size
./bundle/sandbox-macos-permissive-closed.sb	1.03 kB
./bundle/sandbox-macos-permissive-open.sb	890 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB
./bundle/sandbox-macos-restrictive-closed.sb	3.29 kB
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB

_{compressed-size-action}

[gemini-code-assist]

Code Review

This pull request introduces a user-friendly way to handle image attachments from clipboard paste and drag-and-drop, using [Image #X] placeholders. The implementation includes a new useClipboardImages hook for managing images, validation for size and format, and updates to the input prompt for handling and highlighting. The code is well-structured and includes comprehensive tests for the new hook. I have a couple of suggestions to improve performance by parallelizing asynchronous operations in loops, which will improve UI responsiveness when handling multiple images.

[psinha40898]

Hey Jack!

I think that #4924 means that this UX is implemented to an extent -- but it does so through the same .gemini-clipboard directory mechanism that is already in place -- relying on file paths and regular expressions. A benefit of that implementation is how the InputPrompt handles cursor movement (expanding and collapsing).

More broadly, those transformations could easily be generalized to handle additional cases of "we want to abbreviate this type of input" although I'm not sure if that makes it a clever abstraction or a premature one.

(As an aside, #15527 would have these directories exist in ~/.gemini/tmp/${project_hash} as a deliberate extension of the normal workspace)

Notably, that current implementation does not extend that transformation beyond the InputPrompt buffer and into history. To me, shipping that fix seemed to be a correct next step. And it does not validate file size.

I just thought it could be helpful for me to articulate these observations on account of the overlapped concerns, I could be mistaken in which case feel free to ignore.

Please accept my apologies if I was mistaken.