Skip to content

Add button for Attachments (Image, PDF, etc.) in Text Chat#52

Open
ashiquemukkil wants to merge 13 commits intoAzure:insidersfrom
ashiquemukkil:chat-attachment-button
Open

Add button for Attachments (Image, PDF, etc.) in Text Chat#52
ashiquemukkil wants to merge 13 commits intoAzure:insidersfrom
ashiquemukkil:chat-attachment-button

Conversation

@ashiquemukkil
Copy link

This PR introduces functionality for users to upload attachments (images, PDFs, etc.) along with their text input in the chat interface. The enhancement leverages GPT-4O's multimodal capabilities to effectively process and respond to multimedia queries.

Key Changes

  1. Frontend:
    Added an attachment upload button to the chat UI.
    Display of uploaded files as thumbnails or links in the chat.
  2. Backend:
    Upload API to upload file to blob storage.
Screenshot 2024-12-28 at 5 10 04 PM

Changes required at GPT-RAG-ORCHESTRATOR :

  • Extra plugging to process attachments
  • Need to maintain cache with attachment URL

@ashiquemukkil
Copy link
Author

@ashiquemukkil please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree

@ashiquemukkil ashiquemukkil marked this pull request as draft December 30, 2024 17:40
@ashiquemukkil ashiquemukkil marked this pull request as ready for review December 30, 2024 17:43
@placerda
Copy link
Collaborator

placerda commented Jan 7, 2025

Hi @ashiquemukkil could you list at a high-level what changes will this frontend feature requires in the orchestrator and bicep (for example attachment container creation, role assignments)?

@ashiquemukkil
Copy link
Author

Hi @ashiquemukkil could you list at a high-level what changes will this frontend feature requires in the orchestrator and bicep (for example attachment container creation, role assignments)?

Changes to the Orchestrator:

  1. Include the "file" field (representing the blob name) alongside the existing fields: conversation_id, answer, data_points, and thoughts orc endpoint. The value for "file" should be null if there is no attachment.
  2. Introduce an additional field in the Cosmos DB (cache) to store attachments.
  3. Add a new plugin to extract information from images.

bicep file change

  1. add one more container with the same privilege as 'documents', and add life cycle management to delete data after a day

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants