RAG prompt - how and where

[KhaoticMind]

Hi people,
One question that I have is where to put the RAG prompt, instructing the LLM to answer questions ONLY based on the sources.

Should it be a system prompt at the benining or should we pass it at the end of the chat history as a new message? Or really edit the last user message to insert the formated prompt?
If at the end can it be a new "system" prompt or it will only work with a "user" prompt?

Also, any pointers on good proven RAG prompts that will ground the LLM?

Thanks!

[pamelafox]

In the RAG apps that I've worked on, we put the instructions in the system message, and then the sources itself in the final user message, after the user question.

Here are the system prompts we've used:
https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/app/backend/approaches/chatreadretrieveread.py#L58

https://github.com/Azure-Samples/rag-postgres-openai-python/blob/main/src/backend/fastapi_app/prompts/answer.txt

I have done evaluations with variations on those prompts, but none with particular better results.