There is a task - the LLM receives a large piece of text as input - for example, several thousand lines (modern LLMs easily support such a context), in this context, the model must find a paragraph with relevant information about some area (for example, regarding healthy eating) and write an answer taking into account (mixing) the additional information (that is currently available It is presented in the form of several dozen pdf files of several pages each) i.e. there is not too much additional information.
In theory, this is a purely RAG task. But on the other hand, given the amount of context that LLM can accept as input, it may be simple to structure the information from the pdf in the form of “questions and answers” and include it in the prompt, which will look like ==> (a paragraph of text/question… at the same time, keep in mind that: (information from the pdf)), without bothering with RAG, but with the same final result.
I would like to hear the opinions of those who have used RAG before, and in general, please, any thoughts on this!
If you don’t want to disclose private data (from your PDFs) to the LLM but keep it under control of those owning the documents, that would speak for RAG. RAG is, amongst others, about data sovereignty. It would allow for a much more nuanced control of data access than an LLM, let alone using public components.
@Germaneli
Thank you for expressing your opinion!
I’ve been challenged with this question as well. If we send the extracts from the PDF files over to the LLM, the LLM already has access to the private data. If we have the embedding locally and use the LLM we still have some data sent over to the LLM and may be over a period of time most of the data that was supposed to be private has gone over to the LLM. Hence I’m not convinced that the implementation of RAG with an external LLM is viable. If we’re not concerned about data privacy we can always use the LLM directly is my view. Given this unless RAG is implemented with a local LLM I don’t believe it is truly RAG.
I agree. RAG makes most sense in a localized environment, when data sovereignty is the driver.
Another consideration would be between RAG and Finetuning/Training.
Finetuning/Training an LLM, even regularly, with additional data/inputs is costly, especially for SME the effort can be forbidden. Feeding local/added/private data into the LLM via RAG can be much more sustainable and a good alternative to constantly having to update/retrain the LLM with your data.