Hello
Which best precision can be expected from a RAG ?
I mean, given a test set of N Questions & Expected answers, of course being representative of the questions the RAG is supposed to handle,
Precision = {good answers}/N %
I am aware this question applies to a multiplicy of situations
Still, a range could be useful
Thank you
hi @asalem66
what do you mean by which best accuracy i.r.t. RAG?
are asking about the approach of using semantic or keyword search or to hybrid search of re-ranker or RRF?
PLEASE explain your query briefly i.r.t. what accuracy you are asking for comparison.
Regards
Dr. Deepti
There is a good reason why accuracy is not an appropriate measure for information retrieval problems. In almost all circumstances, the data is extremely skewed: normally over 99.9% of the documents are in the nonrelevant category. A system tuned to maximize accuracy can appear to perform well by simply deeming all documents nonrelevant to all queries. Even if the system is quite good, trying to label some documents as relevant will almost always lead to a high rate of false positives. However, labeling all documents as nonrelevant is completely unsatisfying to an information retrieval system user. Users are always going to want to see some documents, and can be assumed to have a certain tolerance for seeing some false positives providing that they get some useful information. The measures of precision and recall concentrate the evaluation on the return of true positives, asking what percentage of the relevant documents have been found and how many false positives have also been returned.
Hello
I mean, given a test set of N Questions & Expected answers, of course being representative of the questions the RAG is supposed to handle,
Precision = {good answers}/N %
Thank you
This is a duplicate posting. Closing the thread.
RAG more uses
probability and top-k like features to select the best answer.
if you ask me with respect to precision, then also it would be still be based on probability of answers being as most relative to the temperature which selects the exact or top answer if temperature is 0 and if temperature is 1 the variability of response will differ.