qa is a RetrievalQA object and we apply examples to qa and assign to predictions. We compare the real answer and predicted answers to get a grade. I’m assuming “real answer” is the truth that the LLM predicted answer compares itself to. But the “real answer” is predictions[“answer”] so where does it come from and how/why is it accurate?
The “real answer” is created by the LLM with the whole document (by selecting each to produce the QA pairs automatically for the evaluation set). The “predicted answer” is obtained through the QAChain (with the retrieval process -embeddings, vectors), entering the Q’s of the evaluation set.
1 Like