Module2 Lab: Retrieval metrics: Recall: false negatives calculation unclear

Course Q&A Retrieval Augmented Generation
1

Hi.

Module2 Lab: Retrieval metrics:
The false negatives calculation in the given code works out to
true_positives - true_positives, which will always be 0. Therefore, the Recall will always be 1.0

What am I missing?

I am referring to the following code snippet in the jupyter notebook:

false_negatives = sum(
             newsgroups_train.target_names[df.iloc[idx]["category"]] == desired_category 
             for idx in top_results
         ) - true_positives

Thanks in advance for your help.

1 Like
2

I believe this is an error in the code. To calculate false negatives, we need to find documents that have the correct category, but are not included in the top k. So the expression should not sum over top_results, in my opinion, but over all documents, and then subtract the true positives.

1 Like
3

Thank you! That clears it up.

4

I wish I saw your post earlier. I just posted about the same issue one minute ago. I think there is an error in the way how false negatives were computed in the provided lab. Happy that I am not the only that noticed the issue. Link to my post: Computer false negatives in C1M2_ungraded_Lab_2

1 Like
5

I hear you. @janhendrik and I believe that the false negatives calculation (and, therefore, recall) in the provided lab is not how they are calculated in the practical world. Perhaps the lab is using an assumption for some reason that might have to do with the top_k parameter in the sense that if top_k is a larger number then there would be fewer false negatives. But this line of thinking is… impractical?

I guess we can hope that one of the mentors will clear this up sooner rather than later. Till then,

false negatives = all desired_category docs in dataframe - true positives

6

@Community-Team, who is the lead technologist for the RAG course?

1 Like
7

Hello @TMosh ,

@lucas.coutinho is OOO. I will reach out to the team.

8

Hi!

Yes, there is an error in the code - in fact it should be Precision@K and Recall@K instead of just precision and recall. We’re working on updating the ungraded lab with this new version.

Thanks,

Lucas

1 Like