NLP Course 3 Week 3: Confusion matrix

Hi!

So I have a question about Exercise 4. I get the right values for the confusion matrix but some of the are at the wrong place. Same for the half of the examples in the unit test. This is what I get:

Accuracy 0.7259765625 Confusion matrix: [[1506 4876] [1300 2558]]

Expected Result

Accuracy ~0.725

Confusion matrix:

[[4876 1506]
[1300 2558]]

And

Wrong confusion matrix for threshold=0.7 and batch_size=512
Expected:[[1624 525]
[ 473 878]],
Got:[[ 525 1624]
[ 473 878]].

Wrong confusion matrix for threshold=0.75 and batch_size=512
Expected:[[1747 421]
[ 612 720]],
Got:[[ 421 1747]
[ 612 720]].

Wrong confusion matrix for threshold=0.7 and batch_size=256
Expected:[[1647 521]
[ 468 864]],
Got:[[ 521 1647]
[ 468 864]].

Wrong confusion matrix for threshold=0.8 and batch_size=256
Expected:[[1857 311]
[ 802 530]],
Got:[[ 311 1857]
[ 802 530]].

4 tests passed
4 tests failed

What could be a reason for that?
Thanks!

1 Like

Hi @Taisiya_Kopytova

Make sure that you correctly calculate and assign the values of TP, FP, TN, and FN as below:

[[TN, FP],
 [FN, TP]]

Hope it helps! Feel free to ask if you need further assistance.

1 Like

Hi @Alireza_Saei

Thank you for replying!

I use tf.math.confusion_matrix, so I can’t control the calculation of TP, FP, TN, and FN.

1 Like

Your code snippet looks correct. However, make sure threshold is set appropriately for your problem. It should reflect the threshold above which you classify a sample as positive. Also, double-check that y_test and y_pred are of the correct shape and type. They should be compatible with the operations you are performing, especially with tf.math.confusion_matrix.

If these conditions are met and the rest of your implementation is correct.

1 Like

@Alireza_Saei I’m not sure what you mean by setting the threshold correctly. If it is correct in the snippet, what else could be a problem? And given that the accuracy value is calculated correctly, the threshold is not a problem.
The shape for y_test and y_pred in the first example is (10240,).

Would you mind to check that everything is calculated correctly from your side? It looks like that for the example and unit tests TP, FP, TN, and FN were calculated without the use of tf.math.confusion_matrix and I suspect that this is where the problem could lie.

1 Like

@Alireza_Saei I believe there might be a bag in unit tests and the grader. Check another similar post

1 Like

Send your codes via personal DM. Do not post codes on public post thread. kindly follow community guidelines.
@Taisiya_Kopytova

please remove any code snippet shared here using edit option

@Deepti_Prasad Thank you for reminding me.

1 Like

Hi @Alireza_Saei

Kindly do not confirm codes are correct if you do not have access to the complete codes provided by learner, this can misguide learner as he ended responding on another post.

There is issue with his codes which does require review to his overall codes.

Regards
DP

Hi @Taisiya_Kopytova

You somehow mixed up two code lines

  1. Check if d>threshold to make predictions
    Here you were only suppose to check if cosine similarity is greater than threshold but you ended up adding the accuracy part of the code to this code line. Also do not hard code the path by introducing res = d > threshold (I KNOW THE HINT FROM PREVIOUS CELLS TELLS YOU TO DO SO.
    INSTEAD OF y_test == res, you only need to use tf.cast(d>threshold with the datatype).

  2. Next your accuracy code is incorrect
    take the average of correct predictions to get the accuracy
    accuracy = tf.math.reduce_mean(y_pred)

You need to use tf.reduce_mean to the tf.cast when your actual target is absolute equal to the prediction with the datatype.

Regards
DP

@Deepti_Prasad
Thank you very much for the explanation! It worked with the unit tests but the grader is complaining now :slight_smile:

1 Like

This could be related to other code cells @Taisiya_Kopytova, so send your assignment for code review. Passing a unit test doesn’t confirm you always will pass the assignment. There could be other issue like using global variable instead of local variable. You can share the grader output here without sharing the codes.

Regards
DP

@Deepti_Prasad I’ve restarted the kernel and rerun all cells and the problem was solved. Now the code passes the grader. So I think you’re right that it was some local/global variable issue

2 Likes

Hi @Deepti_Prasad

Sure, I will make sure to see all the code before guiding the learners and be more careful from now on! Thanks for letting me know.

1 Like