C3W3_Assignment: Different behaviour between unit tests and autograder

Here’s a funny one.

This happens in exercise 4: Classify.

If I write my confusion matrix in a certain way, I get the unit tests passing BUT the autograder will fail with these errors:

Failed test case: Wrong confusion matrix for threshold=0.7 and batch_size=512.
Expected:
[[1624  525]
 [ 473  878]],
but got:
[[ 525 1624]
 [ 473  878]].

Failed test case: Wrong confusion matrix for threshold=0.75 and batch_size=512.
Expected:
[[1747  421]
 [ 612  720]],
but got:
[[ 421 1747]
 [ 612  720]].

Failed test case: Wrong confusion matrix for threshold=0.7 and batch_size=256.
Expected:
[[1647  521]
 [ 468  864]],
but got:
[[ 521 1647]
 [ 468  864]].

Failed test case: Wrong confusion matrix for threshold=0.8 and batch_size=256.
Expected:
[[1857  311]
 [ 802  530]],
but got:
[[ 311 1857]
 [ 802  530]].

If I write the code in a different way, I get the following errors in the unit tests AND the grader will give a 50/50 score:

7/7 [==============================] - 1s 6ms/step
7/7 [==============================] - 0s 5ms/step
14/14 [==============================] - 0s 4ms/step
14/14 [==============================] - 0s 4ms/step
Wrong confusion matrix for threshold=0.7 and batch_size=512
	Expected:[[1624  525]
 [ 473  878]],
	Got:[[ 525 1624]
 [ 473  878]].

Wrong confusion matrix for threshold=0.75 and batch_size=512
	Expected:[[1747  421]
 [ 612  720]],
	Got:[[ 421 1747]
 [ 612  720]].

Wrong confusion matrix for threshold=0.7 and batch_size=256
	Expected:[[1647  521]
 [ 468  864]],
	Got:[[ 521 1647]
 [ 468  864]].

Wrong confusion matrix for threshold=0.8 and batch_size=256
	Expected:[[1857  311]
 [ 802  530]],
	Got:[[ 311 1857]
 [ 802  530]].

4 tests passed
4 tests failed

Just to be clear, this has very little to do with my code and much to do with the fact that the unit tests and the autograder expect a completely opposite behaviour

1 Like

The chance is that there something wrong with your code, the audotgrader and unit tests may run different tests but not oppose each, thats not possible!

As a start if you are not full pass its good to go through your code solution again!

1 Like

Hi Gent,

I’ve had this same discussion in these forums last week regarding the graded exercise in week 1. After a lot of back and forth, it turned out that the issue was with the unit tests, as I was saying from the start. This time I’m going to save myself a lot of frustration and just say this:

You can believe or not. I don’t care: I’ve already passed the course with a total score of 98.68% and a 100% for week 3. If you want a screenshot of both errors, I can send you one.
But I refuse to waste my time trying to convince you.

2 Likes

No need to convince me, I am not a judge. Why am I saying what I am saying is because if you do all the code properly then you pass both. Now if you found it out last week and a mentor already saw it, then they should have sent a github ticket and the team will take care of it in due time.

Its not impossible to have bugs there I can not say that, maybe if there is another bug but it has to be proven!

2 Likes

I am not actually the best mentor for this course but do send me the errors in private I will have a look at them and raise an issue if I think there is a problem. Thank you Miguel.

1 Like

Hi @Miguel_Alonso

You getting an error and failing a test, once in a while there can be error from the staff part, not always.

So it is totally your choice if you want to find the solution to your error and you posting this topic tells you are looking for solution, and for any programmer, the basic first step is what @gent.spah has responded to you. you choose to follow that suggestion is totally dependent on you.

Your error does show there is a problem in your code as you can see for the threshold the confusion matrix value is incorrect.

Kindly check if you have recalled pred correctly using the model.predict function for test_gen.
then for n_feat, you use shape to the previous defined model prediction.

Now here is where you probably might be needing to have another look on v1 and v2 on how you recalled.
Use the below image hint to recall you v1, v2 respectively.
image

Also cross verify your code for take the average of correct predictions to get the accuracy where you make sure your actual targets are absolute equal to the predictions

Regards
DP

1 Like

@Miguel_Alonso Just wanted to let you know that yet another time I encountered the same issue as you, and here it was said that my code snippet looks correct.

2 Likes

@Taisiya_Kopytova The respective mentors who responded is not NLP mentor. You can send codes screenshot via personal DM for review.

Also kindly refrain from posting codes on public post threads, it is against community guidelines.

Hey @Taisiya_Kopytova Thanks for that, at least I know it’s not just me :slight_smile:

The way I solved it, in case it helps, is by switching what you send in the predictions parameter of the confusion matrix function.
You can try step by step: First use only the predictions that are above the threshold, if that doesn’t work, then send the tensor after you match it with the actual predictions. In my case one of those worked while the other produced the same results you get.

1 Like