Confusion about loss values

[Bernardo_Ramalho]

Hey everybody,

I’m currently doing the [C1_W2_Lab_1_beyond_hello_world](https://github.com/https-deeplearning-ai/tensorflow-1-public/blob/main/C1/W2/ungraded_labs/C1_W2_Lab_1_beyond_hello_world.ipynb) notebook.

In exercise 2 we are asked to check the difference between 1024 neurons and 512 neurons in the second layer. I’m getting a bit confused because I thought that the loss value would be lower and it is in the fitting process (0.0258 vs 0.0278). But when I go to evaluate, the loss function for 1024 neurons is 0.0853 and 0.0713 for the 512 neurons. To put more confusion in my head, when I ran exercise 3 which only has 64 neurons the loss value was 0.0843.

Is this normal? And if so why does it happen?

[alvaroramajo]

Hi, @Bernardo_Ramalho!

Normally, larger models take more time to optimize than smaller ones but can usually reach a better accuracy (not always). In this case, the 1024 neurons model may need more epochs to get to a lower loss value than the 512 neurons one. On the other hand, the 64 neurons model may be too simple to get a good accuracy and loss value in this dataset.