Choosing a model based on training and validation errors

In the Optional Lab for Model Evaluation and Selection, in the last part where NNs are applied for Classification, it is suggested that if the CV error is the same for 2 models (in this case NNs 2 and 3 in the workbook), we should choose the model with the lower training error, which makes sense.

But in this particular case, NN 2 is simpler than NN 3 (5 layers vs 6 layers), both give same CV error and NN 3 obviously has lower training error, being the more complex NN. So, both NN 2 and NN 3 fit the CV set equally well, but NN 2 is simpler.

So could a case also be made for choosing NN 2 rather than NN 3 because it’s simpler? Like in a real-world scenario, if 2 NNs give the same CV error, but one is far simpler than the other, wouldn’t it be preferable to choose the simpler one? (considering computational resources required etc)

Hello @TanaySontakke,

I think you have made a very good point. I would choose NN2 (with 749 trainable parameters which is less than NN3 with 869 parameters). I will share this with the course team too.

Btw, that CV set only has 40 samples and uses a discrete-type error, so it is easier for us to see the same CV error from two NNs.

Cheers,
Raymond