Measuring 'errors' in NN

AI Discussions
1

So this has been kind of nagging at me only because it hasn’t directly come up (at least yet) in the DLS class.

But I’m guessing we can still measure ‘error’ (MSE, RMSE, MAE, and/or your Confusion Matrix for multiclass) still dealing with neural nets, no ?

At the same time I have come to the point where ‘we don’t have to write all our functions’.

So if someone can point me to a TF function, or something in Scikit, based on the class that we’ve been taking… Well then ‘bonus points’.

I just want to ensure I am making an ‘apples to apples’ comparison.

2

Here is the top level page for loss functions in TF. I found that by googling “tf loss functions”.

If you scan the list, you’ll find that they provide many different ones, including quite a few variants of cross entropy loss, mean squared error and many more.

But the higher level point here is that what kind of loss function you choose to use depends on the nature of the model you are building. If the model is making binary or categorical classifications, then you need an appropriate variant of cross entropy loss. If your model is a “regression”, meaning that the output is a prediction of some continuous numeric value like a stock price or a temperature, then you need a “distance” based loss function like MSE or MAE.

The only reason we haven’t seen MSE used up to this point in DLS is that everything we’ve done so far are classifiers, not regressions.

One other high level point to make about classifiers is that loss is not really the “gold standard” in any case: it is prediction accuracy. For a classifier, you can directly evaluate that and express it as a percentage. But having an appropriate loss function is critical for achieving good accuracy, because it drives backpropagation, of course.

1 Like
3

https://www.tensorflow.org/api_docs/python/tf/math/confusion_matrix

1 Like
4

@paulinpaloalto Actually Paul, perhaps you have necessarily scrambled my brain here a bit.

I mean I knew/understood the importance of loss functions in NN’s. That is your ABCs.

I am just thinking about how do we judge the model ‘postmortem’, as compared to some other ML technique.

So reading back on what was said, yes you always have ‘accuracy’. That is kind of simple and easy…

But we also want to know how these errors are distributed don’t we ?

5

Well, my high level point was if you want to judge the quality of the trained model and you trained it using a cross entropy loss function, because it is a classifier, then exactly how is a different loss function like MSE relevant to that goal? If MSE is relevant, because you were training a regression model, then you would have used MSE as the loss function to drive back prop and how would cross entropy loss be relevant in that case?

For a classifier, you use accuracy as the way to assess the actual performance of your model once it is trained.

Yes, there may be some additional insight you can gain by analyzing the accuracy on a more fine-grained level. You can look at metrics like precision, recall, sensitivity and F1 scores, which do versions of that. Those are looking at false negatives, false positives and the like and combining them in various ways. Depending on the purpose of your model, you may care more false negatives than false positives or vice versa. E.g. if you are doing medical diagnosis for some disease like cancer, a false negative may have life-threatening consequences. If you’re just making movie recommendations, then “not so much”. Well, it’s not life-threatening either way in that case, but maybe a false positive is actually more to be avoided there.