Why is my ANN's metric value on the last epoch different than the metric's score after predicting on the same training set?

Hi, below you can see my neural network where the metric in question is ROC_AUC, and you will notice that after the final epoch the AUC score is 0.8849, but when I run my NN on the same training set that it trained on it gives a slightly different AUC score of 0.887…


I would like to know why this is the case?
I would expect to receive the same AUC score on the training set since I am using the same dataset the model was trained on.

I of course understand that the test set will have a different value.

Thank you :slight_smile:

1 Like

It might be possible for your model to perform better on the test set than on the training set, if the test set is a much simpler version of the training set, or better say the distribution on the test set is similar to the training set but not as much variety as it exists on the training set.

1 Like

Thanks, I know this. But, my question has nothing to do with the test set.

1 Like

Where does your function roc_auc_score() come from?
Do you know that it computes AUC the same way that TensorFlow/Keras does?

1 Like

It comes from sklearn.metrics. That’s a good point. I guess it must compute it slightly differently. I wonder why though because it is a mathematical formula at the end of the day so I would think that that would not change between different libraries.

1 Like

Hello @Daniel_Fourie,

Perhaps it’s because one number was accumulated as we train the model, while another was computed after training the model? Consider that the states of the model were different, even the same metric should result in different values? What do you think?

Cheers,
Raymond

1 Like

I understand what you are saying, thanks. I thought that after the epoch is done then the auc metric is calculated, i.e. when the model is finished training. That’s why I thought it would be the same value. When you say accumulated, how does it work exactly? Is the metric auc calculated after every batch and then after all batches are done for the epoch the average auc for that epoch is given?
Thanks

Hello @Daniel_Fourie

That’s the idea. For the exact work flow, you will need to read the code implementation (starting with “view code at github” in this link). Based on the doc’s description, it may not accumulate auc value directly, but the auc’s four contributing factors - true/fase positives/negatives.

Cheers,
Raymond