Hello,
Can you explain the benefit of having the MSE ratio of 20% or less between Training and Validation accuracy as a guardrail or evaluation criteria?
Though there are multiple hyper parameter options that can help achieve 80% accuracy on both training and validation, the 20% MSE ratio criteria limits the scope on hyper parameters and eliminates other options.
From the overall perspective, if the loss is too high, then you are doing something very wrong!
One can have high loss in training and validation and still achieve 20 points MSE difference between them. So, I don’t think it is related to measuring loss.
Instead, the loss is contained by the accuracy expectations on training and validation performance.
I hope the MSE difference is not an arbitrary measure that the DLAI team randomly introduced for no reason.
MSE is key metrics to measure or evaluate model accuracy. As you know it measures the avg. of squared differences between prediction and actual values, thus acts as an indicator of model prediction error.
In regression model, smaller the mse, better the model’s predictive accuracy.
If the MSE was measured on only one outcome, for example Training Accuracy, then it would rely upon the loss between true and predicted values of that outcome.
However, that is not the case here.
Instead the criteria used for evaluation in the notebook is a comparison of two different outcomes (in this case, Training accuracy and Validation accuracy) against each other.
For example, if the training accuracy is just 10% and the validation accuracy is just 15%, the difference of 5 between them will still pass the 20 points criteria used in the evaluation, even though both 10% and 15% accuracies are abysmal performance results.
Therefore, the evaluation of comparing two different outcomes is not a measure of loss. Hope this clears up what the question is about.
The unit test used in the notebook is measuring the difference between the training accuracy and validation accuracy as follows:
mse = mean_squared_error(np.array(acc) * 100, np.array(val_acc) * 100)
It will be helpful to know the reasons, from the designers of this course, behind the purpose of using this comparison evaluation criteria.
if you notice the unittest clearly mentions squared difference of training accuracy (here in equation it is mentioned as accuracy) so the predicted values and validation accuracy would give a measurement accuracy on how correctly the model did.
if lower value as you mentioned in created post 20% or less of mse will be considered as good model prediction between the predicted values and actual value.
by using MSE, you can fine-tune your model’s hyperparameters to achieve lower error rates, often involves cross-validation techniques to ensure that the model performs well not just on the training data but also on unseen data.
mse cannot be stated as a mandatory measure or arbitrary measure but as a metric measure to check how well your model has performed on training data by comparing it performance on the validation data, and how well it will perform on data used after it might be used by others (unseen data)
Restating the question again for other inputs or thoughts:
Lets say one is able to build about 6 different models that meet the 80% accuracy requirement on both training and validation.
Now, what is the purpose of requiring 20 point MSE difference in training and validation accuracies? In what ways this additional requirement is forcing the resultant model to be better than what accuracy requirement could not achieve?
I am optimistic that the 20 points or less MSE evaluation was not a purposeless and random graded assignment introduced by whoever designed the notebook.
Your point of question focusing on training and validation accuracy in related to the model accuracy only.
Let’s take your own example.
You mentioned one creates 6 different model with 80% accuracy on training as well as validation accuracy. But in turn this 80% accuracy must have come with the requirement of loss to be as close to as to 0.
So out of this 6 model say model 2 and model 5 had the loss closest to 0 and accuracy of model 2 in related training accuracy was 85 and validation accuracy was 89 where as for model 5, training accuracy was 81 and validation accuracy was 90
Now apply the mse equation you mentioned before and tell me which model will have least model prediction error.
Here Higher the mse value, then higher the model prediction error and vice versa.
The significance of adding mse as metric measure was also introduced for the significance in models where loss and accuracy seems to be based on desired outcome like achieving loss as close to 0 and accuracy as close to 99% and yet model failing on testing data due to complexity of features, or imbalanced data, or where the prediction features were selected based on incorrect parameters to just create a successful model accuracy on validation data but when tested on real-world data, model failed as it didn’t include all the complexity of data. Hence helping one to fine tune hyperparameter based on the required error between actual and predicted values to be as minimal as possible and also achieved the desired accuracy.
Regards
DP
Also remember MSE loss function and the MSE metric both calculate the Mean Squared Error, still differ a bit in their significance.
MSE loss is used during training to adjust the model’s weights by minimizing the difference between predicted and true values.
where as MSE metric evaluates how well the model performs on training or validation data after fitting. So the equation with accuracy is basically checking if there is no overfitting or underfitting of features of your data used in model. @rocki
The overfitting / underfitting seems to be the likely case of the MSE based evaluation rather than losses or prediction errors. @Deepti_Prasad
Correct @rocki
Hello, looks like I filed this under wrong Tensorflow course. This should be under Tensorflow Developer Professional specialization, Convolutional Neural Networks course, Week 2.
no issues, use the 
open next to header of topic and move to the right category.
In case you are not able to do, let me know. I will do it.
Just wanted you to know you can edit your created post whenever you want.
Regards
DP
@Deepti_Prasad , I was looking for that icon but can’t seem to locate it. May be I don’t have permissions. Can you please take care of it and also paste an image to show where that icon usually resides for future reference. Thanks.
moved it to right category and here is the FAQ thread on how to move a post to another category
FAQ section has all the extra details for how to use this discourse community to your better benefit. Do take out time and go through different threads or use the search tool to explore like how to create a post, can I DM directly to mentors, how to edit a post etcetera.
Keep Learning @rocki
Regards
DP