Test set generalisation error larger than CV error?

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

Hi there, I’m wondering how you would handle a situation when your model is trained and compared with the CV set. Suppose the most optimal model is found using the training data and CV data. Then consider attempting to find the generalisation error using the test set. What would happen if the test MSE is much larger than your CV error? Would you consider your model to not be accurate? How would you go about retraining the model? What would cause this?

Thanks in advance!!

2

Likely this means either your data set was too small to learn a general solution, or your random sampling to create the training/validation/test sets was not very good.

1 Like