In Week 3’s Model selection and training/cross validation/test sets video, Prof. Ng suggests choosing the model with the lowest cross validation set cost, and then seeing if this model generalizes well to the test set. My question is: Are there scenarios where the cross validation set cost is the lowest for a particular model but the cost calculated for the test set is not the lowest? (This would mean another model won’t have the lowest cost for the cross validation set but have the lowest cost for the test set.)
Thank you for your response. So, what you mean is, when we choose the model with the lowest cross validation set cost, this usually will lead to a test set cost that is “good enough” for us to be confident in using the model even though test set cost might not be the lowest?
You should verify that the test set cost is good enough. Roughly equal in magnitude to the validation and training set costs is a good goal. Some small variation is possible, due to statistical differences in the data sets.
Decisions like this are guided by your experience. There aren’t any hard-and-fast rules.
