C2W3 Methods for Model Selection

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

I am currently learning the third video for the subchapter Advice for applying machine learning. I know that to select a model, one of the ways introduced here is to divide three subsets (Training, Testing, and CV). We then compare the training and CV costs and select the best model to use. After that, we can then test the performance of this chosen model.

So, what if our chosen model performs poorly on the testing cost? Is this the best model that we have compared to all those eliminated models in the previous stage and should just stay with it; or is there a chance that some other models in the previous stage can turn out to be a better model than this initially chosen one?

2

A few thoughts:

  • If you have overfit the training set, you can fix that by increasing the regularization during training. But the results from the CV set should have caught that issue.
  • Perhaps you got an unlucky random split when you created the test set. You could try randomizing the splits again and start over on training and optimization.
  • It may be time to consider a more complicated model. Perhaps add some new features by combining existing features in a non-linear manner.
  • Perhaps you don’t have enough data to create a reliable model. If you can’t collect more data, you can perhaps expand your dataset by creating augmentations.
1 Like