I am working on a problem where accuracy score on training set is around 90%. But on Dev set, I am getting an accuracy of 70%. So I understand this is a high variance problem and I need to use regularization and simplify my model.
If I choose to do that, is it possible(generally) to attain a final accuracy on dev set beyond the original 90%? Or will I have to settle for something lower?
(My classes are fairly balanced. Hence I am considering accuracy as my evaluation metric)
Thank you.
Prof Ng spends a lot of time explaining these issues in Course 3. My take would be that the first question you need to address is the 90% training accuracy. I would say that in and of itself is a high bias problem. Yes, you have overfitting relative to the dev set, but I think you need higher training accuracy first.
But maybe I’m getting ahead of myself there: the first question you need to answer is what would be an adequate accuracy on the test set, right? That’s really your final goal. Is 90% accuracy on the test set going to be adequate for your system? But even to achieve that, I would guess you’ll need to get the model to have more variance on the training data first and then you can try regularization to make the dev accuracy closer to the training accuracy.