Hi all, as in the title, what do we do when we have both high variance and a high bias when building ML algorithms? I think in the lectures solutions were only discussed with respect to high variance and high bias separately but some of them seem to be contradictory if we apply them all. Thanks in advance!
Hi @Jerry_Zheng,
For example, when we have a high bias problem, we increase the size of our NN, but if we over-increase it, then it can become a high variance problem, so we need to balance. It is like in gradient descent we take a small step at a time hoping not to overshoot the optimal weights (parameters), it is very similar in hyper-parameters tuning (e.g. adjusting NN size) that we don’t want to overshoot.
Now we have many tools for high variance, and many tools as well for high bias. Among them, probably the one that is unlikely harmful is collecting correctly-labeled and representative data. For the rest, we need to fine tune them such that we get the lowest gap between the costs of train and cv sets; and also the lowest cost.
Now focus on your question, if you have both high bias and variances (or just one of them), do the error analysis to establish an understanding of your dataset. For example, if there is a substantial difference between your training and cv sets, depending on whether the difference is foundamental, you might need to add data to your training set, or hope a larger NN can discover more common features between the two datasets. Of course, even if you think your datasets are fine, you still have a table full of bias and variance reduction techniques that you have learnt in the course for your exploration.
If you make a change out of the error analysis, then train a new model and see what problems are left, and repeat this process, then hopefully after some rounds, you will solve both the high bias and high variance problem. There is no fixed rule as to which problem should be solved first because it is an iteration process anyway, and it can also be problem-dependent.
Lastly, even if your costs gap is close to zero and the cost looks pretty good, and you can conclude that you have no high bias and variance problem, you might still want to further fine-tune to push the limit. We cannot always know what cost level is good enough.
Cheers,
Raymond