Bias Variance tradeoff

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
4

Hey @ajaykumar3456,

As @javier pointed out exact actions would higly depend on the model we are working on. The process is repetitive meaning that:

  1. We make a decision on what actions to apply to reduce bias or variance in the model.
  2. We estimate errors on a train set and a dev set (it sometimes called a validation set).
  3. If we are not satisfied with results, we make a decision on what actions to apply again.

To understand what problem we are facing, we estimate an error on a train set and an error a dev set and then compare these errors:

  • If training set error is high, we have a high bias problem.
  • If training set error is low, but dev set error is high, we have a high variance problem.
  • If both of the errors are high, we have high bias and high variance.

The rule of thumb to achieve a low error on the train set first. It also means that we address a high bias problem first. To do that we can:

  • Increase model capasity (e.g. increase number of hidden units and/or layers).
  • Increase mini-batch size.
  • Use additional features.
  • Just train our model for longer time.

If only we have got a low error on the training set, we start working on reducing the error on the dev set. It also means that we start addressing a high variance problem. For that purpose we can:

  • Collect more data.
  • Apply regularization techniques.
5 Likes