Bias and variance , tradeoff

hi ,
please mention ,Real time example for bias, variance and tradeoff , it would better to understand

Hi there,

as an example, let‘s assume you want to predict the value of the S&P 500 in ten years.
Very roughly speaking as an example: let‘s assume 7% return per year is your assumption. So a possible model could be future_p = today_p *1.07^{t}. With t = 10 years you might estimate: 1.07^{10}= ~200% meaning we might expect a doubling in value if our model would be correct as an estimate for the value in 10 years.

What would underfitting mean?
Well: if your model would not be exponential as in our case but e.g. linear, it might be too simple to learn the complex cause effects. If a network has too few parameters or is regulated too much, this can also lead to underfitting with a strong bias. This bias can represent the systematic error in your residuum.

High variance means the exact difference: your model is too complex (and maybe not regulated sufficiently) and is overfitting on noise instead of learning the actual behaviour. Often it goes hand in hand with having too little data given the feature space (I tried to explain this here more in detail: )

An example for overfitting is using an oscillating model (e.g. a High polynomial function) for a rather linear relationship where the noise affects the model too much:

Also the Wikipedia article might be helpful:

The bias–variance dilemma or bias–variance problem is the conflict in trying to simultaneously minimize these two sources of error that prevent supervised learningalgorithms from generalizing beyond their training set:[1][2]

  • The bias error is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).
  • The variance is an error from sensitivity to small fluctuations in the training set. High variance may result from an algorithm modeling the random noise in the training data (overfitting).
    Bias–variance tradeoff - Wikipedia


To further Christian’s point, the bias-variance tradeoff is typically an issue for classical ML, where an increase in model complexity causes a decrease in bias but an increase in variance. This increase in variance can be somewhat alleviated by adding a regularization term to the cost function even in classical machine learning (at a cost of losing some information). That being said, I think the main point of Andrew Ng’s videos in week 1 is to say that deep learning has a lot more tools than just adding regularization terms for handling bias and variance, and thus can more effectively get a handle on both of them.