So, while Prof Ng’s discussion of ferreting out (and understanding the difference between) bias and variance in comparing results of your train/dev/test sets, I was at the same time somewhat surprised – Is there not some formulae or calculation that can actually give you a quantitative measure of bias and variance on your model (similar to say how you can calculate specificity and sensitivity) ? His depicition sounds very empirical.
I mean it at least seems like these measures would something calculable-- And in particular, rather than the two class binary classification he uses to illustrate, I am thinking with regards to much larger data sets with many more features that cannot be easily visualized.
Well i will try to address some points in your question and sure other mentors can respond and add more clarifications than me.
In traditional machine learning theory, there isn’t a single, universally agreed-upon formula or calculation that directly quantifies bias and variance in the same way specificity and sensitivity measure aspects of model performance. However, there are several techniques and metrics that can provide insights into these aspects of model behavior.
For instance, techniques like cross-validation can help in estimating both bias and variance indirectly. By examining the performance of a model on different subsets of the data, we can gain insights into how well it generalizes to unseen data (variance) and how accurately it captures the underlying patterns in the data (bias).
Additionally, there are metrics such as mean squared error (MSE) and its components (bias squared and variance) that provide a way to decompose the overall error of a model into these two components. While these metrics may not offer a direct “bias score” or “variance score” like specificity and sensitivity, they can still be valuable in assessing and understanding the behavior of a model.
Furthermore, in the context of larger datasets with many features, techniques like regularization can help in mitigating both bias and variance by controlling the complexity of the model. Regularization techniques like L1 and L2 regularization introduce penalty terms that discourage overly complex models, thereby helping to address overfitting (high variance) and underfitting (high bias).
@Jamal022 thank you for your input and I’ve been thinking about what you said.
I am familar with MSE and RMSE, at least with regards to traditional ML models (though I would have to think a little about how to apply these to neural nets-- if it is just the same, comparing predictions to outputs and ignoring everything that happens in between (?)).
Further I can/do understand how regularization/CV can assist in preventing overfitting.
But, I’m still a little curious if you can offer any insight/intuition as to why there isn’t any ‘universally agreed-upon formula’ for bias and variance ? I mean, conceptually and directly, on the surface at least, it doesn’t seem as if these measures should be ‘mystical’ or something.
I guess I am just not sure yet in the bias/variance concept if I am just missing/not seeing something (?).
@rmwkwok thanks for the input. I mean, hmm… Not that I love to admit at one time I took exams with pencil, paper, and a calculator (not a computer… I guess I am getting that old), the description reminds me alot of, in a traditional model, your ‘degrees of freedom’. Do you happen to know if the concepts are related ?
Hello @Nevermnd, you mean whether the concepts in the Wikipedia are related to your question of how to determine Bias and Variance quantitatively? Yes, I think so! These concepts do not change over time.
If you are satisfied with whether there are formula for computing Bias and Variance, then yes, the wikipedia page has shown you that.
If you are asking why we do it that way, then I would say, it works and is a simple way.
However, if you start to ask why not doing it quantitatively, then you will need to dive a little bit into the maths and figure out which is the difficult part. Are you ready for that?
@rmwkwok Thank you very much for setting me on the right direction/one to explore further !
Again, Prof. Ng’s depiction was rather clear and inuitive-- Yet he did not seem to lead into any direct reference has to ‘how’ this might be done, other than empirically, thus it left me a bit perplexed because it certainly seemed like concept(s) that could be directly calculated/measured.
I will follow-up with you, after looking into it, if I still have questions, but I appreciate the help/reference.
