Gradient Descent in Built-in Python/R Libraries for Models

Dear all,

After going through the Week-1 lectures on gradient descent in linear regression, I would like to know if the built-in models e.g. Linear Regression in Statsmodel library in Python or an lm() function in R always implement gradient descent automatically or we have to implement it on our own separately? Highly appreciate if anyone can clarify.
Thank you!

Hi Ghalib,

Thanks for the question! I know Statsmodel uses methods like SVD and coordinate descent to solve Linear regression, but I am not sure where they will use gradient descent. SVD and coordinate descent are other approaches for the problem. In order to solve a linear regression problem, given those packages, you don’t have to implement your own gradient descent unless you want to because those existing methods are all possible for the job.

Ofcourse we need to implement our own gradient descent for the assignments of our courses. :wink:

I never use R so I can’t comment on that :slight_smile:

Cheers!
Raymond

Hi Raymond,

Thank you very much for your reply. This is very helpful.
However, I am still not clear about the advantage of using gradient descent in linear regression cases. For example, I have now implemented on the same datasets linear regression model in Scikit Learn with simple OLS Regression as well as with Stochastic Gradient Descent Regression. Besides, also implemented the gradient descent function provided in the course 1. I am getting the same values for the estimators. Appreciate your feedback.

Best regards,
Ghalib

Hi Ghalib,

It’s great to hear that you are trying out different packages and you are able to obtain the same result.

The advantages are (1) it works; (2) it’s scalable to a very large dataset while classical OLS method can’t - unless you have unlimited memory; (3) it’s less stochastic which means your cost/weights can converge more smoothly.

Also, our MLS is for preparing us for breaking into AI, and gradient descent is a method you can bring over to solving neural networks, so we learn it. If you take the deep learning specialization, you will hear about gradient descent, mini-batch gradient descent and stochastic gradient descent again.

Keep trying, and cheers,
Raymond