Gradient descent convergence test

Course Q&A Machine Learning Specialization Supervised ML: Regression and Classification
1

Instead of choosing a small value of epsilon, why is it not desirable to calculate the rate of change of the cost value change as a slope or differentiation? If we do that - then can we have a mathematical limit x that would always guarantee convergence?

I understand that a small epsilon is an empirical approximation of the same but isn’t there a mathematical way to model it?

2

It’s a perceptive question! I’m not a mentor for MLS and don’t know what Professor Ng says about convergence there, but this is discussed also in DLS (the next series you probably would take after MLS). There are a couple of high level points worth mentioning:

  1. There is no such thing as guaranteed convergence once you get to real Neural Networks. The solution surfaces are too complex. Here’s a DLS thread about this.
  2. Yes, there are more sophisticated techniques for dynamically managing the incremental learning rate based on derivatives and other similar techniques. If you want a preview of some of those ideas, try googling “Adam optimization” and get an idea how that works. Adam is one of the algorithms covered in DLS C2.

So “hold that thought” and take DLS next and you’ll learn more about all that.

3

Yes you could. But it will only work if you know the cost function is convex.
And you still have to set some arbitrary threshold of what “good enough” is.

So this has the same limitation as watching the decrease the cost value.

4

Thanks. Very helpful. And it totally makes sense to model the learning rate itself as a function with its counterbalanced parameters!