Value of J(w) further from theoretical value, rather than closer, when reducing epsilon

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

In the lecture and course material we learn that the reason for J(w), or k, not being calculated as the exact theoretical value (e.g. 6 in the given examples) is that epsilon is not infinitesimally small. In the optional lab, we are invited to decrease epsilon further and to observe the error in calculated k vs theoretical k decreasing as a result. For very small epsilon values, I observe k to increase rather than decrease.

For example, with epsilon = 0.0000001, k = 6.000000087880153.
With epsilon = 0.000000000001, k = 6.000533403494046 .

Is this perhaps due to floating point error? Or have a misunderstood something?

2

You are right to suspect that floating-point errors play a role here. As you reduce epsilon to minimal values, the precision limitations of floating-point arithmetic in computers can cause significant problems. What you see is likely due to floating-point errors that become more pronounced as epsilon decreases beyond a certain point.

The key takeaway is that while smaller epsilon values generally lead to more accurate derivatives, this holds only up to a certain limit. Beyond that, numerical precision issues can cause inaccuracies.

1 Like
3

Thanks for clarifying

1 Like