Hello, I have a question about numerical approximation. What happens if we have ReLu function which is non-differentiable at 0? Could we still use the two-side difference to calculate the derivative since the cost function is non-differentiable at some points? Thank you.

best,

Right, for our purposes the non-differentiability of ReLU at 0 doesnâ€™t really cause any problem. You can use either 0 or 1 as the value of the derivative at 0 and everything just works. Also ReLU does have a value at 0, right? So the â€śfinite differenceâ€ť approximation of the derivative works with no problems.

2 Likes