Derivative not defined when z = 0

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning
1

Hi Sir,

In Lecture video (Derivation of Activation functions week 3), in the below bold statement we had some doubts. can u please help to clarify ?

  1. why it was talking about like z very small value like 0.00000…is this small value not exactly equal to 0 value. When we do implementation in software, is the lecturer trying to say we dont end up z exactly equal to zero value but end up with very ssmall value? Both 0.00000 and 0 are same value right… why the context small value meaning here and not getting intutition behind it ?

But you can think of it as that, the chance of z being exactly 0.000000.It’s so small that it almost doesn’t matter where you set the derivative to be equal to when z is equal to 0

2

Hi Anhu: I’d add that the gradients we get are always estimates from the sample of cases in the current batch (you’ll hear more about batches of examples in course 2). There’s a low probability of hitting “exactly” the 0.0 value, but when that does inevitably happen, it’ll just be a small perturbation on the gradient used in that step of gradient descent.

1 Like