W2_A2_Ex6 optimizing log(0) error

But it is an interesting point that in a more general case, it might be necessary to defend yourself against this “saturation” problem. In 64 bit floats, it only takes z > 36 to saturate sigmoid. In 32 floats it’s even less. Here’s a thread which discusses a general way to do that. Eventually we will convert to using packages like TensorFlow and they have sophisticated implementations for defending against this.