In very deep NNs, gradients can get enormous and very close to 0, but can't they get extremally small?

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning
1

Gradients are vanishing when they get very close to 0. They explode when they get very large.
I didn’t hear anything about them getting very small (negative), but what about this:
image
The result on position [0][1] is -69.197.

2

When they talk about vanishing or exploding gradients, they mean the absolute values. -69 would be considered a large gradient value. The point is that gradients are derivatives (slope of the curve or surface). A big negative number means the surface is going steeply downward at that point.

1 Like