In very deep NNs, gradients can get enormous and very close to 0, but can't they get extremally small?

Gradients are vanishing when they get very close to 0. They explode when they get very large.
I didn’t hear anything about them getting very small (negative), but what about this:
The result on position [0][1] is -69.197.

When they talk about vanishing or exploding gradients, they mean the absolute values. -69 would be considered a large gradient value. The point is that gradients are derivatives (slope of the curve or surface). A big negative number means the surface is going steeply downward at that point.

1 Like