In the last lab of the first week, it is said “Note that the gradient points away from the minimum”, how does this statement is derived?
In my understanding, the gradient is equal to the partial derivative. Take the first picture below for example, the partial derivative(actually it is derivative since only 1 parameter) is the slope. The slope can be thought of as pointing away from the minimum, but also can be thought of point to the minimum. Thus, I am confused by this statement. Thanks!
The gradient points in the direction of steepest increase of the cost function. So it points in the opposite direction that we want to go. That’s why we multiply the gradient vector by -1 when we do gradient descent: to move in the direction of steepest decrease of the cost.
1 Like
Let me add one more perspective
Since you mentioned about the left graph, there are 2 gradients - +250 and -250. The positive/negative signs define our arrow directions on the right graph. -250 points in the negative w direction, and +250 the positive w direction. Therefore, both of them are pointing away from the minimum.
Cheers!