In gradient descent algorithm, we update parameters like this:
w = w-αdJ/dw
b = b-αdJ/db
What is the exact role of those partial derivatives?
Does the actual value(or magnitude) of these derivatives matter?
Or is it just representing which direction to take a step?
Hello @allegro6335,
Well the partial derivatives play a crucial role in determining how to update the parameters (w and b) of a machine learning model. it’s simply provide information about the slope or gradient of the cost function (J) with respect to each parameter (w and b) at a specific point in the parameter space. So If the derivative is positive, you move in one direction; if it’s negative, you move in the opposite direction.
Now by coming to your second part of your question “Does the actual value(or magnitude) of these derivatives matter?”
The answer is “Yes it matters” because it determines how big or small your steps should be when adjusting parameters. Larger derivatives suggest larger steps, and smaller derivatives suggest smaller steps.
So, yes, the actual value (magnitude) of these derivatives matters because it helps control the size of the steps you take during parameter updates, which affects how quickly your model learns and converges to a good solution.
I hope it makes sense now,
Regards,
Jamal