My intuition is that GD would converge faster if you update the parameters one by one because the second update will be more “accurate” if it considers the first parameter update - the first parameter update will slightly change the shape of the curve for the second parameter. Of course, it’s compu…

Hey @xl0 , Welcome to the community. Computational Efficiency is indeed one of the major reasons why simultaneous updates are better than non-simultaneous updates. But if we keep that aside, it’s hard to judge which one is better and why. I guess I agree with your point of view that [image] xl0…

Hello @xl0 and @Elemento , this is a very interesting discussion. My two cents, and definitly not an answer but merely some inputs from me: In short, I think updating all parameters at the same time is the default when we have no knowledge about the parameter space except for the first order gradien…

Hey @rmwkwok , The following point that you have made seems interesting [image] rmwkwok: In short, I think updating all parameters at the same time is the default when we have no knowledge about the parameter space except for the first order gradients. I am assuming this means that if we hav…

Interesting. I need to think about it. But first of all, why are the optimized b not -10, but 0 in both approaches?

An interesting discussion – My 2 cents Lets go back to what Gradient Descent does: At every point on the J Vs (w,b) curve, we are trying to find the steepest descent. So, wherever we are CURRENTLY on the Cost curve there is a SPECIFIC VALUE for w,b - And from that EXACT POINT, we are in the quest …

@Elemento , I made some changes. Changed your code cell #4 into this # Number of examples m = X.shape[0] # Initial Values init_w = np.random.rand(2, 1) init_b = [0] # Number of iterations num_iters = 30000 # Learning Rate lr = 1e-2 X_mean = X.mean(axis=0) X_std = X.std(axis=0) X = (X-X_mean)/…

@Elemento I updated my last reply. Please take a look.

[image] Elemento: This raises another query in my mind. Do you think that the knowledge of the parameter space would affect the sequential order in which we should update the parameters, for instance, updating w2, then b, then w3 or some order of similar sorts? I think the graph I shared in m…

Hey @rmwkwok , Thanks a lot for pointing out the errors that I made in this kernel. [image] rmwkwok: dj_dw_0 = (1/m) * np.sum(X[ : , [0]] * err) dj_dw_1 = (1/m) * np.sum(X[ : , [1]] * err) These indeed escaped my attention completely. Leaving out on scaling was intentional although. Performi…

Why is it important to update both parameters at the same time?

Course Q&A Machine Learning Specialization Supervised ML: Regression and Classification

rmwkwok June 22, 2022, 1:39pm 11

Hey @Elemento,

Here’s the deviation for the case of one w, but the same logic applies to more.

1 Like

Interpreting coefficients of scaled features

Topic		Replies	Views
Gradient descent in NN: Order of updating weights in different layers Advanced Learning Algorithms week-module-2	16	1070	August 31, 2023
Course 1 week 2 gradient descent Neural Networks and Deep Learning coursera-platform	2	598	September 24, 2021
Multiple Linear Regression Supervised ML: Regression and Classification week-module-2	2	625	July 1, 2022
Simultaneous update of parameters w & b in Gradient Descent in Multiple Linear Regression Supervised ML: Regression and Classification week-module-2	5	686	May 30, 2024
Gradient Descent Query Advanced Learning Algorithms week-module-3	1	351	September 16, 2023

Why is it important to update both parameters at the same time?

Related topics