I understood very poorly this slide [изображение] If I understand correct, that if we initialize W that will be the same for each layer and if it will be less than 1 or much more, this influence of this W will be exponentially high in reducing or increasion of results of finish layers. But how i…

“Vanishing gradients” means that the gradients trend toward zero before the cost has been minimized, so the weights are no longer being significantly modified, and the cost becomes stuck. “Exploding gradients” means that the gradients become increasingly large, which increases the weights, and the …

First, understand what gradient is. Gradient is the derivative of loss with respect to parameters (W, b) (also A and Z). So, the gradient depends on the parameters. Now if our gradient updates the W to a very large number, our cost will just move back and forth and never reach the minimum. That is c…

So, do I understand correct, that the getting of vanishing/exploding gradients is possible only if in deep neuron network, for example more than 100 layers, all W in 100 layers will be less or more than zero? In this case they will influence on each other and last layers multiplies in calculations N…

[image] someone555777: possible only if in deep neuron network, for example more than 100 layers, all W in 100 layers will be less or more than zero? It’s more complex than that. Besides architecture and initialisation of weights among others also hyperparameters or the activation function pl…

[image] someone555777: So, if NN from 100 neurons for example, than if 50 of neurons will be with 0.5 W and remaining 50 with 1.5 W everything will be more less fine? Without problem of vanishing/exploding gradients? In addition to following all the links from Christian, I think there is a fu…

Vanishing gradient When training a deep neural network, the process involves calculating gradients that indicate the necessary adjustments to be made to the network’s weights to minimize the error between predicted and actual outcomes. These gradients are then propagated backward through the layers,…

So, what is vanishing/exploding gradient?

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun

Christian_Simonis May 9, 2023, 6:52pm 4

This thread might be worth a look, @someone555777: Vanishing/Exploding gradients C2W1 - #2 by Christian_Simonis

Hope that helps!

Best regards
Christian

1 Like

Topic		Replies	Views
Question on weight initialization and exploding/vanishing gradients Improving Deep Neural Networks: Hyperparameter tun coursera-platform	9	697	May 23, 2021
Vanishing / Exploding Gradients Improving Deep Neural Networks: Hyperparameter tun week-module-1 , coursera-platform	5	568	January 11, 2024
Vanishing / Exploding Gradients : week1 Improving Deep Neural Networks: Hyperparameter tun coursera-platform	3	575	June 19, 2021
The problem of expolding/vanishing Improving Deep Neural Networks: Hyperparameter tun coursera-platform	2	529	March 5, 2022
Week 1: Weight Initialization - Effect on Activations vs Gradients Improving Deep Neural Networks: Hyperparameter tun coursera-platform	1	661	June 23, 2021

So, what is vanishing/exploding gradient?

Related topics