When we implemented Logistic Regression in Week 2 of Course 1, Prof Ng tells us that it suffices to use zero initialization for the w and b values that will be learned through Gradient Descent and we will still get a valid solution. But when we get to real Neural Networks in Week 3 with the 2 layer …

Hi! Your explanation helps a lot! Could you explain more on why using a sigmoid which “doesn’t give 0 outpu for 0 input” still can’t change the result? Is it still about math?

Yes, all this is about math. Once the output values from the hidden layers are not zero anymore, the math gets a bit more complicated. What is the case is the rows of the weights are all equal within any given layer and the result is that the outputs are all equal at each given level. Then the gradi…

I tried running gradient descent with initial random biases, and 0 weights. I tried a few different learning rates, tried scaling the biases from 10^-3 to 100, and also ran gradient descent for up to 150,000 iterations. I found that the weights were changing so slowly that they virtually had no eff…

Hi, Divy. It’s great that you are doing that. You always learn something interesting when you try your own experiments! Which exercise did you use as the vehicle for this experiment? I just tried it using the Planar Data exercise in C1 Week 3. My results were just as good as with the original initi…

I also tried this experiment with the 2 layer model in Course 1 Week 4 Assignment 2. Just using the normal distribution scaled by 0.01 for the bias vectors and not changing any of the other hyperparameters, I get 97.6% training accuracy after 2500 iterations and 66% test accuracy. Not that much wors…

Symmetry Breaking versus Zero Initialization

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning

Topic		Replies	Views
Week 1, Programming Assignment initialization, Exercise 1 - initialize_parameters_zeros Improving Deep Neural Networks: Hyperparameter tun coursera-platform	8	879	December 15, 2023
Random Initalization in Neural Networks Neural Networks and Deep Learning week-module-3 , coursera-platform	15	147	September 11, 2024
Randomly initialize parameter b instead of W Neural Networks and Deep Learning coursera-platform	6	714	August 23, 2022
Week 3 Random Initialization Neural Networks and Deep Learning coursera-platform	6	729	May 6, 2022
Course 2 Week 1 PA 1: Why does zero weight cause no change in loss Improving Deep Neural Networks: Hyperparameter tun coursera-platform	2	676	January 5, 2022

Symmetry Breaking versus Zero Initialization

Related topics