Why zero initializations fails?

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

If W and b are zeros as initialized, i understand that this neural network may reduce to a logistic linear regression, but why W and b can not be learnt as the loss function be minimized?

2

Hello @Yuan4,

Let’s use these two slides from Andrew’s lecture titled “Forward and Backward Propagation” in C1 W4:

The first one, and note that I have added a^{[l]} above the arrows of the forward propagation.

image
image961Ă—651 88.3 KB

The second one, and note that I have added some equation numbers to aid our walk-through in below.

image
image968Ă—651 125 KB

OK. Let’s walk through the forward and backward propagation to see why initializing all weights and biases to zero won’t work in this 3-layer neural network .

Forward phase

  1. a^{[0]} is non-zero. That’s fine. :white_check_mark:
  2. a^{[1]} = 0 because all weights and biases are zero. BAD. :frowning:
  3. a^{[2]} = 0 because all weights and biases are zero. :frowning:
  4. a^{[3]} = 0 because all weights and biases are zero. :frowning:

Backward phase

  1. da^{[3]} is non-zero because the errors are non-zero. :white_check_mark:
    • dz^{[3]} is non-zero by equation (1) and da^{[3]} \ne 0. :white_check_mark:
    • dw^{[3]} = 0 by equation (2) and a^{[2]} = 0. :frowning:
    • db^{[3]} is non-zero by equation (3). :white_check_mark:
  2. da^{[2]}=0 by equation (4) and w^{[3]}=0. :frowning:
    • dz^{[2]}=0 is zero by equation (1) and da^{[2]}=0. :frowning:
    • dw^{[2]}=0 by equation (2) and dz^{[2]}=0. :frowning:
    • db^{[2]}=0 by equation (3) and dz^{[2]}=0. :frowning:
  3. da^{[1]}=0 by equation (4) and w^{[2]}=0. :frowning:
    • dz^{[1]}=0 is zero by equation (1) and da^{[1]}=0. :frowning:
    • dw^{[1]}=0 by equation (2) and dz^{[1]}=0. :frowning:
    • db^{[1]}=0 by equation (3) and dz^{[1]}=0. :frowning:

Therefore, only the the bias in the 3rd layer can be updated, and the rest can’t because their gradients are always zero. In other words, our neural network can’t learn any further. :frowning: This is why we don’t want to initialize them to zeros for a multi-layer network.

However, zero initialization isn’t always failing, if we had only the output layer, then the following would happen instead:

Forward phase

  1. a^{[0]} is non-zero. That’s fine. :white_check_mark:
  2. a^{[1]} = 0 because all weights and biases are zero. :frowning:

Backward phase

  1. da^{[1]} is non-zero because the errors are non-zero. :white_check_mark:
    • dz^{[1]} is non-zero by equation (1) and da^{[1]} \ne 0. :white_check_mark:
    • dw^{[1]} is non-zero by equation (2) and a^{[0]} \ne 0. :white_check_mark:
    • db^{[1]} is non-zero by equation (3). :white_check_mark:

There is a very relevant topic called “symmetry breaking” and I am going to refer you to this excellent post by our mentor Paul.

It is a common practice to initialize the weights randomly and the biases to zero.

Cheers,
Raymond

PS: I hope I didn’t make any typo. Please let me know if there is any :wink: Thank you!

2 Likes
3

Thanks a lot! I understand now, :smile:

4

You are welcome @Yuan4!

Raymond