Course 2 Week 1 PA 1: Why does zero weight cause no change in loss

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

The notes in the week 1’s first programming assignment state that initializing the weights to zero prevents the neural net from breaking symmetry. It thus becomes like a logistic regression. When the code is run, the loss does not decrease. So logistic regression loss will not decrease if weights are initialized to zero.

However, in course 1 we initialize our weights in logistic regression to zero, and we get a decreasing loss.

Why is this?

2

As always, it goes back to the math. The math is different in the two cases. Here’s a thread which explains this in some detail.

1 Like
3

Thank you, that makes sense!