In week 3, in the video about the initialization of Weights, it was mentioned that we can not initialize the matrix to zeroes because this will result in symmetry between the nodes.
Does this mean that every that any W matrix that results in equal Z in all nodes will create the same problem? for example, if W= [[1,2],[1,2]] will it be the same as all zeros?
Welcome to the Specialization @aesmat. I like your intuition, which I think is based on linear dependence between rows/columns of the weight matrix. And my initial conjecture would be yes. Two ways to find out. Pencil and paper using a simple example, such as the one that you propose. Another, would be empirical, using a computational approach, i.e. try it out. You’d be looking for a flat cost function (cost plotted against iterations). I have done neither! 
By the way, there will be more about initialization coming up in the second course. Sit tight.
2 Likes
Yes, it’s not just all zero values that are a problem: any symmetric values are a problem. Here’s a thread that discusses this in more detail and (in particular) addresses the question of why this kind of initial value symmetry is not a problem in the Logistic Regression case.
1 Like