Week 3 Random Initialization

Yes, you can do it either way and still “break symmetry”. But the common practice is to randomly set the W values and zero the b values. My guess is that this gives you faster learning but this is an experimental science: you can try it both ways and see what happens. Well, I guess there are three different methods to compare: random W and zero b, zero W and random b and then both of them random.

Here’s a thread which goes into a bit more detail about Symmetry Breaking and mentions the issue that you point out.

3 Likes