Hello @Christopher_Badman
All the weights are randomly initialized, hence they will start out with different values…which puts them on different learning paths.
End result: They do not converge to the same value, as you mentioned. Rather, they end up with different values - And in this whole process they would have learnt different features.
This random initialization of the weights is what breaks their symmetry - Check out the notebook in this article, and you can see for yourself