They are supposed to be initialized randomly, and then have different values throughout the training process, unless we knowingly overly strongly regularize them that push them to zeros, or we are suffering from some unwanted problems.
Cheers 
Raymond
PS: this is a post that explains why neurons can diversify and in which there is another link to a post that demonstrates how you can make neurons not to diversify