Hi, I have the following naive question: How does a neural network learn different patterns at each node. Let’s say I have a 2 layer neural network: 1 input layer taking 3 input features, 1 hidden layer and one output layer, and in the hidden layer I have 3 nodes/neurons. If every node in the hidden layer takes the same, all input features, even if I initialise the weights randomly I would still imagine that every node will “converge” to the same weights that would minimize the overall cost function. In this case, how do we end up with every node learning different patterns? What am I missing here?
Random initialization is the answer. It gives each neuron different starting weights; thus, each neuron produces different activations, receives different gradient updates, and follows a different optimization trajectory. So, the neurons do not generally converge to the same weights
The way I think about this is that we are trying to impose a certain condition on the network so that, given an input, it produces a desired output. The weights are randomly initialized at the beginning, and during training, the neurons in the intermediate layers gradually adapt their weights so that the network can transform the input into the desired output. In a sense, the network is being constrained by the target output, and the weights are adjusted accordingly until the network learns a configuration that produces the desired result.
Hello @ZinaSi, I would like to focus on just two parts of your message because there are already great responses that address the big question in the title.
You were only thinking about the forward pass. However, weights update happens in the back propagation. In the forward pass, yes, each node receives the same input, but in the backward pass, the nodes receive different gradients because each node’s output pairs up with a different weight in the output layer, and the gradient depends on that output layer weight.
There must be some condition for convergence to happen. For example, loss can converge because there is local minimum. If we want the nodes to converge to the same set of weights, we need to impose a condition such as adding a term like w_1 - w_2 in the loss function such that the loss becomes smaller when w_1 = w_2, but obviously we are not doing that. Without that, all that the gradient descent cares is to tune the weights to minimize the loss which is the error between predictions and labels, period.
In short, we need a reason (condition) for such convergence to happen, but there is none, so they won’t converge.