What makes different neurons calculate different parameters within a layer?

From what I understood from one of the lectures of MLS course 2 week 1, the neurons in a layer are like logistic regression units.

Since each such logistic regression unit is getting the same feature vector or input layer as input, why do they output different activations? Or what makes them do that?

2 Likes

The units in a hidden layer have the same input features, but they each use different weights. So they each give different outputs to the following layer.

Hi @Rohit_Pandey1

welcome to the community and thanks for your question!

Here you can find an illustration that might be relevant for you:

You can see that each neuron has individual weights and bias that are learned by the neuron within the training process. These parameters are different for each neuron and they lead (via activation function) also to a different output of each neuron.

Best regards
Christian

1 Like

Hi @Rohit_Pandey1

Each neuron in a layer receives the same input feature vector, but the neurons are different because they have different weights and biases. The weights and biases are learned during the training process and are different for each neuron.

When the input feature vector is multiplied by the weights and added with the bias of a neuron, the output is a scalar value, also called the activation. The activation function is then applied to the scalar value to produce the final output of the neuron.

The activation function is a non-linear function that is applied element-wise to the output of the previous layer, and it’s used to introduce non-linearity in the network. Common activation functions include sigmoid, ReLU, and tanh. The choice of activation function depends on the task and architecture of the network.

For example, the sigmoid activation function squashes the input values into the range of 0 and 1, making it useful for binary classification tasks. On the other hand, ReLU (Rectified Linear Unit) activation function sets negative values to zero, making it useful for deep networks.

Since each neuron has its own set of weights and biases, and the activation function is applied to the output of the previous layer, each neuron will produce a different output, even though they receive the same input feature vector. This allows the network to learn different features from the input data, which helps to improve its ability to generalize to new data.

Regards
Muhammad John Abbas

1 Like

Thanks @Christian_Simonis .

I am still not sure I completely understand how different neurons of the same layer learn different weights and biases during training, but I guess it has something to do with the neurons in the next layer? Back-propagation?

For instance,


here the professor mentioned that it was possible to find what exactly each neuron was recognizing in the input image.

I guess I will go ahead with the course for now and come back to this later because it looks like it involves some advanced mathematics.

Thanks for your patience.

Regards,
Rohit

1 Like

Hi @Rohit_Pandey1,

I guess it’s fair to go ahead as you suggested. I would still recommend to think and reflect about the Initialization of parameters a bit. Possible questions you want to give some thoughts:

  • what happens if all initialized parameters would be the same?
  • what if they are too large?
  • what if they are too small?
  • strategies for proper Initialization

On this note you can find a lot of useful answers here and can practice also a little bit with toy examples: Initializing neural networks - deeplearning.ai

Please let us know if something is unclear or if you need support, @Rohit_Pandey1!

Best regards
Christian

2 Likes

You could also apply a heatmap approach to interpret in a visual way and see in which areas the layer were activated and contributed to the output, see also: Get Heatmap from CNN ( Convolution Neural Network ), AKA CAM | by Seachaos | tree.rocks

Also: In figure 1) you find a nice illustrative visualization how low level features like edges are hierarchically combined and enhanced to describe more advanced patterns to finally form objects: see the following Source.

see this Thread.

Hope that helps, @Rohit_Pandey1!

Best regards
Christian

1 Like

Thank you for the resources!

1 Like

Sure! You are welcome. If you have further questions, pls. do not hesitate to ask :slightly_smiling_face:

Happy learning!

Best regards
Christian

1 Like