How do Neurons and activations actually work?

Generally speaking, no there is no such a way.
Exception? Yes, if you go to C2 W1 Lab “Coffee Roasting in Tensorflow”, you find the following plots that try to visualize what each of the 3 trained neurons are doing.

On the other hand, you can still draw this plot before the training, and just to show how the sample labels are distributed in the input features space (Temperature and Duration), and then you will find that those red crosses are forming a triangle, and given such insight, I would choose 3 neurons accurately because we need only 3 linear boundaries to form a triangle.

However, in practice, we can’t always visualize our dataset like this, and that’s a reason why generally speaking, we can’t accurately choose the number of layers and neurons-per-layer.

So what do we do? That is a long story and not covered in this MLS, but in course 2 of the deep learning specialization when hyperparameter tuning is discussed. In short, the simplest basic is, this involves trying different options, evaluating the trials and look for the best option.

Intead of saying “pick”, I would say the reason why a trained NN knows how to transform the input features into the right abstract neuron outputs is because of gradient descent.

Our gradient descent algorithm optimizes our neurons by minimizing the log loss. So whatever final state you see in the neurons after the training, it is merely a product of minimized cost given the training dataset. You can say “if the neurons look like that, then the cost is minimized, but if it looks like something else, the cost becomes worse”.

Yes, it’s the maths the drives the change of the weights in a neuron, and you have learnt that in course 1 when we talk about gradient descent and gradients.

In summary, the maths drives the change of weights into the direction of minimizing the cost, and the maths doesn’t predict how the weights will finally look like, and it doesn’t predict how many layers and neurons are needed. No matter how many layers and neurons you give to the neural network, the maths works in the same way – trying to change the weights of all available neurons in order to minimize the cost. This is very machanical :wink:

Cheers,
Raymond

3 Likes