I just saw the lecture on neural network activation functions and I was wondering if most of the times, we should use a ReLu function for hidden layers and linear or logistic for the last layer depending on our goal, or there is more ways of making a neural network?
Additionally, in the lecture, we see that using only linear regression functions in the hidden layers is useless because using a simple linear regression gives the same result, why is it different with the ReLu function?
Hi @kaki178925 ,
There are several activation functions available, including the ReLU, tanh, and SELU functions. The choice of activation function largely depends on the specific problem you are trying to solve and your goals.
The reason why the ReLU function is commonly used for hidden layers and it is different from linear activation is that it is non-linear. The ReLU function returns the input value if it is positive, and returns zero if the input value is negative. This introduces non-linearity into the output of a neuron, which allows the neural network to learn non-linear or complex relationships between the inputs and outputs. On the other hand, linear or logistic activation functions are used for the output layer as they are better suited for regression or binary classification problems.
Best Regards,
Mujassim
Hi @kaki178925!
In addition to Mujassim’s reply, read my this article to understand more about different activation functions and when to use them.
Best,
Saif.