In 3 week I got this note in quiz
No. As seen in lecture the output of the tanh is between -1 and 1, it thus centers the data which makes the learning simpler for the next layer.
Can you explain, please, how could the choice between -1 and 1 be better than 0 and 1 in sigmoid function for hidden units for future computations? Or give me link on what lecture it was discussed, please.
We use normalization techniques, such as MinMaxScaling, to ensure that the data is on a similar range, such as -1 to 1. This helps to speed up the learning process. The activation function tanh naturally produces outputs that fall between -1 and 1, which acts as a form of normalization and also facilitates the learning process for the next layer.
Please refer to this video where this topic is discussed.
In a related question, sometimes people ask if we could even use tanh as the activation at the output layer in a binary classification and then use >= 0 as “Yes” and < 0 as “No”. But the question then is what you would use as a loss function, since cross entropy loss is specifically designed to work with sigmoid. Here’s a thread which discusses that in more detail and also ends up showing that there is a very close relationship mathematically between sigmoid and tanh.
thanks for video. Indeed, NG said that it can be better. But it is still unclear for me — why? I understand, that it is more universal — this is ok. But better? About future computations didn’t understood too. I passed 2 course and can’t understand how calculatoins can be easier with tanh. Is it connected with normalization, as I understand?