According to tf.keras.layers.Dense | TensorFlow v2.13.0, if no activation is applied, it is: a(x) = x. Then why bother to even have this layer with no activation function?
And by calculating the params for the user model, clearly the final layer without activation function has W and b values. So there is some activation function applied?
What function is used when no activation function?
When no activation function is specified, layer output is w^T \cdot X + b
Linear activation is useful when output layer is required to predict regression output in range [-\infty, \infty] (eg: temperature at a desert based on number of hours since midnight).
So when no activation function is specified, Linear activation function is used.(default)
Then how to understand this:
Activation is applied on top of the affine transformation i.e. w^T \cdot X + b. Think of it as the following pseudocode:
class Dense:
def forward_pass(self, X):
output = w @ X + b
if self.activation is not None:
output = self.do_activation(output)
return output
Thank you so much!
So if specify Linear activation function, it’s actually == no activation function?
Yes.
Thanks!