Non-linearity in Z

I understand that deep learning models are highly nonlinear by applying nonlinear activation functions in every layer and neuron. However, the values fed to these activation functions (denoted as Z in the course) seem to be always linear. So if the inputs are x1, x2, and x3,

Z = w1*x1 + w2*x2 + w3*x3 + b

My question is: is there any evidence that introducing non-linearity to Z will enhance NN performance? For example, if Z is defined as:

Z = w1*x1 + w2*x2 + w3*x3 + w4*x1^2 + w5*x1*x3 +  b

Or is this kind of non-linearity already accounted for, implicitly, in the NN?

Thank you,

These transformations are also linear, you could create new features what you have done here or scale them or else. This is the case of a polynomial regression.

What is referred to as non-linearity in NN’s, is meant that this linear expression goes through an activation and this makes it non-linear.

1 Like