Non-linearity in Z

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning
1

I understand that deep learning models are highly nonlinear by applying nonlinear activation functions in every layer and neuron. However, the values fed to these activation functions (denoted as Z in the course) seem to be always linear. So if the inputs are x1, x2, and x3,

Z = w1*x1 + w2*x2 + w3*x3 + b

My question is: is there any evidence that introducing non-linearity to Z will enhance NN performance? For example, if Z is defined as:

Z = w1*x1 + w2*x2 + w3*x3 + w4*x1^2 + w5*x1*x3 +  b

Or is this kind of non-linearity already accounted for, implicitly, in the NN?

Thank you,
Amir

2

These transformations are also linear, you could create new features what you have done here or scale them or else. This is the case of a polynomial regression.

What is referred to as non-linearity in NN’s, is meant that this linear expression goes through an activation and this makes it non-linear.

1 Like