Why do you need Non-Linear Activation Functions? -> ReLU is still linear

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning
1

In Week 3 → Why do you need Non-Linear Activation Functions?

I have one question → I think ReLU is still a linear function of input parameters, because all it is doing is just discarding some values i.e. ones which are negative. But overall it will be still a linear function of input parameters. Please suggest.

2

In mathematics, there is no such thing as “almost linear”: a function is either linear or it’s not. And ReLU is not linear: it is “piecewise” linear, but that is non-linear.

Here’s another thread from a while ago that discusses this same question.

1 Like