In mathematics, there is no such thing as “almost linear”: a function is either linear or it’s not. And ReLU is not linear: it is “piecewise” linear, but that is non-linear.
Here’s another thread from a while ago that discusses this same question.
1 Like