Hi everyone! I am finding it hard to give my self an intuitive feel for why ReLU is so good as an activation function. The main complication I am facing is the following: We learn in the lecture that linear activation functions are redundant as they don’t introduce non-linearities and it ends up wi…

I think of ReLU as being an “if” statement in the computation the network performs. That’s non-linear-enough to get round the pointlessness of composing two linear functions. In the first layer of a network, each neuron does a split by a hyperplane (line in week 3’s planar classification) in the in…

Thanks for the reply, In which course’s week 3 is this planar classification of input space talked about, I am in course 1 week 3 and didn’t come across it yet? I do like the if statement analogy, but I’m still unable to get a sense of why how ReLU forms this hyperplane boundary when it is literally…

Take a look at this good forum discussion with some interesting reading sources here: [image] ReLU activation function Neural Networks and Deep Learning One additional thing, a beautiful thing about neural networks, is that even if you have some “dead neurons” the rest of …

Course 1, Week 3 assignment “Planar Data Classification with one hidden layer” is what I was referring to. Relu is “piecewise linear”, but not linear. When ReLU is being used as activation the boundary is the point where the biassed linear combination is zero. It would be an interesting experiment…

Thanks, @GordonRobinson , I now think I have an intuition for how it does the trick. I was obssessing over how its piecewise linear but not how that boundary of different derivatives splits input planes. Very cool. I am still however interested in a mathematical proof for personal satisfaction. Simil…

I think this medium post also explains it well. I guess in a nutshell, with many short straight lines, u can approximate any function. :thinking:

Okay this was EXACTLY what I needed. Thank you SO much!!

Why is ReLU any better than Linear

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning

neurogeek April 22, 2021, 1:44pm 4

Take a look at this good forum discussion with some interesting reading sources here:

Take a look!

Topic		Replies	Views
C1_W3-Non-Linear_Activation_Function Neural Networks and Deep Learning coursera-platform	1	557	May 18, 2021
How non linear is ReLU? Neural Networks and Deep Learning coursera-platform	4	832	March 17, 2023
Choice of activation function Advanced Learning Algorithms week-module-2	7	706	November 21, 2022
Differences between ReLU and linear for positive values Advanced Learning Algorithms week-module-2	4	768	January 16, 2023
Why do we need Activation function Neural Networks and Deep Learning coursera-platform	4	556	February 16, 2023

Why is ReLU any better than Linear

Related topics