Activation function: ReLU

[ahmedemam]

When creating a post, please add:

• Week 4
  *Coursera | Online Courses & Credentials From Top Educators. Join for Free | Coursera
• why was l1[z1<0] = 0 and not just taking taking max between 0 and l1 value itself not the z1, what’s the idea here? For context l1 is W2.T x yhat-y, and z is W1 x X + bias1

[TMosh]

Sorry, I don’t understand your question.

Maybe it would help if you posted a screen capture image from one of the lectures or labs.

[jyadav202]

Hi!
l1[z1<0] is a step function implementing ReLu derivative and gradient multiplication.
What I mean is that during the calculation of gradients using chain rule, we arrive at point when we have to perform :

\begin{align} Gradient * Derv(activation func) &= W_2^T(\hat y - y) * Relu'(Z_1) \\ &= \begin{cases} W_2^T(\hat y - y)*1 , & \text{if } Z_1 > 0\\ W_2^T(\hat y - y)*0 , & \text{if } Z_1 < 0\\ \end{cases} \end{align}

In python we can implement the above condition using a step function l1.step(z1) or l1[z1<0] = 0

[jyadav202]

Also note that we are taking gradient of Relu. In your question, perhaps you understood that we have to apply Relu.

[ahmedemam]

Oh this makes sense now thank you.