# Derivative of Relu in output layer

**URL:** <https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444>\
**Category:** Neural Networks and Deep Learning\
**Tags:** coursera-platform\
**Created:** [November 19, 2022, 9:52am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444 "2022-11-19T09:52:41Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 4:04am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/21 "2022-11-24T04:04:22Z")

</div>

> [@paulinpaloalto](#):
>
> it does indicate that ReLU is not the right choice for your output layer activation.

I just removed the ReLU from the output layer and used leaky\_relu. But I got the same result, I guess, it is because I did not define the leaky\_relu\_backward and instead, use relu\_backward.  
I copied the Relu Backward’s code below but what will be the leaky\_relu\_backward? Derivative of `A = np.maximum(0.01*Z,Z)`?

```
  def relu_backward(dA, cache):
      Z = cache
      dZ = np.array(dA, copy=True) # just converting dz to a correct object.
      dZ[Z <= 0] = 0
        
      assert (dZ.shape == Z.shape)
        
      return dZ

```

> [@paulinpaloalto](#):
>
> Removing ReLU at the output layer and then just letting MSE punish those bad negative values might help. At least that would be one experiment I would try …

Do you mean remove the relu and just use linear function, like below?  
Z^l = A^l = np.dot(W,A^{l-1}) + b^l)

> [@rmwkwok](#):
>
> Just looking at these plots I will first relate them to high variance as Rashmi pointed out. Would you mind sharing your notebook with me in a Direct Message? I am interested in this example.

I sent it to you, Raymond.

In the last, I would like to thank all of you for your unbounded and limitless time and guidance. Highly indebted to you.  
Saif.

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 5:00am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/22 "2022-11-24T05:00:37Z")

</div>

Thank you @saifkhanengr.

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 5:09am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/23 "2022-11-24T05:09:27Z")

</div>

I spot something quickly. Lucky me 😜

I added these print lines

```python
def linear_forward(A, W, b):
    ..... # your code
    print('linear_forward A', A.shape)
    print('linear_forward W', W.shape)
    print('linear_forward b', b.shape)
    print('linear_forward Z', Z.shape)
    return Z, cache

```

and found these:

```python
linear_forward A (20, 1)
linear_forward W (7, 20)
linear_forward b (7, 1)
linear_forward Z (7, 1)

```

The thing is, the shape of your `W` should be `(number of features, number of neurons)` which means (1, 7), although my preference is `(number of neurons, number of features)`.

I will hold on reading your notebook, because it seems you have something to change and check. Checking shapes can be an interesting exercise 😉

Cheers,  
Raymond

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 5:14am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/24 "2022-11-24T05:14:02Z")

</div>

Btw, `(number of neurons, number of features)` is meant for the first dense layer. The second dense layer should have `(number of neurons in this layer, number of neurons in the last layer)`

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 5:32am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/25 "2022-11-24T05:32:35Z")

</div>

Yes, my W’s and b’s are (number of neurons, number of features) and (number of neurons, 1) respectively.

BTW, waiting for you to read the whole notebook.

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 5:53am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/26 "2022-11-24T05:53:05Z")

</div>

> [@saifkhanengr](#):
>
> Yes, my W’s and b’s are (number of neurons, number of features) and (number of neurons, 1) respectively.

The shape of your `W` in the first dense layer is `linear_forward W (7, 20)`. 20 is the number of samples. Did I misunderstand anything? Or would you like to check all the shapes in your notebook?

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 7:34am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/27 "2022-11-24T07:34:03Z")

</div>

I just check all the shapes of my file. They are:  
print(f"shape of X is {X.shape}“)  
print(f"shape of Y is {Y.shape}”)  
print(f"number of neurons in hidden layer is {n\_h}“)  
print(f"shape of W1 is {W1.shape}”)  
print(f"shape of b1 is {b1.shape}“)  
print(f"shape of A1 is {A1.shape}”)  
print(f"shape of W2 is {W2.shape}“)  
print(f"shape of b2 is {b2.shape}”)  
print(f"shape of A2 is {A2.shape}")

```
shape of X is (20, 1)
shape of Y is (20, 1)
number of neurons in hidden layer is 7
shape of W1 is (7, 20)
shape of b1 is (7, 1)
shape of A1 is (7, 1)
shape of W2 is (20, 7)
shape of b2 is (20, 1)
shape of A2 is (20, 1)

```

All good or not?

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 7:37am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/28 "2022-11-24T07:37:59Z")

</div>

> [@saifkhanengr](#):
>
> `shape of W1 is (7, 20)`

What are the meaning of 7 and 20? Number of features? Number of samples? Number of neurons?

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 7:42am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/29 "2022-11-24T07:42:33Z")

</div>

7 is the number of neurons and 20 is the number of samples (rows of input).

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 7:44am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/30 "2022-11-24T07:44:12Z")

</div>

> [@saifkhanengr](#):
>
> Yes, my W’s and b’s are (number of neurons, number of features) and (number of neurons, 1) respectively.

OK, but you said this which is correct but contradicting to your printing result. Do you see what you need to change?

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 7:47am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/31 "2022-11-24T07:47:04Z")

</div>

Oh. Number of features is 1 (column of X). So, I need to change that. Currently, I am using (number of neurons, number of samples) = (7,20) for W1 but it should be (number of neurons, number of features) = (7,1). Right?

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 7:47am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/32 "2022-11-24T07:47:51Z")

</div>

Exactly! And also your W2 and b2, they have problem too.

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 7:48am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/33 "2022-11-24T07:48:09Z")

</div>

> [@saifkhanengr](#):
>
> it should be (number of neurons, number of features) = (7,1). Right?

Doing this gives error

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 7:56am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/34 "2022-11-24T07:56:53Z")

</div>

Saif, I have been following this conversation and seen a lot of your effort. I can guess the error you are seeing, but I hope you will try googling the error message first and debug it yourself. Some said writing the code takes only 30% of the whole coding time but debugging takes 70%. I don’t want to do that major work for you.

Now you know the right thing to do, so it’s time to do it right. I know you will try it first, right?

Raymond

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 8:01am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/35 "2022-11-24T08:01:58Z")

</div>

I highly appreciate your time and efforts, Raymond. Thanks a lot. I love to learn to fish instead of taking fish. Thanks, man…

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [November 24, 2022, 8:02am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/36 "2022-11-24T08:02:54Z")

</div>

> [@saifkhanengr](#):
>
> I love to learn to fish instead of taking fish

We are the same 😉 Take your time debugging, and I am following this topic. Just please make sure all shapes are correct 😉

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 24, 2022, 8:05am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/37 "2022-11-24T08:05:04Z")

</div>

Sure… Will update here.

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [November 24, 2022, 5:33pm UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/38 "2022-11-24T17:33:56Z")

</div>

> [@saifkhanengr](#):
>
> Do you mean remove the relu and just use linear function, like below?  
> Z^l = A^l = np.dot(W,A^{l-1}) + b^l) Zl=Al=np.dot(W,Al−1)+bl)Z^l = A^l = np.dot(W,A^{l-1}) + b^l)

Yes, that was what I was recommending. You can try LeakyReLU as well, but I think it’s worth trying just omitting the output activation function altogether.

If you implement LeakyReLU, here’s one way to code its derivative:

```auto
def leakyreluprime(Z, slope = 0.05):
    G = np.where(Z > 0, 1, slope)
    return G

```

Of course that is implemented as a separate function call. If you build it “in situ” by analogy to the way `relu_backward` works, you’re doing two things at once:

dZ = dA \* g'(Z)

But the same idea can be adapted …

One point to emphasize here is that if you just duplicate the code in `relu_backward` to make `leaky_relu_backward`, be sure to understand the importance of the way they implemented this line:

> [@saifkhanengr](#):
>
> `dZ = np.array(dA, copy=True)`

If you “short-circuit” that by eliminating the “copy” there:

dZ = dA

that is a disaster, because you’re about to overwrite some of the values in dZ. Because of the way that parameter passing works in python and the way object assignments work, doing it without the copy modifies the global value of dA. See [this post](https://community.deeplearning.ai/t/week-3-program-assignment-update-parameters/4016/2) and [this later reply](https://community.deeplearning.ai/t/week-3-program-assignment-update-parameters/4016/5) on that thread.

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [November 29, 2022, 11:02am UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/39 "2022-11-29T11:02:36Z")

</div>

Hello @paulinpaloalto @rmwkwok @Rashmi! I hope you are doing well. Every time I ask silly questions and you consistently guide me. Thank you for that.

> [@paulinpaloalto](#):
>
> Yes, that was what I was recommending. You can try LeakyReLU as well, but I think it’s worth trying just omitting the output activation function altogether.

I am omitting the relu activation function in the last layer. I copied many functions (from DLS course 1, week 3 and 4 assignments), so, for simplicity, I defined a new function named `no_relu` and then just changed the name in other functions.  
This is how I defined that:

```auto
def no_relu(Z):
    A = Z
    assert(A.shape == Z.shape)
    cache = Z 
    return A, cache

```

Now I need to define its derivative too.  
I did like this:

```auto
def no_relu_backward(dA, cache):
    Z = cache
    dZ = np.array(dA, copy=True) # just converting dz to a correct object.
    dZ = 1
    assert (dZ.shape == Z.shape)
    return dZ

```

Just making it 1. Is it correct or not? I am getting error at that point.

Your most unintelligent student,  
Saif.

---

<div class="post-metadata">

**Author:** ![Rashmi](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rashmi/32/17812_2.png) [@Rashmi](https://community.deeplearning.ai/u/Rashmi)\
**Post date:** [November 29, 2022, 12:20pm UTC](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444/40 "2022-11-29T12:20:40Z")

</div>

Hello SaifKhanengr

It’s good to see you experimenting in different ways! Initially you tried ReLu and then LeakyRelu, where you received almost the same results. Now, you have omitted the last layer (outer activation function). But what was your purpose to check these experiments? If it is to check the binary classification, then how can you just build on 1 as an output? It always checks between 0 and 1.  
If you are doing so, then what about 0? And that makes me wonder, how dZ=1 will provide you actual results?  
You can expand it beyond to check more of it. Thanks!

[Previous page](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444.md?page=1)

[Next page](https://community.deeplearning.ai/t/derivative-of-relu-in-output-layer/232444.md?page=3)