# Week 3: Why dZ^\[1\] = W^\[2\]T dZ^\[2\] \* g^\[1\]'(Z^\[1\])

**URL:** <https://community.deeplearning.ai/t/week-3-why-dz-1-w-2-t-dz-2-g-1-z-1/272747>\
**Category:** Neural Networks and Deep Learning\
**Tags:** coursera-platform\
**Created:** [February 12, 2023, 8:28am UTC](https://community.deeplearning.ai/t/week-3-why-dz-1-w-2-t-dz-2-g-1-z-1/272747 "2023-02-12T08:28:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![yujin\_lee2](https://avatars.discourse-cdn.com/v4/letter/y/eb8c5e/32.png) [@yujin\_lee2](https://community.deeplearning.ai/u/yujin_lee2)\
**Post date:** [February 12, 2023, 8:28am UTC](https://community.deeplearning.ai/t/week-3-why-dz-1-w-2-t-dz-2-g-1-z-1/272747/1 "2023-02-12T08:28:58Z")

</div>

![image](https://global.discourse-cdn.com/dlai/original/3X/5/c/5c9e6d6a7e10a1569f6ed0baa48a83f93b55ca67.png)

I think dZ[1] = dZ[2] W[2]T \* g[1]'(Z[1]).

What’s wrong with my calculations?

chain rule? derivative matrix?

---

<div class="post-metadata">

**Author:** ![AbdElRhaman\_Fakhry](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/abdelrhaman_fakhry/32/65768_2.png) [@AbdElRhaman\_Fakhry](https://community.deeplearning.ai/u/AbdElRhaman_Fakhry)\
**Post date:** [February 12, 2023, 4:26pm UTC](https://community.deeplearning.ai/t/week-3-why-dz-1-w-2-t-dz-2-g-1-z-1/272747/2 "2023-02-12T16:26:35Z")

</div>

Hi @yujin_lee2  
Welcome to the community!

The equations in this image

 ![image](https://global.discourse-cdn.com/dlai/original/3X/5/2/522707f5230eb7c0fb005d2ae6bfcedb81e41cca.jpeg)

is only for the last layer to calculate dz^{[l]}, dw^{[l]} and , db^{[l]} and the activation function here is softmax if it isn’t softmax these equations will be change as it is brief.

To get the original equation of how to calculate dz^{[l]}, dw^{[l]} and , db^{[l]} and from they came is

dZ^{[l]} = \frac{\partial \mathcal{L^{[l]}} }{\partial A^{[l]}}\* \frac{\partial \mathcal{A^{[l]}} }{\partial Z^{[l]}}  
dW^{[l]} = \frac{\partial \mathcal{L^{[l]}} }{\partial A^{[l]}}\* \frac{\partial \mathcal{A^{[l]}} }{\partial Z^{[l]}} \* \frac{\partial \mathcal{ Z^{[l]}} }{\partial W^{[l]}}  
dB^{[l]} = \frac{\partial \mathcal{L^{[l]}} }{\partial A^{[l]}}\* \frac{\partial \mathcal{A^{[l]}} }{\partial Z^{[l]}} \* \frac{\partial \mathcal{ Z^{[l]}} }{\partial B^{[l]}}.

That’s called chain rule in the derivatives so that here we remove the step of calculating da^{[l]}

**But** if the last layer **isn’t softmax** we should change this equations to calculate the appropriate equations according to the activation function for example if the last layer activation function is **sigmoid** we should calculate the

dAL = \frac{\partial \mathcal{L^{[L]}}}{\partial A^{[L]}} = -\frac{y^{[1]}}{a^{[1]}} +\frac{1-y^{[1]}}{1-a^{[1]}} ...-\frac{y^{[m]}}{a^{[m]}} +\frac{1-y^{[m]}}{1-a^{[m]}}

and calculate that dZ^{[l]} = \frac{\partial \mathcal{L^{[l]}} }{\partial Z^{[l]}} according to( dZ^{[l]} = \frac{\partial \mathcal{L^{[l]}} }{\partial A^{[l]}}\* \frac{\partial \mathcal{A^{[l]}} }{\partial Z^{[l]}} ) if you want the equation of the dZ^{[l]} of the sigmoid function will be dZ^{[l]} =\frac{\partial\mathcal{L^{[l]}} }{\partial Z^{[l]}} = A^{1}(1-A^{1}) ... A^{m}(1-A^{m})

so concretely we must have dA^{[for \ each \ layer \ except \ layer \ 0]} to get all ( dZ^{[l]},dW^{[l]}, db^{[l]}) by chain rule  
**Also**  
If the activation function of the layer is changed the equations also changed according to this image

 ![image](https://global.discourse-cdn.com/dlai/original/3X/0/e/0e5a308e98296e1d7202cb6ce2540faccaf56c0c.png)

Cheers,  
Abdelrahman

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [February 12, 2023, 5:11pm UTC](https://community.deeplearning.ai/t/week-3-why-dz-1-w-2-t-dz-2-g-1-z-1/272747/3 "2023-02-12T17:11:44Z")

</div>

Well, notice the dimensions on dZ^{[2]} and W^{[2]T}: they are n^{[2]} x m and n^{[1]} x n^{[2]} respectively, right? So that matrix multiply (dot product style) in the formula you show is not going to work. But it does work if the formula is as Prof Ng and AbdElRhaman show it.

Taking derivatives is a bit more complicated when you’re working with matrices than scalars. This is beyond the scope of this course (by design), but you can find links to background information on matrix calculus and on these specific derivations on [this thread.](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979)

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [February 13, 2023, 3:22am UTC](https://community.deeplearning.ai/t/week-3-why-dz-1-w-2-t-dz-2-g-1-z-1/272747/4 "2023-02-13T03:22:26Z")

</div>

Hello @yujin_lee2,

As @paulinpaloalto suggested, we often resort to checking the matrices’ indices to make sure all symbols are in order. As he also pointed out, we can’t always do matrix calculus in the way we do scalar calculus. As [wikipedia](https://en.wikipedia.org/wiki/Matrix_calculus) very well summaried:

> Note that exact equivalents of the scalar product rule and chain rule do not exist when applied to matrix-valued functions of matrices.

We can do a simple exercise to verify that indeed we can’t use chain rule like that. Let’s say we define matrices Z, W, A and a scalar L this way,

![image](https://global.discourse-cdn.com/dlai/original/3X/1/3/1371dd266325ab4e2ed59091b274bea9d05af959.png)

Obviously,

![image](https://global.discourse-cdn.com/dlai/original/3X/e/8/e89f103f041e3431e8e6b6aa650382d0459028a5.png)

Can our scalar “chain rule” recovers the same result?

 ![image](https://global.discourse-cdn.com/dlai/original/3X/e/c/eccb388540bb5d67e1b1712cb333a1bd15b30e7d.png)

The answer is no. But the following will do:

![image](https://global.discourse-cdn.com/dlai/original/3X/c/1/c128ee8f3d1cb9dbccf102e1c9e008215773cdd1.png)

Lastly, in my step (5), I said “this is wrong”, because a matrix-by-matrix derivative like that won’t result in a 2x2 matrix. Because as [the same wikipedia page](https://en.wikipedia.org/wiki/Matrix_calculus) says:

> Matrix calculus refers to a number of different notations that use matrices and vectors to collect the derivative of each component of the dependent variable with respect to each component of the independent variable.

So to correct that wrong thing, it is going to be a tensor that organizes 16 derivative results.

I think [the wikipedia page](https://en.wikipedia.org/wiki/Matrix_calculus) has a lot of useful examples if you want to dig deeper.

Cheers,  
Raymond