# Course 1: Week 3 (backpropagation intuition) **URL:** \ **Category:** Neural Networks and Deep Learning\ **Tags:** coursera-platform\ **Created:** [May 21, 2021, 9:18am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979 "2021-05-21T09:18:35Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Giridhar](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@Giridhar](https://community.deeplearning.ai/u/Giridhar)\ **Post date:** [May 21, 2021, 9:18am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/1 "2021-05-21T09:18:35Z")
Dear Mentors/Friends, Backpropagation of dz[1] is W[2]T dz[2] \* g[1]’ (z[1]) g[1]’(Z[1]) can be any activation function. dz[1] is calculated as dL/dz[1] (Derivative of DL : Loss, w.r.t dz[i]) I am confused with how this equation is obtained: W[2]T dz[2] \* g[1]’ (z[1]). There was no explanation of this in the videos as well. (As far as I know). Could anyone explain this? ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [May 21, 2021, 3:57pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/2 "2021-05-21T15:57:03Z")
Prof Ng has specifically designed these courses so that they do not require the students to know any calculus (even univariate calculus, let alone matrix calculus), so he does not cover the derivations of a lot of the formulas which involve calculus. If you would like to dig deeper and have the math background, there are lots of resources available. [Here’s a local thread with a bibliography](https://community.deeplearning.ai/t/deep-learning-bibliography/76), which includes text books that cover the actual math behind all this. One book that is more math oriented is _Goodfellow et al_, which is listed there. Here are some good websites that will also cover the derivations of back propagation: [Here’s a website from Cornell](http://www.cs.cornell.edu/courses/cs5740/2016sp/resources/backprop.pdf) that covers the derivation. Here’s a good [introduction to the matrix calculus](https://explained.ai/matrix-calculus/index.html) you need in order to follow the above. [The Matrix Cookbook](https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf) from Univ of Waterloo is also a valuable resource for general Linear Algebra topics as well as matrix calculus. Here are [some notes from Stanford CS231n](http://cs231n.github.io/optimization-2/) that give a good overview and insights on back propagation. Here’s a bit [deeper dive on the math](http://cs231n.stanford.edu/vecDerivs.pdf) also from Stanford CS231n. [Here are notes from EECS 442](https://web.eecs.umich.edu/%7Ejustincj/teaching/eecs442/notes/linear-backprop.html) at Univ of Michigan. Mentor Jonas Slalin also covers all this and more on [his website](https://jonaslalin.com/2021/12/10/feedforward-neural-networks-part-1/). That’s just the first page in his series. ---
**Author:** ![Giridhar](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@Giridhar](https://community.deeplearning.ai/u/Giridhar)\ **Post date:** [May 21, 2021, 3:58pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/3 "2021-05-21T15:58:21Z")
Thanx a lot Paul for sharing the resources. Thank you again. ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [May 21, 2021, 4:21pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/4 "2021-05-21T16:21:08Z")
The links above should cover everything, but there are plenty more good sources for this type of information out on the web. Googling around a bit should find plenty, although you also get a lot of [Medium.com](http://Medium.com) articles by people who don’t really know that much. Stanford has been particularly generous in making a lot of the supporting material for their graduate CS courses related to ML/DL publicly available. In addition to CS231n material listed above, here are a few more: The [lectures from CS231n](https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv) as a YouTube channel. [CS224n top level site](http://web.stanford.edu/class/cs224n/) with lots of links. [CS230 syllabus.](http://cs230.stanford.edu/syllabus/) [Deep learning notes](https://stanford.edu/~shervine/teaching/cs-229/cheatsheet-deep-learning) (“cheat sheet”) from CS229. [Overall CS229 website](http://cs229.stanford.edu/). This is the class that Professor Andrew taught for many years that gives the Stanford CS Grad Student version of the original Stanford Machine Learning course on Coursera. He’s handed it off to some other professors now, but it’s a great place to start. ---
**Author:** ![Nay](https://avatars.discourse-cdn.com/v4/letter/n/8e7dd6/32.png) [@Nay](https://community.deeplearning.ai/u/Nay)\ **Post date:** [July 2, 2021, 12:42am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/5 "2021-07-02T00:42:34Z")
Hello friend, Did you figure out why dz[1] = W[2]T dz[2] \* g[1]’ (z[1])? I was thinking about it too the entire time I was watching that lecture video. With the chain rule in mind, it seems that Prof. Ng got da[1] = W[2]T dz[2]. I’m not sure how he got that. Thank you, Nay ---
**Author:** ![jeffreywang](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jeffreywang/32/3849_2.png) [@jeffreywang](https://community.deeplearning.ai/u/jeffreywang)\ **Post date:** [July 3, 2021, 3:15am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/6 "2021-07-03T03:15:51Z")
I am wondering too if anyone has figured out. I am the type of person who feels a lingering sense of malaise when I implement functions that I don’t completely understand—and some of the particular derivations baffle me. I have taken multivariable calculus and linear algebra but am still struggling to figure it out, even with the Cornell resource. If one of the mentors could go through the process of outlining it with Prof. Ng’s notation/framework, that would be incredibly helpful! (Sort of like the [dL/dZ logistic regression derivation](https://community.deeplearning.ai/t/derivation-of-dl-dz/165)—which is the most viewed post in the thread, for what it’s worth 😀) ---
**Author:** ![duhang](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@duhang](https://community.deeplearning.ai/u/duhang)\ **Post date:** [July 8, 2021, 6:11am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/7 "2021-07-08T06:11:12Z")
It would be great if anyone can share how to derive dZ1 and dA1. Thanks a lot! ---
**Author:** ![Yaswanth64](https://avatars.discourse-cdn.com/v4/letter/y/91b2a8/32.png) [@Yaswanth64](https://community.deeplearning.ai/u/Yaswanth64)\ **Post date:** [July 17, 2021, 1:29pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/9 "2021-07-17T13:29:11Z")
Not sure if you guys have figured out how dz[1] is calculated but here is the calculation which might help someone who comes here. So the goal is to minimize loss with respect to z1 which is dL/dz1 and this can be written as `dL/da2` \* `da2/dz2` \* `dz2/da1` \* `da1/dz1` using chain rule. Remember that this term `dL/da2` \* `da2/dz2` is loss with respect to dz2 which is `dL/dz2` = `a2-y`. You can refer [this](https://community.deeplearning.ai/t/derivation-of-dl-dz/165) wonderful post to know how this is derived if you are not sure. Now our equation is `(a2-y)` \* `dz2/da1` \* `da1/dz1` `dz2/da1` = `d/da1 w2a1+b` because `z2` is derived from w2a1+b derivative of `w2a1+b` with respect to `a1` is `w2` `da1/dz1` = `d/dz1 sigmoid(z1)` derivative of `sigmoid(z1)` is `sigmoid(z1) * (1-sigmoid(z1))` Finally everything put together, `dL/da2` \* `da2/dz2` \* `dz2/da1` \* `da1/dz1` becomes `(a2-y)` \* `w2` \* `sigmoid(z1) * (1-sigmoid(z1))` which Prof. Andrew has given as `w2` \* `a2-y` (which is loss with respect to z2 so named it as dz2) and the final term `sigmoid(z1) * (1-sigmoid(z1))` is denoted as g prime (z1). Hope this helps as I couldn’t use math notation but just plain text. P.S: Please note that `da1/dz1` can change depending on the activation function used. Here I have assumed activation function at hidden layer is sigmoid and in one of the assignments tanh is used. So a portion of `dz1` changes. ---
**Author:** ![wmartin](https://avatars.discourse-cdn.com/v4/letter/w/e5b9ba/32.png) [@wmartin](https://community.deeplearning.ai/u/wmartin)\ **Post date:** [August 4, 2021, 12:51am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/11 "2021-08-04T00:51:33Z")
Excellent write-up! Thank you so much! ---
**Author:** ![LimXiuXian96](https://avatars.discourse-cdn.com/v4/letter/l/5daacb/32.png) [@LimXiuXian96](https://community.deeplearning.ai/u/LimXiuXian96)\ **Post date:** [August 4, 2021, 12:00pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/12 "2021-08-04T12:00:46Z")
![16280783805496386594903113729980](https://global.discourse-cdn.com/dlai/original/2X/3/338f6b48b251c952dfc6145a2be96b5a4c23f05c.jpeg) Please correct if anything wrong. ---
**Author:** ![isaacchook](https://avatars.discourse-cdn.com/v4/letter/i/6de8d8/32.png) [@isaacchook](https://community.deeplearning.ai/u/isaacchook)\ **Post date:** [August 6, 2021, 3:15pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/13 "2021-08-06T15:15:52Z")
I got the same answer as well. Instead of A[2] - Y at the end of dz[1], I used dz[2] to simplify. You have a beautiful handwriting. ---
**Author:** ![msaid](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/msaid/32/7136_2.png) [@msaid](https://community.deeplearning.ai/u/msaid)\ **Post date:** [September 8, 2021, 1:07pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/14 "2021-09-08T13:07:05Z")
i’'m still puzzled with the transpose in the equations, did any one managed to understand this ? ---
**Author:** ![AYOUBKECHCHOUR](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/ayoubkechchour/32/6980_2.png) [@AYOUBKECHCHOUR](https://community.deeplearning.ai/u/AYOUBKECHCHOUR)\ **Post date:** [September 10, 2021, 3:58pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/15 "2021-09-10T15:58:18Z")
dear friends , Professor Andrew and the mentors are doing great work to generalize this specialization for people how doesn’t understands maths but if you want the explanation : ![image](https://global.discourse-cdn.com/dlai/original/2X/a/a9b54c0611648884464ca0925841c8ea2b6c0e8e.jpeg) i think the key to master backpropagation, u need to understand the matrices multiplication and equality rules , if 2 matrices are equal they need to have the same shape , you can’t mutilply 2 matrices if they don’t have the same shape, or a shape like ( (a,b)\*(a,c)) if they have a shape like that : ((a,b) \* (c,a)) you need to force the second to be (a,c) and of course we can add python broadcasting to the first 2 Mr @paulinpaloalto can verify it Happy learning ! ---
**Author:** ![ahussein](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@ahussein](https://community.deeplearning.ai/u/ahussein)\ **Post date:** [September 11, 2021, 7:47pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/16 "2021-09-11T19:47:49Z")
I am not sure but I think this is a great question. ---
**Author:** ![raju8901](https://avatars.discourse-cdn.com/v4/letter/r/f04885/32.png) [@raju8901](https://community.deeplearning.ai/u/raju8901)\ **Post date:** [October 25, 2021, 11:27pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/17 "2021-10-25T23:27:11Z")
Thank You. I had a hard time understanding how they came up with this formula. ---
**Author:** ![chonkymonkey](https://avatars.discourse-cdn.com/v4/letter/c/c67d28/32.png) [@chonkymonkey](https://community.deeplearning.ai/u/chonkymonkey)\ **Post date:** [December 18, 2021, 5:49pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/18 "2021-12-18T17:49:52Z")
Hello everyone, This derivation confused me so much that I spent the whole day reading up on derivatives. This seems to answer my confusion, and I hope it will answer someone else’s as well some day. I believe the issue stems from the assumption that z and dz (dL/dz) have the same dimensions, whereas the derivative of a scalar with respect to a column vector is a row vector. If we take that to be true, then the dimensions match as is shown in the picture below : ![image](https://global.discourse-cdn.com/dlai/original/2X/9/9a6449b905fe1da6c4a82c7ac5ebddbcabe2eb1b.jpeg) Good luck and have fun ---
**Author:** ![Peter\_Pen](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/peter_pen/32/14110_2.png) [@Peter\_Pen](https://community.deeplearning.ai/u/Peter_Pen)\ **Post date:** [February 4, 2022, 8:29am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/19 "2022-02-04T08:29:56Z")
One good source is that: > **[Deep Learning](https://mitpress.mit.edu/books/deep-learning-1)** > > An accessible introduction to the artificial intelligence technology that enables computer vision, speech recognition, machine translation, and driverless cars.Deep learning is an artificial intelligence technology that enables computer vision,... Chapter 6, it explains back prop. quite well and clear I found (attached) images also very helpful ![dl1](https://global.discourse-cdn.com/dlai/original/2X/2/21ec61c63a2274a09ee716e3e50e87a5e8edae63.png) ![dl2](https://global.discourse-cdn.com/dlai/original/2X/b/b2b63d3db1045a75211791305f99e101a7b53f99.png) ---
**Author:** ![Gatik\_Gola](https://avatars.discourse-cdn.com/v4/letter/g/a4c791/32.png) [@Gatik\_Gola](https://community.deeplearning.ai/u/Gatik_Gola)\ **Post date:** [March 9, 2022, 2:42am UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/20 "2022-03-09T02:42:05Z")
This is beautiful! Thanks a bunch ---
**Author:** ![yrraadi](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/yrraadi/32/36081_2.png) [@yrraadi](https://community.deeplearning.ai/u/yrraadi)\ **Post date:** [April 26, 2022, 8:01pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/21 "2022-04-26T20:01:26Z")
why is dw[2], db[2], dw[1], db[1] obtained by dividing by m? ![image](https://global.discourse-cdn.com/dlai/original/3X/3/e/3ee56b5fff8581565b68247c5f5c2b6d9baaa4ae.jpeg) ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [April 26, 2022, 8:27pm UTC](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979/22 "2022-04-26T20:27:18Z")
Because those gradients are w.r.t. J, the cost. Remember that J is the average of the L loss values across all the samples. That is where the factor of \frac {1}{m} comes from. The notation Prof Ng uses is slightly ambiguous. For example the dZ^{[l]} values there do not have the factor of \frac {1}{m} because they are just “Chain Rule” factors at a given layer that are used to later compute dW^{[l]} and db^{[l]}. For example the first value there (which is special because it is for the output layer) is really: dZ^{[2]} = \displaystyle \frac {\partial L}{\partial Z^{[2]}} But then we have: dW^{[2]} = \displaystyle \frac {\partial J}{\partial W^{[2]}} Notice the difference of J versus L in the numerator there. [Next page](https://community.deeplearning.ai/t/course-1-week-3-backpropagation-intuition/7979.md?page=2)