This is exactly the diagram showing the flow for
and it might be useful when going through the “intuition for backpropagation” video or doing the assignment of week 3.
It clearly replaces my earlier backpropagation diagrams, which are just too hard to read.
Have fun!
Available in PNG, SVG, GRAPHML at
So, I can see that the diagram can be divided into top / middle / bottom parts.
Arrows go from left to right in the top part, showing the forward prop. Arrows, on the other hand, go from right to left in the bottom part for back prop.
In the middle part, downward arrows show values are cached from the top and then used for the bottom. Upward arrows show the updates.
I think the diagram might be slightly simplified by skipping the operation names such as 
, because, for example, you would just have 
directly instead of adding names in between.
Also, I mentioned 4 kinds of arrows above, so I wonder if it would be interesting to you to see different colors/types of arrows for them.
There is just one additional type of relation you might add to the diagram. If you watch the computational graph video again (this DLS video or this MLS video), you see that each math operation in the forward prop is related to a derivative operation in the back prop. For example,
However, I am not sure if it is better to add these relations as it will make the diagram more complex.
Lastly, in my first two jobs, I used AutoCAD and SolidWork a lot, so at that time I read many things about how to draw. Your diagram reminded me of how lines can be jumped when they cross the others. Check this out for an example. This is particularly useful when all arrows have the same color and style. I mean, if you like it and your diagram tool supports it, then why not?
Cheers,
Raymond
Thanks Raymond, I will have to consider these suggestion 
Sadly, there is no way to implement line jumps in yEd, I will have to use something more complex/powerful.
Meanwhile, I have moved the elements around to align them according to the logic of what we want to do and added some notes.
Also added a new diagram which show the structure when you cannot simplify due to not necessarily using “sigma” and “log-likelihood loss function” at the end of feedforward pipeline.
Sure. Please ignore the line jumps if that feature is not available. I was just sharing some thoughts, as a diagram lover.
Both look great! Really. If someone wants to walk through gradient descent, your diagrams are going to be very helpful! I have bookmarked it.
Cheers,
Raymond
