Hello everyone,
Iโm currently taking the Neural Networks and Deep Learning course, and since Iโm one of those people who canโt fully trust that Iโve understood something until every detail โclicks into placeโ, I decided to attempt a stepโbyโstep derivation of the backpropagation formulae from scratch.
I created a supplement document that Iโm sharing below for anyone who might be interested.
A few disclaimers before you dive in:
- This derivation is heavy on vector calculus. Like, โI had to blow dust off old neuronsโ heavy.
- Following the notations in the course led me to some transposed Jacobians, andโฆ those are not very pretty.
- My vector calculus skills were a bit rusty going in, so this may not be the most elegant derivation ever written, but everything seemingly checks out in the end.
- No LLMs were used in producing the document. So if you find mistakes, those are 100% my own. Please flag them so I can improve.
PS: I canโt say Iโm proud of my handwriting on a tablet, but I am proud of the document itself. I havenโt yet seen a derivation this detailed anywhere, and if it helps even one other person get that โaha!โ moment, it was worth every scribbled matrix.
Logistic Regression from A to Z.pdf (3.0 MB)