Derivation of Backpropagation in RNNs

Course Q&A Deep Learning Specialization Sequence Models
1

Can anybody helps me in derivation of those variables in back propagation -

Screenshot 2024-05-25 at 9.29.07 AM
Screenshot 2024-05-25 at 9.29.07 AM1920×1247 105 KB

2

Hey there @mahesh-mantri ,

I assume that you want explanation for each one of them:

  • dtanh → Derivative of tanh activation function w.r.t. the input to the activation function.

  • dW_{ax} → Gradient of the loss w.r.t. the weight matrix connecting the input to the current hidden state.

  • dW_{aa} → Gradient of the loss w.r.t. the weight matrix connecting the previous hidden state to the current hidden state.

  • db_a → The gradient of the loss w.r.t. the bias term in the hidden state update.

  • dx^{(t)} → The gradient of the loss w.r.t. the input at time step t.

  • da_{prev} → Gradient of the loss w.r.t. the previous hidden state.

Feel free to ask if you need further assistance!

3

Hello @Alireza_Saei , I was asking for how we can derive these formulae , want to understand the math behind them.

4

These are some basic derivatives. You can find the formulas almost everywhere! Try to understand them yourself, but if you ever feel stuck, feel free to ask, and I can explain them!

5

This requires first that you understand calculus. If you do, then the derivatives are straightforward.