Hi,
It may be a silly question but why for the LSTM cell, we have the element wise multiplication using “*” for c_next and a_next and the dot product does not work ?
I did the dot multiplication and it did not work, then I tried using the * symbol for element wise multiplication.
I’m still confused despite the fact that I thought I knew.
Thank you
Here is an overview of LSTM cell.
And, I marked three locations in red dotted line.
All are related with calculations of a_next and c_next. And, important thing is all are designed to do Hadamard product (wlement-wise product), since it’s a gate function.
\Gamma_f, \Gamma_u, and \Gamma_o defines which element in a_prev, c_prev should be passed to the next step. And, those are output of “sigmoid”, i.e, in the range of 0~1. (It’s like a mask for each element.) So, element-wise operation (Hadamard product) is required for those gate variables.
Hope this helps.
Nobu has done a great job of illuminating the specifics of the LSTM logic. Here’s a thread about the general question “When do I use np.dot versus elementwise multiply”. One important “take away” from that thread is you need to carefully observe the notational convention that Prof Ng uses when he writes mathematical formulas: he always uses * to indicate elementwise multiply.
