LSTM peephole connections variation

Course Q&A Deep Learning Specialization Sequence Models
1

In the LSTM video, the peephole connections variation is introduced. My question regarding the peephole connections is about the 1-1 relationship, which Andrew mentions, between the memory cell c of the previous time step and the gate of the current time step. For example, the 1st element of c affects only the 1st element of the update date.

The formula for calculating the update gate for the peephole variation is:

Γu=σ(Wu[a<t-1>,x<t>,c<t-1>]+bu),
with Wu=[Wua||Wux||Wuc]

This can be further decomposed to:

Γu=σ(Wua a<t-1> + Wux x<t> + Wuc c<t-1> + bu)

c and Γu are vectors of the same dimension. Let the dimension of the vectors be n.
Therefore, the Wuc matrix is an nxn matrix.

Wuc c<t-1> = | w11*c1+ w12*c2+ ... + w1n *cn | 
             | w21*c1+ w22*c2+ ... + w2n *cn |
             | ....                          |
             | ....                          |
             | wn1*c1+ wn2*c2+ ... + wnn *cn |

However, this means there is not a 1-1 relationship between c and the gate values. Instead there is a 1-1 relationship between Wuc c<t-1> and the gate values.

2

Hi soul,

Hope that you solved your doubt but in case you did not, I will try to help you understand this variation and the meaning of that relationship. As I understand it the peephole architecture what it does is “to let the gate layers look at the cell state”, that is what 1-1 relationship means. You can read further about it here:

Happy learning