Gated Recurrent Unit gates

[Anand_Kumar3]

Hi @Deepti_Prasad ,

your explanation here:

@Deepti_Prasad

So basically in the activation sequence gamma u makes you to decide when to update the memory sequence based on 0 or 1. when gamma u is equal to 0, it is telling the memory sequence to not update and remember the initial value.

if gamma u is equal to zero, so it’s just setting C^t equal to the old value even as you scan the model.

matches with my observation:

@Anand_Kumar3

If Γ𝑢 = 0, then:
ℎ<𝑡>= ℎ<𝑡−1>

if Γ𝑢 = 1, then:
ℎ<𝑡> = 𝑐<𝑡>

But this was mentioned otherwise in @arvyzukai comments and from the results he shared:

@arvyzukai

For example,

• if the update value is 0, this means that the hidden state for next step is completely the new candidate hidden state (everything is updated).
• if the update value is 1, this means that the hidden state for next step is completely old hidden state (nothing is updated).
• if the update value is 0.5, this means that the hidden state for next step is half old and half new candidate hidden state (equally weighted).

That’s what I wanted to check if I was getting it wrong, or the equations has typo in them. Still have doubts on the same.

The equations are taken from here: https://www.coursera.org/learn/sequence-models-in-nlp/supplement/t5L3H/gated-recurrent-units