Output gate and activation unit a<t>

In the programming exercise 1, it describe the output gate as follow

πšβŸ¨π‘‘βŸ©=πšͺβŸ¨π‘‘βŸ©βˆ—tanh(πœβŸ¨π‘‘βŸ©)

  • The output gate
  1. acts like a β€œmask” that either preserves the values of tanh(πœβŸ¨π‘‘βŸ©)or
  2. keeps those values from being included in the hidden state πšβŸ¨π‘‘βŸ©

I could get point 1, since the output gate values from 0 to 1. Thus, when output gate is 1, c is preserved.
Yet for point 2, I wonder why it could keep values from πšβŸ¨π‘‘βŸ©. Should it be πšβŸ¨π‘‘-1⟩?
Since πšβŸ¨π‘‘βŸ© should be the dependent variable, its means its outcome is determined by the variables , output gate and cell state πœβŸ¨π‘‘βŸ©. So how come it could β€œkeep” its value? Hope mentors or classmates could help. Many thanks.

Hi jackchan.hk,

Point 2. is formulated unfortunately. β€œThe output gate keeps values from being included” is similar in meaning to β€œThe output gate stops the inclusion of values”. So for clarity, point 2. might read:

β€œ2. stops the inclusion of those values in the hidden state a”

I hope this clarifies.

1 Like

Thanks so much reinoudbosch

This explanation sound and improve much the readibility, love it!!!