# Sampling and RNNs

**URL:** <https://community.deeplearning.ai/t/sampling-and-rnns/409632>\
**Category:** NLP with Sequence Models\
**Tags:** week-module-1\
**Created:** [August 6, 2023, 10:45am UTC](https://community.deeplearning.ai/t/sampling-and-rnns/409632 "2023-08-06T10:45:23Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\
**Post date:** [August 7, 2023, 2:09pm UTC](https://community.deeplearning.ai/t/sampling-and-rnns/409632/5 "2023-08-07T14:09:59Z")

</div>

> [@Kamal\_Nayan](#):
>
> 1)GRU units learn from previous inputs also and for dealing with the vanishing gradient problem in this we make a memory cell and a update gate . The update gate keeps helps us remember the value.

Well… I think you are correct if I interpret your sentences correctly:

Intuitively, the reset gate controls how much of the previous state we might still want to remember. Likewise, an update gate would allow us to control how much of the new state is just a copy of the old state.

> [@Kamal\_Nayan](#):
>
> 2)Sampling basically means getting the probabilities of all the characters that can be at that position and then choosing the best one.And this is done by training our model with our dataset.

Not quite. There are different sampling techniques (_during inference_). The “greedy” sampling is as you said - choosing the best one. While there are other sampling techniques which have some parameter (usually called temperature), that helps you control how “greedy” you want to be. For example, if you had the probabilities of [0.1, 0.2, 0.7], the “greedy” version would always choose the character at the third position (0.7), while others, depending on the temperature (and other) settings might sample from the distribution [0.1, 0.2, 0.7] or [0.0001, 0.05, 0.9499] or other.

And usually that (sampling) has nothing to do with training the model on some dataset.

> [@Kamal\_Nayan](#):
>
> 3)And about my question of update gate , check if the following is correct.  
> basically if our update value is low then c ~= c , so if c is not changing much then our model is trained by values that were almost similar to the earlier layers Am i correct with this understanding

If I understand you correctly - no.

First, _update_ gate value is not c (c in your picture is the **hidden state** , or H\_t in my previous illustration), but update is z (\Gamma\_u in your picture, Z\_t in my illustration), which **influences** the _hidden_ state (but is not the same).

Second, the update gate value is calculated at every step (from previous hidden state and current input).

Third, it depends how you understand layers - I think you confuse layers vs. steps ([check this post](https://community.deeplearning.ai/t/creating-a-gru-model-using-trax/165167/4)).

So, if you understand what I wrote, then the update value controls how much this **step** (hidden state of this step c\_t) is a copy of the _previous_ hidden state (c\_{t-1}).

Cheers

---

_[View the full topic](https://community.deeplearning.ai/t/sampling-and-rnns/409632)._