# Generating gibberish chat with the ReformerLM

**URL:** <https://community.deeplearning.ai/t/generating-gibberish-chat-with-the-reformerlm/296976>\
**Category:** NLP with Attention Models\
**Tags:** week-module-4\
**Created:** [March 26, 2023, 2:28pm UTC](https://community.deeplearning.ai/t/generating-gibberish-chat-with-the-reformerlm/296976 "2023-03-26T14:28:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cmosguy](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/cmosguy/32/18724_2.png) [@cmosguy](https://community.deeplearning.ai/u/cmosguy)\
**Post date:** [March 26, 2023, 2:28pm UTC](https://community.deeplearning.ai/t/generating-gibberish-chat-with-the-reformerlm/296976/1 "2023-03-26T14:28:40Z")

</div>

So, when I create a dialog the `temperature=0.2`, which is supposed to be more randomness, correct?

Also, the output of the dialog has some gibberish, see below where Person 2 responds with **“south, south, south, south”**. Is this an issue with the model itself, or how well it was trained in the first place? This is using the pretrained model associated with the lab.

Or, are my expectations too high? Is this what is expected with a pretrained dataset of ~10k dialogue.

```plaintext
Person 1: Are there theatres in town?
Person 2: : There are 4 theatres in town. Two are in the centre of town and 1 in the south. Do you have a preference? 
Person 1: I would like the south please. 
Person 2: There are 4 theatres in the south, south, south, and south. Which would you prefer? 
Person 1: I would prefer the south please. 
Person 2: I would recommend the nusha. Would you like more information? 
Person 1: Yes, could I need to know the area, and area of town.

```

---

<div class="post-metadata">

**Author:** ![Elemento](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/elemento/32/854_2.png) [@Elemento](https://community.deeplearning.ai/u/Elemento)\
**Post date:** [March 27, 2023, 5:23am UTC](https://community.deeplearning.ai/t/generating-gibberish-chat-with-the-reformerlm/296976/2 "2023-03-27T05:23:22Z")

</div>

Hey @cmosguy,

> [@cmosguy](#):
>
> Or, are my expectations too high? Is this what is expected with a pretrained dataset of ~10k dialogue.

I guess you have answered your query yourself. The data in the dataset is completely static, there is no way for the model to access the web, and compared to what is fed to LLMs of today, the data source for this model is miniscule. Despite of this, the model is at least able to produce comprehensible conversations, with a few discrepancies, as you just found out. I guess your expectations are a bit influenced by the recent developments in LLMs, like ChatGPT, etc. Nonetheless, I am sure you would be able to appreciate what the model has learned that too from a static dataset of just 10,000 dialogues. I hope this helps.

Cheers,  
Elemento

---

<div class="post-metadata">

**Author:** ![cmosguy](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/cmosguy/32/18724_2.png) [@cmosguy](https://community.deeplearning.ai/u/cmosguy)\
**Post date:** [March 27, 2023, 3:47pm UTC](https://community.deeplearning.ai/t/generating-gibberish-chat-with-the-reformerlm/296976/3 "2023-03-27T15:47:53Z")

</div>

Thanks @Elemento, yes I am able to now comprehend the scale at which the ChatGPT has been trained on. The essence is that the dataset is so large, that the model appears to be magical because it has trained on so many sequences of tokens.

Thanks again for the support!