Generating gibberish chat with the ReformerLM

Course Q&A Natural Language Processing NLP with Attention Models
1

So, when I create a dialog the temperature=0.2, which is supposed to be more randomness, correct?

Also, the output of the dialog has some gibberish, see below where Person 2 responds with “south, south, south, south”. Is this an issue with the model itself, or how well it was trained in the first place? This is using the pretrained model associated with the lab.

Or, are my expectations too high? Is this what is expected with a pretrained dataset of ~10k dialogue.

Person 1: Are there theatres in town?
Person 2: : There are 4 theatres in town. Two are in the centre of town and 1 in the south. Do you have a preference? 
Person 1: I would like the south please. 
Person 2: There are 4 theatres in the south, south, south, and south. Which would you prefer? 
Person 1: I would prefer the south please. 
Person 2: I would recommend the nusha. Would you like more information? 
Person 1: Yes, could I need to know the area, and area of town.
2

Hey @cmosguy,

I guess you have answered your query yourself. The data in the dataset is completely static, there is no way for the model to access the web, and compared to what is fed to LLMs of today, the data source for this model is miniscule. Despite of this, the model is at least able to produce comprehensible conversations, with a few discrepancies, as you just found out. I guess your expectations are a bit influenced by the recent developments in LLMs, like ChatGPT, etc. Nonetheless, I am sure you would be able to appreciate what the model has learned that too from a static dataset of just 10,000 dialogues. I hope this helps.

Cheers,
Elemento

1 Like
3

Thanks @Elemento, yes I am able to now comprehend the scale at which the ChatGPT has been trained on. The essence is that the dataset is so large, that the model appears to be magical because it has trained on so many sequences of tokens.

Thanks again for the support!