Why do we need to have the dataset from same distribution?

Short Course Q&A Reinforcement Learning from Human Feedback
1

In “dataset for RL training” its stated that we need to have the dataset from same distribution at the end of the video.

My question is why? if its a summarization task, can’t we have it sourced from different sources with different distribution or topics? or i got the phrase wrong?

2

Models are trained on the characteristics of the training set.
So it’s important that the training data resemble the data you’ll find in real use of the model later. Otherwise you will not get good results when using the model.