FineTuninig LLMS

Hello,
I have created an application which takes a context and generates question and answer from the context. For this I have chosen T5 model ( as it is a sequence to sequence task). Initially, I was improving in my task (the dataset size which I tried initially was of 1000 examples later I increased the dataset size to 25000 ) but after few epochs (around 2nd epoch) I am unable to improve my rouge score at all. I tried to train for 50 epochs but the result is the same. My rouge score was not at all improving( I got around 0.3 in rouge-1, 0.18 in rouge-2). The result was my model generating repeated choices for multiple choice questions.

Further implementation details are given here GitHub - sujith2303/Quizbot.ai.

  1. What can be the reason for not improving my rouge scores and the model’s quality of generating questions?

  2. Tried using few shot at inference but there is not use. Why don’t few shot learning apply here?

  3. If I use different temperatures and change the number of questions to return parameters, there is not change. It is generating same questions. For instance , if I use return sequences as 10, it is generating 10 same questions. Is there any other way to generate different output?