How to you actually adjust the parameters using PyTorch? Top P, Top K, Temp, etc. didn’t impact my output after I used few shot prompting. Is this just bc the notebook file and output text was so simple?
You can configure Top P, Top K parameter in GenerationConfig. To watch impact result, maybe you need to use with do_sample=True
1 Like
To observe the impact of Top P, Top K and Temperature, GenerationConfig must be instantiated with do_sample = True. The configurations only work when the random weighting strategy is used, which is when the value of the parameter do_sample is set to True. Note that the variable do_sample is optional and defaults to False when omitted.
1 Like