In the Transformers in Practice course, I am at Visualization 1.3: How Sampling Shapes Output.
I don’t see any difference in the multiple-completion outputs when I set the temperature to 2 and Top-K to 1 or 5.
If Top-K is 1, it should produce the same result across all completions, as per my understanding so far. However, the outputs vary significantly and are not meaningful.
Is the visualization broken, or am I missing something?
Hi Piyush_Raut,
This can depend on the prompt you are using. The default prompt that is provided leads to a very high probability for ‘The’ as the next token. You can also see this in the multiple-completion when you look at the probability.
See if you can use a prompt that allows for more variability in completion.
As an example, I used ‘How would you respond to a question on a forum?’ which led to very different completions with shifts in Temperature and Top-K.
Thank you! I can now see the difference between the Top-K = 5 and Top-K = 1 responses with different prompts.
However, I still don’t understand why I get different results in multiple-completions when I set Temperature = 2 and Top-K = 1.
Wouldn’t Top-K = 1 restrict the sampling to just the single token with the highest probability? And since temperature doesn’t change the ranking of the tokens, shouldn’t the highest probability token always be selected?
attached image of temp:2, top-K:1 for the prompt:“How would you respond to a question on a forum?”
the different output after each completion of loop is because the model has randomness introduced probably, so for each loop the token selection is different for the prompt response. Randomness is introduced to avoid repetitive loop of model learning introducing randomness of selection of token in the given set of token, so if you had select top-k as 1 and temperature being 2; the temperature will become more flat and top-k although filtering the single token, for each loop, the randomness must have select any token in the given set of token which must have in the backend codes. this is a standard practice to avoid repetitive loop or token utilisation as the model wouldn’t learn anything with each loop without randomness.
Regards
Dr. Deepti
Thanks for the reply. Just to make sure I understand, are you saying the backend isn’t actually enforcing the topk = 1 setting?
The way I see it is that with topk= 1, only a single token survives filtering ( 100% probability after renormalization), and since we do sampling after filtering, the choice of sampling strategy becomes irrelevant. whether it’s random sampling or greedy decoding, there’s only one token available to pick. temperature reshapes the distribution but can’t change which token is ranked #1.
Hi Piyush_Raut,
When I set top-K to 1 I always get ‘When’ as the output token. Did you click reset?
Thanks for checking! I think we might be testing different settings though. I am asking about temperature = 2 and topK = 1 set together.
Here’s what I noticed: with that combination, the “Choose Your Own Completion” block gives me “When” as the first token every time, even after multiple runs. But the “Multiple Completions” block still produces different outputs. So maybe that block isn’t applying topK (maybe a bug in that code).
Could you try that exact combination in the “Multiple Completions” block and see if you get “When” as first token in all ?
Here’s what I did and found just now.
This gives me ‘When’ with probability 46.3% and weight of 100.0%
This gave me 8 trials, each of which predicting ‘When’
If you don’t want to restart the notebook, you have to click Reset. But this moves the sliders back to their original position, so you will have to tweak these back to the values you want. This is what I did before I responded to your previous post. This also gave me ‘When’ for both the distribution and the 8 trials.
I am still getting same results, couldn’t find reset button. Attaching a screenshot of the same but it’s not needed to continue on this as my doubt is solved.
Thanks for running that detailed testing that it should be starting with “When” always . We can close this.
Thanks again for confirming this.
The Run Trials button changes into a Reset button.
Anyway, thanks for your questions which may be helpful for others taking the course!
