Its mentioned when p=0.3, it will pick the; 0.1 and 0.2 (because they add up to .3).
What if probabilities are as eg:
apple 0.28
banana 0.10
cake 0.02
etc.
If its top p = 0.3;
would it pick all three?
as apple + banana = 0.38 > 0.3 as well as
apple + cake = 0.3
Probably not, it will pick only the 2.
Per my reading, the top p picks the one with cumulative probability greater or equal to the top p value. so in this case, 0.38 will be chosen because it is greater than the top p value(0.38>0.3)
Based on my reading from the following resources:
I also think it should be first two tokens - apple + banana. The top_p parameter sets the threshold on cumulative probability of tokens so that we can restrict tokens from being considered for next token prediction.
I think there is a mistake in that video.
Top-k sampling refers to selecting the next token randomly from a specified number, k, of tokens with the highest probabilities.
Top-p sampling refers to selecting the next token randomly from the smallest set of tokens for which the cumulative probability exceeds (or is equal) a specified value, p. and that’s why 0.1 and 0.2 where selected
The video says the cumulative probability should be <=p, but by logic >=p makes more sense. Because we want the o/p to be creative, but we are limiting its randomness by putting probability constraints. so >=p should be correct right?