Question on Top P Sampling ( Generative Configuration)

Its mentioned when p=0.3, it will pick the; 0.1 and 0.2 (because they add up to .3).

What if probabilities are as eg:

apple 0.28
banana 0.10
cake 0.02
etc.

If its top p = 0.3;
would it pick all three?

as apple + banana = 0.38 > 0.3 as well as
apple + cake = 0.3

Probably not, it will pick only the 2.

Per my reading, the top p picks the one with cumulative probability greater or equal to the top p value. so in this case, 0.38 will be chosen because it is greater than the top p value(0.38>0.3)

2 Likes

Based on my reading from the following resources:

I also think it should be first two tokens - apple + banana. The top_p parameter sets the threshold on cumulative probability of tokens so that we can restrict tokens from being considered for next token prediction.

I think there is a mistake in that video.

Top-k sampling refers to selecting the next token randomly from a specified number, k, of tokens with the highest probabilities.

Top-p sampling refers to selecting the next token randomly from the smallest set of tokens for which the cumulative probability exceeds (or is equal) a specified value, p. and that’s why 0.1 and 0.2 where selected