Hey all, I am going through the ungraded labs (Module 4 : Exploring LLM capabilities) and am at Section 3.2 (Nucleus or top-p sampling). When I run the cells with top_p = 0, I am expecting the three outputs to be the same but they are not. I can get them to be identical if I set the temperature to zero as well.
Any help in explaining this discrepancy is much appreciated!
Hi achhotray.
In my case, I run the cell with top_p = 0, and I always get the same result.
This was the result it showed me:
When we set top_p = 0, or temperature = 0, the “greed decoding” method is most likely used to select the next token. This causes the model to choose the word with the highest probability, and the model always behaves similarly, so you’ll get the same result.
In the introduction in section 3.1 of the C1M4_Ungraded_lab_1 notebook, it talks about “greed decoding.”
Hi @ribarola : thanks for your reply.
The discrepancy I mention is that all three responses should have been exactly the same which aren’t. However, I get exactly the same responses when I run the query with temperature set to zero.
I also find it puzzling that when you run the identical cell with the exact parameters you get a different response that I do (I assume you are running the same environment as I have) - which should not be the case.
If I may ask, how many times did you run the cell?
