def calculate_perplexity(sentence, n_gram_counts, n_plus1_gram_counts, vocabulary_size, start_token='<s>', end_token = '<e>', k=1.0): seems simple and straight-forward. However, unit tests fail… Any idea what I might be missing?
I added the same instrumentation to my code and here’s what I see when I run that test:
n = 1 N = 6
word i probability 0.2222222222222222 product_pi 4.5
word like probability 0.25 product_pi 18.0
word a probability 0.3333333333333333 product_pi 54.0
word cat probability 0.3333333333333333 product_pi 162.0
word <e> probability 0.3333333333333333 product_pi 486.0
perplexity 2.8039657955522013
n = 1 N = 6
word i probability 0.2222222222222222 product_pi 4.5
word like probability 0.25 product_pi 18.0
word a probability 0.3333333333333333 product_pi 54.0
word dog probability 0.1111111111111111 product_pi 486.0
word <e> probability 0.125 product_pi 3888.0
perplexity 3.965406456500188
n = 2 N = 8
word in probability 0.08 product_pi 12.5
word sunset probability 0.09090909090909091 product_pi 137.5
word sky probability 0.09090909090909091 product_pi 1512.5
word is probability 0.09090909090909091 product_pi 16637.5
word red probability 0.09090909090909091 product_pi 183012.5
word -2 probability 0.09090909090909091 product_pi 2013137.5
perplexity 6.137396479150367
n = 2 N = 7
word i probability 0.08 product_pi 12.5
word like probability 0.09090909090909091 product_pi 137.5
word a probability 0.09090909090909091 product_pi 1512.5
word cat probability 0.13043478260869565 product_pi 11595.833333333334
word -2 probability 0.13043478260869565 product_pi 88901.38888888889
perplexity 5.0931554910158665
All tests passed
Notice that all the probability values are different. Are you sure that your estimate_probability function is correct and that you’re calling it correctly (not referencing global variables)?
Also notice that I’m handling the indexing slightly differently: I don’t see the token <s>, but the token <e> in my logic.
{moderator edit - solution code removed}
All local variables.
Your call to estimate_probability looks correct, but the way you compute product_pi is not. Well, you might compensate later in code we can’t see, but please compare your code to the math formula shown in the instructions and in that comment.
I know. Either I use 1/ or **(-1/N) doesn’t make any difference at all.
Did you also fix the indexing issue that I pointed out?
I already pointed out the mistake in how you compute product_pi.
But notice that your probability values on the individual words are also different than the values that I showed.
w3_unittest.test_estimate_probability(estimate_probability) all tests pass. So I assume there is no problem with the word probability estimation at all?
If the tests pass, then that is probably not where the problem is. Maybe it’s time to look at your full notebook. We can’t do that on a public thread, but check your DMs for a message from me.