Language Model Evaluation issue

Course Q&A Natural Language Processing NLP with Probabilistic Models
1

Video ‘Language Model Evaluation’
Looks like quiz in the video doesn’t have correct answer. ‘Mary likes cats’ => m = 3 and answer should be log PP(W) = (-1/3) * (-113)
image

2

Hi Kiril,

I was also wondering about this, although I would say the correct answer is (-1/5)*113. This is what you get when you include the start- and end-of-sentence tokens as argued in this presentation.

I’ll make an issue of this at the backend and see where it leads.

Thanks!

3

Fix it also this in Practice quiz.

4

Thanks! I have added this to the issue.

5

Hi Kiril,

It turns out perplexity is presented in the video ‘Language Model Evaluation’ at 1:57 as m including </s> but not <s>. This would seem reasonable if you are sure you will get a sentence but do not know how long the sentence will be. From this perspective, the answer log PP(W) = (-1/4)*(-113) is in line with the lecture.