Vocabulary size differs from your preloaded V dimension of W1,W2 and b2 for testing

Course Q&A Natural Language Processing NLP with Probabilistic Models
1

After preprocessing the corpus, I get a vocabulary size of 5775, while the V dimension of your preloaded parameters in unit_test functions is 5778.
That leads to failing several unit_test functions.

2

How did you preprocess the corpus? You should be getting 5778 for the vocab size.

4

I just run your notebook code.
Cheers
Antonino

screenshot.jpg
screenshot.jpg1630×999 286 KB

5

Thats strange. Did it start like this? Maybe you can restart the kernel?

6

I’ve restarted the kernel, and nothing has changed. After the preprocessing, I get 60976 tokens. Is it your number? Then computing fdist = nltk.FreqDist(word for word in data), I get 5775 unique words.

7

I am facing the similar issue. My vocabulary size is 5775 instead of 5778