# C3 W1 assignment: Vocabulary contains 29608 words instead of 29714

**URL:** <https://community.deeplearning.ai/t/c3-w1-assignment-vocabulary-contains-29608-words-instead-of-29714/148426>\
**Category:** Natural Language Processing in TensorFlow\
**Tags:** week-module-1\
**Created:** [June 27, 2022, 10:57am UTC](https://community.deeplearning.ai/t/c3-w1-assignment-vocabulary-contains-29608-words-instead-of-29714/148426 "2022-06-27T10:57:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jpgeek](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jpgeek](https://community.deeplearning.ai/u/jpgeek)\
**Post date:** [June 27, 2022, 10:57am UTC](https://community.deeplearning.ai/t/c3-w1-assignment-vocabulary-contains-29608-words-instead-of-29714/148426/1 "2022-06-27T10:57:13Z")

</div>

My `fit_tokenizer()` function seems to be returning the wrong number of words in the word\_index.

`len(word_index)` is 29608 words instead of the expected 29714. I have verified that my  
`remove_stopwords()`  
is working correctly. Even if it were not, I should get more words rather than less. Is there some way to get the reference dict of words so that I can compare, or does anyone have any other ideas for debugging this?

---

<div class="post-metadata">

**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\
**Post date:** [June 27, 2022, 12:53pm UTC](https://community.deeplearning.ai/t/c3-w1-assignment-vocabulary-contains-29608-words-instead-of-29714/148426/2 "2022-06-27T12:53:23Z")

</div>

Here you go:  
[word\_index.js](https://community.deeplearning.ai/uploads/short-url/kll97kx4rAncAMj7jGvRPnqYdAU.js) (516.3 KB)

---

<div class="post-metadata">

**Author:** ![jpgeek](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jpgeek](https://community.deeplearning.ai/u/jpgeek)\
**Post date:** [June 27, 2022, 4:34pm UTC](https://community.deeplearning.ai/t/c3-w1-assignment-vocabulary-contains-29608-words-instead-of-29714/148426/3 "2022-06-27T16:34:05Z")

</div>

@balaji.ambresh , once again, thank you so much for your extremely prompt and thorough help.

I diffed the keys and am quite confused now - the keys in the list that you so kindly sent include the stopwords. However the input parameter: `sentences` is assigned by the call to `parse_data_from_file` which specifically is supposed to strip the stopwords. Shouldn’t the stopwords have been removed from the word\_index?

---

<div class="post-metadata">

**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\
**Post date:** [June 27, 2022, 5:32pm UTC](https://community.deeplearning.ai/t/c3-w1-assignment-vocabulary-contains-29608-words-instead-of-29714/148426/4 "2022-06-27T17:32:29Z")

</div>

That is correct. We remove stopwords from sentences.

Tensorflow `Tokenizer` does tokenization differently:  
It strips special characters like `-`. Here’s an example:

```auto
ids = tokenizer.texts_to_sequences(['high-definition tv a-la-carte'])[0]
print(ids)
print([tokenizer.index_word[id] for id in ids])

# output
[9, 10, 2, 139, 140, 141]
['high', 'definition', 'tv', 'a', 'la', 'carte']

```

You can set the `filters` field in [Tokenizer](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer) to customize the behavior.

---

<div class="post-metadata">

**Author:** ![jpgeek](https://avatars.discourse-cdn.com/v4/letter/j/9fc348/32.png) [@jpgeek](https://community.deeplearning.ai/u/jpgeek)\
**Post date:** [June 27, 2022, 11:27pm UTC](https://community.deeplearning.ai/t/c3-w1-assignment-vocabulary-contains-29608-words-instead-of-29714/148426/5 "2022-06-27T23:27:49Z")

</div>

Thanks again.

So the stopwords that were included in the file that you sent were just the remnants of hyphenated words that were split apart by the tokenizer. That makes it clear.