# Unexpected output for fit\_tokenizer function

**URL:** <https://community.deeplearning.ai/t/unexpected-output-for-fit-tokenizer-function/255559>\
**Category:** Natural Language Processing in TensorFlow\
**Tags:** week-module-3, week-module-2, week-module-4\
**Created:** [January 11, 2023, 1:23pm UTC](https://community.deeplearning.ai/t/unexpected-output-for-fit-tokenizer-function/255559 "2023-01-11T13:23:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bruce\_Wileman](https://avatars.discourse-cdn.com/v4/letter/b/779978/32.png) [@Bruce\_Wileman](https://community.deeplearning.ai/u/Bruce_Wileman)\
**Post date:** [January 11, 2023, 1:23pm UTC](https://community.deeplearning.ai/t/unexpected-output-for-fit-tokenizer-function/255559/1 "2023-01-11T13:23:59Z")

</div>

When testing the fit\_tokenizer function, I get:

Vocabulary contains 27284 words

token NOT included in vocabulary

Instead of the expected:

Vocabulary contains 27285 words

token included in vocabulary

Any ideas why this might be the case?

---

<div class="post-metadata">

**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\
**Post date:** [January 11, 2023, 2:57pm UTC](https://community.deeplearning.ai/t/unexpected-output-for-fit-tokenizer-function/255559/2 "2023-01-11T14:57:08Z")

</div>

Set the [oov\_token](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer#args_1)

---

<div class="post-metadata">

**Author:** ![Bruce\_Wileman](https://avatars.discourse-cdn.com/v4/letter/b/779978/32.png) [@Bruce\_Wileman](https://community.deeplearning.ai/u/Bruce_Wileman)\
**Post date:** [January 12, 2023, 12:12pm UTC](https://community.deeplearning.ai/t/unexpected-output-for-fit-tokenizer-function/255559/3 "2023-01-12T12:12:33Z")

</div>

Great, that worked! Thanks! 👌