# C3W1-Assignment -\> too much words in vocab and wrong shape **URL:** \ **Category:** Natural Language Processing in TensorFlow\ **Created:** [January 24, 2024, 7:53am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416 "2024-01-24T07:53:29Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![RoWe84](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rowe84/32/90701_2.png) [@RoWe84](https://community.deeplearning.ai/u/RoWe84)\ **Post date:** [January 24, 2024, 7:53am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/1 "2024-01-24T07:53:29Z")
Hello everyone, i got stuck within the first assignment. The parse\_data\_from\_file function works as expected but the fit\_tokenizer gave me an output of Vocabulary contains 30888 words token included in vocabulary Inside the function the Tokenizer gets only the OOV parameter. As a consequence the next function get\_padded\_sequences gave me this output. First padded sequence looks like this: [0 0 0 … 87 87 8] Numpy array of all sequences has shape: (2225, 2439) This means there are 2225 sequences in total and each one has a size of 2439 With a mismatch in the number sequence and in the desired size of the array (2439 instead of 2438). Even if i try to do a post padding the numbers are not correct and obviously no change in shape. Can someone give me a hint where to look at? Many thanks in advance! ---
**Author:** ![Deepti\_Prasad](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/deepti_prasad/32/165831_2.png) [@Deepti\_Prasad](https://community.deeplearning.ai/u/Deepti_Prasad)\ **Post date:** [January 24, 2024, 9:00am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/2 "2024-01-24T09:00:39Z")
Hello @RoWe84, Can you share the output you got for parse data from file grader cell? Also make sure for your fit\_tokenizer grader cell, if you have recalled the fit on the sentence code correctly. Regards DP ---
**Author:** ![RoWe84](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rowe84/32/90701_2.png) [@RoWe84](https://community.deeplearning.ai/u/RoWe84)\ **Post date:** [January 24, 2024, 10:00am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/3 "2024-01-24T10:00:59Z")
Thanks for your fast reply! I’ve added the output as pictures. ![DLTF-C3W1-1](https://global.discourse-cdn.com/dlai/original/3X/a/3/a36d49be9aff17c53ef1f05b502167dfa87d35ef.png) ![DLTF-C3W1-2](https://global.discourse-cdn.com/dlai/original/3X/6/8/6816d2cad7a499957ffc2f2dd5b43ce3790d6d60.png) Haven’t seen something wrong honestly. ---
**Author:** ![Deepti\_Prasad](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/deepti_prasad/32/165831_2.png) [@Deepti\_Prasad](https://community.deeplearning.ai/u/Deepti_Prasad)\ **Post date:** [January 24, 2024, 10:39am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/4 "2024-01-24T10:39:59Z")
fit\_tokenizer output image please ---
**Author:** ![RoWe84](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rowe84/32/90701_2.png) [@RoWe84](https://community.deeplearning.ai/u/RoWe84)\ **Post date:** [January 24, 2024, 11:05am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/5 "2024-01-24T11:05:24Z")
![DLTF-C3W1-3](https://global.discourse-cdn.com/dlai/original/3X/e/6/e6469e241d31432593fa736b763a1ef5e554392d.png) ---
**Author:** ![Deepti\_Prasad](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/deepti_prasad/32/165831_2.png) [@Deepti\_Prasad](https://community.deeplearning.ai/u/Deepti_Prasad)\ **Post date:** [January 24, 2024, 11:11am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/6 "2024-01-24T11:11:59Z")
how did you recall fit in the sentence code line ---
**Author:** ![RoWe84](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rowe84/32/90701_2.png) [@RoWe84](https://community.deeplearning.ai/u/RoWe84)\ **Post date:** [January 24, 2024, 11:25am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/7 "2024-01-24T11:25:20Z")
After creating a tokenizer instance with the OOV parameter i basically called {Correct Code: Removed by Moderater} ---
**Author:** ![Deepti\_Prasad](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/deepti_prasad/32/165831_2.png) [@Deepti\_Prasad](https://community.deeplearning.ai/u/Deepti_Prasad)\ **Post date:** [January 24, 2024, 11:30am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/8 "2024-01-24T11:30:26Z")
How did you instantiate tokenizer class Also check if in your parse data codes, if you have remove\_stopwords for both labels(row[0]) and sentences (row[1]) as per row ---
**Author:** ![RoWe84](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rowe84/32/90701_2.png) [@RoWe84](https://community.deeplearning.ai/u/RoWe84)\ **Post date:** [January 24, 2024, 12:07pm UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/9 "2024-01-24T12:07:47Z")
The tokenizer class was instantiated this way from tensorflow.keras.preprocessing.text import Tokenizer tokenizer = Tokenizer(oov\_token=“”) // the OOV is not displayed but in the code. I’ve checked the parse data function. I called the remove stopwords only for the sentences and not for the labels. Just for testing i added the function call to the labels with no effect on the vocab size nor the shape. ---
**Author:** ![RoWe84](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rowe84/32/90701_2.png) [@RoWe84](https://community.deeplearning.ai/u/RoWe84)\ **Post date:** [January 24, 2024, 12:09pm UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/10 "2024-01-24T12:09:37Z")
hard to answer without posting lines of code. Sorry for that! ---
**Author:** ![Deepti\_Prasad](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/deepti_prasad/32/165831_2.png) [@Deepti\_Prasad](https://community.deeplearning.ai/u/Deepti_Prasad)\ **Post date:** [January 24, 2024, 12:36pm UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/11 "2024-01-24T12:36:33Z")
Can you share an image of parse data grader cell codes as well as the first grader code cell via personal DM. Click on name and then message. I am more certain issue could be in these cells. Also the instantiate tokenizer describes you to use oov\_token=’ '(here the OOV need to be used with \< \> I hope you have used the same. ---
**Author:** ![RoWe84](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rowe84/32/90701_2.png) [@RoWe84](https://community.deeplearning.ai/u/RoWe84)\ **Post date:** [January 25, 2024, 5:55am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/12 "2024-01-25T05:55:02Z")
Problem solved - you pointed me into the right direction. Many thanks! The issue was in the parse data grader cell. If i would have read the instructions carefully it would have saved me some time. I’ve used row[1:] for building the sentences which was incorrect. After changing the rest was argument parsing. ---
**Author:** ![Deepti\_Prasad](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/deepti_prasad/32/165831_2.png) [@Deepti\_Prasad](https://community.deeplearning.ai/u/Deepti_Prasad)\ **Post date:** [January 25, 2024, 10:20am UTC](https://community.deeplearning.ai/t/c3w1-assignment-too-much-words-in-vocab-and-wrong-shape/553416/13 "2024-01-25T10:20:47Z")
Hello @RoWe84, Mistakes and debugging is a part of programming and you catched the part with hints provided and corrected the issue. Happy to help!!! Keep learning!!! Regards DP