Unable to Exclude OOV Tokens

Course Q&A TensorFlow Developer Professional Natural Language Processing in TensorFlow
1

When creating a post, please add:

For Exercise 4: fit_label_encoder, my function is making all my OOV Tokens None. To exclude the OOV tokens from my vocabulary, I am setting the oov_token parameter to None in my tf.keras.layers.StringLookup function. However, I’m getting that the first element of my Vocabulary of labels is None. How can I fix this?

image
image1225×296 9.86 KB

1 Like
2

As a hint, these are the arguments according to the Tensorflow documentation:

tf.keras.layers.StringLookup(
    max_tokens=None,
    num_oov_indices=1,
    mask_token=None,
    oov_token='[UNK]',
    vocabulary=None,
    idf_weights=None,
    invert=False,
    output_mode='int',
    pad_to_max_tokens=False,
    sparse=False,
    encoding='utf-8',
    name=None,
    **kwargs
)

In order to exclude OOV tokens, your argument of interest should be num_oov_indices. Hope this helps.

7 Likes
3

Oh, I see. Thanks!

4

i am still getting the same error by setting the num_oov_indices to 1

5

thanks i was able to get it by setting it to the right valule

1 Like
6

Props to you for finding the right answer yourself.

1 Like
7

I still can’t rule it out. Can you tell me how you did it?

8

{moderator edit: solution code removed}
no OOV indices are generated

9

Sorry I had to remove the solution code as it is not allowed by the community guidelines. Mentors can only provide hints so the person learning can work towards the solution. This is to ensure the integrity of the course. If the learner follows this topic and tries a few times, I’m sure the learner will figure out the answer. This way, learning is more effective. Thank you.

1 Like
10

was able to resolve issue after seeing @lukmanaj comment.
But before coming to forum, I gone through documentation.
Unfortunately the parameter description in tensorflow didn’t make sense to me at that time. I feel tensorflow documentation about this parameter need little more clarification on its usages.

1 Like