About notebook C2_W2_lecture_nb_1_strings_tags

Course Q&A Natural Language Processing NLP with Probabilistic Models
1

How would you classify an unseen token that contains both digit and punctuation for example?

2

Hi @Natalie_Caruana

In reality you would have to decide yourself how you would handle the situation you described.

In the Lab notebook the token you described would be classified as “–unk_digit–”, because the “priorities” are:

  1. if any is a digit - (this is the case you descibed)
  2. if any is a punctuation character
  3. if any is an upper case character
  4. if word ends with any noun suffix
  5. if word ends with any verb suffix
  6. if word ends with any adjective suffix
  7. if word ends with any adverb suffix
1 Like