Hello everyone,
I have a question about the SoftMax formulation parameters(e_c, T_theta_t)
Here’s how I currently understand it:
The context word embedding vector e_c is computed as E @ o_c, where o_c is a one-hot vector for the context word and E is the input embedding matrix (learnable parameters).
e_c is the context word embedding vector and computed as: o_c * E
where o_c is a one-hot vector for the context word.
E matrix is learnable parameters of the model, and will be updated in each batch/mini
batch, with shape(number_of_features, vocab_size)
Now, my question is about “θ_t”
Is “θ_t” the output embedding vector corresponding to word t, just like e_c is the
embedding vector for the context word? like (θ_t = o_t * E) ?
To clarify further with an example:
If both the context word and the target word are the same, e.g., context = "Noah" and target = "Noah", does that mean e_c == θ_t in that case?
Target word embeddings and Context word embeddings are two different sets of embeddings and they are both initialized randomly, so e_c is not equal to θ_t.
To take an embedding, we do e_c = E @ o_c for the context word, and, similarly, θ_t = Θ @ o_tfor the target word.
Note the difference in the order of the variables and the operators in your post, but the above is how I would write them.
Thank you so much for your clarification, and apologies for my late reply. I have one more question, please: the final values of the embedding matrix that we use afterwards — are they considered E or Θ?
If you just need a set of embedding for your words, then the target word embedding can be your first choice.
Moreover, in some studies, people found the aggregation of the two could do better. For example, GloVe, which is work published after Word2Vec and is also covered by this course, has mentioned the following in its paper:
