Multiheaded Attention - Number of heads and Dim of heads

[arvyzukai]

Hi @nmurugesh

@nmurugesh

In Transformer Summary assignment, the dimension of the heads is calculated by dividing the feature embedding by the number of heads.

Yes, this is true - this is how you calculate with how large dimension each head will operate on.

@nmurugesh

In other words, each head is part of the initial word embedding for the q,k,v. Is this the right method or is this a specific implementation adopted for this assignment?

Not entirely true. Each head get its own projection, or in others words, each head gets its own compressed/transformed version of embeddings.

@nmurugesh

In Deep Learning Specialization (course 4 week 4), Andrew Ng explains the multiheads as several equal and parallel computations where there are separate learned q.W,k.W,v.W for each head and NOT just reshaping of the original embedding and then concatenating them.

In the implementation of the multihead in transformer summary assignment, the only benefits seems that each head can be computed parallel as against computing all as single head but that is not the original idea behind multihead attention.

As in my previous point I think you misinterpret the code. I recently answered a similar question which might help illustrate the point that even the operation is single (to get Q, K, V) but the underlying channels of information are separate. Or in other words emb \cdot W_q is a single operation (for efficiency) which produces one output, but later the output is split up (isolated) for different heads.

@nmurugesh

To check this, I calculated multihead attention for some sample q,k,v first for entire embedding (without d_head = d_feature / n_heads) and then for several heads and subsequent concatenation. I find the results are same whether it is single head or multihead (except for the differences that might arise due to linear layer during multiple iterations of the decoder block).

I would doubt the results if the training is done long enough for a dataset large enough - multi-head should be superior to single head. In other words, if your results are same there must be some underlying reason.

@nmurugesh

Kindly clarify whether the d_head should be d_feature / n_heads or each d_head should be same as dimension of word embedding of q / x.

The head dimension is definitely d_feature / n_heads. To convince yourself can :

• check the Attention Is All You Need, in particular:

In this work we employ h = 8 parallel attention layers, or heads. For each of these we use
dk = dv = dmodel/h = 64. Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality.

• or check other DL frameworks implementations, like pytorch MultiheadAttention documentation, in particular:

• num_heads – Number of parallel attention heads. Note that embed_dim will be split across num_heads (i.e. each head will have dimension embed_dim // num_heads).