W4 A1 number of heads vs

[jakhon77]

Can someone help me understand why changing the number of heads does not affect the output dimension? As far as I know, the lecture states that the heads are concatenated, which should impact the output dimension. Thank you in advance!

[Deepti_Prasad]

concatenation of head is to the weight, which if you see when head(i) is changed the attention weight is applied to all 3 multinhead attention, allowing the relative q, k and v value to be selected and getting the same output as the head(i) output.

Self-attention in the Transformer Network

We are not assigning q=k=v=x in encoder, but we create a matrix using the q, k and v relative to a given x, so it’s x position is determined from its ability to relate independently as well as positional to all other inputs(x’s), allowing x to be determined in decoder as well as transformer model using the similar matrix used in encoder for a given a input.