Attention Output shape

[TMosh]

Also, I think you will be interested in the discussion at this thread:

Understanding multi-headed attention - C5W4A1

This is a question about understanding the inputs /outputs of the mha layer calls in the decoder blocks that form the core of the transformers model in the final assignment of the DLS. The second multi-head attention (MHA) block in the decoder portion of the transformer involves two inputs, according to the lectures and the figure below. Two matrices of key and value vectors derived from the encoder, i.e. K and V (which are ultimately derived from the input sentence) A matrix of query values …