I have some confusion regarding the MultiHeadAttention layer. When we call an MHA layer with multiple heads and query, key, value all equal to X, using the lecture’s notation, is this equivalent to something of this form (ignoring the mask and batches) Attention(Q,K,V) = Attention(W_i^QX,W_i^KX,W_i^…

In that case you should scale by 1/sqrt(d_k). Where do you see d_model? In the code, I reference the last dimension of K, which probably is d_k. But if you run attention without the linear transformation, i.e. on X and not X * W^k_i, then you actually end up with d_k = d_model in the code.

Question about MultiHeadAttention layer

Course Q&A Deep Learning Specialization Sequence Models

jonaslalin August 22, 2021, 1:35pm 2

Let me think about this one more time

Clarification regarding attention and self attention

Topic		Replies	Views
C5 W4 A1: Question about MultiHeadAttention Sequence Models	2	732	August 4, 2021
Multiheaded attention question Sequence Models week-4	1	265	January 6, 2024
DLS 5 Week 4 Ex 4 Sequence Models	1	533	December 12, 2021
Question about multi-head attention Sequence Models	2	618	June 25, 2021
Understanding multi-headed attention - C5W4A1 Sequence Models	1	658	May 10, 2022

Question about MultiHeadAttention layer

Related topics