# C5W4 Transformers Assignment/MultiHeadAttention & Concern About Q,K and V dimensions

**URL:** <https://community.deeplearning.ai/t/c5w4-transformers-assignment-multiheadattention-concern-about-q-k-and-v-dimensions/31169>\
**Category:** Sequence Models\
**Tags:** coursera-platform\
**Created:** [August 25, 2021, 1:35pm UTC](https://community.deeplearning.ai/t/c5w4-transformers-assignment-multiheadattention-concern-about-q-k-and-v-dimensions/31169 "2021-08-25T13:35:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abboura](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/abboura/32/430_2.png) [@Abboura](https://community.deeplearning.ai/u/Abboura)\
**Post date:** [August 25, 2021, 1:35pm UTC](https://community.deeplearning.ai/t/c5w4-transformers-assignment-multiheadattention-concern-about-q-k-and-v-dimensions/31169/1 "2021-08-25T13:35:48Z")

</div>

Hello,  
**First,** I liked a lot, the last assignment of the new added module 4, about Transformers, I (maybe the best in the specialization), especially when working with the classes (…): Really Excellent!  
But,  
**I have a concern about Q, K and V:**  
First, since in the Assignment, we were not invited to implement the MultiHeadAttention (I would have liked to see a part of the assignment to implement the MultiHeadAttention (by ourselves), because it is the core of transformers: this is where the magic happens (the parallelization of computations …) , I searched my self to understand how the parallelization is done: Splitting Q,K and V…(I ended up by understanding it.

**Second** , by examining the function  
**scaled\_dot\_product\_attention(q, k, v, mask)** of the Assignment: I found like a **contradiction**

- In fact, by examining the computations (within this function of the assignment)  
we should have this equality so that the computations work in all cases  
**seq\_len\_k = seq\_len\_v** , because to do the multiplication between attention\_weights and v, **it won’t be possible if seq\_len\_k is different from seq\_len\_v**  
==\> so,only two arguments **seq\_len\_k\_v** and **seq\_len\_q** are enough (not 3 different params)

- Then, By examining the MultiHeadAttention implementation(by Searching…), we can immediately realize that the parallelization of the computations (which characterizes the transformers), was possible “roughly” by using the two key functions: tf.reshape, and tf.transpose, and computing their **last\_dimension(depth)= original\_depth\_before\_splitting //number\_of\_heads**…) and it turns out that **Q, K and V have the same dimensions**. In other words: the same depth, and the same seqence\_length (that said **depth\_v** (in our case, in the function scaled\_dot\_product\_attention(q, k, v, mask) ) should not be different from **depth** =\> we can therefore merge them into a single depth parameter “depth”(not two params), similarly for the length of sequences: we can merge them into a single parameter: **seq\_len** (not 3 different parameters, **which gives the impression that they can be different** ). If I am wrong please enlighten me.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [April 21, 2022, 3:33am UTC](https://community.deeplearning.ai/t/c5w4-transformers-assignment-multiheadattention-concern-about-q-k-and-v-dimensions/31169/2 "2022-04-21T03:33:10Z")

</div>

I hope that you were able to find the answer to your question.
