# W4, Exercise 3 - scaled\_dot\_product\_attention: what does the dimension of q, k, v mean?

**URL:** <https://community.deeplearning.ai/t/w4-exercise-3-scaled-dot-product-attention-what-does-the-dimension-of-q-k-v-mean/48878>\
**Category:** Sequence Models\
**Tags:** coursera-platform\
**Created:** [October 12, 2021, 4:15am UTC](https://community.deeplearning.ai/t/w4-exercise-3-scaled-dot-product-attention-what-does-the-dimension-of-q-k-v-mean/48878 "2021-10-12T04:15:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![larryleguo](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/larryleguo/32/125983_2.png) [@larryleguo](https://community.deeplearning.ai/u/larryleguo)\
**Post date:** [October 12, 2021, 4:15am UTC](https://community.deeplearning.ai/t/w4-exercise-3-scaled-dot-product-attention-what-does-the-dimension-of-q-k-v-mean/48878/1 "2021-10-12T04:15:57Z")

</div>

I get the code done and passed, but I still do not understand the dimensions of q, k, v:  
q – query shape == (…, seq\_len\_q, depth)  
k – key shape == (…, seq\_len\_k, depth)  
v – value shape == (…, seq\_len\_v, depth\_v)

1. What is the depth and depth\_v?
2. Is seq\_len\_k the number of words in the sentence?
3. Is seq\_len\_q the number of questions to ask? How is it selected? Is it a super parameter to tune?
4. Does the … represent the folds pf multi-head computations?

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [April 21, 2022, 3:35am UTC](https://community.deeplearning.ai/t/w4-exercise-3-scaled-dot-product-attention-what-does-the-dimension-of-q-k-v-mean/48878/2 "2022-04-21T03:35:38Z")

</div>

Were you able to find answers to your questions?

---

<div class="post-metadata">

**Author:** ![Jason\_Li1](https://avatars.discourse-cdn.com/v4/letter/j/8491ac/32.png) [@Jason\_Li1](https://community.deeplearning.ai/u/Jason_Li1)\
**Post date:** [June 21, 2022, 11:19pm UTC](https://community.deeplearning.ai/t/w4-exercise-3-scaled-dot-product-attention-what-does-the-dimension-of-q-k-v-mean/48878/3 "2022-06-21T23:19:57Z")

</div>

Could a mentor shed some light on this question? Having the same issue myself…

---

<div class="post-metadata">

**Author:** ![anon57530071](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@anon57530071](https://community.deeplearning.ai/u/anon57530071)\
**Post date:** [June 22, 2022, 12:29am UTC](https://community.deeplearning.ai/t/w4-exercise-3-scaled-dot-product-attention-what-does-the-dimension-of-q-k-v-mean/48878/4 "2022-06-22T00:29:33Z")

</div>

Here is an overview of an Encoder process.

 ![image](https://global.discourse-cdn.com/dlai/original/3X/6/0/6003db70d4a7fb6bcd0ce55e91d6a5c4e15bf13a.jpeg)

You can see the process for multi-head attention including scaled dot product attention at the center of this diagram.

Most important variable is “embedding\_dim” which is equal to “model dimension”, d. Embedding, positional\_encoding, fully connected layers and multi-head attention layers are using this value as one of input/output shape of data. (Another key variable is “seq\_len”)  
(As of today, Jun 22, there are incorrect description about “embedded\_dim”. For example, the shape of “3rd parameter for encoder\_layer\_out” is described as (batch\_size, input\_seq\_len, fully\_connected\_dim). But, the last one should be “embedding\_dim”.)

“query size”, d\_k which is “depth” in this assignment is calculated as follows.

d\_k = \frac{d\_{model}}{h} = \frac{embedding\_dim}{num\_heads}, \ \ \ \ \ num\_heads: number of heads for multi-head attention layer

Answers to questions:

1. depth and depth\_v are “query size” which is calculated by the above equation.
2. seq\_len\_q, seq\_len\_k, and seq\_len\_v are essentially same, and are equal to seq\_len. seq\_len is “number of words” + padding.
3. seq\_len\_q is equal to seq\_len, and is “number of words”+padding. If you look at Q, each row of “seq\_len” is represented as q^{\<i\>}. In this sense, it can be also said that “questions to ask”.
4. “…” is same as other input that we saw in the past exercises, I suppose. In my chart above, it is for a single sample (sentence), which includes multiple words. But, of course, the system can accept multiple samples at one time. I suppose this represents “number of input sentences”.

As you see, this “scaled\_dot\_product\_attention” is not used in this exercise, since it is included in Keras MultiHeadAttension. That makes learners difficult to understand input/output parameters.

Hope this helps.