# Confusion about Q, K, and V matrices

**URL:** <https://community.deeplearning.ai/t/confusion-about-q-k-and-v-matrices/426146>\
**Category:** NLP with Attention Models\
**Tags:** week-module-2\
**Created:** [August 23, 2023, 4:36pm UTC](https://community.deeplearning.ai/t/confusion-about-q-k-and-v-matrices/426146 "2023-08-23T16:36:03Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\
**Post date:** [August 24, 2023, 12:48pm UTC](https://community.deeplearning.ai/t/confusion-about-q-k-and-v-matrices/426146/6 "2023-08-24T12:48:55Z")

</div>

No problem, I’m happy to elaborate @Juan_Olano !

I raised my point because I understand these two statements as different:

> [@Juan\_Olano](#):
>
> The Query (Q) is often derived from the word for which you want to calculate the attention score.

vs:

> [@Juan\_Olano](#):
>
> attention mechanism will Query each word against the rest of the words…

In other words, I find the second statement as _true_, while the first one as _confusing_. I wanted to point out that each word in a sequence is “communicating” (my preference as a word of choice, vs., you using the word “querying”) with every other word (including itself).

* * *

> [@Juan\_Olano](#):
>
> Although this may be happing in parallel for each and all embeddings, if we were going to serialize it to try to gain some intuition, the serialized version of it would be one word (or more technically one embedding that comes from a token) versus all other words (embeddings).

If I understand you correctly, then yes, if we go one row (word) at a time in the Attention matrix we find the percentage of how much value from each and every other token in a sequence (from matrix V) to “integrate”.

I find this picture (thanks to Elemento) in [this DLS thread](https://community.deeplearning.ai/t/w4-a1-is-there-a-typo-in-multi-head-attention-slides/135478/2) very concise:

 ![image](https://global.discourse-cdn.com/dlai/original/3X/e/2/e29001946c070aa6878cd7620295d93679116628.jpeg)

In particular, what I mean when saying “one row at a time” is that this:  
 ![image](https://global.discourse-cdn.com/dlai/original/3X/e/0/e00f990274f4ac58cb93b4b589458f0f3a04fef3.png)

results in a square matrix (6 x 6) in this case, and not (1 x 6)… but as you say, we can go line by line in a serialized manner, we can see how much “attention” _this_ word is paying to each other word (including itself).

So the resulting matrix of this head, for example, could be:

![image](https://global.discourse-cdn.com/dlai/original/3X/0/7/07c8aec367af8e25be49cdcb52858f9b077ce8e3.png)

Which, if we interpret as you say in serialized fashion, the first word would accumulate most (79%) from the V of word 2, while words 4, 5 and 6 would mostly equally pay attention to each other and most of its values be average of V^{\<4\>}, V^{\<5\>} and V^{\<6\>} rows.

Even though this picture is about Encoder part (where “Self Attention” is used), @Anthony_Wu was asking about the Decoder (and the “Cross Attention” part) where calculations are similar but Q comes from one sentence and the K and V comes from another (note a not important detail: often even different length, but it must be padded in any case).

But in reality, this matrix is a “single” result for each head (and as a side note, most often attention is concentrated to the tokens in the same position):  
 ![image](https://global.discourse-cdn.com/dlai/original/3X/7/d/7dfaf8e1ed5785361d908b911ba9f60433fcacf6.png)

* * *

Anyways, I think your understanding is correct, but I wanted to point out these points because I found them a bit confusing in your first response.

Cheers

---

_[View the full topic](https://community.deeplearning.ai/t/confusion-about-q-k-and-v-matrices/426146)._