# Sequence length, d\_head variable

**URL:** <https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762>\
**Category:** NLP with Attention Models\
**Tags:** week-module-2\
**Created:** [April 27, 2023, 1:57pm UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762 "2023-04-27T13:57:52Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fei\_Li](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@Fei\_Li](https://community.deeplearning.ai/u/Fei_Li)\
**Post date:** [April 27, 2023, 1:57pm UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762/1 "2023-04-27T13:57:52Z")

</div>

Hello, I am trying to understand the sequence length variable from the assignment

 ![image](https://global.discourse-cdn.com/dlai/original/3X/1/3/133c69cb044721d1eeb473505a20cd98f97eacf9.png)

I reviewed the reading on “Multihead Attention”

I am guessing sequence length is how many words are in a sentence. if I use the picture from the reading, that would be n, if I have word1, word2, … word n

 ![image](https://global.discourse-cdn.com/dlai/original/3X/5/3/53ab2beb647e5070272e11f1b0578791ba2fb264.png)

I am also not quite sure about the d\_head variable in my coding pic (1st pic), is it the number of columns for Z in the pic below? which largely depends on the Wo?

 ![image](https://global.discourse-cdn.com/dlai/original/3X/9/1/91f8be17deb3ba42f6eb0008ddc5235032bb74b8.png)

---

<div class="post-metadata">

**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\
**Post date:** [April 27, 2023, 5:20pm UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762/2 "2023-04-27T17:20:13Z")

</div>

Hi @Fei_Li

> [@Fei\_Li](#):
>
> I am guessing sequence length is how many words are in a sentence. if I use the picture from the reading, that would be n, if I have word1, word2, … word n

Yes. In the picture the sequence length is 2 it would be padded (to like 32) in a more real scenario.

> [@Fei\_Li](#):
>
> I am also not quite sure about the d\_head variable in my coding pic (1st pic), is it the number of columns for Z in the pic below? which largely depends on the Wo?

Yes, the `d_head` highly depends on Wo or more generally on embedding dimension - if the embedding dim is 32 and there are 4 heads, then the `d_head` would be 8.

Each head “operates” in its own vector of 8 numbers, then these numbers of each head are concatenated - back to vector of 32. The subsequent linear transformation Wo is applied on to it to get Z.

---

<div class="post-metadata">

**Author:** ![Fei\_Li](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@Fei\_Li](https://community.deeplearning.ai/u/Fei_Li)\
**Post date:** [April 28, 2023, 2:40am UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762/3 "2023-04-28T02:40:11Z")

</div>

Sure, thank you. Based on your explanation, I also want to make sure that I understand “d\_model” in multi-head attention. Appreciate your help.

In the picture [Z1 Z2 …Zn] \*Wo=Z, there are n heads, right? Then say we have a sequence of words, we apply a different set of WQ, WK, and WV to get a different head.

So if I use your example, that would be for each word we use 4 heads, and for each head, we use 8 numbers, which is d\_head. This 8, should be d\_model if we don’t use multi-head right? That is we use 8 numbers to represent a word before the multi-head mechanism.  
But now, since we are using the multi-head mechanism to represent a word, now each word is represented by 32 numbers(our new d\_model), which comes from 4 heads, each of them having 8 numbers.

---

<div class="post-metadata">

**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\
**Post date:** [April 28, 2023, 7:09am UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762/4 "2023-04-28T07:09:32Z")

</div>

> [@Fei\_Li](#):
>
> This 8, should be d\_model if we don’t use multi-head right?

If I understand you correctly, then this number would be 32. In other words,

- if the embedding dimension is 32 and we have 1 head, then the head dimension is 32;
- if the embedding dimension is 32 and we have 4 heads, then the head dimensions is 8;

* * *

To illustrate what happens with more concrete steps (this time with embedding dimension 16):

For example, input is “word1 word2” → tokenize(input) → [2, 54] → Embed([2, 54]) → 2D matrix of shape (2, 16):

```auto
[[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
 [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]]

```

\*Note: the values here are just _indexes_, as the real values would be more normal (like 0.23, -0.10 etc.)

→ get\_Q()=Embed([2, 54]) \cdot W\_q → Q :

```auto
[[0,1,2,3,|4,5,6,7,|8,9,10,11,|12,13,14,15],
 [0,1,2,3,|4,5,6,7,|8,9,10,11,|12,13,14,15]]

```

→ get\_K()=Embed([2, 54]) \cdot W\_k → K:

```auto
[[0,1,2,3,|4,5,6,7,|8,9,10,11,|12,13,14,15],
 [0,1,2,3,|4,5,6,7,|8,9,10,11,|12,13,14,15]]

```

→ get\_V()=Embed([2, 54]) \cdot W\_v → V:

```auto
[[0,1,2,3,|4,5,6,7,|8,9,10,11,|12,13,14,15],
 [0,1,2,3,|4,5,6,7,|8,9,10,11,|12,13,14,15]]

```

\*Note: “|” is just for illustration purpose, in reality it would be a simple matrix as before _but_ later split into a 3D tensor (for each head according to these “|”) (in this week’s assignment, these splits (crazy 🙂 reshapes and transposes) happen in `compute_attention_heads_closure`. So, “|” indicates each head’s “working” dimensions or splits (4 numbers here, 8 in the previous post, 3 in your pictures).

Now when you get attention weights (by softmax(Q \cdot K^T) you can represent them like a 3D tensor:

```auto
[
 [[0,1], # Q_0_0 @ K_1_0^T, Q_0_0 @ K_0_0^T
  [0,1]], # Q_1_0 @ K_1_0^T, Q_1_0 @ K_0_0^T

 [[0,1],
  [0,1]], 

 [[0,1],
  [0,1]], 

 [[0,1],
  [0,1]], 
]

```

When you apply this attention on V (attention \cdot V) the output is Z\_1, Z\_2, Z\_3, Z\_4 which after concatenation (again “crazy” reshapes and transposes in `compute_attention_output_closure`) becomes a 2D matrix of shape (2, 16) (like in your bottom picture). Also note that order of these attention dimensions do not match Assignment but are just for illustration.

---

<div class="post-metadata">

**Author:** ![Fei\_Li](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@Fei\_Li](https://community.deeplearning.ai/u/Fei_Li)\
**Post date:** [April 28, 2023, 12:44pm UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762/5 "2023-04-28T12:44:49Z")

</div>

Thank you very much for your detailed explanation.

I guess I am getting there.

> [@arvyzukai](#):
>
> (this time with embedding dimension 16)

then this time, d\_model is 16.

> [@arvyzukai](#):
>
> tokenize(input) → [2, 54] → Embed([2, 54])

this “54” is how many numbers we use to represent a word which we call embedding size?

I am getting there, right? 🤔

---

<div class="post-metadata">

**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\
**Post date:** [April 28, 2023, 12:50pm UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762/6 "2023-04-28T12:50:39Z")

</div>

> [@Fei\_Li](#):
>
> then this time, d\_model is 16.

Yes

> [@Fei\_Li](#):
>
> this “54” is how many numbers we use to represent a word which we call embedding size?

No, this 54 is a random integer value that I thought for “word2”. In other words “word2” tokenize(“word2”) → 54, tokenize “word1” → 2

> [@Fei\_Li](#):
>
> I am getting there, right? 🤔

I think so 😉

---

<div class="post-metadata">

**Author:** ![Fei\_Li](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@Fei\_Li](https://community.deeplearning.ai/u/Fei_Li)\
**Post date:** [April 28, 2023, 12:53pm UTC](https://community.deeplearning.ai/t/sequence-length-d-head-variable/315762/7 "2023-04-28T12:53:53Z")

</div>

> [@Fei\_Li](#):
>
> this “54” is how many numbers we use to represent a word which we call embedding size?

Oh, okay okay.

Thank you very very much for your help. 👍