# Multiheaded Attention - Number of heads and Dim of heads

**URL:** <https://community.deeplearning.ai/t/multiheaded-attention-number-of-heads-and-dim-of-heads/317195>\
**Category:** NLP with Attention Models\
**Tags:** week-module-2\
**Created:** [April 28, 2023, 2:18pm UTC](https://community.deeplearning.ai/t/multiheaded-attention-number-of-heads-and-dim-of-heads/317195 "2023-04-28T14:18:18Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![Elemento](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/elemento/32/854_2.png) [@Elemento](https://community.deeplearning.ai/u/Elemento)\
**Post date:** [May 5, 2023, 7:34am UTC](https://community.deeplearning.ai/t/multiheaded-attention-number-of-heads-and-dim-of-heads/317195/4 "2023-05-05T07:34:12Z")

</div>

Hey @nmurugesh and @arvyzukai,  
I spent more than an hour understanding your takes on this, since both seemed to be correct to me 😂 and I guess, both of them are indeed correct as well. What Prof Andrew mentioned in DLS C5 W4 is correct, and what is mentioned in NLP C4 W2 is also correct. It’s just the use of notation that is causing the confusion. I will be referring to one DLS thread [here](https://community.deeplearning.ai/t/w4-a1-is-there-a-typo-in-multi-head-attention-slides/135478/2).

I believe that PyTorch and Tensorflow uses different notation for the same thing. I have borrowed the below image from the reference.

![image](https://global.discourse-cdn.com/dlai/original/3X/b/e/be3b01ec49ced8b2b5944de4c9fa66dd354bbefc.png)

As you can see in W^Q that there is indeed a split in `embedding_dim` for the `n_heads` along the horizontal direction, but there is no split in the vertical direction. So, what @arvyzukai has been mentioning all along is the split along the horizontal direction, and what @nmurugesh has been mentioning all along is the no-split across the vertical direction.

In DLS, Prof Andrew took the matrices for the different heads as W\_i^Q, W\_i^K, W\_i^V, i.e., there were `3 * n_heads` matrices each of dimension `(embedding_dim // n_heads, embedding_dim)`. In other words, there was no split, since the matrices taken were already in their correct shapes.

But in NLP, we have taken them as `3` matrices only, each of dimension `(embedding_dim, embedding_dim)`, and for `axis = 0`, we are considering a logical split so that the first dimension is split into `embedding_dim // n_heads`.

In both the cases, if you will see, after performing concatenation, we will get 3 matrices of dimensions `(embedding_dim, embedding_dim)`. Let me know what you guys think about this.

Cheers,  
Elemento

---

_[View the full topic](https://community.deeplearning.ai/t/multiheaded-attention-number-of-heads-and-dim-of-heads/317195)._