# Scaled\_dot\_product\_attention q, k, and v dimensions not correct

**URL:** <https://community.deeplearning.ai/t/scaled-dot-product-attention-q-k-and-v-dimensions-not-correct/391801>\
**Category:** Sequence Models\
**Tags:** coursera-platform\
**Created:** [July 20, 2023, 6:54pm UTC](https://community.deeplearning.ai/t/scaled-dot-product-attention-q-k-and-v-dimensions-not-correct/391801 "2023-07-20T18:54:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cjcabit](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcabit](https://community.deeplearning.ai/u/cjcabit)\
**Post date:** [July 20, 2023, 6:54pm UTC](https://community.deeplearning.ai/t/scaled-dot-product-attention-q-k-and-v-dimensions-not-correct/391801/1 "2023-07-20T18:54:28Z")

</div>

I’m not sure the scaled\_dot\_product\_attention function can be done correctly.  
q, k and v must have matching leading dimensions and k and v must have matching penultimate dimensions. That’s the only way to complete this exercise. Well, they simply don’t.  
q shape (3, 4)  
k shape (4, 4)  
v shape (4, 2)  
At no point is there any indication that I am meant to manipulate the dimensions of q, k, or v in any way other than transposing them.  
qk can only be shape (3, 4) or shape (4, 3)  
Therefore qk times v can only be shape (3, 2) or shape (2, 3)  
The required output shape is (3, 4)  
On all previous functions, I passed. Where’s my mistake? I’m really not sure what I’m missing.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [July 20, 2023, 7:14pm UTC](https://community.deeplearning.ai/t/scaled-dot-product-attention-q-k-and-v-dimensions-not-correct/391801/2 "2023-07-20T19:14:49Z")

</div>

> [@cjcabit](#):
>
> q, k and v must have matching leading dimensions and k and v must have matching penultimate dimensions.

You’ve found an error in the doctext. None of that is true. I’ll submit a ticket to fix the issue.

If you look at the figure in the instructions, you need to compute Q times the transpose of K.

---

<div class="post-metadata">

**Author:** ![cjcabit](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcabit](https://community.deeplearning.ai/u/cjcabit)\
**Post date:** [July 20, 2023, 7:22pm UTC](https://community.deeplearning.ai/t/scaled-dot-product-attention-q-k-and-v-dimensions-not-correct/391801/3 "2023-07-20T19:22:12Z")

</div>

tf.matmul(q, tf.transpose(k)) Ends up being shape (3, 4) and v remains shape (4, 2).  
Meaning the only output I can get is (3, 2) or the transpose of that. The output needs to be shape (3, 4). I’m still confused as to what mistake I am making.

---

<div class="post-metadata">

**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\
**Post date:** [July 20, 2023, 7:35pm UTC](https://community.deeplearning.ai/t/scaled-dot-product-attention-q-k-and-v-dimensions-not-correct/391801/5 "2023-07-20T19:35:25Z")

</div>

> [@cjcabit](#):
>
> The output needs to be shape (3, 4).

Also, this is not true.

---

<div class="post-metadata">

**Author:** ![saifkhanengr](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/saifkhanengr/32/152858_2.png) [@saifkhanengr](https://community.deeplearning.ai/u/saifkhanengr)\
**Post date:** [July 21, 2023, 5:29am UTC](https://community.deeplearning.ai/t/scaled-dot-product-attention-q-k-and-v-dimensions-not-correct/391801/6 "2023-07-21T05:29:05Z")

</div>

> [@cjcabit](#):
>
> tf.matmul(q, tf.transpose(k)) Ends up being shape (3, 4) and v remains shape (4, 2).

Please note that we do the `matmul` between `attention_weights` and `v`, not the `matmul_qk` and `v`. We get `attention_weights` after applying the softmax and then multiply the `v`. Below is the output of my notebook:

```auto
q shape: (3, 4)
k shape: (4, 4)
v shape: (4, 2)
matmul_qk shape: (3, 4)
attention_weights shape: (3, 4)
output shape: (3, 2)
q shape: (3, 4)
k shape: (4, 4)
v shape: (4, 2)
matmul_qk shape: (3, 4)
attention_weights shape: (1, 3, 4)
output shape: (1, 3, 2)
All tests passed

```