# C5W4 Transformer Network Exercise 3 - scaled\_dot\_product\_attention **URL:** \ **Category:** Sequence Models\ **Tags:** week-module-4, coursera-platform\ **Created:** [March 4, 2024, 4:41pm UTC](https://community.deeplearning.ai/t/c5w4-transformer-network-exercise-3-scaled-dot-product-attention/583562 "2024-03-04T16:41:42Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Kejun\_Zhang](https://avatars.discourse-cdn.com/v4/letter/k/f6c823/32.png) [@Kejun\_Zhang](https://community.deeplearning.ai/u/Kejun_Zhang)\ **Post date:** [March 4, 2024, 4:41pm UTC](https://community.deeplearning.ai/t/c5w4-transformer-network-exercise-3-scaled-dot-product-attention/583562/1 "2024-03-04T16:41:42Z")
I got this error message as below. I also print out the dimension of q, k , v. The equation for attention matrics is softmax(q\*K.T) and dot product with v. If the first part of dimension is (3,4), and v’s dimension is (4,2), the dimension of the dot product should be (3,2). Why the expected dimension is (3,4)? ## q shape [3, 4] k shape [4, 4] v shape [4, 2] None matmul\_qk shape [3, 4] dk 4 tf.Tensor( [[0.2589478 0.42693272 0.15705977 0.15705977] [0.2772748 0.2772748 0.2772748 0.16817567] [0.33620113 0.33620113 0.12368149 0.2039163]], shape=(3, 4), dtype=float32) AssertionError Traceback (most recent call last) in 1 # UNIT TEST ----\> 2 scaled\_dot\_product\_attention\_test(scaled\_dot\_product\_attention) ~/work/W4A1/public\_tests.py in scaled\_dot\_product\_attention\_test(target) 57 attention, weights = target(q, k, v, None) 58 assert tf.is\_tensor(weights), “Weights must be a tensor” —\> 59 assert tuple(tf.shape(weights).numpy()) == (q.shape[0], k.shape[1]), f"Wrong shape. We expected ({q.shape[0]}, {k.shape[1]})" 60 assert np.allclose(weights, [[0.2589478, 0.42693272, 0.15705977, 0.15705977], 61 [0.2772748, 0.2772748, 0.2772748, 0.16817567], AssertionError: Wrong shape. We expected (3, 4) ---
**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\ **Post date:** [March 4, 2024, 7:54pm UTC](https://community.deeplearning.ai/t/c5w4-transformer-network-exercise-3-scaled-dot-product-attention/583562/2 "2024-03-04T19:54:40Z")
Perhaps your output shape is incorrect. It should be (3,2) for the first test, and (1, 3,2) for the second test. ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [March 5, 2024, 5:07am UTC](https://community.deeplearning.ai/t/c5w4-transformer-network-exercise-3-scaled-dot-product-attention/583562/3 "2024-03-05T05:07:14Z")
> [@Kejun\_Zhang](#): > > softmax(q\*K.T) That’s probably where your problems originate. That’s not what the math formula says. When Prof Ng writes QK^T that does not mean elementwise multiply (\*), right? When Prof Ng omits the operator, it means dot product style matrix multiply. ---
**Author:** ![Kejun\_Zhang](https://avatars.discourse-cdn.com/v4/letter/k/f6c823/32.png) [@Kejun\_Zhang](https://community.deeplearning.ai/u/Kejun_Zhang)\ **Post date:** [March 5, 2024, 3:37pm UTC](https://community.deeplearning.ai/t/c5w4-transformer-network-exercise-3-scaled-dot-product-attention/583562/4 "2024-03-05T15:37:13Z")
Thanks. I realized that I had erroneously equated the attention\_weight to output. After fixing that, the error was resolved.