# Padding mask dimmensions **URL:** \ **Category:** Sequence Models\ **Tags:** coursera-platform\ **Created:** [May 3, 2023, 5:44pm UTC](https://community.deeplearning.ai/t/padding-mask-dimmensions/323786 "2023-05-03T17:44:00Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Rodolfo\_Novarini](https://avatars.discourse-cdn.com/v4/letter/r/51bf81/32.png) [@Rodolfo\_Novarini](https://community.deeplearning.ai/u/Rodolfo_Novarini)\ **Post date:** [May 3, 2023, 5:44pm UTC](https://community.deeplearning.ai/t/padding-mask-dimmensions/323786/1 "2023-05-03T17:44:00Z")
Why does the padding mask has one additional dimension? From the description in C5\_W4\_A1\_Transformer\_Subclass\_v1 Section 2.1 it’s not obvious to me that the result needs to have an extra dimension. Can you explain the purpose and how are the values of this extra dimension being calculated? I can see the code but I do not understand what it’s doing: seq = 1 - tf.cast(tf.math.equal(decoder\_token\_ids, 0), tf.float32) seq[:, tf.newaxis, :] Thanks, ---
**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\ **Post date:** [May 4, 2023, 6:36pm UTC](https://community.deeplearning.ai/t/padding-mask-dimmensions/323786/2 "2023-05-04T18:36:59Z")
Thanks for bringing this up. The staff have been notified about this. ---
**Author:** ![Rodolfo\_Novarini](https://avatars.discourse-cdn.com/v4/letter/r/51bf81/32.png) [@Rodolfo\_Novarini](https://community.deeplearning.ai/u/Rodolfo_Novarini)\ **Post date:** [May 4, 2023, 6:53pm UTC](https://community.deeplearning.ai/t/padding-mask-dimmensions/323786/3 "2023-05-04T18:53:44Z")
Not sure this was the intended reply for this topic as I was asking for help in understanding what the code is doing and why. To expand on my question, can you help me understand this example? x = tf.constant([[7., 6., 0., 0., 1.], [1., 2., 3., 0., 0.], [0., 0., 0., 4., 5.]]) print((1-create\_padding\_mask(x))\* -1.0e9) print(x + (1 - create\_padding\_mask(x)) \* -1.0e9) → tf.Tensor( [[[-0.e+00 -0.e+00 -1.e+09 -1.e+09 -0.e+00]] [[-0.e+00 -0.e+00 -0.e+00 -1.e+09 -1.e+09]] [[-1.e+09 -1.e+09 -1.e+09 -0.e+00 -0.e+00]]], shape=(3, 1, 5), dtype=float32) tf.Tensor( [[[7.e+00 6.e+00 -1.e+09 -1.e+09 1.e+00] [1.e+00 2.e+00 -1.e+09 -1.e+09 0.e+00] [0.e+00 0.e+00 -1.e+09 -1.e+09 5.e+00]] [[7.e+00 6.e+00 0.e+00 -1.e+09 -1.e+09] [1.e+00 2.e+00 3.e+00 -1.e+09 -1.e+09] [0.e+00 0.e+00 0.e+00 -1.e+09 -1.e+09]] [[-1.e+09 -1.e+09 -1.e+09 0.e+00 1.e+00] [-1.e+09 -1.e+09 -1.e+09 0.e+00 0.e+00] [-1.e+09 -1.e+09 -1.e+09 4.e+00 5.e+00]]], shape=(3, 3, 5), dtype=float32) Why is the 3x5 tensor being transformed into a 3x3x5 tensor? I can see the original x tensor with -1.e+09 fill where there was a zero before, but what are all these other values? It looks that some kind of casting happens when adding a 3x5 tensor to a 3x3x5 tensor but: 1. I do not understand really what is going on. Help would be appreciated here. 2. I do not understand what is the purpose and the effect of having the output of the mask in this way. ---
**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\ **Post date:** [May 4, 2023, 7:14pm UTC](https://community.deeplearning.ai/t/padding-mask-dimmensions/323786/4 "2023-05-04T19:14:42Z")
Sorry for not being clear. What I meant was that the staff have been notified to update the documentation and the function (if required). For starters, ```auto x = tf.constant([[7., 6., 0., 0., 1.], [1., 2., 3., 0., 0.], [0., 0., 0., 4., 5.]]) ``` doesn’t make sense to me since `0` is usually used as a padding token. This means that 0s should be present at the trailing part of each inner array. The following example is more inline with the transformer architecture: If integer encoded and padded batch looks like this for 2 sentences: ```auto tf.constant([[12, 13, 0, 0, 0], [11, 12, 1, 0, 0]]) ``` the padding mask should be ```auto ``` Please implement `scaled_dot_product_attention` and you’ll see how mask is used. ---
**Author:** ![Rodolfo\_Novarini](https://avatars.discourse-cdn.com/v4/letter/r/51bf81/32.png) [@Rodolfo\_Novarini](https://community.deeplearning.ai/u/Rodolfo_Novarini)\ **Post date:** [May 4, 2023, 9:30pm UTC](https://community.deeplearning.ai/t/padding-mask-dimmensions/323786/5 "2023-05-04T21:30:34Z")
I can see that in scaled\_dot\_product\_attention the mask is: 1. treated to replace ones with zeros and zeros with -1.0e9, using (1. - mask) \* -1.0e9 2. added to scaled\_attention\_logits with shape seq\_len\_q x seq\_len\_k Also, when I run your example through ‘create\_padding\_mask’ I get: tf.Tensor( [[[1. 1. 0. 0. 0.]] [[1. 1. 1. 0. 0.]]], shape=(2, 1, 5), dtype=float32) Instead of the 2x5x5 you show above. And when I run this by ‘x + (1 - create\_padding\_mask(x)) \* -1.0e9’ I get: tf.Tensor( [[[1.2e+01 1.3e+01 -1.0e+09 -1.0e+09 -1.0e+09] [1.1e+01 1.2e+01 -1.0e+09 -1.0e+09 -1.0e+09]] [[1.2e+01 1.3e+01 0.0e+00 -1.0e+09 -1.0e+09] [1.1e+01 1.2e+01 1.0e+00 -1.0e+09 -1.0e+09]]], shape=(2, 2, 5), dtype=float32) But I still don’t get why are we adding an additional dimension and how works the addition of two tensors of (2,1,5) and (2, 5) shapes. ---
**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\ **Post date:** [May 4, 2023, 10:46pm UTC](https://community.deeplearning.ai/t/padding-mask-dimmensions/323786/6 "2023-05-04T22:46:25Z")
The example provided is regarding the suggested behavior and not the current notebook behavior. As far as adding dissimilar shapes is concerned, please see this link on [broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html).