# Intuition about the application of padding masks and look-ahead masks in Transformer's encoder/decoder **URL:** \ **Category:** Sequence Models\ **Tags:** coursera-platform\ **Created:** [August 26, 2021, 5:00pm UTC](https://community.deeplearning.ai/t/intuition-about-the-application-of-padding-masks-and-look-ahead-masks-in-transformers-encoder-decoder/31413 "2021-08-26T17:00:16Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![LuBinLiu](https://avatars.discourse-cdn.com/v4/letter/l/6de8d8/32.png) [@LuBinLiu](https://community.deeplearning.ai/u/LuBinLiu)\ **Post date:** [August 26, 2021, 5:00pm UTC](https://community.deeplearning.ai/t/intuition-about-the-application-of-padding-masks-and-look-ahead-masks-in-transformers-encoder-decoder/31413/1 "2021-08-26T17:00:16Z")
From the [Tensorflow tutorial](https://www.tensorflow.org/text/tutorials/transformer), the shape of the padding mask is `(batch_size, 1, 1, seq_len)` and look-ahead mask is `(batch_size, 1, seq_len, seq_len)` which fed into `scaled_dot_product` function along with q,v,k. For input X with shape `(batch_size, seq_len)` of integer tokens that are then converted to an embedding, the resulting Tensor, let’s call X' has shape `(batch_size, seq_len, d_model)`. In the call method of `MultiHeadAttention` class of the tutorial we have: ``` def call(self, v, k, q, mask): batch_size = tf.shape(q)[0] q = self.wq(q) # (batch_size, seq_len, d_model) k = self.wk(k) # (batch_size, seq_len, d_model) v = self.wv(v) # (batch_size, seq_len, d_model) q = self.split_heads(q, batch_size) # (batch_size, num_heads, seq_len_q, depth) k = self.split_heads(k, batch_size) # (batch_size, num_heads, seq_len_k, depth) v = self.split_heads(v, batch_size) # (batch_size, num_heads, seq_len_v, depth) scaled_attention, attention_weights = scaled_dot_product_attention( q, k, v, mask) .... ``` If q=v=k=X' are input into the above `call`, then input into `scaled_dot_product` in the last line would be tensors of shape `(batch_size, num_head, seq_len, depth )`.Then, in the first line of `scaled_dot_product`: ``` matmul_qk = tf.matmul(q, k, transpose_b=True) ``` would result in `matmul_qk` having a shape of `(batch_size, num_head, seq_len, seq_len )`, and this Tensor then has the masked applied via: ``` if mask is not None: scaled_attention_logits += (mask * -1e9) ``` I understand that the shape of the masks (via broadcasting) matches that of the shape of `matmul_qk` above, but after all of the transformations of the original X, I’m having a hard time visualizing how the padding and look-ahead masks are doing what they are intended to do. For instance, how is the `(batch_size, 1, 1, seq_len)` padding mask created based on padded `0`s in the original input X of integers tokens end up masking padded values in `matmul_qk`? ---
**Author:** ![edwardyu](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/edwardyu/32/82_2.png) [@edwardyu](https://community.deeplearning.ai/u/edwardyu)\ **Post date:** [September 2, 2021, 8:43am UTC](https://community.deeplearning.ai/t/intuition-about-the-application-of-padding-masks-and-look-ahead-masks-in-transformers-encoder-decoder/31413/2 "2021-09-02T08:43:01Z")
Hi @LuBinLiu , To make it easy to understand, let’s simplify the problem, say seq\_len\_q = 5, seq\_len\_k = seq\_len\_v = 6, batch\_size = num\_head = 1, and the length of example sentence is 4, e.g., _[This] [is] [source] [seq] [pad] [pad]_. **Note** : The mask value in the [tutorial](https://www.tensorflow.org/text/tutorials/transformer#masking), 1 means padding, which is different from [MultiHeadAttention](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MultiHeadAttention#args_1) api. So the masking operation is as below: ![image](https://global.discourse-cdn.com/dlai/original/2X/7/74d3e454451b999d3f4b8f56026247bb2935673e.png) After taking softmax, it becomes: ![image](https://global.discourse-cdn.com/dlai/original/2X/5/51b955d3226ca77e857a38b7abf93365373f66af.png) Pij represents how much Qi pays attention to Kj. You can see the attention weights of _[pad]_ are zero. ---
**Author:** ![LuBinLiu](https://avatars.discourse-cdn.com/v4/letter/l/6de8d8/32.png) [@LuBinLiu](https://community.deeplearning.ai/u/LuBinLiu)\ **Post date:** [September 2, 2021, 4:03pm UTC](https://community.deeplearning.ai/t/intuition-about-the-application-of-padding-masks-and-look-ahead-masks-in-transformers-encoder-decoder/31413/3 "2021-09-02T16:03:30Z")
So the last dimension (which has dimension seq\_len\_k = length of each sentence) can be interpreted as the sentence dimension? ---
**Author:** ![edwardyu](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/edwardyu/32/82_2.png) [@edwardyu](https://community.deeplearning.ai/u/edwardyu)\ **Post date:** [September 3, 2021, 1:58am UTC](https://community.deeplearning.ai/t/intuition-about-the-application-of-padding-masks-and-look-ahead-masks-in-transformers-encoder-decoder/31413/4 "2021-09-03T01:58:23Z")
I would say that _seq\_len\_k_ is the sentence length you want to pay attention to. For instance, the 2nd MultiHeadAttention layer, _mha2_ in the Transformer DecoderLayer, _seq\_len\_k = (encoder sentence length)_, and _seq\_len\_q = (decoder sentence length)_. Its mask is created based on the encoder sentence. Take a look at _create\_masks_ function in [Transformer class](https://www.tensorflow.org/text/tutorials/transformer#create_the_transformer) in the tutorial, you will see dec\_padding\_mask is same as enc\_padding\_mask, both are created based on the encoder input sentence. It’s because enc\_padding\_mask is for encoder self-attention to pay attention to itself, but dec\_padding\_mask is for decoder to pay attention to encoder.